FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Build System

 
 
LinkBack Thread Tools
 
Old 08-31-2010, 03:56 PM
Toshio Kuratomi
 
Default email notifications

On Tue, Aug 31, 2010 at 09:54:07AM +0200, Florian La Roche wrote:
> Hello Toshio,
>
> koji already has a generic routing called fixEncoding() and I am currently
> running koji with the patch from https://bugzilla.redhat.com/show_bug.cgi?id=614767.
>
> Maybe your below code to use "errors='replace'" should go into the
> generic koji function? (Haven't looked into this myself if it makes sense...)
>
Took a look -- fixEncoding() is almost as robust as my code[1]_. I think I
prefer my code but the result of either transformation will be mangled byte
strings. Basically we're being handed byte strings without a guarantee that
the strings are utf8. So we first assume that they are utf8. If not, both
pieces of code try someway to coerce the output to utf8. My code substitutes a
replacement character for every illegal byte. So a string could end up looking
like this 'café' (in some non-utf8 encoding): 'caf�'.

The fixEncoding() code fallsback to an encoding that covers the whole
spectrum of possible bytes. So the output will substitute a random
character (or characters if the invalid characters are a multi-byte
sequence) in place of the unknown bytes.

It probably makes sense to use fixEncoding() everywhere as the patch in your
bug report does. It could make sense to change fixEncoding to incorporate
ideas from my code. The result would look like this:

import warnings
def fixEncoding(value, from_encoding=None, fallback=None):
# fallback is used for backwards compatibility
if not from_encoding:
if fallback:
warnings.warn('fixEncoding() no longer takes a fallback'
' keyword arg. Use from_encoding instead.',
DeprecationWarning, stacklevel=2)
from_encoding = fallback
else:
from_encoding = 'utf8'

if isinstance(value, unicode):
# value is already unicode, so just convert it
# to a utf8-encoded str
# Note: with python3, this can fail unless you use an error
# argument because a unicode string could have been created using
# the surrogateescape error handler.
return value.encode('utf8', 'replace')
else:
# value is a str but may not be valid utf8 (encoded in latin1, for
# instance). Note that the string is almost certain to be mangled
# in these instances unless you know what encoding the string is in
# and have set from_encoding to that encoding.
return value.decode(from_encoding, 'replace').encode('utf8', 'replace')

[1]: fixEncoding's ``if not value: return value`` statement will lead to
unicode errors in circumstances like this::

monkeywrench = fixEncoding(u')
name = fixEncoding(u'café')
combined = monkeywrench + name

The empty string in monkeywrench won't be converted into a byte string.
When you then try to combine the unicode string in monkeywrench with the
byte string in name python will try to convert name into a unicode string
with the ascii codec and fail.

-Toshio

> regards,
>
> Florian La Roche
>
>
>
> > dgilmore says that we haven't applied any changes to the koji running in
> > Fedora Infrastructure but that he needs to pull in a patch to hotfix our
> > instance with. Since I didn't see a patch in the upstream git repository,
> > I'm attaching one here.
> >
> > -Toshio
>
> > From a3c0248a10c5ddbe9a34f76b995fbf10c34e87f5 Mon Sep 17 00:00:00 2001
> > From: Toshio Kuratomi <toshio@fedoraproject.org>
> > Date: Fri, 20 Aug 2010 11:17:43 -0400
> > Subject: [PATCH] Fix Unicode traceback with notification email.
> >
> > ---
> > builder/kojid | 5 ++++-
> > 1 files changed, 4 insertions(+), 1 deletions(-)
> >
> > diff --git a/builder/kojid b/builder/kojid
> > index 2f7f480..77ded36 100755
> > --- a/builder/kojid
> > +++ b/builder/kojid
> > @@ -3455,7 +3455,10 @@ Status: %(status)s
> >
> > message = self.message_templ % locals()
> > # ensure message is in UTF-8
> > - message = message.encode('utf-8')
> > + if isinstance(message, unicode):
> > + message = message.encode('utf-8')
> > + else:
> > + message = unicode(message, encoding='utf8', errors='replace').encode('utf8')
> >
> > server = smtplib.SMTP(options.smtphost)
> > #server.set_debuglevel(True)
> > --
> > 1.7.2.1
> --
> buildsys mailing list
> buildsys@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/buildsys
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 
Old 10-05-2010, 04:01 PM
Ville Skytt
 
Default email notifications

On Friday 20 August 2010, Toshio Kuratomi wrote:
> On Fri, Aug 20, 2010 at 05:06:33PM +0300, Ville Skytt wrote:
> > On Friday 13 August 2010, Toshio Kuratomi wrote:
> > > On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skytt wrote:
> > > > Ping? Still no notification mails for me, for example the
> > > > modplugtools builds done yesterday, and I haven't seen any comments
> > > > when this might get fixed.
> > >
> > > I know practically nothing about the koji code but that code seems very
> > > wrong. This is earlier in the class:: [...]
> >
> > Thanks for taking the time to look into this. Maybe some changes have
> > been made already - at least I received a build failure mail from today
> > for a failed javasqlite build. I didn't receive one for example for
> > successful seeker and rpmlint builds yesterday though
>
> dgilmore says that we haven't applied any changes to the koji running in
> Fedora Infrastructure but that he needs to pull in a patch to hotfix our
> instance with. Since I didn't see a patch in the upstream git repository,
> I'm attaching one here.

Thanks again. I'm still not receiving any mails for successful builds,
hopefully someone can do something about that soon.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys
 

Thread Tools




All times are GMT. The time now is 08:20 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org