email notifications
 

Hallo,

With koji-1.4 running on RHEL6-beta2, some email notifications
are not sent out correctly:

Traceback (most recent call last):
File "/usr/sbin/kojid", line 1437, in runTask
response = (handler.run(),)
File "/usr/sbin/kojid", line 1513, in run
return self.handler(*self.params,**self.opts)
File "/usr/sbin/kojid", line 3649, in handler
message = message.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)

Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
cronie, amanda, akonadi.

regards,

Florian La Roche

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

> Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> cronie, amanda, akonadi.


The following change seems to work (tested with ctags and crontabs):

--- builder/kojid
+++ builder/kojid
@@ -3455,7 +3455,7 @@ Status: %(status)s

message = self.message_templ % locals()
# ensure message is in UTF-8
- message = message.encode('utf-8')
+ message = koji.fixEncoding(message)

server = smtplib.SMTP(options.smtphost)
#server.set_debuglevel(True)
@@ -3646,7 +3646,7 @@ Build Info: %(weburl)s/buildinfo?buildID
subject = self.subject_templ % locals()
message = self.message_templ % locals()
# ensure message is in UTF-8
- message = message.encode('utf-8')
+ message = koji.fixEncoding(message)

server = smtplib.SMTP(options.smtphost)
# server.set_debuglevel(True)

I'll put this into a bugzilla and also upload my current
koji rpm to http://jur-linux.org/rpms/el-updates/6/ again.

regards,

Florian La Roche

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> Hallo,
>
> With koji-1.4 running on RHEL6-beta2, some email notifications
> are not sent out correctly:
>
> Traceback (most recent call last):
> File "/usr/sbin/kojid", line 1437, in runTask
> response = (handler.run(),)
> File "/usr/sbin/kojid", line 1513, in run
> return self.handler(*self.params,**self.opts)
> File "/usr/sbin/kojid", line 3649, in handler
> message = message.encode('utf-8')
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
>
> Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> cronie, amanda, akonadi.

And that's the reason why I didn't received any notification from Fedora
Koji about successful builds ...


Dan


--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Wednesday 21 July 2010, Dan Horák wrote:
> Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> > Hallo,
> >
> > With koji-1.4 running on RHEL6-beta2, some email notifications
> > are not sent out correctly:
> >
> > Traceback (most recent call last):
> > File "/usr/sbin/kojid", line 1437, in runTask
> >
> > response = (handler.run(),)
> >
> > File "/usr/sbin/kojid", line 1513, in run
> >
> > return self.handler(*self.params,**self.opts)
> >
> > File "/usr/sbin/kojid", line 3649, in handler
> >
> > message = message.encode('utf-8')
> >
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546:
> > ordinal not in range(128)
> >
> > Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> > cronie, amanda, akonadi.
>
> And that's the reason why I didn't received any notification from Fedora
> Koji about successful builds ...

Neither have I, and not getting those mails has already started to interfere
with my workflow which I'd rather not change... when can we expect a fix for
this in the Fedora koji instance?
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Wed, Jul 21, 2010 at 04:34:11PM +0200, Dan Horák wrote:
> Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> > Hallo,
> >
> > With koji-1.4 running on RHEL6-beta2, some email notifications
> > are not sent out correctly:
> >
> > Traceback (most recent call last):
> > File "/usr/sbin/kojid", line 1437, in runTask
> > response = (handler.run(),)
> > File "/usr/sbin/kojid", line 1513, in run
> > return self.handler(*self.params,**self.opts)
> > File "/usr/sbin/kojid", line 3649, in handler
> > message = message.encode('utf-8')
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
> >
> > Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> > cronie, amanda, akonadi.
>
> And that's the reason why I didn't received any notification from Fedora
> Koji about successful builds ...

Hello Dan,

right, nobody normally looks into the web pages to see this failure,
but the log file also contains a "WARNING" about these and I've
wanted a clean rebuild of all rpm packages.

regards,

Florian La Roche

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Wednesday 21 July 2010, Ville Skyttä wrote:
> On Wednesday 21 July 2010, Dan Horák wrote:
> > Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> > > Hallo,
> > >
> > > With koji-1.4 running on RHEL6-beta2, some email notifications
> > > are not sent out correctly:
> > >
> > > Traceback (most recent call last):
> > > File "/usr/sbin/kojid", line 1437, in runTask
> > >
> > > response = (handler.run(),)
> > >
> > > File "/usr/sbin/kojid", line 1513, in run
> > >
> > > return self.handler(*self.params,**self.opts)
> > >
> > > File "/usr/sbin/kojid", line 3649, in handler
> > >
> > > message = message.encode('utf-8')
> > >
> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > > 546: ordinal not in range(128)
> > >
> > > Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> > > cronie, amanda, akonadi.
> >
> > And that's the reason why I didn't received any notification from Fedora
> > Koji about successful builds ...
>
> Neither have I, and not getting those mails has already started to
> interfere with my workflow which I'd rather not change... when can we
> expect a fix for this in the Fedora koji instance?

Ping? Still no notification mails for me, for example the modplugtools builds
done yesterday, and I haven't seen any comments when this might get fixed.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
> On Wednesday 21 July 2010, Ville Skyttä wrote:
> > On Wednesday 21 July 2010, Dan Horák wrote:
> > > Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> > > > Hallo,
> > > >
> > > > With koji-1.4 running on RHEL6-beta2, some email notifications
> > > > are not sent out correctly:
> > > >
> > > > Traceback (most recent call last):
> > > > File "/usr/sbin/kojid", line 1437, in runTask
> > > >
> > > > response = (handler.run(),)
> > > >
> > > > File "/usr/sbin/kojid", line 1513, in run
> > > >
> > > > return self.handler(*self.params,**self.opts)
> > > >
> > > > File "/usr/sbin/kojid", line 3649, in handler
> > > >
> > > > message = message.encode('utf-8')
> > > >
> > > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> > > > 546: ordinal not in range(128)
> > > >
> > > > Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> > > > cronie, amanda, akonadi.
> > >
> > > And that's the reason why I didn't received any notification from Fedora
> > > Koji about successful builds ...
> >
> > Neither have I, and not getting those mails has already started to
> > interfere with my workflow which I'd rather not change... when can we
> > expect a fix for this in the Fedora koji instance?
>
> Ping? Still no notification mails for me, for example the modplugtools builds
> done yesterday, and I haven't seen any comments when this might get fixed.
>
I know practically nothing about the koji code but that code seems very
wrong. This is earlier in the class::

message_templ =
"""From: %(from_addr)s
Subject: %(nvr)s %(result)s %(operation)s by %(user_name)s
To: %(to_addrs)s
X-Koji-Package: %(pkg_name)s
X-Koji-NVR: %(nvr)s
X-Koji-User: %(user_name)s
X-Koji-Status: %(status)s
%(tag_headers)s

Package: %(pkg_name)s
NVR: %(nvr)s
User: %(user_name)s
Status: %(status)s
%(operation_details)s
%(nvr)s %(result)s %(operation)s by %(user_name)s
%(failure_info)s
"""

Then in this method::
message = self.message_templ % locals()
# ensure message is in UTF-8
message = message.encode('utf-8')

Notice that message_templ is a byte string? If all of the variables in
locals() is also a byte string then message will be a byte string
(if locals() contains unicode strings it will either become a unicode string
or throw a traceback if it can't decode some other byte string to unicode).

The encode method of byte strings is pretty much useless. It allows you to
take a byte string that's all ascii and transform it into another byte
string. You can use it if you want to rot13 it but not much else. If the
byte string contains any bytes that aren't in the ascii subset, it'll throw
the error given above.

How to solve this depends on what you're actually trying to achieve. If
it's really to make sure that the byte string only contains valid utf8
data and you don't care what the non-utf8 data becomes I'd do it like this:

# Note this will be unsafe if locals contains strings of type <unicode>
message = self.message_templ % locals()
message = unicode(message, encoding='utf8', errors='replace').encode('utf8')

That takes the byte string that is the message, turns it into a unicode
string by treating the bytes as 'utf8', replacing any invalid bytes with �,
and then reencoding it to utf8.

If you can't guarantee that you start off with only byte strings, you could
use kitchen.text.converters.to_unicode to do the work or copy the essentials
of what it does::

message = self.message_templ % locals()
if isinstance(message, unicode):
message = message.encode('utf8')
else:
message = unicode(message, encoding='utf8', errors='replace').encode('utf8')

-Toshio
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Friday 13 August 2010, Toshio Kuratomi wrote:
> On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
> > On Wednesday 21 July 2010, Ville Skyttä wrote:
> > > On Wednesday 21 July 2010, Dan Horák wrote:
> > > > Florian La Roche pÃ*Å¡e v ÄŒt 15. 07. 2010 v 06:00 +0200:
> > > > > Hallo,
> > > > >
> > > > > With koji-1.4 running on RHEL6-beta2, some email notifications
> > > > > are not sent out correctly:
> > > > >
> > > > > Traceback (most recent call last):
> > > > > File "/usr/sbin/kojid", line 1437, in runTask
> > > > >
> > > > > response = (handler.run(),)
> > > > >
> > > > > File "/usr/sbin/kojid", line 1513, in run
> > > > >
> > > > > return self.handler(*self.params,**self.opts)
> > > > >
> > > > > File "/usr/sbin/kojid", line 3649, in handler
> > > > >
> > > > > message = message.encode('utf-8')
> > > > >
> > > > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in
> > > > > position 546: ordinal not in range(128)
> > > > >
> > > > > Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs,
> > > > > cronie, amanda, akonadi.
> > > >
> > > > And that's the reason why I didn't received any notification from
> > > > Fedora Koji about successful builds ...
> > >
> > > Neither have I, and not getting those mails has already started to
> > > interfere with my workflow which I'd rather not change... when can we
> > > expect a fix for this in the Fedora koji instance?
> >
> > Ping? Still no notification mails for me, for example the modplugtools
> > builds done yesterday, and I haven't seen any comments when this might
> > get fixed.
>
> I know practically nothing about the koji code but that code seems very
> wrong. This is earlier in the class:: [...]

Thanks for taking the time to look into this. Maybe some changes have been
made already - at least I received a build failure mail from today for a
failed javasqlite build. I didn't receive one for example for successful
seeker and rpmlint builds yesterday though :(
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

On Fri, Aug 20, 2010 at 05:06:33PM +0300, Ville Skyttä wrote:
> On Friday 13 August 2010, Toshio Kuratomi wrote:
> > On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
> > >
> > > Ping? Still no notification mails for me, for example the modplugtools
> > > builds done yesterday, and I haven't seen any comments when this might
> > > get fixed.
> >
> > I know practically nothing about the koji code but that code seems very
> > wrong. This is earlier in the class:: [...]
>
> Thanks for taking the time to look into this. Maybe some changes have been
> made already - at least I received a build failure mail from today for a
> failed javasqlite build. I didn't receive one for example for successful
> seeker and rpmlint builds yesterday though :(

dgilmore says that we haven't applied any changes to the koji running in
Fedora Infrastructure but that he needs to pull in a patch to hotfix our
instance with. Since I didn't see a patch in the upstream git repository,
I'm attaching one here.

-Toshio
From a3c0248a10c5ddbe9a34f76b995fbf10c34e87f5 Mon Sep 17 00:00:00 2001
From: Toshio Kuratomi <toshio@fedoraproject.org>
Date: Fri, 20 Aug 2010 11:17:43 -0400
Subject: [PATCH] Fix Unicode traceback with notification email.

---
builder/kojid | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/builder/kojid b/builder/kojid
index 2f7f480..77ded36 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -3455,7 +3455,10 @@ Status: %(status)s

message = self.message_templ % locals()
# ensure message is in UTF-8
- message = message.encode('utf-8')
+ if isinstance(message, unicode):
+ message = message.encode('utf-8')
+ else:
+ message = unicode(message, encoding='utf8', errors='replace').encode('utf8')

server = smtplib.SMTP(options.smtphost)
#server.set_debuglevel(True)
--
1.7.2.1

--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

email notifications
 

Hello Toshio,

koji already has a generic routing called fixEncoding() and I am currently
running koji with the patch from https://bugzilla.redhat.com/show_bug.cgi?id=614767.

Maybe your below code to use "errors='replace'" should go into the
generic koji function? (Haven't looked into this myself if it makes sense...)

regards,

Florian La Roche



> dgilmore says that we haven't applied any changes to the koji running in
> Fedora Infrastructure but that he needs to pull in a patch to hotfix our
> instance with. Since I didn't see a patch in the upstream git repository,
> I'm attaching one here.
>
> -Toshio

> From a3c0248a10c5ddbe9a34f76b995fbf10c34e87f5 Mon Sep 17 00:00:00 2001
> From: Toshio Kuratomi <toshio@fedoraproject.org>
> Date: Fri, 20 Aug 2010 11:17:43 -0400
> Subject: [PATCH] Fix Unicode traceback with notification email.
>
> ---
> builder/kojid | 5 ++++-
> 1 files changed, 4 insertions(+), 1 deletions(-)
>
> diff --git a/builder/kojid b/builder/kojid
> index 2f7f480..77ded36 100755
> --- a/builder/kojid
> +++ b/builder/kojid
> @@ -3455,7 +3455,10 @@ Status: %(status)s
>
> message = self.message_templ % locals()
> # ensure message is in UTF-8
> - message = message.encode('utf-8')
> + if isinstance(message, unicode):
> + message = message.encode('utf-8')
> + else:
> + message = unicode(message, encoding='utf8', errors='replace').encode('utf8')
>
> server = smtplib.SMTP(options.smtphost)
> #server.set_debuglevel(True)
> --
> 1.7.2.1
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys