Koji and Lustre
 

Hey guys,

We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :


An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>


Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object

return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data

return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__

return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod

raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>


The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.


Example (see 'Device:' the value after the '/'):

# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000

# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48

Hope it helps, lost a good amount of time on that one :)

Cheers,
Thomas
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Koji and Lustre
 

On 07/12/2010 09:18 PM, Thomas Guthmann wrote:
> Hey guys,
>
> We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
> when used with a Lustre filesystem. This function parses all attributes
> of every file of a build and is used when you want to display
> build.log/root.log through the web interface. Everything returned by
> listTaskOutput is returned through XML-RPC and as a result we had this :
>
> An error has occurred while processing your request.
> Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
> limits'>

Note that the httpd error_log on the hub should have shown the full
server side traceback.

> - if attr == 'st_size':
> + if attr == 'st_size' or attr == 'st_dev':

Certainly this will work, though I question the utility of reporting the
entire os.stat result. Most of them are meaningless on a remote host. It
looks like we only ever use st_size in koji itself. I think only that
and the timestamps are appropriate, so we could just drop the rest entirely.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Koji and Lustre
 

Hey guys,

I've just upgraded to koji 1.6 and I still have this issue with Lustre.

Because it's not maintstream, you will find the new patch in attachment
(it's the same patch than before without Hunks warning).


Thomas


--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys

Koji and Lustre
 

On 09/07/2011 02:02 AM, Thomas Guthmann wrote:
> Hey guys,
>
> I've just upgraded to koji 1.6 and I still have this issue with Lustre.
>
> Because it's not maintstream, you will find the new patch in attachment
> (it's the same patch than before without Hunks warning).

The patch looks fine. I'll put it in the next release.
--
buildsys mailing list
buildsys@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/buildsys