FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor


 
 
LinkBack Thread Tools
 
Old 06-12-2010, 03:24 PM
Christopher Brown
 
Default Hey Presto!

You don't seem to be working all that good!

I'm sure I'm not the only one seeing this. In particular:

Transaction Summary
================================================== ==============================
Install 1 Package(s)
Upgrade 85 Package(s)

Total download size: 207 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates/prestodelta | 17 kB 00:00
Processing delta metadata
Download delta size: 2.1 M
(1/10): ModemManager-0.3-9.git20100409.fc12_0.3-13.git20 | 126 kB 00:00
(2/10): NetworkManager-0.8.0-6.git20100408.fc12_0.8.1-0. | 510 kB 00:00
(3/10): NetworkManager-glib-0.8.0-6.git20100408.fc12_0.8 | 106 kB 00:00
(4/10): NetworkManager-gnome-0.8.0-6.git20100408.fc12_0. | 301 kB 00:00
(5/10): NetworkManager-pptp-0.7.997-3.git20100120.fc12_0 | 47 kB 00:00
(6/10): NetworkManager-vpnc-0.7.996-4.git20090921.fc12_0 | 56 kB 00:00
(7/10): gstreamer-plugins-bad-free-0.10.18-1.fc12_0.10.1 | 339 kB 00:00
(8/10): gstreamer-plugins-good-0.10.22-1.fc12_0.10.23-1. | 427 kB 00:00
(9/10): libmtp-1.0.2-1.fc12_1.0.3-2.fc12.x86_64.drpm | 33 kB 00:00
(10/10): wpa_supplicant-0.6.8-8.fc12_0.6.8-9.fc12.x86_64 | 191 kB 00:00
Finishing rebuild of rpms, from deltarpms
<delta rebuild> | 4.5 MB 00:03
Presto reduced the update size by 54% (from 4.5 M to 2.1 M).
Package(s) data still to download: 202 M



...which includes the most recent openoffice.org update - a prime
candidate for drpm goodness.

Am I missing something obvious?

--
Christopher Brown
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:27 PM
drago01
 
Default Hey Presto!

On Sat, Jun 12, 2010 at 5:24 PM, Christopher Brown
<snecklifter@gmail.com> wrote:
> You don't seem to be working all that good!
>
> I'm sure I'm not the only one seeing this. In particular:
>
> Transaction Summary
> ================================================== ==============================
> Install * * * 1 Package(s)
> Upgrade * * *85 Package(s)
>
> Total download size: 207 M
> Is this ok [y/N]: y
> Downloading Packages:
> Setting up and reading Presto delta metadata
> updates/prestodelta * * * * * * * * * * * * * * * * * * *| *17 kB * * 00:00
> Processing delta metadata
> Download delta size: 2.1 M
> (1/10): ModemManager-0.3-9.git20100409.fc12_0.3-13.git20 | 126 kB * * 00:00
> (2/10): NetworkManager-0.8.0-6.git20100408.fc12_0.8.1-0. | 510 kB * * 00:00
> (3/10): NetworkManager-glib-0.8.0-6.git20100408.fc12_0.8 | 106 kB * * 00:00
> (4/10): NetworkManager-gnome-0.8.0-6.git20100408.fc12_0. | 301 kB * * 00:00
> (5/10): NetworkManager-pptp-0.7.997-3.git20100120.fc12_0 | *47 kB * * 00:00
> (6/10): NetworkManager-vpnc-0.7.996-4.git20090921.fc12_0 | *56 kB * * 00:00
> (7/10): gstreamer-plugins-bad-free-0.10.18-1.fc12_0.10.1 | 339 kB * * 00:00
> (8/10): gstreamer-plugins-good-0.10.22-1.fc12_0.10.23-1. | 427 kB * * 00:00
> (9/10): libmtp-1.0.2-1.fc12_1.0.3-2.fc12.x86_64.drpm * * | *33 kB * * 00:00
> (10/10): wpa_supplicant-0.6.8-8.fc12_0.6.8-9.fc12.x86_64 | 191 kB * * 00:00
> Finishing rebuild of rpms, from deltarpms
> <delta rebuild> * * * * * * * * * * * * * * * * * * * * *| 4.5 MB * * 00:03
> Presto reduced the update size by 54% (from 4.5 M to 2.1 M).
> Package(s) data still to download: 202 M
>
>
>
> ...which includes the most recent openoffice.org update - a prime
> candidate for drpm goodness.
>
> Am I missing something obvious?

We don't generate deltas for packages with a size of >= 100MB ....
which kind of makes it useless for this case but it seems that delta
generation is to expensive to do for such large packages on the re-eng
boxes.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:27 PM
drago01
 
Default Hey Presto!

On Sat, Jun 12, 2010 at 5:24 PM, Christopher Brown
<snecklifter@gmail.com> wrote:
> You don't seem to be working all that good!
>
> I'm sure I'm not the only one seeing this. In particular:
>
> Transaction Summary
> ================================================== ==============================
> Install * * * 1 Package(s)
> Upgrade * * *85 Package(s)
>
> Total download size: 207 M
> Is this ok [y/N]: y
> Downloading Packages:
> Setting up and reading Presto delta metadata
> updates/prestodelta * * * * * * * * * * * * * * * * * * *| *17 kB * * 00:00
> Processing delta metadata
> Download delta size: 2.1 M
> (1/10): ModemManager-0.3-9.git20100409.fc12_0.3-13.git20 | 126 kB * * 00:00
> (2/10): NetworkManager-0.8.0-6.git20100408.fc12_0.8.1-0. | 510 kB * * 00:00
> (3/10): NetworkManager-glib-0.8.0-6.git20100408.fc12_0.8 | 106 kB * * 00:00
> (4/10): NetworkManager-gnome-0.8.0-6.git20100408.fc12_0. | 301 kB * * 00:00
> (5/10): NetworkManager-pptp-0.7.997-3.git20100120.fc12_0 | *47 kB * * 00:00
> (6/10): NetworkManager-vpnc-0.7.996-4.git20090921.fc12_0 | *56 kB * * 00:00
> (7/10): gstreamer-plugins-bad-free-0.10.18-1.fc12_0.10.1 | 339 kB * * 00:00
> (8/10): gstreamer-plugins-good-0.10.22-1.fc12_0.10.23-1. | 427 kB * * 00:00
> (9/10): libmtp-1.0.2-1.fc12_1.0.3-2.fc12.x86_64.drpm * * | *33 kB * * 00:00
> (10/10): wpa_supplicant-0.6.8-8.fc12_0.6.8-9.fc12.x86_64 | 191 kB * * 00:00
> Finishing rebuild of rpms, from deltarpms
> <delta rebuild> * * * * * * * * * * * * * * * * * * * * *| 4.5 MB * * 00:03
> Presto reduced the update size by 54% (from 4.5 M to 2.1 M).
> Package(s) data still to download: 202 M
>
>
>
> ...which includes the most recent openoffice.org update - a prime
> candidate for drpm goodness.
>
> Am I missing something obvious?

We don't generate deltas for packages with a size of >= 100MB ....
which kind of makes it useless for this case but it seems that delta
generation is to expensive to do for such large packages on the re-eng
boxes.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:35 PM
"Richard W.M. Jones"
 
Default Hey Presto!

On Sat, Jun 12, 2010 at 05:27:32PM +0200, drago01 wrote:
> We don't generate deltas for packages with a size of >= 100MB ....
> which kind of makes it useless for this case but it seems that delta
> generation is to expensive to do for such large packages on the re-eng
> boxes.

It's because the program that generates the delta RPMs reads the whole
RPMs into memory, according to:

http://lwn.net/Articles/329484/

Anyone know if there's a genuine reason why it does it, or if it's
just a Simple Matter Of Programming to fix it? (And can point us to
the actual bit of code that could be fixed ...)

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:38 PM
Jonathan Dieter
 
Default Hey Presto!

On Sat, 2010-06-12 at 16:24 +0100, Christopher Brown wrote:
> You don't seem to be working all that good!
>
> I'm sure I'm not the only one seeing this. In particular:
>
> Transaction Summary
> ================================================== ==============================
> Install 1 Package(s)
> Upgrade 85 Package(s)
>
> Total download size: 207 M
> Is this ok [y/N]: y
> Downloading Packages:
> Setting up and reading Presto delta metadata
> updates/prestodelta | 17 kB 00:00
> Processing delta metadata
> Download delta size: 2.1 M
> (1/10): ModemManager-0.3-9.git20100409.fc12_0.3-13.git20 | 126 kB 00:00
> (2/10): NetworkManager-0.8.0-6.git20100408.fc12_0.8.1-0. | 510 kB 00:00
> (3/10): NetworkManager-glib-0.8.0-6.git20100408.fc12_0.8 | 106 kB 00:00
> (4/10): NetworkManager-gnome-0.8.0-6.git20100408.fc12_0. | 301 kB 00:00
> (5/10): NetworkManager-pptp-0.7.997-3.git20100120.fc12_0 | 47 kB 00:00
> (6/10): NetworkManager-vpnc-0.7.996-4.git20090921.fc12_0 | 56 kB 00:00
> (7/10): gstreamer-plugins-bad-free-0.10.18-1.fc12_0.10.1 | 339 kB 00:00
> (8/10): gstreamer-plugins-good-0.10.22-1.fc12_0.10.23-1. | 427 kB 00:00
> (9/10): libmtp-1.0.2-1.fc12_1.0.3-2.fc12.x86_64.drpm | 33 kB 00:00
> (10/10): wpa_supplicant-0.6.8-8.fc12_0.6.8-9.fc12.x86_64 | 191 kB 00:00
> Finishing rebuild of rpms, from deltarpms
> <delta rebuild> | 4.5 MB 00:03
> Presto reduced the update size by 54% (from 4.5 M to 2.1 M).
> Package(s) data still to download: 202 M
>
>
>
> ...which includes the most recent openoffice.org update - a prime
> candidate for drpm goodness.
>
> Am I missing something obvious?

Old deltarpms are being deleted after each push. This means that the
openoffice.org deltarpms were only available for a day or two before
being deleted. Yes, this is a bug.

https://bugzilla.redhat.com/show_bug.cgi?id=598584

Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:43 PM
John5342
 
Default Hey Presto!

On Sat, Jun 12, 2010 at 16:35, Richard W.M. Jones <rjones@redhat.com> wrote:
> On Sat, Jun 12, 2010 at 05:27:32PM +0200, drago01 wrote:
>> We don't generate deltas for packages with a size of >= 100MB ....
>> which kind of makes it useless for this case but it seems that delta
>> generation is to expensive to do for such large packages on the re-eng
>> boxes.
>
> It's because the program that generates the delta RPMs reads the whole
> RPMs into memory, according to:
>
> http://lwn.net/Articles/329484/
>
> Anyone know if there's a genuine reason why it does it, or if it's
> just a Simple Matter Of Programming to fix it? *(And can point us to
> the actual bit of code that could be fixed ...)

There was in fact a request for volunteers about a month ago:

http://lists.fedoraproject.org/pipermail/devel/2010-May/136090.html

--
There are 10 kinds of people in the world: Those who understand binary
and those who don't...
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-12-2010, 03:48 PM
Jonathan Dieter
 
Default Hey Presto!

On Sat, 2010-06-12 at 16:35 +0100, Richard W.M. Jones wrote:
> On Sat, Jun 12, 2010 at 05:27:32PM +0200, drago01 wrote:
> > We don't generate deltas for packages with a size of >= 100MB ....
> > which kind of makes it useless for this case but it seems that delta
> > generation is to expensive to do for such large packages on the re-eng
> > boxes.
>
> It's because the program that generates the delta RPMs reads the whole
> RPMs into memory, according to:
>
> http://lwn.net/Articles/329484/
>
> Anyone know if there's a genuine reason why it does it, or if it's
> just a Simple Matter Of Programming to fix it? (And can point us to
> the actual bit of code that could be fixed ...)

For the record, openoffice.org-core comes under the size limit for
deltarpm generation (which I think is closer to 200MB, but I may be
wrong), which means we normally *do* get openoffice.org-core deltarpms.
In my other email, I explained why we don't have them right now.

As for the reason why, deltarpm currently compares *all* of the
uncompressed old rpm against *all* of the uncompressed new rpm. This
gives you the best possible delta at the expense of memory usage. I
would like to allow deltarpm to split both old and new rpms into block
and delta each block separately, but it would involve some very creative
reworking on how deltarpm uses pseudo-files for all of it's work (see
cfile.[ch] for the pseudo-file structure).

I don't know if that's clear enough, feel free to ask if it's not.

Jonathan
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-13-2010, 04:15 PM
Christopher Brown
 
Default Hey Presto!

On 12 June 2010 16:48, Jonathan Dieter <jdieter@gmail.com> wrote:
> On Sat, 2010-06-12 at 16:35 +0100, Richard W.M. Jones wrote:
>> On Sat, Jun 12, 2010 at 05:27:32PM +0200, drago01 wrote:
>> > We don't generate deltas for packages with a size of >= 100MB ....
>> > which kind of makes it useless for this case but it seems that delta
>> > generation is to expensive to do for such large packages on the re-eng
>> > boxes.
>>
>> It's because the program that generates the delta RPMs reads the whole
>> RPMs into memory, according to:
>>
>> http://lwn.net/Articles/329484/
>>
>> Anyone know if there's a genuine reason why it does it, or if it's
>> just a Simple Matter Of Programming to fix it? *(And can point us to
>> the actual bit of code that could be fixed ...)
>
> For the record, openoffice.org-core comes under the size limit for
> deltarpm generation (which I think is closer to 200MB, but I may be
> wrong), which means we normally *do* get openoffice.org-core deltarpms.
> In my other email, I explained why we don't have them right now.
>
> As for the reason why, deltarpm currently compares *all* of the
> uncompressed old rpm against *all* of the uncompressed new rpm. *This
> gives you the best possible delta at the expense of memory usage. *I
> would like to allow deltarpm to split both old and new rpms into block
> and delta each block separately, but it would involve some very creative
> reworking on how deltarpm uses pseudo-files for all of it's work (see
> cfile.[ch] for the pseudo-file structure).
>
> I don't know if that's clear enough, feel free to ask if it's not.

Clear as quartz - thanks Jonathan. I do indeed recall us getting
deltas on OO.org previously which is why I raised the question. I
think the presto plugin is a huge improvement to Fedora and provides
wins on so many levels that it really is something that needs fixing.
With any luck we will see this for RHEL and CentOS 6.

Yeah, I'd love to have the brain-fu to sort this but I'm afraid its
beyond my meagre intellect. Happy to pick off an item from someone's
Amazon wishlist to get this bumped up the priority list.

Cheers

--
Christopher Brown
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-14-2010, 06:21 AM
Adam Miller
 
Default Hey Presto!

I'm with Christopher, this topic is a bit outside my realm of knowledge but I would be willing to donate monitarily to those who are willing and able.


This actually brings up something a bit more, but would a "Vote with your dollars" bounty system be possible? Such that users donate money to an idea they like and a contributor gets the pool of money that particular idea gained when they complete whatever is needed to implement the request? .... just an idea, not sure what all logistics would be needed for setting something like that into motion.



Thoughts?


-AdamM (From Android)


On Jun 13, 2010 11:15 AM, "Christopher Brown" <snecklifter@gmail.com> wrote:


On 12 June 2010 16:48, Jonathan Dieter <jdieter@gmail.com> wrote:

> On Sat, 2010-06-12 at 16:35 +010...
Clear as quartz - thanks Jonathan. I do indeed recall us getting

deltas on OO.org previously which is why I raised the question. I

think the presto plugin is a huge improvement to Fedora and provides

wins on so many levels that it really is something that needs fixing.

With any luck we will see this for RHEL and CentOS 6.



Yeah, I'd love to have the brain-fu to sort this but I'm afraid its

beyond my meagre intellect. Happy to pick off an item from someone's

Amazon wishlist to get this bumped up the priority list.



Cheers



--

Christopher Brown


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinf...



--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 06-14-2010, 08:24 AM
Michael Schroeder
 
Default Hey Presto!

On Sat, Jun 12, 2010 at 04:35:42PM +0100, Richard W.M. Jones wrote:
> On Sat, Jun 12, 2010 at 05:27:32PM +0200, drago01 wrote:
> > We don't generate deltas for packages with a size of >= 100MB ....
> > which kind of makes it useless for this case but it seems that delta
> > generation is to expensive to do for such large packages on the re-eng
> > boxes.
>
> It's because the program that generates the delta RPMs reads the whole
> RPMs into memory, according to:
>
> http://lwn.net/Articles/329484/

It currenlty uses three big chunks of memory
- the source rpm payload (unpacked)
- the target rpm payload (unpacked)
- a hash over the source rpm (exactly the size of the unpacked
source rpm payload)

So it needs roughly three times the size of the unpacked payload.

> Anyone know if there's a genuine reason why it does it, or if it's
> just a Simple Matter Of Programming to fix it? (And can point us to
> the actual bit of code that could be fixed ...)

It's not that hard to fix, there's no need to keep the "target"
rpm in memory at all. The "source" rpm can be limited to some
max size with the down side that the end of the "target" rpm
cannot match the start of the "source" rpm anymore. This shouldn't
do much harm in the real world.

I've already looked at the code, it shouldn't be much work to
implement. I'll try to do it this or next week.

Cheers,
Michael.

--
Michael Schroeder mls@suse.de
SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 01:55 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org