FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 04-12-2011, 10:46 AM
Raphael Hertzog
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

Hi Bill,

On Tue, 12 Apr 2011, Bill Allombert wrote:
> > I get errors from cron because of popcon:
> > /etc/cron.daily/popularity-contest:
> > popcon: file /var/lib/dpkg/info/liblouis2.list is missing
> >
> > That file doesn't exist, it's really /var/lib/dpkg/info/liblouis2:i386.list
> > due to the multiarch-enabled dpkg that I'm running.
> >
> > You should not access those files, the proper interface is dpkg-query -L <pkg>.
>
> Hello Raphaël,
>
> This is not possible: forking dpkg for all installed packages would be way to slow and
> resource intensive. We need a better option.

There's no better alternative. You can use dpkg-query --control-path to
get the path of other non-internal control files there:
$ dpkg-query --control-path liblouis2
/var/lib/dpkg/info/liblouis2:i386.md5sums
/var/lib/dpkg/info/liblouis2:i386.postrm
/var/lib/dpkg/info/liblouis2:i386.shlibs
/var/lib/dpkg/info/liblouis2:i386.postinst

We deliberately skip .list as we don't guarantee that we're always going
to use .list and there's no guaranty that the format of the file won't be
extended to store more information. You should not read those files
directly.

As a short term hack, you can try to infer the .list from the above but
it's definitely not going to work on the long term. If you don't get an
answer, because the package has no extra control files then you're left to
guess.

At some point in the future, you should be able to use libdpkg to write a
dedicated program and avoid the cost of multiple parsing of the status
database (the fork itself is not so costly I think, the parsing is) but
we're not really there yet.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110412104659.GA14466@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110412104659.GA14466@rivendell.home.ouaza.com
 
Old 04-12-2011, 12:32 PM
Bill Allombert
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Tue, Apr 12, 2011 at 12:46:59PM +0200, Raphael Hertzog wrote:
> Hi Bill,
>
> On Tue, 12 Apr 2011, Bill Allombert wrote:
> > > I get errors from cron because of popcon:
> > > /etc/cron.daily/popularity-contest:
> > > popcon: file /var/lib/dpkg/info/liblouis2.list is missing
> > >
> > > That file doesn't exist, it's really /var/lib/dpkg/info/liblouis2:i386.list
> > > due to the multiarch-enabled dpkg that I'm running.
> > >
> > > You should not access those files, the proper interface is dpkg-query -L <pkg>.
> >
> > Hello Raphal,
> >
> > This is not possible: forking dpkg for all installed packages would be way to slow and
> > resource intensive. We need a better option.
>
> There's no better alternative. You can use dpkg-query --control-path to
> get the path of other non-internal control files there:
> $ dpkg-query --control-path liblouis2
> /var/lib/dpkg/info/liblouis2:i386.md5sums
> /var/lib/dpkg/info/liblouis2:i386.postrm
> /var/lib/dpkg/info/liblouis2:i386.shlibs
> /var/lib/dpkg/info/liblouis2:i386.postinst
>
> We deliberately skip .list as we don't guarantee that we're always going
> to use .list and there's no guaranty that the format of the file won't be
> extended to store more information. You should not read those files
> directly.

Hello Raphael,
I have a hard time identifying what you mean by 'we' and 'you' in the above sentences.
I do not read these file. This is what the script popularity-contest do and have
done since well before I was the maintainer of popcon or you the maintainer of dpkg.
There is a single project, Debian, and Debian developers need to figure out a
way to get popularity-contest and dpkg to work together nicely, and set up a transition
period to allow partial upgrades.

Cheers,
--
Bill. <ballombe@debian.org>

Imagine a large red swirl here.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110412123245.GL6352@yellowpig">http://lists.debian.org/20110412123245.GL6352@yellowpig
 
Old 04-12-2011, 09:06 PM
Raphael Hertzog
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

Hi,

On Tue, 12 Apr 2011, Bill Allombert wrote:
> > We deliberately skip .list as we don't guarantee that we're always going
> > to use .list and there's no guaranty that the format of the file won't be
> > extended to store more information. You should not read those files
> > directly.
>
> Hello Raphael,
> I have a hard time identifying what you mean by 'we' and 'you' in the above sentences.

we = the dpkg developers
you = popcon / the popcon maintainer who controls the code in popcon

> I do not read these file. This is what the script popularity-contest do and have
> done since well before I was the maintainer of popcon or you the maintainer of dpkg.

This doesn't make it any less wrong.

> There is a single project, Debian, and Debian developers need to figure out a
> way to get popularity-contest and dpkg to work together nicely, and set up a transition
> period to allow partial upgrades.

We'll gladly add Breaks to help people switch to fixed versions of
software. But for this we need a fixed popcon first.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110412210626.GA18119@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110412210626.GA18119@rivendell.home.ouaza.com
 
Old 04-20-2011, 01:56 PM
Bill Allombert
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Tue, Apr 12, 2011 at 11:06:26PM +0200, Raphael Hertzog wrote:
> Hi,
>
> On Tue, 12 Apr 2011, Bill Allombert wrote:
> > > We deliberately skip .list as we don't guarantee that we're always going
> > > to use .list and there's no guaranty that the format of the file won't be
> > > extended to store more information. You should not read those files
> > > directly.
> >
> > Hello Raphael,
> > I have a hard time identifying what you mean by 'we' and 'you' in the above sentences.
>
> we = the dpkg developers
> you = popcon / the popcon maintainer who controls the code in popcon
>
> > I do not read these file. This is what the script popularity-contest do and have
> > done since well before I was the maintainer of popcon or you the maintainer of dpkg.
>
> This doesn't make it any less wrong.

But that means I did not do anything wrong, and also that maybe it was not
wrong by the standard of the dpkg maintainers of the time.

And saying that it is wrong does not provide any help.

On my system with 1200 packages (far below the average popcon submitter),
traditional popcon take 2s. Once patched with the attached patch to use dpkg -L,
it take 30s. Slowing down 15 times popcon is not acceptable.

Cheers,
--
Bill. <ballombe@debian.org>

Imagine a large red swirl here.
Index: popularity-contest
================================================== =================
--- popularity-contest (rvision 641)
+++ popularity-contest (copie de travail)
@@ -99,7 +99,7 @@
/^.*installed *(.+)$/ or next;
my $pkg=$1;
$popcon{$pkg}=[0,0,$pkg,"<NOFILES>"];
- open FILES, "$dpkg_db/$pkg.list" or
+ open FILES, "-|", "dpkg -L $pkg" or
do { print STDERR "popcon: file $dpkg_db/$pkg.list is missing
"; next; };
my $bestatime = undef;
while (<FILES>)
 
Old 04-20-2011, 07:11 PM
Raphael Hertzog
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, 20 Apr 2011, Bill Allombert wrote:
> On my system with 1200 packages (far below the average popcon submitter),
> traditional popcon take 2s. Once patched with the attached patch to use dpkg -L,
> it take 30s. Slowing down 15 times popcon is not acceptable.

You can invoke "dpkg -L" less often by giving multiple packages as
parameters. Each set of files will be separated by an empty line.

This will greatly improve the performance.

(BTW, popcon is mainly run from cron so interactive performance is not so
critical.)

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110420191159.GB18649@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110420191159.GB18649@rivendell.home.ouaza.com
 
Old 04-20-2011, 07:29 PM
Bill Allombert
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, Apr 20, 2011 at 09:11:59PM +0200, Raphael Hertzog wrote:
> On Wed, 20 Apr 2011, Bill Allombert wrote:
> > On my system with 1200 packages (far below the average popcon submitter),
> > traditional popcon take 2s. Once patched with the attached patch to use dpkg -L,
> > it take 30s. Slowing down 15 times popcon is not acceptable.
>
> You can invoke "dpkg -L" less often by giving multiple packages as
> parameters. Each set of files will be separated by an empty line.

Good, is that behaviour documented somewhere ?

> This will greatly improve the performance.

Great, send me a patch and I will benchmark it.

> (BTW, popcon is mainly run from cron so interactive performance is not so
> critical.)

The issue is not interactive performances but waste of system resource. Users
will complain.

Cheers,
--
Bill. <ballombe@debian.org>

Imagine a large red swirl here.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110420192928.GC13243@yellowpig">http://lists.debian.org/20110420192928.GC13243@yellowpig
 
Old 04-20-2011, 08:51 PM
Raphael Hertzog
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, 20 Apr 2011, Bill Allombert wrote:
> On Wed, Apr 20, 2011 at 09:11:59PM +0200, Raphael Hertzog wrote:
> > On Wed, 20 Apr 2011, Bill Allombert wrote:
> > > On my system with 1200 packages (far below the average popcon submitter),
> > > traditional popcon take 2s. Once patched with the attached patch to use dpkg -L,
> > > it take 30s. Slowing down 15 times popcon is not acceptable.
> >
> > You can invoke "dpkg -L" less often by giving multiple packages as
> > parameters. Each set of files will be separated by an empty line.
>
> Good, is that behaviour documented somewhere ?

man dpkg-query shows that multiple parameters are allowed, it does not
explain that an empty line is the separator though.

> > This will greatly improve the performance.
>
> Great, send me a patch and I will benchmark it.

Dude, I'm not the maintainer of popcon...

> The issue is not interactive performances but waste of system resource. Users
> will complain.

I'm a user too, and I complain about an error message that popcon spits
out right now once a week... pick your poison.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110420205157.GC18649@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110420205157.GC18649@rivendell.home.ouaza.com
 
Old 04-20-2011, 09:08 PM
Bill Allombert
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, Apr 20, 2011 at 10:51:57PM +0200, Raphael Hertzog wrote:
> On Wed, 20 Apr 2011, Bill Allombert wrote:
> > On Wed, Apr 20, 2011 at 09:11:59PM +0200, Raphael Hertzog wrote:
> > > On Wed, 20 Apr 2011, Bill Allombert wrote:
> > > > On my system with 1200 packages (far below the average popcon submitter),
> > > > traditional popcon take 2s. Once patched with the attached patch to use dpkg -L,
> > > > it take 30s. Slowing down 15 times popcon is not acceptable.
> > >
> > > You can invoke "dpkg -L" less often by giving multiple packages as
> > > parameters. Each set of files will be separated by an empty line.
> >
> > Good, is that behaviour documented somewhere ?
>
> man dpkg-query shows that multiple parameters are allowed, it does not
> explain that an empty line is the separator though.

Well, surely we cannot rely on such undocumented behaviour.

Cheers,
--
Bill. <ballombe@debian.org>

Imagine a large red swirl here.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110420210851.GE13243@yellowpig">http://lists.debian.org/20110420210851.GE13243@yellowpig
 
Old 04-20-2011, 09:45 PM
Guillem Jover
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, 2011-04-20 at 23:08:51 +0200, Bill Allombert wrote:
> On Wed, Apr 20, 2011 at 10:51:57PM +0200, Raphael Hertzog wrote:
> > On Wed, 20 Apr 2011, Bill Allombert wrote:
> > > On Wed, Apr 20, 2011 at 09:11:59PM +0200, Raphael Hertzog wrote:
> > > > You can invoke "dpkg -L" less often by giving multiple packages as
> > > > parameters. Each set of files will be separated by an empty line.
> > >
> > > Good, is that behaviour documented somewhere ?
> >
> > man dpkg-query shows that multiple parameters are allowed, it does not
> > explain that an empty line is the separator though.
>
> Well, surely we cannot rely on such undocumented behaviour.

<http://git.debian.org/?p=dpkg/dpkg.git;a=commitdiff;h=e6b6ff088>

regards,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110420214508.GA3888@gaara.hadrons.org">http://lists.debian.org/20110420214508.GA3888@gaara.hadrons.org
 
Old 04-21-2011, 08:54 PM
Bill Allombert
 
Default Bug#622322: popularity-contest: popcon tries to access dpkg internal files and fails with multiarch: same packageso

On Wed, Apr 20, 2011 at 04:10:58PM -0500, Jonathan Nieder wrote:
> Hi,
>
> Bill Allombert wrote:
>
> > This is not possible: forking dpkg for all installed packages would be way to slow and
> > resource intensive. We need a better option.
>
> Bonus points if this interface has an option to point to the file on
> disk rather than asking the caller to take care of tracking down
> diversions.
>
> Of course alternative methods might be possible; I ask the above
> because I am worried about the memory usage from
> "dpkg-query -L $(list-all-packages)".

Another issue with 'dpkg-query -L $(list-all-packages)' is that it is not portable
to system with a command-line length limit.

I suppose popcon will have to process packages by chunk of 100, say.

Cheers,
--
Bill. <ballombe@debian.org>

Imagine a large red swirl here.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110421205411.GK13243@yellowpig">http://lists.debian.org/20110421205411.GK13243@yellowpig
 

Thread Tools




All times are GMT. The time now is 09:11 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org