FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 03-05-2010, 10:16 PM
Till Maas
 
Default Fight bugs, not FESCo

On Mon, Mar 01, 2010 at 10:42:32AM -0600, Mike McGrath wrote:

> It looks like popcon has like 93000 profiles? Smolt has 1.8 million [1]
> and even at that level without package data we have horrible performance
> issues. If I were to add packages with my knowledge of db's, smolt would
> become useless within a month because the thing would be completely
> unavailable.
>
> If someone *really* wants to do this and knows more about databases then I
> do, I'll help them through it. It's a high bar though and not to be taken
> lightly.

Imho for the beginning, there is no need to be able to query complete
profiles, but it would be enough to have a count per package. A simple
implementation for this would be:

1) clients send a plaintext list of installed packages and a UUID every
X days or by user request
2) file is stored in UUID.$timestamp (or it is stored as a BLOB in the
DB)
3) once a day a crawler reads all files and counts for each package how
often they are installed, this is stored in a DB for easy querying
4) all files older than X days are deleted

rpm -qa xz compressed uses 17K on my system, for 1.8 million profiles
this would require 31GB of storage, but this amount of storage would be
needed at least of every approach if we need this details.

The only improvement I can think of would be to only report the leaves
and compute the dependencies on the server, then we can use the output
of "package-cleanup --leaves --all", which is xz-compressed only 4K on
my system or 7.4 GB for 1.8 million profiles.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-05-2010, 11:18 PM
Mike McGrath
 
Default Fight bugs, not FESCo

On Sat, 6 Mar 2010, Till Maas wrote:

> On Mon, Mar 01, 2010 at 10:42:32AM -0600, Mike McGrath wrote:
>
> > It looks like popcon has like 93000 profiles? Smolt has 1.8 million [1]
> > and even at that level without package data we have horrible performance
> > issues. If I were to add packages with my knowledge of db's, smolt would
> > become useless within a month because the thing would be completely
> > unavailable.
> >
> > If someone *really* wants to do this and knows more about databases then I
> > do, I'll help them through it. It's a high bar though and not to be taken
> > lightly.
>
> Imho for the beginning, there is no need to be able to query complete
> profiles, but it would be enough to have a count per package. A simple
> implementation for this would be:
>
> 1) clients send a plaintext list of installed packages and a UUID every
> X days or by user request
> 2) file is stored in UUID.$timestamp (or it is stored as a BLOB in the
> DB)
> 3) once a day a crawler reads all files and counts for each package how
> often they are installed, this is stored in a DB for easy querying
> 4) all files older than X days are deleted
>
> rpm -qa xz compressed uses 17K on my system, for 1.8 million profiles
> this would require 31GB of storage, but this amount of storage would be
> needed at least of every approach if we need this details.
>
> The only improvement I can think of would be to only report the leaves
> and compute the dependencies on the server, then we can use the output
> of "package-cleanup --leaves --all", which is xz-compressed only 4K on
> my system or 7.4 GB for 1.8 million profiles.
>

I'm happy to provide a current dump of the database if you want to
populate it with sample data and see how things look?

-Mike
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-06-2010, 08:04 AM
Till Maas
 
Default Fight bugs, not FESCo

On Fri, Mar 05, 2010 at 06:18:37PM -0600, Mike McGrath wrote:
> On Sat, 6 Mar 2010, Till Maas wrote:
>
> > On Mon, Mar 01, 2010 at 10:42:32AM -0600, Mike McGrath wrote:

> > Imho for the beginning, there is no need to be able to query complete
> > profiles, but it would be enough to have a count per package. A simple
> > implementation for this would be:
> >
> > 1) clients send a plaintext list of installed packages and a UUID every
> > X days or by user request
> > 2) file is stored in UUID.$timestamp (or it is stored as a BLOB in the
> > DB)
> > 3) once a day a crawler reads all files and counts for each package how
> > often they are installed, this is stored in a DB for easy querying
> > 4) all files older than X days are deleted
> >
> > rpm -qa xz compressed uses 17K on my system, for 1.8 million profiles
> > this would require 31GB of storage, but this amount of storage would be
> > needed at least of every approach if we need this details.
> >
> > The only improvement I can think of would be to only report the leaves
> > and compute the dependencies on the server, then we can use the output
> > of "package-cleanup --leaves --all", which is xz-compressed only 4K on
> > my system or 7.4 GB for 1.8 million profiles.
> >
>
> I'm happy to provide a current dump of the database if you want to
> populate it with sample data and see how things look?

Yes, please.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-06-2010, 08:34 AM
Frank Murphy
 
Default Fight bugs, not FESCo

On 06/03/10 09:04, Till Maas wrote:
> On Fri, Mar 05, 2010 at 06:18:37PM -0600, Mike McGrath wrote:
>> On Sat, 6 Mar 2010, Till Maas wrote:
>>
>>> On Mon, Mar 01, 2010 at 10:42:32AM -0600, Mike McGrath wrote:
>
>>> Imho for the beginning, there is no need to be able to query complete
>>> profiles, but it would be enough to have a count per package. A simple
>>> implementation for this would be:
>>>
>>> 1) clients send a plaintext list of installed packages and a UUID every
>>> X days or by user request
>>> 2) file is stored in UUID.$timestamp (or it is stored as a BLOB in the
>>> DB)
>>> 3) once a day a crawler reads all files and counts for each package how
>>> often they are installed,

What about uninstalled?

Update bring in upd to X, but package Y,Z. come in.
User removes Y,Z without breaking anything.

--snipped--

--
Regards,

Frank Murphy
UTF_8 Encoded
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-06-2010, 09:04 AM
Till Maas
 
Default Fight bugs, not FESCo

On Sat, Mar 06, 2010 at 09:34:08AM +0000, Frank Murphy wrote:
> On 06/03/10 09:04, Till Maas wrote:
> > On Fri, Mar 05, 2010 at 06:18:37PM -0600, Mike McGrath wrote:
> >> On Sat, 6 Mar 2010, Till Maas wrote:
> >>
> >>> On Mon, Mar 01, 2010 at 10:42:32AM -0600, Mike McGrath wrote:
> >
> >>> Imho for the beginning, there is no need to be able to query complete
> >>> profiles, but it would be enough to have a count per package. A simple
> >>> implementation for this would be:
> >>>
> >>> 1) clients send a plaintext list of installed packages and a UUID every
> >>> X days or by user request
> >>> 2) file is stored in UUID.$timestamp (or it is stored as a BLOB in the
> >>> DB)
> >>> 3) once a day a crawler reads all files and counts for each package how
> >>> often they are installed,
>
> What about uninstalled?
>
> Update bring in upd to X, but package Y,Z. come in.
> User removes Y,Z without breaking anything.

I am not really sure what problem you are seeing. What is supposed to
break?

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-06-2010, 10:19 AM
Frank Murphy
 
Default Fight bugs, not FESCo

On 06/03/10 10:04, Till Maas wrote:
--snipped--
>>>>> DB)
>>>>> 3) once a day a crawler reads all files and counts for each package how
>>>>> often they are installed,
>>
>> What about uninstalled?
>>
>> Update bring in upd to X, but package Y,Z. come in.
>> User removes Y,Z without breaking anything.
>
> I am not really sure what problem you are seeing. What is supposed to
> break?
>
> Regards
> Till
>


Take XFCE, user removes GDM install Slim.
GDM was installed now is not,
nothing is broken.

Will GDM being uninstalled be accouted for,
or will the install just matter?

Like XFCE update tends to pull in nautilus.

--
Regards,

Frank Murphy
UTF_8 Encoded
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 03-06-2010, 11:11 AM
Till Maas
 
Default Fight bugs, not FESCo

On Sat, Mar 06, 2010 at 11:19:27AM +0000, Frank Murphy wrote:
> On 06/03/10 10:04, Till Maas wrote:
> --snipped--
> >>>>> DB)
> >>>>> 3) once a day a crawler reads all files and counts for each package how
> >>>>> often they are installed,
> >>
> >> What about uninstalled?
> >>
> >> Update bring in upd to X, but package Y,Z. come in.
> >> User removes Y,Z without breaking anything.
> >
> > I am not really sure what problem you are seeing. What is supposed to
> > break?
> >
> > Regards
> > Till
> >
>
>
> Take XFCE, user removes GDM install Slim.
> GDM was installed now is not,
> nothing is broken.
>
> Will GDM being uninstalled be accouted for,
> or will the install just matter?

Yes, it will be honoured. The clients will send a list of installed
packages. So after GDM is uninstalled, it will not be send anymore. The
next time the crawler runs after the client updated his list, the GDM
count will be effectively decremented.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 07:07 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org