FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 08-11-2008, 03:40 PM
Jeroen van Meeuwen
 
Default Default Groups, Default Packages, CD Sets and DVD Size

Hi there,

there's a number of issues which you may or may not agree with, and are
very real concerns. I'll first try to explain what the issue is;


The CD sets in Fedora 10 Alpha, but also Fedora 8 and 9 Re-Spins,
require all discs in order to be installed. I say installed, but what I
really mean is next, next, next, finish. Eg., this includes the "Office
and Productivity" shortcut to the office, games and sound-and-video
comps groups of packages.


This problem may need a little more detailed description of 1) CD sets
are composed, and 2) how the installation procedures selects
groups/packages to install -in a default install.


FWIW, when I use the term "default install" in this email, it defines
"next, next, finish" without checking or unchecking any package or group
related settings.


Also, FWIW, I'm not putting in all the details in composing- and
installing-logic, so that only the bare flow of logic related to the
problem set forth remains (or at least I'm attempting to do so).


== The compose ==

Compose tools get input via a kickstart package manifest, which
basically describes what groups and packages should be available during
the installation. It selects all the relevant groups and packages,
resolves the dependencies (inclusive or exclusive -see PS.), puts these
in a tree and may or may not create a DVD from that tree. In the case of
CD sets however, the packages need to be split over multiple discs and
here's how that works (the process is called package ordering for those
of you who didn't know that already):


From a list of groups, the compose tools first select @core and @base
since these are most commonly (e.g. "always") installed. It resolves
dependencies for the packages selected (e.g. the mandatory and default
packages in the groups @core and @base). The transaction is ordered and
the order in which yum will want to install the packages is spit out.


Then, another set of groups is selected, depsolved, ordered is if it
were for a real transaction, and spit out in the correct order.


This continues until all groups the compose tool knows about are
selected and spit out, and the package ordering process will then spit
out "the remainder of packages".


Note that the next generation of compose tools is going to change this
package ordering process to match the behavior of the installation
procedure, but I'll continue with more about that later.


== The installation procedure ==

By the time the installation procedure has gained the necessary input
wrt. which packages to install, it has also selected a number of default
groups (as well as "Office and Productivity" when performing a default
installation). For this purpose, it uses the "default" (True/False)
attribute in the available -possibly aggregated from multiple
repositories, but not in this case- comps.xml. Given no additional input
during the default installation, these are the groups selected to be
installed[1] (from today's x86_64 rawhide):


-> office
-> admin-tools
-> editors
-> input-methods
-> fonts
-> text-internet
-> gnome-desktop
-> core
-> base
-> hardware-support
-> games
-> java
-> base-x
-> graphics
-> dial-up
-> printing
-> sound-and-video
-> graphical-internet

Resulting in a required RPM payload (on the media) of 2.88 GB, using
exclusive dependency resolving.


2.88 GB in RPMs spans 5 CDs. This means, that a default installation of
Fedora Rawhide today, when using CDs, would require 5 discs minimum.


== However ==

However, the default installation requires all 7 CDs, because of how the
compose tools resolve dependencies (inclusive) during the compose of the
media, and pull in more then is minimally required to complete the
actual transaction of a default installation. The compose tools do so
for good reason:


- one cannot know what package is the user-preferred package for any
given required capability (eg. for fictive capability 'web-client',
there's firefox, iceweasel, elinks, wget, curl, emacs, emacs, emacs,
foo, bar and baz)


- one cannot predict on what installed system one is performing an
upgrade, and to be able to close the transaction certain considerations
must be met justifying the need for inclusive dependency resolving when
composing the media (set) or installation tree.


- it makes the released media apply to N+X use-cases where the package
set or transaction payload during the installation is controlled in a
more granular fashion then the selection dialogs allow (by means of a
kickstart package manifest maybe?), which I guess applies more to
businesses or advanced users using Fedora then it does to Joe Average users.


Basically what I'm saying is that the 2.88 GB is spread over more then
the minimal amount of discs it would fit on, since the complete package
payload when using inclusive dependency resolving grows to 3.66 GB, or 7
discs. So, the compose process spits out 7 discs, each of which contain
a part of the 2.88 GB sized RPM payload needed for a default installation.


== Next Generation of Package Ordering ==

So, what package ordering is going to do -instead of having a static
list of groups to add to a transaction, resolve, spit out the packages-
is use the "default" parameter to groups in comps.xml as well (and then
instead of exclusive dependency resolving like it does now, move to
inclusive dependency resolving as well). This makes the "which packages
and groups are in a default installation" a little less hard to maintain
and the package ordering will almost automagically match up with
comps.xml (of which the installation procedure also uses the default
parameter to groups!)


== DVD and Sizes ==

This leads me to another concern which may or may not be an immediate
issue but requires attention from those in the decision making chain as
well as generally interested people; the size of the DVD ISO is getting
to it's maximum allowed size (just under 4GB for those who use FAT
systems as their downloaded data partition), providing just a default
(eg. not including anything in addition to the default).


== Advise needed ==

There's several ways this can be solved, but I'm not sure what is the
most advisable (some are not feasible I'm sure, I'm just brainstorming
here):


1) reduce the number of mandatory and default packages per group in
comps.xml


2) reduce the number of groups in comps.xml that have "default" set to True

3) Revisit how comps is formatted; Example: Keep the "default" for
compose decisions, but add an "install" attribute for installation
decision making. Install set to True may require default set to True as
well for the group to even be included on the media.


4) Split the packages that are mandatory or default in comps groups,
into smaller packages providing what the group needs and another set of
smaller packages belonging to the group as to reduce the number of
dependencies needing to be met when the compose or installation
procedure selects a group. See also PS2.


5) Or, compared to 4, revisit the Requires in mandatory and default
packages and the Provides in the packages that provide the required
capabilities so that it abstracts from the requires/provides matching
with too many other packages (related to the inclusive dependency
resolving which will then make for a thinner RPM payload on the composed
media)


6) Have the compose tools as well as the installation procedures not
depend on the default attribute to groups anymore, at all.


Thank you very much for reading this message so far, and please don't
hesitate to ask any questions if I wasn't clear enough in how this works
(though I also hope I didn't overdo it for those who already knew) or
make a remark or two when you think I'm wrong about something ;-)


Ideas and insight on the topic very much appreciated,

Kind regards,

Jeroen van Meeuwen
-kanarip

PS. Inclusive dependency resolving is grabbing _all_ packages that
provide a required capability. Exclusive dependency resolving is
grabbing the one best fit (this is YUM dependency resolving). I'm taking
a few shortcuts here but I hope that's OK.


PS2. Selecting all 18 default groups in rawhide results in 79 mandatory
packages, 302 default packages, and (after depsolving) 1386 packages in
total, being a 3.66 GB payload


[1] the find-default-groups.py run against a rawhide yum configuration
[jmeeuwen@ghandalf scripts]$ ./find-default-groups.py -c
../unity/conf/conf.d/revisor-rawhide-x86_64-respin.conf -r

Default groups:
-> office
-> admin-tools
-> editors
-> input-methods
-> fonts
-> text-internet
-> gnome-desktop
-> core
-> base
-> hardware-support
-> games
-> java
-> base-x
-> graphics
-> dial-up
-> printing
-> sound-and-video
-> graphical-internet
3.66 GB (3934874324 archive_size)
79 mandatory packages, 302 default packages, 1386 packages in total
(after depsolving)


The actual script can be found at
https://fedorahosted.org/revisor/browser/scripts/find-default-groups.py


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-13-2008, 01:49 AM
Jesse Keating
 
Default Default Groups, Default Packages, CD Sets and DVD Size

On Mon, 2008-08-11 at 17:40 +0200, Jeroen van Meeuwen wrote:
> However, the default installation requires all 7 CDs, because of how the
> compose tools resolve dependencies (inclusive) during the compose of the
> media, and pull in more then is minimally required to complete the
> actual transaction of a default installation. The compose tools do so
> for good reason:
>
> - one cannot know what package is the user-preferred package for any
> given required capability (eg. for fictive capability 'web-client',
> there's firefox, iceweasel, elinks, wget, curl, emacs, emacs, emacs,
> foo, bar and baz)
>
> - one cannot predict on what installed system one is performing an
> upgrade, and to be able to close the transaction certain considerations
> must be met justifying the need for inclusive dependency resolving when
> composing the media (set) or installation tree.
>
> - it makes the released media apply to N+X use-cases where the package
> set or transaction payload during the installation is controlled in a
> more granular fashion then the selection dialogs allow (by means of a
> kickstart package manifest maybe?), which I guess applies more to
> businesses or advanced users using Fedora then it does to Joe Average users.
>
> Basically what I'm saying is that the 2.88 GB is spread over more then
> the minimal amount of discs it would fit on, since the complete package
> payload when using inclusive dependency resolving grows to 3.66 GB, or 7
> discs. So, the compose process spits out 7 discs, each of which contain
> a part of the 2.88 GB sized RPM payload needed for a default installation.

I've been playing a bit in this area, and noticed that if I make one
small change to pkgorder, at least on i386, a default 'next next next'
install based on a compose done with the Fedora config and it only needs
the first 3 media. This isn't bad really, given the growth of the
distro.

>
> == Next Generation of Package Ordering ==
>
> So, what package ordering is going to do -instead of having a static
> list of groups to add to a transaction, resolve, spit out the packages-
> is use the "default" parameter to groups in comps.xml as well (and then
> instead of exclusive dependency resolving like it does now, move to
> inclusive dependency resolving as well). This makes the "which packages
> and groups are in a default installation" a little less hard to maintain
> and the package ordering will almost automagically match up with
> comps.xml (of which the installation procedure also uses the default
> parameter to groups!)

Hrm, I hadn't thought about applying inclusive depsolving (or
depresolving as I call it) here. That could have some interesting
effects and warrants some investigation.

>
> == DVD and Sizes ==
>
> This leads me to another concern which may or may not be an immediate
> issue but requires attention from those in the decision making chain as
> well as generally interested people; the size of the DVD ISO is getting
> to it's maximum allowed size (just under 4GB for those who use FAT
> systems as their downloaded data partition), providing just a default
> (eg. not including anything in addition to the default).

It's worse than that. We overflowed the DVD size on PPC, and as such we
added a set of excludes to the manifest so that we would fit again.

> == Advise needed ==
>
> There's several ways this can be solved, but I'm not sure what is the
> most advisable (some are not feasible I'm sure, I'm just brainstorming
> here):
>
> 1) reduce the number of mandatory and default packages per group in
> comps.xml

This is a great start, something that hasn't really been done outside of
core/base recently that I'm aware of.

>
> 2) reduce the number of groups in comps.xml that have "default" set to True

This one is going to be a bit tougher to get people to agree upon I'd
imagine, but I'm willing to give it a go.

>
> 3) Revisit how comps is formatted; Example: Keep the "default" for
> compose decisions, but add an "install" attribute for installation
> decision making. Install set to True may require default set to True as
> well for the group to even be included on the media.

What exactly does this accomplish?

>
> 4) Split the packages that are mandatory or default in comps groups,
> into smaller packages providing what the group needs and another set of
> smaller packages belonging to the group as to reduce the number of
> dependencies needing to be met when the compose or installation
> procedure selects a group. See also PS2.
>
> 5) Or, compared to 4, revisit the Requires in mandatory and default
> packages and the Provides in the packages that provide the required
> capabilities so that it abstracts from the requires/provides matching
> with too many other packages (related to the inclusive dependency
> resolving which will then make for a thinner RPM payload on the composed
> media)

This is probably a more interesting case, something that we want to do
for folks like OLPC as well.

>
> 6) Have the compose tools as well as the installation procedures not
> depend on the default attribute to groups anymore, at all.

What does this accomplish?

--
Jesse Keating
Fedora -- Freedom˛ is a feature!
identi.ca: http://identi.ca/jkeating
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-14-2008, 09:38 PM
Jeroen van Meeuwen
 
Default Default Groups, Default Packages, CD Sets and DVD Size

Jesse Keating wrote:

On Mon, 2008-08-11 at 17:40 +0200, Jeroen van Meeuwen wrote:
However, the default installation requires all 7 CDs, because of how the
compose tools resolve dependencies (inclusive) during the compose of the
media, and pull in more then is minimally required to complete the
actual transaction of a default installation. The compose tools do so
for good reason:


- one cannot know what package is the user-preferred package for any
given required capability (eg. for fictive capability 'web-client',
there's firefox, iceweasel, elinks, wget, curl, emacs, emacs, emacs,
foo, bar and baz)


- one cannot predict on what installed system one is performing an
upgrade, and to be able to close the transaction certain considerations
must be met justifying the need for inclusive dependency resolving when
composing the media (set) or installation tree.


- it makes the released media apply to N+X use-cases where the package
set or transaction payload during the installation is controlled in a
more granular fashion then the selection dialogs allow (by means of a
kickstart package manifest maybe?), which I guess applies more to
businesses or advanced users using Fedora then it does to Joe Average users.


Basically what I'm saying is that the 2.88 GB is spread over more then
the minimal amount of discs it would fit on, since the complete package
payload when using inclusive dependency resolving grows to 3.66 GB, or 7
discs. So, the compose process spits out 7 discs, each of which contain
a part of the 2.88 GB sized RPM payload needed for a default installation.


I've been playing a bit in this area, and noticed that if I make one
small change to pkgorder, at least on i386, a default 'next next next'
install based on a compose done with the Fedora config and it only needs
the first 3 media. This isn't bad really, given the growth of the
distro.



I'm extremely curious what the change is as I've been digging through
this for hours.



== Next Generation of Package Ordering ==

So, what package ordering is going to do -instead of having a static
list of groups to add to a transaction, resolve, spit out the packages-
is use the "default" parameter to groups in comps.xml as well (and then
instead of exclusive dependency resolving like it does now, move to
inclusive dependency resolving as well). This makes the "which packages
and groups are in a default installation" a little less hard to maintain
and the package ordering will almost automagically match up with
comps.xml (of which the installation procedure also uses the default
parameter to groups!)


Hrm, I hadn't thought about applying inclusive depsolving (or
depresolving as I call it) here. That could have some interesting
effects and warrants some investigation.



I've done this investigation and I found the following:

Put to extremes, just to get the point across without nitpicking
percentages, in principle the inclusive dependency resolving during
package ordering results in a "tighter" set of packages on, say, disc
1-3, including the packages not used by a default install, but
potentially used by a relatively large number of non-default installs
and upgrades, whereas using exclusive dependency resolving during
package ordering supposedly stuffs the default install onto a minimal
number of discs (say, 1 & 2), but you might need disc 5, 6 & 7 if you
make a small change in what is going to be installed.



== Advise needed ==

There's several ways this can be solved, but I'm not sure what is the
most advisable (some are not feasible I'm sure, I'm just brainstorming
here):


1) reduce the number of mandatory and default packages per group in
comps.xml


This is a great start, something that hasn't really been done outside of
core/base recently that I'm aware of.


2) reduce the number of groups in comps.xml that have "default" set to True


This one is going to be a bit tougher to get people to agree upon I'd
imagine, but I'm willing to give it a go.



Do we know, per group or overall, who can do this?

3) Revisit how comps is formatted; Example: Keep the "default" for
compose decisions, but add an "install" attribute for installation
decision making. Install set to True may require default set to True as
well for the group to even be included on the media.


What exactly does this accomplish?



Having "defaults" included on the media but having "installs" being
selected by the installation procedure so that the compose process can
put them on the first few discs. Does that make sense? It still requires
reducing the groups installed by default though.


4) Split the packages that are mandatory or default in comps groups,
into smaller packages providing what the group needs and another set of
smaller packages belonging to the group as to reduce the number of
dependencies needing to be met when the compose or installation
procedure selects a group. See also PS2.


5) Or, compared to 4, revisit the Requires in mandatory and default
packages and the Provides in the packages that provide the required
capabilities so that it abstracts from the requires/provides matching
with too many other packages (related to the inclusive dependency
resolving which will then make for a thinner RPM payload on the composed
media)


This is probably a more interesting case, something that we want to do
for folks like OLPC as well.

6) Have the compose tools as well as the installation procedures not
depend on the default attribute to groups anymore, at all.


What does this accomplish?



Not so much it'd sorta obsolete the default attribute (iirc the
compose/ordering/install is the very purpose of the default attr. isn't
it?).


-Jeroen

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 

Thread Tools




All times are GMT. The time now is 12:04 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org