I've mainly said out of this discussion until now, because I've been
quite busy.

The root problem here is that there are starting to be a lot of cases
where rule run by udev require that /usr [1] and potentially /var [2] or
more are available when the udev rule runs.

To the best of my knowledge, udev has no way of knowing that the
relevant directory structures might not yet be available.

Moving the binaries out of /usr into / is a band-aid at best, and
potentially a giant nightmare because we have to trace down every binary
run by every rule. Then data-files also need to be considered.

There are only two solutions available, one of which has two variants.
1. Allow udev to delay running a rule until the volumes are available.
2. Ensure /usr and /var are available when udev starts, two possible
ways:
2.1. Merge /usr and /var to /.
2.2. Initramfs to mount them.

Gregkh et al can speak to #1, and I think they will concur with me that
it's not a reasonable change to put in udev (it belongs in whatever is
starting the services and mounting those directory structures).

That leaves the only reasonable solution as #2. In terms of minimal
impact, I propose that we offer users with a static system an absolutely
minimal initramfs, that _just_ mounts the required directories. No
modules, no LVM, no MD, no crypto etc - if you want that functionality,
go and use genkernel or dracut. If your fstab contains a line like:
/dev/sdXN /usr ...
Then this initramfs is for you.

The minimal initramfs would do the following.

1. Mount devtmpfs/sysfs/procfs as needed to access devices.
2. Mount real_root to /newroot
3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
4.1. If /newroot/etc/initramfs.mount does not exist
Assume it contains only: /usr /var
5. Mount the combined items from said files
6. pivot_root.

This initramfs explicitly needs to be rebuilt extremely seldom, because
it reads everything from the real root. (Having the debug interface like
genkernel initramfs would be nice in case the user breaks their fstab,
but isn't required).

There's a couple of other tweaks/improvements that might be useful, like
supporting UUIDs and LABELs for devices.

[1]:
Besides the binaries run from /usr, I've seen broken rules because they
failed USB/PCI identification, because /usr/share/misc/pci.ids and
/usr/share/misc/usb.ids were on /usr.

[2]:
These are the ones on my desktop that reference /var:
app-admin/system-config-printer-common: /lib/udev/udev-configure-printer
sys-fs/udev: /lib/udev/udev-acl
sci-geosciences/gpsd: /lib/udev/gpsd.hotplug
/lib/udev/rules.d/90-alsa-restore.rules:
- alsactl restore needs /var/lib/alsa/asound.state




On Thu, Aug 04, 2011 at 12:00:09PM -0400, Rich Freeman wrote:
> On Thu, Aug 4, 2011 at 11:46 AM, Greg KH <gregkh@gentoo.org> wrote:
> > Also, again, this is an upstream issue, based on the packages you have
> > installed, not anything that has changed in the distro itself. *Upstream
> > is also working to resolve the issue already, by mounting /usr from the
> > initramfs, to keep this sane, so people shouldn't really have to worry
> > about this too much, unless they don't use an initramfs...
>
> Agree. The news was a little shocking to me actually - I had no idea
> people were doing this. I suspect it just happened as things like
> udev went from enhancements used when you inserted something in a
> pcmcia slot to core features used to get everything from your hard
> drive to your mouse to work.
>
> The more I think about it, the more it seems like we're stuck going
> the initramfs route unless we want to become Gentoo vs the world.
> Dropping support for udev clearly isn't going to be a practical
> option, and the number of changes we'd have to make to get it and its
> dependencies out of /usr is going to be a challenge.
>
> I suspect that in the end we're either going to end up requiring
> initramfs, or we're going to end up implementing what is otherwise in
> dracut in openrc to get those drives mounted much earlier.
>
> Gentoo is a bit unusual in not requiring initramfs in the first place.
> We can get away it mostly because everybody customizes their
> kernels/grub/etc anyway. If you're a binary distro and want a one
> bzImage/grub.conf fits all then you need a fancy initramfs to make it
> work.
>
> Rich
>

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85

* Robin H. Johnson schrieb am 05.08.11 um 02:46 Uhr:
[...]
> That leaves the only reasonable solution as #2. In terms of minimal
> impact, I propose that we offer users with a static system an absolutely
> minimal initramfs, that _just_ mounts the required directories. No
> modules, no LVM, no MD, no crypto etc - if you want that functionality,
> go and use genkernel or dracut. If your fstab contains a line like:
> /dev/sdXN /usr ...
> Then this initramfs is for you.
>
> The minimal initramfs would do the following.
>
> 1. Mount devtmpfs/sysfs/procfs as needed to access devices.
> 2. Mount real_root to /newroot
> 3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
> 4.1. If /newroot/etc/initramfs.mount does not exist
> Assume it contains only: /usr /var
> 5. Mount the combined items from said files
> 6. pivot_root.
>

That sounds like a good compromise to me!

Another thing to consider:

/etc/init.d/localmount should check whats already mounted and leave
that out. But it will act as fallback if the minimal initramfs fails
to mount /usr or /var for any reason.

That way anybody migrating to that "minitramfs" will not risc an
unbootable system.

-Marc
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134

On Fri, Aug 5, 2011 at 6:16 AM, Marc Schiffbauer <mschiff@gentoo.org> wrote:
> * Robin H. Johnson schrieb am 05.08.11 um 02:46 Uhr:
> [...]
>> That leaves the only reasonable solution as #2. In terms of minimal
>> impact, I propose that we offer users with a static system an absolutely
>> minimal initramfs, that _just_ mounts the required directories. *No
>> modules, no LVM, no MD, no crypto etc - if you want that functionality,
>> go and use genkernel or dracut. If your fstab contains a line like:
>> /dev/sdXN /usr ...
>> Then this initramfs is for you.
>>
>> The minimal initramfs would do the following.
>>
>> 1. Mount devtmpfs/sysfs/procfs as needed to access devices.
>> 2. Mount real_root to /newroot
>> 3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
>> 4.1. If /newroot/etc/initramfs.mount does not exist
>> * * *Assume it contains only: /usr /var
>> 5. Mount the combined items from said files
>> 6. pivot_root.
>>
>
> That sounds like a good compromise to me!

Why would we build yet another initramfs vs just making dracut work
reliably? You can already build dracut without support for
lvm+raid+luks/etc.

If we're going to require an initramfs then we should make sure that
ALL gentoo-provided solutions work before we expand the need for a
mounted /usr. The genkernel team already mentioned that they plan to
switch to dracut, so we really just need to get dracut working
properly.

That said, nobody is preventing anybody from re-inventing the wheel if
they wish to do so. I just wouldn't just offer it up as an example of
a perfectly acceptable migration strategy, when we've had a lvm+raid
howto for years that wouldn't be compatible with it.

Rich

* Rich Freeman schrieb am 05.08.11 um 14:42 Uhr:
> On Fri, Aug 5, 2011 at 6:16 AM, Marc Schiffbauer <mschiff@gentoo.org> wrote:
> > * Robin H. Johnson schrieb am 05.08.11 um 02:46 Uhr:
> > [...]
> >> That leaves the only reasonable solution as #2. In terms of minimal
> >> impact, I propose that we offer users with a static system an absolutely
> >> minimal initramfs, that _just_ mounts the required directories. *No
> >> modules, no LVM, no MD, no crypto etc - if you want that functionality,
> >> go and use genkernel or dracut. If your fstab contains a line like:
> >> /dev/sdXN /usr ...
> >> Then this initramfs is for you.
> >>
> >> The minimal initramfs would do the following.
> >>
> >> 1. Mount devtmpfs/sysfs/procfs as needed to access devices.
> >> 2. Mount real_root to /newroot
> >> 3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
> >> 4.1. If /newroot/etc/initramfs.mount does not exist
> >> * * *Assume it contains only: /usr /var
> >> 5. Mount the combined items from said files
> >> 6. pivot_root.
> >>
> >
> > That sounds like a good compromise to me!
>
> Why would we build yet another initramfs vs just making dracut work
> reliably? You can already build dracut without support for
> lvm+raid+luks/etc.

If dracut will have some sort of minimalistic mode where it would
generate such an initrd that would be ok IMO.

OTOH the initrd that Robin described would be a very static solution
with almost no dependencies, so if genkernel had a USE flag like
"dracut" it would be possible to build it without dracut
dependency and thus would allow for smaller systems.

-Marc
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134

On Fri, Aug 5, 2011 at 7:42 AM, Rich Freeman <rich0@gentoo.org> wrote:
> On Fri, Aug 5, 2011 at 6:16 AM, Marc Schiffbauer <mschiff@gentoo.org> wrote:
>> * Robin H. Johnson schrieb am 05.08.11 um 02:46 Uhr:
>> [...]
>>> That leaves the only reasonable solution as #2. In terms of minimal
>>> impact, I propose that we offer users with a static system an absolutely
>>> minimal initramfs, that _just_ mounts the required directories. *No
>>> modules, no LVM, no MD, no crypto etc - if you want that functionality,
>>> go and use genkernel or dracut. If your fstab contains a line like:
>>> /dev/sdXN /usr ...
>>> Then this initramfs is for you.
>>>
>>> The minimal initramfs would do the following.
>>>
>>> 1. Mount devtmpfs/sysfs/procfs as needed to access devices.
>>> 2. Mount real_root to /newroot
>>> 3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
>>> 4.1. If /newroot/etc/initramfs.mount does not exist
>>> * * *Assume it contains only: /usr /var
>>> 5. Mount the combined items from said files
>>> 6. pivot_root.
>>>
>>
>> That sounds like a good compromise to me!
>
> Why would we build yet another initramfs vs just making dracut work
> reliably? *You can already build dracut without support for
> lvm+raid+luks/etc.
>
> If we're going to require an initramfs then we should make sure that
> ALL gentoo-provided solutions work before we expand the need for a
> mounted /usr. *The genkernel team already mentioned that they plan to
> switch to dracut, so we really just need to get dracut working
> properly.
>
> That said, nobody is preventing anybody from re-inventing the wheel if
> they wish to do so. *I just wouldn't just offer it up as an example of
> a perfectly acceptable migration strategy, when we've had a lvm+raid
> howto for years that wouldn't be compatible with it.
>
> Rich
>
>

In point of fact all modern Linux kernels have an initramfs built in
now, that when empty is effectively bypassed, so there is no wheel
reinvention. To quote the docs [1]
"
All 2.6 Linux kernels contain a gzipped "cpio" format archive, which is
extracted into rootfs when the kernel boots up. After extracting, the kernel
checks to see if rootfs contains a file "init", and if so it executes it as PID
1. If found, this init process is responsible for bringing the system the
rest of the way up, including locating and mounting the real root device (if
any). If rootfs does not contain an init program after the embedded cpio
archive is extracted into it, the kernel will fall through to the older code
to locate and mount a root partition, then exec some variant of /sbin/init
out of that.
"
[1] /usr/src/linux/Documentation/filesystems/ramfs-rootfs-initramfs.txt

While dracut may help with the process of creating the initramfs, its
really not a terribly complicated endeavor, and from further reading
(specifically as related to the desired removal of md autodetection),
it is the full intention of the kernel upstream that initramfs be the
new path forward in handling things such as separate /usr, raid
volumes, and so on. Further, dracut will introduce yet another dep in
base-system (I am guessing here), that is not more than perhaps a
convenience. I note here, that I have not used dracut as the well
documented method of creating an initramfs is straight forward enough
that I did not require anything too fancy to handle mounting raid1
volumes, and providing a recovery environment with networking and ssh.

This, at least to me, seems like an excellent opportunity to nicely
document what can be done with an initramfs (in basic and advanced
forms, as there are some really fancy things one can do with
initramfs's), and how Gentoo is recommending their usage in the cases
outlined by Robin and others.

So, I am +1 on robbat2's solution and -1 on dracut. That said, I am
fully willing to alter my position when presented with a better
argument. Lastly, I do have a few minor concerns about how this
"default" initramfs will be dealt with, however those can wait.
Mainly, I simply desire the flexibility to replace the Gentoo-shipped
version with a custom version easily.

Thanks and kind regards,
Matt

--
Matthew W. Summers
Gentoo Foundation Inc.

On 08/05/2011 04:12 PM, Marc Schiffbauer wrote:
> * Rich Freeman schrieb am 05.08.11 um 14:42 Uhr:
>> On Fri, Aug 5, 2011 at 6:16 AM, Marc Schiffbauer <mschiff@gentoo.org> wrote:
>>> * Robin H. Johnson schrieb am 05.08.11 um 02:46 Uhr:
>>> [...]
>>>> That leaves the only reasonable solution as #2. In terms of minimal
>>>> impact, I propose that we offer users with a static system an absolutely
>>>> minimal initramfs, that _just_ mounts the required directories. No
>>>> modules, no LVM, no MD, no crypto etc - if you want that functionality,
>>>> go and use genkernel or dracut. If your fstab contains a line like:
>>>> /dev/sdXN /usr ...
>>>> Then this initramfs is for you.
>>>>
>>>> The minimal initramfs would do the following.
>>>>
>>>> 1. Mount devtmpfs/sysfs/procfs as needed to access devices.
>>>> 2. Mount real_root to /newroot
>>>> 3. Read /newroot/etc/initramfs.mount and /newroot/etc/fstab
>>>> 4.1. If /newroot/etc/initramfs.mount does not exist
>>>> Assume it contains only: /usr /var
>>>> 5. Mount the combined items from said files
>>>> 6. pivot_root.
>>>>
>>>
>>> That sounds like a good compromise to me!
>>
>> Why would we build yet another initramfs vs just making dracut work
>> reliably? You can already build dracut without support for
>> lvm+raid+luks/etc.
>
> If dracut will have some sort of minimalistic mode where it would
> generate such an initrd that would be ok IMO.
>
> OTOH the initrd that Robin described would be a very static solution
> with almost no dependencies, so if genkernel had a USE flag like
> "dracut" it would be possible to build it without dracut
> dependency and thus would allow for smaller systems.
>
> -Marc

To clarify,

By dependencies in dracut you mean udev? And by smaller systems you mean
systems without udev?

Then yes, such minimal initramfs should propably be covered in the
embedded's documentation, but otherwise trying to avoid dracut is
reinventing the wheel...

On Fri, Aug 5, 2011 at 9:25 AM, Matthew Summers
<quantumsummers@gentoo.org> wrote:
> In point of fact all modern Linux kernels have an initramfs built in
> now, that when empty is effectively bypassed, so there is no wheel
> reinvention. To quote the docs [1]

Yes, but that embedded initramfs doesn't actually do much of anything
right now. It may exist from an architecture standpoint, but it
doesn't mean that there isn't a lot of work to be done which would be
redundant with work already done elsewhere.

I can see where you might want a simpler solution than dracut as an
alternative for more embedded setups/etc. It probably would be
cleaner to just create a new solution than modify dracut to get rid of
udev/etc.

In any case, as long as a solution exists for md+lvm+luks+/usr before
we start breaking more stuff than is already broken, then we should be
fine. Having more than one optional solution is fine. While I don't
think that gentoo needs to be another Ubuntu, having reasonable
out-of-the-box support for one of the major desktop environments
running on md+lvm+luks seems pretty useful.

Rich

On Fri, Aug 05, 2011 at 08:25:19AM -0500, Matthew Summers wrote:
> This, at least to me, seems like an excellent opportunity to nicely
> document what can be done with an initramfs (in basic and advanced
> forms, as there are some really fancy things one can do with
> initramfs's), and how Gentoo is recommending their usage in the cases
> outlined by Robin and others.

I'm all in favor of documenting what an initramfs does (or at least what it
is supposed to do), how it works, how to create one, how to debug issues
while booting with one, etc.

That said, I'm a bit hesitant to describing that we "recommend" it
regardless of the situation. What is wrong with describing when? At least
inform our users that the udev rules have evolved to more than just "detect
and mknod" scripts and that they are now relying on files and binaries
available in other locations, like /usr and /var.

How does the tool that creates an initramfs know which files to copy from
/usr and /var anyhow?

Also, how well does this play with all our profiles (so not only the popular
architectures)? What about SELinux and/or grSecurity's RBAC model? Are these
supported throughout the initramfs?

Wkr,
Sven Vermeulen

On Fri, Aug 05, 2011 at 09:57:08AM -0400, Rich Freeman wrote:
> In any case, as long as a solution exists for md+lvm+luks+/usr before
> we start breaking more stuff than is already broken, then we should be
> fine. Having more than one optional solution is fine. While I don't
> think that gentoo needs to be another Ubuntu, having reasonable
> out-of-the-box support for one of the major desktop environments
> running on md+lvm+luks seems pretty useful.

My proposed minimal initramfs is an alternative to dracut for users with
trivial systems. For _ALL_ of the other cases, genkernel/dracut should
be used. Patches to add any of that other support to the minimal
initramfs will be rejected.

Definition of a trivial system:
/, /usr, /var are contained on or within device nodes available in
devtmpfs at the time the kernel invokes the initramfs.

Examples of non-trivial systems:
- {/, /usr, /var} on any of LVM, LUKS, NFS, iSCSI
- {/, /usr, /var} on MD that is not auto-assembled by the kernel [1].
- etc.

The initramfs would be rebuilt extremely infrequently, since it doesn't
contain any kernel modules, and reads the actual configuration from the
real root fs.

[1]: Upstream MD kernel folk have said everybody wanting to do MD
detection should migrate to initramfs for detection, as there at lot of
corner cases in detection where userspace is needed to make a sane
decision. Also if you use non-msdos partition tables, or large MDs.

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85

Hi,

my knowledge of booting from an initramfs is limited right now, so keep
that in mind. However, I will attempt to answer some of your questions.

On Fri, Aug 05, 2011 at 10:06:48PM +0200, Sven Vermeulen wrote:
> I'm all in favor of documenting what an initramfs does (or at least what it
> is supposed to do), how it works, how to create one, how to debug issues
> while booting with one, etc.
>
> That said, I'm a bit hesitant to describing that we "recommend" it
> regardless of the situation. What is wrong with describing when? At least
> inform our users that the udev rules have evolved to more than just "detect
> and mknod" scripts and that they are now relying on files and binaries
> available in other locations, like /usr and /var.

It looks like the situation where we will have to have one is if /usr
and /var are not on the same file system as /, because of how udev has
evolved.

> How does the tool that creates an initramfs know which files to copy from
> /usr and /var anyhow?

My understanding is that nothing gets copied from /usr and /var, and it
doesn't have to.

Here is my basic understanding of how the boot sequence works:

1) rootfs is mounted on /. Rootfs contains the contents of the
initramfs.
2) the init program inside the initramfs is run.
3) This init program will mount the real /usr and /var.
4) This init program will then mount the real root on top of rootfs.
(there is a call that does this, but I'll need to look that up again). I
vaguely remember reading that you don't mount the real root on /newroot
then pivet_root as mentioned earlier in this thread; I think that is
part of the old initrd method.
5) This init program will now run /sbin/init or which ever init program
the user wants to run.

> Also, how well does this play with all our profiles (so not only the popular
> architectures)? What about SELinux and/or grSecurity's RBAC model? Are these
> supported throughout the initramfs?

This one I can't answer.

Regards,

William