Testing of filesystem identification code (blkid/vol_id)
 

I'd appreciate some testing of the following packages in my PPA:

deb http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main
deb-src http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main

<https://launchpad.net/~scott/+archive/ppa>

util-linux 2.15~rc1-1ubuntu1~blkid3
libblkid1
libblkid-dev
bsdutils
mount
util-linux
util-linux-locales

e2fsprogs 1.41.4-1ubuntu2~blkid1
e2fsprogs
e2fslibs
libuuid1
uuid-dev
uuid-runtime
libss2
ss-dev
libcomerr2
comerr-dev
e2fsck-static

udev 140-2~blkid1
udev
libudev0
libudev-dev

devmapper 2:1.02.27-4ubuntu5~blkid1
dmsetup
dmeventd
libdevmapper1.02.1
libdevmapper-dev
libdevmapper-event1.02.1

mdadm 2.6.7.1-1ubuntu8~blkid1
mdadm

initramfs-tools 0.92bubuntu26~blkid1
initramfs-tools


Obviously, only upgrade those you have installed. All the version
numbers are greater than in jaunty at the time of mailing, so APT should
do the right thing.

If your computer works after a reboot, then everything's OK :-)


Background:

Up until now, we've had two different libraries and tools for examining
the content of block devices and finding out information about the
filesystem contained therein. Most particularly, the UUID, LABEL and
TYPE of that filesystem.

The first and oldest of these is blkid/libblkid and comes with the
e2fsprogs source package, which contains the tools for the ext2, etc.
filesystem.

The second and newest of these is vol_id/libvolume-id and comes with the
udev source package.

Why we have two at all is a bit of a story, but fundamentally both
libraries behave slightly differently and the maintainers never
persuaded each other to adopt or integrate the other view.

blkid is designed to answer the question "which filesystem has UUID or
LABEL foo?". It keeps an on-disk cache of known mappings, and if you
ask for a mapping it doesn't know, it scans every block device on the
system looking for it.

vol_id is designed to answer the question "what is in this block
device?". You give it a known block device, and it tells you what's in
it.


At first, this might seem ok, except that both have different levels of
filesystem coverage (vol_id has more wide coverage, including of RAID
and LVM; blkid knows about different versions of ext4). Not only that,
but sometimes a block device will have the metadata of two different
filesystems (due to a bad format), and each might give a different
answer as to which is the right one.

So we should just pick one, right?

That's what we've been doing up until now, but it hasn't gone quite
smoothly.


Since udev knows when devices are added to and removed from the system,
manages the /dev tree, and runs programs such as LVM and mdadm, it also
logically probes the filesystem content.

Thus it needs a tool that behaves like vol_id (indeed, this is why
vol_id was written and why it's in the udev package). HAL also uses the
libvolume-id library for its probing.

But vol_id doesn't provide the reverse lookup, however udev uses its
information to maintain the /dev/disk/by-uuid and /dev/disk/by-label
directories of symlinks.

So blkid UUID and LABEL lookups could be replaced by filesystem path
lookups.

That's what we've done until now; our versions of the tools in
util-linux (mount, swapon, etc.) were patched to translate UUID= and
LABEL= lookups into /dev/disk/by-{uuid,label}/* path lookups.

initramfs-tools did the same thing.

Perhaps irritatingly, our fsck tool comes from e2fsprogs and was not
patched. Likewise initscripts called "findfs" which comes from blkid.

So we had some bugs where the two disagreed.

Also this has a shortcoming that the /dev/disk tree is only updated when
udev sees a change in the physical block device. Running dd, mkfs,
mkswap, etc. on a block device does not update the symlinks, even though
the UUID and LABEL may have changed.


So there's been a number of parallel efforts to truly fix this.

Firstly udev now watches block devices; if they are written to, it acts
as if there has been a change to the physical block device. Thus
the /dev/disk/by-uuid and /dev/disk/by-label symlinks *are* now changed
by dd, mkfs, mkswap, etc.

This is already in jaunty.


The next change is what's in the PPA.

The libblkid library from e2fsprogs and the libvolume-id library from
udev have been merged. This new library, still called libblkid, and
backwards compatible in API with it, is part of util-linux-ng (our
util-linux source package).

This newly merged library supports the old libblkid cache API, and also
supports a probe a single block device API which is along similar lines
to the old libvolume-id API (but software[0] does need patching).

Along with the library, the blkid and findfs tools have been moved to
util-linux-ng as well. blkid supports a new "-p" (probe) option for
probing a single block device. Also when performing a simple UUID or
LABEL to device lookup, both tools now first check udev's symlinks
rather than scanning.

The various tools in util-linux-ng (mount, swapon, etc.) all use this
new library.

Finally fsck itself has been moved from e2fsprogs to util-linux, and
uses this new library as well.


So we now have a single filesystem probing library and tool.


Along with the updates to util-linux, my PPA contains updates to udev,
dmsetup and mdadm that change the rules to run "blkid" instead of
"vol_id".

It also contains an update to initramfs-tools to use blkid in all of the
places it previously used vol_id. Mostly these are to lookup the UUID
of a given filesystem; but also in the root mounting code to check that
a block device contains a valid filesystem.


HAL has not yet been patched, it will still use libvolume-id.

Scott

[0] basically, only HAL
--
Scott James Remnant
scott@canonical.com
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Testing of filesystem identification code (blkid/vol_id)
 

FYI, my amd64 box survived a reboot...I have enough issues on my Thinkpad X301,
so I'll wait for someone else to test it. ;)

-Robbie

On 03/24/2009 12:39 PM, Scott James Remnant wrote:
> I'd appreciate some testing of the following packages in my PPA:
>
> deb http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main
> deb-src http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main
>
> <https://launchpad.net/~scott/+archive/ppa>
>
> util-linux 2.15~rc1-1ubuntu1~blkid3
> libblkid1
> libblkid-dev
> bsdutils
> mount
> util-linux
> util-linux-locales
>
> e2fsprogs 1.41.4-1ubuntu2~blkid1
> e2fsprogs
> e2fslibs
> libuuid1
> uuid-dev
> uuid-runtime
> libss2
> ss-dev
> libcomerr2
> comerr-dev
> e2fsck-static
>
> udev 140-2~blkid1
> udev
> libudev0
> libudev-dev
>
> devmapper 2:1.02.27-4ubuntu5~blkid1
> dmsetup
> dmeventd
> libdevmapper1.02.1
> libdevmapper-dev
> libdevmapper-event1.02.1
>
> mdadm 2.6.7.1-1ubuntu8~blkid1
> mdadm
>
> initramfs-tools 0.92bubuntu26~blkid1
> initramfs-tools
>
>
> Obviously, only upgrade those you have installed. All the version
> numbers are greater than in jaunty at the time of mailing, so APT should
> do the right thing.
>
> If your computer works after a reboot, then everything's OK :-)
>
>
> Background:
>
> Up until now, we've had two different libraries and tools for examining
> the content of block devices and finding out information about the
> filesystem contained therein. Most particularly, the UUID, LABEL and
> TYPE of that filesystem.
>
> The first and oldest of these is blkid/libblkid and comes with the
> e2fsprogs source package, which contains the tools for the ext2, etc.
> filesystem.
>
> The second and newest of these is vol_id/libvolume-id and comes with the
> udev source package.
>
> Why we have two at all is a bit of a story, but fundamentally both
> libraries behave slightly differently and the maintainers never
> persuaded each other to adopt or integrate the other view.
>
> blkid is designed to answer the question "which filesystem has UUID or
> LABEL foo?". It keeps an on-disk cache of known mappings, and if you
> ask for a mapping it doesn't know, it scans every block device on the
> system looking for it.
>
> vol_id is designed to answer the question "what is in this block
> device?". You give it a known block device, and it tells you what's in
> it.
>
>
> At first, this might seem ok, except that both have different levels of
> filesystem coverage (vol_id has more wide coverage, including of RAID
> and LVM; blkid knows about different versions of ext4). Not only that,
> but sometimes a block device will have the metadata of two different
> filesystems (due to a bad format), and each might give a different
> answer as to which is the right one.
>
> So we should just pick one, right?
>
> That's what we've been doing up until now, but it hasn't gone quite
> smoothly.
>
>
> Since udev knows when devices are added to and removed from the system,
> manages the /dev tree, and runs programs such as LVM and mdadm, it also
> logically probes the filesystem content.
>
> Thus it needs a tool that behaves like vol_id (indeed, this is why
> vol_id was written and why it's in the udev package). HAL also uses the
> libvolume-id library for its probing.
>
> But vol_id doesn't provide the reverse lookup, however udev uses its
> information to maintain the /dev/disk/by-uuid and /dev/disk/by-label
> directories of symlinks.
>
> So blkid UUID and LABEL lookups could be replaced by filesystem path
> lookups.
>
> That's what we've done until now; our versions of the tools in
> util-linux (mount, swapon, etc.) were patched to translate UUID= and
> LABEL= lookups into /dev/disk/by-{uuid,label}/* path lookups.
>
> initramfs-tools did the same thing.
>
> Perhaps irritatingly, our fsck tool comes from e2fsprogs and was not
> patched. Likewise initscripts called "findfs" which comes from blkid.
>
> So we had some bugs where the two disagreed.
>
> Also this has a shortcoming that the /dev/disk tree is only updated when
> udev sees a change in the physical block device. Running dd, mkfs,
> mkswap, etc. on a block device does not update the symlinks, even though
> the UUID and LABEL may have changed.
>
>
> So there's been a number of parallel efforts to truly fix this.
>
> Firstly udev now watches block devices; if they are written to, it acts
> as if there has been a change to the physical block device. Thus
> the /dev/disk/by-uuid and /dev/disk/by-label symlinks *are* now changed
> by dd, mkfs, mkswap, etc.
>
> This is already in jaunty.
>
>
> The next change is what's in the PPA.
>
> The libblkid library from e2fsprogs and the libvolume-id library from
> udev have been merged. This new library, still called libblkid, and
> backwards compatible in API with it, is part of util-linux-ng (our
> util-linux source package).
>
> This newly merged library supports the old libblkid cache API, and also
> supports a probe a single block device API which is along similar lines
> to the old libvolume-id API (but software[0] does need patching).
>
> Along with the library, the blkid and findfs tools have been moved to
> util-linux-ng as well. blkid supports a new "-p" (probe) option for
> probing a single block device. Also when performing a simple UUID or
> LABEL to device lookup, both tools now first check udev's symlinks
> rather than scanning.
>
> The various tools in util-linux-ng (mount, swapon, etc.) all use this
> new library.
>
> Finally fsck itself has been moved from e2fsprogs to util-linux, and
> uses this new library as well.
>
>
> So we now have a single filesystem probing library and tool.
>
>
> Along with the updates to util-linux, my PPA contains updates to udev,
> dmsetup and mdadm that change the rules to run "blkid" instead of
> "vol_id".
>
> It also contains an update to initramfs-tools to use blkid in all of the
> places it previously used vol_id. Mostly these are to lookup the UUID
> of a given filesystem; but also in the root mounting code to check that
> a block device contains a valid filesystem.
>
>
> HAL has not yet been patched, it will still use libvolume-id.
>
> Scott
>
> [0] basically, only HAL
>


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Testing of filesystem identification code (blkid/vol_id)
 

On Tuesday 24 March 2009 18:39:00 Scott James Remnant wrote:
> I'd appreciate some testing of the following packages in my PPA:
>
> deb http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main
> deb-src http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main

Boots and works fine on x200s with /boot on primary partition, / on plain LVM
and /home on LVM on top of dm-crypt volume. All filesystems are ext4.

Regards,
Luka

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Testing of filesystem identification code (blkid/vol_id)
 

Hi,

On Tue, Mar 24, 2009 at 05:39:00PM +0000, Scott James Remnant wrote:
> I'd appreciate some testing of the following packages in my PPA:
>
> deb http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main
> deb-src http://ppa.launchpad.net/scott/ppa/ubuntu jaunty main

> Along with the updates to util-linux, my PPA contains updates to udev,
> dmsetup and mdadm that change the rules to run "blkid" instead of
> "vol_id".

I gave this a shot, and it was bad news for my configuration (LVM
root on RAID1 on SATA). It seems that blkid hung, (each of the
about 12? 16? instances) spinning at 100% CPU, and never brought up
the md devices. Once I brought them up manually, LVM was activated
without a hitch. Due to the hung blkids, I assume udevsettle failed
to ever finish, so my system hung again at "loading hardware drivers".
After getting past that[1], I manually mounted everything, killed the
blkids, and downgraded to stock jaunty again for now. :)

-Kees

[1] I hit ctrl-alt-del, and that seemed to continue the boot, rather than
rebooting... is that expected behavior from upstart?

--
Kees Cook
Ubuntu Security Team

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel