Hi there!

Is there an advantage in putting the portage tree on an extra partition?

Currently, I'm using reiserfs, because I read that it is efficient when
using many small files. On the other hand I also heard that it tends to
get slower with every emerge --sync.

Space is no longer an argument in these days, at least for my desktop
machine. But I would like to optimize for speed -- emerge -DputnVj
@world takes quite a while to calculate, I assume this is because so many
ebuild files have to be accessed.

Any tips on this? Does it make sense to use a special file system just
for the portage tree? What would be best? Would it help to re-create this
file system from time to time in case it gets slower with every sync? Or
wouldn't I notice a difference if I just used a big ext4 partition for
all portage related stuff?

Anyone using a compressed RAM file system for that?

Wonko

Alex Schuster wrote:
> Hi there!
>
> Is there an advantage in putting the portage tree on an extra partition?
>
> Currently, I'm using reiserfs, because I read that it is efficient when
> using many small files. On the other hand I also heard that it tends to
> get slower with every emerge --sync.
>
> Space is no longer an argument in these days, at least for my desktop
> machine. But I would like to optimize for speed -- emerge -DputnVj
> @world takes quite a while to calculate, I assume this is because so many
> ebuild files have to be accessed.
>
> Any tips on this? Does it make sense to use a special file system just
> for the portage tree? What would be best? Would it help to re-create this
> file system from time to time in case it gets slower with every sync? Or
> wouldn't I notice a difference if I just used a big ext4 partition for
> all portage related stuff?
>
> Anyone using a compressed RAM file system for that?
>
> Wonko
>
>


I have mine on its own partition. Faster, not sure but most likely. I
use ext3 for mine.

Since I am redoing my partitions, I'm looking forward to reading what
others say.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!

Miss the compile output? Hint:
EMERGE_DEFAULT_OPTS="--quiet-build=n"

Am 10.03.2012 14:30, schrieb Alex Schuster:
> Hi there!
>
> Is there an advantage in putting the portage tree on an extra partition?
>

Yes. It allows you to use a smaller and more appropriate block size like
1k or 2k which decreases internal fragmentation. It also increases
locality of data, meaning that you won't scatter your files all over
your 2TB hard disk. Ext* and co. have mechanisms to prevent this but it
still helps to enforce it.

> Currently, I'm using reiserfs, because I read that it is efficient when
> using many small files. On the other hand I also heard that it tends to
> get slower with every emerge --sync.
>

Yes, that's a problem of every file system. Reiserfs (especially without
notail) and btrfs are more prone to this due to their internal organization.

> Space is no longer an argument in these days, at least for my desktop
> machine. But I would like to optimize for speed -- emerge -DputnVj
> @world takes quite a while to calculate, I assume this is because so many
> ebuild files have to be accessed.
>

Not just ebuilds. You also have to consider /var/cache/edb and
/var/db/pkg. Be careful with the latter one. You don't want to loose its
content.

> Any tips on this? Does it make sense to use a special file system just
> for the portage tree? What would be best? Would it help to re-create this
> file system from time to time in case it gets slower with every sync? Or
> wouldn't I notice a difference if I just used a big ext4 partition for
> all portage related stuff?
>
> Anyone using a compressed RAM file system for that?
>
> Wonko
>

Recreating it certainly helps. I don't find it worth the effort. though.

Regards,
Florian Philipp

On Mar 10, 2012 8:33 PM, "Alex Schuster" <wonko@wonkology.org> wrote:

>

> Hi there!

>

> Is there an advantage in putting the portage tree on an extra partition?

>

> Currently, I'm using reiserfs, because I read that it is efficient when

> using many small files. On the other hand I also heard that it tends to

> get slower with every emerge --sync.

>

> Space is no longer an argument in these days, at least for my desktop

> machine. But I would like to optimize for speed -- emerge -DputnVj

> @world takes quite a while to calculate, I assume this is because so many

> ebuild files have to be accessed.

>

> Any tips on this? Does it make sense to use a special file system just

> for the portage tree? What would be best? Would it help to re-create this

> file system from time to time in case it gets slower with every sync? Or

> wouldn't I notice a difference if I just used a big ext4 partition for

> all portage related stuff?

>

> Anyone using a compressed RAM file system for that?

>


This had been my burning question when I was deploying the company's production server, and forced me to do some research:


* reiserfs is amazingly fast for reads, but suffers on simultaneous writes

* reiserfs does not have inode limits

* reiserfs' notail affects performance greatly depending on the nature of the system: I/O-bound (use notail) or CPU-bound (don't use notail)

* reiserfs, if mounted without notail, is very space-efficient


So, I end up with the following mix:


* ext2 for /boot

* reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use tmpfs)

* ext4 for everything else


This cocktail has been serving me well. I don't need advanced filesystems like ZFS, XFS, or btrfs, because my servers are virtualized, and the advanced features (e.g., snapshot) is handled by the underlying hypervisor (XenServer) and SAN Storage (we use NetApp).



Rgds,

On Mar 10, 2012 10:09 PM, "Pandu Poluan" <pandu@poluan.info> wrote:

>

>

> On Mar 10, 2012 8:33 PM, "Alex Schuster" <wonko@wonkology.org> wrote:

> >

> > Hi there!

> >

> > Is there an advantage in putting the portage tree on an extra partition?

> >

> > Currently, I'm using reiserfs, because I read that it is efficient when

> > using many small files. On the other hand I also heard that it tends to

> > get slower with every emerge --sync.

> >

> > Space is no longer an argument in these days, at least for my desktop

> > machine. But I would like to optimize for speed -- emerge -DputnVj

> > @world takes quite a while to calculate, I assume this is because so many

> > ebuild files have to be accessed.

> >

> > Any tips on this? Does it make sense to use a special file system just

> > for the portage tree? What would be best? Would it help to re-create this

> > file system from time to time in case it gets slower with every sync? Or

> > wouldn't I notice a difference if I just used a big ext4 partition for

> > all portage related stuff?

> >

> > Anyone using a compressed RAM file system for that?

> >

>

> This had been my burning question when I was deploying the company's production server, and forced me to do some research:

>

> * reiserfs is amazingly fast for reads, but suffers on simultaneous writes

> * reiserfs does not have inode limits

> * reiserfs' notail affects performance greatly depending on the nature of the system: I/O-bound (use notail) or CPU-bound (don't use notail)

> * reiserfs, if mounted without notail, is very space-efficient

>

> So, I end up with the following mix:

>

> * ext2 for /boot

> * reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use tmpfs)

> * ext4 for everything else

>

> This cocktail has been serving me well. I don't need advanced filesystems like ZFS, XFS, or btrfs, because my servers are virtualized, and the advanced features (e.g., snapshot) is handled by the underlying hypervisor (XenServer) and SAN Storage (we use NetApp).


>

> Rgds,


Okay, I did a mixup:


If the system is I/O-bound, *don't* use notail (saves on disk read/write).


If the system is CPU-bound, *use* notail (saves on having to 'unpack' the tail from the metadata).


In my situation, the bottleneck is the SAN Storage, so I don't use notail.


Rgds,

On Sat, 10 Mar 2012 14:30:15 +0100, Alex Schuster wrote:

> Any tips on this? Does it make sense to use a special file system just
> for the portage tree? What would be best? Would it help to re-create
> this file system from time to time in case it gets slower with every
> sync?

I use an ext2 filesystem for portage, it's still the fastest out there.
Journals are unnecessary because its such a small filesystem, and if it
does get damaged I can just reformat and sync again.


--
Neil Bothwick

Physics is like sex: sure, it may give some practical results, but
that's not why we do it. Richard Feynman

On Sat, 10 Mar 2012 22:09:26 +0700
Pandu Poluan <pandu@poluan.info> wrote:

> On Mar 10, 2012 8:33 PM, "Alex Schuster" <wonko@wonkology.org> wrote:
> >
> > Hi there!
> >
> > Is there an advantage in putting the portage tree on an extra
> > partition?
> >
> > Currently, I'm using reiserfs, because I read that it is efficient
> > when using many small files. On the other hand I also heard that it
> > tends to get slower with every emerge --sync.
> >
> > Space is no longer an argument in these days, at least for my
> > desktop machine. But I would like to optimize for speed -- emerge
> > -DputnVj @world takes quite a while to calculate, I assume this is
> > because so many ebuild files have to be accessed.
> >
> > Any tips on this? Does it make sense to use a special file system
> > just for the portage tree? What would be best? Would it help to
> > re-create this file system from time to time in case it gets slower
> > with every sync? Or wouldn't I notice a difference if I just used a
> > big ext4 partition for all portage related stuff?
> >
> > Anyone using a compressed RAM file system for that?
> >
>
> This had been my burning question when I was deploying the company's
> production server, and forced me to do some research:
>
> * reiserfs is amazingly fast for reads, but suffers on simultaneous
> writes
> * reiserfs does not have inode limits
> * reiserfs' notail affects performance greatly depending on the
> nature of the system: I/O-bound (use notail) or CPU-bound (don't use
> notail)
> * reiserfs, if mounted without notail, is very space-efficient
>
> So, I end up with the following mix:
>
> * ext2 for /boot
> * reiserfs for /usr/portage and /var/tmp (RAM is at premium; can't use
> tmpfs)
> * ext4 for everything else
>
> This cocktail has been serving me well. I don't need advanced
> filesystems like ZFS, XFS, or btrfs, because my servers are
> virtualized, and the advanced features (e.g., snapshot) is handled by
> the underlying hypervisor (XenServer) and SAN Storage (we use NetApp).
>
> Rgds,

That's very close to what I do (though not for the same
extensively-researched reasons . I added an extra bit of twiddling
in make.conf:

DISTDIR="/usr/local/distfiles" # On /.
PKGDIR="/usr/local/packages" # On /.
PORTDIR="/mnt/portage/gentoo" # /mnt/portage is reiserfs and has /layman too

This way the requirements for the portage partition grow much more
gradually (changed that due to overflow once), and on the random
chance that reiserfs gets corrupted, I don't lose all my
fetch-restricted distfiles.

- Bryan

On Sat, Mar 10, 2012 at 03:35:05PM +0000, Neil Bothwick wrote:
> On Sat, 10 Mar 2012 14:30:15 +0100, Alex Schuster wrote:
>
> > Any tips on this? Does it make sense to use a special file system just
> > for the portage tree? What would be best? Would it help to re-create
> > this file system from time to time in case it gets slower with every
> > sync?
>
> I use an ext2 filesystem for portage, it's still the fastest out there.
> Journals are unnecessary because its such a small filesystem, and if it
> does get damaged I can just reformat and sync again.

I use an ext2 partition in a 500MB file image on most of my computers.
Its important to check the inode count on such small filesytem, as
mke2fs' default inode ration for such size is 4096, which is too
low for portage:

dd bs=$((500*1024*1024)) count=1 if=/dev/zero of=/usr/img_portage
mke2fs -f -b1024 -i2048 /usr/img_portage

fstab:
/usr/img_portage /usr/portage/ ext2 loop,noatime 0 0
(this is from desktop, on servers I usually only mount it manually when
emerging)

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 469M 306M 139M 69% /usr/portage

# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 256032 152044 103988 60% /usr/portage


yoyo

On Sat, 10 Mar 2012 19:36:07 +0100, YoYo Siska wrote:

> > I use an ext2 filesystem for portage, it's still the fastest out
> > there. Journals are unnecessary because its such a small filesystem,
> > and if it does get damaged I can just reformat and sync again.
>
> I use an ext2 partition in a 500MB file image on most of my computers.

I used to do that but, after switching to LVM, it was simpler to use an
LV.

> Its important to check the inode count on such small filesytem, as
> mke2fs' default inode ration for such size is 4096, which is too
> low for portage:
>
> dd bs=$((500*1024*1024)) count=1 if=/dev/zero of=/usr/img_portage
> mke2fs -f -b1024 -i2048 /usr/img_portage

I use similar arguments for mke2fs.


--
Neil Bothwick

Why do they call it a TV set when you only get one?

YoYo Siska writes:

> On Sat, Mar 10, 2012 at 03:35:05PM +0000, Neil Bothwick wrote:

> > I use an ext2 filesystem for portage, it's still the fastest out
> > there. Journals are unnecessary because its such a small filesystem,
> > and if it does get damaged I can just reformat and sync again.

Replaying a reiserfs journal in case of an unclean reboot also takes about
the same time as an whole e2fsck, so I switched to ext2. There was no
real need to make the switch, I just wanted to re-create this file system
that has been synced very often now.

> I use an ext2 partition in a 500MB file image on most of my computers.

I also did this in the past, on systems where I did not use LVM. Nowadays
I prefer the latter.

> Its important to check the inode count on such small filesytem, as
> mke2fs' default inode ration for such size is 4096, which is too
> low for portage:

Yes, happened to me more than once...

> mke2fs -f -b1024 -i2048 /usr/img_portage

That's what I did. Well, without the container file.

Thanks to all who replied! I learnt something, like so often when reading
here.

Wonko