Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo User (http://www.linux-archive.org/gentoo-user/)
-   -   Linux, reiserfs and file fragmentation (http://www.linux-archive.org/gentoo-user/55312-linux-reiserfs-file-fragmentation.html)

Dale 02-14-2008 10:06 AM

Linux, reiserfs and file fragmentation
 
Hi,

I'm not wanting to start a flame or anything but I have a question, or
two, on file fragmentation. I have three hard drives here. This is how
they are partitioned at the moment:



root@smoker / # df
Filesystem 1K-blocks Used
Available Use% Mounted on
/dev/hda6 19530340 7689400 11840940
40% /
/dev/hda1 189339 7741
171822 5% /boot
/dev/hda7 4882532 1805104
3077428 37% /usr/portage
/dev/hda8 4882532 1402172
3480360 29% /home
/dev/hdb1 78145768 15171752
62974016 20% /data

root@smoker / #



So you won't freak out, I removed the system generated file systems. I
use reiserfs on everything except /boot. It has ext2. I found a
program, script really, that will tell how fragmented a file system is.
This is what it reports:



root@smoker / # /root/fragck.pl /usr/portage/
2.93016639526831% non contiguous files, 1.10476912422558 average
fragments.
root@smoker / #
/root/fragck.pl /data/
4.64526022181977% non contiguous files, 1.08609726757175 average
fragments.

root@smoker / #
/root/fragck.pl /home/
statfs: No such file or directory
Use of uninitialized value in pattern match (m//) at /root/fragck.pl
line 32.
Use of uninitialized value in concatenation (.) or string at
/root/fragck.pl line 41.
: not understand for
/home/dale/.googleearth/Registry/google/gecommonsettings/User/layers/hiking
and mountain bike trails

.
6.4034151547492% non contiguous files, 2.34827463536108 average fragments.
root@smoker / #



The big one:


root@smoker / # /root/fragck.pl /
statfs: No such file or directory
sh: en_1ca_4278190280: command not found
sh: wm_6: command not found
sh: wp_#usr#kde#3.5#share#wallpapers#here-gear.svgz_1196427186:
command not found

sh: blm_0: command not found
sh: 01.png: command not found
Use of uninitialized value in pattern match (m//) at /root/fragck.pl
line 32.
Use of uninitialized value in concatenation (.) or string at
/root/fragck.pl line 41.
: not understand for
/var/tmp/kdecache-root/background/143x115_bm_0;en_1ca_4278190280;wm_6;wp_#usr#kde#3. 5#share#wallpapers#here-gear.svgz_1196427186;blm_0;01.png

.
statfs: No such file or directory
sh: en_1ca_4278190280: command not found
sh: wm_6: command not found
sh: wp_#usr#kde#3.5#share#wallpapers#here-gear.svgz_1192962680:
command not found

sh: blm_0: command not found
sh: 01.png: command not found
Use of uninitialized value in pattern match (m//) at /root/fragck.pl
line 32.
Use of uninitialized value in concatenation (.) or string at
/root/fragck.pl line 41.
: not understand for
/var/tmp/kdecache-root/background/143x115_bm_0;en_1ca_4278190280;wm_6;wp_#usr#kde#3. 5#share#wallpapers#here-gear.svgz_1192962680;blm_0;01.png

.
7.03979903669298% non contiguous files, 1.22497799280025 average
fragments.
root@smoker / #


Please pardon the error message. I use the script, I didn't write it. :/

I understand that doing the root directory is sort of a sum of all the
others so it may be a little misleading to say the least.


My questions; is this badly fragmented? How can I "unfragment" all the
files and not bork something up badly?

My opinion on this tho, considering this install is about 4 years old,
not to bad. I've seen worse on a windoze rig shortly after a install. ;-)


Thanks for any advice/info you can provide. Let me know if you need
more info too.


Dale

:-) :-)
--

gentoo-user@lists.gentoo.org mailing list

Michal 'vorner' Vaner 02-14-2008 10:31 AM

Linux, reiserfs and file fragmentation
 
Hello

On Thu, Feb 14, 2008 at 05:06:43AM -0600, Dale wrote:
> My questions; is this badly fragmented? How can I "unfragment" all the
> files and not bork something up badly?
> My opinion on this tho, considering this install is about 4 years old, not
> to bad. I've seen worse on a windoze rig shortly after a install. ;-)

I would guess the fragmented files are the big ones. And, with average
of 2 fragments per file, it is not too much. If you have a movie with
30MB fragments, then it is no problem.

Unless you hear lot of rattling noise from the HDD, you could leave it
as is.

And the surest way to defragment a filesystem is take everything out and
put it back again. It will write the files one after another and will
have no reason to split them.

--
XML is like violence. If it doesn't solve your problem, use more.

Michal 'vorner' Vaner
--
gentoo-user@lists.gentoo.org mailing list

Dale 02-14-2008 11:01 AM

Linux, reiserfs and file fragmentation
 
Michal 'vorner' Vaner wrote:

Hello

On Thu, Feb 14, 2008 at 05:06:43AM -0600, Dale wrote:

My questions; is this badly fragmented? How can I "unfragment" all the
files and not bork something up badly?
My opinion on this tho, considering this install is about 4 years old, not
to bad. I've seen worse on a windoze rig shortly after a install. ;-)



I would guess the fragmented files are the big ones. And, with average
of 2 fragments per file, it is not too much. If you have a movie with
30MB fragments, then it is no problem.

Unless you hear lot of rattling noise from the HDD, you could leave it
as is.

And the surest way to defragment a filesystem is take everything out and
put it back again. It will write the files one after another and will
have no reason to split them.




So if for example I copied everything over to a different hard drive and
then copied everything back, it would be "defragmented" then?


I would think of something like this:

Boot some live CD.
Mount old and backup drives.
Copy old drive to a backup drive using cp -av yada yada.
Make a new file system on the old drive to make sure all is clean.
Copy everything back over from the backup to the old drive using cp -av
yada yada.


I would also take the opportunity to redo a few partitions while I was
able to.


The biggest slow down by the way is when logging into KDE the first
time. It takes a long while and that drive is just a getting it. The
light just stays on while loading everything up.


Your thoughts and others if needed.

Dale

:-) :-) :-)
--

gentoo-user@lists.gentoo.org mailing list

Thomas Kahle 02-14-2008 11:57 AM

Linux, reiserfs and file fragmentation
 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

my 2 cents:

| So if for example I copied everything over to a different hard drive and
| then copied everything back, it would be "defragmented" then?

I think so yes, but still I would not do it as I think you will hardly
notice the difference, but there is a good chance to screw things up.

| I would think of something like this:
|
| Boot some live CD.
| Mount old and backup drives.
| Copy old drive to a backup drive using cp -av yada yada.

Its very important to do this as root and preserve all the file
permissions and symbolic links exactly as they are on the drive.
In particular the backup file system must support all this.
(You cannot backup to a FAT file system, etc.)

the cp option "-b" could help, but surely you should read
man cp
and
man mount

| Make a new file system on the old drive to make sure all is clean.
| Copy everything back over from the backup to the old drive using cp -av
| yada yada.
|
| I would also take the opportunity to redo a few partitions while I was
| able to.

If you do so don't forget to update /etc/fstab
and the configuration of the bootloader !

| The biggest slow down by the way is when logging into KDE the first
| time. It takes a long while and that drive is just a getting it. The
| light just stays on while loading everything up.

I personally think this is not due to fragmentation.
On loading KDE just preloads some big libraries (it is a big program :)
and this takes some time.
Furthermore the libraries are loaded with "LD_BIND_NOW=true", which
makes the linker resolve all the symbols when KDE starts. (KDE takes
longer to load, but later the programs are loaded faster).
You can google for that to learn what it means.

Hope it helps a little
Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHtDqdrpEWPKIUt7MRAkV6AKCLrm/tVj7KjM4ElHCqc0Zf/gRoxwCdEI1F
jtPhK0NUmeRBcP5giKMb2JI=
=Dg/q
-----END PGP SIGNATURE-----
--
gentoo-user@lists.gentoo.org mailing list

Volker Armin Hemmann 02-14-2008 12:17 PM

Linux, reiserfs and file fragmentation
 
On Donnerstag, 14. Februar 2008, Dale wrote:

>
> The biggest slow down by the way is when logging into KDE the first
> time. It takes a long while and that drive is just a getting it. The
> light just stays on while loading everything up.

do you use prelink?
--
gentoo-user@lists.gentoo.org mailing list

Dale 02-14-2008 12:53 PM

Linux, reiserfs and file fragmentation
 
Thomas Kahle wrote:

Hi,

my 2 cents:

| So if for example I copied everything over to a different hard drive and
| then copied everything back, it would be "defragmented" then?

I think so yes, but still I would not do it as I think you will hardly
notice the difference, but there is a good chance to screw things up.

| I would think of something like this:
|
| Boot some live CD.
| Mount old and backup drives.
| Copy old drive to a backup drive using cp -av yada yada.

Its very important to do this as root and preserve all the file
permissions and symbolic links exactly as they are on the drive.
In particular the backup file system must support all this.
(You cannot backup to a FAT file system, etc.)

the cp option "-b" could help, but surely you should read
man cp
and
man mount

| Make a new file system on the old drive to make sure all is clean.
| Copy everything back over from the backup to the old drive using cp -av
| yada yada.
|
| I would also take the opportunity to redo a few partitions while I was
| able to.

If you do so don't forget to update /etc/fstab
and the configuration of the bootloader !

| The biggest slow down by the way is when logging into KDE the first
| time. It takes a long while and that drive is just a getting it. The
| light just stays on while loading everything up.

I personally think this is not due to fragmentation.
On loading KDE just preloads some big libraries (it is a big program :)
and this takes some time.
Furthermore the libraries are loaded with "LD_BIND_NOW=true", which
makes the linker resolve all the symbols when KDE starts. (KDE takes
longer to load, but later the programs are loaded faster).
You can google for that to learn what it means.

Hope it helps a little
Thomas



I did a little test. Something fishy here. I did a test with the /data
partition. I store pictures and documents there and it was fragmented.
I cp -av to another reiserfs formatted partition then remade the file
system and copied it back using basically the same command just in
reverse. This is what I got now:


root@smoker / # /root/fragck.pl /data/
3.88457269700333% non contiguous files, 1.04344379261138 average fragments.
root@smoker / #

That is not a lot better than it was before. It was 4.6% before. How
is that? I copied it over then ran the command right after without even
touching the files.


Any ideas? Is there a limit to the fragmenting "smallness"?

Dale

:-) :-)
--

gentoo-user@lists.gentoo.org mailing list

Daniel Iliev 02-14-2008 12:57 PM

Linux, reiserfs and file fragmentation
 
On Thu, 14 Feb 2008 06:01:16 -0600
Dale <dalek1967@bellsouth.net> wrote:

> Michal 'vorner' Vaner wrote:
> > Hello
> >
> > On Thu, Feb 14, 2008 at 05:06:43AM -0600, Dale wrote:
> >
> >> My questions; is this badly fragmented? How can I "unfragment"
> >> all the files and not bork something up badly?
> >> My opinion on this tho, considering this install is about 4 years
> >> old, not to bad. I've seen worse on a windoze rig shortly after a
> >> install. ;-)
> >
> > I would guess the fragmented files are the big ones. And, with
> > average of 2 fragments per file, it is not too much. If you have a
> > movie with 30MB fragments, then it is no problem.
> >
> > Unless you hear lot of rattling noise from the HDD, you could leave
> > it as is.
> >
> > And the surest way to defragment a filesystem is take everything
> > out and put it back again. It will write the files one after
> > another and will have no reason to split them.
> >
> >
>
> So if for example I copied everything over to a different hard drive
> and then copied everything back, it would be "defragmented" then?
>
> I would think of something like this:
>
> Boot some live CD.
> Mount old and backup drives.
> Copy old drive to a backup drive using cp -av yada yada.
> Make a new file system on the old drive to make sure all is clean.
> Copy everything back over from the backup to the old drive using cp
> -av yada yada.
>
> I would also take the opportunity to redo a few partitions while I
> was able to.
>
> The biggest slow down by the way is when logging into KDE the first
> time. It takes a long while and that drive is just a getting it.
> The light just stays on while loading everything up.
>
> Your thoughts and others if needed.
>
> Dale
>
> :-) :-) :-)





If you haven't already done this, you could try [1] for faster KDE boot.
I believe it'll bring you much bigger application start-up boost than
defragmenting your FS.

Please, notice that I'm not saying that defragmentation is pointless.
Just the opposite: I believe fragmentation leads to a perceivable (and
actually measurable) performance hit.


[1] http://www.gentoo.org/doc/en/prelink-howto.xml


--
Best regards,
Daniel
--
gentoo-user@lists.gentoo.org mailing list

Dale 02-14-2008 01:00 PM

Linux, reiserfs and file fragmentation
 
Volker Armin Hemmann wrote:

On Donnerstag, 14. Februar 2008, Dale wrote:



The biggest slow down by the way is when logging into KDE the first
time. It takes a long while and that drive is just a getting it. The
light just stays on while loading everything up.



do you use prelink?



I did use it at one time. It didn't seem to speed anything up. I can't
recall when I stopped using it tho. It's been a while. Think it would
help now?


It is usually when I first login that it takes so long. After the first
time, it only takes about 5 or 6 seconds. Not real bad but I was
thinking about my poor five year old drive. o_O


Thanks.

Dale

:-) :-) :-)
--
gentoo-user@lists.gentoo.org mailing list

Uwe Thiem 02-14-2008 01:16 PM

Linux, reiserfs and file fragmentation
 
On Thursday 14 February 2008, Thomas Kahle wrote:
> Hi,
>
> my 2 cents:
> | So if for example I copied everything over to a different hard
> | drive and then copied everything back, it would be "defragmented"
> | then?
>
> I think so yes, but still I would not do it as I think you will
> hardly notice the difference, but there is a good chance to screw
> things up.

Yes, everything will be defragmented. In addition, it will leave gaps
between files. So if a file lateron grows it will not immediately
fragment.

>
> | I would think of something like this:
> |
> | Boot some live CD.
> | Mount old and backup drives.
> | Copy old drive to a backup drive using cp -av yada yada.
>
> Its very important to do this as root and preserve all the file
> permissions and symbolic links exactly as they are on the drive.
> In particular the backup file system must support all this.
> (You cannot backup to a FAT file system, etc.)
>
> the cp option "-b" could help, but surely you should read
> man cp
> and
> man mount

The easiest way to preserve all permissions and symlinks is to use tar
instead of cp. If you do so, read "man tar" of course.

> | The biggest slow down by the way is when logging into KDE the
> | first time. It takes a long while and that drive is just a
> | getting it. The light just stays on while loading everything up.
>
> I personally think this is not due to fragmentation.
> On loading KDE just preloads some big libraries (it is a big
> program :) and this takes some time.
> Furthermore the libraries are loaded with "LD_BIND_NOW=true", which
> makes the linker resolve all the symbols when KDE starts. (KDE
> takes longer to load, but later the programs are loaded faster).
> You can google for that to learn what it means.

There are two ways to speed up KDE load time. First, prelink
everything (something like "prelink -avmR"). Second, you can
configure kdm to preload as much of KDE as possible. So while you are
still staring at your login screen or typing your user name and
password, it loads as much as it can.

BTW, KDE 4 starts significantly faster than 3.5.

Uwe

--
Informal Linux Group Namibia:
http://www.linux.org.na/
SysEx (Pty) Ltd.:
http://www.SysEx.com.na/
--
gentoo-user@lists.gentoo.org mailing list

Willie Wong 02-14-2008 01:21 PM

Linux, reiserfs and file fragmentation
 
On Thu, Feb 14, 2008 at 07:53:57AM -0600, Penguin Lover Dale squawked:
> I did a little test. Something fishy here. I did a test with the /data
> partition. I store pictures and documents there and it was fragmented.
> I cp -av to another reiserfs formatted partition then remade the file
> system and copied it back using basically the same command just in
> reverse. This is what I got now:
>
> root@smoker / # /root/fragck.pl /data/
> 3.88457269700333% non contiguous files, 1.04344379261138 average fragments.
> root@smoker / #
>
> That is not a lot better than it was before. It was 4.6% before. How
> is that? I copied it over then ran the command right after without even
> touching the files.

Is the /data partition on reiser? Did you enable tailpacking?
As I don't know what the script you ran actually does, I don't know
how it handles block suballocation... Tail packing is something that
can conceivably introduce data fragmentation in place of internal
fragmentation.

W
--
Pintsize: Curses! HOURS in there and I STILL don't have mutant ice powers!
Pintsize: Sorry waffles, you can't be my sidekick until I have some
superhero powers to fight crime with... What? "Waffle powers"?
Somehow I don't see "soaking up syrup" or "browning in a toaster"
getting us a lot of hot supervillain ladies.
Sortir en Pantoufles: up 433 days, 12:47
--
gentoo-user@lists.gentoo.org mailing list


All times are GMT. The time now is 01:22 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.