Dennis Clarke <dclarke@blastwave.org> wrote:

>
> A little data never hurts. Even if the numbers mean little.
>
> test 1 - Debian Linux 6.0.5 on x86_64

Given the fact, that you did not run star -no-fifo, you compare an insecure
implementation (gtar never calls fsync(2)) with a secure by default
implementation (star).

Also note: ZFS has one real problem: it is really slow if you force it to grant
a stable sate on the medium. This is why ZFS is aprox. 4x slower in your test
star (without -no-fsync) compared to gtar.

ext3 is slow with star (without -no-fsync) because ext3 it is not optimized.
ext3 is fast with gtar because it cheats.

Solaris with UFS:

bzip2 -d < /tmp/linux-3.5.1.tar.bz2 > /dev/null
19.909r 19.770u 0.110s 99% 0M 0+0k 0st 0+0io 0pf+0w


OK, 19.77 seconds user CPU time.....

star -xp -xdot -time < /tmp/linux-3.5.1.tar.bz2
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 70.477sec (6647 kBytes/sec)
1:10.489r 23.020u 9.010s 45% 0M 0+0k 0st 0+0io 0pf+0w

star -xp -xdot -time -no-fsync < /tmp/linux-3.5.1.tar.bz2
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 74.161sec (6317 kBytes/sec)
1:14.174r 21.840u 4.640s 35% 0M 0+0k 0st 0+0io 0pf+0w

Only half of the System CPU time here because fsync(2) calls are missing...

gtar --totals -xf /tmp/linux-3.5.1.tar.bz2
Gesamtzahl gelesener Bytes: 479733760 (458MiB, 4,5MiB/s)
1:42.658r 23.150u 5.530s 27% 0M 0+0k 0st 0+0io 0pf+0w

gtar does not have a FIFO and thus is slower than star.


Now uncompressed (you see that the user CPU time for bzip2 is missing):

star -xp -xdot -time < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 70.438sec (6651 kBytes/sec)
1:10.449r 0.520u 8.190s 12% 0M 0+0k 0st 0+0io 0pf+0w

star -xp -xdot -time -no-fsync < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 86.624sec (5408 kBytes/sec)
1:26.636r 0.300u 3.960s 4% 0M 0+0k 0st 0+0io 0pf+0w

gtar --totals -xf /tmp/linux-3.5.1.tar
Gesamtzahl gelesener Bytes: 479733760 (458MiB, 4,7MiB/s)
1:38.829r 0.440u 4.570s 5% 0M 0+0k 0st 0+0io 0pf+0w


Now ZFS being on a single disk like the UFS test before (let us omit
the test with the compressed archive):

star -xp -xdot -time < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 394.783sec (1186 kBytes/sec)
6:34.795r 0.580u 8.250s 2% 0M 0+0k 0st 0+0io 0pf+0w

As expected: ZFS is slow if you force it to grant a stable state.

star -xp -xdot -time -no-fsync < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 11.085sec (42259 kBytes/sec)
11.096r 0.290u 4.380s 42% 0M 0+0k 0st 0+0io 0pf+0w

gtar --totals -xf /tmp/linux-3.5.1.tar
Gesamtzahl gelesener Bytes: 479733760 (458MiB, 39MiB/s)
11.929r 0.360u 4.260s 38% 0M 0+0k 0st 0+0io 0pf+0w

As you see, if you permit star to be as insecure as gtar always is, star is
always faster than gtar.

Now the same on a thumper (ZFS on RAIDZ2):

star -xp -xdot -time < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 349.575sec (1340 kBytes/sec)
5:49.595r 0.690u 11.270s 3% 0M 0+0k 0st 0+0io 0pf+0w

star -xp -xdot -time -no-fsync < /tmp/linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
star: Total time 4.626sec (101251 kBytes/sec)
4.654r 0.330u 4.990s 115% 0M 0+0k 0st 0+0io 0pf+0w

gtar --totals -xf /tmp/linux-3.5.1.tar
Total bytes read: 479733760 (458MiB, 85MiB/s)
5.510r 0.430u 4.130s 82% 0M 0+0k 0st 0+0io 0pf+0w

Jörg

--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni)
joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

> Dennis Clarke <dclarke@blastwave.org> wrote:
>
> >
> > A little data never hurts. Even if the numbers mean little.
> >
> > test 1 - Debian Linux 6.0.5 on x86_64
>
> Given the fact, that you did not run star -no-fifo, you compare an
> insecure
> implementation (gtar never calls fsync(2)) with a secure by default
> implementation (star).

Comparison numbers are only valid of the tests run are the same.

So here is the UFS test once more without the compression and
with -no-fifo :

jupiter-sparc-SunOS5.10 # ptime /opt/schily/bin/star -x -xdir -xdot -no-fifo -U file=../linux-3.5.1.tar
star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).

real 27:44.237
user 2.031
sys 42.419

Not a good result.

dc
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Dennis Clarke <dclarke@blastwave.org> wrote:

> > Given the fact, that you did not run star -no-fifo, you compare an
> > insecure
> > implementation (gtar never calls fsync(2)) with a secure by default
> > implementation (star).
>
> Comparison numbers are only valid of the tests run are the same.
>
> So here is the UFS test once more without the compression and
> with -no-fifo :
>
> jupiter-sparc-SunOS5.10 # ptime /opt/schily/bin/star -x -xdir -xdot -no-fifo -U file=../linux-3.5.1.tar
> star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
>
> real 27:44.237
> user 2.031
> sys 42.419
>
> Not a good result.

So try to think about the reasons..... star is definitely not the reason.
The fact that you spend 10x the amount of expectes SYS CPU seems to lead to a
problem on your system.

Also the USER CPU time is 8x the expected amount. Did you run this test in
_very_ old hardware?

Jörg

--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni)
joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

> > Comparison numbers are only valid of the tests run are the same.
> >
> > So here is the UFS test once more without the compression and
> > with -no-fifo :
> >
> > jupiter-sparc-SunOS5.10 # ptime /opt/schily/bin/star -x -xdir -xdot
> -no-fifo -U file=../linux-3.5.1.tar
> > star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
> >
> > real 27:44.237
> > user 2.031
> > sys 42.419
> >
> > Not a good result.
>
> So try to think about the reasons..... star is definitely not the reason.
> The fact that you spend 10x the amount of expectes SYS CPU seems to
> lead to a
> problem on your system.
>
> Also the USER CPU time is 8x the expected amount. Did you run this
> test in
> _very_ old hardware?
>
> Jörg

It would be reasonable to think of a Sun Fire V480 as old hardware yes, but
not *very* old. I do have *very* old if you would like me to test there?

The server runs fine, is patched up to date. The UFS filesystem that was
used is actually the root filesystem and it is a metadevice mirror of
the two internal disks.

jupiter-sparc-SunOS5.10 # metastat d0
d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: geometric (-g)
Write option: parallel (default)
Size: 33560448 blocks (16 GB)

d10: Submirror of d0
State: Okay
Size: 33560448 blocks (16 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t0d0s0 0 No Okay Yes


d20: Submirror of d0
State: Okay
Size: 33560448 blocks (16 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c1t1d0s0 0 No Okay Yes


Device Relocation Information:
Device Reloc Device ID
c1t0d0 Yes id1,ssd@n20000004cfb6f0ff
c1t1d0 Yes id1,ssd@n20000004cfa10ec9

However I do also have :

jupiter-sparc-SunOS5.10 # luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
Node WWN:20000004cfb6f0ff Device Typeisk device
Logical Path:/dev/rdsk/c1t0d0s2
Node WWN:20000004cfa10ec9 Device Typeisk device
Logical Path:/dev/rdsk/c1t1d0s2
Node WWN:20000018625de4c6 Device Typeisk device
Logical Path:/dev/rdsk/c2t16d0s2
Node WWN:20000018625d599d Device Typeisk device
Logical Path:/dev/rdsk/c2t17d0s2
Node WWN:20000018625f11cb Device Typeisk device
Logical Path:/dev/rdsk/c2t18d0s2
Node WWN:500000e0148b6620 Device Typeisk device
Logical Path:/dev/rdsk/c2t19d0s2
Node WWN:20000014c3a51579 Device Typeisk device
Logical Path:/dev/rdsk/c2t20d0s2
Node WWN:20000018625fe1a2 Device Typeisk device
Logical Path:/dev/rdsk/c2t21d0s2
Node WWN:20000014c3a8ee8b Device Typeisk device
Logical Path:/dev/rdsk/c2t22d0s2
Node WWN:20000018629c6c36 Device Typeisk device
Logical Path:/dev/rdsk/c5t0d0s2
Node WWN:2000000c5042657d Device Typeisk device
Logical Path:/dev/rdsk/c5t1d0s2
Node WWN:20000018629c2baf Device Typeisk device
Logical Path:/dev/rdsk/c5t2d0s2
Node WWN:2000000087b10c29 Device Typeisk device
Logical Path:/dev/rdsk/c5t3d0s2
Node WWN:20000018625dda0c Device Typeisk device
Logical Path:/dev/rdsk/c5t4d0s2
Node WWN:500000e010f28870 Device Typeisk device
Logical Path:/dev/rdsk/c5t5d0s2
Node WWN:20000018625dfa91 Device Typeisk device
Logical Path:/dev/rdsk/c5t6d0s2

Since I have a pile of fibre disks I can perhaps isolate one of
them and test with UFS on a single disk.

This one may do fine :

jupiter-sparc-SunOS5.10 # luxadm display 2000000087b10c29
DEVICE PROPERTIES for disk: 2000000087b10c29
Status(Port B): O.K.
Vendor: HITACHI
Product ID: HUS1030FASUN300G
WWN(Node): 2000000087b10c29
WWN(Port B): 2200000087b10c29
Revision: 2A08
Serial Num: 49VXAJNS 55VAXAJNSA
Unformatted capacity: 286102.281 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c5t3d0s2
/devices/pci@8,600000/pci@2/SUNW,qlc@4/fp@0,0/ssd@w2200000087b10c29,0:c,raw

I can try to isolate that and use it with UFS but no hurry, I have other
things to do also.

Dennis

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Dennis Clarke <dclarke@blastwave.org> wrote:

>
> > > Comparison numbers are only valid of the tests run are the same.
> > >
> > > So here is the UFS test once more without the compression and
> > > with -no-fifo :
> > >
> > > jupiter-sparc-SunOS5.10 # ptime /opt/schily/bin/star -x -xdir -xdot
> > -no-fifo -U file=../linux-3.5.1.tar
> > > star: 46849 blocks + 0 bytes (total of 479733760 bytes = 468490.00k).
> > >
> > > real 27:44.237
> > > user 2.031
> > > sys 42.419
> > >
> > > Not a good result.
> >
> > So try to think about the reasons..... star is definitely not the reason.
> > The fact that you spend 10x the amount of expectes SYS CPU seems to
> > lead to a
> > problem on your system.
> >
> > Also the USER CPU time is 8x the expected amount. Did you run this
> > test in
> > _very_ old hardware?
> >
> > Jörg
>
> It would be reasonable to think of a Sun Fire V480 as old hardware yes, but
> not *very* old. I do have *very* old if you would like me to test there?

Well, this machine is 11 years old now.

This explains the large amount of CPU time.

> The server runs fine, is patched up to date. The UFS filesystem that was
> used is actually the root filesystem and it is a metadevice mirror of
> the two internal disks.

A simple mirror is slow.

However, a 2300 GB FCAL drive should be faster.

But... did you turn on logging?

Jörg

--
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
js@cs.tu-berlin.de (uni)
joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

> Well, this machine is 11 years old now.
> This explains the large amount of CPU time.

Quad 900MHz UltraSparc III processors are more than
enough to handle a simple filesystem.

> > The server runs fine, is patched up to date. The UFS filesystem that
> was
> > used is actually the root filesystem and it is a metadevice mirror of
> > the two internal disks.
>
> A simple mirror is slow.

I don't agree.

jupiter-sparc-SunOS5.10 # pwd
/usr/local/src
jupiter-sparc-SunOS5.10 # which mkfile
/usr/sbin/mkfile

jupiter-sparc-SunOS5.10 # ptime /usr/sbin/mkfile 1024m one_gig.dat

real 30.874
user 0.206
sys 5.574
jupiter-sparc-SunOS5.10 # ls -l
total 3035664
-rw-r--r-- 1 root root 479733760 Aug 9 15:44 linux-3.5.1.tar
-rw------T 1 root root 1073741824 Aug 13 20:00 one_gig.dat
jupiter-sparc-SunOS5.10 # rm one_gig.dat
jupiter-sparc-SunOS5.10 #


Meanwhile in another xterm I ran iostat :

$ iostat -xc -d ssd0 ssd2 5
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.2 4.8 1.5 46.2 0.0 0.1 34.2 0 3 1 2 0 97
ssd2 0.1 4.6 1.2 46.1 0.0 0.1 34.7 0 3
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 9.2 0.0 5.0 0.0 0.2 20.7 0 5 0 1 0 99
ssd2 0.0 6.4 0.0 3.6 0.0 0.1 15.5 0 3
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1 0 99
ssd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.4 111.4 2.0 12546.4 6.3 4.3 94.9 25 36 1 2 0 97
ssd2 0.0 104.6 0.0 12476.0 5.9 4.1 96.1 24 31
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.8 326.4 6.4 37426.6 28.6 13.1 127.4 80 100 0 6 0 93
ssd2 0.0 317.8 0.0 37423.8 26.4 12.5 122.6 76 91
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.2 322.6 1.6 35021.9 23.2 12.5 110.7 73 98 0 17 0 83
ssd2 1.2 312.8 9.6 35104.7 23.0 12.5 113.1 73 92
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 2.2 294.8 20.8 30385.4 16.8 11.9 96.8 64 98 0 11 0 89
ssd2 1.4 284.8 11.2 30418.4 16.1 11.2 95.4 63 90
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 314.0 0.0 33055.0 20.3 12.2 103.6 69 98 0 5 0 95
ssd2 1.8 300.2 14.4 32987.6 19.1 11.7 102.0 67 89
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.4 312.8 3.2 33398.0 26.7 12.8 126.0 75 98 0 6 0 94
ssd2 1.6 304.4 12.8 33424.0 24.0 11.9 117.1 71 89
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 2.4 278.6 19.2 28227.5 24.8 10.8 126.9 63 94 1 7 0 92
ssd2 1.0 259.2 8.0 28187.7 21.3 9.6 118.6 57 75
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 9.4 0.0 27.6 0.0 0.2 25.1 0 6 0 1 0 99
ssd2 0.0 6.6 0.0 26.2 0.0 0.1 15.3 0 3
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 18.0 0.0 62.3 0.0 0.4 21.2 0 11 0 1 0 98
ssd2 0.2 13.2 0.2 59.9 0.0 0.2 16.8 0 7
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.4 29.0 3.2 307.1 0.0 0.8 26.2 0 15 0 2 0 98
ssd2 0.2 29.0 1.6 306.7 0.0 0.8 29.1 0 15
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 0.8 0.0 0.4 0.0 0.0 10.0 0 1 0 5 0 95
ssd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
ssd0 0.0 1.8 0.0 1.2 0.0 0.0 7.8 0 1 0 1 0 99
ssd2 0.0 0.6 0.0 0.6 0.0 0.0 7.1 0 0
^C$
$

bulk throughput would be 1024m/30.874sec =

jupiter-sparc-SunOS5.10 # dc
9k
1024 30.874 / p
33.167066139

Not bad.

> However, a 2300 GB FCAL drive should be faster.

Probably, but not much.

> But... did you turn on logging?

ufs logging ? Yes. Always. However that would be the same with the gtar and star tests also.

Dennis

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos