I want to play movies without hangs
 

Hi there!

Strange things are going on here.

I've written here in the past about my performance problems. My dual-core
had trouble playing movies without stuttering when there was I/O. It was
mainly swapping that caused this, and 8 G were not enough for me running
KDE4.

Then my hardware broke, and I got new one, except for the system hard
drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing movies
got even worse!

The videos do not need to have high quality. When I do this, I get
interruptions, sometimes for more than a whole second:

# dd if=/dev/zero of=/tmp/argh bs=10M count=1000

My whole system is encrypted, but the same happens with unencrypted
partitions. All are on LVM. When I write to another drive, there is no
effect. Throughput is around 50-60 MB/s.

Any ideas where to look? I think I'll create a completely fresh
kernel .config with genkernel, maybe my own .config has some weird
problem. But I tried similar things in the past already, getting a kernel
from a live cd, to no effect.

I put cache = 10240 into .mplayer/config to get 10 MB of video cached,
but I see no effect.

Playing music with Amarok is no problem.

My SATA drives are in AHCI mode, here's some dmesg info about that:

ahci 0000:00:11.0: version 3.0
ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ahci 0000:00:11.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA
mode
ahci 0000:00:11.0: flags: 64bit ncq sntf ilck led clo pmp pio slum part
ccc sxs
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
ata1: SATA max UDMA/133 abar m1024@0xff70b000 port 0xff70b100 irq 22
ata2: SATA max UDMA/133 abar m1024@0xff70b000 port 0xff70b180 irq 22
ata3: SATA max UDMA/133 abar m1024@0xff70b000 port 0xff70b200 irq 22
ata4: SATA max UDMA/133 abar m1024@0xff70b000 port 0xff70b280 irq 22
ahci 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ahci 0000:02:00.0: irq 43 for MSI/MSI-X
ahci: SSS flag set, parallel bus scan disabled
ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA
mode
ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part
ccc sxs
ahci 0000:02:00.0: setting latency timer to 64
scsi4 : ahci
scsi5 : ahci
ata5: SATA max UDMA/133 abar m512@0xff600000 port 0xff600100 irq 43
ata6: SATA max UDMA/133 abar m512@0xff600000 port 0xff600180 irq 43
pata_atiixp 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
scsi6 : pata_atiixp
scsi7 : pata_atiixp
ata7: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xf000 irq 14
ata8: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xf008 irq 15

(ata7/8 is the additional PATA controller, seen with the pata_atiixp
driver. I have one drive there, but it is not being used.)

When my new girl-friend comes over and we want to watch a movie, and it
stutters... she will ask why I don't simply use Windows to get
better performance, her five year old PC would do this just fine. Wat do
I tell her? WHAT DO I TELL HER??


And then there's what happened yesterday. A world update was going on,
with libreoffice, firefox, wine and thunderbird emerging in parallel, all
big packages. I have the PORTAGE_TMPDIR on a 5GB tmpfs, only libreoffice
is being compiled on disk. Suddenly, my system became very unresponsive,
the mouse had disappeared, the KDE widgets did not update, and xosview
showed a load of 23. All 4 cores were at 100%, the type of usage was
io-wait. How can I find out in such a case which processes are waiting
for I/O? top showed nothing. The Ctrl-Esc task viewer of KDE showed some
processes being 'inactive on hard drive', does this men those are the
waiting tasks? They varied, they were mostly Akonadi stuff. I stopped
akonadi, and after a while the load dropped. But this may be a
coincidence.

After all had calmed down, I had 2G of swap in use. 16G total RAM, all
being used of course, but only 8G being needed according to the -/+
buffers/cache line in free -m, the other 8G are cache. Does my Linux
somehow prefer to have this much cache, even if tmpfs stuff gets put into
swap? I have vm.swappiness = 0 in /etc/sysctl.conf.

Is there a command to show me what processes the memory in swap belongs
to?

Wonko

I want to play movies without hangs
 

On Thu, Feb 16, 2012 at 7:29 AM, Alex Schuster <wonko@wonkology.org> wrote:
> Hi there!
>

Hi back at ya.

> How can I find out in such a case which processes are waiting
> for I/O? top showed nothing.

iotop is your friend.

I'll write more when I get some time to think

HTH,
Mark

I want to play movies without hangs
 

On Thu, Feb 16, 2012 at 9:29 AM, Alex Schuster <wonko@wonkology.org> wrote:
> I've written here in the past about my performance problems. My dual-core
> had trouble playing movies without stuttering when there was I/O. It was
> mainly swapping that caused this, and 8 G were not enough for me running
> KDE4.
>
> Then my hardware broke, and I got new one, except for the system hard
> drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
> RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing movies
> got even worse!

You don't mention anything about video card or video driver setup.
That's the first thing I would suspect.

What video card? What drivers? Are you using hardware accelerated
movie playback?

I want to play movies without hangs
 

Paul Hartman writes:

> On Thu, Feb 16, 2012 at 9:29 AM, Alex Schuster <wonko@wonkology.org>
> wrote:

> > Then my hardware broke, and I got new one, except for the system hard
> > drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
> > RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing
> > movies got even worse!
>
> You don't mention anything about video card or video driver setup.
> That's the first thing I would suspect.
>
> What video card? What drivers? Are you using hardware accelerated
> movie playback?

Sorry. Radeon HD 4250 onboard graphics, using the open source radeon
driver. Hardware acceleration is working fine. As I wrote, it doesn't
matter which quality the videos are. There is not much CPU being used at
all, around 5% to 20%, so this is not the bottleneck.

Wonko

I want to play movies without hangs
 

On Thu, Feb 16, 2012 at 10:49 AM, Alex Schuster <wonko@wonkology.org> wrote:
> Paul Hartman writes:
>
>> On Thu, Feb 16, 2012 at 9:29 AM, Alex Schuster <wonko@wonkology.org>
>> wrote:
>
>> > Then my hardware broke, and I got new one, except for the system hard
>> > drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
>> > RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing
>> > movies got even worse!
>>
>> You don't mention anything about video card or video driver setup.
>> That's the first thing I would suspect.
>>
>> What video card? What drivers? Are you using hardware accelerated
>> movie playback?
>
> Sorry. Radeon HD 4250 onboard graphics, using the open source radeon
> driver. Hardware acceleration is working fine. As I wrote, it doesn't
> matter which quality the videos are. There is not much CPU being used at
> all, around 5% to 20%, so this is not the bottleneck.
>
> * * * *Wonko
>

I wonder if you copy the movie to /dev/shm first (so disk I/O is not
an issue) does it still have problems? At least this can potentially
eliminate disk I/O as the cause if something else weird is going on.
:)

For the problem of massive amounts of RAM consumed, that's strange.
Are you compiling debug symbols? That can make the RAM usage (in
linking especially) explode...

I want to play movies without hangs
 

Mark Knecht writes:

> On Thu, Feb 16, 2012 at 7:29 AM, Alex Schuster <wonko@wonkology.org>
> wrote:

> > How can I find out in such a case which processes are waiting
> > for I/O? top showed nothing.
>
> iotop is your friend.

I had called it, but didn't spot the problem there. I don't remember
exactly what the output was, I had expected to see some process show a
large value in the IO column, but that was not the case. I THINK! Various
processes appeared, mostly Akonadi stuff.

I should have logged this, I don't remember this s well. I have caught
some illness, and had fever, which did not help my memory. There were
some kworker processes listed on top, but I don't remember whether in
iotop or in top.

Hmm. Now I just started Akonadi again, and get a lot I/O in iotop, and
xosview again shows much iowait CPU activity for a while. But that calmed
down after a minute.

> I'll write more when I get some time to think

Thanks :)

Wonko

I want to play movies without hangs
 

On Thu, Feb 16, 2012 at 9:34 AM, Alex Schuster <wonko@wonkology.org> wrote:
> Mark Knecht writes:
>
>> On Thu, Feb 16, 2012 at 7:29 AM, Alex Schuster <wonko@wonkology.org>
>> wrote:
>
>> > How can I find out in such a case which processes are waiting
>> > for I/O? top showed nothing.
>>
>> iotop is your friend.
>
> I had called it, but didn't spot the problem there. I don't remember
> exactly what the output was, I had expected to see some process show a
> large value in the IO column, but that was not the case. I THINK! Various
> processes appeared, mostly Akonadi stuff.
>
> I should have logged this, I don't remember this s well. I have caught
> some illness, and had fever, which did not help my memory. There were
> some kworker processes listed on top, but I don't remember whether in
> iotop or in top.
>
> Hmm. Now I just started Akonadi again, and get a lot I/O in iotop, and
> xosview again shows much iowait CPU activity for a while. But that calmed
> down after a minute.
>
>> I'll write more when I get some time to think
>
> Thanks :)
>
> * * * *Wonko
>

Sorry. I was rushing then as now. If you start iotop and then hit the
'o' key it will show only processes actually doing io. If you're hang
is truly an iowait then my experience is that it should at least
identify what process is having the problem.

HTH,
Mark

I want to play movies without hangs
 

On Thu, Feb 16, 2012 at 04:29:48PM +0100, Alex Schuster wrote

> Then my hardware broke, and I got new one...

I had ***EXACTLY THE SAME PROBLEM ON A FRESH INSTALL***. In My case
it was a 4+ year old Dell with onboard Intel GPU that was having
problems playing NHL Gamecenter Live streams at the slowest speed. I
solved the problem and sped up everything by doing...
1) emerge system
2) emerge world
3) rebuild the kernel and reboot

A fresh install will have the stage 3 binaries built with
lowest-common-denominator x86 or amd64 code (depending if you chose 32
or 64 bit install). This is necessary in order to allow the install
code to run on all CPUs with the target platform. The downside is that
you lose all the optimisations that make Gentoo scream. Rebuilding the
install as described above builds optimized (i.e. faster) binaries. My
CFLAGS line in /etc/make.conf is...

CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe"
CXXFLAGS="${CFLAGS}"

Before rebuilding your system, go over your USE flags to make sure
you've got the maximum optimization. To find out what your CPU
supports, execute the command

grep flags /proc/cpuinfo | head -1

This will define the limits what your system can support. For
instance, mplayer can use the following flags...

waltdnes@d530 ~ $ emerge -pv mplayer

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R ] media-video/mplayer-1.0_rc4_p20110322-r1 USE="X a52 alsa ass dga encode gif jpeg mmx mmxext mng mp3 opengl png quicktime real rtmp sse sse2 ssse3 theora truetype win32codecs x264 xv xvid xvmc -3dnow -3dnowext -aalib (-altivec) -amr (-aqua) -bidi -bindist -bl -bluray -bs2b -cddb -cdio -cdparanoia -cpudetection -custom-cpuopts -debug -dirac -directfb -doc -dts -dv -dvb -dvd -dvdnav (-dxr3) -enca (-esd) -faac -faad -fbcon -ftp -ggi -gsm -iconv -ipv6 -jack -joystick -jpeg2k -ladspa -libcaca -libmpeg2 -lirc -live -lzo -mad -md5sum -mpg123 -nas -network -nut -openal -osdmenu -oss -pnm -pulseaudio -pvr -radio -rar -rtc -samba -schroedinger -sdl -shm -speex -tga -toolame -tremor -twolame -unicode -v4l -vdpau -vidix -vorbis -vpx -xanim -xinerama -xscreensaver -zoran" VIDEO_CARDS="-mga -s3virge -tdfx -vesa" 0 kB

Your CPU will obviously support a different set of USE flags than
mine. Check the files /usr/portage/profiles/use.desc for a list of
global flags and /usr/portage/profiles/use.local.desc for
package-specific flags.

--
Walter Dnes <waltdnes@waltdnes.org>

I want to play movies without hangs
 

Paul Hartman writes:

> I wonder if you copy the movie to /dev/shm first (so disk I/O is not
> an issue) does it still have problems? At least this can potentially
> eliminate disk I/O as the cause if something else weird is going on.
> :)

Yes, this helps. As does copying the movie to another partition than that
on which I to the I/O with my dd if=/dev/zero of= command. If I dd to
this partition, tough, it happens again.

> For the problem of massive amounts of RAM consumed, that's strange.

It's been so for years... but with 16 G of RAM it's no longer an issue.
Well, unless this weird problem happened wth parallel emerges on tmpfs. It
doesn't happen every time though.

> Are you compiling debug symbols? That can make the RAM usage (in
> linking especially) explode...

No. I sometimes enable it, but only when I hunt a bug and want to produce
better bug reports.
And even if the emerge would need very much memory, shouldn't this be
taken from the 8 G of caches being used, instead of starting to swap?

Wonko

I want to play movies without hangs
 

Walter Dnes writes:

> On Thu, Feb 16, 2012 at 04:29:48PM +0100, Alex Schuster wrote
>
> > Then my hardware broke, and I got new one...
>
> I had ***EXACTLY THE SAME PROBLEM ON A FRESH INSTALL***. In My case
> it was a 4+ year old Dell with onboard Intel GPU that was having
> problems playing NHL Gamecenter Live streams at the slowest speed. I
> solved the problem and sped up everything by doing...
> 1) emerge system
> 2) emerge world
> 3) rebuild the kernel and reboot

Good idea, Walter! But not in my case. The system had been set up long
ago, and I did an emerge -e @world in the past already.

> A fresh install will have the stage 3 binaries built with
> lowest-common-denominator x86 or amd64 code (depending if you chose 32
> or 64 bit install). This is necessary in order to allow the install
> code to run on all CPUs with the target platform. The downside is that
> you lose all the optimisations that make Gentoo scream. Rebuilding the
> install as described above builds optimized (i.e. faster) binaries. My
> CFLAGS line in /etc/make.conf is...
>
> CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe"
> CXXFLAGS="${CFLAGS}"

I had those, when I upgraded the hardware:
CFLAGS="-march=k8-sse3 -mfpmath=sse -O2 -pipe"
Or something very silimar.

But I also just did an emerge -e @world on the new system, using more
sophisticated CFLAGS. I got them by doing like suggested on[*], using
what -march=natve would do. And adding support for this graphite
stuff. They are:
CFLAGS="-pipe -march=amdfam10 -O2
-floop-interchange -floop-strip-mine -floop-block
-msse -msse2 -msse3 -msse4 -msse4.1 -msse4.2 -m3dnow
-mcx16 -msahf -maes -mpclmul -mpopcnt -mabm -mlwp -mavx
--param l1-cache-size=16 --param l1-cache-line-size=64
--param l2-cache-size=2048"

> Before rebuilding your system, go over your USE flags to make sure
> you've got the maximum optimization. To find out what your CPU
> supports, execute the command
>
> grep flags /proc/cpuinfo | head -1
>
> This will define the limits what your system can support. For
> instance, mplayer can use the following flags...
>
> waltdnes@d530 ~ $ emerge -pv mplayer
>
> These are the packages that would be merged, in order:
>
> Calculating dependencies... done!
> [ebuild R ] media-video/mplayer-1.0_rc4_p20110322-r1 USE="X a52
> alsa ass dga encode gif jpeg mmx mmxext mng mp3 opengl png quicktime
> real rtmp sse sse2 ssse3 theora truetype win32codecs x264 xv xvid xvmc
> -3dnow -3dnowext -aalib (-altivec) -amr (-aqua) -bidi -bindist -bl
> -bluray -bs2b -cddb -cdio -cdparanoia -cpudetection -custom-cpuopts
> -debug -dirac -directfb -doc -dts -dv -dvb -dvd -dvdnav (-dxr3) -enca
> (-esd) -faac -faad -fbcon -ftp -ggi -gsm -iconv -ipv6 -jack -joystick
> -jpeg2k -ladspa -libcaca -libmpeg2 -lirc -live -lzo -mad -md5sum
> -mpg123 -nas -network -nut -openal -osdmenu -oss -pnm -pulseaudio -pvr
> -radio -rar -rtc -samba -schroedinger -sdl -shm -speex -tga -toolame
> -tremor -twolame -unicode -v4l -vdpau -vidix -vorbis -vpx -xanim
> -xinerama -xscreensaver -zoran" VIDEO_CARDS="-mga -s3virge -tdfx -vesa"
> 0 kB
>
> Your CPU will obviously support a different set of USE flags than
> mine. Check the files /usr/portage/profiles/use.desc for a list of
> global flags and /usr/portage/profiles/use.local.desc for
> package-specific flags.

These are my USE flags for mplayer, they should be fine:
[ebuild R ] media-video/mplayer-1.0_rc4_p20120213 USE="3dnow
3dnowext X a52 aalib alsa ass cdio dga directfb dts dv dvb dvd dvdnav
enca encode faad fbcon ftp ggi gif iconv ipv6 jack jpeg jpeg2k live mad
mmx mmxext mng mp3 nas network openal opengl osdmenu oss png pnm
quicktime rar real rtc samba sdl shm speex sse sse2 ssse3 theora toolame
tremor truetype twolame unicode vorbis x264 xinerama xscreensaver xv xvid
(-altivec) (-aqua) -bidi -bindist -bl -bluray -bs2b -cddb -cdparanoia
-cpudetection -debug -doc (-dxr3) (-esd) -faac -gsm -joystick -ladspa
-libcaca -libmpeg2 -lirc -lzo -md5sum -nut -pulseaudio -pvr -radio -rtmp
-tga -v4l -vdpau (-vidix) (-win32codecs) -xanim -xvmc -zoran"
VIDEO_CARDS="-mga -s3virge -tdfx" 0 kB

Now I'm bulding a new kernel, using genkernel, and without providing a
custom made .config. Just in case I have some weird setting somewhere
(debug output for SCSI stuff or something like that).

[later...]

So I did. Argh. I thought genkernel was smart enough to generate a
working kernel from scratch, if no existing .config would be given. But
the initramfs could not open my encrypted root partition, until I compiled
XTS and AES directly into the kernel, not only as modules. Genkernel did
not include modules for my NIC, somewhat annoying because I had to wait
several minutes for mysql to start, until I could open a root shell. KDM
was already running at that time, but I only saw a blank screen, because
the radeon stuff was not compiled with KMS. There's also something
wrong with my hardware clock. And iotop does not work, the kernel is
missing CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT
and CONFIG_TASK_IO_ACCOUNTING.

But now I have KDE running again. As soon as I do my dd command, mplayer
stutters, sometimes hanging for seconds. So it's probably not a flaw in
my kernel .config. Too bad, I hoped that was the problem.

I'm out of ideas now. This is really annoying, my system is fast, mplayer
is using around 20% only, the system is mostly idle, but when there is
I/O, videos do not run smoothly. And I do not want to copy every movie I
want to watch to another drive first.

BTW, when I do the dd if=/dev/zero of=/home/argh bs=10M count=1000
command, and interrupt with Ctrl-C, it takes some seconds until it stops.
Shouldn't this happen immediately, unless I specify a very large block
size? Writing speed is okay, 100 MB/s.

Time to go to sleep now.
[*] http://en.gentoo-wiki.com/wiki/Safe_Cflags#-march.3Dnative

Wonko