FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 11-12-2011, 08:24 PM
Bastien ROUCARIES
 
Default /tmp as tmpfs and consequence for imaging software

Hello,

Recently debian put /tmp under tmpfs.

Even if it increase reponsivness under desktop, it ruin completly
sciene and imaging software that do some off loading on /tmp.

For instance using gscan2pdf on 60pages document create more than 1.2G
of image file under /tmp and crash du to missing space.

What are the solution for this kind of problem ?

Thanks

Bastien


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAE2SPAZZY-mbOXaez2yhtoOq+juU0hPJC_8aB0QD4vgEGo14Bg@mail.gmai l.com">http://lists.debian.org/CAE2SPAZZY-mbOXaez2yhtoOq+juU0hPJC_8aB0QD4vgEGo14Bg@mail.gmai l.com
 
Old 11-12-2011, 09:08 PM
Adam Borowski
 
Default /tmp as tmpfs and consequence for imaging software

On Sat, Nov 12, 2011 at 10:24:00PM +0100, Bastien ROUCARIES wrote:
> Hello,
>
> Recently debian put /tmp under tmpfs.
>
> Even if it increase reponsivness under desktop, it ruin completly
> sciene and imaging software that do some off loading on /tmp.
>
> For instance using gscan2pdf on 60pages document create more than 1.2G
> of image file under /tmp and crash du to missing space.
>
> What are the solution for this kind of problem ?

You need to increase the swap size by the amount you'd use for /tmp. Or, if
you have plenty of RAM, merely increase the cap but then it may have no
place to be swapped to if you'd want the RAM for other uses.

Which raises a question what is a reasonable /tmp size. I doubt a system
where someone even considers scanning a 1.2G document would be so tight to
not afford 2.4G of RAM+swap (the /tmp cap is still set to 50%, right?).
Thus, this suggests it might be a good idea to adjust d-i defaults.

--
1KB // Yo momma uses IPv4!
 
Old 11-12-2011, 09:12 PM
Samuel Thibault
 
Default /tmp as tmpfs and consequence for imaging software

Adam Borowski, le Sat 12 Nov 2011 23:08:08 +0100, a écrit :
> You need to increase the swap size by the amount you'd use for /tmp.

Well, the idea of such case is precisely to *not* use swap, but real
disks. Such software already know how to manage its memory and
disk-backed memory (thusly stored in /tmp)

Samuel


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20111112221236.GT4465@type.famille.thibault.fr">ht tp://lists.debian.org/20111112221236.GT4465@type.famille.thibault.fr
 
Old 11-12-2011, 09:25 PM
Josselin Mouette
 
Default /tmp as tmpfs and consequence for imaging software

Le samedi 12 novembre 2011 à 23:12 +0100, Samuel Thibault a écrit :
> Adam Borowski, le Sat 12 Nov 2011 23:08:08 +0100, a écrit :
> > You need to increase the swap size by the amount you'd use for /tmp.
>
> Well, the idea of such case is precisely to *not* use swap, but real
> disks. Such software already know how to manage its memory and
> disk-backed memory (thusly stored in /tmp)

Practically speaking, the only significant difference is that files are
not forced to disk as early. Otherwise, if you have a large enough swap,
pages of a file on a tmpfs that are not used enough will be swapped. And
pages of a file on a regular filesystem that are used enough will be
kept in the buffer cache.

OTOH, for a wide range of applications that do a lot of small writes,
using tmpfs is a huge gain.

--
.'`. Josselin Mouette
: :' :
`. `'
`-
 
Old 11-12-2011, 09:35 PM
Ben Hutchings
 
Default /tmp as tmpfs and consequence for imaging software

On Sat, 2011-11-12 at 22:24 +0100, Bastien ROUCARIES wrote:
> Hello,
>
> Recently debian put /tmp under tmpfs.
>
> Even if it increase reponsivness under desktop, it ruin completly
> sciene and imaging software that do some off loading on /tmp.
>
> For instance using gscan2pdf on 60pages document create more than 1.2G
> of image file under /tmp and crash du to missing space.
>
> What are the solution for this kind of problem ?

This problem has little to do with use of tmpfs; it is not reasonable to
assume that /tmp has that much space. Programs should generally allow
use of /tmp to be overridden by $TMPDIR and should handle disk-full
errors gracefully.

Ben.

--
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.
 
Old 11-12-2011, 09:39 PM
Geoffrey Thomas
 
Default /tmp as tmpfs and consequence for imaging software

On Sat, 12 Nov 2011, Bastien ROUCARIES wrote:


Hello,

Recently debian put /tmp under tmpfs.

Even if it increase reponsivness under desktop, it ruin completly
sciene and imaging software that do some off loading on /tmp.

For instance using gscan2pdf on 60pages document create more than 1.2G
of image file under /tmp and crash du to missing space.

What are the solution for this kind of problem ?


Can you make this software use /var/tmp instead of /tmp, and would that
address your issue?


There's some discussion about /var/tmp vs. /tmp, as part of a larger
discussion on /tmp as tmpfs, in http://bugs.debian.org/630615 .


--
Geoffrey Thomas
http://ldpreload.com
geofft@ldpreload.com



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: alpine.DEB.2.00.1111121728460.28002@tyger.mit.edu" >http://lists.debian.org/alpine.DEB.2.00.1111121728460.28002@tyger.mit.edu
 
Old 11-12-2011, 11:28 PM
Roger Leigh
 
Default /tmp as tmpfs and consequence for imaging software

On Sat, Nov 12, 2011 at 10:24:00PM +0100, Bastien ROUCARIES wrote:
> Recently debian put /tmp under tmpfs.
>
> Even if it increase reponsivness under desktop, it ruin completly
> sciene and imaging software that do some off loading on /tmp.
>
> For instance using gscan2pdf on 60pages document create more than 1.2G
> of image file under /tmp and crash du to missing space.
>
> What are the solution for this kind of problem ?

As mentioned elsewhere in this thread, this is discussed in detail
in #630615.

As touched on in the bug report, I think that being able to store
1.2GiB on /tmp is an unrealistic expectation. To qualify, I mean
to expect that to work *by default*. If you want to store such
large amounts of data, you will need to configure your system to
handle that, either by:

- provisioning of more swap and raising of the TMP_SIZE limit.
- disabling RAMTMP and using a disc-backed filesystem (either the
rootfs or dedicated /tmp mount).

Again, as mentioned in the report, due to the wide variation in
disc partitioning, filesystem utilisation and RAM capacity between
systems, we don't currently make *any* guarantees regarding a
minimum amount of space available in /tmp, when using a disc-backed
/tmp. If the rootfs fills up, /tmp will cease to allow creation of
new files. When using tmpfs, we do at least make a minimum
guarantee of having a certain amount of storage available (which
might albeit be used by other users).


I'm not sure that I can really add more at this point than which
was already included in the bug report. As a general rule, I think
it's fair to say that if you want to *guarantee* the availability of
that much storage, the defaults will not typically be sufficient.a But
the defaults are just defaults--you are free to configure your system
to satisfy your needs as you see fit.


Regards,
Roger

--
.'`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20111113002850.GX28537@codelibre.net">http://lists.debian.org/20111113002850.GX28537@codelibre.net
 
Old 11-12-2011, 11:59 PM
Adam Borowski
 
Default /tmp as tmpfs and consequence for imaging software

On Sat, Nov 12, 2011 at 11:25:12PM +0100, Josselin Mouette wrote:
> Le samedi 12 novembre 2011 à 23:12 +0100, Samuel Thibault a écrit :
> > Adam Borowski, le Sat 12 Nov 2011 23:08:08 +0100, a écrit :
> > > You need to increase the swap size by the amount you'd use for /tmp.
> >
> > Well, the idea of such case is precisely to *not* use swap, but real
> > disks. Such software already know how to manage its memory and
> > disk-backed memory (thusly stored in /tmp)
>
> Practically speaking, the only significant difference is that files are
> not forced to disk as early.

That, and you don't have to do umpteen metadata writes (including barriers!)
for every file written to. Real filesystems work with the assumption that
both the files need to be durable and the filesystem itself must be
consistent even after a crash, including caring about the disk controller
"maliciously" reordering writes. We're speaking of ~ seven seeks + writes
for creating an one-block file. Log-structured filesystems might not have
such journal churn but are still worse than tmpfs.

> Otherwise, if you have a large enough swap, pages of a file on a tmpfs
> that are not used enough will be swapped. And pages of a file on a
> regular filesystem that are used enough will be kept in the buffer cache.

In other words, tmpfs never[1] performs worse than an actual filesystem.
Usually the files won't be ever actually written to the disk, and if they
will, tmpfs will work equivalently (anonymous vs file-backed pages), minus
crash consistency costs.

There's a configuration issue caused by taking space from a different pool,
though.


[1]. Currently, if there are multiple concurrent writers of large files,
there will be fragmentation smart real filesystems have workarounds for.
This could be avoided by hinting the kernel that if it swaps down a block,
it should consider the next block of the same file rather than strictly
following LRU order. Such logic is far simpler than what real filesystems
have to do.

--
1KB // Yo momma uses IPv4!


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20111113005928.GC1888@angband.pl">http://lists.debian.org/20111113005928.GC1888@angband.pl
 
Old 11-13-2011, 02:04 AM
Carlos Alberto Lopez Perez
 
Default /tmp as tmpfs and consequence for imaging software

On 12/11/11 23:25, Josselin Mouette wrote:
> Le samedi 12 novembre 2011 à 23:12 +0100, Samuel Thibault a écrit :
>> Adam Borowski, le Sat 12 Nov 2011 23:08:08 +0100, a écrit :
>>> You need to increase the swap size by the amount you'd use for /tmp.
>>
>> Well, the idea of such case is precisely to *not* use swap, but real
>> disks. Such software already know how to manage its memory and
>> disk-backed memory (thusly stored in /tmp)
>
> Practically speaking, the only significant difference is that files are
> not forced to disk as early. Otherwise, if you have a large enough swap,
> pages of a file on a tmpfs that are not used enough will be swapped. And
> pages of a file on a regular filesystem that are used enough will be
> kept in the buffer cache.
>
There are another differences:


When the system is swapping heavily, if you are not using a preempt
kernel the hole system will become so unresponsive while the swapping
process is taking place that even your mouse pointer will stop moving.
And Debian kernel is no preempt.

So, if having /tmp with tmpfs will make your system to swap more often
(huge files on /tmp) then any performance gain by tmpfs will be buried
by this swapping hell.


Also, if you are near to run out of virtual memory (RAM+SWAP) for
whatever reason: few ram, many apps open... an application can trigger
the OOMKiller by simply writing data into /tmp if you are using tmpfs.


> OTOH, for a wide range of applications that do a lot of small writes,
> using tmpfs is a huge gain.
>

Linux already has a disk cache buffer that works very nice for this case
of doing lot of small read/writes into a file. Also, when its time to
flush the data to disk, or read data no previously cached, the kernel IO
scheduler will take care of optimizing it in order to reduce the disk
spinning and improve the performance.


So... while is true that tmpfs is faster than using the disk for /tmp,
it isn't such big deal.


And IMHO I don't think that this performance gain outweighs so clearly
the problems exposed that justify making tmpfs on /tmp the default on
Debian.
 
Old 11-13-2011, 05:33 AM
Ben Hutchings
 
Default /tmp as tmpfs and consequence for imaging software

On Sun, 2011-11-13 at 04:04 +0100, Carlos Alberto Lopez Perez wrote:
> On 12/11/11 23:25, Josselin Mouette wrote:
> > Le samedi 12 novembre 2011 à 23:12 +0100, Samuel Thibault a écrit :
> >> Adam Borowski, le Sat 12 Nov 2011 23:08:08 +0100, a écrit :
> >>> You need to increase the swap size by the amount you'd use for /tmp.
> >>
> >> Well, the idea of such case is precisely to *not* use swap, but real
> >> disks. Such software already know how to manage its memory and
> >> disk-backed memory (thusly stored in /tmp)
> >
> > Practically speaking, the only significant difference is that files are
> > not forced to disk as early. Otherwise, if you have a large enough swap,
> > pages of a file on a tmpfs that are not used enough will be swapped. And
> > pages of a file on a regular filesystem that are used enough will be
> > kept in the buffer cache.
> >
> There are another differences:
>
>
> When the system is swapping heavily, if you are not using a preempt
> kernel the hole system will become so unresponsive while the swapping
> process is taking place that even your mouse pointer will stop moving.
> And Debian kernel is no preempt.

Linux does not busy-wait for disk I/O and will keep switching between
tasks regardless of whether preemption is enabled. Further, since
version 2.6.31-1~experimental.1, all official kernel packages have
PREEMPT_VOLUNTARY enabled.

The problem you are describing is that physical memory becomes almost
full with the working set, dirty pages and write buffers and it is not
possible to write data to disk fast enough to keep up with tasks that
generate it. As this happens, the cache may have to shrink and reads
will then miss the cache more often. Further, the kernel must block
tasks that write to disk rather than allocating more write buffers for
asynchronous writeback. Suddenly many of your tasks become blocked on
the disk I/O queue. (Note that Linux 3.2 should improve I/O scheduling
in this case so that the slowdown isn't quite so bad.)

> So, if having /tmp with tmpfs will make your system to swap more often
> (huge files on /tmp) then any performance gain by tmpfs will be buried
> by this swapping hell.

But a very similar problem can occur with I/O to a conventional
filesystem. And it's worse in some ways - the kernel has to write
filesystem blocks back in the right order to keep the filesystem
consistent, which is not a concern for tmpfs.

> Also, if you are near to run out of virtual memory (RAM+SWAP) for
> whatever reason: few ram, many apps open... an application can trigger
> the OOMKiller by simply writing data into /tmp if you are using tmpfs.

In theory, yes. However the size of tmpfs is limited.

> > OTOH, for a wide range of applications that do a lot of small writes,
> > using tmpfs is a huge gain.
> >
>
> Linux already has a disk cache buffer that works very nice for this case
> of doing lot of small read/writes into a file. Also, when its time to
> flush the data to disk, or read data no previously cached, the kernel IO
> scheduler will take care of optimizing it in order to reduce the disk
> spinning and improve the performance.

That doesn't really help that much once the system is short of memory,
though.

> So... while is true that tmpfs is faster than using the disk for /tmp,
> it isn't such big deal.
>
>
> And IMHO I don't think that this performance gain outweighs so clearly
> the problems exposed that justify making tmpfs on /tmp the default on
> Debian.

Ben.

--
Ben Hutchings
Never attribute to conspiracy what can adequately be explained by stupidity.
 

Thread Tools




All times are GMT. The time now is 02:26 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org