FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 05-25-2012, 11:04 AM
Thorsten Glaser
 
Default Moving /tmp to tmpfs makes it useful

Henrique de Moraes Holschuh dixit:

>On Fri, 25 May 2012, Thomas Goirand wrote:
>> for small files, and in that case, it's faster. In reality, it's
>> not that much faster, thanks to Linux caching of the filesystem,
>
>Under heavy filesystem IO load, yes it is. By several orders of magnitude.

Not just that.

It’s faster because the files don’t even get written to disc if
they’re kept for, say, five minutes, and aren’t just short-lived.

If you’ve got a slow hard disc (say low-end spec or old system),
RAM’s still cheaper for tmpfs: no matter whether tmpfs or a real
on-disc filesystem, buffer cache would still get it, AND the writes
would be very slow (especially for short-lived files, _if_ the system
is otherwise busy and would not wait before writing them to disc),
and the system could concentrate on getting the files we want to
disc instead of the temporary ones.

And then there’s the CF card usecase. Or SSD, nowadays, I guess
(my IBM X40 just got two CF cards to replace the HDD). There, you’d
absolutely want to minimise disc writes. And just the chance that
it could not end up there (except as part of swap) is better then
the chance it could end up there (plus journal writes).

>In fact it is the other way. We have /var/tmp for the large file since
>about forever, and important platforms that have /tmp in memory since the
>early 2000's (Solaris)....

Right. /tmp always had size limitations, it used to be no bigger
than a few MiB, and people were ALWAYS tought to use /var/tmp for
files that are large, should not be removed across reboots or from
a cronjob pruning /tmp files older than 7 days, or a combination
of these. (Uglities like /usr/tmp notwithstanding, although I’ve
mostly seen that as compat symlink to /var/tmp later.)

I’m absolutely for /tmp as tmpfs by default. Even, no, especially
on low-end systems. Heck, be lucky it’s tmpfs and not Classical
BSDs’ mfs (basically a variable-size 4.2FFS ramdisc, not just
an object store in the buffer cache).

Every tool either supports setting TMPDIR=/var/tmp before running
it or is buggy. Go fix these instead, and then just run them with
that if you need them to process files you don’t want in tmpfs.

bye,
//mirabilos
--
[...] if maybe ext3fs wasn't a better pick, or jfs, or maybe reiserfs, oh but
what about xfs, and if only i had waited until reiser4 was ready... in the be-
ginning, there was ffs, and in the middle, there was ffs, and at the end, there
was still ffs, and the sys admins knew it was good. -- Ted Unangst über *fs


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: Pine.BSM.4.64L.1205251057460.4146@herc.mirbsd.org" >http://lists.debian.org/Pine.BSM.4.64L.1205251057460.4146@herc.mirbsd.org
 
Old 05-25-2012, 12:50 PM
Serge
 
Default Moving /tmp to tmpfs makes it useful

2012/5/25 Thorsten Glaser wrote:

> It’s faster because the files don’t even get written to disc if
> they’re kept for, say, five minutes, and aren’t just short-lived.

Theoretically, yes. But I'm not asking to forbid everybody to use
tmpfs. We're talking about defaults and about the most used test
cases. Can you name some popular real-world program that write
only small files and become noticeably faster when /tmp is on tmpfs?

> Every tool either supports setting TMPDIR=/var/tmp before running
> it or is buggy. Go fix these instead

Do I understand you right? You suggest every tool that need large
tmp files to use /var/tmp instead? Ok, then... what popular tools
should use /tmp now?

> And then there’s the CF card usecase. Or SSD, nowadays, I guess
> (my IBM X40 just got two CF cards to replace the HDD). There, you’d
> absolutely want to minimise disc writes.

/tmp on tmpfs won't help in that case. You do not reduce number of disk
writes, you just move them to other directories. As you suggested, all
the programs will still use files i.e. in /var/tmp. Or they'll write to
swap if tmpfs is large enough. Or they'll just break, if it's small.

But you'll get the same number of SSD writes (or maybe even more,
because of other applications being heavily swapped as well).

Again, I'm not asking to drop this feature. I'm asking to have it disabled
*by default* but supported by debian installer, so really smart people,
that know what may be broken by it, but really need it, could enable it.

--
Serge


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAOVenEp+mr3yNi=4obCG2qtztFjuAJ_SvcUS1Y-7iJ5mNhj9GA@mail.gmail.com">http://lists.debian.org/CAOVenEp+mr3yNi=4obCG2qtztFjuAJ_SvcUS1Y-7iJ5mNhj9GA@mail.gmail.com
 
Old 05-25-2012, 12:56 PM
Andrey Rahmatullin
 
Default Moving /tmp to tmpfs makes it useful

On Fri, May 25, 2012 at 03:50:31PM +0300, Serge wrote:
> /tmp on tmpfs won't help in that case. You do not reduce number of disk
> writes, you just move them to other directories. As you suggested, all
> the programs will still use files i.e. in /var/tmp. Or they'll write to
> swap if tmpfs is large enough. Or they'll just break, if it's small.
Why do you assume tmpfs contents is always written to the swap?

--
WBR, wRAR
 
Old 05-25-2012, 01:49 PM
Will Daniels
 
Default Moving /tmp to tmpfs makes it useful

On 25/05/12 13:50, Serge wrote:


Again, I'm not asking to drop this feature. I'm asking to have it disabled
*by default* but supported by debian installer, so really smart people,
that know what may be broken by it, but really need it, could enable it.


+1

On 25/05/12 13:52, Ted Ts'o wrote:

So what? If you write to a normal file system, it goes into the page
cache, which is pretty much the same as writing into tmpfs. In both
cases if you have swap configured, the data will get pushed to disk;


That's not at all the same, the page cache is more temporary, it's getting
flushed to disk pretty quick if memory is tight (presumably) but in the same
situation using tmpfs going to swap is surely going to be more disruptive?


I won't pretend to know the details half as well as other commentators but it
seems only logical that you'd end up pushing more memory from other running
processes onto disk as well as (or instead of) the tmpfs memory, which is going
to have to get reloaded at some point.


And anyway, not everybody uses swap, in which case this "default" is not
entirely viable. I, for one, had no idea this had become default for Debian and
I think it's likely to be one of those things that jumps out to bite people who
weren't expecting it at some inconvenient moment.


I'm sure the project veterans and more attentive readers of this list are tired
of recurring arguments like this, but usually if something is recurring it is
for a reason. Given my general "no swap" preference, I'm glad this has come up
again so that I'm aware of it this time.


The tmpfs setup seems far more appropriate as a performance tweak for admins
than as a default. Where there is plenty of RAM, buffer cache makes the
difference largely negligible. But where there isn't an abundance of RAM, it
could needlessly cause problems (especially without swap).


Not big problems perhaps, and not likely to many, but that could still be a few
thousand people with a project like Debain, so I just don't see the issue with
leaving /tmp on disk, why complicate the matter?


-Will


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FBF8DDA.9070107@willdaniels.co.uk">http://lists.debian.org/4FBF8DDA.9070107@willdaniels.co.uk
 
Old 05-25-2012, 02:08 PM
Serge
 
Default Moving /tmp to tmpfs makes it useful

2012/5/25 Andrey Rahmatullin wrote:

> Why do you assume tmpfs contents is always written to the swap?

Most programs, using /tmp for temporary files, may write *large* files
there. So most programs fill tmpfs with large files. So tmpfs grows and
swaps out (probably swapping out other applications as well).

I really can't think of any popular application that write a lot of
small-only files and can benefit from /tmp being on tmpfs. And I
know a lot of *popular* apps that will have troubles because of that.

If default settings cause trouble in a lot of cases and cause no benefits,
than it's a bad default, and it should be fixed.

--
Serge


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAOVenEqkgq7pUD1xeneRvFxwzRh3qXk_xD9eWeUz7nKjDT2Ch g@mail.gmail.com">http://lists.debian.org/CAOVenEqkgq7pUD1xeneRvFxwzRh3qXk_xD9eWeUz7nKjDT2Ch g@mail.gmail.com
 
Old 05-25-2012, 02:20 PM
Andrey Rahmatullin
 
Default Moving /tmp to tmpfs makes it useful

On Fri, May 25, 2012 at 05:08:37PM +0300, Serge wrote:
> > Why do you assume tmpfs contents is always written to the swap?
>
> Most programs, using /tmp for temporary files, may write *large* files
> there. So most programs fill tmpfs with large files. So tmpfs grows and
> swaps out (probably swapping out other applications as well).
>
> I really can't think of any popular application that write a lot of
> small-only files and can benefit from /tmp being on tmpfs. And I
> know a lot of *popular* apps that will have troubles because of that.
/tmp 8,0G 60M 8,0G 1% /tmp
Does this count as "large files"?
As "a lot of small-only files"?


--
WBR, wRAR
 
Old 05-25-2012, 04:21 PM
Thorsten Glaser
 
Default Moving /tmp to tmpfs makes it useful

Serge dixit:

>cases. Can you name some popular real-world program that write
>only small files and become noticeably faster when /tmp is on tmpfs?

Most shell scripts using << (here documents). mc, out of all things,
as long as the temporary files (e.g. of a tarball) fit inside it.
And countless others. The majority, in any case. What you want is
to optimise for a corner case.

>> Every tool either supports setting TMPDIR=/var/tmp before running
>> it or is buggy. Go fix these instead
>
>Do I understand you right? You suggest every tool that need large
>tmp files to use /var/tmp instead?

I do not suggest it. I say that every tool that doesn’t honour when
the user sets TMPDIR=/var/tmp is broken (but that’s nothing new, as
it has always been like that), and that the user should set that in
some corner cases like yours.

>Ok, then... what popular tools should use /tmp now?

/tmp (on tmpfs) should still be the default, as that’s the common
case, and one that can be optimised for well.

>/tmp on tmpfs won't help in that case. You do not reduce number of disk
>writes, you just move them to other directories. As you suggested, all

No. You reduce the number of disc writes quite a bit, as all current
filesystems use journalling and write more additional metadata than
paging would ever use. (And, from the other mail, the read accesses
could be fixed (for the slow HDD case) or don’t matter that much (for
the CF/SSD case).)

>the programs will still use files i.e. in /var/tmp. Or they'll write to
>swap if tmpfs is large enough. Or they'll just break, if it's small.

Ever heard of statfs?

>But you'll get the same number of SSD writes (or maybe even more,
>because of other applications being heavily swapped as well).

No. The common case wouldn’t swap them a lot, besides read-only
things (.text and .rodata, plus some bits) would not need to
be written anyway. So you’d still get less.

>Again, I'm not asking to drop this feature. I'm asking to have it disabled
>*by default* but supported by debian installer, so really smart people,
>that know what may be broken by it, but really need it, could enable it.

And I’m proving that it should be *enabled by default*, so that
really smart people like you, who need it disabled in corner
cases, can do that. And I’m suggesting that file managers that
extract archives apply some heuristics (size of the archive vs.
size and current usage of /tmp) to determine whether they should
use /tmp or /var/tmp.

>Most programs, using /tmp for temporary files, may write *large* files

No. The absolute majority writes *small* files there.

>there. So most programs fill tmpfs with large files. So tmpfs grows and
>swaps out (probably swapping out other applications as well).

You can always limit the size of your tmpfs? I think in Debian it
is already sized to a quite small (IMHO) portion of the main memory
by default.

>I really can't think of any popular application that write a lot of
>small-only files and can benefit from /tmp being on tmpfs. And I

$GUI_BROWSER_DU_JOUR with XTaran’s unburden-home-dir, for example.

bye,
//mirabilos
--
"Using Lynx is like wearing a really good pair of shades: cuts out
the glare and harmful UV (ultra-vanity), and you feel so-o-o COOL."
-- Henry Nelson, March 1999


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: Pine.BSM.4.64L.1205251615020.20173@herc.mirbsd.org ">http://lists.debian.org/Pine.BSM.4.64L.1205251615020.20173@herc.mirbsd.org
 
Old 05-25-2012, 07:24 PM
Thomas Goirand
 
Default Moving /tmp to tmpfs makes it useful

On 05/25/2012 10:20 PM, Andrey Rahmatullin wrote:
> /tmp 8,0G 60M 8,0G 1% /tmp
> Does this count as "large files"?
> As "a lot of small-only files"?
>
Exactly how is this a practical explanation and example? :/
Are you saying that in *your case* /tmp is almost unused?
That doesn't even remotely help to write this. Serge made
a much better point with common apps.

Thomas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FBFDC52.20202@debian.org">http://lists.debian.org/4FBFDC52.20202@debian.org
 
Old 05-25-2012, 07:26 PM
Thomas Goirand
 
Default Moving /tmp to tmpfs makes it useful

On 05/26/2012 12:21 AM, Thorsten Glaser wrote:
> What you want is
> to optimise for a corner case.
>
Ah... So Mysql, flash videos, mc, image editors,
cd burners, and all the (very common) examples
he gave are just "corner cases" in your book?
We must have very different readings...

Thomas


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FBFDD03.3050206@debian.org">http://lists.debian.org/4FBFDD03.3050206@debian.org
 
Old 05-26-2012, 12:32 AM
Miles Bader
 
Default Moving /tmp to tmpfs makes it useful

Serge <sergemdev@gmail.com> writes:
>> Every tool either supports setting TMPDIR=/var/tmp before running
>> it or is buggy. Go fix these instead
>
> Do I understand you right? You suggest every tool that need large
> tmp files to use /var/tmp instead?

That's not a new suggestion, it's been standard practice for nigh-on
_decades_...

-miles

--
Do not taunt Happy Fun Ball.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87zk8vdbnx.fsf@catnip.gol.com">http://lists.debian.org/87zk8vdbnx.fsf@catnip.gol.com
 

Thread Tools




All times are GMT. The time now is 07:14 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org