Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian Development (http://www.linux-archive.org/debian-development/)
-   -   Summary: Moving /tmp to tmpfs makes it useless (http://www.linux-archive.org/debian-development/674275-summary-moving-tmp-tmpfs-makes-useless.html)

Wouter Verhelst 06-18-2012 09:42 PM

Summary: Moving /tmp to tmpfs makes it useless
 
On Wed, Jun 13, 2012 at 04:14:52AM +0300, Serge wrote:
> 2012/6/10 Wouter Verhelst wrote:
>
> >> A lot of people (including you) said that tmpfs makes things faster. But
> >> there were no examples of popular use-cases becoming faster because
> >> of /tmp on tmpfs, so I had nothing to quote.
> >
> > You're not even trying.
> >
> > if tmpfs is faster than (say) ext4, then anything which uses /tmp will
> > obviously speed up.
>
> That's not true. Only applications, that are limited by /tmp speed will
> become faster then. Do you know such applications?

Any application which performs I/O anywhere has a chance of being
limited by it.

If you write to /tmp on disk and someone or something calls "sync" at
precisely the wrong moment, you're stuck, and your performance suffers.
Not so with tmpfs.

I'm not saying the speedup will be extreme, but it will be there, and
(cumulated over loads of programs using /tmp) it will be significant
enough to be noticeable.

> > Can I provide a use case where this will matter? Not necessarily. But
> > then, can you provide a use case where this will *not* matter? Really?
>
> Yes. Everything.

Oh, interesting. You have the data to back that claim up?

> Every popular /tmp usage that most users expect to work
> is limited either by CPU (gcc compiling)

I don't think compiling C code has been CPU bound since before I was
born (and I was born in the late 70s, so that's quite a while). C++ is a
different matter, but still.

> or by network speed (browser or flash temporaries), or is just too
> fast already (bash heredoc).

You make it sound as if those three are the only uses of /tmp. That's
just not true.

Here are some other real-world uses of /tmp, which are (or can be) I/O
bound and where the size is significant enough that it makes a
difference:
- The nbd test suite stores fairly large files in $TMPDIR on which it
then runs nbd-server (yes, maybe that's cheating since I maintain nbd;
still, it's been that way for quite some time now, certainly since
before tmpfs was the default).
- Any data transformation or filtering which needs to be done in
multiple passes over a file would use a temporary file for
intermediate results, which it then reads in again for the next pass.
Multi-pass video transcoding is an example of this, and which
(depending on the codecs used and the hardware on which it runs) could
certainly be I/O bound.
- using mc on a tar.gz which was compressed with --fast

I could probably go on if I really wanted to, but that's fairly boring.

The point is that neither you nor I can reasonably be expected to list
all possible uses of /tmp; and that RAM is faster than disk, so that
when you access a tmpfs you're going to be somewhat faster than when you
access a disk-backed filesystem, at least until you start swapping (if
not longer).

Now whether that advantage outweighs the disadvantages you've outlined
is something we can talk about. However, whether RAM is faster than disk
isn't; and the fact that avoiding disk access will result in speedups,
no matter how small, isn't up for discussion either.

[...]
> >> Nobody could provide examples or numbers of how much /tmp on tmpfs
> >> reduces amount of writes, and tests showed that tmpfs+swap may even
> >> increase amount of writes (hence not always good for SSD),
> >
> > True, but then swapping to an SSD is the "best" idea since "640kB is
> > enough for everyone" :-)
>
> Hm, it's a bad idea to use tmpfs with swap... And it's not a good idea
> to use tmpfs without swap, since it would be too small and may even
> trigger OOM killer. When is it good to use tmpfs then? ;)

I never used tmpfs on a system with swap, and I've not often seen the
OOM killer start doing its job. My current machine has 4GiB of RAM,
which seems to be more than enough. The only exception was when I was
trying to run one VM too many, which then tried to eat up 110% of my
RAM, but that wasn't a good idea in any case.

[...]
> >> tmpfs does not have 5% overflow safety,
> >
> > Because it doesn't need it.
> > The 5% overflow safety exists for two reasons:
> > - to avoid excessive fragmentation (which is not relevant for tmpfs)
> > - to allow you to clean up when the filesystem does fill up.
>
> You missed one more reason. When user fills the entire /tmp on disk, it
> does not break system services, since root can still use those 5%.

Most system services do not run as root, so that's not a real advantage.
However, even if they did, your statement is still wrong:

> User cannot break the system filling /tmp on disk. But he can do that
> if he fills /tmp on tmpfs. So /tmp on tmpfs adds one more point of
> failure for servers.

No, that's not true. The real danger in filling up /tmp is not that
other processes can't write temporary files anymore (causing a minority
of processes to hang or die; those who just happen to need temporary
storage at that point in time), but that no process can write any file
anymore (causing a significant majority of processes to hang or die).

--
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120618214206.GY25995@grep.be">http://lists.debian.org/20120618214206.GY25995@grep.be

Uoti Urpala 06-19-2012 01:22 AM

Summary: Moving /tmp to tmpfs makes it useless
 
Wouter Verhelst wrote:
> I don't think compiling C code has been CPU bound since before I was
> born (and I was born in the late 70s, so that's quite a while). C++ is a
> different matter, but still.

Bullshit. GCC uses a lot of CPU unless you compile without optimization,
and is surprisingly slow even if you do (nowhere near linear disk access
speed).



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/1340068942.23020.5.camel@glyph.nonexistent.invalid

Goswin von Brederlow 06-19-2012 10:41 AM

Summary: Moving /tmp to tmpfs makes it useless
 
Wouter Verhelst <wouter@debian.org> writes:

> On Wed, Jun 13, 2012 at 04:14:52AM +0300, Serge wrote:
>> User cannot break the system filling /tmp on disk. But he can do that
>> if he fills /tmp on tmpfs. So /tmp on tmpfs adds one more point of
>> failure for servers.
>
> No, that's not true. The real danger in filling up /tmp is not that
> other processes can't write temporary files anymore (causing a minority
> of processes to hang or die; those who just happen to need temporary
> storage at that point in time), but that no process can write any file
> anymore (causing a significant majority of processes to hang or die).

So tmpfs for /tmp increases the chance of accidentally filling up /tmp
and causing a moderate failure but prevents malicious filling up of /tmp
to cause wide spread failure. :)

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/87ipend1nb.fsf@frosties.localnet

Stephan Seitz 06-20-2012 01:18 PM

Summary: Moving /tmp to tmpfs makes it useless
 
On Mon, Jun 18, 2012 at 11:42:06PM +0200, Wouter Verhelst wrote:

If you write to /tmp on disk and someone or something calls "sync" at
precisely the wrong moment, you're stuck, and your performance suffers.
Not so with tmpfs.


Maybe, but we are talking about defaults. Please correct me, but I think
that most Debian systems are in some way single user systems.



I'm not saying the speedup will be extreme, but it will be there, and
(cumulated over loads of programs using /tmp) it will be significant
enough to be noticeable.


If you train the user to keep away from /tmp, because it may be too small
with tmpfs, how much programs will use it then?



- Any data transformation or filtering which needs to be done in
multiple passes over a file would use a temporary file for
intermediate results, which it then reads in again for the next pass.
Multi-pass video transcoding is an example of this, and which
(depending on the codecs used and the hardware on which it runs) could
certainly be I/O bound.


I agree, but only if your tmpfs is big enough to hold the file. Ripping
DVDs or BDs will exceed any tmpfs limit on most systems.



The point is that neither you nor I can reasonably be expected to list
all possible uses of /tmp; and that RAM is faster than disk, so that
when you access a tmpfs you're going to be somewhat faster than when you
access a disk-backed filesystem, at least until you start swapping (if
not longer).


Nobody disagrees that RAM is faster than disk, but I hope you don’t
disagree as well that most people will have more disk space than RAM.



Now whether that advantage outweighs the disadvantages you've outlined
is something we can talk about. However, whether RAM is faster than disk


Fine let’s talk. Why can’t we find a compromise? Additional to our disk
/tmp we create a /ramtmp (so the name suggests that this tmp is
a ramdisk) with tmpfs. This should be doable in time for Wheezy. The
release notes should mention it. And those who wish can patch their
programs to use the ramdisk if they think their program uses only small
temporary files. In this way, we get some data and experience. And we
have both worlds. /tmp on disk for even large temporary files and /ramtmp
as fast ramdisk.



No, that's not true. The real danger in filling up /tmp is not that
other processes can't write temporary files anymore (causing a minority
of processes to hang or die; those who just happen to need temporary
storage at that point in time), but that no process can write any file
anymore (causing a significant majority of processes to hang or die).


But this will only happen if your /tmp is not a separate partition. And
it can happen with /var as well. I’ve seen programs logging so fast that
/var had no space left breaking the logging and the database.


We can now start another discussion what should be the best partition
layout for a default installation, but /tmp is not the only problem. And
/var/tmp exists as well for everyone writeable.


Stephan

--
| Stephan Seitz E-Mail: stse@fsing.rootsland.net |
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |

Carlos Alberto Lopez Perez 06-20-2012 07:08 PM

Summary: Moving /tmp to tmpfs makes it useless
 
On 20/06/12 15:18, Stephan Seitz wrote:
>>
>
> Fine let’s talk. Why can’t we find a compromise? Additional to our disk
> /tmp we create a /ramtmp (so the name suggests that this tmp is a
> ramdisk) with tmpfs. This should be doable in time for Wheezy. The
> release notes should mention it. And those who wish can patch their
> programs to use the ramdisk if they think their program uses only small
> temporary files. In this way, we get some data and experience. And we
> have both worlds. /tmp on disk for even large temporary files and
> /ramtmp as fast ramdisk.

I really think this is the way to go :)


--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
Carlos Alberto Lopez Perez http://neutrino.es
Igalia - Free Software Engineering http://www.igalia.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~

Wouter Verhelst 06-21-2012 07:06 AM

Summary: Moving /tmp to tmpfs makes it useless
 
On Wed, Jun 20, 2012 at 03:18:55PM +0200, Stephan Seitz wrote:
> On Mon, Jun 18, 2012 at 11:42:06PM +0200, Wouter Verhelst wrote:
> >If you write to /tmp on disk and someone or something calls "sync" at
> >precisely the wrong moment, you're stuck, and your performance suffers.
> >Not so with tmpfs.
>
> Maybe, but we are talking about defaults. Please correct me, but I
> think that most Debian systems are in some way single user systems.

Not in my experience.

> >I'm not saying the speedup will be extreme, but it will be there, and
> >(cumulated over loads of programs using /tmp) it will be significant
> >enough to be noticeable.
>
> If you train the user to keep away from /tmp, because it may be too
> small with tmpfs, how much programs will use it then?

This argument is circular.

If you train the user to keep away from /tmp, because it may be too slow
without tmpfs, how many programs will use it then?

> >- Any data transformation or filtering which needs to be done in
> > multiple passes over a file would use a temporary file for
> > intermediate results, which it then reads in again for the next pass.
> > Multi-pass video transcoding is an example of this, and which
> > (depending on the codecs used and the hardware on which it runs) could
> > certainly be I/O bound.
>
> I agree, but only if your tmpfs is big enough to hold the file.
> Ripping DVDs or BDs will exceed any tmpfs limit on most systems.

Not necessarily. Ffmpeg's two-pass video encoding uses a temporary file
to which it writes statistics about the video file being processed
(mainly data about how fast the image moves, etc), which it can then use
to optimize the encoder in the second pass. This statistics file
contains data such as "at time index X, Y% of the image changes" in
ASCII form, and hence need not be much larger than some tens of
megabytes for a full-sized DVD.

The _output_ of the encoder (the encoded MPEG file) in the first pass is
irrelevant, and can in fact be thrown away. That's the whole point of
doing two-pass encoding.

So that leaves you with reading a whole lot of data from disk, and
writing a temporary file to RAM.

> >The point is that neither you nor I can reasonably be expected to list
> >all possible uses of /tmp; and that RAM is faster than disk, so that
> >when you access a tmpfs you're going to be somewhat faster than when you
> >access a disk-backed filesystem, at least until you start swapping (if
> >not longer).
>
> Nobody disagrees that RAM is faster than disk, but I hope you don’t
> disagree as well that most people will have more disk space than
> RAM.

That leads us to the crux of the discussion: we are both right. You are
right in that /tmp on disk lowers the chance of /tmp running out of
space, which is a real problem. I, and others arguing in favour of
tmpfs, are right that placing /tmp on tmpfs will speed up things and (if
not using any swap space) will reduce usage of an SSD, both of which
are real improvements.

The question is: what matters most? To me, the performance improvements
of tmpfs are significant enough to warrant making it the default.
Clearly, you disagree.

> >Now whether that advantage outweighs the disadvantages you've outlined
> >is something we can talk about. However, whether RAM is faster than disk
>
> Fine let’s talk. Why can’t we find a compromise? Additional to our
> disk /tmp we create a /ramtmp (so the name suggests that this tmp is
> a ramdisk) with tmpfs. This should be doable in time for Wheezy. The
> release notes should mention it. And those who wish can patch their
> programs to use the ramdisk if they think their program uses only
> small temporary files. In this way, we get some data and experience.
> And we have both worlds. /tmp on disk for even large temporary files
> and /ramtmp as fast ramdisk.

While I think a compromise would be wonderful, I don't think this is it.
Additionally, I don't think this is technically and aesthetically a very
good solution.

> >No, that's not true. The real danger in filling up /tmp is not that
> >other processes can't write temporary files anymore (causing a minority
> >of processes to hang or die; those who just happen to need temporary
> >storage at that point in time), but that no process can write any file
> >anymore (causing a significant majority of processes to hang or die).
>
> But this will only happen if your /tmp is not a separate partition.

Yes; but if you're going to make /tmp be a separate partition, then your
argument that there's more space on disk doesn't really hold anymore,
either, since now /tmp is much much smaller than your disk (I've never
seen a system with a separate partition for /tmp use more than a few
gigs for that partition).

> And it can happen with /var as well. I’ve seen programs logging so
> fast that /var had no space left breaking the logging and the
> database.

Absolutely, I'm not contesting that (in fact, I've recently had a very
similar situation at a customer). But this discussion is not about /var,
it's about /tmp.

--
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120621070630.GK21287@grep.be">http://lists.debian.org/20120621070630.GK21287@grep.be

Stephan Seitz 06-21-2012 01:46 PM

Summary: Moving /tmp to tmpfs makes it useless
 
On Thu, Jun 21, 2012 at 09:06:30AM +0200, Wouter Verhelst wrote:

Maybe, but we are talking about defaults. Please correct me, but I
think that most Debian systems are in some way single user systems.

Not in my experience.


So most of your Debian systems have several users working at the same
time on the same system? Okay, then you have a different user base.



If you train the user to keep away from /tmp, because it may be too
small with tmpfs, how much programs will use it then?

This argument is circular.


I beg to disagree.


If you train the user to keep away from /tmp, because it may be too slow
without tmpfs, how many programs will use it then?


But the disk will not be so slow to break applications like they would if
you run out of disk space. So we weight a time advantage against an
enough space advantage. And since the user don’t know about tmpfs they
won’t think that there /tmp is slow.


Besides disk /tmp is still faster than NFS /home. ;-)


I agree, but only if your tmpfs is big enough to hold the file.
Ripping DVDs or BDs will exceed any tmpfs limit on most systems.

Not necessarily. Ffmpeg's two-pass video encoding uses a temporary file
to which it writes statistics about the video file being processed


Ah, thank you for the explanation, but


to optimize the encoder in the second pass. This statistics file
contains data such as "at time index X, Y% of the image changes" in
ASCII form, and hence need not be much larger than some tens of
megabytes for a full-sized DVD.


if the file is only about some tens of megabytes, do you really gain so
much speed?



Nobody disagrees that RAM is faster than disk, but I hope you don’t
disagree as well that most people will have more disk space than
RAM.

That leads us to the crux of the discussion: we are both right. You are
right in that /tmp on disk lowers the chance of /tmp running out of
space, which is a real problem. I, and others arguing in favour of
tmpfs, are right that placing /tmp on tmpfs will speed up things and (if
not using any swap space) will reduce usage of an SSD, both of which
are real improvements.


I still think that the SSD problem is not a valid concern as long as we
don’t have solutions for /var and log daemons. There is more traffic in
/var than in /tmp.

If we really want to optimize for SSDs we should do something like:
- /tmp and /var/tmp on tmpfs
- no local logging or the possibility to enter a log server
- patching any applications like iceweasel to store their cache on
a tmpfs

- running trim daemon for discard (maybe)

This could be part of an enhanced laptop mode because it would safe
battery time for normal disks as well.



The question is: what matters most? To me, the performance improvements
of tmpfs are significant enough to warrant making it the default.
Clearly, you disagree.


Yes, because the performance improvements will only get you speed but
risk breakage of the application because /tmp is now smaller than it was
before tmpfs times. On the other hand disk /tmp will make some things
slower, but in the end it is safer. And I think users understand the
problem of a full disk much easier than how to expand a full ram disk.



Fine let’s talk. Why can’t we find a compromise? Additional to our
disk /tmp we create a /ramtmp (so the name suggests that this tmp is
a ramdisk) with tmpfs. This should be doable in time for Wheezy. The
release notes should mention it. And those who wish can patch their
programs to use the ramdisk if they think their program uses only
small temporary files. In this way, we get some data and experience.
And we have both worlds. /tmp on disk for even large temporary files
and /ramtmp as fast ramdisk.

While I think a compromise would be wonderful, I don't think this is it.
Additionally, I don't think this is technically and aesthetically a very
good solution.


Well, why? Other suggestions were to write daemons monitoring tmpfs and
increasing it if necessary together with swap space. Or writing a daemon
to blend tmpfs and disk together.
Besides that these daemons must be written and tested first it sounds
much to complicated to me.
With my suggestion you have both worlds available without manual
configuration by the user, and we can encourage our users to test it and
report to us if they find it useful or not. I see the advantages of tmpfs
but I wouldn’t replace my disk tmp with it, only have it as alternative.



But this will only happen if your /tmp is not a separate partition.

Yes; but if you're going to make /tmp be a separate partition, then your
argument that there's more space on disk doesn't really hold anymore,
either, since now /tmp is much much smaller than your disk (I've never
seen a system with a separate partition for /tmp use more than a few
gigs for that partition).


While you are right that /tmp on a separate partition is smaller than the
disk, you are wrong (at least in my cases) that tmpfs would now be bigger
than disk /tmp. My smallest disk is 250GB, my biggest 2TB, still all my
systems have only 4GB RAM. At my company there are some systems with 8GB
RAM, some even with 16GB (eclipse user), but the disks are at least
500GB.


/tmp has always been about 10-20GB big, because I expect users to store
big temporary images in /tmp. I won’t get this with tmpfs and I would
have to train the users to change their behaviour.



And it can happen with /var as well. I’ve seen programs logging so
fast that /var had no space left breaking the logging and the
database.

Absolutely, I'm not contesting that (in fact, I've recently had a very
similar situation at a customer). But this discussion is not about /var,
it's about /tmp.


Yes, it is, but if tmpfs is seen as an advantage because /tmp can’t block
the system anymore and prevent DOS attacks (among others), this doesn’t
sound so good anymore if you can as easily block the system by filling
/var/tmp or /var/log.


Stephan

--
| Stephan Seitz E-Mail: stse@fsing.rootsland.net |
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |

Tomasz Rybak 06-21-2012 02:05 PM

Summary: Moving /tmp to tmpfs makes it useless
 
Dnia 2012-06-21, czw o godzinie 09:06 +0200, Wouter Verhelst pisze:
[ cut ]
> Yes; but if you're going to make /tmp be a separate partition, then your
> argument that there's more space on disk doesn't really hold anymore,
> either, since now /tmp is much much smaller than your disk (I've never
> seen a system with a separate partition for /tmp use more than a few
> gigs for that partition).
>

All my computers have 1 user. On all my non-laptop machines
I have /tmp as separate partitions which sizes are varying
between 9GB and almost 20GB. I try to at least have enough
space on /tmp to be able to store both DVD image (iso9660)
and data for it.
I have discovered moving /tmp to tmpfs on my laptop.
I was making backup and got message that I run out of space
in /tmp. After discovering the reason I have just changed
my scripts to use /var/tmp instead of /tmp.

From this discussion it looks like there is no easy compromise.
As a (power) user I would prefer to have debconf question
during upgrade, e.g. "We are proposing for you to move
/tmp to use tmpfs. On your system you have X MB of RAM
which means that your tmpfs will have Y MB. You can also
leave /tmp as is to use hard disk which will be slower,
can cause SSD wear-out, but will allow for storing large files."

Best regards.
--
Tomasz Rybak <tomasz.rybak@post.pl> GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak

Russ Allbery 06-21-2012 03:39 PM

Summary: Moving /tmp to tmpfs makes it useless
 
Stephan Seitz <stse+debian@fsing.rootsland.net> writes:
> On Thu, Jun 21, 2012 at 09:06:30AM +0200, Wouter Verhelst wrote:

>>> Maybe, but we are talking about defaults. Please correct me, but I
>>> think that most Debian systems are in some way single user systems.

>> Not in my experience.

> So most of your Debian systems have several users working at the same
> time on the same system? Okay, then you have a different user base.

Far and away the majority of my Debian systems are zero-user systems.
They're servers.

Of the remaining ones, I think I have about 10x as many multiuser systems
as single-user systems.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87k3z0ad31.fsf@windlord.stanford.edu">http://lists.debian.org/87k3z0ad31.fsf@windlord.stanford.edu

David Weinehall 06-21-2012 08:20 PM

Summary: Moving /tmp to tmpfs makes it useless
 
On Wed, Jun 20, 2012 at 09:08:51PM +0200, Carlos Alberto Lopez Perez wrote:
> On 20/06/12 15:18, Stephan Seitz wrote:
> >>
> >
> > Fine let’s talk. Why can’t we find a compromise? Additional to our disk
> > /tmp we create a /ramtmp (so the name suggests that this tmp is a
> > ramdisk) with tmpfs. This should be doable in time for Wheezy. The
> > release notes should mention it. And those who wish can patch their
> > programs to use the ramdisk if they think their program uses only small
> > temporary files. In this way, we get some data and experience. And we
> > have both worlds. /tmp on disk for even large temporary files and
> > /ramtmp as fast ramdisk.
>
> I really think this is the way to go :)

As do I. Not because I think diskbased /tmp is something worth keeping
(on the contrary I consider tmpfs-based /tmp to be superior), but
because I think it'd be impossible to convince some people that /tmp
isn't a random dumping ground for anything and everything.

I think that having a tmpfs file system, and mandating that long-term
all software packaged for Debian -- where possible -- except in special
cases, and leave /tmp primarily as a dumping ground for the user
(If it wasn't for NFS *shudder* I'd advocate /home/<user>/tmp for this
purpose instead).


Regards: DAvid
--
/) David Weinehall <tao@debian.org> /) Rime on my window (
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ // Diamond-white roses of fire //
) http://www.acc.umu.se/~tao/ (/ Beautiful hoar-frost (/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120621202003.GC25829@suiko.acc.umu.se">http://lists.debian.org/20120621202003.GC25829@suiko.acc.umu.se


All times are GMT. The time now is 10:54 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.