FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 09-06-2012, 01:27 PM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> I do get it. I CLEARED #1 and #2, there is no usage of #3 and #4 is not
> large enough here to matter. So, it is left with #5.
>
> See the point? The test was a NORMAL emerge with portages work
> directory on tmpfs and a NORMAL emerge with portages work directory on
> disk and compare the results. The test resulted in little if any
> difference.
>
> If I ran the test and did not clear the cache, then I would expect
> skewed results because after the first emerge, some files would be
> cached in ram and the drive would not be used. If you clear the cache,
> then it has to take the same steps regardless of whether it was run
> first, second or third time.

What you want to measure is the difference of times required by emerge
whether you use a real disk or tmpfs as backend.

What you would expect is a difference because a disk is much slower than
RAM.

What you see is no difference. You won't conclude that disk is as fast
as RAM, right? Can you explain why you don't see much difference? No.

Here is the explanation: if you have enough RAM, the emerge rapidity
will NOT rely on the disk rapidity whatever storage backend you use. It
will only rely on the RAM rapidity because of the kernel cache.

Now, pretending that whatever backend you use (real disk or tmpfs) never
changes the emerge time is WRONG because of the persistence strategy
used by the kernel for the kernel cache.

When having lot of RAM like you have, the persistence strategy of the
kernel cache is NEVER raised in the process.

This is exactly what your tests demonstrate demonstrate: if you have
enough RAM, the persistence strategy of kernel cache is not raised, so
everything happens in RAM, so the emerge times do not differ.

--
Nicolas Sebrecht
 
Old 09-06-2012, 01:46 PM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> Not quite. The theory is that if you put portages work directory on
> tmpfs, then all the writes and such are done in ram which is faster.

No! This is too much simplistic view to explain what you see.

In practice, _all_ the writes always happen in RAM whatever backend
storage you use.

The difference you could see is if there is not enough RAM for the
kernel cache, it will have to wait for the backend storage.

--
Nicolas Sebrecht
 
Old 09-06-2012, 02:07 PM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 07:48:59 -0500, Dale wrote:
>
>> I don't think that is correct. I am clearing the files in ram. That's
>> the point of drop_caches is to clear the kernels cache files. See post
>> to Nicolas Sebrecht a bit ago.
> Take a step back Dale and read the posts again. This is not about the
> state of the cache at the start of the emerge but during it. You may
> clear the cache before starting, but that doesn't stop is filling up
> again as soon as the emerge reaches src_unpack().
>
> This has nothing to do with caching the data from the previous emerge
> run, it is all from the currently running emerge. You may think you are
> unpacking the tarball to disk and then loading those files into the
> compiler, but you are only using the copies that are cached when you
> unpack.
>
>


Then take a look at it this way. If I emerge seamonkey with portage's
work directory on disk and it takes 12 minutes, the first time. Then I
clear the caches and emerge seamonkey again while portage's work
directory is on tmpfs and it is 12 minutes. Then repeat that process a
few times more. If the outcome of all those emerges is 12 minutes,
regardless of the order, then putting portages work directory on tmpfs
makes no difference at all in that case. The emerge times are exactly
the same regardless of emerge using cache or not or portage's work
directory being on tmpfs or not. I don't care if emerge uses cache
DURING the emerge process because it is always enabled in both tests.
The point is whether portage's work directory is on tmpfs or not makes
emerges faster.

The thing about what you are saying is that I ran those tests with the
files in memory. What I am saying is this, that is not the case. I am
clearing that memory with the drop_cache command between each test. You
claim that cache is affecting the timing but I am clearing the very same
cache the same as a reboot would. The emerge times whether portage's
work directory is on tmpfs or not didn't change enough to make a
difference. That is what I am saying the tests resulted in. It was not
what I expected but it is what I got. It is also what others got as well.

I provided a link to the information that should be as clear as it
gets. Can you provide a link that shows that the command does not clear
the kernel cache? I'm going by what I linked to on kernel.org. Since
they are the ones that make the kernels, I think they should know what
it is and what it does.

Here is some more links with the same info really:

http://linux-mm.org/Drop_Caches

http://www.linuxinsight.com/proc_sys_vm_drop_caches.html

http://bjdean.id.au/wiki/LinuxMemoryManagement

Those are all the first links in a google search for "drop_caches
kernel". See if you can find anything that says otherwise.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 02:20 PM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 06/09/12, Dale wrote:
>
>> Not quite. The theory is that if you put portages work directory on
>> tmpfs, then all the writes and such are done in ram which is faster.
> No! This is too much simplistic view to explain what you see.
>
> In practice, _all_ the writes always happen in RAM whatever backend
> storage you use.
>
> The difference you could see is if there is not enough RAM for the
> kernel cache, it will have to wait for the backend storage.
>


OK. Step by step here so hopefully you and Neil can follow.

Freshly booted system.
Clear caches just to be sure

emerge foo with portages work directory on tmpfs
clear caches again
emerge foo with portages work directory on disk
clear caches again.
emerge foo with portages work directory on tmpfs
clear caches again
emerge foo with portages work directory on disk

You repeat this enough times and you see that it doesn't matter if
portage's work directory is on disk or on tmpfs. As I said before, when
I have portage's work directory on disk, I see the drive light blinking
like crazy so it is doing something, reading or writing. When portage's
work directory is on tmpfs, it only blinks when I first start the
process which should be unpacking the tarball and then at the end when
it is installing the package. In between that, it is just the normal
stuff of my wallpaper changing or it checking my emails. So, it may
store something in ram as it does in both cases but it is also storing
things on the drive or else the light would not be blinking so much.

I'm not real big on rebooting but you and Neil are about to make me test
this and reboot between each and every test. If nothing else, just to
show that drop_caches does the same as rebooting like kernel.org says,
except for the programs are still actually running.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 02:26 PM
Michael Mol
 
Default aligning SSD partitions

On Thu, Sep 6, 2012 at 10:07 AM, Dale <rdalek1967@gmail.com> wrote:
> Neil Bothwick wrote:
>> On Thu, 06 Sep 2012 07:48:59 -0500, Dale wrote:
>>
>>> I don't think that is correct. I am clearing the files in ram. That's
>>> the point of drop_caches is to clear the kernels cache files. See post
>>> to Nicolas Sebrecht a bit ago.
>> Take a step back Dale and read the posts again. This is not about the
>> state of the cache at the start of the emerge but during it. You may
>> clear the cache before starting, but that doesn't stop is filling up
>> again as soon as the emerge reaches src_unpack().
>>
>> This has nothing to do with caching the data from the previous emerge
>> run, it is all from the currently running emerge. You may think you are
>> unpacking the tarball to disk and then loading those files into the
>> compiler, but you are only using the copies that are cached when you
>> unpack.
>>
>>
>
>
> Then take a look at it this way. If I emerge seamonkey with portage's
> work directory on disk and it takes 12 minutes, the first time. Then I
> clear the caches and emerge seamonkey again while portage's work
> directory is on tmpfs and it is 12 minutes. Then repeat that process a
> few times more. If the outcome of all those emerges is 12 minutes,
> regardless of the order, then putting portages work directory on tmpfs
> makes no difference at all in that case. The emerge times are exactly
> the same regardless of emerge using cache or not or portage's work
> directory being on tmpfs or not. I don't care if emerge uses cache
> DURING the emerge process because it is always enabled in both tests.
> The point is whether portage's work directory is on tmpfs or not makes
> emerges faster.
>
> The thing about what you are saying is that I ran those tests with the
> files in memory. What I am saying is this, that is not the case. I am
> clearing that memory with the drop_cache command between each test.

Dale, here's what you're missing:

emerge first downloads the source tarball and drops it on disk. Once
the tarball has been placed on disk, the time required to read the
tarball back into memory is negligible; it's a streamed format.

The next step is what's important: the tarball gets extracted into
PORTAGE_TEMP. At that moment onward, all the files that were inside
that tarball are in your file cache until something bumps it out.

If you have enough RAM, then the file will not be bumped out as a
consequence of build-time memory usage. As a consequence, if you have
enough ram, you won't see much (if any) difference in build times if
you're comparing tmpfs to a normal filesystem...which means tmpfs (for
you) won't have any benefit beyond being self-cleaning on a reboot or
remount.

So your drop_cache has no influence over build times, since the only
cache behavior that matters is whatever happens between the time
emerge unpacks the tarball and the time emerge exits.

To see the difference, try something like "watch drop_cache" leave
that running while you let a few builds fly. You should see an
increase in build times.

--
:wq
 
Old 09-06-2012, 02:27 PM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 06/09/12, Dale wrote:
>
>> I do get it. I CLEARED #1 and #2, there is no usage of #3 and #4 is not
>> large enough here to matter. So, it is left with #5.
>>
>> See the point? The test was a NORMAL emerge with portages work
>> directory on tmpfs and a NORMAL emerge with portages work directory on
>> disk and compare the results. The test resulted in little if any
>> difference.
>>
>> If I ran the test and did not clear the cache, then I would expect
>> skewed results because after the first emerge, some files would be
>> cached in ram and the drive would not be used. If you clear the cache,
>> then it has to take the same steps regardless of whether it was run
>> first, second or third time.
> What you want to measure is the difference of times required by emerge
> whether you use a real disk or tmpfs as backend.
>
> What you would expect is a difference because a disk is much slower than
> RAM.
>
> What you see is no difference. You won't conclude that disk is as fast
> as RAM, right? Can you explain why you don't see much difference? No.
>
> Here is the explanation: if you have enough RAM, the emerge rapidity
> will NOT rely on the disk rapidity whatever storage backend you use. It
> will only rely on the RAM rapidity because of the kernel cache.
>
> Now, pretending that whatever backend you use (real disk or tmpfs) never
> changes the emerge time is WRONG because of the persistence strategy
> used by the kernel for the kernel cache.
>
> When having lot of RAM like you have, the persistence strategy of the
> kernel cache is NEVER raised in the process.
>
> This is exactly what your tests demonstrate demonstrate: if you have
> enough RAM, the persistence strategy of kernel cache is not raised, so
> everything happens in RAM, so the emerge times do not differ.
>

The end result is this, it doesn't matter if portage's work directory is
on tmpfs or not. You just concluded that yourself which is what I have
been saying. It doesn't matter WHY it doesn't matter, it just matters
that it DOESN'T matter. It takes just as long on a system with
portage's work directory on tmpfs as it does on tmpfs. Very little
difference at all. The variance I had was minimal at best. It was
basically seconds of difference not minutes.

I might add, I got the same results on my older system which has a LOT
less ram. I think it only has 2Gbs or so.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 02:33 PM
Michael Mol
 
Default aligning SSD partitions

On Thu, Sep 6, 2012 at 10:20 AM, Dale <rdalek1967@gmail.com> wrote:
> Nicolas Sebrecht wrote:
>> The 06/09/12, Dale wrote:
>>
>>> Not quite. The theory is that if you put portages work directory on
>>> tmpfs, then all the writes and such are done in ram which is faster.
>> No! This is too much simplistic view to explain what you see.
>>
>> In practice, _all_ the writes always happen in RAM whatever backend
>> storage you use.
>>
>> The difference you could see is if there is not enough RAM for the
>> kernel cache, it will have to wait for the backend storage.
>>
>
>
> OK. Step by step here so hopefully you and Neil can follow.
>
> Freshly booted system.
> Clear caches just to be sure
>
> emerge foo with portages work directory on tmpfs
> clear caches again
> emerge foo with portages work directory on disk
> clear caches again.
> emerge foo with portages work directory on tmpfs
> clear caches again
> emerge foo with portages work directory on disk
>
> You repeat this enough times and you see that it doesn't matter if
> portage's work directory is on disk or on tmpfs.

If you have enough RAM, then this is certainly true. Nobody is
disputing that. They've been trying to explain that there's a
difference when you _don't_ have that much RAM, and they've been
trying to explain the mechanism behind that.


--
:wq
 
Old 09-06-2012, 02:37 PM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 09:07:30 -0500, Dale wrote:

> I don't care if emerge uses cache
> DURING the emerge process because it is always enabled in both tests.
> The point is whether portage's work directory is on tmpfs or not makes
> emerges faster.

It does not, if you have enough RAM, precisely because of the part you
claim not to care about.

> The thing about what you are saying is that I ran those tests with the
> files in memory. What I am saying is this, that is not the case.

No, that is not what I am saying. Those files were loaded into memory
when you ran the test AFTER you cleared the previously cached files. The
number of times you run the test is irrelevant, as is whether you start
with an empty cache or not. All that matters is that the kernel caching
all the files used during the emerge makes the storage medium used
irrelevant.

Like I said, take a step back, a deep breath and a break of an hour or
two. Then read the posts again without your preconceptions of what you
think we are trying to say (which is not what we are actually saying).
Only when you have done that can this discussion proceed beyond the
current tit-for-tat exchanges of misunderstanding.


--
Neil Bothwick

Always remember you're unique, just like everyone else.
 
Old 09-06-2012, 02:51 PM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> Then take a look at it this way. If I emerge seamonkey with portage's
> work directory on disk and it takes 12 minutes, the first time. Then I
> clear the caches and emerge seamonkey again while portage's work
> directory is on tmpfs and it is 12 minutes. Then repeat that process a
> few times more. If the outcome of all those emerges is 12 minutes,
> regardless of the order, then putting portages work directory on tmpfs
> makes no difference at all in that case.

We fully agree with you, here.

> The emerge times are exactly
> the same regardless of emerge using cache or not or portage's work
> directory being on tmpfs or not. I don't care if emerge uses cache
> DURING the emerge process because it is always enabled in both tests.

But you *should* care. If you don't have enough memory, the kernel will
reclaim memory from the pagecache, so the whole process rapidity won't
only rely on RAM rapidity anymore.

> The point is whether portage's work directory is on tmpfs or not makes
> emerges faster.
>
> The thing about what you are saying is that I ran those tests with the
> files in memory. What I am saying is this, that is not the case. I am
> clearing that memory with the drop_cache command between each test. You
> claim that cache is affecting the timing but I am clearing the very same
> cache the same as a reboot would. The emerge times whether portage's

We do agree with you that you droped the cache between the tests with
almost the same effect of a reboot.

> The emerge times whether portage's
> work directory is on tmpfs or not didn't change enough to make a
> difference.

Yes, we agree. You droped the cache which is expected to get correct
tests.

What we are saying is that you droped the cache but did NOT DISABLED the
VM caches (kernel cache). You say that you don't care of that one
because it was involved in all the tests. We say that you might not care
in some contexts, not for all the contexts. You reach the context where
it does not matter much, fine.

--
Nicolas Sebrecht
 
Old 09-06-2012, 04:00 PM
Paul Hartman
 
Default aligning SSD partitions

On Thu, Sep 6, 2012 at 9:20 AM, Dale <rdalek1967@gmail.com> wrote:
> OK. Step by step here so hopefully you and Neil can follow.
>
> Freshly booted system.
> Clear caches just to be sure
>
> emerge foo with portages work directory on tmpfs
> clear caches again
> emerge foo with portages work directory on disk
> clear caches again.
> emerge foo with portages work directory on tmpfs
> clear caches again
> emerge foo with portages work directory on disk

I think, based on what the others are saying, that for you to more
accurately test it, you should not use "emerge" but rather use
"ebuild" to run (and time) the individual steps involved in emerging a
package (unpacking, preparing, compiling, installing), clearing disk
caches in-between each step. So, for example, after sources are
unpacked to tmpfs, clear caches before compilation begins -- this way
the source files have to be read from disk rather than from cache/RAM.
 

Thread Tools




All times are GMT. The time now is 03:53 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org