FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 09-06-2012, 06:57 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 05/09/12, Dale wrote:
> Michael Mol wrote:
> > On Wed, Sep 5, 2012 at 11:17 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> >> On Wed, 05 Sep 2012 07:52:45 -0500, Dale wrote:
> >>
> >>>>> I might also add, I see no speed improvements in putting portages
> >>>>> work directory on tmpfs. I have tested this a few times and the
> >>>>> difference in compile times is just not there.
> >>>> Probably because with 16GB everything stays cached anyway.
> >>> I cleared the cache between the compiles. This is the command I use:
> >>>
> >>> echo 3 > /proc/sys/vm/drop_caches
> >> But you are still using the RAM as disk cache during the emerge, the data
> >> doesn't stay around long enough to need to get written to disk with so
> >> much RAM for cache.
> > Indeed. Try setting the mount to write-through to see the difference.
> >
> >
>
> When I run that command, it clears all the cache. It is the same as if
> I rebooted. Certainly you are not thinking that cache survives a reboot?

You missed the point. One of the first thing emerge will do is to
uncompress the package. At this time, all the files are cached in RAM.
Hence, everything needed for the build/compilation will come from the
cache like it would do with tmpfs.

--
Nicolas Sebrecht
 
Old 09-06-2012, 09:13 AM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Wed, 05 Sep 2012 12:54:51 -0500, Dale wrote:
>
>>>>>>> I might also add, I see no speed improvements in putting portages
>>>>>>> work directory on tmpfs. I have tested this a few times and the
>>>>>>> difference in compile times is just not there.
>>>>>> Probably because with 16GB everything stays cached anyway.
>>>>> I cleared the cache between the compiles. This is the command I
>>>>> use:
>>>>>
>>>>> echo 3 > /proc/sys/vm/drop_caches
>>>> But you are still using the RAM as disk cache during the emerge, the
>>>> data doesn't stay around long enough to need to get written to disk
>>>> with so much RAM for cache.
>>> Indeed. Try setting the mount to write-through to see the difference.
>> When I run that command, it clears all the cache. It is the same as if
>> I rebooted. Certainly you are not thinking that cache survives a
>> reboot?
> You clear the cache between the two emerge runs, not during them.

If I recall correctly the process, I cleared the cache, ran emerge with
the portage work directory on disk. Then cleared cache again and run on
tmpfs. If you think that the cache would make any difference for the
second run then it would be faster just because of that *while using
tmpfs* since that was the second run. Thing is, it wasn't faster. In
some tests it was actually slower.

I'm trying to catch on to why you think that clearing the cache means it
is still there. That's the whole point of clearing cache is that it is
gone. When I was checking on drop_caches, my understanding was that
clearing that was the same as a reboot. This is from kernel.org

drop_caches

Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

According to that, the 3 option clears all cache. One site I found that
on even recommended running sync first just in case something in ram was
not yet written to disk.

>
>> If you are talking about ram on the drive itself, well, when it is on
>> tmpfs, it is not on the drive to be cached. That's the whole point of
>> tmpfs is to get the slow drive out of the way. By the way, there are
>> others that ran tests with the same results. It just doesn't speed up
>> anything since drives are so much faster nowadays.
> Drives are still orders of magnitude slower than RAM, that's why using
> swap is so slow. What appears to be happening here is that because
> files are written and then read again in short succession, they are still
> in the kernel's disk cache, so the speed of the disk is irrelevant. Bear
> in mind that tmpfs is basically a cached disk without the disk, so you
> are effectively comparing the same thing twice.
>
>

I agree that they are slower but for whatever reason, it doesn't seem to
matter as much as one thought. I was expecting a huge difference in
this with using tmpfs being much faster. Thing is, that is NOT what I
got. Theory meets reality and it was not what I expected or what others
expected either. Putting portages work directory on tmpfs doesn't make
much if any difference in emerge times.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 09:15 AM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 05/09/12, Dale wrote:
>> Michael Mol wrote:
>>> On Wed, Sep 5, 2012 at 11:17 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
>>>> On Wed, 05 Sep 2012 07:52:45 -0500, Dale wrote:
>>>>
>>>>>>> I might also add, I see no speed improvements in putting portages
>>>>>>> work directory on tmpfs. I have tested this a few times and the
>>>>>>> difference in compile times is just not there.
>>>>>> Probably because with 16GB everything stays cached anyway.
>>>>> I cleared the cache between the compiles. This is the command I use:
>>>>>
>>>>> echo 3 > /proc/sys/vm/drop_caches
>>>> But you are still using the RAM as disk cache during the emerge, the data
>>>> doesn't stay around long enough to need to get written to disk with so
>>>> much RAM for cache.
>>> Indeed. Try setting the mount to write-through to see the difference.
>>>
>>>
>> When I run that command, it clears all the cache. It is the same as if
>> I rebooted. Certainly you are not thinking that cache survives a reboot?
> You missed the point. One of the first thing emerge will do is to
> uncompress the package. At this time, all the files are cached in RAM.
> Hence, everything needed for the build/compilation will come from the
> cache like it would do with tmpfs.
>

You miss this point not me. I *cleared* that cache. From kernel.org:

drop_caches

Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

I can confirm this is done with free, top or htop. See my reply to Neil for more on this.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 09:47 AM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 04:15:23 -0500, Dale wrote:

> > You missed the point. One of the first thing emerge will do is to
> > uncompress the package. At this time, all the files are cached in RAM.
> > Hence, everything needed for the build/compilation will come from the
> > cache like it would do with tmpfs.
> >
>
> You miss this point not me. I *cleared* that cache. From kernel.org:

Sorry Dale, but you are missing the point. You cleared the cache before
running emerge, then ran emerge. The first thing emerge did was unpack
the tarball and populate the disk cache. All clearing the disk cache did
was make sure there was plenty of space to cache the new data, thus
speeding up the process.


--
Neil Bothwick

"A snooze button is a poor substitute for no alarm clock at all."
 
Old 09-06-2012, 10:03 AM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 04:15:23 -0500, Dale wrote:
>
>>> You missed the point. One of the first thing emerge will do is to
>>> uncompress the package. At this time, all the files are cached in RAM.
>>> Hence, everything needed for the build/compilation will come from the
>>> cache like it would do with tmpfs.
>>>
>> You miss this point not me. I *cleared* that cache. From kernel.org:
> Sorry Dale, but you are missing the point. You cleared the cache before
> running emerge, then ran emerge. The first thing emerge did was unpack
> the tarball and populate the disk cache. All clearing the disk cache did
> was make sure there was plenty of space to cache the new data, thus
> speeding up the process.
>
>

Then explain to me why it was at times slower while on tmpfs? Trust me,
I ran this test many times and in different orders and it did NOT make
much if any difference.

I might add, the cache on the drive I was using is nowhere near large
enough to cache the tarball for the package. Heck, the cache on my
current system drive is only 8Mbs according to hdparm. That is not much
since I tested using much larger packages. You can't cache files larger
than the cache.

Do I need to run a test, reboot, run the test again to show this is not
making much if any difference? I mean, really? o_O

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 10:11 AM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 04:15:23 -0500, Dale wrote:
>
>>> You missed the point. One of the first thing emerge will do is to
>>> uncompress the package. At this time, all the files are cached in RAM.
>>> Hence, everything needed for the build/compilation will come from the
>>> cache like it would do with tmpfs.
>>>
>> You miss this point not me. I *cleared* that cache. From kernel.org:
> Sorry Dale, but you are missing the point. You cleared the cache before
> running emerge, then ran emerge. The first thing emerge did was unpack
> the tarball and populate the disk cache. All clearing the disk cache did
> was make sure there was plenty of space to cache the new data, thus
> speeding up the process.
>
>

One other thing, I am not just clearing the *disk* cache. I am clearing
all the *SYSTEM* cache. I can have all 16Gbs of memory in use, either
by programs or as cache, then run that command and it is then only using
what is in use by programs. It clears everything else. That includes
any cache that was stored there, disk or otherwise.

You need to run free, run the command to clear and then run free again
so you can see for yourself. If it was just me, I could think I am
wrong but this was tested by others too with the same results.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 10:18 AM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 05:03:55 -0500, Dale wrote:

> >> You miss this point not me. I *cleared* that cache. From
> >> kernel.org:
> > Sorry Dale, but you are missing the point. You cleared the cache
> > before running emerge, then ran emerge. The first thing emerge did
> > was unpack the tarball and populate the disk cache. All clearing the
> > disk cache did was make sure there was plenty of space to cache the
> > new data, thus speeding up the process.

> Then explain to me why it was at times slower while on tmpfs? Trust me,
> I ran this test many times and in different orders and it did NOT make
> much if any difference.

So it was slower at times, but not by much? That's just general variances
caused by multi-tasking, wind direction etc.

> I might add, the cache on the drive I was using is nowhere near large
> enough to cache the tarball for the package. Heck, the cache on my
> current system drive is only 8Mbs according to hdparm.

We're not talking about drive caches, the kernel caches filesystem access
long before it gets anywhere the drive. So all the real work is done in
RAM if you have enough, whether you are using a hard drive filesystem or
tmpfs. All your test demonstrates is that if you have enough RAM, it
doesn't make much difference where you put PORTAGE_TMPDIR.


--
Neil Bothwick

Evolution stops when stupidity is no longer fatal!
 
Old 09-06-2012, 10:20 AM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 05:11:01 -0500, Dale wrote:

> You need to run free, run the command to clear and then run free again
> so you can see for yourself. If it was just me, I could think I am
> wrong but this was tested by others too with the same results.

I'm not saying your test results are wrong, I'm explaining why I think
they are what they are. Have you tried running free *during* the emerge?
I expect you'll find plenty of cache in use then.


--
Neil Bothwick

We are Pentium of Borg. You will be approximated. Resistance may or may
not be futile, except on every other Tuesday when it is a definite maybe.
 
Old 09-06-2012, 10:41 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> Then explain to me why it was at times slower while on tmpfs? Trust me,
> I ran this test many times and in different orders and it did NOT make
> much if any difference.

As explained, this is expected if you have enough RAM.

I didn't check but I would expect that files stored in tmpfs are NOT
duplicated in the the kernel cache in order to save RAM. So, the
different times could come from the fact that the kernel will first look
up in the kernel cache and /then/ look up in the tmpfs.

In the scenario without tmpfs and lot of RAM, every unpacked file is
stored in the _kernel cache_ with really fast access much before hitting
the disk or even the disk cache (RAM speed and very few processor
calculation required). While retrieving, the file is found on first look
up from the kernel cache.

In the other scenario with tmpfs and lot of RAM, every unpacked file is
stored in the tmpfs allowing very fast access (due to RAM speed) but
with the price of a first negative result from the kernel cache (and
perhaps additional time needed by the kernel for accessing the file
through the driver of the tmpfs filesystem).

Using tmpfs will still be better as it prevents from writes to the disk
in the spare times, avoiding unnecessary mecanic movements and saving
disk life time.

> I might add, the cache on the drive I was using is nowhere near large
> enough to cache the tarball for the package. Heck, the cache on my
> current system drive is only 8Mbs according to hdparm. That is not much
> since I tested using much larger packages. You can't cache files larger
> than the cache.

The disk cache is out of the scope.

> Do I need to run a test, reboot, run the test again to show this is not
> making much if any difference? I mean, really? o_O

It won't make any difference from the drop cache configuration but it is
still not the point!

--
Nicolas Sebrecht
 
Old 09-06-2012, 10:44 AM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 05:11:01 -0500, Dale wrote:
>
>> You need to run free, run the command to clear and then run free again
>> so you can see for yourself. If it was just me, I could think I am
>> wrong but this was tested by others too with the same results.
> I'm not saying your test results are wrong, I'm explaining why I think
> they are what they are. Have you tried running free *during* the emerge?
> I expect you'll find plenty of cache in use then.
>
>

The point isn't about using cache DURING the emerge. The point was
whether having portages work directory on tmpfs resulted in speed
increases. If you have portages work directory on tmpfs, of course it
uses ram. That's what tmpfs is. It's taking what might normally be put
on the disk and putting it in ram because ram is faster. The point is,
cache or not, having portages work directory on tmpfs doesn't result in
speed improvements as one would expect. Actual tests gave unexpected
results. Tests show that putting portages work directory on tmpfs did
not result in speed increases for emerging packages.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 

Thread Tools




All times are GMT. The time now is 11:31 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org