FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 09-06-2012, 04:27 PM
Dale
 
Default aligning SSD partitions

Michael Mol wrote:
> On Thu, Sep 6, 2012 at 10:07 AM, Dale <rdalek1967@gmail.com> wrote:
>> Neil Bothwick wrote:
>>> On Thu, 06 Sep 2012 07:48:59 -0500, Dale wrote:
>>>
>>>> I don't think that is correct. I am clearing the files in ram. That's
>>>> the point of drop_caches is to clear the kernels cache files. See post
>>>> to Nicolas Sebrecht a bit ago.
>>> Take a step back Dale and read the posts again. This is not about the
>>> state of the cache at the start of the emerge but during it. You may
>>> clear the cache before starting, but that doesn't stop is filling up
>>> again as soon as the emerge reaches src_unpack().
>>>
>>> This has nothing to do with caching the data from the previous emerge
>>> run, it is all from the currently running emerge. You may think you are
>>> unpacking the tarball to disk and then loading those files into the
>>> compiler, but you are only using the copies that are cached when you
>>> unpack.
>>>
>>>
>>
>> Then take a look at it this way. If I emerge seamonkey with portage's
>> work directory on disk and it takes 12 minutes, the first time. Then I
>> clear the caches and emerge seamonkey again while portage's work
>> directory is on tmpfs and it is 12 minutes. Then repeat that process a
>> few times more. If the outcome of all those emerges is 12 minutes,
>> regardless of the order, then putting portages work directory on tmpfs
>> makes no difference at all in that case. The emerge times are exactly
>> the same regardless of emerge using cache or not or portage's work
>> directory being on tmpfs or not. I don't care if emerge uses cache
>> DURING the emerge process because it is always enabled in both tests.
>> The point is whether portage's work directory is on tmpfs or not makes
>> emerges faster.
>>
>> The thing about what you are saying is that I ran those tests with the
>> files in memory. What I am saying is this, that is not the case. I am
>> clearing that memory with the drop_cache command between each test.
> Dale, here's what you're missing:
>
> emerge first downloads the source tarball and drops it on disk. Once
> the tarball has been placed on disk, the time required to read the
> tarball back into memory is negligible; it's a streamed format.
>
> The next step is what's important: the tarball gets extracted into
> PORTAGE_TEMP. At that moment onward, all the files that were inside
> that tarball are in your file cache until something bumps it out.
>
> If you have enough RAM, then the file will not be bumped out as a
> consequence of build-time memory usage. As a consequence, if you have
> enough ram, you won't see much (if any) difference in build times if
> you're comparing tmpfs to a normal filesystem...which means tmpfs (for
> you) won't have any benefit beyond being self-cleaning on a reboot or
> remount.
>
> So your drop_cache has no influence over build times, since the only
> cache behavior that matters is whatever happens between the time
> emerge unpacks the tarball and the time emerge exits.
>
> To see the difference, try something like "watch drop_cache" leave
> that running while you let a few builds fly. You should see an
> increase in build times.
>

But this is what you guys are missing too. If you want to use tmpfs,
you have to have enough ram to begin with. Whether you use tmpfs or
not, you have to have enough ram to do the compile otherwise you start
using swap or it just crashes. Having ram is a prerequisite to using
tmpfs. You can't set tmpfs to 8Gbs on a machine that doesn't have 8Gbs
available and it work. I don't count swap because when you start using
swap, it all goes out the window at that point.

There is another flaw in your assumption above. I already had the
tarballs downloaded BEFORE even the first emerge. I may not be the
sharpest tool in the shed but I do know to download first when trying to
measure a emerge time. I can measure my DSL speed with other tools. lol

What the people wanted to test is if putting portages work directory on
tmpfs would make emerge times faster. It doesn't. The posts people
make admit to that fact now but want to argue the reason. I don't care
about the reason. I just know that it doesn't matter. Putting
portage's work directory on tmpfs does NOT make it faster. For the
purpose of this thread, it would be a good idea to save wear and tear on
the SSD but one should not expect compile times to improve as one would
expect.

I might also add, I didn't always have 16Gbs on this rig. I started
with 4Gbs. Then I went to 8 and later on went to 16Gbs.

Do we all admit that having portage on tmpfs does not make emerge times
faster yet?

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 04:32 PM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 09:07:30 -0500, Dale wrote:
>
>> I don't care if emerge uses cache
>> DURING the emerge process because it is always enabled in both tests.
>> The point is whether portage's work directory is on tmpfs or not makes
>> emerges faster.
> It does not, if you have enough RAM, precisely because of the part you
> claim not to care about.
>
>> The thing about what you are saying is that I ran those tests with the
>> files in memory. What I am saying is this, that is not the case.
> No, that is not what I am saying. Those files were loaded into memory
> when you ran the test AFTER you cleared the previously cached files. The
> number of times you run the test is irrelevant, as is whether you start
> with an empty cache or not. All that matters is that the kernel caching
> all the files used during the emerge makes the storage medium used
> irrelevant.
>
> Like I said, take a step back, a deep breath and a break of an hour or
> two. Then read the posts again without your preconceptions of what you
> think we are trying to say (which is not what we are actually saying).
> Only when you have done that can this discussion proceed beyond the
> current tit-for-tat exchanges of misunderstanding.
>
>


But to use that or tmpfs, you first have to have the ram. The exact
same rig reports that putting portages work directory on tmpfs does NOT
result in faster emerge times. Period. I DO NOT care why that is, I
just know from testing that it does NOT make emerge work any faster.
The only reason to use tmpfs for portage's work directory is to save
wear and tear on a drive. There is no difference in emerge times
otherwise. Others ran their own tests and got the same results.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 04:39 PM
Dale
 
Default aligning SSD partitions

Michael Mol wrote:
> On Thu, Sep 6, 2012 at 10:20 AM, Dale <rdalek1967@gmail.com> wrote:
>> Nicolas Sebrecht wrote:
>>> The 06/09/12, Dale wrote:
>>>
>>>> Not quite. The theory is that if you put portages work directory on
>>>> tmpfs, then all the writes and such are done in ram which is faster.
>>> No! This is too much simplistic view to explain what you see.
>>>
>>> In practice, _all_ the writes always happen in RAM whatever backend
>>> storage you use.
>>>
>>> The difference you could see is if there is not enough RAM for the
>>> kernel cache, it will have to wait for the backend storage.
>>>
>>
>> OK. Step by step here so hopefully you and Neil can follow.
>>
>> Freshly booted system.
>> Clear caches just to be sure
>>
>> emerge foo with portages work directory on tmpfs
>> clear caches again
>> emerge foo with portages work directory on disk
>> clear caches again.
>> emerge foo with portages work directory on tmpfs
>> clear caches again
>> emerge foo with portages work directory on disk
>>
>> You repeat this enough times and you see that it doesn't matter if
>> portage's work directory is on disk or on tmpfs.
> If you have enough RAM, then this is certainly true. Nobody is
> disputing that. They've been trying to explain that there's a
> difference when you _don't_ have that much RAM, and they've been
> trying to explain the mechanism behind that.
>
>

But, if you don't have enough ram to compile a package, then you can't
use tmpfs anyway. So, that point is not really a point. If you try to
compile OOo on a machine with 512M of ram, you can't use tmpfs because
you don't have enough ram to even consider it. The amount of ram wasn't
what I was testing, I was testing whether using tmpfs makes it faster
regardless of the amount of ram. It doesn't. Once everything related
to that specific emerge process is loaded, tmpfs doesn't matter. That
is what I been saying this whole time.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 04:44 PM
Dale
 
Default aligning SSD partitions

Paul Hartman wrote:
> On Thu, Sep 6, 2012 at 9:20 AM, Dale <rdalek1967@gmail.com> wrote:
>> OK. Step by step here so hopefully you and Neil can follow.
>>
>> Freshly booted system.
>> Clear caches just to be sure
>>
>> emerge foo with portages work directory on tmpfs
>> clear caches again
>> emerge foo with portages work directory on disk
>> clear caches again.
>> emerge foo with portages work directory on tmpfs
>> clear caches again
>> emerge foo with portages work directory on disk
> I think, based on what the others are saying, that for you to more
> accurately test it, you should not use "emerge" but rather use
> "ebuild" to run (and time) the individual steps involved in emerging a
> package (unpacking, preparing, compiling, installing), clearing disk
> caches in-between each step. So, for example, after sources are
> unpacked to tmpfs, clear caches before compilation begins -- this way
> the source files have to be read from disk rather than from cache/RAM.
>
>

I didn't want to do it that way because how many people actually update
their system that way? I wanted to test doing the same way any other
person would normally do a emerge. I suspect that 99% of users just
type emerge foo and let emerge do it.

I kind of get what they are saying but at the same time using tmpfs
doesn't matter. Once the tarball is read off the drive, it doesn't
matter whether portage is run on a tmpfs or not. The only way it would
is if you ran out of ram and it started using swap. That I disabled
because we all know that when you use swap, it's all over. Who in their
right mind wants to compile a large program and use a LOT of swap? I
hope nobody. lol

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 04:49 PM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 06/09/12, Dale wrote:
>
>> Then take a look at it this way. If I emerge seamonkey with portage's
>> work directory on disk and it takes 12 minutes, the first time. Then I
>> clear the caches and emerge seamonkey again while portage's work
>> directory is on tmpfs and it is 12 minutes. Then repeat that process a
>> few times more. If the outcome of all those emerges is 12 minutes,
>> regardless of the order, then putting portages work directory on tmpfs
>> makes no difference at all in that case.
> We fully agree with you, here.
>

That's good.


>> The emerge times are exactly
>> the same regardless of emerge using cache or not or portage's work
>> directory being on tmpfs or not. I don't care if emerge uses cache
>> DURING the emerge process because it is always enabled in both tests.
> But you *should* care. If you don't have enough memory, the kernel will
> reclaim memory from the pagecache, so the whole process rapidity won't
> only rely on RAM rapidity anymore.

But if you are going to use tmpfs, you have to have the memory
available. It doesn't matter if it is tmpfs or just used in the normal
way. That is my point.

>> The point is whether portage's work directory is on tmpfs or not makes
>> emerges faster.
>>
>> The thing about what you are saying is that I ran those tests with the
>> files in memory. What I am saying is this, that is not the case. I am
>> clearing that memory with the drop_cache command between each test. You
>> claim that cache is affecting the timing but I am clearing the very same
>> cache the same as a reboot would. The emerge times whether portage's
> We do agree with you that you droped the cache between the tests with
> almost the same effect of a reboot.

That's good.

>> The emerge times whether portage's
>> work directory is on tmpfs or not didn't change enough to make a
>> difference.
> Yes, we agree. You droped the cache which is expected to get correct
> tests.
>
> What we are saying is that you droped the cache but did NOT DISABLED the
> VM caches (kernel cache). You say that you don't care of that one
> because it was involved in all the tests. We say that you might not care
> in some contexts, not for all the contexts. You reach the context where
> it does not matter much, fine.
>

Who doing a normal update would cut off the cache? I wouldn't. I know
how to clear it but I don't know how to disable it nor would I or most
likely anyone else in normal use. The point of my test was in a normal
use case of emerge with or without tmpfs and if there is any difference
in the emerge times. There wasn't. Once emerge starts and loads all
the stuff it needs, tmpfs doesn't matter at that point.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 08:21 PM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 11:44:07 -0500, Dale wrote:

> I kind of get what they are saying but at the same time using tmpfs
> doesn't matter. Once the tarball is read off the drive, it doesn't
> matter whether portage is run on a tmpfs or not.

Reading the tarball has nothing to do with this, we are discussing
filesystems for PORTAGE_TMPDIR, not DISTDIR. It's where the source is
unpacked, the object files compiled to, the executables linked to and the
install image created that is relevant to TMPDIR.


--
Neil Bothwick

What's the difference between ignorance and apathy?
I don't know and I don't care
 
Old 09-06-2012, 08:21 PM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 11:32:41 -0500, Dale wrote:

> Others ran their own tests and got the same results.

No one is denying the results, only the reasons given for them.

--
Neil Bothwick

If you can't be kind, be vague.
 
Old 09-06-2012, 09:09 PM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 11:44:07 -0500, Dale wrote:
>
>> I kind of get what they are saying but at the same time using tmpfs
>> doesn't matter. Once the tarball is read off the drive, it doesn't
>> matter whether portage is run on a tmpfs or not.
> Reading the tarball has nothing to do with this, we are discussing
> filesystems for PORTAGE_TMPDIR, not DISTDIR. It's where the source is
> unpacked, the object files compiled to, the executables linked to and the
> install image created that is relevant to TMPDIR.
>
>

Well, on my system, when I run emerge, it has to go read the tarball
from the drive before it can unpack and do all the rest that needs to be
done. I was timing from the time I hit return on the emerge command
till it was done. Actually, I used time to time it for me. ;-)

As I said, I ran these tests on what a typical user would be using.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-06-2012, 10:38 PM
Neil Bothwick
 
Default aligning SSD partitions

On Thu, 06 Sep 2012 16:09:12 -0500, Dale wrote:

> > Reading the tarball has nothing to do with this, we are discussing
> > filesystems for PORTAGE_TMPDIR, not DISTDIR. It's where the source is
> > unpacked, the object files compiled to, the executables linked to and
> > the install image created that is relevant to TMPDIR.

> Well, on my system, when I run emerge, it has to go read the tarball
> from the drive before it can unpack and do all the rest that needs to be
> done.

Of course, but it is reading from a different filesystem that is
unaffected by your choice for $PORTAGE_TMPDIR. It has about as much
relevance as the brand of mouse you are using.


--
Neil Bothwick

"Fascinating," said Spock, watching Kirk's lousy acting.
 
Old 09-06-2012, 11:08 PM
Dale
 
Default aligning SSD partitions

Neil Bothwick wrote:
> On Thu, 06 Sep 2012 16:09:12 -0500, Dale wrote:
>
>>> Reading the tarball has nothing to do with this, we are discussing
>>> filesystems for PORTAGE_TMPDIR, not DISTDIR. It's where the source is
>>> unpacked, the object files compiled to, the executables linked to and
>>> the install image created that is relevant to TMPDIR.
>> Well, on my system, when I run emerge, it has to go read the tarball
>> from the drive before it can unpack and do all the rest that needs to be
>> done.
> Of course, but it is reading from a different filesystem that is
> unaffected by your choice for $PORTAGE_TMPDIR. It has about as much
> relevance as the brand of mouse you are using.
>
>


But whether it is on tmpfs or just regular memory doesn't matter. Once
emerge starts, everything is in ram including portages work directory
which would be on tmpfs here. That's why it doesn't matter if portage
is on tmpfs or not. Once emerge loads up the files, it's the same
thing. That's why using tmpfs doesn't matter. I knew that the whole
time. The amount of ram on a system doesn't matter either. If you have
a system that doesn't have a lot of ram, then you can't really use tmpfs
anyway. That is not something I would recommend to anyone.

I just don't agree that one should *disable* cache to run the test since
no one would disable cache on a normal system. It's not a memory speed
test. It's a test to see if putting it on tmpfs makes it faster. The
fact that emerge loads everything up in memory when it starts is not
relevant for what I am testing. It does that on its own anyway.

Since portage and the kernel does this in the most efficient way
already, I still say putting portage's work directory on tmpfs is not
needed UNLESS a person needs to save wear and tear on a drive, such as
the SSD in this thread. I just don't want someone that is sort of new
to Gentoo and compiling things to think that a package that takes 10
minutes when done on disk will take 3 minutes when on tmpfs. I see that
thinking from time to time, usually on the forums.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 

Thread Tools




All times are GMT. The time now is 11:48 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org