FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 09-07-2012, 07:47 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> But whether it is on tmpfs or just regular memory doesn't matter. Once
> emerge starts, everything is in ram including portages work directory
> which would be on tmpfs here. That's why it doesn't matter if portage
> is on tmpfs or not. Once emerge loads up the files, it's the same
> thing. That's why using tmpfs doesn't matter. I knew that the whole
> time. The amount of ram on a system doesn't matter either. If you have
> a system that doesn't have a lot of ram, then you can't really use tmpfs
> anyway. That is not something I would recommend to anyone.

But you're wrong with this assumption. I guess you never tried to
upgrade a Gentoo system running as server (a working one, with users and
workload).

The amount of memory is only /one/ helper parameter to not see a
difference.

Like I've already said, the issue is all about the persistence strategy
used by the VM to mark memory as pinned, reclaimable or swappable. Where
tmpfs do change the matter is that a file stored in it is not going be
dropped from RAM until there is a unlink(2) call on it or that other
running processes are running out of memory and some page needs to be
swapped (so there is _already_ no more available RAM in the kernel cache).

If not using tmpfs and because memory cache is the first place where the
kernel will free up memory, you don't have to wait for the processes to
run out of available memory to hit a situation where you'll have to wait
for the disk to retrieve files. So, this will affect times.

--
Nicolas Sebrecht
 
Old 09-07-2012, 07:56 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 06/09/12, Dale wrote:

> But this is what you guys are missing too. If you want to use tmpfs,
> you have to have enough ram to begin with. Whether you use tmpfs or
> not, you have to have enough ram to do the compile otherwise you start
> using swap or it just crashes. Having ram is a prerequisite to using
> tmpfs.

This is too minimal overview to get the point. Memory is not a static
place. This is not a cake beeing shared once. Memory is living. See my
other mail.

> There is another flaw in your assumption above. I already had the
> tarballs downloaded BEFORE even the first emerge.

This is not a flaw in assumption. This is negligible.

> What the people wanted to test is if putting portages work directory on
> tmpfs would make emerge times faster.

Come'on. We all understood your goal from the beginning.

> Do we all admit that having portage on tmpfs does not make emerge times
> faster yet?

No. It depends on factors and underlying processes you claim they don't
matter, which is wrong. They *might* be not relevant in some cases.

--
Nicolas Sebrecht
 
Old 09-07-2012, 09:15 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 07/09/12, Nicolas Sebrecht wrote:

> > There is another flaw in your assumption above. I already had the
> > tarballs downloaded BEFORE even the first emerge.
>
> This is not a flaw in assumption. This is negligible.

Fixing myself: s/negligible/out of the scope/

--
Nicolas Sebrecht
 
Old 09-07-2012, 12:25 PM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 06/09/12, Dale wrote:
>
>> But this is what you guys are missing too. If you want to use tmpfs,
>> you have to have enough ram to begin with. Whether you use tmpfs or
>> not, you have to have enough ram to do the compile otherwise you start
>> using swap or it just crashes. Having ram is a prerequisite to using
>> tmpfs.
> This is too minimal overview to get the point. Memory is not a static
> place. This is not a cake beeing shared once. Memory is living. See my
> other mail.

I understand that memory is static but that is NOT what I was testing or
others either. The test it whether putting portage's work directory on
tmpfs makes emerges faster not whether emerge using memory itself makes
it faster. Since when you run emerge it loads everything into ram,
regardless of whether portages work directory is on tmpfs or not, it
doesn't matter. This test is NOT about portage loading things into ram
WHILE emerging, it was about having the work directory on tmpfs and
speed. Since emerge loads everything right after you hit the enter key,
it doesn't matter where the work directory is located.

We wanted to change only one thing for this test, where portage's work
directory was. It was not about how much ram a system has but where
tmpfs was located. To use tmpfs, the system has to have enough ram to
begin with so systems that do not have larger amounts of ram were not
even relevant to the question. If a system has small amounts of ram,
then most likely they can't use tmpfs anyway.

>> There is another flaw in your assumption above. I already had the
>> tarballs downloaded BEFORE even the first emerge.
> This is not a flaw in assumption. This is negligible.

It can make a huge difference. The download times are included in the
emerge times if it is not already in distfiles. So, if a tarball takes
a hour to download, it adds one hour to the emerge time. Depending on
internet speed, it can be more than negligible. I have DSL but it is
the slower package so this can in some cases make a HUGE difference
here. Since I was running my tests here, I know it makes a difference
but you assumed it didn't. That would be incorrect. It does make a
difference and it can be a big one depending on the tarball size.

>
>> What the people wanted to test is if putting portages work directory on
>> tmpfs would make emerge times faster.
> Come'on. We all understood your goal from the beginning.

Well great. We, and I, were only testing one thing not two or three
things. We just wanted to change one setting, not disable a whole bunch
of stuff.


>
>> Do we all admit that having portage on tmpfs does not make emerge times
>> faster yet?
> No. It depends on factors and underlying processes you claim they don't
> matter, which is wrong. They *might* be not relevant in some cases.
>

Actually, they don't matter on my system and since others got the same
results, it doesn't matter. Again, we only wanted to change one
specific setting, tmpfs, nothing else. That was the only thing we were
testing and it was the only thing I tested here and it is the only
results I am reporting. I'm not reporting on how well emerge is using
ram after the command is given.

So, accept it or not, it makes no difference whether portage's work
directory is on tmpfs or not speed wise. You get the same results
either way. In the case of the OP of this thread, it would likely be a
good idea if he can but he should not expect emerge to be any faster.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-07-2012, 12:35 PM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 06/09/12, Dale wrote:
>
>> But whether it is on tmpfs or just regular memory doesn't matter. Once
>> emerge starts, everything is in ram including portages work directory
>> which would be on tmpfs here. That's why it doesn't matter if portage
>> is on tmpfs or not. Once emerge loads up the files, it's the same
>> thing. That's why using tmpfs doesn't matter. I knew that the whole
>> time. The amount of ram on a system doesn't matter either. If you have
>> a system that doesn't have a lot of ram, then you can't really use tmpfs
>> anyway. That is not something I would recommend to anyone.
> But you're wrong with this assumption. I guess you never tried to
> upgrade a Gentoo system running as server (a working one, with users and
> workload).

Actually, I do that a lot here but we were not testing this on a server
but on what a regular user would have. I ran most of my tests while in
single user mode tho. I didn't want the fact that I use KDE and it may
be doing something to use CPU resources or even memory to skew the
results. I went over this in another reply long ago.

>
> The amount of memory is only /one/ helper parameter to not see a
> difference.
>
> Like I've already said, the issue is all about the persistence strategy
> used by the VM to mark memory as pinned, reclaimable or swappable. Where
> tmpfs do change the matter is that a file stored in it is not going be
> dropped from RAM until there is a unlink(2) call on it or that other
> running processes are running out of memory and some page needs to be
> swapped (so there is _already_ no more available RAM in the kernel cache).
>
> If not using tmpfs and because memory cache is the first place where the
> kernel will free up memory, you don't have to wait for the processes to
> run out of available memory to hit a situation where you'll have to wait
> for the disk to retrieve files. So, this will affect times.
>

Swap was disabled when I ran the tests even tho I have it set to not use
swap unless it is a must. Memory is memory whether it is tmpfs or just
being used by a process or as disk cache. I only have one type of
memory in my system here. It is all the same no matter how the system
uses it. Since the only setting changed was where the work directory
was located, then emerge pinned the use of memory the same in both
cases. As it should.

The thing is tho, whether it is using the memory as cache or using it as
tmpfs, it is the same memory. There is no difference. That's the whole
point.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-07-2012, 07:48 PM
Neil Bothwick
 
Default aligning SSD partitions

On Fri, 07 Sep 2012 07:25:42 -0500, Dale wrote:

> Since when you run emerge it loads everything into ram,
> regardless of whether portages work directory is on tmpfs or not, it
> doesn't matter. This test is NOT about portage loading things into ram
> WHILE emerging, it was about having the work directory on tmpfs and
> speed.

Of course it doesn't. The tarball is unpacked from DISTDIR to the work
directory. Then individual source files in the work directory are
compiled to object files, also in the work directory. Then those object
files are linked to executables, also in the work directory. Finally,
everything is install to an image directory, also on TMPDIR. The speed of
the work directory would appear to be of critical importance - but it
isn't, as shown by your tests. The reason for this, and the point
everyone else has been making, is because the files are cached by the
kernel, so the filesystem is less important if you have enough RAM.


--
Neil Bothwick

30 minutes of begging is not considered foreplay.
 
Old 09-10-2012, 10:32 AM
Nicolas Sebrecht
 
Default aligning SSD partitions

The 07/09/12, Dale wrote:

> The thing is tho, whether it is using the memory as cache or using it
> as
> tmpfs, it is the same memory. There is no difference. That's the
> whole
> point.

Feel free to take your own assumptions as undeniable truth. The way the
kernel work with memory is the key, of course.

Now, as long as you blind yourself with statements like that, I'm not
going to respond anymore. I guess you need to make some basic research.

--
Nicolas Sebrecht
 
Old 09-10-2012, 11:13 AM
Dale
 
Default aligning SSD partitions

Nicolas Sebrecht wrote:
> The 07/09/12, Dale wrote:
>
>> The thing is tho, whether it is using the memory as cache or using it
>> as
>> tmpfs, it is the same memory. There is no difference. That's the
>> whole
>> point.
> Feel free to take your own assumptions as undeniable truth. The way the
> kernel work with memory is the key, of course.
>
> Now, as long as you blind yourself with statements like that, I'm not
> going to respond anymore. I guess you need to make some basic research.
>

I understand how the kernel uses memory. That's why it doesn't matter
if you put portage's work directory on tmpfs or not. I been using Linux
for a pretty good long while now. I have a pretty good understanding of
it, especially the things that I use.

Respond or not, I know what I tested and what the results were. They
were not just my tests and results either.

Dale

:-) :-)

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 
Old 09-10-2012, 12:37 PM
Michael Mol
 
Default aligning SSD partitions

On Mon, Sep 10, 2012 at 7:13 AM, Dale <rdalek1967@gmail.com> wrote:

Nicolas Sebrecht wrote:

> The 07/09/12, Dale wrote:

>

>> The thing is tho, whether it is using the memory as cache or using it

>> as

>> tmpfs, it is the same memory. *There is no difference. *That's the

>> whole

>> point.

> Feel free to take your own assumptions as undeniable truth. The way the

> kernel work with memory is the key, of course.

>

> Now, as long as you blind yourself with statements like that, I'm not

> going to respond anymore. I guess you need to make some basic research.

>



I understand how the kernel uses memory. *That's why it doesn't matter

if you put portage's work directory on tmpfs or not. *I been using Linux

for a pretty good long while now. *I have a pretty good understanding of

it, especially the things that I use.



Respond or not, I know what I tested and what the results were. *They

were not just my tests and results either.

Nobody is disagreeing with your test results. In fact, they're not even disagreeing with you that they mean what you think they mean within the context you're testing. They're disagreeing with your extrapolation of your results to other contexts. In short, all other things being equal, your test results work out for someone in the exact same circumstances as yourself...but there are a _lot_ of other things that need to be equal!

Filesystem mount options can have an impact. For example, let's say your filesystem is configured to make writes synchronous, for general data integrity purposes. That would slow PORTAGE_TMP down something _fierce_.

Someone might be tweaking any number of the knobs under 'vm' in /proc. vm.swappiness, vm.dirty_* or vm.min_free_kbytes are ones that caught my eye, but really most of them in there look relevant.

Or consider that someone else might be running drop_caches, or even sync() while your code is running. (Heck, if there's a database, even an sqlite database, on the same filesystem, that's almost a guarantee.)

These may seem to be obvious, but these are the kinds of things people were trying to get you to be willing to acknowledge before you made blanket assertions which covered them.

--
:wq
 
Old 09-10-2012, 01:52 PM
Dale
 
Default aligning SSD partitions

Michael Mol wrote:



On Mon, Sep 10, 2012 at 7:13 AM, Dale <rdalek1967@gmail.com>
wrote:



Nicolas Sebrecht wrote:

> The 07/09/12, Dale wrote:

>

>> The thing is tho, whether it is using the memory
as cache or using it

>> as

>> tmpfs, it is the same memory. *There is no
difference. *That's the

>> whole

>> point.

> Feel free to take your own assumptions as undeniable
truth. The way the

> kernel work with memory is the key, of course.

>

> Now, as long as you blind yourself with statements
like that, I'm not

> going to respond anymore. I guess you need to make
some basic research.

>





I understand how the kernel uses memory. *That's why it
doesn't matter

if you put portage's work directory on tmpfs or not. *I been
using Linux

for a pretty good long while now. *I have a pretty good
understanding of

it, especially the things that I use.



Respond or not, I know what I tested and what the results
were. *They

were not just my tests and results either.





Nobody is disagreeing with your test results. In fact,
they're not even disagreeing with you that they mean what you
think they mean within the context you're testing. They're
disagreeing with your extrapolation of your results to other
contexts. In short, all other things being equal, your test
results work out for someone in the exact same circumstances
as yourself...but there are a _lot_ of other things that need
to be equal!



Filesystem mount options can have an impact. For example,
let's say your filesystem is configured to make writes
synchronous, for general data integrity purposes. That would
slow PORTAGE_TMP down something _fierce_.



Someone might be tweaking any number of the knobs under
'vm' in /proc. vm.swappiness, vm.dirty_* or vm.min_free_kbytes
are ones that caught my eye, but really most of them in there
look relevant.



Or consider that someone else might be running drop_caches,
or even sync() while your code is running. (Heck, if there's a
database, even an sqlite database, on the same filesystem,
that's almost a guarantee.)



These may seem to be obvious, but these are the kinds of
things people were trying to get you to be willing to
acknowledge before you made blanket assertions which covered
them.




--

:wq






Someone could be getting rays from Mars but I am not testing that.*
What I tested was this,* Run emerge with portages work directory on
disk.* Then run same command with portage's work directory on
tmpfs.* Then compare the results.* No other changes except for where
portage's work directory is located, hard drive or ram.* This was
done on a NORMAL system that most ANY user would be using.* I'm not
concerned with some rare or exotic setup, just a normal setup.* If
someone is running some exotic setup, then they need to test that to
see whether it helps or not because I did not test for that sort of
system.* I didn't test for rays from Mars either.* LOL



Dale



:-)* :-)*

--
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
 

Thread Tools




All times are GMT. The time now is 03:04 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org