FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian KDE

 
 
LinkBack Thread Tools
 
Old 05-13-2010, 09:18 AM
Kevin Krammer
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wednesday, 2010-05-12, Mike Kasick wrote:
> On Wed, May 12, 2010 at 10:41:35AM +0200, Kevin Krammer wrote:
> > Since you are writing a bit down that you think it is caused by
> > kres-migrator, where did you get it from (here it seems to be part of
> > the kdepim-runtime package).
>
> Yes, kres-migrator is part of kdepim-runtime. I do have that package
> installed, as it seems to be indirectly depended upon by kde-minimal. It's
> kdepim itself, and its application dependencies (kaddresbook, kalarm, kmail
> knode, knotes, kontact, korganizer, etc.) I don't have installed. Maybe
> the kdepim-runtime dependency itself is a bug--can't say.

It depends.
One possible way would be to check every single application (and potentially
their plugins) for PIM related functionality.
I guess it is just easier to depend on runtime. These processes are run on-
demand only anyway, so in the worst case they consume disk space if nothing is
actually using PIM functionality.

> > kres-migrator is called when an application accesses the KResource
> > framework, e.g. some app accessing the old addressbook API.
> > Not using KDEPIM apps does not necessarily mean non of your other
> > applications access PIM data.
>
> Looks like the culprit here is libkabc. There's a "Default Addressboook"
> created by the library, that's presumably empty. I'm not sure what's
> loading libkabc in the first place. I do know that I didn't even have kabc
> database files (.kde/share/apps/kabc/std.vcf*) until upgrading to 4.4.
> Maybe it's an explicit part of the migration? Or I suppose one of the
> panel widgets I'm using might depend on it now, but I don't believe that's
> the case.

Ah, right. Probably a KRunner plugin for actions on contacts.

> Looked into this a bit. The InnoDB documentation itself is a little
> lacking on describing its particular architecture, but there's an InnoDB
> tuning tutorial [1] that's rather helpful.
>
> These files serve as InnoDB's REDO logs. They serve two purposes. First,
> committed transactions are written to the REDO logs sequentially, so that
> table updates (with possible random seeks) can be done in a write-back mode
> "at leisure."
>
> Second, REDO logs serve as a durability mesaure. Each time the database is
> restarted, the REDO logs are replayed to ensure that recent transactions
> have been properly commited--say if either the database is "kill -9ed" or
> there's other table corruption. They may also be used in recovery, whereby
> if table corruption is found and old tables can be reloaded from backup,
> then the REDO logs can be replayed to bring the tables up to date. You can
> also forward REDO logs to standby (fail-over) servers to ensure their
> database tables are up to date.
>
> The REDO logs themselves contain row updates from insert/update statements.
> So for a given row length, the REDO logs contain the last
> LOG_SIZE/ROW_LENGTH transactions. They're not used in selects or other
> non-mutating accesses.
>
> REDO log size is not an issue of correctness. A small log size might
> result in decreased performance by forcing a burst of inserts/updates to be
> committed to table before completing a transaction. A larger log size may
> also be of benefit in data recovery if database corruption is found, and a
> recent enough table backup is maintained so that the REDO log still
> contains all non-backed up transactions.

Ah, good research, thank you!

> Let's try to quantify this a bit. I'm not exactly sure what kind of
> database workloads Akonadi is targetting, but for PIM applications we're
> looking at managing (1) contacts, (2) calendar entries, (3) "TODO" tasks,
> (4) notes-to-self, etc. It seems to me that each of these things results
> in:

Just for the context, this is for KDE 4.4
4.5 potentially adds (5) emails

> - Table row length on order of 1 kB.

I think the preconfigured threshold for database cached parts is 4KB, though
there can be several such parts per item (depends on the data type).

> - Total number of rows < 10,000 (how many people do you know?)
> - Largely read-only data sets, grows over period of years.
> - A working set (actively updated rows) < 1,000 per day. Probably < 100.

Right, again in in the 4.4 context.
With emails these can easily be surpassed, especially on update rate (mails
come in, get marked read, moved, deleted, etc)

> The part that bothers me is that the Akonadi folks are basically aware of
> the situation, and feel justified in claiming [2] that 100+ MB of disk is
> reasonable. Franlky, if you ask even an arm-chair DBA if using InnoDB with
> these parameters are appropriate for per-user PIM management, they'll look
> at you like your crazy--which is, from what I can tell, the underlying
> reason for so much of the dislike with KDE 4.4.

My take is that there is quite some room for optimizations through input by
people with good knowledge of database systems.
As far as I know the developers had help from some MySQL expert on the initial
configuration, but since MySQL evolves over time the chosen settings might not
be that good anymore.

> I can't imagine that SQLite was really _so bad_ of a target for low-usage
> PIM workloads that the Akonadi folks couldn't have just written a plugin
> for it some time ago and filed some bugs. Afterall, Firefox uses it rather
> extensively, seems like it would've been a perfect fit. But that's another
> story, and we just have to make do with what we have right now.

SQLite wasn't viable until very recently due to deadlocking when transactions
were being using in a multithreading environment.
My most up-to-date information on that is that it mostly works now with a
development version of SQLite and some additional changes elsewhere.
IIRC there is only one deadlock case left.

Performance wise it might be a viable solution for people with low
requirements on PIM, e.g. not for people like myself with several hundret
thousand mails, tens of quite active mailinglist, etc.

Cheers,
Kevin
 
Old 05-13-2010, 09:28 AM
Kevin Krammer
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wednesday, 2010-05-12, Mike Kasick wrote:
> On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote:
> > > - Disable "kres-migrator". Or at least add a debconf option to
> > > kdepim-runtime presenting the option of running kres-migrator by
> > > default or disabling it.
> >
> > Did you do any research as to what the longer-term implications
> > of foregoing this might be? I am not saying there are any, I
> > honestly don't know.
>
> There shouldn't be any long-term implications as long as the old KResource
> code still exists. I'm assuming it's being deprecated as of KDE 4.4, but
> I'd be surprised if it's removed in KDE 4.5, especially since delaying
> migration is an official recommendation.

The KResource API is public API of KDE4, it can at best removed at the first
KDE5 version (which is probably still several years away).
Some of its plugins are public API as well (those from kdepimlibs) so they
stay too.
Problematic are plugins from kdepim, they can be replaced by respective
Akonadi base resource implementations.

> Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the
> data by running "kres-migrator". I actually did this accidentally. The
> migrator _might_ also run if we remove the configuration bit, but I've not
> tried.

Right, I think it will run if it does not find its configuration, thus not
knowing whether it has already run.
Could lead to duplicated resources on Akonadi side though (if it has been run
before).

Cheers,
Kevin
 
Old 05-13-2010, 07:44 PM
Nicolas Alvarez
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

Mike Kasick wrote:
> On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote:
>
>> I think that disk usage is an issue in most software, not just KDE apps.
>> For example, look at ~/.thumbnails, and you will see that is probably
>> filled with many thumbnails of images that you deleted long ago or that
>> you saw just once. Same thing for apps that store some data about all
>> files that you open, without limit and without expiration.
>
> I purge .thumbnails occasionally for this reason. I also wasn't happy to
> see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge
> that now on apt-get upgrades. Most recently I saw .xsession-errors grow
> to ~600 MB in just under a few days, so I've /dev/nulled that too except
> for when I need it.

As a KDE4.3 user with 1TB of disk, I'm way more annoyed at
kio_http_cache_cleaner starting regularly and eating my computer's I/O than
the HTTP cache growing big. I can easily notice when the the cache cleaner
is running from the HD light, the HD noise, and the general system lag.

There *has* to be a better way to clean an HTTP cache than to stat every
single file in the cache to then delete a tiny fraction of them. Someone
told me 4.4 improved in this area, but I don't know specifics, and remain
skeptical.

--
Nicolas

(I read mailing lists through Gmane. Please don't Cc me on replies; it makes
me get one message on my newsreader and another on email.)


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: hshkqb$8ov$1@dough.gmane.org">http://lists.debian.org/hshkqb$8ov$1@dough.gmane.org
 
Old 05-13-2010, 08:16 PM
Christoph Burgmer
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

> As a KDE4.3 user with 1TB of disk, I'm way more annoyed at
> kio_http_cache_cleaner starting regularly and eating my computer's I/O than
> the HTTP cache growing big. I can easily notice when the the cache cleaner
> is running from the HD light, the HD noise, and the general system lag.
>
> There *has* to be a better way to clean an HTTP cache than to stat every
> single file in the cache to then delete a tiny fraction of them. Someone
> told me 4.4 improved in this area, but I don't know specifics, and remain
> skeptical.

Officially this problem has been solved. The only difference I see though is that
killing the process will instantly create a new one. So for me it got worse,
rather then better.

One would think there was another way, after all other browsers seem to get by
without that much IO. I guess it's just an unpopular area of KDE so nobody
really wants to sit down and fix it. Never had any issues with 3.5, not sure
what was changed afterwards.

-Christoph


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201005132216.14037.christoph.burgmer@stud.uni-karlsruhe.de">http://lists.debian.org/201005132216.14037.christoph.burgmer@stud.uni-karlsruhe.de
 
Old 05-14-2010, 04:37 PM
Allan Sandfeld Jensen
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Thursday 13 May 2010, Nicolas Alvarez wrote:
> As a KDE4.3 user with 1TB of disk, I'm way more annoyed at
> kio_http_cache_cleaner starting regularly and eating my computer's I/O than
> the HTTP cache growing big. I can easily notice when the the cache cleaner
> is running from the HD light, the HD noise, and the general system lag.
>
> There *has* to be a better way to clean an HTTP cache than to stat every
> single file in the cache to then delete a tiny fraction of them. Someone
> told me 4.4 improved in this area, but I don't know specifics, and remain
> skeptical.

If stat updates the files' atime, you may get a lot better performance by
simply mounting the relevant partition with relatime or noatime (which for
some reason is still not default).

Mvh
Allan


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201005141837.20338.debian@carewolf.com">http://lists.debian.org/201005141837.20338.debian@carewolf.com
 
Old 05-14-2010, 05:10 PM
"Boyd Stephen Smith Jr."
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Friday 14 May 2010 11:37:19 Allan Sandfeld Jensen wrote:
> On Thursday 13 May 2010, Nicolas Alvarez wrote:
> > There *has* to be a better way to clean an HTTP cache than to stat every
> > single file in the cache to then delete a tiny fraction of them. Someone
> > told me 4.4 improved in this area, but I don't know specifics, and remain
> > skeptical.
>
> If stat updates the files' atime, you may get a lot better performance by
> simply mounting the relevant partition with relatime or noatime (which for
> some reason is still not default).

noatime "breaks" a few applications. I recommend against it.

I've not seen similar issues with relatime, although it is theoretically
possible. I mount using relatime, normally.
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/
 
Old 05-14-2010, 05:55 PM
Allan Sandfeld Jensen
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Friday 14 May 2010, Boyd Stephen Smith Jr. wrote:
> On Friday 14 May 2010 11:37:19 Allan Sandfeld Jensen wrote:
> > On Thursday 13 May 2010, Nicolas Alvarez wrote:
> > > There *has* to be a better way to clean an HTTP cache than to stat
> > > every single file in the cache to then delete a tiny fraction of them.
> > > Someone told me 4.4 improved in this area, but I don't know specifics,
> > > and remain skeptical.
> >
> > If stat updates the files' atime, you may get a lot better performance by
> > simply mounting the relevant partition with relatime or noatime (which
> > for some reason is still not default).
>
> noatime "breaks" a few applications. I recommend against it.
>
> I've not seen similar issues with relatime, although it is theoretically
> possible. I mount using relatime, normally.

It breaks mutt, relatime was invented to fix the mutt issue. A lot of netbooks
and machines with only SSDs use noatime now, if noatime broke anything serious
at any point, those issues will have been solved by now, SSDs have been out
for some time and they need noatime.

Also possibly if you don't like the mount flags:
chattr -R +A /var/tmp

Also add
chattr -R +A $KDEHOME

The later is good for the config files.

`Allan


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201005141955.46390.debian@carewolf.com">http://lists.debian.org/201005141955.46390.debian@carewolf.com
 
Old 05-14-2010, 06:45 PM
"Boyd Stephen Smith Jr."
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Friday 14 May 2010 12:55:45 Allan Sandfeld Jensen wrote:
> On Friday 14 May 2010, Boyd Stephen Smith Jr. wrote:
> > noatime "breaks" a few applications. I recommend against it.
> >
> > I've not seen similar issues with relatime, although it is theoretically
> > possible. I mount using relatime, normally.
>
> It breaks mutt, relatime was invented to fix the mutt issue. A lot of
> netbooks and machines with only SSDs use noatime now, if noatime broke
> anything serious at any point, those issues will have been solved by now,
> SSDs have been out for some time and they need noatime.

SSDs marketed for HD replacement do not need noatime. They have re-write
limits such that, if written to at maximum speed for the lifetime of a
comparable HD, less than the number of reserved cells would fail and no data
would be lost. They are roughly 1.5 orders of magnitude more durable than
"thumbdrives" -- they have to be to stand a MS Windows page file or a Linux
swap partition/file.

noatime can increase performance, it does so by breaking the expectations of
at least mutt (possibly others). atime updates are a normal expectation,
since they've been included in the documentation of dozens of UNIX system
calls since before standardization started on POSIX.

relatime is a much better alternative and is roughly equivalent to the OS
deciding to combine atime updates, which is explicitly allowed by the
standards.
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/
 
Old 05-14-2010, 11:26 PM
Richard Hartmann
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 17:36, Christoph Burgmer
<christoph.burgmer@stud.uni-karlsruhe.de> wrote:

> 4.4 was supposed to have major improvements over 4.3. 4.2 was supposed to fix
> major issues of 4.1 and 4.0. Also called, I believe, the first "stable" version
> of KDE 4.

Every minor release has been significantly better than its predecessor.
4.4 might not be perfect, but it's the best KDE version since 3.5.10 and
it's the only one which has a realistic chance of ending up in Squeeze.


Richard


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTikDVaco7HfpDG1MxiYEqnfz9Yb7pQZb9SRPp4Jb@mail .gmail.com">http://lists.debian.org/AANLkTikDVaco7HfpDG1MxiYEqnfz9Yb7pQZb9SRPp4Jb@mail .gmail.com
 
Old 05-15-2010, 05:21 AM
Nicolas Alvarez
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

Allan Sandfeld Jensen wrote:
> On Thursday 13 May 2010, Nicolas Alvarez wrote:
>> As a KDE4.3 user with 1TB of disk, I'm way more annoyed at
>> kio_http_cache_cleaner starting regularly and eating my computer's I/O
>> than the HTTP cache growing big. I can easily notice when the the cache
>> cleaner is running from the HD light, the HD noise, and the general
>> system lag.
>>
>> There *has* to be a better way to clean an HTTP cache than to stat every
>> single file in the cache to then delete a tiny fraction of them. Someone
>> told me 4.4 improved in this area, but I don't know specifics, and remain
>> skeptical.
>
> If stat updates the files' atime, you may get a lot better performance by
> simply mounting the relevant partition with relatime or noatime (which for
> some reason is still not default).

After some stracing: It doesn't just stat the files. It first stats, then
actually opens the files and *reads* the first few bytes of all of them
(where the HTTP cache information is stored).

--
Nicolas

(I read mailing lists through Gmane. Please don't Cc me on replies; it makes
me get one message on my newsreader and another on email.)


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: hslb13$ibk$1@dough.gmane.org">http://lists.debian.org/hslb13$ibk$1@dough.gmane.org
 

Thread Tools




All times are GMT. The time now is 08:35 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org