FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian KDE

 
 
LinkBack Thread Tools
 
Old 05-12-2010, 04:45 PM
Frederik Schwarzer
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

[Alejandro Exojo - Mittwoch 12 Mai 2010 18:28:35]
> El Miércoles, 12 de Mayo de 2010, Christoph Burgmer escribió:
> > Am Mittwoch, 12. Mai 2010 schrieb Alejandro Exojo:
> > > I think that disk usage is an issue in most software, not just KDE apps.
> > > For example, look at ~/.thumbnails, and you will see that is probably
> > > filled with many thumbnails of images that you deleted long ago or that
> > > you saw just once. Same thing for apps that store some data about all
> > > files that you open, without limit and without expiration.
> >
> > You can tell Gwenview to remove .thumbnails on exit.
>
> The point of .thumbnails to be created, I thought it was to have the scaled
> images already generated. If I did this, what's the point of having this
> directory?

If you want to have these images there in case they are needed, keep the
folder, if disk space is more important than loading speed, remove it
regularily.

Regards


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201005121845.25780.schwarzerf@gmail.com">http://lists.debian.org/201005121845.25780.schwarzerf@gmail.com
 
Old 05-12-2010, 04:47 PM
Modestas Vainius
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

Hello,

On trečiadienis 12 Gegužė 2010 18:48:04 Boyd Stephen Smith Jr. wrote:
> Options:
> 1. Conspire to keep the RC-bug count high enough on other packages for
> long enough that KDE 4.5 gets into testing.
> 2. Convince upstream to to release KDE SC 4.4½.0 before the freeze, or
> roll these new features into another KDE SC 4.4.x release before the
> freeze 3. Where possible, get the feature into Debian by working on the
> dependencies outside of "KDE-proper".
>
> I think I'm going to work on 3. Virtuoso, Akonadi, and Strigi (among
> others) may be appropriate for a upgrade before either the freeze or the
> KDE SC 4.5.0 release. The KDE-integration parts might be missing, but as
> long as new versions don't cripple KDE (AFAIK, their only "consumer" in
> Debian) new upstream versions and bug fixes can be accepted until the
> freeze.

Except that kdesupport applications are typically not the problem. It is KDE
integration where majority of bugs are.


--
Modestas Vainius <modestas@vainius.eu>
 
Old 05-12-2010, 05:08 PM
"Boyd Stephen Smith Jr."
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wednesday 12 May 2010 11:47:08 Modestas Vainius wrote:
> On trečiadienis 12 Gegužė 2010 18:48:04 Boyd Stephen Smith Jr. wrote:
> > Virtuoso, Akonadi, and Strigi (among
> > others) may be appropriate for a upgrade before either the freeze or the
> > KDE SC 4.5.0 release. The KDE-integration parts might be missing, but as
> > long as new versions don't cripple KDE (AFAIK, their only "consumer" in
> > Debian) new upstream versions and bug fixes can be accepted until the
> > freeze.
>
> Except that kdesupport applications are typically not the problem. It is
> KDE integration where majority of bugs are.

I propose we drop KDE from stable for the Squeeze release, then. ;P

Seriously though, if the RC-bug list keeps falling, there's no time to wait on
KDE SC 4.5. Let's try and get a high-quality KDE SC 4.4 ready for Squeeze.
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/
 
Old 05-12-2010, 05:19 PM
Mike Kasick
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 10:41:35AM +0200, Kevin Krammer wrote:

> Since you are writing a bit down that you think it is caused by kres-migrator,
> where did you get it from (here it seems to be part of the kdepim-runtime
> package).

Yes, kres-migrator is part of kdepim-runtime. I do have that package
installed, as it seems to be indirectly depended upon by kde-minimal. It's
kdepim itself, and its application dependencies (kaddresbook, kalarm, kmail
knode, knotes, kontact, korganizer, etc.) I don't have installed. Maybe
the kdepim-runtime dependency itself is a bug--can't say.

> kres-migrator is called when an application accesses the KResource framework,
> e.g. some app accessing the old addressbook API.
> Not using KDEPIM apps does not necessarily mean non of your other applications
> access PIM data.

Looks like the culprit here is libkabc. There's a "Default Addressboook"
created by the library, that's presumably empty. I'm not sure what's
loading libkabc in the first place. I do know that I didn't even have kabc
database files (.kde/share/apps/kabc/std.vcf*) until upgrading to 4.4.
Maybe it's an explicit part of the migration? Or I suppose one of the
panel widgets I'm using might depend on it now, but I don't believe that's
the case.

> Anyone know what kind of data is stored in these logs?

Looked into this a bit. The InnoDB documentation itself is a little
lacking on describing its particular architecture, but there's an InnoDB
tuning tutorial [1] that's rather helpful.

These files serve as InnoDB's REDO logs. They serve two purposes. First,
committed transactions are written to the REDO logs sequentially, so that
table updates (with possible random seeks) can be done in a write-back mode
"at leisure."

Second, REDO logs serve as a durability mesaure. Each time the database is
restarted, the REDO logs are replayed to ensure that recent transactions
have been properly commited--say if either the database is "kill -9ed" or
there's other table corruption. They may also be used in recovery, whereby
if table corruption is found and old tables can be reloaded from backup,
then the REDO logs can be replayed to bring the tables up to date. You can
also forward REDO logs to standby (fail-over) servers to ensure their
database tables are up to date.

The REDO logs themselves contain row updates from insert/update statements.
So for a given row length, the REDO logs contain the last
LOG_SIZE/ROW_LENGTH transactions. They're not used in selects or other
non-mutating accesses.

REDO log size is not an issue of correctness. A small log size might
result in decreased performance by forcing a burst of inserts/updates to be
committed to table before completing a transaction. A larger log size may
also be of benefit in data recovery if database corruption is found, and a
recent enough table backup is maintained so that the REDO log still
contains all non-backed up transactions.

Let's try to quantify this a bit. I'm not exactly sure what kind of
database workloads Akonadi is targetting, but for PIM applications we're
looking at managing (1) contacts, (2) calendar entries, (3) "TODO" tasks,
(4) notes-to-self, etc. It seems to me that each of these things results
in:

- Table row length on order of 1 kB.
- Total number of rows < 10,000 (how many people do you know?)
- Largely read-only data sets, grows over period of years.
- A working set (actively updated rows) < 1,000 per day. Probably < 100.

Thus, I would conclude that tables rarely grow larger than 10 MB (1 kB
* 10,000). The number of inserts/updates per table shouldn't exceed
100 kB-1 MB per day. We're also unlikely to see bursty updates anytime
information is manually provided, since it has to be typed in. Bursty
updates would happen on device synchronization, at which point you might
see 100 kB-1MB of table updates in a few-second window.

This means that we should be able to record 1-10 days of update history
with 1 MB transaction logs. And that's under a heavy PIM load.

We also know that InnoDB's "leisure write" pace is 64 pages per second.
Each page is 16 kB. If we're pessimistic, and say that row updates are
randomly distributed enough such that there's only a single updated-row per
page, then it would take up to 1.5s-15.6s to "leisurly flush" 100 kB-1MB of
table updates. So it really only makes sense to increase the REDO log
beyond 1 MB for performance purposes if we expect 1,000 of random-row (1
MB) updates to occur more frequently than once every 15 seconds. That
doesn't really seem plausible with these kinds of workloads. In the even
it _does_ happen, then it just takes a little longer to finish the sync.

So that's my argument for 1 MB REDO logs. Let's look at the other defaults
for a bit:

innodb_log_buffer_size=1M -- Should be large enough to assemble a
transaction in memory. 1 MB fits 1000 1kB-single-row transactions. Fine.

innodb_buffer_pool_size=80M -- Page cache size, including cached read
pages. Seems fairly large and wasteful use of RAM if we expect tables
themselves to grow no larger than 10MB. 8 MB might be a more reasonable
default, and in the event that a database does grow large, the Linux buffer
cache should eliminiate most of the disk fetches (unless the tables are
opened O_DIRECT, I'm not sure about that).

innodb_file_per_table=1 -- Each table (basically, application) uses its own
DB file. Implicitly configured is each table having a minimum size of 10
MB, and growing in 8 MB increments.

Cutting to the chase: by the current configuration, each per-user MySQL
instance will use 80 MB of RAM for a database buffer pool, and will create
10 MB files per-table (per-application), but we expect tables to never
increase beyond that size. Overall seems kind of wasteful for this type of
workload. These are also defaults that are meant for the lower-end of
centralized DB workload-scenarios, not per-user PIM storage. If InnoDB is
to be used long term, we could really benefit from constructing a sample
workload and (asking a DBA to help us in) tuning the inital parameters
appropriately.

But if we're going to go with things as is, I've already made the case for
why a 1 MB REDO log should be sufficient. I would actually claim now,
though, that 5 MB REDO logs wouldn't be unreasonable in this context
either. Turns out that 5 MB REDO logs are the InnoDB default, and that
would mean that REDO logs would occupy the same amount of disk space as an
empty table. Since we're paying a 10 MB penalty for each application to
use Akonadi in the first place, another 10 MB for the logs isn't extremely
egregious.

The part that bothers me is that the Akonadi folks are basically aware of
the situation, and feel justified in claiming [2] that 100+ MB of disk is
reasonable. Franlky, if you ask even an arm-chair DBA if using InnoDB with
these parameters are appropriate for per-user PIM management, they'll look
at you like your crazy--which is, from what I can tell, the underlying
reason for so much of the dislike with KDE 4.4.

I can't imagine that SQLite was really _so bad_ of a target for low-usage
PIM workloads that the Akonadi folks couldn't have just written a plugin
for it some time ago and filed some bugs. Afterall, Firefox uses it rather
extensively, seems like it would've been a perfect fit. But that's another
story, and we just have to make do with what we have right now.

[1] http://mysqldump.azundris.com/archives/78-Configuring-InnoDB-An-InnoDB-tutorial.html
[2] http://techbase.kde.org/Projects/PIM/Akonadi#Akonadi_needs_too_much_space_in_my_home_di rectory.21


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512171951.GA17357@club.cc.cmu.edu">http://lists.debian.org/20100512171951.GA17357@club.cc.cmu.edu
 
Old 05-12-2010, 05:27 PM
Mike Kasick
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote:

> > - Disable "kres-migrator". *Or at least add a debconf option to
> > *kdepim-runtime presenting the option of running kres-migrator by default
> > *or disabling it.
>
> Did you do any research as to what the longer-term implications
> of foregoing this might be? I am not saying there are any, I
> honestly don't know.

There shouldn't be any long-term implications as long as the old KResource
code still exists. I'm assuming it's being deprecated as of KDE 4.4, but
I'd be surprised if it's removed in KDE 4.5, especially since delaying
migration is an official recommendation.

Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the
data by running "kres-migrator". I actually did this accidentally. The
migrator _might_ also run if we remove the configuration bit, but I've not
tried.

> > - Add a warning dialog during kdepim-runtime upgrade stating that
> > *kres-migrator will run by default and potentially consume a large amount
> > *of per-user disk space, unless migration is manually disabled by the
> > *above command.
>
> Personally, I would like to see such a warning, but I am not sure if
> that is acceptable by Debian's standards. apt-listchanges exists for
> a reason.

It extends beyond just a change in package behavior, it's a non-reversible
and potentially aggravating alteration to user data. I've seen similar
warnings before, most recently to rebuild OpenOffice.org .rdb files due to
possible corruption.

> > - Reduce the default size of the InnoDB transaction logs in the Akonadi
> > *per-user MySQL configuration.
>
> Seems like the best bet to me. If need be, the file size could be adapted
> automagically. But that functionality would be in 4.5 at the earliest unless
> it's patched downstream.

Forgot to mention before, apparently there's an Innodb_log_waits counter
that increments when you have write bursts that exceed the REDO log size.
It can be used for feedback to increase the log size if needed. Perhaps
something to suggest manually doing in NEWS.

> Also, what about /root?

Should have same problem, if one runs a KDE desktop as root, which I do
not.


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512172757.GB17357@club.cc.cmu.edu">http://lists.debian.org/20100512172757.GB17357@club.cc.cmu.edu
 
Old 05-12-2010, 05:33 PM
Mike Kasick
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote:

> I think that disk usage is an issue in most software, not just KDE apps.
> For example, look at ~/.thumbnails, and you will see that is probably
> filled with many thumbnails of images that you deleted long ago or that
> you saw just once. Same thing for apps that store some data about all
> files that you open, without limit and without expiration.

I purge .thumbnails occasionally for this reason. I also wasn't happy to
see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge
that now on apt-get upgrades. Most recently I saw .xsession-errors grow to
~600 MB in just under a few days, so I've /dev/nulled that too except for
when I need it.

So yeah, it's a fairly epidemic problem. I probably notice more than most
because I do daily remote-incremental backups over a DSL line, so an
incremental backup of a few hundred MB definitely gets noticed. Sometimes
it's OK, sometimes it's not.


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512173310.GC17357@club.cc.cmu.edu">http://lists.debian.org/20100512173310.GC17357@club.cc.cmu.edu
 
Old 05-12-2010, 05:42 PM
Mike Kasick
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 10:48:43AM +0200, Richard Hartmann wrote:

> > - Disable "kres-migrator". *Or at least add a debconf option to
> > *kdepim-runtime presenting the option of running kres-migrator by default
> > *or disabling it.
>
> Did you do any research as to what the longer-term implications
> of foregoing this might be? I am not saying there are any, I
> honestly don't know.

There shouldn't be any long-term implications as long as the old KResource
code still exists. I'm assuming it's being deprecated as of KDE 4.4, but
I'd be surprised if it's removed in KDE 4.5, especially since delaying
migration is an official recommendation.

Once the DB mess is sorted out (KDE 4.5?), one can manually migrate the
data by running "kres-migrator". I actually did this accidentally. The
migrator _might_ also run if we remove the configuration bit, but I've not
tried.

> > - Add a warning dialog during kdepim-runtime upgrade stating that
> > *kres-migrator will run by default and potentially consume a large amount
> > *of per-user disk space, unless migration is manually disabled by the
> > *above command.
>
> Personally, I would like to see such a warning, but I am not sure if
> that is acceptable by Debian's standards. apt-listchanges exists for
> a reason.

It extends beyond just a change in package behavior, it's a non-reversible
and potentially aggravating alteration to user data. I've seen similar
warnings before, most recently to rebuild OpenOffice.org .rdb files due to
possible corruption.

> > - Reduce the default size of the InnoDB transaction logs in the Akonadi
> > *per-user MySQL configuration.
>
> Seems like the best bet to me. If need be, the file size could be adapted
> automagically. But that functionality would be in 4.5 at the earliest unless
> it's patched downstream.

Forgot to mention before, apparently there's an Innodb_log_waits counter
that increments when you have write bursts that exceed the REDO log size.
It can be used for feedback to increase the log size if needed. Perhaps
something to suggest manually doing in NEWS.

> Also, what about /root?

Should have same problem, if one runs a KDE desktop as root, which I do
not.


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512174253.GA17540@club.cc.cmu.edu">http://lists.debian.org/20100512174253.GA17540@club.cc.cmu.edu
 
Old 05-12-2010, 05:43 PM
Mike Kasick
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wed, May 12, 2010 at 04:51:23PM +0200, Alejandro Exojo wrote:

> I think that disk usage is an issue in most software, not just KDE apps.
> For example, look at ~/.thumbnails, and you will see that is probably
> filled with many thumbnails of images that you deleted long ago or that
> you saw just once. Same thing for apps that store some data about all
> files that you open, without limit and without expiration.

I purge .thumbnails occasionally for this reason. I also wasn't happy to
see my /var/tmp/kdecache-* grow to 400 MBish either, so I regularly purge
that now on apt-get upgrades. Most recently I saw .xsession-errors grow to
~600 MB in just under a few days, so I've /dev/nulled that too except for
when I need it.

So yeah, it's a fairly epidemic problem. I probably notice more than most
because I do daily remote-incremental backups over a DSL line, so an
incremental backup of a few hundred MB definitely gets noticed. Sometimes
it's OK, sometimes it's not.


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100512174327.GB17540@club.cc.cmu.edu">http://lists.debian.org/20100512174327.GB17540@club.cc.cmu.edu
 
Old 05-12-2010, 10:22 PM
Alejandro Exojo
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

El Miércoles, 12 de Mayo de 2010, Frederik Schwarzer escribió:
> If you want to have these images there in case they are needed, keep the
> folder, if disk space is more important than loading speed, remove it
> regularily.

Of course, that's what I do. But a plain user should not be asked to regularly
remove it, should he? It probably should be handled like the trash folder,
which IIRC has a size/time limit.

--
Alex (a.k.a. suy) | GPG ID 0x0B8B0BC2
http://barnacity.net/ | http://disperso.net


--
To UNSUBSCRIBE, email to debian-kde-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201005130022.21523.suy@badopi.org">http://lists.debian.org/201005130022.21523.suy@badopi.org
 
Old 05-12-2010, 10:31 PM
"Boyd Stephen Smith Jr."
 
Default KDE 4.4.3 upgrade eats 141 MB of /home

On Wednesday 12 May 2010 17:22:21 Alejandro Exojo wrote:
> El Miércoles, 12 de Mayo de 2010, Frederik Schwarzer escribió:
> > If you want to have these images there in case they are needed, keep the
> > folder, if disk space is more important than loading speed, remove it
> > regularily.
>
> Of course, that's what I do. But a plain user should not be asked to
> regularly remove it, should he? It probably should be handled like the
> trash folder, which IIRC has a size/time limit.

I think "sweeper" (?) from kdeutils is supposed to handle this, right?
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@iguanasuicide.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.net/ \_/
 

Thread Tools




All times are GMT. The time now is 05:50 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org