FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 01-25-2011, 02:53 PM
Miklos Vajna
 
Default Improving dm-mirror as a final year project

Hi,

I got the possibility to work on dm-mirror as a final year project at
ULX, the Hungarian distributor of Red Hat.

Get my feet wet, I created two small patches:

1) dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case.

2) dm-mirror: allow setting the default mirror

These can help in case one data leg of a mirror is a remote (iSCSI) one,
so the default RR aproach is not optimal for reading. (One may set the
ios count to 0, set the default mirror to the local one, and that will
cause a read speedup.)

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be able to
do so - to get them reviewed.

So the final year project's aim is to improve "the fault tolerance and
performance of dm-mirror". We (I and my mentors) have two ideas in that
area, not counting the above patches:

1) Make the currently hardwired RR read approach modular, that would
allow implementing for example a weighted RR algorithm. (In case one
disk is two times faster than the other one, etc.)

2) From our experiments, it seems that in case the dm-mirror looses one
of its legs and there is a write to the mirror, it gets converted to a
linear volume. It would be nice (not sure how easy) to use the mirror
log to mark the dirty blocks, so that the volume would not be converted
to a linear one: once the other leg is back, it could be updated based
on the mirror log.

The question: what do you (who have much more experience with dm-mirror
than me) think - are these reasonable goals? If not, what would you
improve/change/add/remove to the above list?

Thanks,

Miklos

PS: I'm not subscribed, please keep me in CC.
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-26-2011, 12:58 AM
Miklos Vajna
 
Default Improving dm-mirror as a final year project

Hi,

I got the possibility to work on dm-mirror as a final year project at
ULX, the Hungarian distributor of Red Hat.

Get my feet wet, I created two small patches:

1) dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case.

2) dm-mirror: allow setting the default mirror

These can help in case one data leg of a mirror is a remote (iSCSI) one,
so the default RR aproach is not optimal for reading. (One may set the
ios count to 0, set the default mirror to the local one, and that will
cause a read speedup.)

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be able to
do so - to get them reviewed.

So the final year project's aim is to improve "the fault tolerance and
performance of dm-mirror". We (I and my mentors) have two ideas in that
area, not counting the above patches:

1) Make the currently hardwired RR read approach modular, that would
allow implementing for example a weighted RR algorithm. (In case one
disk is two times faster than the other one, etc.)

2) From our experiments, it seems that in case the dm-mirror looses one
of its legs and there is a write to the mirror, it gets converted to a
linear volume. It would be nice (not sure how easy) to use the mirror
log to mark the dirty blocks, so that the volume would not be converted
to a linear one: once the other leg is back, it could be updated based
on the mirror log.

The question: what do you (who have much more experience with dm-mirror
than me) think - are these reasonable goals? If not, what would you
improve/change/add/remove to the above list?

Thanks,

Miklos

PS: Sorry if this is a duplicate, first I did not subscribe to the list.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-26-2011, 04:24 PM
nishant mungse
 
Default Improving dm-mirror as a final year project

On Tue, Jan 25, 2011 at 10:53 AM, Miklos Vajna <vmiklos@frugalware.org> wrote:

Hi,



I got the possibility to work on dm-mirror as a final year project at

ULX, the Hungarian distributor of Red Hat.



Get my feet wet, I created two small patches:



1) dm-mirror: allow setting ios count to 0



Always read from the default_mirror in that case.



2) dm-mirror: allow setting the default mirror



These can help in case one data leg of a mirror is a remote (iSCSI) one,

so the default RR aproach is not optimal for reading. (One may set the

ios count to 0, set the default mirror to the local one, and that will

cause a read speedup.)



I do not yet have the right to send those patches (I do this in

university time, so the copyright is not mine), but I hope to be able to

do so - to get them reviewed.



So the final year project's aim is to improve "the fault tolerance and

performance of dm-mirror". We (I and my mentors) have two ideas in that

area, not counting the above patches:



1) Make the currently hardwired RR read approach modular, that would

allow implementing for example a weighted RR algorithm. (In case one

disk is two times faster than the other one, etc.)



2) From our experiments, it seems that in case the dm-mirror looses one

of its legs and there is a write to the mirror, it gets converted to a

linear volume. It would be nice (not sure how easy) to use the mirror

log to mark the dirty blocks, so that the volume would not be converted

to a linear one: once the other leg is back, it could be updated based

on the mirror log.



The question: what do you (who have much more experience with dm-mirror

than me) think - are these reasonable goals? If not, what would you

improve/change/add/remove to the above list?



Thanks,



Miklos



PS: I'm not subscribed, please keep me in CC.


--

dm-devel mailing list

dm-devel@redhat.com

https://www.redhat.com/mailman/listinfo/dm-devel


hello all,
There is one doubt regarding the dm-raid1. As far as i know, in dm-raid1 the data is written parallelly* on all the mirrors of mirrorset and if any of the mirror fails to write the data then dm-mirror adds this mirror to fail list by increasing the error count in "fail mirror" function in dm-raid1.

Actually my doubt is where this error count is decremented? i.e after kcpyd or before and where exactly this error count is decremented?


Regards,
Nishant.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-26-2011, 07:04 PM
Malahal Naineni
 
Default Improving dm-mirror as a final year project

nishant mungse [nishantmungse@gmail.com] wrote:
>
> hello all,
> There is one doubt regarding the dm-raid1. As far as i know, in dm-raid1
> the data is written parallelly� on all the mirrors of mirrorset and if any
> of the mirror fails to write the data then dm-mirror adds this mirror to
> fail list by increasing the error count in "fail mirror" function in
> dm-raid1.
> Actually my doubt is where this error count is decremented? i.e after
> kcpyd or before and where exactly this error count is decremented?

There is no actual 'fail list' of mirror legs. The error_count is never
decremented. The 'error_count' is only used to tell if the mirror leg
has encountered errors. The only way to clear that counter is
re-loading the table!

See the comment:

/*
* error_count is used for nothing more than a
* simple way to tell if a device has encountered
* errors.
*/
atomic_inc(&m->error_count);

Hope that helps,
Malahal.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-27-2011, 07:28 AM
nishant mungse
 
Default Improving dm-mirror as a final year project

On Wed, Jan 26, 2011 at 3:04 PM, Malahal Naineni <malahal@us.ibm.com> wrote:


nishant mungse [nishantmungse@gmail.com] wrote:

>

> Â* Â*hello all,

> Â* Â*There is one doubt regarding the dm-raid1. As far as i know, in dm-raid1

> Â* Â*the data is written parallelly� on all the mirrors of mirrorset and if any

> Â* Â*of the mirror fails to write the data then dm-mirror adds this mirror to

> Â* Â*fail list by increasing the error count in "fail mirror" function in

> Â* Â*dm-raid1.

> Â* Â*Actually my doubt is where this error count is decremented? i.e after

> Â* Â*kcpyd or before and where exactly this error count is decremented?



There is no actual 'fail list' of mirror legs. The error_count is never

decremented. The 'error_count' is only used to tell if the mirror leg

has encountered errors. The only way to clear that counter is

re-loading the table!



See the comment:



Â* Â* Â* Â*/*

Â* Â* Â* Â* * error_count is used for nothing more than a

Â* Â* Â* Â* * simple way to tell if a device has encountered

Â* Â* Â* Â* * errors.

Â* Â* Â* Â* */

Â* Â* Â* Â*atomic_inc(&m->error_count);



Hope that helps,

Malahal.



--

dm-devel mailing list

dm-devel@redhat.com

https://www.redhat.com/mailman/listinfo/dm-devel
Hi Malahal

Thanks for reply. I am not able to understand whatÂ* this "INIT_WORK(&ms->kmirrord_work, do_mirror)" function does. ItÂ* is written in mirror_ctr of dm-raid1. What i think is whenever a bio comes dm_mirror will be called if that is case then why this is written in mirror_ctr. Please answer this.


Regards,
Nishant.





--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-01-2011, 03:12 PM
Jonathan Brassow
 
Default Improving dm-mirror as a final year project

On Jan 25, 2011, at 9:53 AM, Miklos Vajna wrote:


Hi,

I got the possibility to work on dm-mirror as a final year project at
ULX, the Hungarian distributor of Red Hat.

Get my feet wet, I created two small patches:

1) dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case.

2) dm-mirror: allow setting the default mirror

These can help in case one data leg of a mirror is a remote (iSCSI)
one,

so the default RR aproach is not optimal for reading. (One may set the
ios count to 0, set the default mirror to the local one, and that will
cause a read speedup.)

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be
able to

do so - to get them reviewed.

So the final year project's aim is to improve "the fault tolerance and
performance of dm-mirror". We (I and my mentors) have two ideas in
that

area, not counting the above patches:

1) Make the currently hardwired RR read approach modular, that would
allow implementing for example a weighted RR algorithm. (In case one
disk is two times faster than the other one, etc.)

2) From our experiments, it seems that in case the dm-mirror looses
one

of its legs and there is a write to the mirror, it gets converted to a
linear volume. It would be nice (not sure how easy) to use the mirror
log to mark the dirty blocks, so that the volume would not be
converted

to a linear one: once the other leg is back, it could be updated based
on the mirror log.

The question: what do you (who have much more experience with dm-
mirror

than me) think - are these reasonable goals? If not, what would you
improve/change/add/remove to the above list?


Would you consider working on the recently added dm-raid.c? It is an
attempt to access the MD personalities through device-mapper. As
such, RAID456 (the initial drop) will be available - in addition to
RAID1. There is much to be done in this area - large projects as well
as small - encompassing performance issues, metadata layout, testing,
etc... There is also a lot of attention being paid to this area.


I think device-mapper mirroring will be used for a while, but it will
likely become deprecated.


brassow

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-09-2011, 04:13 PM
Miklos Vajna
 
Default Improving dm-mirror as a final year project

Hi Jonathan,

On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> Would you consider working on the recently added dm-raid.c? It is an
> attempt to access the MD personalities through device-mapper. As
> such, RAID456 (the initial drop) will be available - in addition to
> RAID1. There is much to be done in this area - large projects as well
> as small - encompassing performance issues, metadata layout, testing,
> etc... There is also a lot of attention being paid to this area.
>
> I think device-mapper mirroring will be used for a while, but it will
> likely become deprecated.

Sure, that sounds interesting. The first part of my MSc thesis should be
done till 2011.05.13 (14 weeks), I have about 2 workdays / week for the
project. Of course the actual development time is a bit less, since I
have to produce paper from this (about 30 pages for the first part).

(The second part should be done in Q3-Q4, when I'll have a bit more
dedicated time.)

Do you have a project idea for such a timeframe?

My mentors are asked: is dm-raid supported in a RHEL version?

As far as I see it was added in commit
9d09e663d5502c46f2d9481c04c1087e1c2da698 after 2.6.37, so it's not part
of RHEL (I created my dm-mirror patches on RHEL5) - and it's a pro for
them if I can do my development based on a RHEL kernel, what they use.

So - given such a timeframe, is it better to develop dm-mirror, which is
fairly stable or is it possible to develop dm-raid, even if it's quite
new?

Thanks,

Miklos

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-14-2011, 04:33 PM
Miklos Vajna
 
Default Improving dm-mirror as a final year project

On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> > I do not yet have the right to send those patches (I do this in
> > university time, so the copyright is not mine), but I hope to be
> > able to
> > do so - to get them reviewed.

Hi,

I'm attaching the two patches. I tested these on RHEL5, but I don't
think there are major changes in newer versions, either.

If that's the only problem, I can test it on latest linux-2.6.git.

Any feedback is welcome.

Thanks,

Miklos
>From 2a5aaeef1caed0019abdce92fdb6e6a4413677f8 Mon Sep 17 00:00:00 2001
From: Miklos Vajna <vmiklos@ulx.hu>
Date: Sat, 25 Sep 2010 14:56:28 +0200
Subject: [PATCH] dm-mirror: allow setting ios count to 0

Always read from the default_mirror in that case. This can help in case
the default_mirror is way faster than the other data leg.

Signed-off-by: Miklos Vajna <vmiklos@ulx.hu>
---
drivers/md/dm-raid1.c | 15 +++++++++++++--
1 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 23c1d65..318899e 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -849,6 +849,15 @@ static struct mirror *choose_mirror(struct mirror_set *ms)
* the first we tried, so we know when we're done.
*/
ret = start_mirror = ms->read_mirror;
+
+ /*
+ * If MIN_READS is zero, then always use the default one.
+ */
+ if (!atomic_read(&ms->rr_ios_set)) {
+ ret = ms->default_mirror;
+ goto use_mirror;
+ }
+
do {
if (likely(!atomic_read(&ret->error_count) &&
!atomic_dec_and_test(&ms->rr_ios)))
@@ -1848,8 +1857,10 @@ static int mirror_message(struct dm_target *ti, unsigned argc, char **argv)

if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
rr_ios_set < 2) {
- DMERR("Round robin read ios have to be > 1");
- return -EINVAL;
+ if (rr_ios_set != 0) {
+ DMERR("Round robin read ios have to be > 1 or 0");
+ return -EINVAL;
+ }
}

md = dm_table_get_md(ti->table);
--
1.5.5.6

>From 02c399b2171dec66d7c014caa1efbc58c0ba8ee1 Mon Sep 17 00:00:00 2001
From: Miklos Vajna <vmiklos@ulx.hu>
Date: Sat, 25 Sep 2010 17:16:11 +0200
Subject: [PATCH] dm-mirror: allow setting the default mirror

This can help in case an other data leg is faster than the default one
and the ios count is set to zero.

Signed-off-by: Miklos Vajna <vmiklos@ulx.hu>
---
drivers/md/dm-raid1.c | 54 +++++++++++++++++++++++++++++++-----------------
1 files changed, 35 insertions(+), 19 deletions(-)

diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 318899e..b746b0c 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -1845,31 +1845,47 @@ static void mirror_resume(struct dm_target *ti)
/* Set round robin ios via message. */
static int mirror_message(struct dm_target *ti, unsigned argc, char **argv)
{
- unsigned rr_ios_set;
struct mirror_set *ms = ti->private;
struct mapped_device *md;

- if (argc != 4 ||
- strncmp(argv[0], "io_balance", strlen(argv[0])) ||
- strncmp(argv[1], "round_robin", strlen(argv[1])) ||
- strncmp(argv[2], "ios", strlen(argv[2])))
- return -EINVAL;
-
- if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
- rr_ios_set < 2) {
- if (rr_ios_set != 0) {
- DMERR("Round robin read ios have to be > 1 or 0");
- return -EINVAL;
+ if (argc == 4 &&
+ !strncmp(argv[0], "io_balance", strlen(argv[0])) &&
+ !strncmp(argv[1], "round_robin", strlen(argv[1])) &&
+ !strncmp(argv[2], "ios", strlen(argv[2]))) {
+ unsigned rr_ios_set;
+ if (sscanf(argv[3], "%u", &rr_ios_set) != 1 ||
+ rr_ios_set < 2) {
+ if (rr_ios_set != 0) {
+ DMERR("Round robin read ios have to be > 1 or 0");
+ return -EINVAL;
+ }
}
+
+ md = dm_table_get_md(ti->table);
+ DMINFO("Setting round robin read ios for "%s" to %u",
+ dm_device_name(md), rr_ios_set);
+ dm_put(md);
+ atomic_set(&ms->rr_ios_set, rr_ios_set);
+ atomic_set(&ms->rr_ios, rr_ios_set);
+ return 0;
}

- md = dm_table_get_md(ti->table);
- DMINFO("Setting round robin read ios for "%s" to %u",
- dm_device_name(md), rr_ios_set);
- dm_put(md);
- atomic_set(&ms->rr_ios_set, rr_ios_set);
- atomic_set(&ms->rr_ios, rr_ios_set);
- return 0;
+ if (argc == 3 &&
+ !strncmp(argv[0], "io_balance", strlen(argv[0])) &&
+ !strncmp(argv[1], "default", strlen(argv[1]))) {
+ unsigned int m;
+ for (m = 0; m < ms->nr_mirrors; m++)
+ if (!strncmp(argv[2], ms->mirror[m].dev->name, strlen(argv[2]))) {
+ ms->default_mirror = &ms->mirror[m];
+ md = dm_table_get_md(ti->table);
+ DMINFO("Setting default device for "%s" to "%s"",
+ dm_device_name(md), argv[2]);
+ dm_put(md);
+ return 0;
+ }
+ }
+
+ return -EINVAL;
}

/*
--
1.5.5.6

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-14-2011, 08:31 PM
Jonathan Brassow
 
Default Improving dm-mirror as a final year project

On Feb 14, 2011, at 11:33 AM, Miklos Vajna wrote:

On Tue, Feb 01, 2011 at 10:12:36AM -0600, Jonathan Brassow <jbrassow@redhat.com
> wrote:

I do not yet have the right to send those patches (I do this in
university time, so the copyright is not mine), but I hope to be
able to
do so - to get them reviewed.


Hi,

I'm attaching the two patches. I tested these on RHEL5, but I don't
think there are major changes in newer versions, either.

If that's the only problem, I can test it on latest linux-2.6.git.

Any feedback is welcome.


Thanks for the patches. I've seen the first one before (slightly
different) - I'll discuss it with others whether to include in rhel5.
There is no read-balancing in rhel6/upstream.


The second patch addresses which device should be primary. This can
be done when creating the mirror. I'm not sure how much benefit there
is to doing this additional step. Most people will access dm mirrors
through LVM - not through the dm message interface. If it makes sense
upstream - and you can argue for it - though, I would consider it.


Wether changes are going into rhel5 or rhel6, we still like it when
they go upstream first. We generally don't like feature inversion.


If you have any interest in dm-raid, these are some of the things that
need to be done:
1) Definition of new MD superblock: Some of this is started, and I've
got a working version, but I'm sure there are pieces missing related
to offsets that must be tracked for RAID type conversion, etc.
2) Bitmap work: The bitmap keeps track of which areas of the array
are being written. Right now, I take all the bitmap code "as-is".
There are a number of things in this area to be improved. Firstly, we
don't necessarily need all the fields in the bitmap superblock -
perhaps this could be streamlined and added to the new MD superblock.
Secondly, things are way too slow. I get a 10x slowdown when using a
bitmap with RAID1 through device-mapper. This could be due to the
region-size chosen, the bitmap being at a stupid offset, or something
else. This problem could be solved by trial-and-error or through
profiling and reason... seems like a great small project.
3) Conversion code: New device-mapper targets (very simple small
ones) must be written to engage the MD RAID conversion code (like when
you change RAID4 to RAID5, for example)

4) Failure testing
5) LVM code: to handle creation of RAID devices
6) dmeventd code: to handle device failures

This is an incomplete list, but does have a variety of complexity and
duration.


brassow

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-15-2011, 11:52 AM
Miklos Vajna
 
Default Improving dm-mirror as a final year project

On Mon, Feb 14, 2011 at 03:31:00PM -0600, Jonathan Brassow <jbrassow@redhat.com> wrote:
> Thanks for the patches. I've seen the first one before (slightly
> different) - I'll discuss it with others whether to include in rhel5.
> There is no read-balancing in rhel6/upstream.

Oh, do I read the code correctly that rhel6/upstream always reads from
the first mirror and switches only in case there is a read failure?

> The second patch addresses which device should be primary. This can
> be done when creating the mirror. I'm not sure how much benefit there
> is to doing this additional step. Most people will access dm mirrors
> through LVM - not through the dm message interface. If it makes sense
> upstream - and you can argue for it - though, I would consider it.

Is there such a messaging interface for lvm as well? I choosed this way
as in this case I did not have to alter the metadata.

One useful use case I can imagine is when both data legs of the mirror
are provided by iscsi and the administrator does not realise what is the
faster leg and her bad decision is found out only after there is some
data on the mirror.

My patch allows one to just set the first mirror in that case, without
saving data, recreating the mirror and restoring data. (Unless I missed
some other neat trick on how to do so.)

> Wether changes are going into rhel5 or rhel6, we still like it when
> they go upstream first. We generally don't like feature inversion.

Sure - I was not aware at all that the round robin part of the code is
RHEL5-specific.

> If you have any interest in dm-raid, these are some of the things that
> need to be done:

Thanks for the list - I must admit that some of the points are Chinese
to me; I'm not that familiar with the codebase, just with the basic LVM
commands anc concepts.

> 1) Definition of new MD superblock: Some of this is started, and I've
> got a working version, but I'm sure there are pieces missing related
> to offsets that must be tracked for RAID type conversion, etc.
> 2) Bitmap work: The bitmap keeps track of which areas of the array
> are being written. Right now, I take all the bitmap code "as-is".
> There are a number of things in this area to be improved. Firstly, we
> don't necessarily need all the fields in the bitmap superblock -
> perhaps this could be streamlined and added to the new MD superblock.
> Secondly, things are way too slow. I get a 10x slowdown when using a
> bitmap with RAID1 through device-mapper. This could be due to the
> region-size chosen, the bitmap being at a stupid offset, or something
> else. This problem could be solved by trial-and-error or through
> profiling and reason... seems like a great small project.
> 3) Conversion code: New device-mapper targets (very simple small
> ones) must be written to engage the MD RAID conversion code (like when
> you change RAID4 to RAID5, for example)
> 4) Failure testing
> 5) LVM code: to handle creation of RAID devices
> 6) dmeventd code: to handle device failures

Before choosing from this list:

I first have to evaluate the current status of dm-raid so that we can
decide with my mentors if the topic of my thesis should be dm-mirror or
dm-raid (ie. if dm-raid is mature enough that I can write a thesis about
it). Where is the newest version of dm-raid.c? I saw the upstream kernel
has a single commit from this January, but I guess the rawhide / rhel
kernel contained this earlier - maybe there is a newer version than
upstream somewhere?

Also, is there any documentation on dm-raid? Google found
http://www.linux-archive.org/device-mapper-development/454656-dm-raid-wrapper-target-md-raid456.html
but maybe there is now a better way to create raid4 than using
gime_raid.pl?

And a last question: is support for raid1 a planned feature? I think
that would be interesting as well. (If dm-raid is going to replace
dm-mirror in the long run.)

Thanks!

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 08:09 PM.

VBulletin, Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org