FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 06-30-2012, 10:44 PM
Vincent Danjean
 
Default OpenCL in Debian

Hi,

During the Debian sprint at Grenoble, we talk about the OpenCL
situation in Debian and I decided to wrote a summary. The sprint
organizer, Jerome Kieffer, tells me I will include it in the
report of the sprint. However, I would like to also put it here
so that more people can react about it.
Note that, due to the fact that some software have been written
very recently (I'm thinking to ocl-icd where I'm also upstream)
and to the fact that some bug in package dependencies have been
discovered and fixed after I wrote this document (ie very recently),
the situation of OpenCL packages in wheezy will mainly depends on
the decisions of the release team to allow or not freeze exceptions.

Summary of the situation of OpenCL in Debian
********************************************

OpenCL goals
============

>From wikipedia:
Open Computing Language (OpenCL) is a framework for writing programs that
execute across heterogeneous platforms consisting of central processing unit
(CPUs), graphics processing unit (GPUs), and other processors.
OpenCL includes a language (based on C99) for writing kernels (functions
that execute on OpenCL devices), plus application programming interfaces
(APIs) that are used to define and then control the platforms.
[...]
OpenCL is an open standard maintained by the non-profit technology
consortium Khronos Group. It has been adopted by Intel, Advanced Micro
Devices, Nvidia, and ARM Holdings.

What is important for Debian is that:
- OpenCL is a standard maintained by the Khronos Group
- OpenCL is a language (for target platforms) *and* an API.
- there will be several implementations (we can hope that, at least one for
processors will be free)
- API of OpenCL is design to support several implementations at the same time
- API of OpenCL offert the possibility to choose the used platform (and OpenCL
vendor) at runtime (instead of compilation/link-time as this is the case for
MPI)
- API of OpenCL offers the possibility to use vendor extensions, looking for
address function at runtime (here again by opposition to compile/runtime)

Mechanisms used to archieve all these goals
===========================================

An application using OpenCL can compile with, and link directly at, a specific
implementation (as in the MPI world). However, doing this will forbid most of
the runtime choices presented in the previous paragraph.

The classical situation for OpenCL is that:
- each vendors (ie OpenCL implementor) must provide:
- an OpenCL ICD (Installable Client Driver) as a shared library with
some convention/features/symbols (see later)
- the path of this library in a vendor-specific text file in
/etc/OpenCL/vendors/ directory
- the application is compiled with
- some OpenCL headers. This can be:
- the official Kronos OpenCL headers (all official features and extensions,
no private vendor extension)
- some vendors headers, generally based on the previous one but with
some extensions (functions, constants, ...) added
- OpenCL ICD Loader which offer the OpenCL API to the applications. The
ICD Loader is mainly a call dispatcher. Kronos group provides (nonfree)
sources of one to its members.
- most vendors (amd, intel, nvidia at least) provide a ICD Loader
- ocl-icd provides a free one after some reverse engeenering of the
previously cited ones.

ICD Loaders and ICD
===================
In therory (see below), any ICD Loader should be able to load and use any ICD.
An ICD must implement some mandatory functions. One of these functions allows
the ICD Loader to get the address of a structure which contains the address of
all ICD implementation of OpenCL functions.
Writing such an ICD Loader is easy once you know the order of the functions
in the structure (list and order that the Khronos group restrict to its members
and that the ocl-icd projet reverse engeenered)
The Khronos group ensures that any functions added to this structure will
never be removed (so that their won't be incompatibility between to ICD
Loaders).

In pratice, we can really use the ICD Loader from Intel to use the NVidia ICD.
But, there is no support for the ICD to declare which version of the structure
it provides. An ICD can offers (through the structure) more functions. It
happens when the ICD provides an OpenCL version greater than the one provided
by the ICD Loader. This is not really a problem but the an OpenCL application
using the more recent OpenCL version will not be able to link to the ICD
Loader. It will then fail at link time.
But there is also the opposite: the ICD Loader supports a greater version of
OpenCL than the ICD. In this case, an OpenCL application using features from
the new version will call the ICD Loader that will call the "function" at the
address present in the memory after the end of the (old) structure provider by
the ICD. This generally leads to a segfault.
The ICD Loader cannot do anything about that: it does not have the
information. Even if ICDs declare a supported OpenCL version, it is the
OpenCL version for which the support is full. At the time of writing, the Intel
ICD declare to support OpenCL 1.1 but provides to the ICD Loader the structure
of OpenCL 1.2 *with* some address of functions specific to OpenCL 1.2 (but not
all of them, hence the no declaration of OpenCL 1.2 support).
This situation is also a problem for application (PyCL ?) that detect at
compilation time which version of OpenCL is supported. They will get the
OpenCL version of the ICD Loader, but not the one of the ICD itself that
will be loaded at runtime.

An ICD not implementing one function of the structure puts a NULL pointer.
But non-free ICD Loaders (cnot ocl-icd for now) do not do any checks for
non-NULL value before dispatching the call. So it is again easy to get segfault
with some combination of program/ICD.


About compatibility, each ICD must also implement the support for the OpenCL
*language* (for the code dedicated to the target platforms). Of course, as the
norm is not complete, or not fully implemented or implemented with extension,
the portability of OpenCL code is not as good as one can hope. But there is
little to do here with respect to Debian packaging.

Current packaging situation
===========================
Brief overview of sources packages/OpenCL implementations:
* AMD and NVidia OpenCL implementation are packaged into the non-free section.
* Intel is not packaged. It is not redistribuable, so its packaging would
involve a 'installer' package.
Intel ICD Loader support OpenCL 1.2
Intel ICD support OpenCL 1.1 and some functions of OpenCL 1.2
* Khronos OpenCL headers 1.2 are packaged in main
* Ocl-icd project is packaged in main
* Pocl, a free implementation of OpenCL, will be packaged in main. Its state
does not seem yet good enough to be able to run non simple OpenCL
applications. This should change in the future.

Overview of binary packages currently in sid (and probably testing) on amd64:
[AMD]
* amd-libopencl1: AMD ICD Loader (support OpenCL 1.2)
install libOpenCL.so.1
shared library
soname libOpenCL.so.1
symbols versionned (with OpenCL version, but no conflicts on symbols)
install libOpenCL.so
symlink to libOpenCL.so.1
Multi-Arch: same
Provides: libopencl1
Conflicts: amd-app, nvidia-libopencl1
=> should Conflicts and Provides libopencl1. See #679038 (fixed in unstable)
* amd-opencl-icd: AMD ICD (support OpenCL 1.2)
Multi-Arch: same
Depends: libopencl1
Enhances: libopencl1
Conflicts: amd-app
* amd-opencl-dev: AMD OpenCL development
Metapackage
Multi-Arch: same
Depends: amd-libopencl1 | libopencl1, opencl-headers (>= 1.2)
* amd-clinfo:
No Multi-Arch
Depends: amd-libopencl1
=> should depends on the virtual package libopencl1. See #679025 (fixed in unstable)

[NVidia]
* nvidia-libopencl1: NVidia ICD Loader (support OpenCL 1.1)
install libOpenCL.so.1.0.0
shared library
soname libOpenCL.so.1
symbols not versionned
install libOpenCL.so.1
symlink to libOpenCL.so.1.0.0
install libOpenCL.so
symlink to libOpenCL.so.1
Multi-Arch: same
Provides: libopencl1
Recommends: opencl-icd | nvidia-opencl-icd
Conflicts: nvidia-libopencl1-dev
=> should Conflicts and Provides libopencl1. See #679047 (fixed in unstable)
* nvidia-libopencl1-ia32: 32-bit NVidia ICD Loader (support OpenCL 1.1)
install lib32/libOpenCL.so.1.0.0
shared library
soname libOpenCL.so.1
symbols not versionned
install lib32/libOpenCL.so.1
symlink to libOpenCL.so.1.0.0
install lib32/libOpenCL.so
symlink to libOpenCL.so.1
No Multi-Arch
Provides: lib32opencl1
Recommends: opencl-icd | nvidia-opencl-icd-ia32
=> the Recommends to opencl-icd seems wrong to me
=> this package should disappear (replaced by nvidia-libopencl1:i386)
* nvidia-opencl-common: common files for NVidia ICD (support OpenCL 1.1)
Multi-Arch: foreign
Recommends: nvidia-opencl-icd | nvidia-opencl-icd-ia32
* nvidia-opencl-icd: NVidia ICD (support OpenCL 1.1)
Multi-Arch: same
Provides: opencl-icd
Depends: nvidia-opencl-common, libopencl1
Enhances: libopencl1
* nvidia-opencl-icd-ia32: 32-bit NVidia ICD (support OpenCL 1.1)
No Multi-Arch
Provides: opencl-icd
Depends: nvidia-opencl-common, lib32opencl1
Enhances: lib32opencl1
=> the Provides opencl-icd seems wrong to me
=> this package should disappear (replaced by nvidia-opencl-icd:i386)
* nvidia-opencl-dev: NVidia OpenCL development
Metapackage
No Multi-Arch
Provides: opencl-dev
Depends: nvidia-libopencl1 | libopencl1, opencl-headers (>= 1.1)

[ocl-icd] (version 1.3-2 currently in NEW or in my own repo)
* ocl-icd-libopencl1: ocl-icd ICD Loader (support OpenCL 1.2)
install libOpenCL.so.1.0.0
shared library
soname libOpenCL.so.1
symbols versionned (with OpenCL version, but no conflicts on symbols)
install libOpenCL.so.1
symlink to libOpenCL.so.1.0.0
install libOpenCL.so
symlink to libOpenCL.so.1.0.0
Multi-Arch: same
Provides: libopencl1
Conflicts: libopencl1, amd-app, nvidia-libopencl1-dev
Replaces: libopencl1, amd-app, nvidia-libopencl1-dev
Note: amd-app, nvidia-libopencl1-dev seems to be old/external packages
providing some of these files
Suggests: opencl-icd
Note: would be promoted to a recommends once a free OpenCL implementation
will be available
* ocl-icd-opencl-dev: OpenCL development
install OpenCL.pc
pkg-config file for OpenCL
Multi-Arch: same
Provides: opencl-dev
Depends: opencl-headers (>= 1.2), libopencl1
Conflicts: amd-app, opencl-dev
Replaces: amd-app, opencl-dev
* ocl-icd-dev: devel package to develop an ICD
install ocl_icd.h
headers with the structure with all functions that an ICD must provide
install some examples (skeleton of a ICD source and linker map)
install ocl-icd.pc
No Multi-Arch

[Intel OpenCL SDK]: no public Debian packages (software not redistributable)
* ICD Loader (support OpenCL 1.2): can be installed in Multi-Arch paths
install libOpenCL.so
shared library
soname libOpenCL.so
symbols not versionned
=> problem with wrong soname
=> can still be used by programs that require libOpenCL.so.1 by creating
a symlink (the dynamic linker do not check the soname)
* ICD (support OpenCL 1.1 + some 1.2 functions)
* OpenCL headers
based on the Khronos ones. Intel's one seem a little bit outdated but
include some constants/... that seems to be Intel specific...


Virtual packages:
* opencl-icd: existing virtual package for OpenCL ICD
ensure an OpenCL ICD is installed
a package providing opencl-icd should (must?):
- Provides: opencl-icd
- Enhances: libopencl1
* opencl-dev: existing virtual package dor OpenCL development
ensure headers and libraries are installed
a package providing opencl-dev should (must?):
- Provides: opencl-dev
- install OpenCL headers
can be done with a (versionned) dependency on the opencl-headers package
- Depends: libopencl1
- Recommends: libgl1-mesa-dev | libgl-dev
GL headers are often required when compiling OpenCL program
=> not done in amd-opencl-dev
* libopencl1: existing virtual package for OpenCL ICD Loaders
ensure an OpenCL ICD Loader is installed
a package providing libopencl1 should (must?):
- Conflicts/Replaces/Provides: libopencl1
http://www.debian.org/doc/debian-policy/ch-relationships.html#s7.6.2
- Recommends: opencl-icd
- install libOpenCL.so.1 in library search path
- install libOpenCL.so in library search path
NOTE: this last one must not be in a -dev package if we want to
support binaries compiled with the Intel SDK
- libOpenCL.so.1 must have libOpenCL.so.1 as soname
[proposed, not right for the intel one]
- libOpenCL.so.1 must have its symbols versionned
[proposed, not right for the intel and nvidia ones]
* libopencl1.1: proposed virtual package
a package providing libopencl1.1 must:
- provides libopencl1
- implements and dispatch all functions from OpenCL 1.1 API
* libopencl1.2: proposed virtual package
a package providing libopencl1.2 must:
- provides libopencl1
- implements and dispatch all functions from OpenCL 1.2 API
NOTE: libopencl1.0 is not proposed as a virtual package:
- the 1.0 API is not really designed to work with ICD Loaders
- most (all?) OpenCL implementations target at least OpenCL 1.1


A few remarks:
* due to the soname of Intel (libOpenCL.so instead of libOpenCL.so.1), it is
better to install the libOpenCL.so link in the libopencl1 packages instead
of installing it in the dev packages
* is it really necessary to keep the three -dev packages?
+ ocl-icd-opencl-dev is free and provides a OpenCL.pc file
+ amd-opencl-dev is a metapackage that install by default the amd-libopencl1
that itself install by default the amd-opencl-icd
+ same for nvidia-opencl-dev
* if we keep the three -dev packages (to ease the installation of the
full AMD and NVidia SDK), do we want to rename ocl-icd-opencl-dev
into opencl-dev (ie use the virtual package name as the binary
package name of the free alternative)
* do we want to keep the various *-libopencl1 (ie ICD loaders) packages in
Debian?
ocl-icd-libopencl1 provides a free alternative
{amd,nvidia}-libopencl1 will be useful:
- when ocl-icd-libopencl1 is not yet updated to the latest version
where as {amd,nvidia}-libopencl1 are
- if {amd,nvidia}-libopencl1 start to implement non standard things
- to ease the reverse engeenering required to improve ocl-icd-libopencl1
when non-free ICD loader implements more functions
* at least, here is a summary of problems/warning for different combinaison of
ICD *Loaders* used at compile-time (CT) and at run-time (RT)
(problems with ICD compatibility themselves are not taken into account here)

| | | | |
RT | ocl-icd | AMD | NVidia | Intel |
| | | | |
CT | | | | |
--------+---------+-------+---------+-------+
| | | | |
ocl-icd | ok | ok | MVS | MVS |
| | | MS | MSO |
--------+---------+-------+---------+-------+
| | | | |
AMD | ok | ok | MVS | MVS |
| | | MS | MSO |
--------+---------+-------+---------+-------+
| | | | |
NVidia | ok | ok | ok | MSO |
| | | | |
--------+---------+-------+---------+-------+
| | | | |
Intel | BSO | BSO | BSO | ok |
| | | MS | |
--------+---------+-------+---------+-------+
MVS: missing version on symbol => a warning from the linker at program start
MS: missing symbols. The linker can fail to find some symbols (NVidia ICD
Loader still only support the 1.1 API)
BSO: the libOpenCL.so link must be present at runtime.
The requested soname is libOpenCL.so
The soname in the loaded library (filename) is libOpenCL.so.1
MSO: wrong soname:
The requested soname is libOpenCL.so.1
The soname in the loaded library (filename) is libOpenCL.so


Many thanks for readers that go until here.
Vincent


--
Vincent Danjean GPG key ID 0x9D025E87 vdanjean@debian.org
GPG key fingerprint: FC95 08A6 854D DB48 4B9A 8A94 0BF7 7867 9D02 5E87
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo: deb http://people.debian.org/~vdanjean/debian unstable main


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FEF8131.7080308@debian.org">http://lists.debian.org/4FEF8131.7080308@debian.org
 
Old 06-30-2012, 11:06 PM
Vincent Danjean
 
Default OpenCL in Debian

Le 01/07/2012 00:44, Vincent Danjean a écrit :
> Note that, due to the fact that some software have been written
> very recently (I'm thinking to ocl-icd where I'm also upstream)
> and to the fact that some bug in package dependencies have been
> discovered and fixed after I wrote this document (ie very recently),
> the situation of OpenCL packages in wheezy will mainly depends on
> the decisions of the release team to allow or not freeze exceptions.

I just see that freeze exceptions have been automatically granted
for packages currently in unstable. So perhaps most of the packaging
work has been done just in time.

Regards,
Vincent

--
Vincent Danjean GPG key ID 0x9D025E87 vdanjean@debian.org
GPG key fingerprint: FC95 08A6 854D DB48 4B9A 8A94 0BF7 7867 9D02 5E87
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo: deb http://people.debian.org/~vdanjean/debian unstable main


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FEF8692.8040200@debian.org">http://lists.debian.org/4FEF8692.8040200@debian.org
 
Old 07-10-2012, 07:40 AM
Vincent Danjean
 
Default OpenCL in Debian

Le 01/07/2012 00:44, Vincent Danjean a écrit :
> [Intel OpenCL SDK]: no public Debian packages (software not redistributable)
> * ICD Loader (support OpenCL 1.2): can be installed in Multi-Arch paths
> install libOpenCL.so
> shared library
> soname libOpenCL.so
> symbols not versionned
> => problem with wrong soname
> => can still be used by programs that require libOpenCL.so.1 by creating
> a symlink (the dynamic linker do not check the soname)

We can hope that Intel will fix its soname in its future SDK:
http://software.intel.com/en-us/forums/showthread.php?t=106234

Regards,
Vincent

--
Vincent Danjean GPG key ID 0x9D025E87 vdanjean@debian.org
GPG key fingerprint: FC95 08A6 854D DB48 4B9A 8A94 0BF7 7867 9D02 5E87
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo: deb http://people.debian.org/~vdanjean/debian unstable main


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FFBDC6E.6010902@free.fr">http://lists.debian.org/4FFBDC6E.6010902@free.fr
 
Old 07-10-2012, 07:45 PM
Tomasz Rybak
 
Default OpenCL in Debian

Dnia 2012-07-01, nie o godzinie 00:44 +0200, Vincent Danjean pisze:
[ cut ]
> * at least, here is a summary of problems/warning for different combinaison of
> ICD *Loaders* used at compile-time (CT) and at run-time (RT)
> (problems with ICD compatibility themselves are not taken into account here)
>
> | | | | |
> RT | ocl-icd | AMD | NVidia | Intel |
> | | | | |
> CT | | | | |
> --------+---------+-------+---------+-------+
> | | | | |
> ocl-icd | ok | ok | MVS | MVS |
> | | | MS | MSO |
> --------+---------+-------+---------+-------+
> | | | | |
> AMD | ok | ok | MVS | MVS |
> | | | MS | MSO |
> --------+---------+-------+---------+-------+
> | | | | |
> NVidia | ok | ok | ok | MSO |
> | | | | |
> --------+---------+-------+---------+-------+
> | | | | |
> Intel | BSO | BSO | BSO | ok |
> | | | MS | |
> --------+---------+-------+---------+-------+
> MVS: missing version on symbol => a warning from the linker at program start
> MS: missing symbols. The linker can fail to find some symbols (NVidia ICD
> Loader still only support the 1.1 API)
> BSO: the libOpenCL.so link must be present at runtime.
> The requested soname is libOpenCL.so
> The soname in the loaded library (filename) is libOpenCL.so.1
> MSO: wrong soname:
> The requested soname is libOpenCL.so.1
> The soname in the loaded library (filename) is libOpenCL.so
>
>
> Many thanks for readers that go until here.
> Vincent

Thanks for extensive summary. I have one dimension to add to
this table though: supported OpenCL versions.
Currently Debian contains opencl-headers in version 1.2.
Many programs when they encounter headers in particular
version assume that they can expect libraries in the same version.
This was problem with PyOpenCL when compiled with NVIDIA ICD loader:
PyOpenCL compilation failed because NVIDIA ICD supports version 1.1
while headers are in version 1.2. This resulted in PyOpenCL wanting
to use non-present functions.
Unfortunately I there is possibility to write:
opencl-headers (= 1.1) & nvidia-libopencl1 |
opencl-headers (= 1.2) & ( amd-libopencl1 | ocl-icl-libopencl1)
in Build-Depends ;-)

Best regards.

--
Tomasz Rybak <tomasz.rybak@post.pl> GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak
 
Old 07-10-2012, 10:19 PM
Vincent Danjean
 
Default OpenCL in Debian

Le 10/07/2012 21:45, Tomasz Rybak a écrit :
> Thanks for extensive summary. I have one dimension to add to
> this table though: supported OpenCL versions.
> Currently Debian contains opencl-headers in version 1.2.
> Many programs when they encounter headers in particular
> version assume that they can expect libraries in the same version.
> This was problem with PyOpenCL when compiled with NVIDIA ICD loader:
> PyOpenCL compilation failed because NVIDIA ICD supports version 1.1
> while headers are in version 1.2. This resulted in PyOpenCL wanting
> to use non-present functions.
> Unfortunately I there is possibility to write:
> opencl-headers (= 1.1) & nvidia-libopencl1 |
> opencl-headers (= 1.2) & ( amd-libopencl1 | ocl-icl-libopencl1)
> in Build-Depends ;-)

Not sure you will get what you want. the OpenCL version supported
by the ICD Loader does not say anything about the OpenCL version
supported by the ICD itself.

So, if you have opencl-headers (= 1.2), amd-libopencl1 and
nvidia-opencl-icd installed, you can compile a program for
OpenCL 1.2 but it will fail at runtime because the ICD Loader
(amd-libopencl1) will call random functions (random addresses)
in nvidia-opencl-icd when OpenCL 1.2 functions are used.
It seems that there is no way that an ICD Loader can know
how many/which functions are available in the ICD implementations :-(

So, I think your build-depends ensure PyOpenCL can be compiled with
one OpenCL version compatible with the ICD Loader installed, but
it does nothing to ensure that it will run correctly (this depends
on the OpenCL version of the ICD(s) installed)

In summary:
With opencl-headers (= 1.1), you can compile OpenCL 1.1 programs
With opencl-headers (= 1.2), you can compile OpenCL 1.2 *and* 1.1 programs
(you will perhaps need to define CL_USE_DEPRECATED_OPENCL_1_1_APIS for 1.1)

With nvidia-libopencl1, you can compile and run OpenCL 1.1 programs
With amd-libopencl1, you can compile and run OpenCL 1.1 *and* 1.2 programs
With ocl-icd-libopencl1, you can compile and run OpenCL 1.1 *and* 1.2 programs
With intel-libopencl1, you can compile and run OpenCL 1.1 *and* 1.2 programs

With nvidia-opencl-icd, you can run OpenCL 1.1 programs
With amd-opencl-icd, you can run OpenCL 1.1 *and* 1.2 programs
With intel-opencl-icd, you can run OpenCL 1.1 programs and call a few 1.2 functions

*-libopencl1 are not co-installable
But *-opencl-icd *are* co-installable (and are choose at runtime by the programs)

About ICD Loaders (*-libopencl1), we can propose that they provide
libopencl-1.1 / libopencl-1.2 if they support it (ie amd-libopencl1 would provide
both). Would you be interested by it?
About ICD, I do not see anything easy. As they are co-installable (by
design: this is the reason of development of a ICD Loader), the selection
must be done at runtime (ie not by dependencies). If we use virtual packages
(opencl-1.1, opencl-1.2) to be sure that a compatible ICD is installed.
But, to my knowledge, there is nothing in the ICD Loader API that can
be used to detect/select the good implementation (but a hard-coded
knowledge of compatible ICD)

Regards,
Vincent


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4FFCAA6F.2040300@free.fr">http://lists.debian.org/4FFCAA6F.2040300@free.fr
 

Thread Tools




All times are GMT. The time now is 04:42 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org