Packaging large Java software stacks ?
Fellow developers,
It is difficult to integrate the large recent Java software stacks (Glassfish, Geronimo...) in a Linux distribution in general. The key reason is that most of those stacks require very precise versions of libraries (JARs) to run and to build. They won't work with the latest version of libraries as those might change APIs and/or key behavior. Java developers are used to pick specific JAR versions and assemble the exact needed stack, they don't want to care about sharing their dependencies with other packages, or about dependencies being upgraded. Tools like Maven help them in this endeavor, and they rely very heavily on this external code : dozens of those JARs are usually needed at runtime, hundreds of those at build-time. That makes most attempts to properly package large Java software in Ubuntu to fail for two reasons. (1) Need precisely-versioned artifacts Those stacks need, as build dependencies and as runtime dependencies, very precise versions of JARs. They won't build or run with a different one. Using a more recent one might break functionality in a creative way. A maven-based build will sometimes require 6 different versions of the same JAR. In our packages, we usually offer only one version, in corner cases one minor version for each major version. In some cases the Java software will run/build with ours, in most cases it won't. (2) Building entirely from source Java "compilation" (in fact bytecoding) requires lots of JARs to be present, because they are used to check all external method signatures. Geronimo and Glassfish builds will require hundreds of them. Packaging all those build dependencies from source is a huge work, and each of those build dependencies, in turn, will require more. Combined with the first problem, this makes packaging those stacks too much work for so little to gain (they are usually easily installable by downloading/unpacking the upstream tarball). I am looking at ways to solve these issues, and would very much like to hear ideas from our awesome community :) Potential solutions to solve (1) include using a parallel delivery mechanism for JARs, separated from the usual Ubuntu packages, something that could hook into Maven repositories... or get ready to create a separate package for each version of each JAR. Potential solutions to solve (2) include evolving our build-from-source policy to accept that JARs that would only be used as method-checking media during Java compilation to be considered part of the source... A solution to workaround both problems would be to avoid targeting our built-from-source repositories for such Java software and pack them with their binary dependencies (or as binary directly)... Please let me know what you think of all this problem. I'm pretty sure the Java world will one day gets its senses back and have a more distribution-friendly dependency system, but we probably need a solution until such day happens... or forget about such Java software altogether. -- Thierry Carrez Ubuntu server team -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Thierry Carrez schrieb:
> Potential solutions to solve (1) include using a parallel delivery > mechanism for JARs, separated from the usual Ubuntu packages, something > that could hook into Maven repositories... or get ready to create a > separate package for each version of each JAR. While I don't know much about Java, JARs and Java packaging, this sounds most like the most ubuntuic or debianic alternative. I realise that it will take huge amount of work and a very active Java team that manages the source packages that are out-of-date, etc. very well. I'm sure there's a lot of interest in getting this solved, but as far as I know we'd need more documentation or "agree on" example packages to make this work easier. What do you think? > Please let me know what you think of all this problem. I'm pretty sure > the Java world will one day gets its senses back and have a more > distribution-friendly dependency system, but we probably need a > solution until such day happens... or forget about such Java software > altogether. The only way I see to go about this is very careful bitching about breaking compatibility over and over again. We've done this successfully in other parts of the Open Source community. :-) Have a great day, Daniel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Daniel Holbach wrote:
> Thierry Carrez schrieb: >> Potential solutions to solve (1) include using a parallel delivery >> mechanism for JARs, separated from the usual Ubuntu packages, something >> that could hook into Maven repositories... or get ready to create a >> separate package for each version of each JAR. > > While I don't know much about Java, JARs and Java packaging, this sounds > most like the most ubuntuic or debianic alternative. I realise that it > will take huge amount of work and a very active Java team that manages > the source packages that are out-of-date, etc. very well. Creating a separate package for each version of each JAR is indeed the most debianic approach, however I'm not sure it's realistic. Glassfish alone needs ~230 separate versions of dependencies to build (of which we have ~30 in the tree currently). Add the dependencies to those dependencies to the mix, and the 1000 mark is probably not very far. And Glassfish has a rather clean dependency approach compared to Geronimo... > I'm sure there's a lot of interest in getting this solved, but as far as > I know we'd need more documentation or "agree on" example packages to > make this work easier. What do you think? I'm not sure there is so much interest. It's quite easy to install Java software from upstream binary tarballs. Even with tomcat6 packaged, lots of Java developers are still unpacking binary tarballs to install their own Tomcat... >> Please let me know what you think of all this problem. I'm pretty sure >> the Java world will one day gets its senses back and have a more >> distribution-friendly dependency system, but we probably need a >> solution until such day happens... or forget about such Java software >> altogether. > > The only way I see to go about this is very careful bitching about > breaking compatibility over and over again. We've done this successfully > in other parts of the Open Source community. :-) Yes, we still have to advocate the right way. Note that from Java developers point of view, their way of doing things (including precise external libraries as part of the package) is the right way, and our way (trying to run several things on top of the same pile of libraries) is wrong. -- Thierry Carrez Ubuntu server team -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
I'll add this disclaimer once for all the thread now: I hope to not
sound too authoritative. If somebody knows a cleverer way, speak up. :-) Thierry Carrez schrieb: > Creating a separate package for each version of each JAR is indeed the > most debianic approach, however I'm not sure it's realistic. Glassfish > alone needs ~230 separate versions of dependencies to build (of which we > have ~30 in the tree currently). Add the dependencies to those > dependencies to the mix, and the 1000 mark is probably not very far. And > Glassfish has a rather clean dependency approach compared to Geronimo... I have a set of questions: - How many of these do we have already? - How many of these ever change? - How many of these break compatibility? - How hard is the dependency? Is this always "== JAR_VERSION" or often a ">= JAR_VERSION" too? > I'm not sure there is so much interest. If it's "hard-core scientific rocket science work", it's going to be tough, but if we can demonstrate how it's done and clearly articulate what needs doing, I'm sure we can generate some interest here. There's a lot of people on the world who use Java and who would be happy to have it distributed by their distribution just like the rest of quality software that we ship. > It's quite easy to install Java > software from upstream binary tarballs. Even with tomcat6 packaged, lots > of Java developers are still unpacking binary tarballs to install their > own Tomcat... I'm not sure if a previous discussion of this ended up with differing outcome, but I think there's no framework for us to allow this. > Yes, we still have to advocate the right way. Note that from Java > developers point of view, their way of doing things (including precise > external libraries as part of the package) is the right way, and our way > (trying to run several things on top of the same pile of libraries) is > wrong. Maybe there needs to be a "Java Open Source Distribution Interest Group" or something. The other distributions will have the same problem like we do. Open Source Distributions essentially value and care about: - building from open source - shipping built binaries and being authoritative about what is shipped and installed on the users machines - stable interfaces and a clear indication (versioning, tools, etc) of changes - <add more items here> It might be a long winding road (and sound a bit demanding too), but if Java wants to be a "first class citizen" in the Open Source world, this needs to be clear. Have a great day, Daniel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Daniel Holbach wrote:
> Thierry Carrez schrieb: >> Creating a separate package for each version of each JAR is indeed the >> most debianic approach, however I'm not sure it's realistic. Glassfish >> alone needs ~230 separate versions of dependencies to build (of which we >> have ~30 in the tree currently). Add the dependencies to those >> dependencies to the mix, and the 1000 mark is probably not very far. And >> Glassfish has a rather clean dependency approach compared to Geronimo... > > I have a set of questions: > - How many of these do we have already? I would say around 500. 99% of them are providing a single version (usually a recent one). > - How many of these ever change? Most of them still regularly change. > - How many of these break compatibility? That's a difficult one to answer, since there is no API versioning in JARs, but apparently a lot of new releases do. The Geronimo developers told me the slightest version change frequently breaks the TCK JavaEE compliance tests, and using the latest is usually not the best. To get an idea of the (runtime) components versioning complexity, see [1]. The -Gnnnnnn versions means it's a Geronimo-patched version of the JAR that is used. [1] http://cwiki.apache.org/GMOxDOC22/component-versions.html > - How hard is the dependency? Is this always "== JAR_VERSION" or often > a ">= JAR_VERSION" too? Dependencies are "== JAR_VERSION" for Maven-based projects. Ant-based things are generally friendlier. Note that the JARs needed at runtime are usually shipped together with the software in the upstream binary release. They really consider JARs as pre-written blocks of their own code rather than as external deps... The Geronimo guys monitor security issues in all components and release a new version of the stack every time there is a security issue with some external thing. >> Yes, we still have to advocate the right way. Note that from Java >> developers point of view, their way of doing things (including precise >> external libraries as part of the package) is the right way, and our way >> (trying to run several things on top of the same pile of libraries) is >> wrong. > > Maybe there needs to be a "Java Open Source Distribution Interest Group" > or something. The other distributions will have the same problem like we do. > > Open Source Distributions essentially value and care about: > - building from open source > - shipping built binaries and being authoritative about what is shipped > and installed on the users machines > - stable interfaces and a clear indication (versioning, tools, etc) of > changes > - <add more items here> The JPackage project has been trying to do that, provide a sufficiently common platform to try to influence smaller upstream projects. The bigger projects (like Geronimo) have much more brittle stacks: they don't see the point of going through a very complex path while they already have a hard job assembling their stack without caring about anything else. -- Thierry Carrez Ubuntu server team -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Hello Thierry,
Thierry Carrez [2009-01-26 10:49 +0100]: > It is difficult to integrate the large recent Java software stacks > (Glassfish, Geronimo...) in a Linux distribution in general. You impressively described why packaging those "properly", i. e. with separate libraries and build dependencies, is not really sensible, if at all possible. It doesn't sound like we could support those kind of builds in our usual "backport patches" way of working anyway, if these components get constantly changed. However, do we actually need to? It doesn't sound like we would ever put those into main, just making it available conveniently for Ubuntu users? > Potential solutions to solve (1) include using a parallel delivery > mechanism for JARs, separated from the usual Ubuntu packages, something > that could hook into Maven repositories... or get ready to create a > separate package for each version of each JAR. So is Maven a system to pull Java libraries/projects from external well-known sources, much like Python eggs or Ruby gems? If those repositories already exist and are maintained, then we should rather think about integrating them better than trying to do the sysiphos work of packaging them again properly. > A solution to workaround both problems would be to avoid targeting our > built-from-source repositories for such Java software and pack them with > their binary dependencies (or as binary directly)... If I remember correctly, e. g. our glassfish package is a giant bundle of jars, i. e. includes everything you need to build and run the package. At this level, packaging seems to be feasible, and working (however ugly it might be wrt. "proper" packaging). > Please let me know what you think of all this problem. I'm pretty sure > the Java world will one day gets its senses back and have a more > distribution-friendly dependency system Stable APIs have worked very well in the C, perl, and Python world. I can't imagine why that should intrinsically not be possible in the Java world. But for projects where the changes are really as dramatic as you point out, I agree that we currently just have to do the "giant jar bundle" workaround, if we need to package it in the first place. However, we do have a fair bunch of Java libraries in the repositories, and even in main, and their API doesn't seem to change that often (if at all). So is this really just a problem of the two big projects you mentioned (glassfish/geronimo) or a wide-spread phenomenon, and the packages we have in main are just the "good" exceptions? Martin -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Thierry Carrez wrote:
> It is difficult to integrate the large recent Java software stacks > (Glassfish, Geronimo...) in a Linux distribution in general. The key > reason is that most of those stacks require very precise versions of > libraries (JARs) to run and to build. They won't work with the latest > version of libraries as those might change APIs and/or key behavior. > Java developers are used to pick specific JAR versions and assemble the > exact needed stack, they don't want to care about sharing their > dependencies with other packages, or about dependencies being upgraded. > Tools like Maven help them in this endeavor, and they rely very heavily > on this external code : dozens of those JARs are usually needed at > runtime, hundreds of those at build-time. > > That makes most attempts to properly package large Java software in > Ubuntu to fail for two reasons. > > (1) Need precisely-versioned artifacts > Those stacks need, as build dependencies and as runtime dependencies, > very precise versions of JARs. They won't build or run with a different > one. Using a more recent one might break functionality in a creative > way. A maven-based build will sometimes require 6 different versions of > the same JAR. In our packages, we usually offer only one version, > in corner cases one minor version for each major version. In some cases > the Java software will run/build with ours, in most cases it won't. In the case of libraries in other languages (most commonly C), we regularly port applications to work with the preferred version of the libraries we ship. When preparing new versions of C libraries, the API and ABI are checked, with the binary package name changed where they differ, to better ensure compatibility. Perhaps we could use the Java Introspection methods to generate some API report, and version Java libraries based on changing APIs? In the case where an incompatibility is not an API change, what sort of differences are encountered? Might these be considered bugs? Is there a case where two applications depend on the same API, but would break if used with different versions providing that API? > (2) Building entirely from source > Java "compilation" (in fact bytecoding) requires lots of JARs to be > present, because they are used to check all external method signatures. > Geronimo and Glassfish builds will require hundreds of them. Packaging > all those build dependencies from source is a huge work, and each of > those build dependencies, in turn, will require more. Combined with the > first problem, this makes packaging those stacks too much work for so > little to gain (they are usually easily installable by > downloading/unpacking the upstream tarball). Well, the same could be said for any interpreted language (python, ruby, perl, etc.), and to a large degree for compiled languages (which are often sufficiently portable that `./configure && make && make install` works). One of the points of building from source is to be able to verify that the provided source matches the provided binary, and allow our users to make further modifications to the software in accordance with the license. In a purely self-serving manner, shipping the source from which the package was built significantly reduces the effort required for distribution-local patches, most notably for security patches. > Potential solutions to solve (1) include using a parallel delivery > mechanism for JARs, separated from the usual Ubuntu packages, something > that could hook into Maven repositories... or get ready to create a > separate package for each version of each JAR. The last option is what is chosen for libraries in other languages, where we ship two or three different upstream versions while waiting for ports to occur. Where possible, we try to restrict this to less than three, and wherever possible to only one. > Potential solutions to solve (2) include evolving our build-from-source > policy to accept that JARs that would only be used as method-checking > media during Java compilation to be considered part of the source... > > A solution to workaround both problems would be to avoid targeting our > built-from-source repositories for such Java software and pack them with > their binary dependencies (or as binary directly)... This has historically been acceptable for some packages in multiverse, but without using the built-from-source method, it is hard for us to comply with licenses that insist that source code be provided on request as we cannot know that the source provided generates the binary provided. This is further complicated in terms of defect management: without being able to ourselves collect a source tree we know to be capable of generating a given binary object, it becomes very difficult for us to address any user-reported problems. This is compounded if there are embedded libraries, as a patch to fix a bug in a library may need to be repeated for several packages embedding that library. Even if such patches are generated, but the use of our build tools to generate the package has not been tested, we cannot be confident of our ability to regenerate a suitable package to distribute to users. Note also that including binary objects in a source limits our ability to patch them using our existing toolset. Variances from upstream distributed software are included in the diff,gz, which is restricted to unicode artifacts. The typical model to handle binary patches is to ship the replacement blob uuencoded, and uudecode at build-time for installation. When changing objects where the preferred form of modification is indeed binary (e.g. PNG files), this is acceptable. Where changing objects where the preferred form of modification are separate source, we would need to find a way to indicate from which source the replacement blob was generated (which is more than just patching the provided source, especially in the case of binary-only embedded libraries). In summary, I'd recommend shipping such stacks that do not require redistribution of source as binary objects in multiverse until such time as they can be built reliably from source within the distribution, and discussing the difficulties of extending the offered license to others with upstream for those stacks licensed with source-distribution licenses (e.g. GPL) if the entire application cannot be built without binary objects within the distribution (as I'm not convinced these can be shipped, although I'm not an archive-admin, so it's not my decision, and I'm certainly not providing any legal advice). My personal experience in working with upstream to get Java software packaged is that upstream is quite willing to help identify what pieces are missing in Ubuntu, help identify which patches may have been applied to any embedded libraries, and test their software against provided system libraries once available (even where versions differ). While this still leaves significant work in getting the necessary components packaged, I believe that these discussions lead to better sharing of patches between upstreams, and overall a better quality of the Java stack within Ubuntu. -- Emmet HIKORY -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Martin Pitt wrote:
> You impressively described why packaging those "properly", i. e. with > separate libraries and build dependencies, is not really sensible, if > at all possible. It doesn't sound like we could support those kind of > builds in our usual "backport patches" way of working anyway, if these > components get constantly changed. > > However, do we actually need to? It doesn't sound like we would ever > put those into main, just making it available conveniently for Ubuntu > users? The top stacks (Geronimo) might become targets for main, based on our support strategy. I agree with you it would be simpler if they weren't. >> Potential solutions to solve (1) include using a parallel delivery >> mechanism for JARs, separated from the usual Ubuntu packages, something >> that could hook into Maven repositories... or get ready to create a >> separate package for each version of each JAR. > > So is Maven a system to pull Java libraries/projects from external > well-known sources, much like Python eggs or Ruby gems? If those > repositories already exist and are maintained, then we should rather > think about integrating them better than trying to do the sysiphos > work of packaging them again properly. Maven is a project management tool that handles build-time dependencies, rather than a more classic runtime library delivery mechanism. Remember Java "binary releases" tarballs usually include all the needed external JARs directly, so there is no need for a runtime dependency delivery system. However Maven makes use of artifact repositories from well-known sources so we could try to leverage those. >> A solution to workaround both problems would be to avoid targeting our >> built-from-source repositories for such Java software and pack them with >> their binary dependencies (or as binary directly)... > > If I remember correctly, e. g. our glassfish package is a giant bundle > of jars, i. e. includes everything you need to build and run the > package. At this level, packaging seems to be feasible, and working > (however ugly it might be wrt. "proper" packaging). Yes, that's how glassfishv2 was packaged (in multiverse). For glassfishv3, an attempt to make a "proper" packaging is under way but hits the two issues I described in this post. >> Please let me know what you think of all this problem. I'm pretty sure >> the Java world will one day gets its senses back and have a more >> distribution-friendly dependency system > > Stable APIs have worked very well in the C, perl, and Python world. I > can't imagine why that should intrinsically not be possible in the > Java world. There is no reason why it would be impossible in the Java world. It's just that the Java platform doesn't help (yet) and the Java developers have decided not to care about "common library dependencies" but to just consider them as "pre-written bricks included in my code". > But for projects where the changes are really as dramatic > as you point out, I agree that we currently just have to do the "giant > jar bundle" workaround, if we need to package it in the first place. > However, we do have a fair bunch of Java libraries in the > repositories, and even in main, and their API doesn't seem to change > that often (if at all). So is this really just a problem of the two > big projects you mentioned (glassfish/geronimo) or a wide-spread > phenomenon, and the packages we have in main are just the "good" > exceptions? I think the Geronimo case is extreme, given the number of dependencies and the brittle nature of JavaEE TCK compliance. But for all projects built with Maven, you will need the precise artifact versions to build, or force it into using a slightly different version, without any easy way of telling if they are at least API-compatible. This may just fail to compile, or fail in strange ways at runtime... This is definitely something you can work on and test for small Maven-based projects, but for medium-sized ones it will be much more difficult, and you can't count on upstream support (they decided to use Maven to avoid caring about those issues). -- Thierry Carrez Ubuntu server team -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Emmet Hikory wrote:
> Thierry Carrez wrote: >> (1) Need precisely-versioned artifacts >> Those stacks need, as build dependencies and as runtime dependencies, >> very precise versions of JARs. They won't build or run with a different >> one. Using a more recent one might break functionality in a creative >> way. A maven-based build will sometimes require 6 different versions of >> the same JAR. In our packages, we usually offer only one version, >> in corner cases one minor version for each major version. In some cases >> the Java software will run/build with ours, in most cases it won't. > > In the case of libraries in other languages (most commonly C), we > regularly port applications to work with the preferred version of the > libraries we ship. When preparing new versions of C libraries, the API > and ABI are checked, with the binary package name changed where they > differ, to better ensure compatibility. Perhaps we could use the Java > Introspection methods to generate some API report, and version Java > libraries based on changing APIs? Yes, that would be an option. I know Sun wants to go in a more stable-API-oriented direction in the future, having acknowledged that the current JAR version numbers don't really mean anything useful. > In the case where an incompatibility > is not an API change, what sort of differences are encountered? Might > these be considered bugs? Is there a case where two applications depend > on the same API, but would break if used with different versions > providing that API? When some project requires a precise version of a JAR, you don't really know if that's : - because it relies on a specific API - because there is a bugfix they rely on - because newer features introduce a regression - because Maven uses =JAR_VERSION types of dependencies by default There might exist some Java library projects with strict versioning rules, like x.y.z where x.y defines API and .z defines bugfixes inside a stable API. But the general case is that they are allowed to (and will) break API and behavior for every version (no matter how minor), since (1) Maven helps developers in depending on a specific JAR version and (2) runtime JARs are shipped within the binary release tarball. >> A solution to workaround both problems would be to avoid targeting our >> built-from-source repositories for such Java software and pack them with >> their binary dependencies (or as binary directly)... > > This has historically been acceptable for some packages in > multiverse, but without using the built-from-source method, it is hard > for us to comply with licenses that insist that source code be provided > on request as we cannot know that the source provided generates the > binary provided. This is further complicated in terms of defect > management: without being able to ourselves collect a source tree we > know to be capable of generating a given binary object, it becomes very > difficult for us to address any user-reported problems. This is > compounded if there are embedded libraries, as a patch to fix a bug in a > library may need to be repeated for several packages embedding that > library. Even if such patches are generated, but the use of our build > tools to generate the package has not been tested, we cannot be > confident of our ability to regenerate a suitable package to distribute > to users. There is the middle alternative, where we would still build from source everything we *distribute*. There are a lot more Java build dependencies than runtime dependencies: most of the Java build dependencies are just used to check external method signatures during the bytecode compilation, much like an API description. So we could still produce the needed runtime JARs from their source by shipping those JAR build dependencies as part of the source package. That way we solve the source code providing hole, and we can have a clean defect management system, for a much smaller packaging cost. Taking numbers from a project I'm working on : - binary distribution: 1 package - build from source all runtime dependencies: 15 packages - build from source all runtime/build deps: ~200 packages (just counting one level deep. Deeper analysis under way, conservative estimates are ~500 packages needed) So you can see that we are definitely still in the "doable" area with the middle solution. If someone knew a way of turning off external method checking during bytecode compilation, or a way of providing a text-based API description to compile against... that would avoid the "binary" thing altogether. -- Thierry Carrez Ubuntu server team -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Packaging large Java software stacks ?
Thierry-
Just curious, but have you (or anyone else) checked how JBoss is packaged and built by and for Red Hat? Not that I'm suggesting JBoss as a replacement for Glassfish/Geronimo/etc, but perhaps they have made some break-throughs in the realm of packaging that we might emulate? :-Dustin -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
| All times are GMT. The time now is 07:18 AM. |
VBulletin, Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.