FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Development

 
 
LinkBack Thread Tools
 
Old 01-27-2009, 01:38 PM
Emmet Hikory
 
Default Packaging large Java software stacks ?

Thierry Carrez wrote:
> Emmet Hikory wrote:
>> In the case of libraries in other languages (most commonly C), we
>> regularly port applications to work with the preferred version of the
>> libraries we ship. When preparing new versions of C libraries, the API
>> and ABI are checked, with the binary package name changed where they
>> differ, to better ensure compatibility. Perhaps we could use the Java
>> Introspection methods to generate some API report, and version Java
>> libraries based on changing APIs?
>
> Yes, that would be an option. I know Sun wants to go in a more
> stable-API-oriented direction in the future, having acknowledged that
> the current JAR version numbers don't really mean anything useful.

Well, there's two factors here: a version number used to track
upstream workflow, and an API version used for dependency consistency.
In C, these are typically differentiated, so that the version of a
library may have little direct relation to the soname (e.g. 3.7.1 could
provide libfoo.s0.1.0.8).

>> In the case where an incompatibility
>> is not an API change, what sort of differences are encountered? Might
>> these be considered bugs? Is there a case where two applications depend
>> on the same API, but would break if used with different versions
>> providing that API?
>
> When some project requires a precise version of a JAR, you don't really
> know if that's :
> - because it relies on a specific API
> - because there is a bugfix they rely on
> - because newer features introduce a regression
> - because Maven uses =JAR_VERSION types of dependencies by default
>
> There might exist some Java library projects with strict versioning
> rules, like x.y.z where x.y defines API and .z defines bugfixes inside a
> stable API. But the general case is that they are allowed to (and will)
> break API and behavior for every version (no matter how minor), since
> (1) Maven helps developers in depending on a specific JAR version and
> (2) runtime JARs are shipped within the binary release tarball.

The issues I'm interested in are the second and third on your list,
as the first is soluable with an API checker, and the last is an
artifact of current social practice. The second is something that can
simply be documentation, and we can easily represent in the archive
using the existing dependencies model. The third is clearly a bug, and
ought be fixed, rather than ignored (which is significantly easier if
the package in question is available in a source form from which we can
create distributable binary artifacts in the regular manner).

> There is the middle alternative, where we would still build from source
> everything we *distribute*. There are a lot more Java build dependencies
> than runtime dependencies: most of the Java build dependencies are just
> used to check external method signatures during the bytecode
> compilation, much like an API description. So we could still produce the
> needed runtime JARs from their source by shipping those JAR build
> dependencies as part of the source package. That way we solve the source
> code providing hole, and we can have a clean defect management system,
> for a much smaller packaging cost.

Well, I see two issues with this approach. The first is that if the
package is built against some arbitrary API which may or may not match
the API provided by other system libraries, there may be runtime issues
unless the jars are also shipped as part of the binary package (thereby
becoming part of that which is distributed). The second is that we
still have no way to address any issues discovered: if there is an
inconsistency of some sort, we are again restricted to patching binary
objects in the hopes of addressing it.

I'm also in support of Martin's suggestion that we ensure that
end-users can use the distribution-unfriendly system if they wish (as we
do for cpan modules, eggs, gems, etc.), but don't use these on the
buildds, and just ship binary blobs until the software in question is
available in such a way that we can effectively maintain it. While this
may relegate some packages to multiverse while this is addressed, it
does more accurately represent our ability to support the application
directly.

--
Emmet HIKORY


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 09:10 AM
Thierry Carrez
 
Default Packaging large Java software stacks ?

Dustin Kirkland wrote:

> Just curious, but have you (or anyone else) checked how JBoss is
> packaged and built by and for Red Hat? Not that I'm suggesting JBoss
> as a replacement for Glassfish/Geronimo/etc, but perhaps they have
> made some break-throughs in the realm of packaging that we might
> emulate?

JBoss 4.2 has a much cleaner dependency approach (builds with ant,
relatively small number of deps, no =VERSION dep). It's actually
packaged in jaunty[1] so it works with our stack You could compare
its clean dependency approach to Tomcat.

So the solution for us might be to continue to do the same, package from
source the sane upstream works (Tomcat, JBoss), do the
big-binary-tarball approach for those we need but that can't build/run
from our stack (Glassfish, Geronimo) and, when we can, encourage them to
have a cleaner dependency approach.

There are a few lessons to learn from RedHat/Fedora approach to Java
packaging though. They have been leading/integrating the JPackage
project[2], which is a nice attempt to provide a consistent RPM-based
platform of Java libraries. They provide a single version of each JAR
and work hard to patch everything to work with them... or they just
leave the unfriendly software out of JPackage.

Note that you could consider that JPackage repositories are built around
JBoss dependency needs. It's not so much difficult when you control both
sides and your platform is built around a few key stacks need. There is
no complete Geronimo 2 or Glassfish in JPackage.

JPackage is apparently losing traction those days. They have been doing
quite an exceptional job to make everything work together but it doesn't
scale very well when you try to add new software (due to dependencies on
different versions of the same JAR, one of the problems we are trying to
solve here) or when you want to upgrade (you basically need a completely
new repository and start the work all over again). Some Java upstream
developers are already using the past when talking about JPackage, and a
few are describing this laudable attempt as a failure (but they are biased).

[1] https://launchpad.net/ubuntu/+source/jbossas4
[2] http://www.jpackage.org/

--
Thierry Carrez
Ubuntu server team

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 12:43 PM
Dustin Kirkland
 
Default Packaging large Java software stacks ?

On Wed, Jan 28, 2009 at 4:10 AM, Thierry Carrez
<thierry.carrez@ubuntu.com> wrote:
> JBoss 4.2 has a much cleaner dependency approach (builds with ant,
> relatively small number of deps, no =VERSION dep). It's actually
> packaged in jaunty[1] so it works with our stack You could compare
> its clean dependency approach to Tomcat.
>
> So the solution for us might be to continue to do the same, package from
> source the sane upstream works (Tomcat, JBoss), do the
> big-binary-tarball approach for those we need but that can't build/run
> from our stack (Glassfish, Geronimo) and, when we can, encourage them to
> have a cleaner dependency approach.
>
> There are a few lessons to learn from RedHat/Fedora approach to Java
> packaging though. They have been leading/integrating the JPackage
> project[2], which is a nice attempt to provide a consistent RPM-based
> platform of Java libraries. They provide a single version of each JAR
> and work hard to patch everything to work with them... or they just
> leave the unfriendly software out of JPackage.

Thierry-

As with all of your messages in this thread, thanks for a concise,
informative summary.

I should also mention one more thing to this list... And that's the
size of the effort at Red Hat, and within the community. RH employs a
throng of JBoss developers, and the Java community within Fedora seems
healthy.

:-Dustin

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 02:25 PM
Matthias Klose
 
Default Packaging large Java software stacks ?

I don't think this is related to Java software in general, but is related to the
practice of development and deployment. It is not unusual that the vendor of an
application server specifies requirements to the operating system down to single
patches, and a vendor using an application server certifying only one specific
version of this AS, plus specific versions of third party software. Plus there
is the hope that you are able to test one build and assume same test results for
all deployments. For this kind of software you usually see not more than one
upload per release cycle in Ubuntu. We certainly can package such software, but
this kind of software is better targeted to the partner archive. We will see
more requests to allow such "pre-tested" software into the archive, especially
if the upstreams are involved in the packaging of the software as well.

Looking at OOo: The source distribution made by upstream includes all system
libraries required by the build. With the time the community around OOo did add
configuration options to build with libraries found on the system instead while
upstream does build with the included libraries to provide their own
distribution independent build. At least here we do have the option to integrate
the build into our environment.

I don't know who did develop maven, but a comparable system is developed in the
python world: setuptools allows you to download things at build time, and set
requirements on version numbers for imports. Now, the download thing can be
turned off, and we do recommend not to use the version stuff for packaging
(using --single-version-externally packaged). Interestingly this system was
sponsored and developed by a company/companies shipping large python
applications in binary form. Again, if we do want to use the libraries found on
the system, we can do so.

In main we try not to duplicate stuff (one thing doing something), and we do
spend a lot of time to demote old libraries and older versions to the universe
section, and removing old stuff from universe. The same rules should apply Java
based software.

Thierry Carrez schrieb:
> Fellow developers,
>
> It is difficult to integrate the large recent Java software stacks
> (Glassfish, Geronimo...) in a Linux distribution in general. The key
> reason is that most of those stacks require very precise versions of
> libraries (JARs) to run and to build. They won't work with the latest
> version of libraries as those might change APIs and/or key behavior.
> Java developers are used to pick specific JAR versions and assemble the
> exact needed stack, they don't want to care about sharing their
> dependencies with other packages, or about dependencies being upgraded.
> Tools like Maven help them in this endeavor, and they rely very heavily
> on this external code : dozens of those JARs are usually needed at
> runtime, hundreds of those at build-time.

So maven basically is a tool to handle badly designed software? I do see the use
case, and maybe it is necessary to do this if you only have the compiled classes
for dependant libs available, but what does hinder us to fix the bugs in the
package and patch the version requirement as well so that maven is happy?

> That makes most attempts to properly package large Java software in
> Ubuntu to fail for two reasons.
>
> (1) Need precisely-versioned artifacts
> Those stacks need, as build dependencies and as runtime dependencies,
> very precise versions of JARs. They won't build or run with a different
> one. Using a more recent one might break functionality in a creative
> way. A maven-based build will sometimes require 6 different versions of
> the same JAR. In our packages, we usually offer only one version,
> in corner cases one minor version for each major version. In some cases
> the Java software will run/build with ours, in most cases it won't.
>
> (2) Building entirely from source
> Java "compilation" (in fact bytecoding) requires lots of JARs to be
> present, because they are used to check all external method signatures.
> Geronimo and Glassfish builds will require hundreds of them. Packaging
> all those build dependencies from source is a huge work, and each of
> those build dependencies, in turn, will require more. Combined with the
> first problem, this makes packaging those stacks too much work for so
> little to gain (they are usually easily installable by
> downloading/unpacking the upstream tarball).

Is it possible to disable these checks during the build? Assuming that we
provide these and assure those by (build-)dependencies?

> I am looking at ways to solve these issues, and would very much like to
> hear ideas from our awesome community
>
> Potential solutions to solve (1) include using a parallel delivery
> mechanism for JARs, separated from the usual Ubuntu packages, something
> that could hook into Maven repositories... or get ready to create a
> separate package for each version of each JAR.

I don't see how such mechanism would be maintained, and what would be different
than packaging different versions of a library. This looks like another
ia32-libs ...

> Potential solutions to solve (2) include evolving our build-from-source
> policy to accept that JARs that would only be used as method-checking
> media during Java compilation to be considered part of the source...

For another software I was told that the upstream doesn't care if the
documentation can be built, so the docs are pre-built, and included in the
binary package. In this case we are not able to patch these docs to fix a bug
(or at least with a higher effort), but maybe the docs are not that important as
the code is. method-checking looks like the same, but how do you ensure (and the
people who review software for main) that these are just used for checking and
not for building the package?

> A solution to workaround both problems would be to avoid targeting our
> built-from-source repositories for such Java software and pack them with
> their binary dependencies (or as binary directly)...

yes, this seems to work well for these packages.

Maybe another way would be to start with a really big source package, including
all sources, together with the committment to integrate this better into the
distribution. Not sure who does want to commit to this approach. Plus it would
not help to increase the stack of available Java packages in the distribution.

Matthias


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 03:16 PM
Matthias Klose
 
Default Packaging large Java software stacks ?

Thierry Carrez schrieb:
> That's a difficult one to answer, since there is no API versioning in
> JARs, but apparently a lot of new releases do. The Geronimo developers
> told me the slightest version change frequently breaks the TCK JavaEE
> compliance tests, and using the latest is usually not the best. To get
> an idea of the (runtime) components versioning complexity, see [1]. The
> -Gnnnnnn versions means it's a Geronimo-patched version of the JAR that
> is used.
>
> [1] http://cwiki.apache.org/GMOxDOC22/component-versions.html

I don't think all of these should translate into packages. Looking at Netbeans
you have hundreds or more components or modules, but just a few packages. I
doubt that we do want to package those all separately.

Matthias

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 03:58 PM
Thierry Carrez
 
Default Packaging large Java software stacks ?

Matthias Klose wrote:
> Thierry Carrez schrieb:
>>
>> That makes most attempts to properly package large Java software in
>> Ubuntu to fail for two reasons.
>>
>> (1) Need precisely-versioned artifacts
>> [...]
>> (2) Building entirely from source
>> Java "compilation" (in fact bytecoding) requires lots of JARs to be
>> present, because they are used to check all external method signatures.
>> Geronimo and Glassfish builds will require hundreds of them. Packaging
>> all those build dependencies from source is a huge work, and each of
>> those build dependencies, in turn, will require more. Combined with the
>> first problem, this makes packaging those stacks too much work for so
>> little to gain (they are usually easily installable by
>> downloading/unpacking the upstream tarball).
>
> Is it possible to disable these checks during the build? Assuming that we
> provide these and assure those by (build-)dependencies?

That would be nice, but I haven't found a way (option in javac or
whatever). I don't see a technical reason why external JARs would be
/needed/ to produce bytecode. And that would very much simplify the
build dependencies problem.

>> Potential solutions to solve (2) include evolving our build-from-source
>> policy to accept that JARs that would only be used as method-checking
>> media during Java compilation to be considered part of the source...
>
> For another software I was told that the upstream doesn't care if the
> documentation can be built, so the docs are pre-built, and included in the
> binary package. In this case we are not able to patch these docs to fix a bug
> (or at least with a higher effort), but maybe the docs are not that important as
> the code is. method-checking looks like the same, but how do you ensure (and the
> people who review software for main) that these are just used for checking and
> not for building the package?

That's the trick. You have to tell which build dependencies happen to be
run by the build process (ant tasks, maven plugins...) and which are
just used for method checks without executing any code. This would
require deep knowledge of the package and there may be loopholes (static
code run when classes are loaded ?).

That brings us back to the possibility to ignore method checks during
bytecode compilation, or to compile against a text description of an API...

--
Thierry Carrez
Ubuntu server team

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 04:02 PM
Emmet Hikory
 
Default Packaging large Java software stacks ?

Matthias Klose wrote:
> Thierry Carrez schrieb:
>> That's a difficult one to answer, since there is no API versioning in
>> JARs, but apparently a lot of new releases do. The Geronimo developers
>> told me the slightest version change frequently breaks the TCK JavaEE
>> compliance tests, and using the latest is usually not the best. To get
>> an idea of the (runtime) components versioning complexity, see [1]. The
>> -Gnnnnnn versions means it's a Geronimo-patched version of the JAR that
>> is used.
>>
>> [1] http://cwiki.apache.org/GMOxDOC22/component-versions.html
>
> I don't think all of these should translate into packages. Looking at Netbeans
> you have hundreds or more components or modules, but just a few packages. I
> doubt that we do want to package those all separately.

In the specific case of NetBeans, there was an analysis of what
existing packages were of general use, and an effort to get any
NetBeans-specific patches applied upstream so they would be included
with Ubuntu. The NetBeans codebase was then broken down into chunks
based on expected reuse (e.g. the platform library is separate from the
API because other applications may use the platform as a high-level
toolkit). In one case, a library became a separate upstream project.
Once the analysis was complete, there were only a few new source
packages, and when uploaded in the right order, they were then
self-sustaining.

Since then, the NetBeans team has been taking advantage of PPAs to
ensure that the stack continues to build against Ubuntu, and can be
tested as Ubuntu adjusts, so that Ubuntu can carry whichever release of
NetBeans is available at FeatureFreeze with some expectation that it
will be useful to users without additional download.

Additionally, NetBeans has a plugin system that allows some
components to be added post-install, at user discretion. While the
packaged NetBeans includes sufficient functionality for most uses,
end-users are able to use the plugin system to further extend the
packaged NetBeans, which reduced the packaging requirements to a
manageable level. At the current time there does not seem to be
sufficient demand for packaging of most of these optional plugins for
them to warrant additional packages in the archive.

I'm not sure how much of that would apply to the stacks in question,
but at least the application of a similar analysis may reduce the number
of packages required, depending on reuse in other environments.

Mind you, this doesn't address the issues with maven, or
incompatibilities with library versions, or any of that.

--
Emmet HIKORY

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 
Old 01-28-2009, 04:20 PM
Thierry Carrez
 
Default Packaging large Java software stacks ?

Matthias Klose wrote:
> Thierry Carrez schrieb:
>> That's a difficult one to answer, since there is no API versioning in
>> JARs, but apparently a lot of new releases do. The Geronimo developers
>> told me the slightest version change frequently breaks the TCK JavaEE
>> compliance tests, and using the latest is usually not the best. To get
>> an idea of the (runtime) components versioning complexity, see [1]. The
>> -Gnnnnnn versions means it's a Geronimo-patched version of the JAR that
>> is used.
>>
>> [1] http://cwiki.apache.org/GMOxDOC22/component-versions.html
>
> I don't think all of these should translate into packages. Looking at Netbeans
> you have hundreds or more components or modules, but just a few packages. I
> doubt that we do want to package those all separately.

These are not the Geronimo internal modules, but the runtime
dependencies from external projects (or related, but separate, Apache
projects) that Geronimo copies into their release tarball.

There is indeed some redundancy (for example libtomcat6-java provides 9
lines of that list, and GShell would probably provide 17) but in most
cases those would probably need to be packaged separately (as they come
from different sources and are released with their own version number).

--
Thierry Carrez
Ubuntu server team

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
 

Thread Tools




All times are GMT. The time now is 10:11 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org