FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Java

 
 
LinkBack Thread Tools
 
Old 02-10-2011, 08:52 PM
Russ Allbery
 
Default Dependence on specific versions

Stefane Fermigier <sf@nuxeo.com> writes:
> On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:

>> For those of us who have been doing this sort of thing for a while,
>> this argument sounds very familiar. I've heard this argument applied
>> to C libraries, Perl modules, Python modules, and most recently Ruby
>> modules. It always sounds persuasive when presented from a stability
>> perspective. It has always proven to be completely wrong in the long
>> run.

> Please develop, unless you want me to believe you only based on your
> reputation, which I won't since I don't know you.

You're advocating for a (very significant) change in Debian Policy, and I
think the burden of proof is on showing why Java libraries should be
treated differently than every other library in Debian. They pose the
same challenges for security support, for ensuring rebuildability of
software, and for keeping the whole distribution moving forward in an
integrated fashion, and therefore the same issues apply.

So, I think you have the burden of proof reveresed. I think you have to
convince Debian that Java needs to be a weird special case, rather than
asking me to convince you that Debian's policy for every other language
should also apply to Java.

You can look at the infrastructure and handling of libraries in every
other language in Debian, and observe that they all follow the system that
we're asking Java libraries to follow as well. Java isn't being singled
out here. Historically, every one of those languages have presented the
same argument that you're presenting. For example, including convenience
copies of C libraries in other applications (zlib was a major offender),
or slightly modified versions of those libraries, or requiring specific
versions, used to be widespread. Application developers argued long and
hard that this was required for stability and that they would handle
security updates.

It wasn't required for stability, and they didn't handle security updates
properly.

Debian, and just about every other Linux distribution, put our collective
feet down and said "no, we're not going to do things that way." And as a
result, the library handling for C applications is now far, far more sane,
reasonable, and maintainable than we could have ever anticipated back when
this started.

Getting there is a lot of really hard work, but I'm convinced that the
same thing will apply in the long run to Java, to Ruby, and to every other
language that's fighting these same issues around stable ABIs and shared
code across multiple projects.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87tygbk150.fsf@windlord.stanford.edu">http://lists.debian.org/87tygbk150.fsf@windlord.stanford.edu
 
Old 02-10-2011, 08:56 PM
Russ Allbery
 
Default Dependence on specific versions

"Jesús M. Navarro" <jesus.navarro@undominio.net> writes:

> If you don't do that already is because you know that your app will fail
> day-in day-out because foo developer (and bar, and zoot...) is not
> sensible enough not to break backwards compatibility between foo_1.2.3
> and foo_1.2.4. Russ is right in that when things go that way, things
> are completly wrong in the long run in so many manners it's not even
> funny. The fact this behaviour is "business as usual" on Java
> development it's only a measure of its average quality.

I also feel like I should say here that many Java developers *do* do this
properly. I've moved Java projects to new versions of common Java
libraries like Spring or various Apache infrastructure libraries and had
them just work. Often even across major releases, but very commonly with
minor releases. I don't think the current state of the art is actually as
dire as all that. Java offers very good mechanisms for information hiding
and minimizing the exposed API so that libraries can adjust for
compatibility issues, and good Java programmers use those facilities.

> Oh! and another hint: "if ain't broken don't fix it" is not such a
> valuable knowledge with regards to open source if only because of the
> outer testing base and the work it takes to jump over more than a few
> releases (exponential, not linear, to the distance).

Right, what you end up with is five or six or more copies of the same
library at different versions, frozen there because no one dares to try to
update and test, and then when that library has a security vulnerability,
you have to backport the fix to all these slightly different versions.
It's a nightmare.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87pqqzk0zw.fsf@windlord.stanford.edu">http://lists.debian.org/87pqqzk0zw.fsf@windlord.stanford.edu
 
Old 02-10-2011, 09:09 PM
Stefane Fermigier
 
Default Dependence on specific versions

On Feb 10, 2011, at 10:40 PM, Jesús M. Navarro wrote:

> Hi, Stefane:
>
> On Thursday 10 February 2011 21:29:34 Stefane Fermigier wrote:
>> On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:
>>> Stefane Fermigier <sf@nuxeo.com> writes:
>>>> Only by fixing version numbers of third-party libraries can you be sure
>>>> that the same build that works today will still work next week, if you
>>>> redo the build on the exact same version of the sources (and Maven, and
>>>> Java, of course), any operating system.
>>>>
>>>> Yes, we do upgrade third-party lib versions from time to time, but only
>>>> when there is a good reason to ("if it ain't broke, don't fix it").
>>>>
>>>> BTW: I used to think like you 3-4 years ago when I discovered Maven, but
>>>> had to change my mind due to the reality.
>>>
>>> For those of us who have been doing this sort of thing for a while, this
>>> argument sounds very familiar. I've heard this argument applied to C
>>> libraries, Perl modules, Python modules, and most recently Ruby modules.
>>> It always sounds persuasive when presented from a stability perspective.
>>> It has always proven to be completely wrong in the long run.
>>
>> Please develop, unless you want me to believe you only based on your
>> reputation, which I won't since I don't know you.
>
> I'll give my opinion here: both Russ and you are right.
>
> Yes, you are right: in order to distribute a product you must have to be able
> to reproduce it unambiguosly from your sources. This implies all your
> dependencies must be hardcoded.
>
> But in the case of Java (and Ruby, and Python), making this on development is
> nothing but a dirty hack to cover the malpractices of the developers.

Which developers, upstream or downstream ?

If I have to work with upstream libraries that ignore, or break, the rules you state below, and are, indeed, good software engineering rules, in theory, what am I to do ?

My (and my team's) experience has taught us that it's better to assumer the worse.

One of my friend, who's the maintainer of the distribute2 Python project, and knows a thing or two about dependency management, send me recently an article with the example of a Python library whose versioning scheme is that each version of his software is named after one of his cats.

(This is, BTW, not much more stupid that the versioning scheme chosen by Debian, where you can't figure out if a "squeeze" is a better or smaller version number than a "sid" or whatever).

Even with more classical versioning, you still have to work around date-based releases, alpha-beta-gamma-rc releases, pre-releases, etc.

All of this can confuse Maven's version comparison function, and give you a very wrong version if no one really cared.

> Your development environment should rely at most on minor versions of its
> dependencies and let the extraversion float as the upstream developer see fit
> because said upstream developer will have the acumen not to break backwards
> compatibility between extraversions and, in fact, will use extraversions only
> for bug fixing (and those you definetly want as soon as they are published).
> Your developers, on the other hand, will need to defend their position if
> your app is happening to be using two different versions of the same
> component and, no, telling "that's what X-tool pushed into my environment",
> or "that's the uberbleeding edge version from the upstream developer out of
> his nightly builds" is not a proper answer.
>
> So, if your app depends on, say, foo_1.2.3, your dependency checking will look
> for foo_1.2 or even "just" for foo_1. This will allow your continous
> building environment to fail early if it happens that foo_1.2.(x+1) or foo_1.
> (x+1) are not backwards compatible, analyze why and act accordingly. Of
> course, once the shinny new version of your app is ready, you will freeze
> your building environment for that version to whatever happens to be the
> exact versions of dependant libs at the moment.

Actually at least one thing is wrong in this model (which I used to think the right one): my continuous integration system (Jenkins, BTW), is not alerted when a new shiny (but buggy) new version of library so and so appears on the central maven repository. So no build will be triggered, and the breakage won't be noticed.

But next time I commit a change to my code base, the build will break, and I will do the logical thing, which is to assume that *I* did break the build, and spend an inordinate amount of time until I realize that the error doesn't come form my commit, but from an event that doesn't appear at all in the commit timeline.

> Oh! and another hint: "if ain't broken don't fix it" is not such a valuable
> knowledge with regards to open source if only because of the outer testing
> base and the work it takes to jump over more than a few releases
> (exponential, not linear, to the distance).

I didn't mean that you don't have from time to time, and in a very controlled manner, to upgrade your dependencies to keep up with upstream developments. But that you shouldn't leave it to chance.

S.

--
Stefane Fermigier, Founder and Chairman, Nuxeo
Open Source, Java EE based, Enterprise Content Management (ECM)
http://www.nuxeo.com/ - +33 1 40 33 79 87 - http://twitter.com/sfermigier
Join the Nuxeo Group on LinkedIn: http://linkedin.com/groups?gid=43314
New Nuxeo release: http://nuxeo.com/dm54
"There's no such thing as can't. You always have a choice."


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 255F9DA9-9A04-4A50-A7B2-D48A33CB818B@nuxeo.com">http://lists.debian.org/255F9DA9-9A04-4A50-A7B2-D48A33CB818B@nuxeo.com
 
Old 02-11-2011, 05:19 AM
Torsten Werner
 
Default Dependence on specific versions

Hi Thomas,

On Thu, Feb 10, 2011 at 10:23 PM, Thomas Zeeman <tjzeeman@xs4all.nl> wrote:
> Actually, you can predict it. Given the above order and no outside influences like a dependencyManagement block in your (parent) pom, *it would be 0.1. See http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html for a more detailed explanation on why this is and what surprising results the dependency mediation can have.

what I wanted to say: you cannot predict that for any non trivial real
work project. Try to look at the dependency chain of maven itself. But
not for too long because of the danger of getting insane.

> If a and c depend on the same part of the API of b and it has changed from .1 to .2 you will be in for either some spectacular failure at run time or some very subtle and hard to notice error in your application behaviour.

Yes, runtime is another step where the whole thing collapses. That is
probably why integration tests are more important in the Java world[*] - the code uses different dependencies during unittest time
(provided by Maven) and after integration (something else).

Torsten
[*] I must confess that I have no hard scientific evidence for my claim.


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTimoCM=1vggqry6dZ9h3ySGJBshe-w-MrOZuBw=3@mail.gmail.com">http://lists.debian.org/AANLkTimoCM=1vggqry6dZ9h3ySGJBshe-w-MrOZuBw=3@mail.gmail.com
 
Old 02-11-2011, 08:35 AM
"Thomas J. Zeeman"
 
Default Dependence on specific versions

On Fri, 11 Feb 2011 07:19:33 +0100, Torsten Werner wrote:

Hi Thomas,

On Thu, Feb 10, 2011 at 10:23 PM, Thomas Zeeman <tjzeeman@xs4all.nl>
wrote:
Actually, you can predict it. Given the above order and no outside
influences like a dependencyManagement block in your (parent) pom, *it
would be 0.1. See
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html
for a more detailed explanation on why this is and what surprising
results the dependency mediation can have.


what I wanted to say: you cannot predict that for any non trivial
real

work project.


I think you can. Given that maven creates its dependency graph as a
tree where any transitive dependency which is also declared at the top
level is cut out at the transitive level and it prioritizes transitive
dependencies on depth in the tree first and on order of declaration in
the pom second. Granted. It is not going to be easy on larger projects,
but with enough patience and a large enough screen to show the entire
tree at once its doable. And yes, I've had to do just that one more
occasions than I would've liked already.



Try to look at the dependency chain of maven itself. But
not for too long because of the danger of getting insane.


Too late. I've had to do just that for a project where I had to
mavenise a largish (~200 jars in the largest war, 300 different
dependencies over all 4 wars), third party application using some
homegrown, ant based build so we could use it as part of a new
application. They were basically falling into all of the pitfalls a
poorly dependency managed project can fall. Not pretty!


If a and c depend on the same part of the API of b and it has
changed from .1 to .2 you will be in for either some spectacular
failure at run time or some very subtle and hard to notice error in
your application behaviour.


Yes, runtime is another step where the whole thing collapses. That is
probably why integration tests are more important in the Java world[*] - the code uses different dependencies during unittest time
(provided by Maven) and after integration (something else).


If only more people would do that...

I've found automated deploys as part of a continuous build process also
helps a lot too, but even that seems to be uncommon still.


Kind regards,
Thomas


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 35c4e8bfbc0b70783ec7c48edbda0988@xs4all.nl">http://lists.debian.org/35c4e8bfbc0b70783ec7c48edbda0988@xs4all.nl
 

Thread Tools




All times are GMT. The time now is 03:57 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org