FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Java

 
 
LinkBack Thread Tools
 
Old 05-25-2011, 01:58 PM
Jonathan Nieder
 
Default Symbols/shlibs files for Java

(+cc: debian-java; please cc me on replies if following up there)
Hi Niels,

Niels Thykier wrote:

> we can still gain a lot
> from automated ABI/API breakage detection and well defined transitions[1].

This means allowing Java packages to use ${shlibsepends} in the same
way packages with binaries do? Sounds excellent.

> [1] Such being able to tell if reverse dependencies have been rebuilt
> against the new version of a library or not.

I had thought that Java doesn't require this in the way that DSOs do
(but I could easily be wrong; see below[*]). According to my
assumptions, either there is an API break or no ABI break at all; in
the former case, this means serious (FTBFS) bugs, while in the latter
case it is nice to rebuild the reverse dependencies in case they run a
test suite to catch problems but it does not need to affect
transitions to testing.

Example: Suppose I rename some important function or method. This
would be analogous to renaming a function in C, which warrants a
soname bump. So in C, there would be two versions of the DSO, one
from before and one after:

libexample.so.1 (in an old libexample1 binary package)
libexample.so.2 (in a new libexample2 binary package)

Before the updated libexample can transition to testing, we need all
packages that used to link to the former to start linking to the
latter, even if they never use the renamed function.

Now on the other hand suppose I decide to add a new function. This
is also a change in ABI, but it is backward-compatible, so it does not
warrant a soname bump. The deb-symbols infrastructure means we can
automatically add versioned dependencies based on functions an
application uses (which is great[**]), but it doesn't enter into
transition tracking much --- there is no need to rebuild packages
before the new version of the library enters testing.

C++ works the same way because constructors, destructors, overloaded
functions, and so on are implemented as functions with mangled names
behind the scenes.

In Java, though, I had thought that there was nothing like a soname
bump when a function gets removed.
[*] I mentioned above that I am probably missing something. It
might be possible to achieve something like the
libexample1/libexample2 case above by installing two versions of a
Java package, setting the classpath appropriately at run-time. If
there is something similar in common use, that's probably the big
piece I was confused about.

[**] This is the part that seems exciting to me for Java.

> As it is now, we manually create a libXY-java package if we suspect
> that the new version of libX-java is not backwards compatible.

Could you give an example? Does this also change the API or only ABI?

> First of all I was hoping that you might have some "Do" or "Don't"
> pointers from when dpkg added support for these things. Secondly, there
> might be some code or infrastructure that could be shared.
> Particularly I am interested in how you handle mapping
> filenames/SONAMES to a package (especially in cases like libc6, where
> there more than one lib in the package).

See deb-shlibs(5) and deb-symbols(5). (I know you already have. Just
listing references for future readers.)

> We also have cases where two packages provide the same library and it
> would be optimal for us to end up with libX-java | libY-java in the
> depends, but I have a feeling that is not entirely trivial to support
> (in a sane way).

The symbols file supports this. It should be possible as long as
the build-deps of packages using these libraries have enough
information to figure it out (so, it can't be "libX-java and libY-java
install files to the same directory and javahelper figures it out").

> I am not too sure that we can recycle the existing formats
> (maybe the shlibs format with s/SONAME/filename/) as we have to check
> for things like classes, return-types, inheritance and method
> overloading as well.

I think checking for ABI breakage is a different and harder problem.
The symbols infrastructure can notice if a symbol was added or
removed, but in C and C++ that is not enough to know if something
(like the return type, as you mention) changed.

Have you talked to the authors of the "ABI compliance checker" at
<http://ispras.linux-foundation.org/index.php/ABI_compliance_checker>?

Thanks and good luck,
Jonathan


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110525135759.GA2352@elie">http://lists.debian.org/20110525135759.GA2352@elie
 
Old 05-25-2011, 01:58 PM
Jonathan Nieder
 
Default Symbols/shlibs files for Java

(+cc: debian-java; please cc me on replies if following up there)
Hi Niels,

Niels Thykier wrote:

> we can still gain a lot
> from automated ABI/API breakage detection and well defined transitions[1].

This means allowing Java packages to use ${shlibsepends} in the same
way packages with binaries do? Sounds excellent.

> [1] Such being able to tell if reverse dependencies have been rebuilt
> against the new version of a library or not.

I had thought that Java doesn't require this in the way that DSOs do
(but I could easily be wrong; see below[*]). According to my
assumptions, either there is an API break or no ABI break at all; in
the former case, this means serious (FTBFS) bugs, while in the latter
case it is nice to rebuild the reverse dependencies in case they run a
test suite to catch problems but it does not need to affect
transitions to testing.

Example: Suppose I rename some important function or method. This
would be analogous to renaming a function in C, which warrants a
soname bump. So in C, there would be two versions of the DSO, one
from before and one after:

libexample.so.1 (in an old libexample1 binary package)
libexample.so.2 (in a new libexample2 binary package)

Before the updated libexample can transition to testing, we need all
packages that used to link to the former to start linking to the
latter, even if they never use the renamed function.

Now on the other hand suppose I decide to add a new function. This
is also a change in ABI, but it is backward-compatible, so it does not
warrant a soname bump. The deb-symbols infrastructure means we can
automatically add versioned dependencies based on functions an
application uses (which is great[**]), but it doesn't enter into
transition tracking much --- there is no need to rebuild packages
before the new version of the library enters testing.

C++ works the same way because constructors, destructors, overloaded
functions, and so on are implemented as functions with mangled names
behind the scenes.

In Java, though, I had thought that there was nothing like a soname
bump when a function gets removed.
[*] I mentioned above that I am probably missing something. It
might be possible to achieve something like the
libexample1/libexample2 case above by installing two versions of a
Java package, setting the classpath appropriately at run-time. If
there is something similar in common use, that's probably the big
piece I was confused about.

[**] This is the part that seems exciting to me for Java.

> As it is now, we manually create a libXY-java package if we suspect
> that the new version of libX-java is not backwards compatible.

Could you give an example? Does this also change the API or only ABI?

> First of all I was hoping that you might have some "Do" or "Don't"
> pointers from when dpkg added support for these things. Secondly, there
> might be some code or infrastructure that could be shared.
> Particularly I am interested in how you handle mapping
> filenames/SONAMES to a package (especially in cases like libc6, where
> there more than one lib in the package).

See deb-shlibs(5) and deb-symbols(5). (I know you already have. Just
listing references for future readers.)

> We also have cases where two packages provide the same library and it
> would be optimal for us to end up with libX-java | libY-java in the
> depends, but I have a feeling that is not entirely trivial to support
> (in a sane way).

The symbols file supports this. It should be possible as long as
the build-deps of packages using these libraries have enough
information to figure it out (so, it can't be "libX-java and libY-java
install files to the same directory and javahelper figures it out").

> I am not too sure that we can recycle the existing formats
> (maybe the shlibs format with s/SONAME/filename/) as we have to check
> for things like classes, return-types, inheritance and method
> overloading as well.

I think checking for ABI breakage is a different and harder problem.
The symbols infrastructure can notice if a symbol was added or
removed, but in C and C++ that is not enough to know if something
(like the return type, as you mention) changed.

Have you talked to the authors of the "ABI compliance checker" at
<http://ispras.linux-foundation.org/index.php/ABI_compliance_checker>?

Thanks and good luck,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110525135759.GA2352@elie">http://lists.debian.org/20110525135759.GA2352@elie
 
Old 05-26-2011, 08:54 AM
Niels Thykier
 
Default Symbols/shlibs files for Java

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-05-25 15:58, Jonathan Nieder wrote:
> (+cc: debian-java; please cc me on replies if following up there)

(assuming you are on debian-dpkg@l.d.o)

> Hi Niels,
>
> Niels Thykier wrote:
>
>> we can still gain a lot
>> from automated ABI/API breakage detection and well defined transitions[1].
>
> This means allowing Java packages to use ${shlibsepends} in the same
> way packages with binaries do? Sounds excellent.
>

I think we would use ${javaepends} (currently used by javahelper
tools), but yes.

>> [1] Such being able to tell if reverse dependencies have been rebuilt
>> against the new version of a library or not.
>
> I had thought that Java doesn't require this in the way that DSOs do
> (but I could easily be wrong; see below[*]). According to my
> assumptions, either there is an API break or no ABI break at all; in
> the former case, this means serious (FTBFS) bugs, while in the latter
> case it is nice to rebuild the reverse dependencies in case they run a
> test suite to catch problems but it does not need to affect
> transitions to testing.
>

Turns out I cannot think of an example of an ABI breakage in Java that
is not also an API breakage (in some sense or another). Maybe someone
else on list can think of one (where symbols/shlibs tracking would be
helpful).
Maybe narrowing the return type of a primitive? Or generalizing an
object type paramenter?

> Example: Suppose I rename some important function or method. This
> would be analogous to renaming a function in C, which warrants a
> soname bump. So in C, there would be two versions of the DSO, one
> from before and one after:
>
> libexample.so.1 (in an old libexample1 binary package)
> libexample.so.2 (in a new libexample2 binary package)
>
> Before the updated libexample can transition to testing, we need all
> packages that used to link to the former to start linking to the
> latter, even if they never use the renamed function.
>
> Now on the other hand suppose I decide to add a new function. This
> is also a change in ABI, but it is backward-compatible, so it does not
> warrant a soname bump. The deb-symbols infrastructure means we can
> automatically add versioned dependencies based on functions an
> application uses (which is great[**]), but it doesn't enter into
> transition tracking much --- there is no need to rebuild packages
> before the new version of the library enters testing.
>
> C++ works the same way because constructors, destructors, overloaded
> functions, and so on are implemented as functions with mangled names
> behind the scenes.
>
> In Java, though, I had thought that there was nothing like a soname
> bump when a function gets removed.
>

There is not, but we are trying to make something like it. Basically we
would include a "fake" ABI version in the jar file and rewrite the
classpath to use the ABI version (as I recall at least).

So something like:
foo.jar -> foo2.jar (development symlink)
foo2.jar (real jar)

If foo2.jar breaks backwards compatibility, we would rename it to
foo3.jar (rename its package to make foo2.jar and foo3.jar to be
co-installable) and migrate reverse-dependencies to foo3.jar.

Classpath-wise a reverse-dependency would be built using "foo.jar" and a
helper tool would then re-write it to fooX.jar during build
(post-compilation) to reflect which "ABI" version of foo it was built
against. (So either foo2.jar or foo3.jar in this example)

I vaguely seem to remember that part of the plan here was to try to get
"binNMU" for arch:all packages as well. But I do not recall the details
here very well, so I hope someone can refresh my memory (Matthew?).

>[*] I mentioned above that I am probably missing something. It
> might be possible to achieve something like the
> libexample1/libexample2 case above by installing two versions of a
> Java package, setting the classpath appropriately at run-time. If
> there is something similar in common use, that's probably the big
> piece I was confused about.
>
> [**] This is the part that seems exciting to me for Java.
>

Certainly, this is definitely useful as well; currently we do it on a
best effort basis, but automatic tends to do a better job here in my
experience with symbols files in C.

>> As it is now, we manually create a libXY-java package if we suspect
>> that the new version of libX-java is not backwards compatible.
>
> Could you give an example? Does this also change the API or only ABI?
>

In these cases both packages will be co-installable and the jars will be
installed side-by-side with manual migration of reverse-dependencies.
So nothing will be broken by default, but it leaves old packages in the
archive such as libservlet2.3-java (recently removed) and
libservet2.4-java (RM TODO).

Other examples I can think of include libasmX-java and libicu4{,.2}-java.

>> First of all I was hoping that you might have some "Do" or "Don't"
>> pointers from when dpkg added support for these things. Secondly, there
>> might be some code or infrastructure that could be shared.
>> Particularly I am interested in how you handle mapping
>> filenames/SONAMES to a package (especially in cases like libc6, where
>> there more than one lib in the package).
>
> See deb-shlibs(5) and deb-symbols(5). (I know you already have. Just
> listing references for future readers.)
>

Been a while since I read them though.

>> We also have cases where two packages provide the same library and it
>> would be optimal for us to end up with libX-java | libY-java in the
>> depends, but I have a feeling that is not entirely trivial to support
>> (in a sane way).
>
> The symbols file supports this. It should be possible as long as
> the build-deps of packages using these libraries have enough
> information to figure it out (so, it can't be "libX-java and libY-java
> install files to the same directory and javahelper figures it out").
>

Okay; I think we have a few spicy cases such as openjdk-6, sun-java6 and
some library package provide some "extra" java API and gcj does not.
Would that imply that openjdk-6, sun-java6 and said library were all
installed during build (or that the symbol file of openjdk-6 "knows"
that its symbols are also available in sun-java6)?

>> I am not too sure that we can recycle the existing formats
>> (maybe the shlibs format with s/SONAME/filename/) as we have to check
>> for things like classes, return-types, inheritance and method
>> overloading as well.
>
> I think checking for ABI breakage is a different and harder problem.
> The symbols infrastructure can notice if a symbol was added or
> removed, but in C and C++ that is not enough to know if something
> (like the return type, as you mention) changed.
>

True, but I believe the symbol infrastructure catches a fair share of
(obvious) broken backwards compatibility. If we can create a format and
a checker that can check slightly less obvious breakages, then all the
better.

> Have you talked to the authors of the "ABI compliance checker" at
> <http://ispras.linux-foundation.org/index.php/ABI_compliance_checker>?
>

No, but thanks for the reminder.

> Thanks and good luck,
> Jonathan
>
>

~Niels

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJN3hVYAAoJEAVLu599gGRCaCAP/ibcy459MuLxCxM5bIkwKb9x
CBgjFUbmQFKWAs+kSOf0oi1ILwljJvUrl4ZpLjFdH5+SxuO0uT phhVTjTj5WZe2K
im/wshU/aoZxCRFbgH1I/jqS1O9Z4XrNZ9e0dLyFzzD6AcpEZ3DmlQb+l+jo5Pj3
3JUrMJeh44dI5UemV4tGDdH9Og/kMYDD9jY0YxlraIG5bY97oyBgFlk2cphMkiXc
1QPvH1u/x6j4mmegyRjhTAtj0JojAmVXw8ZZtmvOFitc9VEmIGIMSiFzOK pu92Tp
G7cZ6d3Vv3l/7IDNv18NBd76Vn0IaYBE1D7kFz/By79vwhPTmFvpQIVj6k7ZIE+j
ZtumHmT8hDBY2i9q2MR0XZzNwMIfm6uAME5AhqL0ZIk9g0p983 PtgQmx9WtrSbRZ
y5Pu7nIWKopKwHEr/IAza1OhadSv50Z2w5mNtUeYN5Bpa5bJrFIOFAJkBnfxu75I
FuxdOWFmOJ7pzNAfpme4tDlJgxvnRNb1MaEjzlqMJd+R7ZjzDZ 9yt5LTt+m+921T
RHDLnW8iPpNHFcTJ7g0YtRIHVE3JqEdhrXZSX0JOJiTfg9y4oc J3s/K6PWijQLgY
h7Vys7xI43Yl2pTBd9mTlaqiaNGtuwW0eBEZPOG2nEslPafdbA wldeuSBYLmLD2C
LPggAt+FCUtwJmjhRWEo
=2unk
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DDE155C.2000607@thykier.net">http://lists.debian.org/4DDE155C.2000607@thykier.net
 
Old 05-26-2011, 08:54 AM
Niels Thykier
 
Default Symbols/shlibs files for Java

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-05-25 15:58, Jonathan Nieder wrote:
> (+cc: debian-java; please cc me on replies if following up there)

(assuming you are on debian-dpkg@l.d.o)

> Hi Niels,
>
> Niels Thykier wrote:
>
>> we can still gain a lot
>> from automated ABI/API breakage detection and well defined transitions[1].
>
> This means allowing Java packages to use ${shlibsepends} in the same
> way packages with binaries do? Sounds excellent.
>

I think we would use ${javaepends} (currently used by javahelper
tools), but yes.

>> [1] Such being able to tell if reverse dependencies have been rebuilt
>> against the new version of a library or not.
>
> I had thought that Java doesn't require this in the way that DSOs do
> (but I could easily be wrong; see below[*]). According to my
> assumptions, either there is an API break or no ABI break at all; in
> the former case, this means serious (FTBFS) bugs, while in the latter
> case it is nice to rebuild the reverse dependencies in case they run a
> test suite to catch problems but it does not need to affect
> transitions to testing.
>

Turns out I cannot think of an example of an ABI breakage in Java that
is not also an API breakage (in some sense or another). Maybe someone
else on list can think of one (where symbols/shlibs tracking would be
helpful).
Maybe narrowing the return type of a primitive? Or generalizing an
object type paramenter?

> Example: Suppose I rename some important function or method. This
> would be analogous to renaming a function in C, which warrants a
> soname bump. So in C, there would be two versions of the DSO, one
> from before and one after:
>
> libexample.so.1 (in an old libexample1 binary package)
> libexample.so.2 (in a new libexample2 binary package)
>
> Before the updated libexample can transition to testing, we need all
> packages that used to link to the former to start linking to the
> latter, even if they never use the renamed function.
>
> Now on the other hand suppose I decide to add a new function. This
> is also a change in ABI, but it is backward-compatible, so it does not
> warrant a soname bump. The deb-symbols infrastructure means we can
> automatically add versioned dependencies based on functions an
> application uses (which is great[**]), but it doesn't enter into
> transition tracking much --- there is no need to rebuild packages
> before the new version of the library enters testing.
>
> C++ works the same way because constructors, destructors, overloaded
> functions, and so on are implemented as functions with mangled names
> behind the scenes.
>
> In Java, though, I had thought that there was nothing like a soname
> bump when a function gets removed.
>

There is not, but we are trying to make something like it. Basically we
would include a "fake" ABI version in the jar file and rewrite the
classpath to use the ABI version (as I recall at least).

So something like:
foo.jar -> foo2.jar (development symlink)
foo2.jar (real jar)

If foo2.jar breaks backwards compatibility, we would rename it to
foo3.jar (rename its package to make foo2.jar and foo3.jar to be
co-installable) and migrate reverse-dependencies to foo3.jar.

Classpath-wise a reverse-dependency would be built using "foo.jar" and a
helper tool would then re-write it to fooX.jar during build
(post-compilation) to reflect which "ABI" version of foo it was built
against. (So either foo2.jar or foo3.jar in this example)

I vaguely seem to remember that part of the plan here was to try to get
"binNMU" for arch:all packages as well. But I do not recall the details
here very well, so I hope someone can refresh my memory (Matthew?).

>[*] I mentioned above that I am probably missing something. It
> might be possible to achieve something like the
> libexample1/libexample2 case above by installing two versions of a
> Java package, setting the classpath appropriately at run-time. If
> there is something similar in common use, that's probably the big
> piece I was confused about.
>
> [**] This is the part that seems exciting to me for Java.
>

Certainly, this is definitely useful as well; currently we do it on a
best effort basis, but automatic tends to do a better job here in my
experience with symbols files in C.

>> As it is now, we manually create a libXY-java package if we suspect
>> that the new version of libX-java is not backwards compatible.
>
> Could you give an example? Does this also change the API or only ABI?
>

In these cases both packages will be co-installable and the jars will be
installed side-by-side with manual migration of reverse-dependencies.
So nothing will be broken by default, but it leaves old packages in the
archive such as libservlet2.3-java (recently removed) and
libservet2.4-java (RM TODO).

Other examples I can think of include libasmX-java and libicu4{,.2}-java.

>> First of all I was hoping that you might have some "Do" or "Don't"
>> pointers from when dpkg added support for these things. Secondly, there
>> might be some code or infrastructure that could be shared.
>> Particularly I am interested in how you handle mapping
>> filenames/SONAMES to a package (especially in cases like libc6, where
>> there more than one lib in the package).
>
> See deb-shlibs(5) and deb-symbols(5). (I know you already have. Just
> listing references for future readers.)
>

Been a while since I read them though.

>> We also have cases where two packages provide the same library and it
>> would be optimal for us to end up with libX-java | libY-java in the
>> depends, but I have a feeling that is not entirely trivial to support
>> (in a sane way).
>
> The symbols file supports this. It should be possible as long as
> the build-deps of packages using these libraries have enough
> information to figure it out (so, it can't be "libX-java and libY-java
> install files to the same directory and javahelper figures it out").
>

Okay; I think we have a few spicy cases such as openjdk-6, sun-java6 and
some library package provide some "extra" java API and gcj does not.
Would that imply that openjdk-6, sun-java6 and said library were all
installed during build (or that the symbol file of openjdk-6 "knows"
that its symbols are also available in sun-java6)?

>> I am not too sure that we can recycle the existing formats
>> (maybe the shlibs format with s/SONAME/filename/) as we have to check
>> for things like classes, return-types, inheritance and method
>> overloading as well.
>
> I think checking for ABI breakage is a different and harder problem.
> The symbols infrastructure can notice if a symbol was added or
> removed, but in C and C++ that is not enough to know if something
> (like the return type, as you mention) changed.
>

True, but I believe the symbol infrastructure catches a fair share of
(obvious) broken backwards compatibility. If we can create a format and
a checker that can check slightly less obvious breakages, then all the
better.

> Have you talked to the authors of the "ABI compliance checker" at
> <http://ispras.linux-foundation.org/index.php/ABI_compliance_checker>?
>

No, but thanks for the reminder.

> Thanks and good luck,
> Jonathan
>
>

~Niels

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJN3hVYAAoJEAVLu599gGRCaCAP/ibcy459MuLxCxM5bIkwKb9x
CBgjFUbmQFKWAs+kSOf0oi1ILwljJvUrl4ZpLjFdH5+SxuO0uT phhVTjTj5WZe2K
im/wshU/aoZxCRFbgH1I/jqS1O9Z4XrNZ9e0dLyFzzD6AcpEZ3DmlQb+l+jo5Pj3
3JUrMJeh44dI5UemV4tGDdH9Og/kMYDD9jY0YxlraIG5bY97oyBgFlk2cphMkiXc
1QPvH1u/x6j4mmegyRjhTAtj0JojAmVXw8ZZtmvOFitc9VEmIGIMSiFzOK pu92Tp
G7cZ6d3Vv3l/7IDNv18NBd76Vn0IaYBE1D7kFz/By79vwhPTmFvpQIVj6k7ZIE+j
ZtumHmT8hDBY2i9q2MR0XZzNwMIfm6uAME5AhqL0ZIk9g0p983 PtgQmx9WtrSbRZ
y5Pu7nIWKopKwHEr/IAza1OhadSv50Z2w5mNtUeYN5Bpa5bJrFIOFAJkBnfxu75I
FuxdOWFmOJ7pzNAfpme4tDlJgxvnRNb1MaEjzlqMJd+R7ZjzDZ 9yt5LTt+m+921T
RHDLnW8iPpNHFcTJ7g0YtRIHVE3JqEdhrXZSX0JOJiTfg9y4oc J3s/K6PWijQLgY
h7Vys7xI43Yl2pTBd9mTlaqiaNGtuwW0eBEZPOG2nEslPafdbA wldeuSBYLmLD2C
LPggAt+FCUtwJmjhRWEo
=2unk
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DDE155C.2000607@thykier.net">http://lists.debian.org/4DDE155C.2000607@thykier.net
 
Old 05-26-2011, 09:06 AM
Florian Weimer
 
Default Symbols/shlibs files for Java

* Niels Thykier:

> Turns out I cannot think of an example of an ABI breakage in Java that
> is not also an API breakage (in some sense or another).

Generalizing a parameter of a method in a final class to take a base
class of the original parameter type. Restricting the return parameter
of a method to a derived class.

These are source-compatible changes, but not binary-compatible.

--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 82aae96dbq.fsf@mid.bfk.de">http://lists.debian.org/82aae96dbq.fsf@mid.bfk.de
 
Old 05-26-2011, 09:06 AM
Florian Weimer
 
Default Symbols/shlibs files for Java

* Niels Thykier:

> Turns out I cannot think of an example of an ABI breakage in Java that
> is not also an API breakage (in some sense or another).

Generalizing a parameter of a method in a final class to take a base
class of the original parameter type. Restricting the return parameter
of a method to a derived class.

These are source-compatible changes, but not binary-compatible.

--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99


--
To UNSUBSCRIBE, email to debian-java-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 82aae96dbq.fsf@mid.bfk.de">http://lists.debian.org/82aae96dbq.fsf@mid.bfk.de
 
Old 05-26-2011, 10:03 AM
Andrey Ponomarenko
 
Default Symbols/shlibs files for Java

On 05/26/2011 12:54 PM, Niels Thykier wrote:


Turns out I cannot think of an example of an ABI breakage in Java that
is not also an API breakage (in some sense or another). Maybe someone
else on list can think of one (where symbols/shlibs tracking would be
helpful).
Maybe narrowing the return type of a primitive? Or generalizing an
object type paramenter?




I know at least 6 kinds of API changes that can break
binary/behavioral (ABI) compatibility and don't break source (API)
compatibility:


Change a non-final class field to be static (causes
IncompatibleClassChangeError exception)
Change the value of a final class constant field (behavioral,
the old value inlined at compile-time to the client code and
will be used instead of the new one)
Change a class field to be non-final (behavioral, the same
effect as in the previous one)
Change a class method to be non-abstract (behavioral)
Override a class method (behavioral)
Move a class method up class hierarchy (behavioral)

Also you might be interested to read about the clirr tool [1],
"Achieving API Binary Compatibility" article [2] or about the
upstream-tracker service for Java [3].



[1] Clirr Tool - checks
Java libraries for binary and source compatibility

[2] "Achieving
API Binary Compatibility" - article about binary compatibility
with notes about source compatibility

[3] Upstream
Tracker for Java - looking for new releases of various
libraries and checking them for backward binary and source
compatibility



--
Andrey Ponomarenko
Department for Operating Systems at ISPRAS
web: http://www.LinuxTesting.org
mail: aponomarenko@ispras.ru
 
Old 05-26-2011, 10:03 AM
Andrey Ponomarenko
 
Default Symbols/shlibs files for Java

On 05/26/2011 12:54 PM, Niels Thykier wrote:


Turns out I cannot think of an example of an ABI breakage in Java that
is not also an API breakage (in some sense or another). Maybe someone
else on list can think of one (where symbols/shlibs tracking would be
helpful).
Maybe narrowing the return type of a primitive? Or generalizing an
object type paramenter?




I know at least 6 kinds of API changes that can break
binary/behavioral (ABI) compatibility and don't break source (API)
compatibility:


Change a non-final class field to be static (causes
IncompatibleClassChangeError exception)
Change the value of a final class constant field (behavioral,
the old value inlined at compile-time to the client code and
will be used instead of the new one)
Change a class field to be non-final (behavioral, the same
effect as in the previous one)
Change a class method to be non-abstract (behavioral)
Override a class method (behavioral)
Move a class method up class hierarchy (behavioral)

Also you might be interested to read about the clirr tool [1],
"Achieving API Binary Compatibility" article [2] or about the
upstream-tracker service for Java [3].



[1] Clirr Tool - checks
Java libraries for binary and source compatibility

[2] "Achieving
API Binary Compatibility" - article about binary compatibility
with notes about source compatibility

[3] Upstream
Tracker for Java - looking for new releases of various
libraries and checking them for backward binary and source
compatibility



--
Andrey Ponomarenko
Department for Operating Systems at ISPRAS
web: http://www.LinuxTesting.org
mail: aponomarenko@ispras.ru
 
Old 05-28-2011, 12:30 PM
Raphael Hertzog
 
Default Symbols/shlibs files for Java

Hi Niels,

On Wed, 25 May 2011, Niels Thykier wrote:
> First of all I was hoping that you might have some "Do" or "Don't"
> pointers from when dpkg added support for these things.

Do not underestimate the task. Apart from that, I'm sorry I'm not sure
what kind of advice I can give you. :-)

> Secondly, there might be some code or infrastructure that could be
> shared.

I would love to generalize the principle of auto-generated dependencies
to cover more than just C libraries but we're far from that, i.e. there's
no infrastructure in place for this and all the code in dpkg-gensymbols
and dpkg-shlibdeps is highly specific to the case of C libraries/binaries.

> Particularly I am interested in how you handle mapping
> filenames/SONAMES to a package (especially in cases like libc6, where
> there more than one lib in the package).

There's nothing magical here. Once we have a SONAME, we find the library
on the system (using the same path that ld.so would use). Once we have
the complete filename, dpkg -S /the/file returns the package name. And
with the package name we're checking the content of
/var/lib/dpkg/info/<pkg>.shlibs (but you have to use dpkg-query
--control-path <pkg> shlibs to find that path).

> We also have cases where two packages provide the same library and it
> would be optimal for us to end up with libX-java | libY-java in the
> depends, but I have a feeling that is not entirely trivial to support
> (in a sane way).

Well, both packages need to provide this dependency. There's no way the
system can know that there is some other libraries that could fulfill the
same role and that it needs to put an alternative in the dependency.

> I intend to have all the tools to support this in the javahelper
> package. I am not too sure that we can recycle the existing formats
> (maybe the shlibs format with s/SONAME/filename/) as we have to check
> for things like classes, return-types, inheritance and method
> overloading as well. But feel free to correct me if symbols files
> already have support for this.

Sorry, I have too few java knowledge to answer this.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News â–¶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110528123049.GA10566@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110528123049.GA10566@rivendell.home.ouaza.com
 
Old 05-30-2011, 11:31 AM
Niels Thykier
 
Default Symbols/shlibs files for Java

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 2011-05-28 14:30, Raphael Hertzog wrote:
> Hi Niels,
>

Hey

(Added d-java to CC)

> On Wed, 25 May 2011, Niels Thykier wrote:
>> First of all I was hoping that you might have some "Do" or "Don't"
>> pointers from when dpkg added support for these things.
>
> Do not underestimate the task. Apart from that, I'm sorry I'm not sure
> what kind of advice I can give you. :-)
>



>> Secondly, there might be some code or infrastructure that could be
>> shared.
>
> I would love to generalize the principle of auto-generated dependencies
> to cover more than just C libraries but we're far from that, i.e. there's
> no infrastructure in place for this and all the code in dpkg-gensymbols
> and dpkg-shlibdeps is highly specific to the case of C libraries/binaries.
>

Could we begin refactoring this towards something similar to the
$NS::Source::Package setup (e.g. $NS::SymbolsFile::$LANG)? Or would you
rather see a different approach to this code-wise

>> Particularly I am interested in how you handle mapping
>> filenames/SONAMES to a package (especially in cases like libc6, where
>> there more than one lib in the package).
>
> There's nothing magical here. Once we have a SONAME, we find the library
> on the system (using the same path that ld.so would use). Once we have
> the complete filename, dpkg -S /the/file returns the package name. And
> with the package name we're checking the content of
> /var/lib/dpkg/info/<pkg>.shlibs (but you have to use dpkg-query
> --control-path <pkg> shlibs to find that path).
>

Aw, I was hoping for dragons and magic. :P But yeah, I should have seen
the dpkg -S; I have been using it before.

>> We also have cases where two packages provide the same library and it
>> would be optimal for us to end up with libX-java | libY-java in the
>> depends, but I have a feeling that is not entirely trivial to support
>> (in a sane way).
>
> Well, both packages need to provide this dependency. There's no way the
> system can know that there is some other libraries that could fulfill the
> same role and that it needs to put an alternative in the dependency.
>

I had a feeling you might say that.

>> I intend to have all the tools to support this in the javahelper
>> package. I am not too sure that we can recycle the existing formats
>> (maybe the shlibs format with s/SONAME/filename/) as we have to check
>> for things like classes, return-types, inheritance and method
>> overloading as well. But feel free to correct me if symbols files
>> already have support for this.
>
> Sorry, I have too few java knowledge to answer this.
>
> Cheers,

So I have been looking at this a bit more; the shlibs format actually
looks fully recyclable, assuming we can somehow tell a "C"-shlibs file
from a "Java"-shlibs file and map "SONAME" to filename accordingly.

... and if I discard my desire to record all access qualifiers and such,
I think the symbols file is mostly re-usable if we encode things right.

But first, a quick Java lecture so we are all more or less on the same
page. I will (where possible) map Java terms to C++. As I understood
Jonathan, C++ maps/mangles all constructors and methods into a flat
function name and builds a C library out of that.
A Java library consists of 0 or more class files stored in a jar (zip)
file. The meta data (such as dependencies) are stored in the manifest
file (plain text file). We can extract almost everything we need from
said class files (method-signatures etc) and the manifest file.

Java does a well-defined mangling of method names in the class files[1].
The mangled method could trivially be prefixed by its class name
(either in binary or source format[2]).

So the parseInt example from [1] could be stored in the symbols file as:

java.lang.Integer.parseInt(Ljava/lang/String;I)I

- - or -

java/lang/Integer.parseInt(Ljava/lang/String;I)I

Which would tell us that the class java.lang.Integer has a method with
the signature "int parseInt(java.lang.String,int)". Personally I would
prefer the second option of those two since it is easier to map to a
file name in the jar file (plus it consistently uses the binary name
instead of mixing source and binary name).
((For the rest of the email I will be using the format resembling the
latter of the two in the example above.))

In that case it is trivial to recycle the current symbols format (modulo
using possibly forbidden characters in the symbol names). Such as:

java/lang/Integer.parseInt(Ljava/lang/String;I)I@Base 1.1

This obviously assumes we can tell a C-symbols file from a Java-symbols
file and map the "SONAME" part accordingly. Since symbols cannot be
versioned like in C, I believe that the @Base part would be redundant
for Java.


The only thing missing is how to handle a regular field / constant; here
I see two "easy" options. Either use <encoded-type><field-name> or
<field-name><delimiter><encoded-type>. Assuming ":" is the delimiter
for the section option, it would like:

Imy/finctional/Code.length@Base 1.1
Ljava/io/PrintStream;java.lang.System.out@Base 1.1

- - or -

my/finctional/Code.length:I@Base 1.1
java/lang/System.out:Ljava/io/PrintStream;@Base 1.1

Encoding the two fictional fields (or constants) "int length;" and
"java.io.PrintStream out;" in the class my.finctional.Code and
java.lang.System (respectively). But I welcome alternatives.
Particularly, for for enums the type of the enum constant would be the
same as the class it is in,

- From there we can extend it to having "class-sections", e.g. something
like:

class java/lang/Integer
parseInt(Ljava/lang/String;I)I@Base 1.1
# other symbols in java/lang/Integer
class java/lang/String
# symbols in java/lang/String

This would reduce the size of the symbols file. From there on we could
always extend the format to include more information and check for
compatibility breakage beyond symbols.

The Java examples for symbols files in this mail are mere suggestions,
so if anyone has a better format to encode it in, please argue for it
and its advantages.

~Niels


[1] An example:

int parseInt(String str, int radix);

is mangled to (in byte-code format):

parseInt(Ljava/lang/String;I)I

Where "java/lang/String" is the binary name of the String class (L and ;
are start and end-markers) and "I" is the binary name of the "int". The
"I" after the end bracket denotes the return type, the types inside the
brackets are the arguments (in order).

Reference:
http://java.sun.com/docs/books/jvms/second_edition/html/ClassFile.doc.html#1169

http://java.sun.com/docs/books/jvms/second_edition/html/ClassFile.doc.html#84645

[2] The (fully qualifying) source format of (e.g.) "String" is
java.lang.String; the binary format is java/lang/String - which is just
a ".class" short of being the filename.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJN44AZAAoJEAVLu599gGRCMpkQAJvnwnttwy Hv6CKs7TjTDk94
UoJ2PfSU0Y6tD+OS1MUk9t1tbSwv7UCTsXJRroaA80iNNC1ncv Yzz9qTaDJzUjJi
MmKcumMNQufSIXQOX9xDSGYVTxLUrpYyZQ7ZxnGbpyfzfz5T+6 0ar4rkmOvfXl49
jdIXfwYBwf0R5ySkhXLZIZaPa8ikohH1fezPx9sd3YNgOEbQ64 saIwNfL+wdxH+c
PwY1d6qCt5OMORYZKNstScahxqtaFsfiCWYDdhlVbaAbc7KVW7 yuans9EHk3r7sN
95yc8T3UA05nrQwFlmdHuELGuzvTWJg5suUTNeIxxdXPM1r5i4 Y40vEl0/3dFWYz
fiC//5r+2mLDd/FgoLfWfx/5j0i3GVM0wVabXpCa+gutyV2E/noKciiNafDz0vNW
girqQT88ra8XIMgTzaaCkP9Zb1G4aimQQ0+1LtYsUo/WOrIH44q0vzBVerJS3KJs
atLIxbDie0wft9uOitcyw7Y6/J2ysol2oJzNGqBHFx6bI4Pqunpdi3eIctyVlQ/F
FP7/AgOxeY47IgITy1XpDMediefXOfwVsu0i9SRJ1k9qXYrn3ss5HO FI6pZfOkfc
MOQDoB94eEnkN/gqkh9lD6lKTuEn7rgBGfV2n8DDAv4CZs+teq4expfbGuGs7p5P
vOSDMz3cDe21/at0dEfS
=mBuA
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DE3801E.6080400@thykier.net">http://lists.debian.org/4DE3801E.6080400@thykier.net
 

Thread Tools




All times are GMT. The time now is 01:32 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org