FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 06-12-2011, 10:07 AM
Ted Dunning
 
Default more about Zookeeper

There has been a bit of talk about Apache Zookeeper recently by Thomas
Koch. He has a bad opinion of
Zookeeper and things it is "not good enough for Debian".

I would like to provide a countervailing view. Thomas and I get along
very well in person, but there are
definitely somethings that we do not always agree on. I would not say
that his opinions on Zookeeper
are entirely without merit, but I do feel that they should be taken a
bit in context. In general, I feel that
more moderate conclusions are to be preferred.

My first point is that stylistics and analysis tools like findBugs are
useful as ways to focus efforts to
improve code quality, but they can hardly be considered the primary
goal. The primary goal is highly
reliable software that can be maintained and which has a living
community around it. Zookeeper has
achieved all three of these. Zookeeper is highly reliable and is used
extensively in high end web
systems and in many other applications. In my own years of experience
with Zookeeper, I very rarely
have to restart Zookeeper and the issues I have had operationally were
almost exclusively when I
screwed up by allowing a disk to fill up (in which case ZK fails safe
and asks for help) or overloaded
the in-memory datastore (in which case ZK fails safe and asks for
help). I have found minor bugs
in startup scripts, but have never found any significant errors in
design or implementation in the
java server or java client. I heard of one really significant error
in the C API which was almost
immediately fixed on discovery. Moreover, ZK has maintained disk
image and wire compatibility
for years now across a half dozen version changes or more. In fact,
rolling upgrades with no
downtime have been possible during that time so there are Zookeeper
clusters around with years
of uptime even though none of the machines underneath the cluster has
been up so long.

Suffice it to say that ZK is widely adopted by both sophisticated and
unsophisticated users with
really excellent operational experience. From the admin point of
view, it is some of the most
reliable and high quality code around.

That leaves the question of internal code quality which is that Thomas
has focussed on in his critiques.

My own recent experience was as lead developer and proposer of the
multi-operation primitive. This
new capability is one of the largest API changes in years for
Zookeeper. As Thomas says, this has
taken nearly 6 months to complete. As Thomas neglects to mention,
however, most of that time was
spent without any significant effort being applied. I did the API
design, java client changes and wire format
extensions in a week or two late last year and then became too busy to
continue. Marshall McMullen
recently stepped up along with Camille Fournier to finish off the C
API changes and server side changes.
They (mostly Marshall) were able to make these changes in a few weeks.
Neither Marshall nor I had
previously been involved much in the internals and were able to make
our changes with only minimal
assistance from the core developers. Code review has taken a few more
weeks, largely due to schedule
conflict on the part of the core developers rather than difficulty.
IF you are curious, you can see the
entire history of the development at
https://issues.apache.org/jira/browse/ZOOKEEPER-965

The overall patch was moderately large involving changes to 28
separate code files. It was a fairly invasive
change because it involved changes in functionality on so many separate levels.

So we have some interesting evidence here.

On one side, automated tools quibble with the style of the code in ZK
and Thomas finds the code design
distasteful.

On the other side, two relatively naive coders (with respect to ZK)
were able to make substantial and invasive
changes without negative impact on other functionality. This is in a
very high performance transactional
storage system. This is the kind of software that is traditionally
some of the most difficult software to maintain.
Many consider real-time control software and certain high performance
device control applications to be more
difficult.

I won't say that Thomas is wrong. The code in ZK does need some
curation to make it better, more readable
and simpler. I am aware of no production code that does not need such help.

In contrast, however, the code as it stands is very well tested, is
extraordinarily stable and by actual demonstration
is maintainable and modifiable by new contributors.

As such, I would disagree strongly with the statement that ZK is not
of sufficient quality to be included in Debian.
Thomas may not want to maintain the Debian packaging, but hopefully
others will. I hope that Thomas' negative
comments do not discourage others from investigating Zookeeper and
drawing their own conclusions.


(and for disclosure sake, I am on the Zookeeper project management committee)


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTimG=boBJMz9V5UOadaMPiXs8et3rA@mail.gmail.com ">http://lists.debian.org/BANLkTimG=boBJMz9V5UOadaMPiXs8et3rA@mail.gmail.com
 
Old 06-12-2011, 10:36 AM
Mike Dupont
 
Default more about Zookeeper

Hi,
this sounds interesting, where can I find packages to test?
mike

On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com> wrote:


Zookeeper

--
James Michael DuPont
Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
 
Old 06-12-2011, 10:49 AM
Ted Dunning
 
Default more about Zookeeper

Debian packages?

Or the original code itself?

On Sun, Jun 12, 2011 at 12:36 PM, Mike Dupont
<jamesmikedupont@googlemail.com> wrote:
> Hi,
> this sounds interesting, where can I find packages to test?
> mike
>
> On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>>
>> Zookeeper
>
>
> --
> James Michael DuPont
> Member of Free Libre Open Source Software Kosova and Albania flossk.org
> flossal.org
>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTinnpqm44ZewBc9cu9Nx7ieOfQ8a8g@mail.gmail.com ">http://lists.debian.org/BANLkTinnpqm44ZewBc9cu9Nx7ieOfQ8a8g@mail.gmail.com
 
Old 06-12-2011, 10:51 AM
Ted Dunning
 
Default more about Zookeeper

Perhaps this is what you need: http://packages.debian.org/sid/zookeeper

On Sun, Jun 12, 2011 at 12:49 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> Debian packages?
>
> Or the original code itself?
>
> On Sun, Jun 12, 2011 at 12:36 PM, Mike *Dupont
> <jamesmikedupont@googlemail.com> wrote:
>> Hi,
>> this sounds interesting, where can I find packages to test?
>> mike
>>
>> On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>>>
>>> Zookeeper
>>
>>
>> --
>> James Michael DuPont
>> Member of Free Libre Open Source Software Kosova and Albania flossk.org
>> flossal.org
>>
>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTi=MEDdidDLepGUv0cJsSP4s-RxsPQ@mail.gmail.com">http://lists.debian.org/BANLkTi=MEDdidDLepGUv0cJsSP4s-RxsPQ@mail.gmail.com
 
Old 06-12-2011, 11:00 AM
Mike Dupont
 
Default more about Zookeeper

Thanks, I would like to try it out. I am currently experimenting with kestrel for creating a distributed openstreetmap rendering system.
http://osmopenlayers.blogspot.com/2011/06/get-started-with-renderbot-net.html



I have looked at hadoop before, but I dont know if it will be good for a p2p network.

mike

On Sun, Jun 12, 2011 at 12:51 PM, Ted Dunning <ted.dunning@gmail.com> wrote:


Perhaps this is what you need: http://packages.debian.org/sid/zookeeper





On Sun, Jun 12, 2011 at 12:49 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

> Debian packages?

>

> Or the original code itself?

>

> On Sun, Jun 12, 2011 at 12:36 PM, Mike *Dupont

> <jamesmikedupont@googlemail.com> wrote:

>> Hi,

>> this sounds interesting, where can I find packages to test?

>> mike

>>

>> On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

>>>

>>> Zookeeper

>>

>>

>> --

>> James Michael DuPont

>> Member of Free Libre Open Source Software Kosova and Albania flossk.org

>> flossal.org

>>

>



--
James Michael DuPont
Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
 
Old 06-12-2011, 11:30 AM
Ted Dunning
 
Default more about Zookeeper

Zookeeper is independent of hadoop except by heritage.

Hadoop should be excellent as a rendering system for maps. Map-reduce
is used extensively by Google for similar operations.

The basic idea is that you scan across all objects emitting a copy of
the object tagged with the tiles that object intersects. In the
reduce phase, the framework has collected all objects that intersect a
single map tile together so that rendering that tile is relatively
simple.

To render only changed tiles is similar. First you do the same
process as above to get a list of all tiles with changes. Then you
scan all objects emitting copies as before, but this time limited to
those tiles that have changes.

For very small numbers of changes, it may be faster to keep an index
and retrieve the involved objects using random access. Once the
fraction of objects involved in changed tiles reaches about 0.1 to 1%
of all objects, however, it will be much faster to simply scan all
objects since you will have sequential disk access.

(but this should really be another thread)


On Sun, Jun 12, 2011 at 1:00 PM, Mike Dupont
<jamesmikedupont@googlemail.com> wrote:
> Thanks, I would like to try it out. I am currently experimenting with
> kestrel for creating a distributed openstreetmap rendering system.
> http://osmopenlayers.blogspot.com/2011/06/get-started-with-renderbot-net.html
>
> I have looked at hadoop before, but I dont know if it will be good for a p2p
> network.
>
> mike
>
> On Sun, Jun 12, 2011 at 12:51 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>>
>> Perhaps this is what you need: http://packages.debian.org/sid/zookeeper
>>
>> On Sun, Jun 12, 2011 at 12:49 PM, Ted Dunning <ted.dunning@gmail.com>
>> wrote:
>> > Debian packages?
>> >
>> > Or the original code itself?
>> >
>> > On Sun, Jun 12, 2011 at 12:36 PM, Mike *Dupont
>> > <jamesmikedupont@googlemail.com> wrote:
>> >> Hi,
>> >> this sounds interesting, where can I find packages to test?
>> >> mike
>> >>
>> >> On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com>
>> >> wrote:
>> >>>
>> >>> Zookeeper
>> >>
>> >>
>> >> --
>> >> James Michael DuPont
>> >> Member of Free Libre Open Source Software Kosova and Albania flossk.org
>> >> flossal.org
>> >>
>> >
>
>
>
> --
> James Michael DuPont
> Member of Free Libre Open Source Software Kosova and Albania flossk.org
> flossal.org
>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTimV1NDx1BMnhZ_5NyCNXh1vUoCtQQ@mail.gmail.com ">http://lists.debian.org/BANLkTimV1NDx1BMnhZ_5NyCNXh1vUoCtQQ@mail.gmail.com
 
Old 06-12-2011, 11:30 AM
Ted Dunning
 
Default more about Zookeeper

Zookeeper is independent of hadoop except by heritage.

Hadoop should be excellent as a rendering system for maps. Map-reduce
is used extensively by Google for similar operations.

The basic idea is that you scan across all objects emitting a copy of
the object tagged with the tiles that object intersects. In the
reduce phase, the framework has collected all objects that intersect a
single map tile together so that rendering that tile is relatively
simple.

To render only changed tiles is similar. First you do the same
process as above to get a list of all tiles with changes. Then you
scan all objects emitting copies as before, but this time limited to
those tiles that have changes.

For very small numbers of changes, it may be faster to keep an index
and retrieve the involved objects using random access. Once the
fraction of objects involved in changed tiles reaches about 0.1 to 1%
of all objects, however, it will be much faster to simply scan all
objects since you will have sequential disk access.

(but this should really be another thread)


On Sun, Jun 12, 2011 at 1:00 PM, Mike Dupont
<jamesmikedupont@googlemail.com> wrote:
> Thanks, I would like to try it out. I am currently experimenting with
> kestrel for creating a distributed openstreetmap rendering system.
> http://osmopenlayers.blogspot.com/2011/06/get-started-with-renderbot-net.html
>
> I have looked at hadoop before, but I dont know if it will be good for a p2p
> network.
>
> mike
>
> On Sun, Jun 12, 2011 at 12:51 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>>
>> Perhaps this is what you need: http://packages.debian.org/sid/zookeeper
>>
>> On Sun, Jun 12, 2011 at 12:49 PM, Ted Dunning <ted.dunning@gmail.com>
>> wrote:
>> > Debian packages?
>> >
>> > Or the original code itself?
>> >
>> > On Sun, Jun 12, 2011 at 12:36 PM, Mike *Dupont
>> > <jamesmikedupont@googlemail.com> wrote:
>> >> Hi,
>> >> this sounds interesting, where can I find packages to test?
>> >> mike
>> >>
>> >> On Sun, Jun 12, 2011 at 12:07 PM, Ted Dunning <ted.dunning@gmail.com>
>> >> wrote:
>> >>>
>> >>> Zookeeper
>> >>
>> >>
>> >> --
>> >> James Michael DuPont
>> >> Member of Free Libre Open Source Software Kosova and Albania flossk.org
>> >> flossal.org
>> >>
>> >
>
>
>
> --
> James Michael DuPont
> Member of Free Libre Open Source Software Kosova and Albania flossk.org
> flossal.org
>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTimV1NDx1BMnhZ_5NyCNXh1vUoCtQQ@mail.gmail.com ">http://lists.debian.org/BANLkTimV1NDx1BMnhZ_5NyCNXh1vUoCtQQ@mail.gmail.com
 
Old 06-12-2011, 11:32 AM
Neil Williams
 
Default more about Zookeeper

On Sun, 12 Jun 2011 12:07:46 +0200
Ted Dunning <ted.dunning@gmail.com> wrote:

> There has been a bit of talk about Apache Zookeeper recently by Thomas
> Koch.

For the benefit of the list, I suspect this is #602694 which is at
least partly a personal thing for the current Debian maintainer.

Debian, in general, cares more about #626020.

> He has a bad opinion of
> Zookeeper and things it is "not good enough for Debian".

Irrespective of disagreements with the current maintainer, zookeeper is
not good enough for Debian stable simply because it has a separate
release-critical bug. #626020 - FTBFS on mips. There is no argument
about this - failing to build from source on a supported architecture
means, by definition, that the software is not of sufficient quality for
Debian. End of.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626020

That one bug is sufficient reason for zookeeper to be excluded from the
next stable release. It's sufficiently bad that if it is left unfixed
when the release freeze starts, there would be a justifiable reason for
removing zookeeper from testing as well as preventing it getting into a
stable release. At that point, it's probably worth removing it from
unstable too. Release critical bugs *matter*.

There are different ways of handling this but handled it must be or
zookeeper in Debian will be history.

> goal. The primary goal is highly
> reliable software that can be maintained and which has a living
> community around it. Zookeeper has
> achieved all three of these.

... so the community need to see what can be done to fix the RC bug.

If the current maintainer isn't happy with the package then someone
else needs to step up or someone like me will seek the removal of the
package instead.

> Suffice it to say that ZK is widely adopted by both sophisticated and
> unsophisticated users with
> really excellent operational experience. From the admin point of
> view, it is some of the most
> reliable and high quality code around.

Thomas: are you going to continue maintaining zookeeper or has your
usage of it changed such that it would be better to orphan it in
Debian? (If so, a wnpp O: bug would be useful.)

> That leaves the question of internal code quality which is that Thomas
> has focussed on in his critiques.

Internal code quality would be at issue with the current FTBFS bug, yes.

> So we have some interesting evidence here.

It can be hard to debug arch-specific bugs but that is about working
with the Debian maintainer and the mips team to debug it. There's no
room for discussion here. Fix the bug or the package will eventually be
removed because it will, by definition, not be good enough for Debian.

> On one side, automated tools quibble with the style of the code in ZK
> and Thomas finds the code design
> distasteful.

That's been sufficient reason in the past for me to get packages
removed from Debian. It depends on the likelihood of RC bugs arising
from design problems, upstream reactions and whether someone in Debian
volunteers to do the Debian side of the work.

> In contrast, however, the code as it stands is very well tested, is
> extraordinarily stable and by actual demonstration
> is maintainable and modifiable by new contributors.
>
> As such, I would disagree strongly with the statement that ZK is not
> of sufficient quality to be included in Debian.

The RC bug argues differently and, in Debian, the RC bug is always more
persuasive than protests from interested parties who lack the time or
motivation to actually fix the problem.

> Thomas may not want to maintain the Debian packaging, but hopefully
> others will. I hope that Thomas' negative
> comments do not discourage others from investigating Zookeeper and
> drawing their own conclusions.

If someone does not step up to do the work (in say about a month or two
because we're quite a way from the release currently), then I'll seek
the removal of zookeeper from testing and unstable as a fix for the RC
bug myself. Forewarned is forearmed etc.

Thanks for putting zookeeper on my radar - I'll be watching it a lot
more closely now and I'll be in touch if it gets to be removed. I've no
interest in zookeeper in and of itself, I just care about quality
assurance in Debian.

A new twist on The Streisand Effect, you might say.
;-)

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/
 
Old 06-12-2011, 11:49 AM
Ted Dunning
 
Default more about Zookeeper

I just looked at this bug and it looks like a configuration error in
the environment rather than a bug in Zookeeper per se.

In particular, Java >= 1.6 is an explicit requirement of the Zookeeper
package while the error being emitted is highly characteristic of an
attempt to compile 1.6 compatible code on a java 1.5 compiler.

This is a relatively basic problem:

http://stackoverflow.com/questions/1678122/must-override-a-superclass-method-errors-after-importing-a-project-into-eclipse
http://stackoverflow.com/questions/2335655/why-is-javac-failing-on-override-annotation

I am surprised that you didn't know about this since you seem to be
otherwise knowledgable.

In any case, it is a bit strong language to blame code quality for a
build system configuration error.

Also, nobody seems to have contacted the actual project maintainers
about this problem. Your comments about lack of motivation are also
somewhat ad hominem (and incorrect).

On Sun, Jun 12, 2011 at 1:32 PM, Neil Williams <codehelp@debian.org> wrote:
>> He has a bad opinion of
>> Zookeeper and things it is "not good enough for Debian".
>
> Irrespective of disagreements with the current maintainer, zookeeper is
> not good enough for Debian stable simply because it has a separate
> release-critical bug. #626020 - FTBFS on mips. There is no argument
> about this - failing to build from source on a supported architecture
> means, by definition, that the software is not of sufficient quality for
> Debian. End of.
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626020

> ...
> As such, I would disagree strongly with the statement that ZK is not
> of sufficient quality to be included in Debian.

> The RC bug argues differently and, in Debian, the RC bug is always more
> persuasive than protests from interested parties who lack the time or
> motivation to actually fix the problem.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTim+FyU7J9gM6F4JBkEHKTPKYb4JZg@mail.gmail.com ">http://lists.debian.org/BANLkTim+FyU7J9gM6F4JBkEHKTPKYb4JZg@mail.gmail.com
 
Old 06-12-2011, 11:49 AM
Ted Dunning
 
Default more about Zookeeper

I just looked at this bug and it looks like a configuration error in
the environment rather than a bug in Zookeeper per se.

In particular, Java >= 1.6 is an explicit requirement of the Zookeeper
package while the error being emitted is highly characteristic of an
attempt to compile 1.6 compatible code on a java 1.5 compiler.

This is a relatively basic problem:

http://stackoverflow.com/questions/1678122/must-override-a-superclass-method-errors-after-importing-a-project-into-eclipse
http://stackoverflow.com/questions/2335655/why-is-javac-failing-on-override-annotation

I am surprised that you didn't know about this since you seem to be
otherwise knowledgable.

In any case, it is a bit strong language to blame code quality for a
build system configuration error.

Also, nobody seems to have contacted the actual project maintainers
about this problem. Your comments about lack of motivation are also
somewhat ad hominem (and incorrect).

On Sun, Jun 12, 2011 at 1:32 PM, Neil Williams <codehelp@debian.org> wrote:
>> He has a bad opinion of
>> Zookeeper and things it is "not good enough for Debian".
>
> Irrespective of disagreements with the current maintainer, zookeeper is
> not good enough for Debian stable simply because it has a separate
> release-critical bug. #626020 - FTBFS on mips. There is no argument
> about this - failing to build from source on a supported architecture
> means, by definition, that the software is not of sufficient quality for
> Debian. End of.
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=626020

> ...
> As such, I would disagree strongly with the statement that ZK is not
> of sufficient quality to be included in Debian.

> The RC bug argues differently and, in Debian, the RC bug is always more
> persuasive than protests from interested parties who lack the time or
> motivation to actually fix the problem.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTim+FyU7J9gM6F4JBkEHKTPKYb4JZg@mail.gmail.com ">http://lists.debian.org/BANLkTim+FyU7J9gM6F4JBkEHKTPKYb4JZg@mail.gmail.com
 

Thread Tools




All times are GMT. The time now is 06:34 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org