FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 07-09-2010, 03:05 PM
Luke Kenneth Casson Leighton
 
Default dpkg minw32

dear debian-pkg,

last year i began cross-compiling packages for win32 using mingw32
(installed under wine and also using mingw32-cross). i spent one week
fighting with libicu38 and 90 other dependencies just to get one
application compiled (webkit). i hunted down a dozen different
obscure bits of libraries and functions, patching things like libbz2
and other libraries that were missing flock and so on.

i don't know if you've ever thought about the implications of running
mingw32 under wine, but fork() on win32 doesn't really exist: the only
person who successfully implemented fork() under win32 was jeremy
allison, for cygwin, by - believe it or not - monkey-patching windows
system memory in undocumented system data structures. and, if you
don't / can't do that, you're forced to use exec(), and that means
_complete_ restart from scratch under bash for example. the use of
exec() means that instantly you get a security system call/check
overhead... but it's much worse than that.

whilst bash shell scripts on a linux system you can shell/exec() one
thousand new shell instances per second, on mingw32 native under
windows that comes down to one PER SECOND OR WORSE, and when running
the same mingw32-compiled bash under wine, it's truly truly madness.
you should see the strace logs: total reload of x11 libraries,
printing subsystem, termcap - reams and reams of unnecessary system
calls, _just_ to start a child bash shell.

by the time you are using ./configure or running autoconf, it can be
up to fifteen to thirty seconds to do a single line of autoconf
output.

i mention this as background because yes it was insane.

against this background, anyone actually making the effort... well...
the effort is completely wasted, because of the lack of formal
packaging on win32, to track dependencies. that _really_ hit home.
so, being a debianite, i looked up "dpkg win32" on google and found a
failed project. so, i tried cross-compiling dpkg myself. i found
that it, too, wasn't going to work, and the reason was that yes,
again: fork and popen are used, to communicate with tar, bzip2, gzip
and so on.

this is in direct contrast to dpkg of some years ago, where it used to
be (statically?) compiled with libtar, libz and libbz2.

so my question is this: why was the decision made to turn dpkg into a
pipe-based architecture, and can that decision be reversed, with a
view to removing any system calls which do not exist under the mingw32
architecture?

many thanks,

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTin3ZvoRBrZpDfRE_uvGFor9nQy_rUxSNvQvLFEn@mail .gmail.com">http://lists.debian.org/AANLkTin3ZvoRBrZpDfRE_uvGFor9nQy_rUxSNvQvLFEn@mail .gmail.com
 
Old 07-09-2010, 11:43 PM
Jonathan Nieder
 
Default dpkg minw32

Hi Luke,

Luke Kenneth Casson Leighton wrote:

> i looked up "dpkg win32" on google and found a
> failed project. so, i tried cross-compiling dpkg myself. i found
> that it, too, wasn't going to work, and the reason was that yes,
> again: fork and popen are used, to communicate with tar, bzip2, gzip
> and so on.
>
> this is in direct contrast to dpkg of some years ago, where it used to
> be (statically?) compiled with libtar, libz and libbz2.

Check again? dpkg 1.1.4 from 1996 (oldest version I have handy) uses
the ‘tar’ binary to build and extract packages. And modern dpkg has
./configure --with/--without-zlib and similar options to decide
whether to use compression libaries or external compression programs.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100709234314.GA14264@burratino">http://lists.debian.org/20100709234314.GA14264@burratino
 
Old 07-10-2010, 12:10 PM
Luke Kenneth Casson Leighton
 
Default dpkg minw32

On Fri, Jul 9, 2010 at 11:43 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi Luke,
>
> Luke Kenneth Casson Leighton wrote:
>
>> * * * * * * * * * * * *i looked up "dpkg win32" on google and found a
>> failed project. *so, i tried cross-compiling dpkg myself. *i found
>> that it, too, wasn't going to work, and the reason was that yes,
>> again: fork and popen are used, to communicate with tar, bzip2, gzip
>> and so on.
>>
>> this is in direct contrast to dpkg of some years ago, where it used to
>> be (statically?) compiled with libtar, libz and libbz2.
>
> Check again? *dpkg 1.1.4 from 1996 (oldest version I have handy) uses
> the ‘tar’ binary to build and extract packages. *And modern dpkg has
> ./configure --with/--without-zlib and similar options to decide
> whether to use compression libaries or external compression programs.

really? _fantastic_. ha. thank you jonathan. harumph - could have
done knowing about that a year ago - should have asked back then,
shouldn't i?

ok. so that still leaves a sticky situation with using the tar
binary instead of libtar, and the fact that mingw32 doesn't have
fork(), and even _remotely_ considering attempting to use win32's
popen would result in a dog's dinner beyond all reasonable acceptable
coding standards. you only have to look at the hoops that python's
win32 implementation of "popen" to understand that (dup of HANDLEs,
system calls to associate with a filedescriptor, pass those across
process boundaries... nightmare).

was there any particular reason why dpkg farms out to tar, rather than
statically linking with libtar?

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTin1JKesiVaBTLaTy0iy-dDCqgAE72MDPAmYHfqf@mail.gmail.com">http://lists.debian.org/AANLkTin1JKesiVaBTLaTy0iy-dDCqgAE72MDPAmYHfqf@mail.gmail.com
 
Old 07-10-2010, 05:37 PM
Jonathan Nieder
 
Default dpkg minw32

Luke Kenneth Casson Leighton wrote:

> ok. so that still leaves a sticky situation with using the tar
> binary instead of libtar, and the fact that mingw32 doesn't have
> fork(), and even _remotely_ considering attempting to use win32's
> popen would result in a dog's dinner beyond all reasonable acceptable
> coding standards.

Have you tried it? I am not sure that running tar would really be
that horrible. It is usually easiest to first get something working,
then optimize.

dpkg needs to be able to spawn processes still, to run package
maintainer scripts.

> you only have to look at the hoops that python's
> win32 implementation of "popen" to understand that (dup of HANDLEs,
> system calls to associate with a filedescriptor, pass those across
> process boundaries... nightmare).

Um, dpkg doesn’t use popen. So the fork() + exec() examples, complete
with appropriate dup et al, should give enough of an example to write
some spawnve()-based code.

See compat/mingw.c in the source to git for some examples users of
the relevant Windows APIs.

> was there any particular reason why dpkg farms out to tar, rather than
> statically linking with libtar?

Probably because it is fairly easy, and tar is likely to be available
everywhere while libarchive et al are not.

Please, try to look at the code and try some things instead of just
being daunted by the task!

Regards,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100710173754.GA1520@burratino">http://lists.debian.org/20100710173754.GA1520@burratino
 

Thread Tools




All times are GMT. The time now is 07:07 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org