libdpkg: m_fork and friends
 

Hi all,

I've already started this thread on debian-win32, however I've moved it
here as my questions are about proposed solutions to os-specific issues.


I've started trying to port dpkg natively to Windows (as opposed to
using Cygwin).


One area that may cause problems is that Windows doesn't have fork
functionality, so implementing m_fork() will be challenging.


From what I've seen so far, the only time m_fork gets used in the dpkg
source is as part of the process of spawning a child to do some work,
and writing Win32 child-spawning code should be pretty simple.


It was suggested on debian-win32 that I could use GLib for the process
spawning. This would simplify code, as there are functions to spawn
synchronously or asynchronously, and using single-string or vector-based
arguments, as well as automatically creating pipes for
stdin/stdout/stderr. On the down side, it does mean introducing a new
dependency to dpkg.


Either GLib or Win32-specific code will can solve the process spawning
issue, and this is all well and good as long as no other packages are
using the m_fork (and possibly m_pipe/m_dup2) code. So my questions are:


- Are other packages supposed to use m_fork, m_pipe, etc?
- If not, do we know of/care about packages that do use it?
- If so, would breaking API compatability to allow a Win32 port be a
problem.


Please note that my last question is badly phrased... it would _only_ be
the Win32 port that would have a changed API, specifically the functions
that can't be (easily) implemented (m_fork, possibly m_pipe/m_dup2)
would be #ifdef'd out, or moved to a separate file that autotools links
in on non-Win32 platforms.


Thanks,

Phil


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

libdpkg: m_fork and friends
 

phil, ian, hi,

http://lists.debian.org/debian-dpkg/2007/12/msg00040.html

apologies i realise this was 5 years ago but was any progress made,
here? 3 years ago i also tried tackling a port of dpkg to mingw32
(actually, MSYS mingw32 runtime running under wine - yes it works it's
just doogggg slow but they found some bugs in wine and fixed them)

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDy0xOr_iVMPsUB803MspU=cmbEOJYNGPA2gdTMtCmWDa A@mail.gmail.com">http://lists.debian.org/CAPweEDy0xOr_iVMPsUB803MspU=cmbEOJYNGPA2gdTMtCmWDa A@mail.gmail.com

libdpkg: m_fork and friends
 

As far as I recall, I gave up and went Linux only (i'm from a predominantly Unix background)

On Friday, March 16, 2012, lkcl luke <luke.leighton@gmail.com> wrote:

> phil, ian, hi,
>
> http://lists.debian.org/debian-dpkg/2007/12/msg00040.html
>
> apologies i realise this was 5 years ago but was any progress made,

> here? *3 years ago i also tried tackling a port of dpkg to mingw32
> (actually, MSYS mingw32 runtime running under wine - yes it works it's
> just doogggg slow but they found some bugs in wine and fixed them)

>
> l.
>

libdpkg: m_fork and friends
 

On Fri, Mar 16, 2012 at 5:33 PM, Phil Lello <phil.lello@googlemail.com> wrote:
> As far as I recall, I gave up and went Linux only (i'm from a predominantly
> Unix background)

*cackle* :)

whoops. it'd kinda be nice to have dpkg help solve the dll hell
thing. but there's another reason: i'm looking at recommending the
use of dpkg for B2G package management. that would actually mean it
would need to be available on a few more platforms than it is right
now.

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDyewSsoaM3Suk7LxtSFbSkC5g5qCDLWWLdNC39aq1yWR A@mail.gmail.com">http://lists.debian.org/CAPweEDyewSsoaM3Suk7LxtSFbSkC5g5qCDLWWLdNC39aq1yWR A@mail.gmail.com

libdpkg: m_fork and friends
 

On Fri, Mar 16, 2012 at 5:41 PM, lkcl luke <luke.leighton@gmail.com> wrote:
> On Fri, Mar 16, 2012 at 5:33 PM, Phil Lello <phil.lello@googlemail.com> wrote:
>> As far as I recall, I gave up and went Linux only (i'm from a predominantly
>> Unix background)
>
> **cackle* :)
>
> *whoops. *it'd kinda be nice to have dpkg help solve the dll hell
> thing. *but there's another reason: i'm looking at recommending the
> use of dpkg for B2G package management. *that would actually mean it
> would need to be available on a few more platforms than it is right
> now.

https://groups.google.com/group/comp.os.ms-windows.programmer.nt.kernel-mode/browse_thread/thread/86837f458b67a79f/eeceb4c546b35314?lnk=st&q=ntcreateprocess+fork&rnu m=4&hl=en#eeceb4c546b35314

apparently it is actually possible to use the low-level
NtCreateProcess function to create a unix version of fork. sort-of.
fuuun....

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDzcnznzWZwHjvcBS6v73nYc88p=WEYnCNXJ+JLHf5uYc w@mail.gmail.com">http://lists.debian.org/CAPweEDzcnznzWZwHjvcBS6v73nYc88p=WEYnCNXJ+JLHf5uYc w@mail.gmail.com

libdpkg: m_fork and friends
 

On Fri, Mar 16, 2012 at 5:51 PM, lkcl luke <luke.leighton@gmail.com> wrote:

> *apparently it is actually possible to use the low-level
> NtCreateProcess function to create a unix version of fork. *sort-of.
> fuuun....

http://blog.airesoft.co.uk/code/fork.cpp

ye gods what a mess! but... yep, it looks doable. *respect* for mr
airesoft, for solving _that_ one.

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDzqRkg601bKAOoFLc1L07Xro318KzB5vrJ1yusnwp5bD Q@mail.gmail.com">http://lists.debian.org/CAPweEDzqRkg601bKAOoFLc1L07Xro318KzB5vrJ1yusnwp5bD Q@mail.gmail.com

libdpkg: m_fork and friends
 

On Fri, Mar 16, 2012 at 6:03 PM, lkcl luke <luke.leighton@gmail.com> wrote:
> On Fri, Mar 16, 2012 at 5:51 PM, lkcl luke <luke.leighton@gmail.com> wrote:
>
>> *apparently it is actually possible to use the low-level
>> NtCreateProcess function to create a unix version of fork. *sort-of.
>> fuuun....
>
> http://blog.airesoft.co.uk/code/fork.cpp
>
> ye gods what a mess! *but... yep, it looks doable. **respect* for mr
> airesoft, for solving _that_ one.

ah i just spoke to him: the code he wrote segfaults when the fork
returns. i started investigating cygwin's implementation instead:
http://cygwin.com/cgi-bin/cvsweb.cgi/~checkout~/src/winsup/cygwin/fork.cc?rev=1.233&content-type=text/plain&cvsroot=src

it's much much worse.

um... is there any chance of redesigning dpkg to not require fork?
popen looks like it could be used in movecontrolfiles; likewise in
extracthalf.

deb_reassemble, again subproc_fork appears to be being used as a
subsitute for popen. and deb_verify. each of those perform an execlp
immediately after the fork, with a subproc_wait_check...

falliblesubprocess, too, seems to be executing a remote command and
waiting for it to complete before continuing.

ok, i'm labouring the point: is there any situation in dpkg where
fork *isn't* used to do the same job as popen?

l.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDwGw3cjOi-chVYsHvhJb7zt+=j3Can1MzZp55UXe+mfnA@mail.gmail.com ">http://lists.debian.org/CAPweEDwGw3cjOi-chVYsHvhJb7zt+=j3Can1MzZp55UXe+mfnA@mail.gmail.com

libdpkg: m_fork and friends
 

lkcl luke writes ("Re: libdpkg: m_fork and friends"):
> um... is there any chance of redesigning dpkg to not require fork?
> popen looks like it could be used in movecontrolfiles; likewise in
> extracthalf.

dpkg was never intended to be portable to not-at-all-unix-like
systems. I'm afraid you are probably onto a loser.

> ok, i'm labouring the point: is there any situation in dpkg where
> fork *isn't* used to do the same job as popen?

You mean, does it fork and then continue in both child and parent
without calling exec ? I don't think there are any irremoveable cases
of this but any removeable cases will probably involve intrusive
patches.

Also note that dpkg depends fundamentally on traditional unix
filesystem semantics, which are not generally available on Windows
(and if anything resembling them is, the performance is very poor).

Ian.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20327.19596.633773.738554@chiark.greenend.org.uk"> http://lists.debian.org/20327.19596.633773.738554@chiark.greenend.org.uk

libdpkg: m_fork and friends
 

On Mon, Mar 19, 2012 at 3:11 PM, Ian Jackson
<ijackson@chiark.greenend.org.uk> wrote:
> lkcl luke writes ("Re: libdpkg: m_fork and friends"):
>> *um... is there any chance of redesigning dpkg to not require fork?
>> popen looks like it could be used in movecontrolfiles; likewise in
>> extracthalf.
>
> dpkg was never intended to be portable to not-at-all-unix-like
> systems. *I'm afraid you are probably onto a loser.

i'm really tempted to take that as a challenge just to prove it can
be done, but equally i fear you may be right. yuck.

>> *ok, i'm labouring the point: is there any situation in dpkg where
>> fork *isn't* used to do the same job as popen?
>
> You mean, does it fork and then continue in both child and parent
> without calling exec ? *I don't think there are any irremoveable cases
> of this but any removeable cases will probably involve intrusive
> patches.

ok, allow me to go through them formally [patience appreciated]

> Also note that dpkg depends fundamentally on traditional unix
> filesystem semantics, which are not generally available on Windows
> (and if anything resembling them is, the performance is very poor).

yeahhh that's fine. performance is never a major issue here :) but
i believe, last time i looked, all the dependencies had been sorted
out (with one exception) by the mingw, msys and git-win32 teams. i
had to track down an implementation of flock but that was about it.

if git, bash and ssh can all be compiled for win32 using mingw32 i'm
sure that dpkg shouldn't be that hard.

anyway. review. this is based on dpkg-1.16.1.2 (notes below) - it
all looks good. the only two i have any queries about are fd_fd_copy
and the loggers.

loggers: although statusfd_add records the pipe's fd, there's no call
to subproc_wait (the pid is thrown away). fd_fd_copy: not sure what
this is actually for: perhaps a simple exec of the "cat" command would
do?

other than those two queries (one of which may have highlighted a bug
- not calling subproc_wait) i'd say it was doable.

the really bigger issue is, as someone earlier highlighted (with the
cygwin thing): some programs still hang on to their DLLs until they're
well and truly killed. however, i'd say (perhaps naively?) that that
was for the postinit and prerm (etc.) scripts to resolve and start
knocking processes on the head. my main curiosity is to see if dpkg
can be press-ganged into service (any service), and worry about those
kinds of issues later on down the line :)

l.

build.c
--------

do_build: line 451. pipe created to tar. wait @ 482
do_build: line 475. compress filter called. wait @ 481
do_build: line 531. pine to tar. wait @ 585
do_build: line 543. compress filter. wait @ 584.
do_build: line 554. find called. wait @ 576.

extract.c
---------

movecotnrolfiles: line 59. command shell. wait @ 63
extracthalf: line 263. fd_fd_copy. wait @ 325.
extracthalf: line 280. decompress. wait @ 323.
extracthalf: line 292. tar. wait @ 320

the purpose of the fd_fd_copy isn't clear to me (as in, i'm not sure
it's needed at all).

split.c
--------

deb_field: line 61. exec to backend. wait @ 81.

archives.c
----------

archivefiles: line 1238. exec to find. wait @ 1274

help.c
-------

do_script: line 270. exec script. wait @ 282.
ensure_pathname_nonexisting: 618. exec rm. wait @ 624

main.c
------

run_status_loggers: line 456. hook command. *no wait*.

processarc.c
------------

deb_reassemble: line 95. execlp splitter. wait @ 101.
deb_verify: line 135. execl verify. wait @ 145.
process_archive: line 428. execlp backend. wait @ 435.
process_archive: line 835. execlp backend. wait @ 862.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAPweEDzjZsp=_o4HbY5qm4+ZMruetY3rXXAAR9CznVc-c5+fUA@mail.gmail.com">http://lists.debian.org/CAPweEDzjZsp=_o4HbY5qm4+ZMruetY3rXXAAR9CznVc-c5+fUA@mail.gmail.com

libdpkg: m_fork and friends
 

lkcl luke wrote:

> performance is never a major issue here :) but
> i believe, last time i looked, all the dependencies had been sorted
> out (with one exception) by the mingw, msys and git-win32 teams. i
> had to track down an implementation of flock but that was about it.

Some examples from "Git for Windows" experience: Windows does not
allow removing a file or directory that is open, nor modifying a DSO
that is in use.

Now compare this to dpkg's requirements, which include the ability to
upgrade packages while they are currently in use (think: libc).

Can you reconcile these? If so, I will be happy to look over your
work and try it out when I am next on Windows. You might be able to
coordinate work on the debian-win32 list. If you run into specific
questions about the design or style of dpkg code, the friendly people
on the debian-dpkg list might be able to help you. However, until you
have something working, I do not expect there to be much interest in
solving Windows-specific problems on the debian-dpkg list or merging
workarounds to Windows API infelicities in the official dpkg.git
repository.

Hope that helps,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120319224144.GB19732@burratino">http://lists.debian.org/20120319224144.GB19732@burratino