Hi all, newbie here.

What I'm trying to accomplish is offline updating packages lists in
/var/lib/apt/lists. As far as I know, there's no supported way of
doing this with apt tools, so I'm trying to do so with a bash script.
Assuming that no synaptic, aptitude, apt-get, etc. process is running,
doing the following will just works (assuming you have saved
/var/lib/apt/lists folder in lists.tar.bz2):

rm -r /var/lib/apt/lists
tar xvjpf lists.tar.bz2 -C /var/lib/apt

I realize this is ugly and unsafe, so I'd like to write the same in a
cleaner way, taking the lock on /var/lib/apt/lists/lock first,
similarly to what apt-get update and other tools do. Surprisingly, it
wasn't so easy as I tough, and the following bash script doesn't work
as I expected:

--------------------------------------
#!/bin/bash
set -e
(
flock --exclusive --nonblock 200
# do something inside /var/lib/apt/lists
sleep 50 # Taking some time to test if the lock works
) 200>/var/lib/apt/lists/lock
--------------------------------------

It doesn't work in the sense that trying to launch apt-get update
during the sleep time will simply works, like no lock was set. What am
I doing wrong?

I hope apt-get is not actually checking which process is taking the
lock and failing just if the lock was taken by apt-get: I think it
would not be a good design and would mean that my task, offline
updating apt packages lists with a provided archive, can't be safely
implemented.

Any idea, workaround? Thanks a lot.

Greetings,
Francesco


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

2010/2/10 Francesco Pretto <ceztkoml@gmail.com>:
> Hi all, newbie here.
>

Uh, forgot to say: I am not sure this is the right mailing list for my
problem. I'm sorry if it wasn't.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Hi Francesco,

Francesco Pretto wrote:

> --------------------------------------
> #!/bin/bash
> set -e
> (
> flock --exclusive --nonblock 200
> # do something inside /var/lib/apt/lists
> sleep 50 # Taking some time to test if the lock works
> ) 200>/var/lib/apt/lists/lock
> --------------------------------------
>
> It doesn't work in the sense that trying to launch apt-get update
> during the sleep time will simply works, like no lock was set. What am
> I doing wrong?

The problem is that apt use fcntl(2) locks, not flock(2) locks.

| $ apt-get source apt
| $ cd apt-*
| $ grep -R "Could not open lock file" . --include='*.cc' -C15 -h
[...]
| int FD = open(File.c_str(),O_RDWR | O_CREAT | O_NOFOLLOW,0640);
[...]
| SetCloseExec(FD,true);
|
| // Aquire a write lock
| struct flock fl;
| fl.l_type = F_WRLCK;
| fl.l_whence = SEEK_SET;
| fl.l_start = 0;
| fl.l_len = 0;
| if (fcntl(FD,F_SETLK,&fl) == -1)
| {
| $

I do not know how to use fcntl locks from a shell script. I would
suggest trying a scripting language like Perl or Python instead.

As for your general question about offline update using APT, I
don’t know if there’s something appropriate available already. I
would recommend looking at reprepro, apt-offline, and
http://batmat.net/apt-offline/ and asking in the deity mailing list
(to reach APT developers) or debian-user (to reach fellow users).

Hope that helps,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Francesco Pretto <ceztkoml@gmail.com> writes:

> Hi all, newbie here.
>
> What I'm trying to accomplish is offline updating packages lists in
> /var/lib/apt/lists. As far as I know, there's no supported way of
> doing this with apt tools, so I'm trying to do so with a bash script.
> Assuming that no synaptic, aptitude, apt-get, etc. process is running,
> doing the following will just works (assuming you have saved
> /var/lib/apt/lists folder in lists.tar.bz2):
>
> rm -r /var/lib/apt/lists
> tar xvjpf lists.tar.bz2 -C /var/lib/apt
>
> I realize this is ugly and unsafe, so I'd like to write the same in a
> cleaner way, taking the lock on /var/lib/apt/lists/lock first,
> similarly to what apt-get update and other tools do. Surprisingly, it
> wasn't so easy as I tough, and the following bash script doesn't work
> as I expected:
>
> --------------------------------------
> #!/bin/bash
> set -e
> (
> flock --exclusive --nonblock 200
> # do something inside /var/lib/apt/lists
> sleep 50 # Taking some time to test if the lock works
> ) 200>/var/lib/apt/lists/lock
> --------------------------------------
>
> It doesn't work in the sense that trying to launch apt-get update
> during the sleep time will simply works, like no lock was set. What am
> I doing wrong?
>
> I hope apt-get is not actually checking which process is taking the
> lock and failing just if the lock was taken by apt-get: I think it
> would not be a good design and would mean that my task, offline
> updating apt packages lists with a provided archive, can't be safely
> implemented.
>
> Any idea, workaround? Thanks a lot.
>
> Greetings,
> Francesco

Not tested and off the top of my head:

apt-get --no-download -o APT::Update::Pre-Invoke::="find /var/lib/apt/lists -type f -a ! -name lock && tar xvjpf lists.tar.bz2 -C /var/lib/apt" update


- Invoking apt-get takes the lock so that is out of the way.

- --no-download prevents apt-get from going online and downloading Index files.

- APT::Update::Pre-Invoke is run before updating the index
files. Although since you download nothing Post-Invoke would work
too. But maybe something else wants to run then already.

MfG
Goswin

PS: Why delete files prior to unpacking? Just unpack the new ones and
let apt clean up any extra files like it already does. That also allows for partial updates.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

2010/2/10 Jonathan Nieder <jrnieder@gmail.com>:
>
> The problem is that apt use fcntl(2) locks, not flock(2) locks.
>

Thanks a lot, I learned something new. This [1] is a great article regarding.

>
> I do not know how to use fcntl locks from a shell script. Â*I would
> suggest trying a scripting language like Perl or Python instead.
>

Yup, seems the the best way to go.


> As for your general question about offline update using APT, I
> don’t know if there’s something appropriate available already. Â*I
> would recommend looking at reprepro, apt-offline, and
> http://batmat.net/apt-offline/ and asking in the deity mailing list
> (to reach APT developers) or debian-user (to reach fellow users).
>

I'll take a look at these tools. I previously tried apt-zip and
aptoncd, but they just solve the problem of collecting debs, completly
ignoring the problem of updating the packages lists.

Greetings,
Francesco

[1] http://www.hackinglinuxexposed.com/articles/20030616.html


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

2010/2/10 Goswin von Brederlow <goswin-v-b@web.de>:
>
> Not tested and off the top of my head:
>
> apt-get --no-download -o APT::Update::Pre-Invoke::="find /var/lib/apt/lists -type f -a ! -name lock && tar xvjpf lists.tar.bz2 -C /var/lib/apt" update
>
>
> - Invoking apt-get takes the lock so that is out of the way.
>
> - --no-download prevents apt-get from going online and downloading Index files.
>
> - APT::Update::Pre-Invoke is run before updating the index
> *files. Although since you download nothing Post-Invoke would work
> *too. But maybe something else wants to run then already.
>

Thanks a lot, it seemed good but unfortunately that --no-download
seems to make apt-get *not* to call shell scripts in
APT::Update::Pre-Invoke and APT::Update::Post-Invoke configuration
items. I don't know other configuration items that
may do the job but I'll try to search.

>
> PS: Why delete files prior to unpacking? Just unpack the new ones and
> let apt clean up any extra files like it already does. That also allows for partial updates.
>

You are right, It was just a quick and ugly attempt. As most works,
they required to do this today, and it must be ready for yesterday,
ehehehehe.

Greetings,
Francesco


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

2010/2/10 Jonathan Nieder <jrnieder@gmail.com>:
> I do not know how to use fcntl locks from a shell script. *I would
> suggest trying a scripting language like Perl or Python instead.
>

Ok, the following is fine with me and don't add dependencies except python.

---------------------------------
#!/usr/bin/python

import fcntl, sys, subprocess

try:
fp = open('/var/lib/apt/lists/lock', 'w')
fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
except IOError:
# Unable to acquire the lock for some reason, report it
# some way and exit
sys.exit(1)

# Launch shell script 'critical.run' in the critical section
subprocess.call('./critical.run', shell=True)
---------------------------------

It can be launched with "sudo python lock.py". With gksudo is even
more integrated with desktop environments and is possible to do error
reporting in gtk with python. It scatters the whole script in more
files, but at least is correct.

Again, thanks Jonathan for the hint about fnctl().

Regards,
Francesco


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org