FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 05-31-2010, 01:40 AM
Cameron Hutchison
 
Default Automatic partial archive mirroring

I have a few debian unstable boxes that I like to keep up-to-date.

Currently, I run apt-cacher-ng (a proxy for apt-get which stores
packages so they don't need to be downloaded again) on my gateway box
(lenny 32-bit), so I don't double download from my couple of other boxes
(sid 64-bit). This works well enough, but I want to make it better.

My internet connection has an off-peak period (2am-8am) where downloads
are not counted in the monthly quota. I want my downloads to occur
during this window, automatically.

The problem is that the sid boxes are not powered-up during this window
so I cannot just simply schedule a cronjob to do an apt-get -d
dist-upgrade.

I would like my lenny box to do this on behalf of my sid boxes. I dont
really want to mirror the whole archive. Is there some way to do an
intelligent partial mirror of the archive of what is installed on other
boxes, without too much overhead of managing a package list? (i.e. I dont
want to have to manually update some package list on my lenny box when I
install a new package on a sid box).

I've looked at apt-proxy, apt-cacher, apt-cacher-ng and approx, but they
all appear to download on demand, not according to a schedule.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 85e.4c0313ab.76bef@getafix.xdna.net">http://lists.debian.org/85e.4c0313ab.76bef@getafix.xdna.net
 
Old 05-31-2010, 03:28 AM
Celejar
 
Default Automatic partial archive mirroring

On Mon, 31 May 2010 01:40:59 -0000
Cameron Hutchison <lists@xdna.net> wrote:

> I have a few debian unstable boxes that I like to keep up-to-date.
>
> Currently, I run apt-cacher-ng (a proxy for apt-get which stores
> packages so they don't need to be downloaded again) on my gateway box
> (lenny 32-bit), so I don't double download from my couple of other boxes
> (sid 64-bit). This works well enough, but I want to make it better.
>
> My internet connection has an off-peak period (2am-8am) where downloads
> are not counted in the monthly quota. I want my downloads to occur
> during this window, automatically.
>
> The problem is that the sid boxes are not powered-up during this window
> so I cannot just simply schedule a cronjob to do an apt-get -d
> dist-upgrade.
>
> I would like my lenny box to do this on behalf of my sid boxes. I dont
> really want to mirror the whole archive. Is there some way to do an
> intelligent partial mirror of the archive of what is installed on other
> boxes, without too much overhead of managing a package list? (i.e. I dont
> want to have to manually update some package list on my lenny box when I
> install a new package on a sid box).
>
> I've looked at apt-proxy, apt-cacher, apt-cacher-ng and approx, but they
> all appear to download on demand, not according to a schedule.

You could use use the output of 'dpkg --get-selections' on the Sid
boxes, pruning it using grep or similar to keep only the lines ending
with 'install'. Combine the results from the various Sid boxes, and
then have the gateway box get those packages. This could be scripted
easily enough, and there would be no need for manually updating
anything.

Celejar
--
foffl.sourceforge.net - Feeds OFFLine, an offline RSS/Atom aggregator
mailmin.sourceforge.net - remote access via secure (OpenPGP) email
ssuds.sourceforge.net - A Simple Sudoku Solver and Generator


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100530232849.13be6e74.celejar@gmail.com">http://lists.debian.org/20100530232849.13be6e74.celejar@gmail.com
 
Old 05-31-2010, 04:09 AM
Cameron Hutchison
 
Default Automatic partial archive mirroring

Celejar <celejar@gmail.com> writes:
>Cameron Hutchison <lists@xdna.net> wrote:

>> Is there some way to do an
>> intelligent partial mirror of the archive of what is installed on other
>> boxes, without too much overhead of managing a package list? (i.e. I dont
>> want to have to manually update some package list on my lenny box when I
>> install a new package on a sid box).

>You could use use the output of 'dpkg --get-selections' on the Sid
>boxes, pruning it using grep or similar to keep only the lines ending
>with 'install'.

ok. that's easy.

>Combine the results from the various Sid boxes,

so's that.

>and then have the gateway box get those packages.

hmmm. How? Is there an existing tool that will do this? Doing it
manually (in a script) would require too much work (essentially
implementing apt-get -d dist-upgrade against a specified package index
and package list).


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: c07.4c03366f.8a7ce@getafix.xdna.net">http://lists.debian.org/c07.4c03366f.8a7ce@getafix.xdna.net
 
Old 05-31-2010, 04:35 AM
Celejar
 
Default Automatic partial archive mirroring

On Mon, 31 May 2010 04:09:19 -0000
Cameron Hutchison <lists@xdna.net> wrote:

> Celejar <celejar@gmail.com> writes:
> >Cameron Hutchison <lists@xdna.net> wrote:
>
> >> Is there some way to do an
> >> intelligent partial mirror of the archive of what is installed on other
> >> boxes, without too much overhead of managing a package list? (i.e. I dont
> >> want to have to manually update some package list on my lenny box when I
> >> install a new package on a sid box).
>
> >You could use use the output of 'dpkg --get-selections' on the Sid
> >boxes, pruning it using grep or similar to keep only the lines ending
> >with 'install'.
>
> ok. that's easy.
>
> >Combine the results from the various Sid boxes,
>
> so's that.
>
> >and then have the gateway box get those packages.
>
> hmmm. How? Is there an existing tool that will do this? Doing it
> manually (in a script) would require too much work (essentially
> implementing apt-get -d dist-upgrade against a specified package index
> and package list).

How about using 'aptitude download', possibly feeding the package list
to aptitude with xargs? The 'download' action downloads to the current
directory, so I suppose that you'd use a temp directory, and that would
have the side effect of getting a copy stored by the cacher.

Celejar
--
foffl.sourceforge.net - Feeds OFFLine, an offline RSS/Atom aggregator
mailmin.sourceforge.net - remote access via secure (OpenPGP) email
ssuds.sourceforge.net - A Simple Sudoku Solver and Generator


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100531003508.36f3ddde.celejar@gmail.com">http://lists.debian.org/20100531003508.36f3ddde.celejar@gmail.com
 
Old 05-31-2010, 05:10 AM
Cameron Hutchison
 
Default Automatic partial archive mirroring

Celejar <celejar@gmail.com> writes:

>Cameron Hutchison <lists@xdna.net> wrote:

>> >and then have the gateway box get those packages.
>>
>> hmmm. How? Is there an existing tool that will do this? Doing it
>> manually (in a script) would require too much work (essentially
>> implementing apt-get -d dist-upgrade against a specified package index
>> and package list).

>How about using 'aptitude download', possibly feeding the package list
>to aptitude with xargs? The 'download' action downloads to the current
>directory, so I suppose that you'd use a temp directory, and that would
>have the side effect of getting a copy stored by the cacher.

The problem is the the dpkg architecture (and distribution) of the host
is different to the targets. 'aptitude download' is going to use the
native dpkg architecture, and is going to look in /etc/apt/sources.list
for the sources to use.

It looks like it may be possible to use something like 'aptitude
extract-cache-subset' on the targets and set Apt::Architecture to the
target on the command line. apt.conf(5) has a number of details of how I
can override most of the host environment.

Hmmm, I can see how this might work now.

* periodically run aptitude extract-cache-subset on the targets and
send that to the host
* every night on the host, run 'aptitude -d dist-upgrade', but
overriding all the directories for configs, indexes, pkgstates, etc and
architecture to point to what has been captured by extract-cache-subset.
Also point aptitude to the local apt-cacher-ng proxy so all downloads
go through it.
* delete the downloaded files, since apt-cacher-ng has already cached
them.

Thanks for the ideas. Perhaps I can knock this up fairly easily. I'll
certainly learn more about apt than I probably want to :-)



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: eb9.4c0344bd.22067@getafix.xdna.net">http://lists.debian.org/eb9.4c0344bd.22067@getafix.xdna.net
 
Old 05-31-2010, 06:27 AM
Andrei Popescu
 
Default Automatic partial archive mirroring

On Mon,31.May.10, 01:40:59, Cameron Hutchison wrote:
>
> I've looked at apt-proxy, apt-cacher, apt-cacher-ng and approx, but they
> all appear to download on demand, not according to a schedule.

Maybe apt-zip or apt-offline (not in lenny) can be used for what you
need.

Regards,
Andrei
--
Offtopic discussions among Debian users and developers:
http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic
 
Old 05-31-2010, 05:11 PM
 
Default Automatic partial archive mirroring

On Mon, May 31, 2010 at 01:40:59AM -0000, Cameron Hutchison wrote:
> I have a few debian unstable boxes that I like to keep up-to-date.
>
> Currently, I run apt-cacher-ng (a proxy for apt-get which stores
> packages so they don't need to be downloaded again) on my gateway box
> (lenny 32-bit), so I don't double download from my couple of other boxes
> (sid 64-bit). This works well enough, but I want to make it better.
>
> My internet connection has an off-peak period (2am-8am) where downloads
> are not counted in the monthly quota. I want my downloads to occur
> during this window, automatically.
>
> The problem is that the sid boxes are not powered-up during this window
> so I cannot just simply schedule a cronjob to do an apt-get -d
> dist-upgrade.
>
> I would like my lenny box to do this on behalf of my sid boxes. I dont
> really want to mirror the whole archive. Is there some way to do an
> intelligent partial mirror of the archive of what is installed on other
> boxes, without too much overhead of managing a package list? (i.e. I dont
> want to have to manually update some package list on my lenny box when I
> install a new package on a sid box).
>
> I've looked at apt-proxy, apt-cacher, apt-cacher-ng and approx, but they
> all appear to download on demand, not according to a schedule.

Take a look at debmirror's options. You can choose to download by
dist, section, protocol, arch, source/no source, include/exclude packages by
regexp matching, etc...
It can be easily scheduled and you can serve the packages afterwards
adding a

deb http://lighttpd.at.lenny.gw sid main contrib

to the other boxes' /etc/apt/sources.list.d/lenny.gw.list, for
example.

Would that be useful for you?

Obvious drawback: you'll possibly be pulling more than you need.

--
Huella de clave primaria: 0FDA C36F F110 54F4 D42B D0EB 617D 396C 448B 31EB
 

Thread Tools




All times are GMT. The time now is 11:41 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org