FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Development

 
 
LinkBack Thread Tools
 
Old 05-04-2010, 09:39 AM
Roman Kyrylych
 
Default dbscripts pkg pools

On 2010-05-03, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> What about a directory structure like this? We could also remove the os
>> subdir and backwards compatibility could be achieved by some symlinks.
>>
>> ftp
>> └── repo
>> ├── arch
>> │ ├── core
>> │ ├── extra
>> │ ├── packages
>> │ └── testing
>> └── community
>> ├── community
>> ├── community-testing
>> └── packages
>>
>> The actual names might not be final, but with this structure we separate
>> our repos from everything else on the ftp and we separate the "official"
>> and community repos from each other.

I like this.

>
> Here is a proposal which is more backwards-compatible and doesn't need a
> complete resync; just a few modificatinos to the db-scripts. (e.g. we need
> to define which repos the cleanup-script should check)
>
> ftp
> ├── core
> ├── extra
> ├── testing
> ├── community
> ├── community-testing
> └── packages
> ├── arch
> │ ├── i686
> │ ├── x86_64
> │ └── any
> └── community
> ├── i686
> ├── x86_64
> └── any
>
> The top dir repo dirs are kept. They include symlinks to the packages in
> the packages dir. Packages from community are kept separate. We now just
> need to modify the db-scripts and define the repository names it should
> work on. (especialy the cleanup-script should only search in core,extra and
> testing but not community for example.
>
> The community, community-testing and packages/community dirs would still
> be rsynced from sigurd.
>
> What do you think about this layout? It wont need any chagne on the
> clients (mirrorlists still work) and no resyncs are needed.

I fail to see why resyncs are not needed in this situation.
Let's suppose we have /foo/bar on a downstream mirror,
and upstream has /foo/bar -> /baz/bar
will rsync guess that it should not download bar and instead just copy
it from foo to baz locally?

--
Roman Kyrylych (*оман Кирилич)
 
Old 05-04-2010, 10:13 AM
Pierre Schmitz
 
Default dbscripts pkg pools

On Tue, 4 May 2010 12:39:33 +0300, Roman Kyrylych
<roman.kyrylych@gmail.com> wrote:
> I fail to see why resyncs are not needed in this situation.
> Let's suppose we have /foo/bar on a downstream mirror,
> and upstream has /foo/bar -> /baz/bar
> will rsync guess that it should not download bar and instead just copy
> it from foo to baz locally?

It's the other way round we keep the old dirs; so packages will be
replaced by symlinks to the packages dir when they got updated.

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 05-04-2010, 11:57 AM
Aaron Griffin
 
Default dbscripts pkg pools

On Tue, May 4, 2010 at 3:26 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Mon, 3 May 2010 11:19:35 -0500, Aaron Griffin <aaronmgriffin@gmail.com>
> wrote:
>>> In the spirit of this, I broke out the "packages" dir to a variable so
>>> that we can change it:
>>>
> http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475 fb4a3
>>>
>>> On gerolde, it'd be packages/arch and on sigurd, packages/community (in
>>> theory)
>
> Yes, that should be fine this way. Could you merge this into master and
> pull to gerolde? Two things I wanted to chagne:
> * add a localconfig file which optionally overrides the config options
> (this way we can still manage the main configs in git, but adjust settings
> for community etc.)

What's wrong with just changing the config file for community? That
was the intent. If you modify it, and commit it locally, everything
will work out fine if you "git pull --rebase". Adding a localconfig
just seems like extra complexity for no real reason.

> * we need to define a list of repos the cleanup-script (especially this
> find call) should look into.

Can we use the get_repos_for_host for this?
 
Old 05-04-2010, 03:16 PM
Roman Kyrylych
 
Default dbscripts pkg pools

On Sun, May 2, 2010 at 19:10, Pierre Schmitz <pierre@archlinux.de> wrote:
> What about a directory structure like this? We could also remove the os
> subdir and backwards compatibility could be achieved by some symlinks.
>
> ftp
> └── repo
> * *├── arch
> * *│** ├── core
> * *│** ├── extra
> * *│** ├── packages
> * *│** └── testing
> * *└── community
> * * * *├── community
> * * * *├── community-testing
> * * * *└── packages
>
> The actual names might not be final, but with this structure we separate
> our repos from everything else on the ftp and we separate the "official"
> and community repos from each other.

Regardless of the internal structure of arch and community trees,
do we _want_ to separate them?
If it's just because of cleanup scripts, access rights etc.,
then it can be solved this way:

arch - packages from gerolde go there
pkg - packages are written/removed by db-scripts here
pkg1.i686.tar.xz
pkg2.any.tar.xz
pkg3.x86_64.tar.xz
repo - db files and symlinks are written/removed by db-scripts here
testing
i686
testing.db.tar.gz
pkg1.i686.tar.xz -> ../../../pkg/pkg1.i686.tar.xz
pkg2.any.tar.xz -> ../../../pkg/pkg2.any.tar.xz
x86_64
core
extra
community - packages from sigurd go there
pkg
repo
community-testing
community
ftp - this is what users will see
iso
pkg - union mount of arch/pkg and community/pkg
repo - union mount of arch/repo and community/repo

The magic:
mount -t aufs -o br=/arch/pkg=ro+wh:/community/pkg=ro+wh none /ftp/pkg
mount -t aufs -o br=/arch/repo=ro+wh:/community/repo=ro+wh none /ftp/repo

Once union mounts hit the upstream it will be even easier:
mount /arch/pkg /ftp/pkg
mount --union /community/pkg /ftp/pkg
mount /arch/repo /ftp/pkg
mount --union /community/repo /ftp/pkg

Pros/cons of this particular scheme:
* no arch/community separation from the user point of view
* no os prefix
* one huge pool of packages
* packages can be moved even between official and community repos
without the need for mirrors/users to redownload them
* all packages must have arch suffix
* a big initial resync is needed

Opinions?

P.S.: this is just an idea, I'm not going to argue about it,
so if you don't like it - just ignore it.

--
Roman Kyrylych (*оман Кирилич)
 
Old 05-07-2010, 12:35 PM
Pierre Schmitz
 
Default dbscripts pkg pools

On Tue, 4 May 2010 06:57:12 -0500, Aaron Griffin <aaronmgriffin@gmail.com>
wrote:
>> * add a localconfig file which optionally overrides the config options
>> (this way we can still manage the main configs in git, but adjust
>> settings
>> for community etc.)
>
> What's wrong with just changing the config file for community? That
> was the intent. If you modify it, and commit it locally, everything
> will work out fine if you "git pull --rebase". Adding a localconfig
> just seems like extra complexity for no real reason.

You are right; let's do it that way.

>> * we need to define a list of repos the cleanup-script (especially this
>> find call) should look into.
>
> Can we use the get_repos_for_host for this?

Yes, good catch. Forgot about that function. Just loop through the repo
dirs and execute the function added by this commit:
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=22c5f235fa25ea3327d95a28c6dc8ae399d 6cb9a

Btw: I cannot checkout your repo "warning: remote HEAD refers to
nonexistent ref, unable to checkout." Could you push it to gerolde/master?
I think we are very close to get this stuff ready for production.

Greetings,

Pierre

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 05-07-2010, 12:43 PM
Pierre Schmitz
 
Default dbscripts pkg pools

On Tue, 4 May 2010 18:16:25 +0300, Roman Kyrylych
<roman.kyrylych@gmail.com> wrote:
> Opinions?
>
> P.S.: this is just an idea, I'm not going to argue about it,
> so if you don't like it - just ignore it.

But I'll add some points why I don't like it that much though: ;-)
* if I got it right this will also require a complete resync
* it's way to complicated
* aufs is slow and crashes the kernel the hard way on heavy load. (that's
why I removed its usage from makechrootpkg)

The overlay magic might look nice in theory but the risk of a major mess
is too high imho.


--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 05-28-2010, 10:13 AM
Pierre Schmitz
 
Default dbscripts pkg pools

On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz <pierre@archlinux.de>
wrote:
> Btw: I cannot checkout your repo "warning: remote HEAD refers to
> nonexistent ref, unable to checkout." Could you push it to
gerolde/master?
> I think we are very close to get this stuff ready for production.

bump.

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 05-28-2010, 06:36 PM
Aaron Griffin
 
Default dbscripts pkg pools

On Fri, May 28, 2010 at 5:13 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> Btw: I cannot checkout your repo "warning: remote HEAD refers to
>> nonexistent ref, unable to checkout." Could you push it to
> gerolde/master?
>> I think we are very close to get this stuff ready for production.
>
> bump.

Pushed
 
Old 06-20-2010, 04:57 PM
Pierre Schmitz
 
Default dbscripts pkg pools

On Fri, 28 May 2010 13:36:04 -0500, Aaron Griffin
<aaronmgriffin@gmail.com> wrote:
> On Fri, May 28, 2010 at 5:13 AM, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz
<pierre@archlinux.de>
>> wrote:
>>> Btw: I cannot checkout your repo "warning: remote HEAD refers to
>>> nonexistent ref, unable to checkout." Could you push it to
>> gerolde/master?
>>> I think we are very close to get this stuff ready for production.
>>
>> bump.
>
> Pushed

Thanks. I made the adjustments. Would be nice if someone would have a
quick look at my changes: http://projects.archlinux.org/dbscripts.git/log/

What I just noticed: We had the ftp-arch group for those devs who are able
to add packages to the core repo. This distinction wont be really doable
with a package pool. So we might want to remove that completely.

What is missing?
1) Do a little test locally (help is more than welcome here)
2) Someone with root access on gerolde/sigurd:
gerolde: install -d -o ftp -g ftp-extra
/srv/ftp/packages/arch/{i686,x86_64,any}
rsync the packages/community dir from sigurd
git pull new db-scripts
sigurd: install -d -o root -g tusers
/srv/ftp/packages/community/{i686,x86_64,any}
git pull new db-scripts
3) Do we need to adjust our rsync modules for our mirrors?

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 06-21-2010, 06:31 PM
Aaron Griffin
 
Default dbscripts pkg pools

On Sun, Jun 20, 2010 at 11:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> 1) Do a little test locally (help is more than welcome here)

Testing the db-scripts is always tedious. Any ideas to speed up
testing would help here.

> 2) Someone with root access on gerolde/sigurd:
> * * gerolde: install -d -o ftp -g ftp-extra
> /srv/ftp/packages/arch/{i686,x86_64,any}
> * * * * * * *rsync the packages/community dir from sigurd
> * * * * * * *git pull new db-scripts
> * * sigurd: *install -d -o root -g tusers
> /srv/ftp/packages/community/{i686,x86_64,any}
> * * * * * * *git pull new db-scripts

Seems right.

> 3) Do we need to adjust our rsync modules for our mirrors?

Yeah, I think so - the package pool dirs need to be in there, or else
the symlinks will be broken.
 

Thread Tools




All times are GMT. The time now is 02:04 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org