On 03/05/10 02:10, Pierre Schmitz wrote:

On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin<aaronmgriffin@gmail.com>
wrote:

How's this?


http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad 2aee5

Looks good to me.


The issue left is the community repo.


Ideas?


Well, if we want to use the same scripts for community and the same layout
we have to put them into different root dirs (FTP_BASE) as you already
suggested. Maybe we should also cleanup the root dir auf our ftp and put
repos into a sub dir.

What about a directory structure like this? We could also remove the os
subdir and backwards compatibility could be achieved by some symlinks.

ftp
└── repo
├── arch
│ ├── core
│ ├── extra
│ ├── packages
│ └── testing
└── community
├── community
├── community-testing
└── packages

The actual names might not be final, but with this structure we separate
our repos from everything else on the ftp and we separate the "official"
and community repos from each other.



That brings back the whole initial rsync hell we are trying to avoid by
not moving everything to the packages directory.


Allan

On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de>
wrote:
> What about a directory structure like this?

This change would be quite extreme though and it would be probably easier
to really issue a complete resync if we actually do that. The
"compatibility symlinks" would just be there to not break "old"
mirrorlists. But in this case it might be worth it.

--
Pierre Schmitz, https://users.archlinux.de/~pierre

On Mon, 03 May 2010 02:16:44 +1000, Allan McRae <allan@archlinux.org>
wrote:
> That brings back the whole initial rsync hell we are trying to avoid by
> not moving everything to the packages directory.

Yes, as I wrote in my other mail. This would be a solution to another
problem: separating the "arch" and community repo. My proposed layout has
its advantages but the transition is really ugly.

--
Pierre Schmitz, https://users.archlinux.de/~pierre

On 03/05/10 02:20, Pierre Schmitz wrote:

On Mon, 03 May 2010 02:16:44 +1000, Allan McRae<allan@archlinux.org>
wrote:

That brings back the whole initial rsync hell we are trying to avoid by
not moving everything to the packages directory.


Yes, as I wrote in my other mail. This would be a solution to another
problem: separating the "arch" and community repo. My proposed layout has
its advantages but the transition is really ugly.


If we are prepared to accept an ugly transition (and a several day
rsync), then I like the layout that you propose.

On 05/02/2010 07:10 PM, Pierre Schmitz wrote:

On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin<aaronmgriffin@gmail.com>
wrote:

How's this?


http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad 2aee5

Looks good to me.


The issue left is the community repo.


Ideas?


Well, if we want to use the same scripts for community and the same layout
we have to put them into different root dirs (FTP_BASE) as you already
suggested. Maybe we should also cleanup the root dir auf our ftp and put
repos into a sub dir.

What about a directory structure like this? We could also remove the os
subdir and backwards compatibility could be achieved by some symlinks.

ftp
└── repo
├── arch
│ ├── core
│ ├── extra
│ ├── packages
│ └── testing
└── community
├── community
├── community-testing
└── packages

The actual names might not be final, but with this structure we separate
our repos from everything else on the ftp and we separate the "official"
and community repos from each other.



if i understand well using this schema we can't have a generic server
link and only one mirrorlist file?


--
Ionut

On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de>
wrote:
> What about a directory structure like this? We could also remove the os
> subdir and backwards compatibility could be achieved by some symlinks.
>
> ftp
> └── repo
> ├── arch
> │Â*Â* ├── core
> │Â*Â* ├── extra
> │Â*Â* ├── packages
> │Â*Â* └── testing
> └── community
> ├── community
> ├── community-testing
> └── packages
>
> The actual names might not be final, but with this structure we separate
> our repos from everything else on the ftp and we separate the "official"
> and community repos from each other.

Here is a proposal which is more backwards-compatible and doesn't need a
complete resync; just a few modificatinos to the db-scripts. (e.g. we need
to define which repos the cleanup-script should check)

ftp
├── core
├── extra
├── testing
├── community
├── community-testing
└── packages
├── arch
│ ├── i686
│ ├── x86_64
│ └── any
└── community
├── i686
├── x86_64
└── any

The top dir repo dirs are kept. They include symlinks to the packages in
the packages dir. Packages from community are kept separate. We now just
need to modify the db-scripts and define the repository names it should
work on. (especialy the cleanup-script should only search in core,extra and
testing but not community for example.

The community, community-testing and packages/community dirs would still
be rsynced from sigurd.

What do you think about this layout? It wont need any chagne on the
clients (mirrorlists still work) and no resyncs are needed.

Greetings,

Pierre

--
Pierre Schmitz, https://users.archlinux.de/~pierre

On Mon, May 3, 2010 at 5:59 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> What about a directory structure like this? We could also remove the os
>> subdir and backwards compatibility could be achieved by some symlinks.
>>
>> ftp
>> └── repo
>> Â* Â* ├── arch
>> Â* Â* │Â*Â* ├── core
>> Â* Â* │Â*Â* ├── extra
>> Â* Â* │Â*Â* ├── packages
>> Â* Â* │Â*Â* └── testing
>> Â* Â* └── community
>> Â* Â* Â* Â* ├── community
>> Â* Â* Â* Â* ├── community-testing
>> Â* Â* Â* Â* └── packages
>>
>> The actual names might not be final, but with this structure we separate
>> our repos from everything else on the ftp and we separate the "official"
>> and community repos from each other.
>
> Here is a proposal which is more backwards-compatible and doesn't need a
> complete resync; just a few modificatinos to the db-scripts. (e.g. we need
> to define which repos the cleanup-script should check)
>
> Â*ftp
> Â*├── core
> Â*├── extra
> Â*├── testing
> Â*├── community
> Â*├── community-testing
> Â*└── packages
> Â* Â* ├── arch
> Â* Â* │ Â* ├── i686
> Â* Â* │ Â* ├── x86_64
> Â* Â* │ Â* └── any
> Â* Â* └── community
> Â* Â* Â* Â* ├── i686
> Â* Â* Â* Â* ├── x86_64
> Â* Â* Â* Â* └── any
>
> The top dir repo dirs are kept. They include symlinks to the packages in
> the packages dir. Packages from community are kept separate. We now just
> need to modify the db-scripts and define the repository names it should
> work on. (especialy the cleanup-script should only search in core,extra and
> testing but not community for example.
>
> The community, community-testing and packages/community dirs would still
> be rsynced from sigurd.
>
> What do you think about this layout? It wont need any chagne on the
> clients (mirrorlists still work) and no resyncs are needed.

In the spirit of this, I broke out the "packages" dir to a variable so
that we can change it:
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475 fb4a3

On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)

On Mon, May 3, 2010 at 11:19 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
> On Mon, May 3, 2010 at 5:59 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
>> On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de>
>> wrote:
>>> What about a directory structure like this? We could also remove the os
>>> subdir and backwards compatibility could be achieved by some symlinks.
>>>
>>> ftp
>>> └── repo
>>> Â* Â* ├── arch
>>> Â* Â* │Â*Â* ├── core
>>> Â* Â* │Â*Â* ├── extra
>>> Â* Â* │Â*Â* ├── packages
>>> Â* Â* │Â*Â* └── testing
>>> Â* Â* └── community
>>> Â* Â* Â* Â* ├── community
>>> Â* Â* Â* Â* ├── community-testing
>>> Â* Â* Â* Â* └── packages
>>>
>>> The actual names might not be final, but with this structure we separate
>>> our repos from everything else on the ftp and we separate the "official"
>>> and community repos from each other.
>>
>> Here is a proposal which is more backwards-compatible and doesn't need a
>> complete resync; just a few modificatinos to the db-scripts. (e.g. we need
>> to define which repos the cleanup-script should check)
>>
>> Â*ftp
>> Â*├── core
>> Â*├── extra
>> Â*├── testing
>> Â*├── community
>> Â*├── community-testing
>> Â*└── packages
>> Â* Â* ├── arch
>> Â* Â* │ Â* ├── i686
>> Â* Â* │ Â* ├── x86_64
>> Â* Â* │ Â* └── any
>> Â* Â* └── community
>> Â* Â* Â* Â* ├── i686
>> Â* Â* Â* Â* ├── x86_64
>> Â* Â* Â* Â* └── any
>>
>> The top dir repo dirs are kept. They include symlinks to the packages in
>> the packages dir. Packages from community are kept separate. We now just
>> need to modify the db-scripts and define the repository names it should
>> work on. (especialy the cleanup-script should only search in core,extra and
>> testing but not community for example.
>>
>> The community, community-testing and packages/community dirs would still
>> be rsynced from sigurd.
>>
>> What do you think about this layout? It wont need any chagne on the
>> clients (mirrorlists still work) and no resyncs are needed.
>
> In the spirit of this, I broke out the "packages" dir to a variable so
> that we can change it:
> http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475 fb4a3
>
> On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)

Though... we'd still need to get rid of that silly "os" dir as part of this.

On Mon, May 3, 2010 at 12:19, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
> Though... we'd still need to get rid of that silly "os" dir as part of this.
>

The rankmirrors script will also need to be updated for that I believe.

On Mon, 3 May 2010 11:19:35 -0500, Aaron Griffin <aaronmgriffin@gmail.com>
wrote:
>> In the spirit of this, I broke out the "packages" dir to a variable so
>> that we can change it:
>>
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475 fb4a3
>>
>> On gerolde, it'd be packages/arch and on sigurd, packages/community (in
>> theory)

Yes, that should be fine this way. Could you merge this into master and
pull to gerolde? Two things I wanted to chagne:
* add a localconfig file which optionally overrides the config options
(this way we can still manage the main configs in git, but adjust settings
for community etc.)
* we need to define a list of repos the cleanup-script (especially this
find call) should look into.

> Though... we'd still need to get rid of that silly "os" dir as part of
> this.

I don't think it's worth to have a complete resync and adjusting client's
configs just to remove this dir. It doesn't hurt that much and who knows
for what reason it was created. So, I would prefer a way with least
breakage.

--
Pierre Schmitz, https://users.archlinux.de/~pierre