dbscripts pkg pools
 

On Wed, 28 Apr 2010 11:56:53 -0500, Aaron Griffin
<aaronmgriffin@gmail.com> wrote:
> Hello all,
> I've had these patches sitting around for a bit:
> http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=pkgpools
>
> I'd like to see if someone else has the time to test them.
>
> I have tested them with the following procedure:
> * created ./testing/{staging,myrepo,packages,package-cleanup}/ with
> necessary subdirs
> * modified config to point to these paths
> * commented out all the SVN malarky in
db-add/db-remove/ftpdir-cleanup/etc
> * changed get_repos_for_host() to return "myrepo"
>
> Testing:
> * Add a package to myrepo
> - Package file in packages/os/*/
> - Relative symlink in myrepo/os/*/
> - In myrepo DB
> * Remove package from repo 'foo'
> - Removed from myrepo DB
> * Run cron-jobs/ftpdir-cleanup
> - Symlink is removed from myrepo/os/*/
> - Package file moved from packages/os/*/ to package-cleanup/
>
> Anyone else want to verify this, or should I push it to master?

Thanks for your heads-up. That was on my todo list but atm I am not really
able to catch up. But those patches are a lot more readable than those from
your working branch which virtually changed every line of code. I'll have a
closer look at those soon.

For now:
* Does this also work with the old layout/packages? (especially the
cleanup and move scripts)
* I still don't know how to handle the community and community-testing
repo. We cannot really put those packages into the same folder as
core,extra,testing. That could break the cleanup-scripts (different svn)
and definitely will break the rsync process from sigurd to gerolde.
* Do all packages have the $arch within their filename? (community might
be a problem here as they are not using db-scripts for that long)

--
Pierre Schmitz, https://users.archlinux.de/~pierre

dbscripts pkg pools
 

On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Wed, 28 Apr 2010 11:56:53 -0500, Aaron Griffin
> <aaronmgriffin@gmail.com> wrote:
>> Hello all,
>> I've had these patches sitting around for a bit:
>> http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=pkgpools
>>
>> I'd like to see if someone else has the time to test them.
>>
>> I have tested them with the following procedure:
>> * created ./testing/{staging,myrepo,packages,package-cleanup}/ with
>> necessary subdirs
>> * modified config to point to these paths
>> * commented out all the SVN malarky in
> db-add/db-remove/ftpdir-cleanup/etc
>> * changed get_repos_for_host() to return "myrepo"
>>
>> Testing:
>> * Add a package to myrepo
>> * *- Package file in packages/os/*/
>> * *- Relative symlink in myrepo/os/*/
>> * *- In myrepo DB
>> * Remove package from repo 'foo'
>> * *- Removed from myrepo DB
>> * Run cron-jobs/ftpdir-cleanup
>> * *- Symlink is removed from myrepo/os/*/
>> * *- Package file moved from packages/os/*/ to package-cleanup/
>>
>> Anyone else want to verify this, or should I push it to master?
>
> Thanks for your heads-up. That was on my todo list but atm I am not really
> able to catch up. But those patches are a lot more readable than those from
> your working branch which virtually changed every line of code. I'll have a
> closer look at those soon.

Yeah, there's two patches in the working branch that are totally
unrelated, so I dumped them for the purposes of these changes.

> For now:
> * Does this also work with the old layout/packages? (especially the
> cleanup and move scripts)

No. What will have to happen is that all packages will need to be
moved to /srv/ftp/packages/os/*/ and symlinked to where they actually
end up (with relative symlinks, for the purposes of rsync). This can
be scripted away fairly quickly.

> * I still don't know how to handle the community and community-testing
> repo. We cannot really put those packages into the same folder as
> core,extra,testing. That could break the cleanup-scripts (different svn)
> and definitely will break the rsync process from sigurd to gerolde.

Hmm, if the new DB scripts are placed on there, it will have it's own
/srv/ftp/packages dir and relative symlinks. This is going to be very
hard to rsync in.

One possibility is to move all of community to a subtree in our ftp
layout, i.e. /srv/ftp/repos/other/{community, ...}. This would be a
fairly drastic change, but would make this easier.

> * Do all packages have the $arch within their filename? (community might
> be a problem here as they are not using db-scripts for that long)

What is this referencing? Is this related to the ARCHES loop I added?

dbscripts pkg pools
 

On 29/04/10 03:42, Aaron Griffin wrote:

On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de> wrote:

<snip>


* Does this also work with the old layout/packages? (especially the
cleanup and move scripts)


No. What will have to happen is that all packages will need to be
moved to /srv/ftp/packages/os/*/ and symlinked to where they actually
end up (with relative symlinks, for the purposes of rsync). This can
be scripted away fairly quickly.


That is going to be a painful initial rsync... Is it just the clean-up
script that can not handle the change?




* Do all packages have the $arch within their filename? (community might
be a problem here as they are not using db-scripts for that long)


All package in [core] and [extra] have $arch in their filename as of a
month or two ago. There is no way community is even close.


Allan

dbscripts pkg pools
 

On Wed, Apr 28, 2010 at 6:39 PM, Allan McRae <allan@archlinux.org> wrote:
> On 29/04/10 03:42, Aaron Griffin wrote:
>>
>> On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de>
>> *wrote:
>
> <snip>
>
>>> * Does this also work with the old layout/packages? (especially the
>>> cleanup and move scripts)
>>
>> No. What will have to happen is that all packages will need to be
>> moved to /srv/ftp/packages/os/*/ and symlinked to where they actually
>> end up (with relative symlinks, for the purposes of rsync). This can
>> be scripted away fairly quickly.
>
> That is going to be a painful initial rsync... *Is it just the clean-up
> script that can not handle the change?

I'm sure we could figure something out... the cleanup script does one
thing that is incompatible: it removes (rm) files from each repo if
not in the DB - this assumes it is a symlink. It should be fairly easy
to replace that with a "if symlink, rm, else mv to cleanup"

Can anyone else see anything incompatible here?

dbscripts pkg pools
 

On Thu, 29 Apr 2010 09:39:52 +1000, Allan McRae <allan@archlinux.org>
wrote:
>>> * Do all packages have the $arch within their filename? (community
might
>>> be a problem here as they are not using db-scripts for that long)
>
> All package in [core] and [extra] have $arch in their filename as of a
> month or two ago. There is no way community is even close.

Forget about this. I thought al packages were in the /packages dir, but
Aaron created subdirs for each arch. So no problem here.

--
Pierre Schmitz, https://users.archlinux.de/~pierre

dbscripts pkg pools
 

On 04/29/2010 02:39 AM, Allan McRae wrote:

On 29/04/10 03:42, Aaron Griffin wrote:

On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de>
wrote:

<snip>


* Does this also work with the old layout/packages? (especially the
cleanup and move scripts)


No. What will have to happen is that all packages will need to be
moved to /srv/ftp/packages/os/*/ and symlinked to where they actually
end up (with relative symlinks, for the purposes of rsync). This can
be scripted away fairly quickly.


That is going to be a painful initial rsync... Is it just the clean-up
script that can not handle the change?



* Do all packages have the $arch within their filename? (community might
be a problem here as they are not using db-scripts for that long)


All package in [core] and [extra] have $arch in their filename as of a
month or two ago. There is no way community is even close.

Allan


that is not a problem. we can always rebuild all of them. the one that
can't be rebuilt we move it into aur.


--
Ionut

dbscripts pkg pools
 

On 03/05/10 00:46, Ionut Biru wrote:

On 04/29/2010 02:39 AM, Allan McRae wrote:

On 29/04/10 03:42, Aaron Griffin wrote:

On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de>
wrote:

<snip>


* Does this also work with the old layout/packages? (especially the
cleanup and move scripts)


No. What will have to happen is that all packages will need to be
moved to /srv/ftp/packages/os/*/ and symlinked to where they actually
end up (with relative symlinks, for the purposes of rsync). This can
be scripted away fairly quickly.


That is going to be a painful initial rsync... Is it just the clean-up
script that can not handle the change?



* Do all packages have the $arch within their filename? (community
might
be a problem here as they are not using db-scripts for that long)


All package in [core] and [extra] have $arch in their filename as of a
month or two ago. There is no way community is even close.

Allan


that is not a problem. we can always rebuild all of them. the one that
can't be rebuilt we move it into aur.


You are a bit keen.... it is over half the community repo (957/1874)
that would need rebuilt.


If you really want to do some mindless rebuilding, see
http://wiki.archlinux.org/index.php/DeveloperWiki:Unimportant_Rebuild_List .


Allan

dbscripts pkg pools
 

On Thu, 29 Apr 2010 12:05:19 -0500, Aaron Griffin
<aaronmgriffin@gmail.com> wrote:
>> That is going to be a painful initial rsync... *Is it just the clean-up
>> script that can not handle the change?
>
> I'm sure we could figure something out... the cleanup script does one
> thing that is incompatible: it removes (rm) files from each repo if
> not in the DB - this assumes it is a symlink. It should be fairly easy
> to replace that with a "if symlink, rm, else mv to cleanup"
>
> Can anyone else see anything incompatible here?

I think its really only the cleanup script that is directly affected.
db-update doesn't need to know about old packages, db-move and db-remove
only work on db files and use mv which should also work with symlinks.

I think it's definetly worth to modify the cleanup script to handle both
layouts...a complete resync will produce a lot of unnessecary traffic.

The issue left is the community repo.

--
Pierre Schmitz, https://users.archlinux.de/~pierre

dbscripts pkg pools
 

On Sun, May 2, 2010 at 9:53 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
> On Thu, 29 Apr 2010 12:05:19 -0500, Aaron Griffin
> <aaronmgriffin@gmail.com> wrote:
>>> That is going to be a painful initial rsync... *Is it just the clean-up
>>> script that can not handle the change?
>>
>> I'm sure we could figure something out... the cleanup script does one
>> thing that is incompatible: it removes (rm) files from each repo if
>> not in the DB - this assumes it is a symlink. It should be fairly easy
>> to replace that with a "if symlink, rm, else mv to cleanup"
>>
>> Can anyone else see anything incompatible here?
>
> I think its really only the cleanup script that is directly affected.
> db-update doesn't need to know about old packages, db-move and db-remove
> only work on db files and use mv which should also work with symlinks.
>
> I think it's definetly worth to modify the cleanup script to handle both
> layouts...a complete resync will produce a lot of unnessecary traffic.

How's this?
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad 2aee5

> The issue left is the community repo.

Ideas?

dbscripts pkg pools
 

On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin <aaronmgriffin@gmail.com>
wrote:
> How's this?
>
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad 2aee5

Looks good to me.

>> The issue left is the community repo.
>
> Ideas?

Well, if we want to use the same scripts for community and the same layout
we have to put them into different root dirs (FTP_BASE) as you already
suggested. Maybe we should also cleanup the root dir auf our ftp and put
repos into a sub dir.

What about a directory structure like this? We could also remove the os
subdir and backwards compatibility could be achieved by some symlinks.

ftp
└── repo
├── arch
│Â*Â* ├── core
│Â*Â* ├── extra
│Â*Â* ├── packages
│Â*Â* └── testing
└── community
├── community
├── community-testing
└── packages

The actual names might not be final, but with this structure we separate
our repos from everything else on the ftp and we separate the "official"
and community repos from each other.

--
Pierre Schmitz, https://users.archlinux.de/~pierre