community back-end rewrite

Hi all,

In the TU meetings there was some discussion about doing a
rewrite/update/fix-up of the [community] back-end. I am not sure who
all is looking at this and what the plans are - hence this email.


Can people who are looking at this or interested in looking at this give
a ping back to this message with whatever plans you have or work you
have already done.


Things I want to flag:
- we should get a git repo for a community-backend project on the arch
server and separate this from the aur site code.
- we should switch SCM tools from CVS. I am strongly in favour of
using the same system as the main repos which would allow much code
reuse and easily get us a testing repo for community packages. (No more
big rebuild issues!).


Allan

On Sun, Nov 30, 2008 at 20:35, Allan McRae <allan@archlinux.org> wrote:
> Hi all,
>
> In the TU meetings there was some discussion about doing a
> rewrite/update/fix-up of the [community] back-end. I am not sure who all is
> looking at this and what the plans are - hence this email.
>
> Can people who are looking at this or interested in looking at this give a
> ping back to this message with whatever plans you have or work you have
> already done.

You're probably referring to me here. I have been considering writing
a tupkg-like tool that will upload and repo-add things, and be able to
have any sort of post-upload hooks that you want. I haven't really
written anything yet. One of the main attractions is that you wouldn't
really need any sort of cron job running to add things, which is one
sucky thing about the current community backend

On Mon, Dec 01, 2008 at 11:35:18AM +1000, Allan McRae wrote:
> Hi all,
>
> In the TU meetings there was some discussion about doing a
> rewrite/update/fix-up of the [community] back-end. I am not sure who all is
> looking at this and what the plans are - hence this email.
>
> Can people who are looking at this or interested in looking at this give a
> ping back to this message with whatever plans you have or work you have
> already done.
>
> Things I want to flag:
> - we should get a git repo for a community-backend project on the arch
> server and separate this from the aur site code. - we should switch SCM
> tools from CVS. I am strongly in favour of using the same system as the
> main repos which would allow much code reuse and easily get us a testing
> repo for community packages. (No more big rebuild issues!).

I'm involved with AUR development and have recently studied and begin to
understand the tupkg code. Some improvements based on that activity will
be put into action soon. I'll continue to help maintain AUR until I feel it
has reached a certain point of efficiency and effectiveness where it no
longer needs major maintenance.

My plan from there is to start design and development on a new system
that can be used for repo maintenance in AUR as well as any other
package repo project, like arch-games, or even maybe official repos.

These were actually some of my main goals in becoming a TU.

I don't there is any benefit from putting tupkg code in a different repo
than the AUR web interface. They are tied together by the DB and
separating them may cause disjointed development and other problems.

However we could organise the source tree and document how the system
works a little better.

Cheers!

Heh. This happens when you don't read all of your e-mails before
answering. So, here's a bit of copy 'n paste from my last message, and
some new thoughts.

On Mon, Dec 1, 2008 at 2:35 AM, Allan McRae <allan@archlinux.org> wrote:
> In the TU meetings there was some discussion about doing a
> rewrite/update/fix-up of the [community] back-end. I am not sure who all is
> looking at this and what the plans are - hence this email.

I think the only work done is in AUR2, but I don't know if they have
touched the backend. Could any of the involved people update us on its
status?

> Can people who are looking at this or interested in looking at this give a
> ping back to this message with whatever plans you have or work you have
> already done.

From my lst mail:
«This is a very important point, IMHO, which we need to discuss with
the AUR2 guys. In fact I think we could, as we say in Italian, "catch
two pigeons with one broad bean" and plan the backend changes to
include the move to SVN. Personally, I'd like to see an independent
RCS layer so that we won't see this situation again when we need to
move to $ZOMGCOOLRCS.»

> Things I want to flag:
> - we should get a git repo for a community-backend project on the arch
> server and separate this from the aur site code. - we should switch SCM
> tools from CVS. I am strongly in favour of using the same system as the
> main repos which would allow much code reuse and easily get us a testing
> repo for community packages. (No more big rebuild issues!).

Here I agree with Loui: it doesn't make sense to separate the AUR and
the [community] backend, they are tightly coupled and so they should
be. The risk is to break things if two groups move independently.
And I'm not really sure we could reuse some code, but writing an SCM
abstraction layer that makes us independent from CVS and SVN could be
userful for the main repos, too. I don't think it's a stupid idea
because we just need the basic operations that every SCM provides...

Corrado

On Mon, Dec 1, 2008 at 5:23 PM, bardo <ilbardo@gmail.com> wrote:
> I think the only work done is in AUR2, but I don't know if they have
> touched the backend. Could any of the involved people update us on its
> status?

What AUR2?

> Here I agree with Loui: it doesn't make sense to separate the AUR and
> the [community] backend, they are tightly coupled and so they should
> be. The risk is to break things if two groups move independently.
> And I'm not really sure we could reuse some code, but writing an SCM
> abstraction layer that makes us independent from CVS and SVN could be
> userful for the main repos, too. I don't think it's a stupid idea
> because we just need the basic operations that every SCM provides...

Also if we're talking about AUR2 I think the plan is definitely to
separate them, but when that happens there won't be two sets of
queries but rather abstracted.

--
Callan Barrett

On Sun, Nov 30, 2008 at 7:35 PM, Allan McRae <allan@archlinux.org> wrote:
> Hi all,
>
> In the TU meetings there was some discussion about doing a
> rewrite/update/fix-up of the [community] back-end. I am not sure who all is
> looking at this and what the plans are - hence this email.
>
> Can people who are looking at this or interested in looking at this give a
> ping back to this message with whatever plans you have or work you have
> already done.
>
> Things I want to flag:
> - we should get a git repo for a community-backend project on the arch
> server and separate this from the aur site code. - we should switch SCM
> tools from CVS. I am strongly in favour of using the same system as the
> main repos which would allow much code reuse and easily get us a testing
> repo for community packages. (No more big rebuild issues!).

To whoever is doing this: Please remain in touch with me for this. I
would *love* it if the community backend just ran the official db
scripts under the 'aur' user. This would not only simplify the code,
but allow us to unify lots of changes.

Additionally, the database itself should be updated the same way the
official database is done - parse the db.tar.gz and
insert/update/delete accordingly.

In short: the community backend should really just accept packages via
upload (authenticated, or course), dump them in a common location
(/home/aur/staging) and then setup a cronjob to run db-update
periodically. This cronjob should also find packages in the db.tar.gz
and not in CVS and run db-remove on them.

If we can collaborate enough on this, we could even use the same
functionality for extra and core.

On Mon, Dec 1, 2008 at 9:55 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
> On Sun, Nov 30, 2008 at 7:35 PM, Allan McRae <allan@archlinux.org> wrote:
>> Hi all,
>>
>> In the TU meetings there was some discussion about doing a
>> rewrite/update/fix-up of the [community] back-end. I am not sure who all is
>> looking at this and what the plans are - hence this email.
>>
>> Can people who are looking at this or interested in looking at this give a
>> ping back to this message with whatever plans you have or work you have
>> already done.
>>
>> Things I want to flag:
>> - we should get a git repo for a community-backend project on the arch
>> server and separate this from the aur site code. - we should switch SCM
>> tools from CVS. I am strongly in favour of using the same system as the
>> main repos which would allow much code reuse and easily get us a testing
>> repo for community packages. (No more big rebuild issues!).
>
> To whoever is doing this: Please remain in touch with me for this. I
> would *love* it if the community backend just ran the official db
> scripts under the 'aur' user. This would not only simplify the code,
> but allow us to unify lots of changes.
>
> Additionally, the database itself should be updated the same way the
> official database is done - parse the db.tar.gz and
> insert/update/delete accordingly.
>
> In short: the community backend should really just accept packages via
> upload (authenticated, or course), dump them in a common location
> (/home/aur/staging) and then setup a cronjob to run db-update
> periodically. This cronjob should also find packages in the db.tar.gz
> and not in CVS and run db-remove on them.
>
> If we can collaborate enough on this, we could even use the same
> functionality for extra and core.

Note that the above regarding db-update requires a move to svn at the
same time. I do not plan to add special code to deal with CVS

>> In short: the community backend should really just accept packages via
>> upload (authenticated, or course), dump them in a common location
>> (/home/aur/staging) and then setup a cronjob to run db-update
>> periodically. This cronjob should also find packages in the db.tar.gz
>> and not in CVS and run db-remove on them.
>>
>> If we can collaborate enough on this, we could even use the same
>> functionality for extra and core.
>
> Note that the above regarding db-update requires a move to svn at the
> same time. I do not plan to add special code to deal with CVS
>
Regarding db-remove: I think is is an _absolute_ requirement that
packages be easily moved back to unsupported. Currently the only way
is to delete it entirely and re-upload as a new package (losing all
related metadata). It's really horrible, and I think contributes to
the glut of useless stuff in community at the moment.

On Mon, Dec 1, 2008 at 1:12 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
>>> In short: the community backend should really just accept packages via
>>> upload (authenticated, or course), dump them in a common location
>>> (/home/aur/staging) and then setup a cronjob to run db-update
>>> periodically. This cronjob should also find packages in the db.tar.gz
>>> and not in CVS and run db-remove on them.
>>>
>>> If we can collaborate enough on this, we could even use the same
>>> functionality for extra and core.
>>
>> Note that the above regarding db-update requires a move to svn at the
>> same time. I do not plan to add special code to deal with CVS
>>
> Regarding db-remove: I think is is an _absolute_ requirement that
> packages be easily moved back to unsupported. Currently the only way
> is to delete it entirely and re-upload as a new package (losing all
> related metadata). It's really horrible, and I think contributes to
> the glut of useless stuff in community at the moment.

Sure, but that'd be more up to the AUR front-end that controls
unsupported. It'd just have to copy the entries from CVS and put them
in the unsupported dir. Should be simple. As for removing them,
perhaps the front-end could even remove them from CVS, but that seems
a little gross.

On Mon, Dec 1, 2008 at 1:33 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
> On Mon, Dec 1, 2008 at 1:12 PM, Daenyth Blank <daenyth+arch@gmail.com> wrote:
>>>> In short: the community backend should really just accept packages via
>>>> upload (authenticated, or course), dump them in a common location
>>>> (/home/aur/staging) and then setup a cronjob to run db-update
>>>> periodically. This cronjob should also find packages in the db.tar.gz
>>>> and not in CVS and run db-remove on them.
>>>>
>>>> If we can collaborate enough on this, we could even use the same
>>>> functionality for extra and core.
>>>
>>> Note that the above regarding db-update requires a move to svn at the
>>> same time. I do not plan to add special code to deal with CVS
>>>
>> Regarding db-remove: I think is is an _absolute_ requirement that
>> packages be easily moved back to unsupported. Currently the only way
>> is to delete it entirely and re-upload as a new package (losing all
>> related metadata). It's really horrible, and I think contributes to
>> the glut of useless stuff in community at the moment.
>
> Sure, but that'd be more up to the AUR front-end that controls
> unsupported. It'd just have to copy the entries from CVS and put them
> in the unsupported dir. Should be simple. As for removing them,
> perhaps the front-end could even remove them from CVS, but that seems
> a little gross.

Hmm, perhaps we could just set a rule such that any removal at all is
copied to the unsupported dir, and then people could simply delete the
package from the front-end.