Linux Archive - CIA replacement

Linux Archive (http://www.linux-archive.org/)

- Gentoo Development (http://www.linux-archive.org/gentoo-development/)

- - CIA replacement (http://www.linux-archive.org/gentoo-development/708658-cia-replacement.html)

Since CIA.vc is dead [1], I think we should be looking into a
replacement service, or host our own [2].
Is infra already looking into this?

1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
2: http://www.donarmstrong.com/posts/switching_to_kgb/
--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Ben de Groot posted on Mon, 01 Oct 2012 16:14:25 +0800 as excerpted:

> Since CIA.vc is dead [1], I think we should be looking into a
> replacement service, or host our own [2].
> Is infra already looking into this?
>
> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html 2:
> http://www.donarmstrong.com/posts/switching_to_kgb/

This has been discussed previously.

Thread: CIA.VC down for the count?
Original post by blueness on August 23, last post on the 26th.

Here's a link to the gmane thread:

http://comments.gmane.org/gmane.linux.gentoo.devel/79499

Old interface:

http://thread.gmane.org/gmane.linux.gentoo.devel/79499

It may be that there are further developments, but a read of the old
thread (which you don't appear to have participated in and didn't mention
so I'm assuming you missed) should help in any case.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

Hi Ben,

On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot <yngwin@gentoo.org> wrote:
> Since CIA.vc is dead [1], I think we should be looking into a
> replacement service, or host our own [2].
> Is infra already looking into this?
>
> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
> 2: http://www.donarmstrong.com/posts/switching_to_kgb/

Maybe someone with good cvs knowledge can contribute a hook for irker
[1], so we can have #gentoo-commits flooding our irc clients again! :)

[1] http://www.catb.org/esr/irker/

Best regards.

--
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Maybe someone with good cvs knowledge can contribute a hook for irker
> [1], so we can have #gentoo-commits flooding our irc clients again! :)

Why exactly are we still using cvs? Rather than building enhancements
for cvs, why not just migrate everything to git, and spend our time
building the git hooks/etc necessary to make this work?

Looking at the tracker [1], we need a pre-upload hook (I'm not quite
sure why), an rsync conversion script, the ability to validate the
converted tree, and documentation. There is still an open bug for
commit signing, and I'm not quite sure why as this was implemented.

It seems like a lot has already been done with validation. Checking
the active tree is pretty trivial - just compare the trees and they
should be the same. I guess we need to check history, but it seems to
me like the risk of problems is low, and if we just keep a backup of
the cvs repository if there is ever a concern about who made some
commit 5 years ago we can always dig it up.

It really seems to me like little remains to be done here. Mostly we
just need somebody to push a decision on things like workflow. A few
of the bugs have comments like "no sense working on this with other
stuff still needed" - which seems to be outdated thinking with so
little left to do.

Am I missing some big concern that just isn't obvious in these bugs?

I also fear that we're refusing to take action on a great solution
because it isn't a perfect solution. Nobody in the world is using
tree-signing with git, and we aren't really using it in cvs either.
We now have the ability to do it with git, but depending on workflow
3rd-party signatures might not end up in the history of head, or we
might not be able to verify them in an automated fashion. Honestly, I
think the appropriate response here is whoop-de-doo. We can't do any
of that stuff with cvs, but moving to git would have a lot of other
benefits. We can always change our processes later once somebody has
a solution for the signing problem. Right now we're making do without
it on cvs, and so is every other project using git. We can also
continue to sign manifests as a workaround, which is what we'll be
doing anyway if we never migrate to git.

The git migration just strikes me as one of those cases where anybody
is free to come up with a reason not to use something, but nobody has
to defend keeping the status quo. I think the question isn't whether
there is anything wrong with using git, but whether the problems with
git are worse than the problems we already have.

But, hey, if somebody wants to write an irc bot that posts cvs
commits, knock yourself out.

Rich

[1] - https://bugs.gentoo.org/show_bug.cgi?id=333531

On Mon, Oct 1, 2012 at 1:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
> <rafaelmartins@gentoo.org> wrote:
>> Maybe someone with good cvs knowledge can contribute a hook for irker
>> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> Why exactly are we still using cvs? Rather than building enhancements
> for cvs, why not just migrate everything to git, and spend our time
> building the git hooks/etc necessary to make this work?
>
> Looking at the tracker [1], we need a pre-upload hook (I'm not quite
> sure why), an rsync conversion script, the ability to validate the
> converted tree, and documentation. There is still an open bug for
> commit signing, and I'm not quite sure why as this was implemented.
>
> It seems like a lot has already been done with validation. Checking
> the active tree is pretty trivial - just compare the trees and they
> should be the same. I guess we need to check history, but it seems to
> me like the risk of problems is low, and if we just keep a backup of
> the cvs repository if there is ever a concern about who made some
> commit 5 years ago we can always dig it up.
>
> It really seems to me like little remains to be done here. Mostly we
> just need somebody to push a decision on things like workflow. A few
> of the bugs have comments like "no sense working on this with other
> stuff still needed" - which seems to be outdated thinking with so
> little left to do.
>
> Am I missing some big concern that just isn't obvious in these bugs?
>
> I also fear that we're refusing to take action on a great solution
> because it isn't a perfect solution. Nobody in the world is using
> tree-signing with git, and we aren't really using it in cvs either.
> We now have the ability to do it with git, but depending on workflow
> 3rd-party signatures might not end up in the history of head, or we
> might not be able to verify them in an automated fashion. Honestly, I
> think the appropriate response here is whoop-de-doo. We can't do any
> of that stuff with cvs, but moving to git would have a lot of other
> benefits. We can always change our processes later once somebody has
> a solution for the signing problem. Right now we're making do without
> it on cvs, and so is every other project using git. We can also
> continue to sign manifests as a workaround, which is what we'll be
> doing anyway if we never migrate to git.
>
> The git migration just strikes me as one of those cases where anybody
> is free to come up with a reason not to use something, but nobody has
> to defend keeping the status quo. I think the question isn't whether
> there is anything wrong with using git, but whether the problems with
> git are worse than the problems we already have.
>
> But, hey, if somebody wants to write an irc bot that posts cvs
> commits, knock yourself out.
>
> Rich
>
> [1] - https://bugs.gentoo.org/show_bug.cgi?id=333531
>

I don't know to what depth this has been discussed in the past, but if
you use git, you also get an HTTP transport, which has a useful
feature: You could simplify updating the tree on end-users's machines
by using caching proxy servers (operating in accelerator mode) on the
various mirrors.

Those of us who have our own, local caching proxy servers (I have
squid running on my network gateway) can reduce the network load even
further by getting cache hits on our local network before even making
queries outside our network. (Personally, I find this a far easier
thing to maintain and do debugging reasoning on than, e.g. sharing a
network mount or running a local rsync server managed by a cron job.)

--
:wq

Well nenolod has written a CIA -> Irker proxy that (I believe) takes
commit messages designed to go to CIA and makes irker read them and
such, but i haven't looked into it:

https://github.com/nenolod/irker-cia-proxy

On 1 October 2012 11:21, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Hi Ben,
>
> On Mon, Oct 1, 2012 at 5:14 AM, Ben de Groot <yngwin@gentoo.org> wrote:
>> Since CIA.vc is dead [1], I think we should be looking into a
>> replacement service, or host our own [2].
>> Is infra already looking into this?
>>
>> 1: http://shadowm.rewound.net/blog/archives/245-CIA.vc-is-dead.html
>> 2: http://www.donarmstrong.com/posts/switching_to_kgb/
>
> Maybe someone with good cvs knowledge can contribute a hook for irker
> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> [1] http://www.catb.org/esr/irker/
>
> Best regards.
>
> --
> Rafael Goncalves Martins
> Gentoo Linux developer
> http://rafaelmartins.eng.br/
>

On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> I don't know to what depth this has been discussed in the past, but if
> you use git, you also get an HTTP transport, which has a useful
> feature: You could simplify updating the tree on end-users's machines
> by using caching proxy servers (operating in accelerator mode) on the
> various mirrors.

The issue I see here is a tradeoff of bandwidth vs CPU. I just ran an
emerge --sync and the total amount of transmitted data was 5M. The
whole tree is 250M, though no doubt with compression that could be
reduced.

Now, one advantage of HTTP is that caching http servers are likely
more ubiquitous in general than rsync servers. But, we have a whole
bunch of rsync servers already, and we don't have a bunch of caching
http servers.

I suspect bandwidth is going to cost more than CPU here.

In any case, not a reason to hold up git, just one more possibility if
we ever move.

Rich

On Mon, Oct 1, 2012 at 2:29 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 11:21 AM, Rafael Goncalves Martins
> <rafaelmartins@gentoo.org> wrote:
>> Maybe someone with good cvs knowledge can contribute a hook for irker
>> [1], so we can have #gentoo-commits flooding our irc clients again! :)
>
> Why exactly are we still using cvs? Rather than building enhancements
> for cvs, why not just migrate everything to git, and spend our time
> building the git hooks/etc necessary to make this work?

It is amazing how a simple thread about a quite simple topic, with an
easy solution, like this, turns to useless discussions about endless
topics, like this git conversion. This mailing-list is getting really
boring. :(

I'm not asking nobody to write any "enhancement for cvs". irker is
already "done", and working fine, but the authors just implemented
support for git and svn, because this is what they use. We *use* CVS
for now, then we would need to fix it to work with CVS, no matter if
we will have git repositories at some time. I stopped believing in
Santa Claus when I was 5 years old.

This should be doable with 10 lines of python, or using the cia->irker
proxy that jdhore mentioned in another email.

> Looking at the tracker [1], we need a pre-upload hook (I'm not quite
> sure why), an rsync conversion script, the ability to validate the
> converted tree, and documentation. There is still an open bug for
> commit signing, and I'm not quite sure why as this was implemented.

Have you ever thought that people may be not really interested on this
move? or don't have the time to work on it? or don't care enough to
spend time on it? or just wants someone else to do the work?

If you want it, go ahead and push it, but in the right places, please.
I'm tired of bikeshedding here in this list. :(

[snip]

Best regards,

--
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/

On Mon, Oct 1, 2012 at 1:54 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Mon, Oct 1, 2012 at 1:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>> I don't know to what depth this has been discussed in the past, but if
>> you use git, you also get an HTTP transport, which has a useful
>> feature: You could simplify updating the tree on end-users's machines
>> by using caching proxy servers (operating in accelerator mode) on the
>> various mirrors.
>
> The issue I see here is a tradeoff of bandwidth vs CPU. I just ran an
> emerge --sync and the total amount of transmitted data was 5M. The
> whole tree is 250M, though no doubt with compression that could be
> reduced.
>
> Now, one advantage of HTTP is that caching http servers are likely
> more ubiquitous in general than rsync servers. But, we have a whole
> bunch of rsync servers already, and we don't have a bunch of caching
> http servers.
>
> I suspect bandwidth is going to cost more than CPU here.
>
> In any case, not a reason to hold up git, just one more possibility if
> we ever move.

It really depends on how efficient 'git pull' is over HTTP, IMO. I
mean, when I do pulls and pushes in my workflows, it doesn't do a full
data send or a full data pull; that's reserved for 'git clone'.

It may also depend on how often the pull is done; rsync doesn't pull a
tree history, just a copy at the time of sync. git pulls new objects,
which may include intermediate versions which are no longer necessary.
(git may be capable of only syncing to 'HEAD' without worrying about
intermediate states, but I don't know. I'm not advanced enough to
definitively say one way or another.)

As for setting up caching proxies at existing mirror sites...It should
be a case of publishing, a single squid (or pick whatever proxy you
prefer to standardize on) configuration file with appropriate ACLs and
copy it all over the place. I'm not on the infra team, so I don't know
what things look like under the hood, but I do know setting up squid
in forward and reverse roles isn't very painful.

--
:wq

On Mon, Oct 1, 2012 at 2:08 PM, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> Have you ever thought that people may be not really interested on this
> move? or don't have the time to work on it? or don't care enough to
> spend time on it? or just wants someone else to do the work?
>

I'd thought of every single one of those. Hence my post on the list
to drum up interest.

> If you want it, go ahead and push it, but in the right places, please.
> I'm tired of bikeshedding here in this list. :(

Where else would one discuss it? Certainly happy to have a separate
thread - it just seemed like an obvious question when somebody brought
up making an enhancement to our cvs infrastructure. I have no
interest in posting this on a list that nobody actually reads, but if
there are a bunch of devs following some other list that is more
appropriate by all means let me know.

This just strikes me as something that is about at the point where we
could "just do it." Obviously the discussion needs to involve at
least some core of developers, since the goal is to get everybody to
stop doing their commits in cvs, and start doing them in git, and I
doubt everybody is going to be happy if somebody convinces infra to
shut down cvs without any discussion first.

Rich