FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 08-06-2008, 03:30 AM
"Yuan Yijun"
 
Default fedorahosted git repo too large

Hi,

I just tried to download revisor git with this command "git pull
http://git.fedorahosted.org/git/revisor master". I have to repeat 4-5
times since it breaks during downloading. The .git folder is about
58MB. After "git gc --aggressive" it becomes only 6MB.

Anyone please run gc on server?

--
bbbush ^_^

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 03:44 AM
Todd Zullinger
 
Default fedorahosted git repo too large

Yuan Yijun wrote:
> I just tried to download revisor git with this command "git pull
> http://git.fedorahosted.org/git/revisor master". I have to repeat
> 4-5 times since it breaks during downloading. The .git folder is
> about 58MB. After "git gc --aggressive" it becomes only 6MB.
>
> Anyone please run gc on server?

Perhaps better would be repack. There was a recent thread on the git
list and one of the developers pointed out an older mail from Linus
where he described gc --aggressive as "mostly dumb" and recommended
that using something like "repack -a -d -f --depth=250 --window=250"
instead.

http://article.gmane.org/gmane.comp.gcc.devel/94613

--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Rome did not create a great empire by having meetings, they did it by
killing all those who opposed them.

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 08:37 AM
Nigel Jones
 
Default fedorahosted git repo too large

On Tue, 2008-08-05 at 23:44 -0400, Todd Zullinger wrote:
> Yuan Yijun wrote:
> > I just tried to download revisor git with this command "git pull
> > http://git.fedorahosted.org/git/revisor master". I have to repeat
> > 4-5 times since it breaks during downloading. The .git folder is
> > about 58MB. After "git gc --aggressive" it becomes only 6MB.
> >
> > Anyone please run gc on server?
>
> Perhaps better would be repack. There was a recent thread on the git
> list and one of the developers pointed out an older mail from Linus
> where he described gc --aggressive as "mostly dumb" and recommended
> that using something like "repack -a -d -f --depth=250 --window=250"
> instead.
>
> http://article.gmane.org/gmane.comp.gcc.devel/94613
That's actually a very useful article and the methods/reasons behind it
sound quite sane and it could be a useful approach for us.

I'll try this out on one of the smaller repos (a copy of course) and see
what happens.

(n.b. I've added f-infrastructure-list to CC's, that's where everyone
that manages the hosted server reads .)

- Nigel
>
> --
> fedora-devel-list mailing list
> fedora-devel-list@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-devel-list

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 08:37 AM
Nigel Jones
 
Default fedorahosted git repo too large

On Tue, 2008-08-05 at 23:44 -0400, Todd Zullinger wrote:
> Yuan Yijun wrote:
> > I just tried to download revisor git with this command "git pull
> > http://git.fedorahosted.org/git/revisor master". I have to repeat
> > 4-5 times since it breaks during downloading. The .git folder is
> > about 58MB. After "git gc --aggressive" it becomes only 6MB.
> >
> > Anyone please run gc on server?
>
> Perhaps better would be repack. There was a recent thread on the git
> list and one of the developers pointed out an older mail from Linus
> where he described gc --aggressive as "mostly dumb" and recommended
> that using something like "repack -a -d -f --depth=250 --window=250"
> instead.
>
> http://article.gmane.org/gmane.comp.gcc.devel/94613
That's actually a very useful article and the methods/reasons behind it
sound quite sane and it could be a useful approach for us.

I'll try this out on one of the smaller repos (a copy of course) and see
what happens.

(n.b. I've added f-infrastructure-list to CC's, that's where everyone
that manages the hosted server reads .)

- Nigel
>
> --
> fedora-devel-list mailing list
> fedora-devel-list@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-devel-list

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 08-06-2008, 09:23 AM
Jeroen van Meeuwen
 
Default fedorahosted git repo too large

Nigel Jones wrote:

On Tue, 2008-08-05 at 23:44 -0400, Todd Zullinger wrote:

Yuan Yijun wrote:

I just tried to download revisor git with this command "git pull
http://git.fedorahosted.org/git/revisor master". I have to repeat
4-5 times since it breaks during downloading. The .git folder is
about 58MB. After "git gc --aggressive" it becomes only 6MB.

Anyone please run gc on server?

Perhaps better would be repack. There was a recent thread on the git
list and one of the developers pointed out an older mail from Linus
where he described gc --aggressive as "mostly dumb" and recommended
that using something like "repack -a -d -f --depth=250 --window=250"
instead.

http://article.gmane.org/gmane.comp.gcc.devel/94613

That's actually a very useful article and the methods/reasons behind it
sound quite sane and it could be a useful approach for us.

I'll try this out on one of the smaller repos (a copy of course) and see
what happens.



We've ended up doing this live as well and I'm happy with the few stabs
I took at seeing if everything still works.


Feel free to make this a regular thing on the revisor repo and I'll
report if anything breaks, so that if it doesn't, this could maybe
become a regular thing to do on all repos?


Kind regards,

Jeroen van Meeuwen
-kanarip

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 08-06-2008, 10:16 AM
"Martin Langhoff"
 
Default fedorahosted git repo too large

On Wed, Aug 6, 2008 at 8:37 PM, Nigel Jones <dev@nigelj.com> wrote:
>> http://article.gmane.org/gmane.comp.gcc.devel/94613

> That's actually a very useful article and the methods/reasons behind it
> sound quite sane and it could be a useful approach for us.

Agreed. git gc on all repos on an infrequent cronjob is a good idea.
But --aggressive is a very bad idea as it throws away delta-chain work
that's been done. Specially bad on larger repos as the potential delta
pairs to evaluate is much much larger.

cheers,



martin (also a git dev)
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 10:16 AM
"Martin Langhoff"
 
Default fedorahosted git repo too large

On Wed, Aug 6, 2008 at 8:37 PM, Nigel Jones <dev@nigelj.com> wrote:
>> http://article.gmane.org/gmane.comp.gcc.devel/94613

> That's actually a very useful article and the methods/reasons behind it
> sound quite sane and it could be a useful approach for us.

Agreed. git gc on all repos on an infrequent cronjob is a good idea.
But --aggressive is a very bad idea as it throws away delta-chain work
that's been done. Specially bad on larger repos as the potential delta
pairs to evaluate is much much larger.

cheers,



martin (also a git dev)
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 
Old 08-06-2008, 10:18 AM
"Yuan Yijun"
 
Default fedorahosted git repo too large

2008/8/6 Todd Zullinger <tmz@pobox.com>:
> Yuan Yijun wrote:
>> I just tried to download revisor git with this command "git pull
>> http://git.fedorahosted.org/git/revisor master". I have to repeat
>> 4-5 times since it breaks during downloading. The .git folder is
>> about 58MB. After "git gc --aggressive" it becomes only 6MB.
>>
>> Anyone please run gc on server?
>
> Perhaps better would be repack. There was a recent thread on the git
> list and one of the developers pointed out an older mail from Linus
> where he described gc --aggressive as "mostly dumb" and recommended
> that using something like "repack -a -d -f --depth=250 --window=250"
> instead.
>
> http://article.gmane.org/gmane.comp.gcc.devel/94613
>

See that. The repack command is so hard to remember, while "git gc" is
fast enough and also runs repack, AND the .git folder becomes 6.6MB
without --aggressive...

I guess the only problem is git version on server is too old, AFAIK
new git will repack automatically when commit or push or before fetch
(?)



--
bbbush ^_^

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 10:51 AM
Nigel Jones
 
Default fedorahosted git repo too large

On Wed, 2008-08-06 at 18:18 +0800, Yuan Yijun wrote:
> 2008/8/6 Todd Zullinger <tmz@pobox.com>:
> > Yuan Yijun wrote:
> >> I just tried to download revisor git with this command "git pull
> >> http://git.fedorahosted.org/git/revisor master". I have to repeat
> >> 4-5 times since it breaks during downloading. The .git folder is
> >> about 58MB. After "git gc --aggressive" it becomes only 6MB.
> >>
> >> Anyone please run gc on server?
> >
> > Perhaps better would be repack. There was a recent thread on the git
> > list and one of the developers pointed out an older mail from Linus
> > where he described gc --aggressive as "mostly dumb" and recommended
> > that using something like "repack -a -d -f --depth=250 --window=250"
> > instead.
> >
> > http://article.gmane.org/gmane.comp.gcc.devel/94613
> >
>
> See that. The repack command is so hard to remember, while "git gc" is
> fast enough and also runs repack, AND the .git folder becomes 6.6MB
> without --aggressive...
git gc, while fast, is not as efficient, we have repos with nearly
100000 objects (anaconda) and while it does a fair job, (and quick - a
tenth of the time a git repack does) I do think that a repack would be a
good idea on a frequent basis (read bi/tri-monthly).

Of course, like beauty the usefulness is in the eyes of the beholder,
and while the majority of repos are very small, there are the few that
are over the 100M mark at present. So different needs mean different
solutions.

- Nigel
>
> I guess the only problem is git version on server is too old, AFAIK
> new git will repack automatically when commit or push or before fetch
> (?)
>
>
>
> --
> bbbush ^_^
>

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 08-06-2008, 10:53 AM
Nigel Jones
 
Default fedorahosted git repo too large

On Wed, 2008-08-06 at 11:23 +0200, Jeroen van Meeuwen wrote:
> Nigel Jones wrote:
> > On Tue, 2008-08-05 at 23:44 -0400, Todd Zullinger wrote:
> >> Yuan Yijun wrote:
> >>> I just tried to download revisor git with this command "git pull
> >>> http://git.fedorahosted.org/git/revisor master". I have to repeat
> >>> 4-5 times since it breaks during downloading. The .git folder is
> >>> about 58MB. After "git gc --aggressive" it becomes only 6MB.
> >>>
> >>> Anyone please run gc on server?
> >> Perhaps better would be repack. There was a recent thread on the git
> >> list and one of the developers pointed out an older mail from Linus
> >> where he described gc --aggressive as "mostly dumb" and recommended
> >> that using something like "repack -a -d -f --depth=250 --window=250"
> >> instead.
> >>
> >> http://article.gmane.org/gmane.comp.gcc.devel/94613
> > That's actually a very useful article and the methods/reasons behind it
> > sound quite sane and it could be a useful approach for us.
> >
> > I'll try this out on one of the smaller repos (a copy of course) and see
> > what happens.
> >
>
> We've ended up doing this live as well and I'm happy with the few stabs
> I took at seeing if everything still works.
>
> Feel free to make this a regular thing on the revisor repo and I'll
> report if anything breaks, so that if it doesn't, this could maybe
> become a regular thing to do on all repos?
Okay, from a server POV it shrunk the 116MB folder down to just 7MB in
less than two minutes (based on a trial run in my homedir), which is
pretty sweet.

A trial with system-config-firewall.git went from ~20M to ~4M.

I also did a trial run of anaconda.git and anaconda-images.git:
anaconda.git:
183M (97745 objects) -> 64M (a third of the original size)
real 26m18.050s
user 23m9.395s
sys 0m6.568s

anaconda-images.git:
54M (1482 objects) -> 41M (didn't expect much here)
real 1m57.944s
user 1m43.466s
sys 0m0.848s

Maybe we should run git repack on the big repos on a bi/tri-monthly
basis, and git gc (which is very fast - <1 minute on the anaconda repo
for example) on a monthly basis.

- Nigel

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
 

Thread Tools




All times are GMT. The time now is 10:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org