On Thu, Oct 09, 2008 at 01:37:14PM -0700, Jesse Keating wrote:
> On Thu, 2008-10-09 at 22:26 +0200, Till Maas wrote:
> > On Thu October 9 2008, Jesse Keating wrote:
> > > On Thu, 2008-10-09 at 18:16 +0200, Till Maas wrote:
> > > > On Thu October 9 2008, Jesse Keating wrote:
> > > > > Only if we allow source repo tags to be a suitable source distribution
> > > > > method. Since fedorahosted still doesn't have an easy to use way of
> > > > > distributing tarballs, and getting the code via a scm checkout is quite
> > > > > easy, it should suffice.
> > > >
> > > > If this is the only obstacle to get this done, I will write a patch for
> > > > Makefile.common that displays the URL for every file in sources to fetch
> > > > it from the lookaside cache. Then this URL can be provided as source
> > > > tarball on a fedorahosted wiki page.
> > >
> > > Why must we insist on tarballs? Unless the tarball includes scm
> > > metadata in it, it's practically useless for upstream patch development.
> >
> > Tarballs allow to easily build the desired software via rpm and test it. With
> > make patch and make rediff and make prep from Makefile.common it is also
> > pretty easy to create and update patches. I have sucessfully created patches
> > for upstream using this. Using directly an upstream scm, it is always a PITA
> > to make the spec build from there, because the spec needs to be heavily
> > modificated or I have to remember to recreate the tarball everytime. It would
> > like it very much, if this would be a lot easier, e.g. some intelligent
> > macros in spec files that allow to use a scm instead of a tarball to be used
> > as Source0 and some magic that allows to define what from the scm should be
> > used, e.g. should uncommited changes be used?
> >
> > Nevertheless, there have to be tarballs in the cvs lookaside cache, therefore
> > it is not much work to link to them. Therefore I do not really see what needs
> > to be discussed here. ;-)
>
> I'm trying to see things from somebody else's point of view. Recently,
> on this list, somebody else was arguing that we in Fedora land should be
> doing away with tarball distribution, and srpm distribution, and instead
> direct everything back to SCM, and to work on RPM so that it just
> understood SCM and left tarballs in the past. Thank you for providing
> some argument against that (:

I like the tar.gz from a release tracking point of view - with pretty
much every distro, whether Linux, or BSD or Windows using pristine
tar.gz + patches, you can write some interesting tools to correlate
packages & patches across distros. I'm working on a toy/tool which
would record the md5sum of every source & patch file from every
released RPM in Fedora, RHEL, Ubuntu, Debian, etc. It will also
import checksums of official releases from major hosting sites like
CPAN, gnome.org, sourceforge etc. You can then correlate all the
md5sums to track the relations between packages in different distros
vs upstream, even if the package naming is completely different. This
lets you answer questions like

- What distros have at least version x.y.z
- When did version x.y.z appear in a distro
- What packages are in distro A but not B
- Is distro X out of date compared to latest upstream release
- As a maintainer in distro A, what patches are other distros
carrying against my packages
- As an upstream maintainer, what patches are distros applying
to my code.
- As an app developer, if I have a choice between libraries A & B,
which is the most widely available in distros.
- What source files claim to be version x.y.z, but don't match
the x.y.z published by upstream (there's a number of surprises
in this area, but in Fedora & CPAN itself :-)

There's probably more you can do, but those were the questions that
motivated me to try and hack some cross-distro data corrrelation tool
together. Even if it doesn't turn into anything serious, its nice that
you can experiment with these kind of things. I fear it'd be harder to
do this kind of thing if everyone just bundled SCM repos in their distros
as source, as you'd need to actually get the source - currnetly I can
just get the checksums from the 'sources' file in CVS without downloading
everything from the lookaside cache, likewise many hosting sites publish
the md5 sums separately from the sources. Then again maybe having the
full SCM repos would let you do some even more interesting correlation.

Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Thu October 9 2008, Jesse Keating wrote:

> I'm trying to see things from somebody else's point of view. Recently,
> on this list, somebody else was arguing that we in Fedora land should be
> doing away with tarball distribution, and srpm distribution, and instead
> direct everything back to SCM, and to work on RPM so that it just
> understood SCM and left tarballs in the past. Thank you for providing
> some argument against that (:

I do not see how I argued against this, because I also wrote that it would be
helpful if one could rpm in combination SCMs efficently.

Regards,
Till
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Thu October 9 2008, Daniel P. Berrange wrote:

> packages & patches across distros. I'm working on a toy/tool which
> would record the md5sum of every source & patch file from every
> released RPM in Fedora, RHEL, Ubuntu, Debian, etc. It will also
> import checksums of official releases from major hosting sites like
> CPAN, gnome.org, sourceforge etc. You can then correlate all the
> md5sums to track the relations between packages in different distros
> vs upstream, even if the package naming is completely different. This
> lets you answer questions like

This is great, I always planned to write a similiar application, but my desire
was not big enough, that I found the time. But I also wanted to track the
contents of the packages to make it easier to spot files on a system, that
are not well known, e.g. malware. But I guess once your application is in a
usable state, it can be easily added. :-)

Regards,
Till
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Thu, 2008-10-09 at 13:37 -0700, Jesse Keating wrote:
> On Thu, 2008-10-09 at 22:26 +0200, Till Maas wrote:
> > On Thu October 9 2008, Jesse Keating wrote:
> > > On Thu, 2008-10-09 at 18:16 +0200, Till Maas wrote:
> > > > On Thu October 9 2008, Jesse Keating wrote:
> > > > > Only if we allow source repo tags to be a suitable source distribution
> > > > > method. Since fedorahosted still doesn't have an easy to use way of
> > > > > distributing tarballs, and getting the code via a scm checkout is quite
> > > > > easy, it should suffice.
> > > >
> > > > If this is the only obstacle to get this done, I will write a patch for
> > > > Makefile.common that displays the URL for every file in sources to fetch
> > > > it from the lookaside cache. Then this URL can be provided as source
> > > > tarball on a fedorahosted wiki page.
> > >
> > > Why must we insist on tarballs? Unless the tarball includes scm
> > > metadata in it, it's practically useless for upstream patch development.
> >
> > Tarballs allow to easily build the desired software via rpm and test it. With
> > make patch and make rediff and make prep from Makefile.common it is also
> > pretty easy to create and update patches. I have sucessfully created patches
> > for upstream using this. Using directly an upstream scm, it is always a PITA
> > to make the spec build from there, because the spec needs to be heavily
> > modificated or I have to remember to recreate the tarball everytime. It would
> > like it very much, if this would be a lot easier, e.g. some intelligent
> > macros in spec files that allow to use a scm instead of a tarball to be used
> > as Source0 and some magic that allows to define what from the scm should be
> > used, e.g. should uncommited changes be used?
> >
> > Nevertheless, there have to be tarballs in the cvs lookaside cache, therefore
> > it is not much work to link to them. Therefore I do not really see what needs
> > to be discussed here. ;-)
>
> I'm trying to see things from somebody else's point of view. Recently,
> on this list, somebody else was arguing that we in Fedora land should be
> doing away with tarball distribution, and srpm distribution, and instead
> direct everything back to SCM, and to work on RPM so that it just
> understood SCM and left tarballs in the past. Thank you for providing
> some argument against that (:
There are many more arguments against directly working from an SCM:
+ tarballs are easy to verify, long-term stable and easy to copy.
+ tarballs don't need a server infrastructure.
+ tarballs need a minimal client infrastructure.
+ tarballs decouple "rpms" from "upstreams".

That said, I find letting rpm directly work from SCMs to be
- unstable/unreliable
- unsafe/a security risk
- resource demanding
- adding too many project dependencies.

=> I actually hardly find an argument speaking for letting rpm work
directly from an SCM.

Ralf



--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Dave Airlie wrote:

On Wed, 2008-10-01 at 18:52 +0200, Jakub Jelinek wrote:


Hi!






MAKEDEV sources almost 500KB of config files on every invocation, and the
inner loop is terribly inefficient. The second patch allows it to shrink
the files to 55KB while expressing the same info and makes the inner loop
more efficient. Depending on how many modprobe and MAKEDEV invocations
are done on your box during bootup, this can or might not make meassurable
difference.




I've just pushed MAKEDEV 3.23-7 into koji should be in rawhide soon.

MAKEDEV is now hosted on git.fedorahosted.org for anyone who cares.

Thanks Jakub.

Dave.


Has anyone made any bootcharts? Is there any word on how much this gains
us at boot time?


--CJD

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

I have tested both on the virtual machine,
I built an rpm
www.ojuba.org/testing/module-init-tools-3.4.1-2.oj1.i386.rpm

and it was really fast

but the MAKEDEV gives errors/warnings like this
don't know how to make device "tty2"

here is a screenshot
http://www.linuxac.org/forum/attachment.php?attachmentid=4658&stc=1&d=122365090 7

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Wed, 2008-10-01 at 13:20 -0400, Kyle McMartin wrote:
> On Wed, Oct 01, 2008 at 06:52:15PM +0200, Jakub Jelinek wrote:
> > Given the recent 5sec boot efforts http://lwn.net/Articles/299483/
> > and Mandriva follow-ups on that http://lwn.net/Articles/300873/,
> > I thought I'd share my modprobe and MAKEDEV speedup patches with
> > a wider community so folks can experiment with them, especially seeing
> > Mandriva folks playing with turning modprobe into a daemon because of its
> > slowness.

> Have you tested things on a big endian machine? In any case, this looks
> really good. Hopefully Jon will apply this upstream soon so we can get
> it in F10.

So I've added an alternative patch from Alan Jenkins to the current git
and in rawhide today, also using a binary index (there's no version info
embedded right now however that is about to be fixed in a later patch).
I think my plan right now is to profile this against Jakub's patch and
pick the winner. So there might be a few more changes this week

Jon.


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, Oct 13, 2008 at 09:52:17AM -0400, Jon Masters wrote:
> On Wed, 2008-10-01 at 13:20 -0400, Kyle McMartin wrote:
> > On Wed, Oct 01, 2008 at 06:52:15PM +0200, Jakub Jelinek wrote:
> > > Given the recent 5sec boot efforts http://lwn.net/Articles/299483/
> > > and Mandriva follow-ups on that http://lwn.net/Articles/300873/,
> > > I thought I'd share my modprobe and MAKEDEV speedup patches with
> > > a wider community so folks can experiment with them, especially seeing
> > > Mandriva folks playing with turning modprobe into a daemon because of its
> > > slowness.
>
> > Have you tested things on a big endian machine? In any case, this looks
> > really good. Hopefully Jon will apply this upstream soon so we can get
> > it in F10.
>
> So I've added an alternative patch from Alan Jenkins to the current git
> and in rawhide today, also using a binary index (there's no version info
> embedded right now however that is about to be fixed in a later patch).
> I think my plan right now is to profile this against Jakub's patch and
> pick the winner. So there might be a few more changes this week

FYI, here is a tiny incremental fix for the patch I've posted earlier
- Mandriva folks reported that with my patch depmod was upset when run
on a tree with no modules at all.
--- module-init-tools-3.4.1/depmod.c 2008-10-01 17:32:14.000000000 +0200
+++ module-init-tools-3.4.1/depmod.c 2008-10-07 11:49:29.000000000 +0200
@@ -651,6 +651,8 @@
d = d->next;
}
}
+ if (strings->dir0.len == -1)
+ strings->dir0.len = 0;
no_dir0:;
/* Assign the remaining up to 255 directory table
entries, starting with most often used ones. */


Jakub

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, 2008-10-13 at 16:10 +0200, Jakub Jelinek wrote:

> FYI, here is a tiny incremental fix for the patch I've posted earlier
> - Mandriva folks reported that with my patch depmod was upset when run
> on a tree with no modules at all.

Though if we ever manage to "demodularize for the win" to that point,
well, it'll be quite exciting!

I'm testing your 3.4.1 patch against today's 3.5 tree and will followup
with some results. Whichever performs best, I'll keep. Sorry I didn't
sort this out sooner, but it's time to move forward and get this sorted
out - and there are more trivial fixes I've got in mind. As of today's
rawhide there are also no longer any Fedora-specific patches left.

Jon.


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, 2008-10-13 at 10:24 -0400, Jon Masters wrote:
> On Mon, 2008-10-13 at 16:10 +0200, Jakub Jelinek wrote:
>
> > FYI, here is a tiny incremental fix for the patch I've posted earlier
> > - Mandriva folks reported that with my patch depmod was upset when run
> > on a tree with no modules at all.
>
> Though if we ever manage to "demodularize for the win" to that point,
> well, it'll be quite exciting!
>
> I'm testing your 3.4.1 patch against today's 3.5 tree and will followup
> with some results. Whichever performs best, I'll keep. Sorry I didn't
> sort this out sooner, but it's time to move forward and get this sorted
> out - and there are more trivial fixes I've got in mind. As of today's
> rawhide there are also no longer any Fedora-specific patches left.

In one test, I ran modprobe about 8000 times on a desktop.

older modprobe: 190.03user 25.19system 3:55.54elapsed 91%CPU
upstream modprobe: 8.65user 19.69system 0:31.08elapsed 91%CPU
Jakob's modprobe: 6.71user 18.05system 0:27.01elapsed 91%CPU

The latter two vie a little between each other, but basically are
comparable. So for now, I'll keep upstream as it is and do some more
testing/write some scripts. Then I'll do some migration to unlocked
non-threaded functions and pull in other parts of Jakob's patch. I'm
thinking there are actually other places to focus on now - more
profiling will be called for and I'll push some more updates.

Older modprobe is 3.4.1, upstream is 3.5 (radix trie implementation),
Jakob's patch uses a hashtable/bucket mechanism.

Jon.


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list