FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Development

 
 
LinkBack Thread Tools
 
Old 04-10-2010, 01:13 AM
Pierre Schmitz
 
Default Sharing my repo-tools

I have finally put the tools I use to sync my local repo and to create
those magic rebuild lists into a git repo. I polished them to increase
usability and portability, but they are not really ready to be run e.g. on
gerolde.

See https://git.archlinux.de/repo-tools.git/tree/ or clone from
http://git.archlinux.de/~pierre/repo-tools.git

syncrepo is a script I just refactored today to keep my local repo in
sync. You can simply change it to sync from any other server. What I am
trying here is to sync all db files first to reduce the problems of repos
not matching each other. I exclude some files I don't need and the links
files I create myself. This script can be run via cron (add >/dev/null to
suppress output)

createlinks is a script which reads all the links to shared libs from the
packages and put them into a database with a format similar to our repo db
files. This script works incremental and can be run via cron. I just
cleaned it up and added the readelf line from the sodeps pacman branch,
which was a little simpler than mine. I only read the files from certain
dirs like /usr/lib; so if you have a lib in /usr/share that wont be
covered. If you uncomment that "(..)& wait" you can check both arches in
parallel. The initial run takes quite long; but I don't know how to speed
this up even more (at least when we stay in bash).

getpkgbase is a simple script stolen from the db-scripts to get the
package base or name from a package file.

with sogrep you can check which packages link against a certain soname.
This is of course really fast as it literally greps through the link
database we create with the script mentioned above.

Maybe these scripts are helpful for someone; suggestions or patches are
welcome.

Pierre

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 04-11-2010, 07:09 PM
Pierre Schmitz
 
Default Sharing my repo-tools

On Sat, 10 Apr 2010 03:13:33 +0200, Pierre Schmitz <pierre@archlinux.de>
wrote:
> I have finally put the tools I use to sync my local repo and to create
> those magic rebuild lists into a git repo. I polished them to increase
> usability and portability, but they are not really ready to be run e.g.
on
> gerolde.

I just added a simple integrity check for a repo:
https://git.archlinux.de/repo-tools.git/tree/checkrepo

Any ideas why writing that md5sums.txt is so slow?

--
Pierre Schmitz, https://users.archlinux.de/~pierre
 
Old 04-11-2010, 07:29 PM
Mauro Santos
 
Default Sharing my repo-tools

On 04/11/2010 08:09 PM, Pierre Schmitz wrote:
> On Sat, 10 Apr 2010 03:13:33 +0200, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> I have finally put the tools I use to sync my local repo and to create
>> those magic rebuild lists into a git repo. I polished them to increase
>> usability and portability, but they are not really ready to be run e.g.
> on
>> gerolde.
>
> I just added a simple integrity check for a repo:
> https://git.archlinux.de/repo-tools.git/tree/checkrepo
>
> Any ideas why writing that md5sums.txt is so slow?
>

Your inner for loop is slowing everything, I have found that FOR loops
in bash are very slow. If the loop will run only a few times it is ok
but if the intended purpose is to run hundreds of times then it is
better to find an alternative.

Maybe you can try to replace the inner loop with something involving
'find ... -exec something' or some recursive grep and mix it with awk, I
have found that it can be many times faster than using FOR loops.
 
Old 04-11-2010, 10:26 PM
Mauro Santos
 
Default Sharing my repo-tools

On 04/11/2010 08:09 PM, Pierre Schmitz wrote:
> On Sat, 10 Apr 2010 03:13:33 +0200, Pierre Schmitz <pierre@archlinux.de>
> wrote:
>> I have finally put the tools I use to sync my local repo and to create
>> those magic rebuild lists into a git repo. I polished them to increase
>> usability and portability, but they are not really ready to be run e.g.
> on
>> gerolde.
>
> I just added a simple integrity check for a repo:
> https://git.archlinux.de/repo-tools.git/tree/checkrepo
>
> Any ideas why writing that md5sums.txt is so slow?
>

Try something like this:

cat `find $tmp/$repo/$arch -iname desc` | egrep -i -A1 "filename|md5" |
tr -d '
' | sed 's/%FILENAME%/
/g' | sed 's/%/ /g' | sed 's/--/ /g' |
awk '{print $3 " " $1}' > $tmp/md5sums.txt

and adapt to your needs. This should be really fast, way faster than
using a for loop to read each file.
 

Thread Tools




All times are GMT. The time now is 04:22 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org