pac-file utility script
 

Hi all,

First a quick introduction, since this is my first message to the list. I am
a CS grad student at James Madison University in Harrisonburg, VA, and I also
do IT work for JMU's art department. I've been using arch for around 2.5
years now.

Anyway, I've written a script that takes advantage of the [repo].files.tar.gz
filelists that are now available to discover which package provides a
particular file, when that package is not installed on the user's machine. It
is similar to debian's apt-file, if you are familiar with that. I would
appreciate any comments on the script, with an eye toward possible inclusion
in Allan's pacman-contrib community package.

The script uses the mirrors defined in the user's /etc/pacman.d/mirrorlist and
downloads the filelists for the repos enabled in /etc/pacman.conf. It is
available from http://threadstates.com/downloads/pac-file.

The only major issue I can see is in the handling of non-standard repos,
including local ones. The repo names are obtained from pacman.conf's "[repo]"
lines, and the name of the repo is substituted for $repo in the mirror url
from the mirrorlist. If a local repo (for example) is defined in pacman.conf
as "[local]", on a 'sync' operation, pac-file will attempt to download
local.files.tar.gz from every mirror enabled in the mirrorlist. This will not
cause the script to fail (it will report that the file is unavailable from any
mirror and continue on), but it will waste time. I am hesitant to limit the
possible repos to the official ones because there might be third party repos
that provides filelists. Suggestions?

Thanks,
Pete

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

Hi Pete,

Can you post the script inline so it would be easier for us to comment on.

Thanks,
Allan



_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

On Thu Jun 26, 2008 at 10:07AM, Allan McRae wrote:
> Hi Pete,
>
> Can you post the script inline so it would be easier for us to comment
> on.

Sure, see below:

#!/bin/bash

#-----------------------------------------------------------
# A utility for discovering which ArchLinux package contains
# a given file. Downloads filelists for all active repos in
# pacman.conf; uses mirrors in order according to pacman's
# mirrorlist file.
#
# Peter Morris 2008
#-----------------------------------------------------------

db=/var/lib/pacman/filelists
pacmanconf=/etc/pacman.conf
mirrorlist=/etc/pacman.d/mirrorlist

#-----------------------------------------------------------
# Print active repos in $pacmanconf.
#-----------------------------------------------------------
function list_repos() {
sed -n '/REPOSITORIES/,$ s/^[(.*)]/1/p' $pacmanconf
}

#-----------------------------------------------------------
# Print active mirrors in $mirrorlist.
#-----------------------------------------------------------
function list_mirrors() {
sed -n 's/^Server = (.*)$/1/p' $mirrorlist
}

#-----------------------------------------------------------
# Download the filelist for repo $1. Assumes $mirrors
# is an array of urls to check for $file.
#-----------------------------------------------------------
function download() {
repo=$1
file="${repo}.files.tar.gz"

i=0
while [[ ! -e "${db}/${file}" ]]; do
if (( $i >= ${#mirrors[@]} )); then
return 1
fi

url=$(echo "${mirrors[$i]}" | sed "s/$repo/$repo/g")
wget --connect-timeout=15 -P "$db" "${url}/${file}"
let i+=1
done
}

#-----------------------------------------------------------
# Decompress $1 into dir $2.
#-----------------------------------------------------------
function decompress() {
file=$1
repo=$2

mkdir "${db}/${repo}"
tar -xzf "${db}/${file}" -C "${db}/${repo}"
rm -f "${db}/${file}"
}

#-----------------------------------------------------------
# Download the filelist for each active repo in
# pacman.conf. Tries mirrors in order until successful.
#-----------------------------------------------------------
function sync() {
mirrors=($(list_mirrors))
for repo in $(list_repos); do
echo "Fetching filelist for $repo"
download "$repo"
if [[ $? == 0 ]]; then
decompress "$file" "$repo"
else
echo
echo "Error: unable to retrieve $file from any mirror."
echo
fi
done
}

#-----------------------------------------------------------
# Print usage message.
#-----------------------------------------------------------
function usage() {
echo "usage: $(basename $0) [options] [pattern]"
echo echo "Search package filelists for [pattern]."
echo "options:"
echo " $(basename $0) [-h --help] Print this help message."
echo " $(basename $0) [-S --sync] Syncronize package filelists."
}

#-----------------------------------------------------------
# Start of `main'.
#-----------------------------------------------------------
case "$1" in
-S|--sync)
if [[ $EUID != 0 ]]; then
echo "error: you cannot perform this operation unless you are root."
exit 1;
fi

echo "Syncronizing ${db}..."; echo
rm -rf "$db" && mkdir -p "$db"
sync
;;

-h|--help)
usage
;;
*)
if [[ $# < 1 ]]; then
echo "not enough parameters"; echo
usage
exit 1
fi

if [[ ! -d "$db" ]]; then
echo "File database "$db" does not exist!"
echo "Have you synchronized the database?"
echo
exit 1
fi

grep -rE "$@" "$db" | sed "s|$db/(.*)/files:|1 |g"
;;
esac

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

pete morris wrote:
> Anyway, I've written a script that takes advantage of the [repo].files.tar.gz
> filelists that are now available to discover which package provides a
> particular file, when that package is not installed on the user's machine. It
> is similar to debian's apt-file, if you are familiar with that. I would
> appreciate any comments on the script, with an eye toward possible inclusion
> in Allan's pacman-contrib community package.
>

No comments on the actual script yet (I will look through it over the
weekend).

I do like the idea of this script and it is a feature users have
requests several times. However, it is currently Arch specific given it
relies on file lists that are only created for the Arch repos. How
about including Pierre's file list generation script in the scripts/
directory along with repo-add etc? I think these two really need to be
added in combination.

I am going to email Pierre and see how general his file list generation
script is.

Allan



_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

On Fri, Jun 27, 2008 at 1:44 AM, Allan McRae <allan@archlinux.org> wrote:
> pete morris wrote:
>> Anyway, I've written a script that takes advantage of the [repo].files.tar.gz
>> filelists that are now available to discover which package provides a
>> particular file, when that package is not installed on the user's machine. It
>> is similar to debian's apt-file, if you are familiar with that. I would
>> appreciate any comments on the script, with an eye toward possible inclusion
>> in Allan's pacman-contrib community package.
>>
>
> No comments on the actual script yet (I will look through it over the
> weekend).
>
> I do like the idea of this script and it is a feature users have
> requests several times. However, it is currently Arch specific given it
> relies on file lists that are only created for the Arch repos. How
> about including Pierre's file list generation script in the scripts/
> directory along with repo-add etc? I think these two really need to be
> added in combination.
>
> I am going to email Pierre and see how general his file list generation
> script is.

It's in the dbscripts git repo:
http://projects.archlinux.org/?p=dbscripts.git;a=blob;f=cron-jobs/createFileLists;h=d6cbd18d9084be1cae733a7a5cf85fd9 aa98d637;hb=HEAD

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

Am Freitag, 27. Juni 2008 08:44:25 schrieb Allan McRae:
> I do like the idea of this script and it is a feature users have
> requests several times. *However, it is currently Arch specific given it
> relies on file lists that are only created for the Arch repos. *How
> about including Pierre's file list generation script in the scripts/
> directory along with repo-add etc? *I think these two really need to be
> added in combination.

ATM the script is separated from repo-add and run by a cron job. I think it
would be best to include it into repo-add. This way it would be quite
independent from arch, more effective because repo-add knows which packages
have changed and filelists and package dbs would allways be consistent. Last
but not least this should simplify the script.

Of course this should only be optional and enabled by a switch
like "repo-add --with-filelists"

--

Pierre Schmitz


Clemens-August-Straße 76
53115 Bonn

Telefon 0228 9716608
Mobil 0160 95269831
Jabber pierre@jabber.archlinux.de
WWW http://www.archlinux.de

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev

pac-file utility script
 

On Fri, Jun 27, 2008 at 6:44 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
> Of course this should only be optional and enabled by a switch
> like "repo-add --with-filelists"
Sure, it must be optional. But couldn't it be enabled by default?
I see a repository filelist as a standard feature (which I often miss
with pacman-based systems)...

--
Geoffroy Carrier

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev