FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Pacman Development

 
 
LinkBack Thread Tools
 
Old 02-11-2008, 07:59 AM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/10, Roman Kyrylych <roman.kyrylych@gmail.com>:
> Hi!
>
> The inability of makepkg to resume broken source downloads annoyed me
> pretty much, so here's a patch attached. ;-)
>

And with the help of Xavier I've remembered the bug that can be
re-opened and closed as "Fixed" now. ;-)
http://bugs.archlinux.org/task/9299

--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-11-2008, 08:33 AM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/11, Roman Kyrylych <roman.kyrylych@gmail.com>:
> 2008/2/10, Roman Kyrylych <roman.kyrylych@gmail.com>:
> > Hi!
> >
> > The inability of makepkg to resume broken source downloads annoyed me
> > pretty much, so here's a patch attached. ;-)
> >
>
> And with the help of Xavier I've remembered the bug that can be
> re-opened and closed as "Fixed" now. ;-)
> http://bugs.archlinux.org/task/9299
>

Previous patch didn't care about path with spaces,
here is a fixed version which restores previous behaviour of putting
paths/urls in quotes (").

--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-11-2008, 12:33 PM
"Dan McGee"
 
Default makepkg: Support for resuming source downloads

On Feb 11, 2008 3:33 AM, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
> Previous patch didn't care about path with spaces,
> here is a fixed version which restores previous behaviour of putting
> paths/urls in quotes (").

Can you please resend this as inline so we can comment on it easier?

-Dan

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-11-2008, 01:10 PM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/11, Dan McGee <dpmcgee@gmail.com>:
> On Feb 11, 2008 3:33 AM, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
> > Previous patch didn't care about path with spaces,
> > here is a fixed version which restores previous behaviour of putting
> > paths/urls in quotes (").
>
> Can you please resend this as inline so we can comment on it easier?
>

OK, I just had some bad experience with Gmail in past.


From 1196a6b6a97c17027608d6c92a3df647deab114c Mon Sep 17 00:00:00 2001
From: Roman Kyrylych <roman@archlinux.org>
Date: Sun, 10 Feb 2008 18:56:47 +0200
Subject: [PATCH] makepkg: Support for resuming source downloads

download to *.part and remove the suffix if successful,
thus md5sums will be checked only for complete files,
and next time makepkg will resume downloading of partial file
instead of complaining about invalid md5sums

Signed-off-by: Roman Kyrylych <roman@archlinux.org>
---
doc/makepkg.conf.5.txt | 3 +++
etc/makepkg.conf.in | 10 +++++-----
scripts/makepkg.sh.in | 13 ++++++++++++-
3 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt
index f6e4b38..ec72e29 100644
--- a/doc/makepkg.conf.5.txt
+++ b/doc/makepkg.conf.5.txt
@@ -37,6 +37,9 @@ Options
well; the download URL is placed on the end of the command. This is more
flexible than the former `FTPAGENT` variable, as any protocol can have a
download agent. Several examples are provided in the default makepkg.conf.
+ All instances of `%u` will be replaced with the download URL. If present,
+ instances of `%o` will be replaced with the local filename, plus a ``.part'
+ extension, which allows yo do file resumes properly.

**CARCH=**"carch"::
Specifies your computer architecture; possible values include such things
diff --git a/etc/makepkg.conf.in b/etc/makepkg.conf.in
index d808270..e8ac6a1 100644
--- a/etc/makepkg.conf.in
+++ b/etc/makepkg.conf.in
@@ -8,11 +8,11 @@
#
#-- The download utilities that makepkg should use to acquire sources
# Format: 'protocol::agent'
-DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3'
- 'http::/usr/bin/wget -c -t 3 --waitretry=3'
- 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate'
- 'rsync::/usr/bin/rsync -z'
- 'scp::/usr/bin/scp -C')
+DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3 -O %o %u'
+ 'http::/usr/bin/wget -c -t 3 --waitretry=3 -O %o %u'
+ 'https::/usr/bin/wget -c -t 3 --waitretry=3
--no-check-certificate -O %o %u'
+ 'rsync::/usr/bin/rsync -z %u %o'
+ 'scp::/usr/bin/scp -C %u %o')

# Other common tools:
# /usr/bin/snarf
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index cecda1d..27bf7fa 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -299,6 +299,16 @@ get_downloadclient() {
echo "$agent"
}

+get_downloadcmd() {
+ local dlagent=$1
+ local netfile=$2
+ local file=$3
+
+ local dlcmd=$(echo $dlagent | sed "s|%o|"$file.part"|" | sed
"s|%u|"$netfile"|")
+
+ echo "$dlcmd"
+}
+
check_deps() {
[ $# -gt 0 ] || return

@@ -476,12 +486,13 @@ download_sources() {
msg2 "$(gettext "Downloading %s...")" "$file"
# fix flyspray bug #3289
local ret=0
- $dlclient "$netfile" || ret=$?
+ $(get_downloadcmd "$dlclient" "$netfile" "$file") || ret=$?
if [ $ret -gt 0 ]; then
error "$(gettext "Failure while downloading %s")" "$file"
plain "$(gettext "Aborting...")"
exit 1
fi
+ mv -f "$SRCDEST/$file.part" "$SRCDEST/$file"
cp -s --remove-destination "$SRCDEST/$file" "$srcdir/"
done

--
1.5.3.7


--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-11-2008, 03:17 PM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/11, Miklos Vajna <vmiklos@frugalware.org>:
> On Mon, Feb 11, 2008 at 04:10:51PM +0200, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
> > + extension, which allows yo do file resumes properly.
>
> typo?
>

Yep. I've took that part from the description of XferCommand and must
have broken the sentence while editing.

--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-11-2008, 08:25 PM
Xavier
 
Default makepkg: Support for resuming source downloads

Roman Kyrylych wrote:
> 2008/2/11, Roman Kyrylych<roman.kyrylych@gmail.com>:
>> 2008/2/10, Roman Kyrylych<roman.kyrylych@gmail.com>:
>>> Hi!
>>>
>>> The inability of makepkg to resume broken source downloads annoyed me
>>> pretty much, so here's a patch attached. ;-)
>>>
>> And with the help of Xavier I've remembered the bug that can be
>> re-opened and closed as "Fixed" now. ;-)
>> http://bugs.archlinux.org/task/9299
>>
>
> Previous patch didn't care about path with spaces,
> here is a fixed version which restores previous behaviour of putting
> paths/urls in quotes (").
>
>

For some reason, adding these quotes broke it.

-> Downloading opera-9.50-20080131.6-shared-qt.i386-1786.tar.bz2...
"http://snapshot.opera.com/unix/snapshot-1786/intel-linux/opera-9.50-20080131.6-shared-qt.i386-1786.tar.bz2":
Unsupported scheme.
==> ERROR: Failure while downloading
opera-9.50-20080131.6-shared-qt.i386-1786.tar.bz2

The original version works fine though. But I didn't check what happens
with spaces..

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-12-2008, 01:50 AM
"Dan McGee"
 
Default makepkg: Support for resuming source downloads

> From 1196a6b6a97c17027608d6c92a3df647deab114c Mon Sep 17 00:00:00 2001
> From: Roman Kyrylych <roman@archlinux.org>
> Date: Sun, 10 Feb 2008 18:56:47 +0200
> Subject: [PATCH] makepkg: Support for resuming source downloads
>
> download to *.part and remove the suffix if successful,
> thus md5sums will be checked only for complete files,
> and next time makepkg will resume downloading of partial file
> instead of complaining about invalid md5sums
>
> Signed-off-by: Roman Kyrylych <roman@archlinux.org>
> ---
> doc/makepkg.conf.5.txt | 3 +++
> etc/makepkg.conf.in | 10 +++++-----
> scripts/makepkg.sh.in | 13 ++++++++++++-
> 3 files changed, 20 insertions(+), 6 deletions(-)
>
> diff --git a/doc/makepkg.conf.5.txt b/doc/makepkg.conf.5.txt
> index f6e4b38..ec72e29 100644
> --- a/doc/makepkg.conf.5.txt
> +++ b/doc/makepkg.conf.5.txt
> @@ -37,6 +37,9 @@ Options
> well; the download URL is placed on the end of the command. This is more
> flexible than the former `FTPAGENT` variable, as any protocol can have a
> download agent. Several examples are provided in the default makepkg.conf.
> + All instances of `%u` will be replaced with the download URL. If present,
> + instances of `%o` will be replaced with the local filename, plus a ``.part'
> + extension, which allows yo do file resumes properly.
Typo, but no big deal. :P

> index d808270..e8ac6a1 100644
> --- a/etc/makepkg.conf.in
> +++ b/etc/makepkg.conf.in
> @@ -8,11 +8,11 @@
> #
> #-- The download utilities that makepkg should use to acquire sources
> # Format: 'protocol::agent'
> -DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3'
> - 'http::/usr/bin/wget -c -t 3 --waitretry=3'
> - 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate'
> - 'rsync::/usr/bin/rsync -z'
> - 'scp::/usr/bin/scp -C')
> +DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3 -O %o %u'
> + 'http::/usr/bin/wget -c -t 3 --waitretry=3 -O %o %u'
> + 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate -O %o %u'
> + 'rsync::/usr/bin/rsync -z %u %o'
> + 'scp::/usr/bin/scp -C %u %o')
Damn, I just wish we could leave what works alone, but that is too hard. :P

Is there anyway we can slightly rework the below so that if neither %o
or %u are found in the DLAGENT, we basically do the former behavior?
Not sure if this is completely feasible, but it would be nice to have
the simple option still available (and especially if someone is using
a download client that doesn't need all this complexity).

> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> index cecda1d..27bf7fa 100644
> --- a/scripts/makepkg.sh.in
> +++ b/scripts/makepkg.sh.in
> @@ -299,6 +299,16 @@ get_downloadclient() {
> echo "$agent"
> }
>
> +get_downloadcmd() {
> + local dlagent=$1
> + local netfile=$2
> + local file=$3
> +
> + local dlcmd=$(echo $dlagent | sed "s|%o|"$file.part"|" | sed
> "s|%u|"$netfile"|")
> +
> + echo "$dlcmd"
> +}
> +
> check_deps() {
> [ $# -gt 0 ] || return
>
> @@ -476,12 +486,13 @@ download_sources() {
> msg2 "$(gettext "Downloading %s...")" "$file"
> # fix flyspray bug #3289
> local ret=0
> - $dlclient "$netfile" || ret=$?
> + $(get_downloadcmd "$dlclient" "$netfile" "$file") || ret=$?
> if [ $ret -gt 0 ]; then
> error "$(gettext "Failure while downloading %s")" "$file"
> plain "$(gettext "Aborting...")"
> exit 1
> fi
> + mv -f "$SRCDEST/$file.part" "$SRCDEST/$file"
> cp -s --remove-destination "$SRCDEST/$file" "$srcdir/"
> done
>
> --
> 1.5.3.7

Someone doesn't run testing!

-Dan

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-12-2008, 08:35 AM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/12, Dan McGee <dpmcgee@gmail.com>:
> > index d808270..e8ac6a1 100644
> > --- a/etc/makepkg.conf.in
> > +++ b/etc/makepkg.conf.in
> > @@ -8,11 +8,11 @@
> > #
> > #-- The download utilities that makepkg should use to acquire sources
> > # Format: 'protocol::agent'
> > -DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3'
> > - 'http::/usr/bin/wget -c -t 3 --waitretry=3'
> > - 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate'
> > - 'rsync::/usr/bin/rsync -z'
> > - 'scp::/usr/bin/scp -C')
> > +DLAGENTS=('ftp::/usr/bin/wget -c --passive-ftp -t 3 --waitretry=3 -O %o %u'
> > + 'http::/usr/bin/wget -c -t 3 --waitretry=3 -O %o %u'
> > + 'https::/usr/bin/wget -c -t 3 --waitretry=3 --no-check-certificate -O %o %u'
> > + 'rsync::/usr/bin/rsync -z %u %o'
> > + 'scp::/usr/bin/scp -C %u %o')
> Damn, I just wish we could leave what works alone, but that is too hard. :P
>
> Is there anyway we can slightly rework the below so that if neither %o
> or %u are found in the DLAGENT, we basically do the former behavior?
> Not sure if this is completely feasible, but it would be nice to have
> the simple option still available (and especially if someone is using
> a download client that doesn't need all this complexity).
>

Yeah, I thought about that. Will try to do something in the next version.

> > diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> > index cecda1d..27bf7fa 100644
> > --- a/scripts/makepkg.sh.in
> > +++ b/scripts/makepkg.sh.in
> > @@ -299,6 +299,16 @@ get_downloadclient() {
> > echo "$agent"
> > }
> >
> > +get_downloadcmd() {
> > + local dlagent=$1
> > + local netfile=$2
> > + local file=$3
> > +
> > + local dlcmd=$(echo $dlagent | sed "s|%o|"$file.part"|" | sed
> > "s|%u|"$netfile"|")
> > +
> > + echo "$dlcmd"
> > +}
> > +
> > check_deps() {
> > [ $# -gt 0 ] || return
> >
> > @@ -476,12 +486,13 @@ download_sources() {
> > msg2 "$(gettext "Downloading %s...")" "$file"
> > # fix flyspray bug #3289
> > local ret=0
> > - $dlclient "$netfile" || ret=$?
> > + $(get_downloadcmd "$dlclient" "$netfile" "$file") || ret=$?
> > if [ $ret -gt 0 ]; then
> > error "$(gettext "Failure while downloading %s")" "$file"
> > plain "$(gettext "Aborting...")"
> > exit 1
> > fi
> > + mv -f "$SRCDEST/$file.part" "$SRCDEST/$file"
> > cp -s --remove-destination "$SRCDEST/$file" "$srcdir/"
> > done
> >
> > --
> > 1.5.3.7
>
> Someone doesn't run testing!

I did test the first version (without " added to dlcmd= and it worked
flawlessly.
The addition of " in the second patch broke it :-)
So please use the version attached in my first message to test that it
works. ;-)

--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-12-2008, 07:39 PM
"Roman Kyrylych"
 
Default makepkg: Support for resuming source downloads

2008/2/12, Roman Kyrylych <roman.kyrylych@gmail.com>:
> I did test the first version (without " added to dlcmd= and it worked
> flawlessly.
> The addition of " in the second patch broke it :-)
> So please use the version attached in my first message to test that it
> works. ;-)

Ok, I'm running the patched makepkg for a couple of days already.
It turned out that "path with spaces" are non-issue due to a simple reason:
all spaces in URLs are converted to %20 anyway. :-)

--
Roman Kyrylych (*оман Кирилич)
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 
Old 02-12-2008, 07:55 PM
Xavier
 
Default makepkg: Support for resuming source downloads

Roman Kyrylych wrote:
> 2008/2/12, Roman Kyrylych<roman.kyrylych@gmail.com>:
>> I did test the first version (without " added to dlcmd= and it worked
>> flawlessly.
>> The addition of " in the second patch broke it :-)
>> So please use the version attached in my first message to test that it
>> works. ;-)
>
> Ok, I'm running the patched makepkg for a couple of days already.
> It turned out that "path with spaces" are non-issue due to a simple reason:
> all spaces in URLs are converted to %20 anyway. :-)
>

Oh, of course. And since the filename is taken from the url anyway, well
it shouldn't be a problem indeed.

_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://archlinux.org/mailman/listinfo/pacman-dev
 

Thread Tools




All times are GMT. The time now is 02:55 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org