Properly handling missing files when downloading files from ftp:// via ISA proxy (emerge/wget)

09-10-2010, 02:43 PM

Hi,

Is there any way to make emerge (wget) correctly behave when it tries
to download a non-existing file from FTP
in a network using ISA as ftp_proxy ? I have one of these at work and
it's really annoying because of situations
like below:

pazuzu ~ # LC_ALL=C wget -T 30 -t 1 --continue -O
/var/cache/src/cvsps-2.1.tar.gz
ftp://ftp.gentoo.mesh-solutions.com/gentoo/distfiles/cvsps-2.1.tar.gz
--2010-09-10 16:31:25--
ftp://ftp.gentoo.mesh-solutions.com/gentoo/distfiles/cvsps-2.1.tar.gz
Connecting to 10.144.1.10:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/var/cache/src/cvsps-2.1.tar.gz'

[ <=>
] 318
--.-K/s in 0s

2010-09-10 16:31:25 (14.9 MB/s) - `/var/cache/src/cvsps-2.1.tar.gz' saved [318]

Wget thinks, that this file was downloaded when really it contains
some crap produced by ISA:

pazuzu ~ # cat /var/cache/src/cvsps-2.1.tar.gz
<HTML>
<meta http-equiv="Content-Type" content="text-html; charset=UTF-8">
<HEAD>
<TITLE>Server error message </TITLE>
</HEAD>
<BODY><H1>ISA Server: extended error message : </H1>
<PRE>200 Switching to Binary mode.
227 Entering Passive Mode (213,203,218,123,116,42)
550 Failed to open file.
</PRE>
</BODY>
</HTML>
pazuzu ~ #

Is there any way to work around this problem ?
Is it neccessary to change ISA configuration to properly respond to an
ftp client when
the file is missing ? If yes, how ?

Best regards,
Maciej Grela

09-10-2010, 03:26 PM

On Fri, Sep 10, 2010 at 9:43 AM, Maciej Grela <maciej.grela@gmail.com> wrote:
> Hi,
>
> Is there any way to make emerge (wget) correctly behave when it tries
> to download a non-existing file from FTP
> in a network using ISA as ftp_proxy ? I have one of these at work and
> it's really annoying because of situations

If you are allowed to bypass the proxy, try --use-proxy=off in your
wget command line.

It seems your FTP is proxied over HTTP and your client needs to
support this. I'm not sure if wget supports HTTP proxies for FTP.

Here is a document from Microsoft about configuring ISA and various clients:
http://technet.microsoft.com/en-us/library/bb794745.aspx

Good luck!

09-10-2010, 04:13 PM

2010/9/10 Paul Hartman <paul.hartman+gentoo@gmail.com>:
> On Fri, Sep 10, 2010 at 9:43 AM, Maciej Grela <maciej.grela@gmail.com> wrote:
>> Hi,
>>
>> Is there any way to make emerge (wget) correctly behave when it tries
>> to download a non-existing file from FTP
>> in a network using ISA as ftp_proxy ? I have one of these at work and
>> it's really annoying because of situations
>
> If you are allowed to bypass the proxy, try --use-proxy=off in your
> wget command line.
>

Unfortunately proxy is the only way to access the Internet.

> It seems your FTP is proxied over HTTP and your client needs to
> support this. I'm not sure if wget supports HTTP proxies for FTP.
>
> Here is a document from Microsoft about configuring ISA and various clients:
> http://technet.microsoft.com/en-us/library/bb794745.aspx
>

I'll read that, thanks.

Br,
Maciej Grela

09-11-2010, 07:44 AM

On Friday 10 September 2010 17:13:44 Maciej Grela wrote:
> 2010/9/10 Paul Hartman <paul.hartman+gentoo@gmail.com>:
> > On Fri, Sep 10, 2010 at 9:43 AM, Maciej Grela <maciej.grela@gmail.com>
wrote:
> >> Hi,
> >>
> >> Is there any way to make emerge (wget) correctly behave when it tries
> >> to download a non-existing file from FTP
> >> in a network using ISA as ftp_proxy ? I have one of these at work and
> >> it's really annoying because of situations
> >
> > If you are allowed to bypass the proxy, try --use-proxy=off in your
> > wget command line.
>
> Unfortunately proxy is the only way to access the Internet.
>
> > It seems your FTP is proxied over HTTP and your client needs to
> > support this. I'm not sure if wget supports HTTP proxies for FTP.
> >
> > Here is a document from Microsoft about configuring ISA and various
> > clients: http://technet.microsoft.com/en-us/library/bb794745.aspx
>
> I'll read that, thanks.

I think that you can try exporting your proxy's http address to ftp like so:

# export ftp_proxy="http://my_proxy.com:1234"

and then run emerge to see if you can get through. If your proxy requires a
username/passwd you'll need to add these on the command line just as the
handbook advises.

--
Regards,
Mick

09-11-2010, 10:22 PM

On Sat, Sep 11, 2010 at 5:44 PM, Mick <michaelkintzios@gmail.com> wrote:

On Friday 10 September 2010 17:13:44 Maciej Grela wrote:

> 2010/9/10 Paul Hartman <paul.hartman+gentoo@gmail.com>:

> > On Fri, Sep 10, 2010 at 9:43 AM, Maciej Grela <maciej.grela@gmail.com>

wrote:

> >> Hi,

> >>

> >> Is there any way to make emerge (wget) correctly behave when it tries

> >> to download a non-existing file from FTP

> >> in a network using ISA as ftp_proxy ? I have one of these at work and

> >> it's really annoying because of situations

> >

> > If you are allowed to bypass the proxy, try --use-proxy=off in your

> > wget command line.

>

> Unfortunately proxy is the only way to access the Internet.

>

> > It seems your FTP is proxied over HTTP and your client needs to

> > support this. I'm not sure if wget supports HTTP proxies for FTP.

> >

> > Here is a document from Microsoft about configuring ISA and various

> > clients: http://technet.microsoft.com/en-us/library/bb794745.aspx

>

> I'll read that, thanks.

I think that you can try exporting your proxy's http address to ftp like so:

# export ftp_proxy="http://my_proxy.com:1234"

and then run emerge to see if you can get through. *If your proxy requires a

username/passwd you'll need to add these on the command line just as the

handbook advises.

--

Regards,

Mick

Hi
You might find the problem is NTLM authentication. I have found*net-proxy/ntlmaps to work around this issue.

Kind regards

Brett Freer
www.rhapsody.com.au