FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 07-08-2008, 03:48 PM
Warren Togami
 
Default InstantMirror Redesign / current best /simplest way to achieve

Arthur Pemberton wrote:


I wanted to try InstantMirror, but was in a rush. I just used a basic
squid setup.

Are their any advantages of using it as a reverse mirror vs. regular
squid usage?



Regular squid wont be able to effectively cache if MirrorManager is
telling you to use random mirrors. If you use MirrorManager you could
receive the same local (reverse proxy) mirror as the first mirror every
time if yum is used from your network block.


refresh_pattern repodata/.*$ 0 0% 0
refresh_pattern .*rpm$ 0 0% 0

Also with any squid.conf you will need these lines in order to guarantee
that your repodata and RPMS stay consistent with your upstream source.
This is because proxies do not handle data changing without changing the
filename.


Warren Togami
wtogami@redhat.com

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 04:02 PM
"Arthur Pemberton"
 
Default InstantMirror Redesign / current best /simplest way to achieve

On Tue, Jul 8, 2008 at 10:48 AM, Warren Togami <wtogami@redhat.com> wrote:
> Arthur Pemberton wrote:
>>
>> I wanted to try InstantMirror, but was in a rush. I just used a basic
>> squid setup.
>>
>> Are their any advantages of using it as a reverse mirror vs. regular
>> squid usage?
>>
>
> Regular squid wont be able to effectively cache if MirrorManager is telling
> you to use random mirrors. If you use MirrorManager you could receive the
> same local (reverse proxy) mirror as the first mirror every time if yum is
> used from your network block.
>
> refresh_pattern repodata/.*$ 0 0% 0
> refresh_pattern .*rpm$ 0 0% 0
>
> Also with any squid.conf you will need these lines in order to guarantee
> that your repodata and RPMS stay consistent with your upstream source. This
> is because proxies do not handle data changing without changing the
> filename.
>
> Warren Togami
> wtogami@redhat.com


Ok thanks, what I did was comment out the mirror list url, and just
use the base url. I'll add those refresh patterns, but doesn't the
second one effectively turn of caching of *.rpm?


--
Fedora 7 : sipping some of that moonshine
( www.pembo13.com )

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 04:09 PM
Warren Togami
 
Default InstantMirror Redesign / current best /simplest way to achieve

Gary Thomas wrote:

Warren Togami wrote:

Arthur Pemberton wrote:


I wanted to try InstantMirror, but was in a rush. I just used a basic
squid setup.

Are their any advantages of using it as a reverse mirror vs. regular
squid usage?



Regular squid wont be able to effectively cache if MirrorManager is
telling you to use random mirrors. If you use MirrorManager you could
receive the same local (reverse proxy) mirror as the first mirror
every time if yum is used from your network block.


refresh_pattern repodata/.*$ 0 0% 0
refresh_pattern .*rpm$ 0 0% 0

Also with any squid.conf you will need these lines in order to
guarantee that your repodata and RPMS stay consistent with your
upstream source. This is because proxies do not handle data changing
without changing the filename.


Since you seem to have this all figured out, could you share
your squid.conf?

Thanks



Unfortunately my squid.conf is a bit too complex to such a point that I
don't understand how it works anymore. The reverse proxy configuration
part is also different between squid in RHEL and the newer squid
versions in Fedora. Search around for the magic options...


Warren

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 04:12 PM
Warren Togami
 
Default InstantMirror Redesign / current best /simplest way to achieve

Arthur Pemberton wrote:


refresh_pattern repodata/.*$ 0 0% 0
refresh_pattern .*rpm$ 0 0% 0

Also with any squid.conf you will need these lines in order to guarantee
that your repodata and RPMS stay consistent with your upstream source. This
is because proxies do not handle data changing without changing the
filename.

Warren Togami
wtogami@redhat.com



Ok thanks, what I did was comment out the mirror list url, and just
use the base url. I'll add those refresh patterns, but doesn't the
second one effectively turn of caching of *.rpm?




Not exactly. It checks with the source server on every request if the
data changed, but it doesn't re-download the entire thing. The only way
we could do this without checking the upstream source is if all
filenames on the mirrors changed every time their contents change. This
is possible and we considered this for repodata, but decided against it
because it would have broke earlier clients. This is also currently not
possible with the RPMS themselves. Hence the need for refresh_pattern
rules.


refresh_pattern images/.*$ 0 0% 0

I just realized that you probably want this additional rule to provide
the same guarantees for stage2.img and other stuff in that directory.


Warren Togami
wtogami@redhat.com

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 04:22 PM
"Arthur Pemberton"
 
Default InstantMirror Redesign / current best /simplest way to achieve

On Tue, Jul 8, 2008 at 11:12 AM, Warren Togami <wtogami@redhat.com> wrote:
> Arthur Pemberton wrote:
>>>
>>> refresh_pattern repodata/.*$ 0 0% 0
>>> refresh_pattern .*rpm$ 0 0% 0
>>>
>>> Also with any squid.conf you will need these lines in order to guarantee
>>> that your repodata and RPMS stay consistent with your upstream source.
>>> This
>>> is because proxies do not handle data changing without changing the
>>> filename.
>>>
>>> Warren Togami
>>> wtogami@redhat.com
>>
>>
>> Ok thanks, what I did was comment out the mirror list url, and just
>> use the base url. I'll add those refresh patterns, but doesn't the
>> second one effectively turn of caching of *.rpm?
>>
>>
>
> Not exactly. It checks with the source server on every request if the data
> changed, but it doesn't re-download the entire thing. The only way we could
> do this without checking the upstream source is if all filenames on the
> mirrors changed every time their contents change. This is possible and we
> considered this for repodata, but decided against it because it would have
> broke earlier clients. This is also currently not possible with the RPMS
> themselves. Hence the need for refresh_pattern rules.
>
> refresh_pattern images/.*$ 0 0% 0
>
> I just realized that you probably want this additional rule to provide the
> same guarantees for stage2.img and other stuff in that directory.
>
> Warren Togami
> wtogami@redhat.com
>

Okay thanks, I had apparently not completely understood how refresh
patterns were used, I am clearer now.

--
Fedora 7 : sipping some of that moonshine
( www.pembo13.com )

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 05:36 PM
Nicolas Mailhot
 
Default InstantMirror Redesign / current best /simplest way to achieve

Le mardi 08 juillet 2008 à 12:12 -0400, Warren Togami a écrit :

> The only way
> we could do this without checking the upstream source is if all
> filenames on the mirrors changed every time their contents change. This
> is possible and we considered this for repodata, but decided against it
> because it would have broke earlier clients.

So repodata is condemned to be broken with proxies just because it was
designed broken ? Please reconsider. There are many places in the world
where you can only access trhough proxies (for good reasons) and yum can
not really be used right now.

Just autogenerate to sets of metadata and deprecate the proxy-unfriendly
version after a few years.

--
Nicolas Mailhot
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 06:08 PM
Jesse Keating
 
Default InstantMirror Redesign / current best /simplest way to achieve

On Tue, 2008-07-08 at 19:36 +0200, Nicolas Mailhot wrote:
> Le mardi 08 juillet 2008 à 12:12 -0400, Warren Togami a écrit :
>
> > The only way
> > we could do this without checking the upstream source is if all
> > filenames on the mirrors changed every time their contents change. This
> > is possible and we considered this for repodata, but decided against it
> > because it would have broke earlier clients.
>
> So repodata is condemned to be broken with proxies just because it was
> designed broken ? Please reconsider. There are many places in the world
> where you can only access trhough proxies (for good reasons) and yum can
> not really be used right now.
>
> Just autogenerate to sets of metadata and deprecate the proxy-unfriendly
> version after a few years.

Repodata will very shortly have a unique name to it, rather than a
static name. repomd.xml will remain unchanged though.

--
Jesse Keating
Fedora -- Freedom² is a feature!
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 06:20 PM
chasd
 
Default InstantMirror Redesign / current best /simplest way to achieve

Warren Togami wrote:


Arthur Pemberton wrote:


I wanted to try InstantMirror, but was in a rush. I just used a basic
squid setup.

Are their any advantages of using it as a reverse mirror vs. regular
squid usage?



Regular squid wont be able to effectively cache if MirrorManager is
telling you to use random mirrors. If you use MirrorManager you could
receive the same local (reverse proxy) mirror as the first mirror
every

time if yum is used from your network block.

refresh_pattern repodata/.*$ 0 0% 0
refresh_pattern .*rpm$ 0 0% 0

Also with any squid.conf you will need these lines in order to
guarantee

that your repodata and RPMS stay consistent with your upstream source.
This is because proxies do not handle data changing without
changing the

filename.


Um, from reading the comments in the squid config file relating to
"refresh_pattern," don't those settings in effect negate caching,
since that will always return STALE ? From the squid config file :


# Basically a cached object is:
#
# FRESH if expires < now, else STALE
# STALE if age > max
# FRESH if lm-factor < percent, else STALE
# FRESH if age < min
# else STALE


Wouldn't you want squid to keep a file at least a couple of hours, if
not a day ?


Settings I use to better cache OS updates using squid :

# Use more than the default 8 MB of RAM
cache_mem 128 MB
# Use 8 MB for the largest object in RAM cache instead of 8KB
maximum_object_size_in_memory 8192 KB

A large percentage of RPMs are less than 8 MB

# Use 30GB for the cache instead of the default 100 MB
cache_dir aufs /var/spool/squid 30000 48 512

If you have an update run that totals close to the default 100 MB,
and you have multiple clients set to update via a cron script,
the cache will thrash if you don't set that higher.

Those settings above are hardware dependent,
if you have more, let squid use more

I also use an ext2 partition for where the cache lives since it
seems the journaling of ext3 can impact squid performance,
particularly on older PPC hardware kjournald will keep rising to the
first line

of "top" and squid can block on I / O.
If the fs gets mangled, only the squid cache is on it
so I can wipe it and re-init the cache

# Use maximum object size of 120 MB not 4 MB - gotta cache openoffice
fer sure.

maximum_object_size 122880 KB

That said, I use InstantMirror with a custom repo file for our Fedora
needs.



Charles Dostale

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 
Old 07-08-2008, 06:34 PM
Warren Togami
 
Default InstantMirror Redesign / current best /simplest way to achieve

chasd wrote:

refresh_pattern repodata/.*$ 0 0% 0
refresh_pattern .*rpm$ 0 0% 0

Also with any squid.conf you will need these lines in order to guarantee
that your repodata and RPMS stay consistent with your upstream source.
This is because proxies do not handle data changing without changing the
filename.


Um, from reading the comments in the squid config file relating to
"refresh_pattern," don't those settings in effect negate caching,
since that will always return STALE ? From the squid config file :


This was already discussed earlier in this thread. This doesn't expire
the object from the cache entirely. It does check the upstream source
for changes but it doesn't need to download the entire file again if it
did not change.



If you have an update run that totals close to the default 100 MB,
and you have multiple clients set to update via a cron script,
the cache will thrash if you don't set that higher.


Right, you need to adjust the cache_dir size and maximum object size. I
personally use a maximum object size of like 4GB because my reverse
proxy is used *only* for a Fedora mirror.


That said, I use InstantMirror with a custom repo file for our Fedora
needs.


Note, InstantMirror behaves exactly as described above with the squid
refresh_pattern rule. InstantMirror checks the original source files
for changes upon every client request.


Due to repodata/*, images/* and sometimes *.rpm files changing without
their contents changing, running a caching proxy without such rules
causes you to occasionally have metadata mismatch failures until the
proxy server naturally refreshes itself sometime later.


Warren Togami
wtogami@redhat.com

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
 

Thread Tools




All times are GMT. The time now is 05:46 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org