FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.

» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian ISP

LinkBack Thread Tools
Old 07-26-2011, 09:29 AM
Marek Podmaka
Default awstats and too many unique hosts

Hello all,

I have one vhost with 12 GB (about 30 million lines) nginx logfile per
day. As it is ad-serving vhost, it has too many unique hosts. Running
sort+uniq on random day says it has 3 millions unique IPs. After
piping through sed script for stripping last part of IP (so I have
only xxx.yyy.zzz.1 then) goes down to 700.000, but that is still too
much for awstats. It processes about 3-7.000 lines per second, but
stays at "Flush history file on disk (unique hosts reach flush limit
of 20000)" for many seconds or even minutes as the number of IPs is
rising. I am using GeoIP.

Is there any other way how to speed this up? I'm not interested in
exact per-IP stats, per country as per provider is OK, but I don't
know how to reduce the unique IPs anymore. Plain reduction (like
making whole /23 to one IP) might jump the IP to different provider
and even country. And I don't know how to easily obtain provider data
from whois (so that I could replace each IP with first IP of
provider's subnet from whois).

Any ideas? Or maybe some awstats tuning? (I'm running awstats 6.7 from

-- bYE, Marki

To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1008638190.20110726112954@marki-online.net">http://lists.debian.org/1008638190.20110726112954@marki-online.net

Thread Tools

All times are GMT. The time now is 10:14 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org