Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian User (http://www.linux-archive.org/debian-user/)
-   -   Slow Script (http://www.linux-archive.org/debian-user/239369-slow-script.html)

"Gorka" 02-03-2009 04:14 PM

Slow Script
 
Hi! I've got a perl script with this for:

for (my $j=0;$j<=$#fichero1;$j++)
{
if (@fichero1[$j] eq $valor1)
{
$token = 1;
}
}

The problem is that fichero1 has 32 millions of records and moreover I've
got to repeat this for several millions times, so this way it would take
years to finish.
Does anybody know a way to optimize this script? Is there any other linux
programing language I could make this more quickly whith?
Thank you!




--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Dave Sherohman 02-03-2009 04:24 PM

Slow Script
 
On Tue, Feb 03, 2009 at 06:14:48PM +0100, Gorka wrote:
> Hi! I've got a perl script with this for:
>
> for (my $j=0;$j<=$#fichero1;$j++)
> {
> if (@fichero1[$j] eq $valor1)
> {
> $token = 1;
> }
> }
>
> The problem is that fichero1 has 32 millions of records and moreover I've
> got to repeat this for several millions times, so this way it would take
> years to finish.
> Does anybody know a way to optimize this script? Is there any other linux
> programing language I could make this more quickly whith?
> Thank you!

Although the Perl could definitely be optimized (and you've already been
shown one way to do so), your core issue is that you're doing several
million passes over 32 million records. That's not going to be fast in
any language. (Even if you can check a million records per second,
that's 32 seconds per pass, or about 6 hours for 1,000 passes, or just
over a year for a million passes.)

Given the small piece of code that you posted and the magnitude of the
numbers you've stated, I strongly suspect that you probably want to use
a database for this, since it's exactly the kind of thing they're
designed to handle, but we can't say much more than that without
additional information about what you're actually trying to accomplish.

--
Dave Sherohman
NomadNet, Inc.
http://nomadnetinc.com/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

"Eugene V. Lyubimkin" 02-03-2009 04:27 PM

Slow Script
 
Gorka wrote:
> Hi! I've got a perl script with this for:
>
> for (my $j=0;$j<=$#fichero1;$j++)
> {
> if (@fichero1[$j] eq $valor1)
> {
> $token = 1;
> }
> }
>
Try

foreach (@fichero1)
{
if ($_ eq $valor1)
{
$token = 1;
break;
}
}

--
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
Ukrainian C++ Developer, Debian Maintainer, APT contributor


All times are GMT. The time now is 11:38 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.