FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 01-25-2009, 12:12 AM
Filipe Brandenburger
 
Default OT? File order on CentOS/Samba server

Oi Miguel,

On Sat, Jan 24, 2009 at 15:24, Miguel Medalha <miguelmedalha@sapo.pt> wrote:
> Thank you for caring to look for and post the code.

No problem! Glad to help.

> At first I became very excited about it. But then I tried it...
>
> It does work. The problem is that it suffers from the same illness as
> runfilex does: it takes forever. The process starts very swiftly but each
> new processed page takes longer and longer until it all slows to a crawl.
> Worse yet, Distiller goes on to use enormous (> 90%) amounts of CPU time.
>
> I just measured the process as folllows, for the same set of files,
> corresponding to a 32 page publication in A3 format:
>
> rundirex: 3m42s
> runfilex: 1h29m54s
> Wikipedia code: 1h14m55s

That is really weird, since it's only sorting a list before starting
the processing, but once the processing is started, it does exactly
the same in both cases (the only difference is that in one case
"filenameforall" is used and in the other case "forall" is used over
an array with the sorted list of files).

Do you have a support contract with Adobe? If you do, I think you
should bring up this issue with them and try to figure out where the
huge performance difference is coming from, since it should not.

> I suppose I will end up creating a FAT32 partition on the server just for
> this purpose.

and:

> I just turned dir_index OFF with tune2fs. Now the directory order is the
> same as the inode order. This makes the order of files predictable and
> in fact turns out to solve my problem.
>
> With dir_index turned OFF on that filesystem, when a copy is made to
> another directory (even from Windows on a Samba share) the
> alphanumeric order is preserved. I will just ask the workstation
> operators to copy the PS files to a new folder when they are all
> ready. Distiller is watching that folder and will process the files in the
> normal way, using the rundirex file.

I don't think turning dir_index off will make the order as predictable
as you want it. It may be a good enough work around for now, but it
might lead to strange problems in the future that you may end up
having to deal with again.

I would really advise you to investigate why when you list the files
in the order you want in the input file it takes so long.

Boa Sorte!
Filipe
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-25-2009, 02:57 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server

> That is really weird, since it's only sorting a list before starting
> the processing, but once the processing is started, it does exactly
> the same in both cases (the only difference is that in one case
> "filenameforall" is used and in the other case "forall" is used over
> an array with the sorted list of files).
>
>

Weird indeed. But repeatable and confirmed again and again. It doesn't
happen only over the network. It happens at the local level too, with
Distiller and all files on the same workstation.

> I don't think turning dir_index off will make the order as predictable
> as you want it. It may be a good enough work around for now, but it
> might lead to strange problems in the future that you may end up
> having to deal with again.
>
>

I don't have a choice right now. Newspapers don't wait, you know

> I would really advise you to investigate why when you list the files
> in the order you want in the input file it takes so long.
>
>

I will certainly investigate it. I am a curious mind, for the worse and
for the better

> Boa Sorte!
>

Agradeço. Saudações!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-25-2009, 03:21 PM
"John"
 
Default OT? File order on CentOS/Samba server

> -----Original Message-----
> From: centos-bounces@centos.org
> [mailto:centos-bounces@centos.org] On Behalf Of Filipe Brandenburger
> Sent: Saturday, January 24, 2009 8:13 PM
> To: CentOS mailing list
> Subject: Re: [CentOS] OT? File order on CentOS/Samba server
>
> Oi Miguel,
>
> On Sat, Jan 24, 2009 at 15:24, Miguel Medalha
> <miguelmedalha@sapo.pt> wrote:
> > Thank you for caring to look for and post the code.
>
> No problem! Glad to help.
>
> > At first I became very excited about it. But then I tried it...
> >
> > It does work. The problem is that it suffers from the same
> illness as
> > runfilex does: it takes forever. The process starts very
> swiftly but each
> > new processed page takes longer and longer until it all
> slows to a crawl.
> > Worse yet, Distiller goes on to use enormous (> 90%)
> amounts of CPU time.
> >
> > I just measured the process as folllows, for the same set of files,
> > corresponding to a 32 page publication in A3 format:
> >
> > rundirex: 3m42s
> > runfilex: 1h29m54s
> > Wikipedia code: 1h14m55s
>
> That is really weird, since it's only sorting a list before starting
> the processing, but once the processing is started, it does exactly
> the same in both cases (the only difference is that in one case
> "filenameforall" is used and in the other case "forall" is used over
> an array with the sorted list of files).
>
> Do you have a support contract with Adobe? If you do, I think you
> should bring up this issue with them and try to figure out where the
> huge performance difference is coming from, since it should not.
>
> > I suppose I will end up creating a FAT32 partition on the
> server just for
> > this purpose.
>
> and:
>
> > I just turned dir_index OFF with tune2fs. Now the directory
> order is the
> > same as the inode order. This makes the order of files
> predictable and
> > in fact turns out to solve my problem.
> >
> > With dir_index turned OFF on that filesystem, when a copy is made to
> > another directory (even from Windows on a Samba share) the
> > alphanumeric order is preserved. I will just ask the workstation
> > operators to copy the PS files to a new folder when they are all
> > ready. Distiller is watching that folder and will process
> the files in the
> > normal way, using the rundirex file.
>
> I don't think turning dir_index off will make the order as predictable
> as you want it. It may be a good enough work around for now, but it
> might lead to strange problems in the future that you may end up
> having to deal with again.
>
> I would really advise you to investigate why when you list the files
> in the order you want in the input file it takes so long.
------
Filipe, it is possible it is taking so long to do a "sort" because when
doing it, it caches it on the client side of Distiller also + does it on the
Samba Server to. IE; Sorts on Both Sides.

I have had this happen in .Net. When doing a sort in .Net the default is to
sort on the client and the server.

JohnStanley

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-25-2009, 03:29 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server

> Filipe, it is possible it is taking so long to do a "sort" because when
> doing it, it caches it on the client side of Distiller also + does it on the
> Samba Server to. IE; Sorts on Both Sides.
>
>
>
I tried it, several times, on a standalone Windows workstation and the
same happens.
I am not saying that the sorting takes too much time; the whole process
takes too much time.

And please note that it also happens with the runfilex.ps code provided
by Adobe, which does not sort but instead presents Distiller with a list
of files to process, instead of letting it rely on the dir order.
Sorting is not the problem here.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-25-2009, 04:36 PM
Les Mikesell
 
Default OT? File order on CentOS/Samba server

Miguel Medalha wrote:
>> Filipe, it is possible it is taking so long to do a "sort" because when
>> doing it, it caches it on the client side of Distiller also + does it on the
>> Samba Server to. IE; Sorts on Both Sides.
>>
>>
>>
> I tried it, several times, on a standalone Windows workstation and the
> same happens.
> I am not saying that the sorting takes too much time; the whole process
> takes too much time.
>
> And please note that it also happens with the runfilex.ps code provided
> by Adobe, which does not sort but instead presents Distiller with a list
> of files to process, instead of letting it rely on the dir order.
> Sorting is not the problem here.

Sounds like a bug in the program. Maybe it runs a separate instance for
each page in that mode and doesn't release any memory until it is all
finished. On something smaller or less complex it might not make much
difference, but if the memory use pushes into swap it will take much longer.

By the way, yet another really-contorted workaround would be to run
VMware server or virtualbox (both free) on the centos box with a windows
guest to get a reliable NTFS network drive. If you have resources to
spare on this server you could even run distiller there so you could
shut down the workstations as soon as the final run starts. It isn't
the most efficient way to do things, but I've had some running that way
for years with no unexpected problems. The only inconvenience is that
at least in the VMware server case on centos 5, whenever you update the
kernel and reboot, you have to run a script that recompiles the vmware
module before the guest will run.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-25-2009, 05:09 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server

>> Sounds like a bug in the program. Maybe it runs a separate instance for
>> each page in that mode and doesn't release any memory until it is all
>> finished. On something smaller or less complex it might not make much
>> difference, but if the memory use pushes into swap it will take much longer.
>>
>>
Yes, that's what it seems to me. As I said before, it starts processing
swiftly but soon each new page takes longer and longer until it crawls.
CPU time reaches 98% and the memory footprint keeps increasing untill
the end of the process. This happens even on a standalone Windows
workstation, not only over the network. I can report this to Adobe but I
don't have too much hope about the attention such a large company is
going to give to such an issue...

>> By the way, yet another really-contorted workaround would be to run
>> VMware server or virtualbox (both free) on the centos box with a windows
>> guest to get a reliable NTFS network drive. If you have resources to
>> spare on this server you could even run distiller there so you could
>> shut down the workstations as soon as the final run starts.

I thought of doing that but it really is not realistic at the moment in
my environment. It is overkill. It would be much easier to put a small
FAT32-formated partition on the server just for that purpose. The PS
files are not kept. After processing they are discarded, only the
resulting PDF is used and archived. For now I will stick with a EXT3
partition with dir_index off and use rundirex like we always did. It
works well this way: 3 to 4 minutes to render a complete publication.

Thank you for your tips. Even if I don't use them now, the information
stays. Maybe it will be needed one of these days.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 10:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org