FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 01-23-2009, 06:43 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

> http://code.google.com/p/samba-dirsort-vfs/
> Did you try that? I think someone recommended it to you.
Well, I did try to compile it but make fails on all the Linux computers
I have access to. They all run CentOS 5.2. It would be nice to have a
.rpm... I am a sysadmin, not a programmer, I am not able to solve most
compile errors.
> (...) think your real problem lies in your processing software in
> the file ordering. I would have a really good look at the software doing it.
>

The problem lies in EXT3. I discovered that if I mv the files to another
directory the files will then appear on the samba shares in
alphanumerical order and will be processed by Acrobat Distiller
accordingly. The move can even be done by Windows Explorer working on
the Samba share.

This seems a bit strange to me. Why doesn't EXT3 present the files in
alphanumerical order after they are first created one by one but then
presents them alphanumerically after a bulk move to another directory?

Also, I connected a FAT32 formated USB flash drive to the server and
directed Distiller to there. The files are correctly processed at the
first trial. I suppose I will install a smallish FAT32 formated IDE disk
on the server just for this purpose.

Thank you to all who answered my questions. We form a great community
indeed!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 06:48 PM
Steve Thompson
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

On Fri, 23 Jan 2009, Miguel Medalha wrote:

> This seems a bit strange to me. Why doesn't EXT3 present the files in
> alphanumerical order after they are first created one by one but then
> presents them alphanumerically after a bulk move to another directory?

This sounds to me like the dir_index option was applied to a file system
that didn't originally have it and an fsck -Df wasn't run at the time.

Steve

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 06:55 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

> This sounds to me like the dir_index option was applied to a file system
> that didn't originally have it and an fsck -Df wasn't run at the time.

That may well be the most relevant information given here! I will
*certainly* give it a try.

Thank you!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 07:22 PM
Les Mikesell
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

Miguel Medalha wrote:
>
>> (...) think your real problem lies in your processing software in
>> the file ordering. I would have a really good look at the software doing it.
>>
>
> The problem lies in EXT3. I discovered that if I mv the files to another
> directory the files will then appear on the samba shares in
> alphanumerical order and will be processed by Acrobat Distiller
> accordingly. The move can even be done by Windows Explorer working on
> the Samba share.
>
> This seems a bit strange to me. Why doesn't EXT3 present the files in
> alphanumerical order after they are first created one by one but then
> presents them alphanumerically after a bulk move to another directory?

Directories grow as they are filled the first time. If you use a shell
script with a wildcard to do the move, the shell will sort the list on
the command line as it expands it, so the names are linked into the new
directory in sorted order. However if you repeat this in the same
directory instead of creating new ones each time it may not continue to
work as existing empty slots may be reused in a different order.

> Also, I connected a FAT32 formated USB flash drive to the server and
> directed Distiller to there. The files are correctly processed at the
> first trial. I suppose I will install a smallish FAT32 formated IDE disk
> on the server just for this purpose.

Did you consider sharing a directory from the machine running distiller
and cifs-mounting it on the linux side to get ntfs behavior? Also, I'm
curious about the timing of the runs. It doesn't sound like the file
operations are grouped atomically. How do you ensure that the whole set
is present when distiller starts, or that only one set is present? If I
were doing it, I'd probably create a new tmp directory for each set of
files (which should fix the ordering as a side effect) and rename it to
the expected name after all files are present so you see all of them or
none. Or, I might put cygwin sshd on the windows box and use scp or
rsync to copy the files over in a batch, then start the Distiller run
(if you can start it from the command line).

--
Les Mikesell
lesmikesell@gmail.com

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 07:57 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

> Did you consider sharing a directory from the machine running distiller
> and cifs-mounting it on the linux side to get ntfs behavior?
That is out of question. The Windows machines are graphic workstations
which are not all connected all the time and the Distiller service is
essential to the network.

> Also, I'm curious about the timing of the runs. It doesn't sound like the file
> operations are grouped atomically. How do you ensure that the whole set
> is present when distiller starts, or that only one set is present?
This is a very peculiar implementation. As I said om my first post, we
are a newspaper and, as all newspapers, we don't have a fixed time to
close the edition. It closes when it is ready, that's all.

The PDFs for print are automatically produced one by one from PostScript
files. The PS files fall on a folder watched by Acrobat Distiller and
after being stable for more than 10 seconds the conversion begins. Each
one contains only one page, which will then be joined to others to form
a plan for a platesetter.

When all the pages have been produced, one of the graphics people places
a special text file on a folder watched by Distiller and it begins to
bulk process all the individual PS files: downsampling images,
converting the color space to sRGB, consolidating font subsets, creating
bookmarks and indexes, etc. The result is a multipage PDF for electronic
distribution, containing the whole newspaper in the sRGB color space.

This always worked flawlessly until some days ago I replaced the win2k
server with a new CentOS/Samba one. Everything worked better and faster
except... the pages on this last PDF were in what seemed like an
aleatory order. Ordering them by hand is a time consuming and error
prone process, specially when everybody is now tired... Producing a
newspaper is a pretty tense work, you know.

The difficulty with the scripted solutions proposed here is that we
cannot know in advance at what time this process will take place and
what the number of pages involved will be. At the end of each issue
every minute counts. A watching process would have to poll the status of
the workflow for several hours with very small intervals, which would be
a waste of processor cicles. And not a very elegant thing to do, I feel.


I am (for now...) convinced that the tip given to me here about
dir_index and the use of fsck -fD will solve this problem.
Monday I will know. It will be a loooong wait for me.

Thank you again.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 09:30 PM
Les Mikesell
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

Miguel Medalha wrote:
>> Did you consider sharing a directory from the machine running distiller
>> and cifs-mounting it on the linux side to get ntfs behavior?
> That is out of question. The Windows machines are graphic workstations
> which are not all connected all the time and the Distiller service is
> essential to the network.

I was under the impression that the Distiller app was running under
Windows. If it isn't, it doesn't make much sense for it to expect NTFS
filesystem semantics.

> When all the pages have been produced, one of the graphics people places
> a special text file on a folder watched by Distiller and it begins to
> bulk process all the individual PS files:

[...]

> The difficulty with the scripted solutions proposed here is that we
> cannot know in advance at what time this process will take place and
> what the number of pages involved will be.

Can't the trigger operation of placing the special text file be replaced
by that person starting the script instead (perhaps click a button on a
web page or something similar)?

> At the end of each issue
> every minute counts. A watching process would have to poll the status of
> the workflow for several hours with very small intervals, which would be
> a waste of processor cicles. And not a very elegant thing to do, I feel.

While I wouldn't call it elegant, filesystem caching makes such things
efficient enough that you'll never notice them running. If you need a
script that looks for a file to appear or expands a wildcard in a
directory, go ahead and use one as long as you can sleep for at least a
few seconds in the loop. It's cheaper than having a person rearrange
something.

--
Les Mikesell
lesmikesell@gmail.com




>
>
> I am (for now...) convinced that the tip given to me here about
> dir_index and the use of fsck -fD will solve this problem.
> Monday I will know. It will be a loooong wait for me.
>
> Thank you again.
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 10:40 PM
Steve Thompson
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

On Fri, 23 Jan 2009, Miguel Medalha wrote:

>
>> This sounds to me like the dir_index option was applied to a file system
>> that didn't originally have it and an fsck -Df wasn't run at the time.
>
> That may well be the most relevant information given here! I will *certainly*
> give it a try.

I based my speculation on some observations I had made on some of my own
systems when I implemented dir_index. It so happens that, on that system
at least, a "find /foo -print" returns the filenames in sorted order.
Unfortunately, it isn't true on another system that I just checked. So now
I will go and stand in the corner

Steve
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 10:51 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

> I was under the impression that the Distiller app was running under
> Windows. If it isn't, it doesn't make much sense for it to expect NTFS
> filesystem semantics.
>
>

Yes, Distiller is running under Windows. When pages start to get ready,
one of the graphic operators opens Distiller on his/her workstation
which then starts watching a folder *on the server*.

> Can't the trigger operation of placing the special text file be replaced
> by that person starting the script instead (perhaps click a button on a
> web page or something similar)?
>
>
Yes, that would be a possibility. But those people have strong rooted
habits and they are not in the least technically minded. As such, I
would prefer to keep a workflow that has been functioning very well.

(By the way, that "special text file" is a snippet of PostScript code
that instructs Distiller on where to find the files and how to process
them. It would be needed anyway.)

Perhaps this obstacle will be removed by applying the correct parameters
to the EXT3 file system, as suggested by William Maltby and Steve
Thompson above in this thread: mount option "dir_index" followed by a
"fsck -Df". I will try this Monday.

Thank you for answering.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-23-2009, 10:56 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

>
> I based my speculation on some observations I had made on some of my
> own systems when I implemented dir_index. It so happens that, on that
> system at least, a "find /foo -print" returns the filenames in sorted
> order. Unfortunately, it isn't true on another system that I just
> checked. So now I will go and stand in the corner
>



Anyway, your tip gave me some precious direction. Monday I will
investigate and then report.
Thank you!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 01-24-2009, 02:25 PM
Miguel Medalha
 
Default OT? File order on CentOS/Samba server -- SOLVED (kind of...)

> Hi,
>
> You might want to try to look into the Distiller side of things.
>

That's what I always did. I am a DTP guy.
> 1) I believe you are using Rundirex.txt file to convert all the .ps's
> into one .pdf. This page from Adobe confirms that it will take the
> files in directory order under Windows:
>
> http://kb.adobe.com/selfservice/viewContent.do?externalId=318674
> "-- Acrobat Distiller for Windows will process the files in the order
> in which you put them into the folder and create the PDF pages in the
> order in which it processes the files."
> "-- Acrobat Distiller for Mac OS will process the files in alphabetical order."
> (one solution would be getting a mac, hehehe).
>
> Strange that you never hit the wrong order problem before, since
> according to that page, you should...
>
>
Regardless of what that paper says, Distiller has ALWAYS processed the
files in alphabetical order under Windows. I have been doing so since
2000 and Acrobat Distiller 4. We are now at 9. I refer, of course, to
the use of rundirx.

> 2) That page also talks about Runfilex.ps file, which is basically the
> same, only you have to list each .ps file in the order you want them
> to be included.
>

I already addressed that on my first post. I tried runfilex.ps but then
Distiller takes 30 to 40 minutes to do the same job that it now does in
3 to 4 minutes, which really is not an option for a newspaper at closing
time.

I will do some more experiences, from the Distiller side and the Linux
side, and I will report here.

Thank you for your answers.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 03:57 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org