FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 09-01-2011, 08:41 PM
Austin Godber
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Hello,
Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
I am getting very sporadic throughput with moderately sized files (0.5-2GB) on ext3. *I have tried most of the mount time tuning options:


* noatime* trying different journal types* setting commit=120 - helped a little
Even after these*optimizations*it doesn't seem like the raid array is working as it should. *After a few 1GB writes (dd from /dev/zero to the raid) kjournald runs for an hour and later writes are really slow. *Not only that, using tw_cli is very slow ... whereas tw_cli is superfast if kjournald is not churning away.


So it goes something like this
* write one 1GB file (486 MB/s)* writes another (223 MB/s)* any writes before the 120s commit kicks in, is ~200MB/s

* commit kicks in and kjournald starts churning* writes are all over the map - 6-85MB/s
Maybe this is just the way it is, but it did not seem to be the case this same hardware was running Fedora (Core) 9 and I have a similar machine where this does not seem to be the case (I can't experiment on it at the moment).


The one thing I do not know, since I did not create the RAID or ext3 filesystem is whether the stride and stripe-width were properly selected to match the 64k chunk size of the raid array. *I don't know how to tell from tune2fs ... output below ... its either not there or by another name.


Any help or suggestions are appreciated.
Austin
Useful Information==============
============== /proc/mounts =====================


/dev/sda1 /tonga_raid ext3 rw,noatime,errors=continue,nouser_xattr,noacl,comm it=120,barrier=1,data="" 0 0

========= *tw_cli /c2 show diag =====================


### Time Stamp: * * * *12:34:18 01-Sep-2011### Host Name: * * * * tonga### Host Architecture: x86_64 (64 bit)### OS Version: * * * *Linux 2.6.32-71.29.1.el6.x86_64

### Model: * * * * * * 9650SE-8LPML### Serial #: * * * * *L326025A8221043### Controller ID: * * 2### CLI Version: * * * 2.00.11.016### API Version: * * * 2.08.00.017

### Driver Version: * *2.26.02.014RH### Firmware Version: *FE9X 3.08.00.016### BIOS Version: * * *BE9X 3.08.00.004### Available Memory: *224MB
================================================== ========================

Diagnostic Information on Controller //.../c2/...--------------------------------------------------------------------------Event Trigger and Log Information:Triggered Event(s) =*

* * ctlreset (controller soft reset)** * fwassert (firmware assert)** * driveerr (drive error)*Diagnostic log save mode = -Parameter table does not exist*



========== tune2fs -l /dev/sda1 =================tune2fs 1.41.12 (17-May-2010)Filesystem volume name: * /dataLast mounted on: * * * * *<not available>

Filesystem UUID: * * * * *3c5f6dbb-d5dc-4f85-bc70-9b761c89c86eFilesystem magic number: *0xEF53Filesystem revision #: * *1 (dynamic)Filesystem features: * * *has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file

Filesystem flags: * * * * signed_directory_hash*Default mount options: * *user_xattr aclFilesystem state: * * * * cleanErrors behavior: * * * * *ContinueFilesystem OS type: * * * Linux

Inode count: * * * * * * *427245568Block count: * * * * * * *1708965879Reserved block count: * * 85448293Free blocks: * * * * * * *809305423Free inodes: * * * * * * *426287552

First block: * * * * * * *0Block size: * * * * * * * 4096Fragment size: * * * * * *4096Reserved GDT blocks: * * *616Blocks per group: * * * * 32768Fragments per group: * * *32768

Inodes per group: * * * * 8192Inode blocks per group: * 512Filesystem created: * * * Tue Sep *9 09:57:44 2008Last mount time: * * * * *Thu Sep *1 12:40:01 2011Last write time: * * * * *Thu Sep *1 12:40:01 2011

Mount count: * * * * * * *11Maximum mount count: * * *-1Last checked: * * * * * * Tue Aug 30 23:12:49 2011Check interval: * * * * * 0 (<none>)Reserved blocks uid: * * *0 (user root)

Reserved blocks gid: * * *0 (group root)First inode: * * * * * * *11Inode size: * * * * *256Journal inode: * * * * * *8

Default directory hash: * teaDirectory Hash Seed: * * *0b324311-93a9-4c23-bf15-40965792029bJournal backup: * * * * * inode blocks============================================ ===============



=================== *tw_cli info c2 *============================
Unit *UnitType *Status * * * * %RCmpl *%V/I/M *Stripe *Size(GB) *Cache *AVrfy------------------------------------------------------------------------------

u0 * *RAID-5 * *OK * * * * * * - * * * - * * * 64K * * 6519.19 * OFF * *OFF * *
Port * Status * * * * * Unit * Size * * * *Blocks * * * *Serial---------------------------------------------------------------

p0 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1631953 * **p1 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1622428 * **p2 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1639721 * **

p3 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1636054 * **p4 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1621694 * **p5 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1636292 * **

p6 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1637586 * **p7 * * OK * * * * * * * u0 * * 931.51 GB * 1953525168 * *WD-WCASJ1637516*
================================================== =========



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 09:05 PM
Marcelo Beckmann
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Em 01-09-2011 17:41, Austin Godber escreveu:
>
> =================== tw_cli info c2 ============================
>
> Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache
> AVrfy
> ------------------------------------------------------------------------------
> u0 RAID-5 OK - - 64K 6519.19 OFF
> OFF

I have the same controller on Centos 5.

Did you try to active Cache on 3ware?

[17:56:04 root@backup ~]# lspci | grep 3ware
01:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
[17:56:11 root@backup ~]# tw_cli /c4 show

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache
AVrfy
------------------------------------------------------------------------------
u0 RAID-5 OK - - 64K 5587.9 RiW
ON


In the past I had a scenario where I started with Cache OFF and had a
poor performance, specially for write. After set Cache RiW I got better
performance.

I didn't test that controller on CentOS 6, but is good to know if there
is some problem, because my company sells equipment with that controller.


Best regards,

--
Marcelo Beckmann
Suporte Corporativo - suporte@webers.com.br
Webers Tecnologia - http://www.webers.com.br
Curitiba (PR) (41) 3094-6600
Rio de Janeiro (RJ) (21) 4007-1207
São Paulo (SP) (11) 4007-1207

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 09:54 PM
Austin Godber
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Hello Marcelo,
Thank you for the suggestion. *I had not yet tried activating the cache since I was unsure whether that was a good idea or not.
Since you have experience with this card, do you have any recommendations for what I should expect or avoid? *Have you used EXT3 with success or are you using XFS or something else?


Austin

On Thu, Sep 1, 2011 at 2:05 PM, Marcelo Beckmann <marcelo.beckmann@webers.com.br> wrote:


Em 01-09-2011 17:41, Austin Godber escreveu:

>

> =================== *tw_cli info c2 *============================

>

> Unit *UnitType *Status * * * * %RCmpl *%V/I/M *Stripe *Size(GB) *Cache

> *AVrfy

> ------------------------------------------------------------------------------

> u0 * *RAID-5 * *OK * * * * * * - * * * - * * * 64K * * 6519.19 * OFF

> *OFF



I have the same controller on Centos 5.



Did you try to active Cache on 3ware?



[17:56:04 root@backup ~]# lspci | grep 3ware

01:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01)

[17:56:11 root@backup ~]# tw_cli /c4 show



Unit *UnitType *Status * * * * %RCmpl *%V/I/M *Stripe *Size(GB) *Cache

AVrfy

------------------------------------------------------------------------------

u0 * *RAID-5 * *OK * * * * * * - * * * - * * * 64K * * 5587.9 * *RiW

ON





In the past I had a scenario where I started with Cache OFF and had a

poor performance, specially for write. After set Cache RiW I got better

performance.



I didn't test that controller on CentOS 6, but is good to know if there

is some problem, because my company sells equipment with that controller.





Best regards,



--

Marcelo Beckmann

Suporte Corporativo - suporte@webers.com.br

Webers Tecnologia - http://www.webers.com.br

Curitiba * * * (PR) (41) 3094-6600

Rio de Janeiro (RJ) (21) 4007-1207

São Paulo * * *(SP) (11) 4007-1207



_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 10:58 PM
Craig White
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:

> Hello,
>
> Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
----
use RAID 10

Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.

Craig
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 11:17 PM
Austin Godber
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Hi Craig,
Thanks for the suggestion. *I would if I could. *I'd also probably try another file system. *Though the good news is, enabling the write cache on that array has improved things significantly. *Which, in my case, was:


tw_cli*/c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. *For now I am happy with the situation, but I'd be interested to hear the experiences of others.


Austin

On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:




On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:



> Hello,

>

> Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?

----

use RAID 10



Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.



Craig

_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 11:47 PM
Craig White
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

I'm surprised that you can actually turn it on without a battery. I suspect that this is not a write-through/write-back cache but be forewarned that if there's no battery, it's possible that things you thought were written to the hard drive on shutdown/restart/hang/crash might not ever be written to the hard drive(s)

Craig

On Sep 1, 2011, at 4:17 PM, Austin Godber wrote:

> Hi Craig,
>
> Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache on that array has improved things significantly. Which, in my case, was:
>
> tw_cli /c2/u0 set cache=on
>
> Now, if only I had the battery backup unit for the card.
>
> Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
>
> Austin
>
> On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:
>
> On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
>
> > Hello,
> >
> > Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
> ----
> use RAID 10
>
> Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-01-2011, 11:49 PM
Tom Bishop
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Keep in mind you really only want to enable the cache if you have a
bbc, otherwise you are risking your data since it can/will cache
writes...just something to keep in mind.

On 9/1/11, Austin Godber <godber@gmail.com> wrote:
> Hi Craig,
>
> Thanks for the suggestion. I would if I could. I'd also probably try
> another file system. Though the good news is, enabling the write cache on
> that array has improved things significantly. Which, in my case, was:
>
> tw_cli /c2/u0 set cache=on
>
> Now, if only I had the battery backup unit for the card.
>
> Thanks, everyone for their suggestions. For now I am happy with the
> situation, but I'd be interested to hear the experiences of others.
>
> Austin
>
> On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:
>
>>
>> On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
>>
>> > Hello,
>> >
>> > Does anyone have experience using a 3ware 9650SE series raid controller
>> on CentOS 6.0?
>> ----
>> use RAID 10
>>
>> Unless something has changed, RAID 5 is notoriously slow on the 3Ware
>> controllers. Whatever you do will only incrementally speed things up. If
>> performance is desired, RAID 5 is not the way to go.
>>
>> Craig
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-02-2011, 12:43 AM
Austin Godber
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

At this point the card is pretty much useless without that cache enabled. *Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. *If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.


I will, for the sake of not being an idiot, look into buying the BBUs.
Austin

On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop <bishoptf@gmail.com> wrote:


Keep in mind you really only want to enable the cache if you have a

bbc, otherwise you are risking your data since it can/will cache

writes...just something to keep in mind.



On 9/1/11, Austin Godber <godber@gmail.com> wrote:

> Hi Craig,

>

> Thanks for the suggestion. *I would if I could. *I'd also probably try

> another file system. *Though the good news is, enabling the write cache on

> that array has improved things significantly. *Which, in my case, was:

>

> tw_cli /c2/u0 set cache=on

>

> Now, if only I had the battery backup unit for the card.

>

> Thanks, everyone for their suggestions. *For now I am happy with the

> situation, but I'd be interested to hear the experiences of others.

>

> Austin

>

> On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:

>

>>

>> On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:

>>

>> > Hello,

>> >

>> > Does anyone have experience using a 3ware 9650SE series raid controller

>> on CentOS 6.0?

>> ----

>> use RAID 10

>>

>> Unless something has changed, RAID 5 is notoriously slow on the 3Ware

>> controllers. Whatever you do will only incrementally speed things up. If

>> performance is desired, RAID 5 is not the way to go.

>>

>> Craig

>> _______________________________________________

>> CentOS mailing list

>> CentOS@centos.org

>> http://lists.centos.org/mailman/listinfo/centos

>>

>

_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-02-2011, 05:51 PM
Craig White
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Data loss could conceivably occur on shutdown or restart too - just saying... You are assuming that the data that doesn't get written to disk is going to be non-essential... I wish you good luck with that. I think if one doesn't want to be an idiot, one would not enable a cache that has no means to ensure that the cache is written to disk.

I think your take away from all of this is somewhat misdirected. Not having a BBU simply means that your writes really should always be synchronous/immediate. That shouldn't really be a problem and shouldn't impose a large performance penalty.

Your performance issue relates more to the fact that RAID 5 implementation on the 3Ware cards is rather poor and modes such as RAID 10 (RAID 0 + 1) will give you much more speed that you realize. If you also consider on the surprisingly higher rates of failure with loss of data possibility when reconstructing a missing/dead drive on a RAID 5 setup you really should be re-examining your storage strategy.

Craig

On Sep 1, 2011, at 5:43 PM, Austin Godber wrote:

> At this point the card is pretty much useless without that cache enabled. Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.
>
> I will, for the sake of not being an idiot, look into buying the BBUs.
>
> Austin
>
> On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop <bishoptf@gmail.com> wrote:
> Keep in mind you really only want to enable the cache if you have a
> bbc, otherwise you are risking your data since it can/will cache
> writes...just something to keep in mind.
>
> On 9/1/11, Austin Godber <godber@gmail.com> wrote:
> > Hi Craig,
> >
> > Thanks for the suggestion. I would if I could. I'd also probably try
> > another file system. Though the good news is, enabling the write cache on
> > that array has improved things significantly. Which, in my case, was:
> >
> > tw_cli /c2/u0 set cache=on
> >
> > Now, if only I had the battery backup unit for the card.
> >
> > Thanks, everyone for their suggestions. For now I am happy with the
> > situation, but I'd be interested to hear the experiences of others.
> >
> > Austin
> >
> > On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:
> >
> >>
> >> On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
> >>
> >> > Hello,
> >> >
> >> > Does anyone have experience using a 3ware 9650SE series raid controller
> >> on CentOS 6.0?
> >> ----
> >> use RAID 10
> >>
> >> Unless something has changed, RAID 5 is notoriously slow on the 3Ware
> >> controllers. Whatever you do will only incrementally speed things up. If
> >> performance is desired, RAID 5 is not the way to go.
> >>
> >> Craig
> >> _______________________________________________
> >> CentOS mailing list
> >> CentOS@centos.org
> >> http://lists.centos.org/mailman/listinfo/centos
> >>
> >
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos

--
Craig White ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ craig.white@ttiltd.com
1.800.869.6908 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ www.ttiassessments.com

Need help communicating between generations at work to achieve your desired success? Let us help!

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-02-2011, 07:03 PM
Austin Godber
 
Default CentOS 6.0 and 3ware 9650SE series RAID Performance

Thank you for the clarification Craig, I am re-examining my storage strategy, thus my email. *Sadly this machine has been in service a number of years and already contains more data than is possible in a RAID 10 configuration. *If I had the spare space and resources I'd be thrilled to switch to RAID 10. *That is not the case, however. *Given that, I am faced with the choice between having the machine entirely unusable between now and when I can acquire a BBU or using it with the risk of data loss/corruption, I will chose the latter. *Neither data loss nor corruption would go unnoticed in the period that this will be necessary.


Does that sound*entirely*unreasonable?
Austin



On Fri, Sep 2, 2011 at 10:51 AM, Craig White <craig.white@ttiltd.com> wrote:


Data loss could conceivably occur on shutdown or restart too - just saying... You are assuming that the data that doesn't get written to disk is going to be non-essential... I wish you good luck with that. I think if one doesn't want to be an idiot, one would not enable a cache that has no means to ensure that the cache is written to disk.





I think your take away from all of this is somewhat misdirected. Not having a BBU simply means that your writes really should always be synchronous/immediate. That shouldn't really be a problem and shouldn't impose a large performance penalty.





Your performance issue relates more to the fact that RAID 5 implementation on the 3Ware cards is rather poor and modes such as RAID 10 (RAID 0 + 1) will give you much more speed that you realize. If you also consider on the surprisingly higher rates of failure with loss of data possibility when reconstructing a missing/dead drive on a RAID 5 setup you really should be re-examining your storage strategy.





Craig



On Sep 1, 2011, at 5:43 PM, Austin Godber wrote:



> At this point the card is pretty much useless without that cache enabled. *Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. *If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.



>

> I will, for the sake of not being an idiot, look into buying the BBUs.

>

> Austin

>

> On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop <bishoptf@gmail.com> wrote:

> Keep in mind you really only want to enable the cache if you have a

> bbc, otherwise you are risking your data since it can/will cache

> writes...just something to keep in mind.

>

> On 9/1/11, Austin Godber <godber@gmail.com> wrote:

> > Hi Craig,

> >

> > Thanks for the suggestion. *I would if I could. *I'd also probably try

> > another file system. *Though the good news is, enabling the write cache on

> > that array has improved things significantly. *Which, in my case, was:

> >

> > tw_cli /c2/u0 set cache=on

> >

> > Now, if only I had the battery backup unit for the card.

> >

> > Thanks, everyone for their suggestions. *For now I am happy with the

> > situation, but I'd be interested to hear the experiences of others.

> >

> > Austin

> >

> > On Thu, Sep 1, 2011 at 3:58 PM, Craig White <craig.white@ttiltd.com> wrote:

> >

> >>

> >> On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:

> >>

> >> > Hello,

> >> >

> >> > Does anyone have experience using a 3ware 9650SE series raid controller

> >> on CentOS 6.0?

> >> ----

> >> use RAID 10

> >>

> >> Unless something has changed, RAID 5 is notoriously slow on the 3Ware

> >> controllers. Whatever you do will only incrementally speed things up. If

> >> performance is desired, RAID 5 is not the way to go.

> >>

> >> Craig

> >> _______________________________________________

> >> CentOS mailing list

> >> CentOS@centos.org

> >> http://lists.centos.org/mailman/listinfo/centos

> >>

> >

> _______________________________________________

> CentOS mailing list

> CentOS@centos.org

> http://lists.centos.org/mailman/listinfo/centos

>

> _______________________________________________

> CentOS mailing list

> CentOS@centos.org

> http://lists.centos.org/mailman/listinfo/centos



--

Craig White ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ craig.white@ttiltd.com

1.800.869.6908*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ www.ttiassessments.com



Need help communicating between generations at work to achieve your desired success? Let us help!



_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 08:16 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org