Last days we had some crash on redhat system related to high io, I dont
see a evident cause so perhaps someone on the list can give some pointers
where to look for.
We have several sybase database running, and database backup process
Seems there was a heavy io operations and sybase crashed
We have atop running, where we can see sybase using ram, and dying, also
on sar we see heavy io.
No swap has used I search the logs for oom kill's but didn't found any
The machine was very unresponsive when the issue occur, we have redhat
cluster running and cluster token got lost from this node during this
time.
I'd like to know if something killed the app, and how to fine tune
memory/disk access to prevent this
output from atop and sar follow,
thanks in advanced
28Ag
12:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
04:10:01 AM 1 656 6.36 7.08 4.59
04:20:01 AM 3 656 6.59 6.47 5.52
04:30:01 AM 3 658 6.03 6.37 5.96
04:40:01 AM 3 658 8.30 8.03 6.96
04:50:01 AM 3 665 9.74 8.11 7.42
05:00:01 AM 3 664 4.44 6.53 7.28
05:10:02 AM 2 664 6.45 6.85 6.99
12:00:01 AM CPU %user %nice %system %iowait %steal
%idle
03:50:01 AM all 2.29 0.00 2.73 0.88 0.00
94.10
04:00:01 AM all 2.31 0.00 2.77 0.93 0.00
93.99
04:10:01 AM all 5.46 0.00 5.57 10.47 0.00
78.49
04:20:01 AM all 7.18 0.00 6.33 6.76 0.00
79.73
04:30:01 AM all 5.88 0.00 5.90 6.87 0.00
81.35
04:40:01 AM all 5.16 0.00 5.77 8.09 0.00
80.98
04:50:01 AM all 4.88 0.00 5.35 7.82 0.00
81.95
05:00:01 AM all 4.59 0.00 5.29 7.95 0.00
82.17
05:10:02 AM all 4.99 0.00 5.64 7.09 0.00
82.28
05:20:01 AM all 4.36 0.00 4.50 6.52 0.00
84.62
05:30:01 AM all 0.01 0.00 0.02 0.13 0.00
99.84
tps Total number of transfers per second that were issued to
physical devices. A transfer is an I/O request to a physical device.
rtps Total number of read requests per second issued to physical
devices.
wtps Total number of write requests per second issued to physical
devices.
bread/s Total amount of data read from the devices in blocks per
second. 512 bytes.
bwrtn/s Total amount of data written to devices in blocks per second.
12:00:01 AM tps rtps wtps bread/s bwrtn/s
1:10:01 AM 7996.72 321.85 7674.87 9120.26 187734.23
01:20:01 AM 7260.18 239.13 7021.05 7151.23 129573.36
01:30:01 AM 10541.89 333.87 10208.02 3724.24 260710.61
01:40:01 AM 8298.28 59.59 8238.69 949.20 200802.65
01:50:01 AM 12443.27 30.84 12412.43 141.97 310002.60
02:00:01 AM 10878.28 134.28 10744.00 3120.10 252048.75
02:10:01 AM 7191.81 245.43 6946.39 4632.28 155588.73
03:40:01 AM 9971.54 2.17 9969.37 9.80 194855.04
03:50:01 AM 9615.50 3.80 9611.70 16.28 195641.46
04:00:01 AM 10615.95 2.11 10613.84 9.04 201471.62
04:10:01 AM 10579.62 3619.13 6960.49 130869.08 143336.99
04:20:01 AM 10598.33 2381.38 8216.94 98763.15 176191.20
04:30:01 AM 11677.21 3535.63 8141.58 233305.68 174777.83
04:40:01 AM 10890.26 3150.32 7739.95 112731.87 170365.89
04:50:01 AM 12778.49 4054.31 8724.18 137983.84 184082.99
05:00:01 AM 13507.21 3830.58 9676.63 104143.93 203367.17
05:10:02 AM 12670.16 2266.25 10403.91 103696.19 210170.15
05:20:01 AM 5380199.36 6489408.45 6061020.12 6428917.69 3338784.18
05:30:01 AM 33.70 0.11 33.59 1.55 373.81
High IO issue
