[oclug] Kernel locks, delays on querying /proc/pid/stat and qlogic driver

[Bart Trojanowski]

* Christo Kutrovsky <kutrovsky+oclug at pythian.com> [040417 00:56]:
> We have a system with 8 CPUs (XEON with HT=on so 16 visible) and 18gb
> of RAM...

Allow me to recover my jaw from the floor.

> ... The system is running kernel 2.4.9-e.37summit and it's primarily
> purpose is to run an Oracle database. 

Do you know if Oracle will run on 2.6 based kernel?  I know that a
higher "responsiveness feeling" can be obtained from an upgrade to 2.6.
Not sure if it will help in this instance.

It may be worth a try.

> It is a datawarehouse style system, so lots of IO going to a SAN with
> 700Gb+ of storage used.

What kind of SAN connection?  SCSI, SCSI over IP, NFS, something else?

> The code was rather simple and I took a guess and assumed that
> "read_lock" is where the most time is spend, since that's the only
> /proc/pid/stat extra function call "stat" has over all other files. 

It could also be stolen from you in the context-switch between
user-space and kernel and back out.  IIRC, the kernel will attempt to
run outstanding post-interrupt (bottom halves & tasklets) processing at
this time.  If you have poorly behaving drivers the processing of these
outstanding events could be done with a spin-lock and prevent other
things from happening... although this alone would not explain the other
7 CPUs not running your 'ls'.

> So I concluded that the qlogic driver (i2x00-v6.06.60-fo) must be
> causing the problems as this is the only thing that is different on
> the system.

This is a good conclusion.

> Any hints are welcome. Including pure theories.

Cannot offer much other then suggesting you use a newer kernel (even a
newer 2.4 kernel if you are not willing to try a 2.6), and talk to
RedHat.  You are paying for support with the advanced server, no?

-Bart

-- 
				WebSig: http://www.jukie.net/~bart/sig/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://tux.oclug.on.ca/pipermail/oclug/attachments/20040417/fd842a60/attachment.bin