ASM, block devices and SCSI emulation.

Quite recently, I learned that Oracle ASM is now supporting block
devices. The whole process is described here:

http://docs.oracle.com/cd/E11882_01/install.112/e17212/storage.htm#CDEBFDEH

This deserves few additional observations. The main engine that enables
one to add block devices to SCSI is udev. Essentially, one creates SCSI
devices by using a configuration file like this one, pasted directly
from the manual above:

# vi /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL==”sd?1″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”,
RESULT==”14f70656e66696c00000000″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd?2″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”,
RESULT==”14f70656e66696c00000001″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd?3″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”,
RESULT==”14f70656e66696c00000002″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

What does that do? This enables udev to create “SCSI devices” for each
and every SATA or FC disk on the system. You sse, when Linux was much,
much younger, there was a problem with now largely forgotten devices
called “CD burners”. Those vere EIDE devices and each and every one had
to have its own driver. Commercial packages like Roxio were used to
support as many of those devices as possible and, needless to say, Linux
was lagging behind. Then the Linux guys got an excellent idea: why not
emulate SCSI on all EIDE devices and write one program to burn them all,
using just SCSI command set. The generic SCSI device emulator, embodied
in the “sg” module was born and it still lives on today. That is what
the configuration file above is utilizing to create SCSI disks from the
standard SATA devices. The data is sent to the device using IOCTL
defined in /usr/include/scsi/scsi_ioctl. That is the same interface once
used by the program X-CD-Roast, the first popular Linux CD burning
program.

There is, however, a little thing that one must be aware about SCSI
emulation. It is an additional driver and driver requests are usually
spawned by the processor interrupts. Device driver, itself activated by
an interrupt, spawns another interrupt, frequently called “soft
interrupt” because it’s not spawned by a hardware, with the arguments
which direct kernel to select the apropriate drives, in this case “sg”.
That is happening for every request to device. Using “sg” doubles the
number of disk interrupts caused by the particular device it is attached
to. And interrupts are expensive. Not only do they operate in kernel
mode, they also prevent other interrupts from being delivered. Each
interrupt has its IPL (Interrupt Priority Level) and during the
interrupt handler execution (aka “driver”), all interrupts with the IPL
lower than the currently executing one are blocked. The clock interrupt
is at the lowest priority level, which means that any disk interrupt
will prevent the time sharing system during its execution. Rasing the
number of interrupts is not a trivial matter. It doesn’t really matter
for my desktop computer and burning CD’s, which nobody does any more,
but for my database server it greatly increases the price of my IO. On
LinkedIn RAC group, a guy reported performance drop as much as 25% as
compared to ASMLib, which uses raw devices. He was talking about zLinux
on an IBM mainframe which is particularly sensitive because of the
mainframe IO architecture which is all about the smart channels and
avoiding interrupts. Raw devices, which is the interface presented to
ASM by ASMLib, do not increase the number of interrupts as they are not
emulating anything.

About these ads

About mgogala

I am a long time Oracle DBA, who has worked on very large databases. I have worked with OPS and RAC since its inception.I am also a published book writer, having published two books about PHP. This blog is about the challenges and adventures in my professional life. Sorry, no family pictures here.
This entry was posted in Miscellaneous Oracle and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s