Cluster:Direct access is not working

From: Udo Grabowski <udo.grabowski_at_imk.fzk.de>
Date: Tue, 07 Aug 2001 18:29:07 +0200

Hello managers !

After killing advfsd, I finally was able to do some performance
tests on our TruCluster 5.1/EMA 12000 (HSG80) 8 member system.
Monitoring throughput using portperfshow on the switch, I noticed
that access to a disk on the HSG80 always goes through the current
server of the disk, although that disk is a direct access (don't
mix this up with direct IO!) disk on a shared bus (fibre channel
SCSI 2) as drdmgr shows:

drdmgr dsk4:

    View of Data from member imkdec43 as of 2001-08-07:18:10:13

                    Device Name: dsk4
                    Device Type: Direct Access IO Disk
                  Device Status: OK
              Number of Servers: 7
                    Server Name: imkdec41
                   Server State: Server
                    Server Name: imkdec42
                   Server State: Server
                    Server Name: imkdec43
                   Server State: Server
                    Server Name: imkdec44
                   Server State: Server
                    Server Name: imkdec45
                   Server State: Server
                    Server Name: imkdec46
                   Server State: Server
                    Server Name: imkdec47
                   Server State: Server
             Access Member Name: imkdec43
            Open Partition Mask: 0x1 < a >
   Statistics for Client Member: imkdec43
      Number of Read Operations: 203871
     Number of Write Operations: 11568
           Number of Bytes Read: 14998511616
        Number of Bytes Written: 94765056

----
cfsmgr /Incoming  (is on dsk4)
  Domain or filesystem name = /Incoming
  Server Name = imkdec47
  Server Status : OK
The disk has a domain#fileset which is mounted on /Incoming, the server
for / is currently the imkdec41. In the documents is stated explicitly:
     "Because dsk<n> is a direct-access I/O device on the shared bus, all three
  systems on the bus serve it. This means that, when any member on the shared bus
  accesses the disk, the access is directly from the member to the device."
Which it is definitely not in our case ! Even when I force high throughput
with several large files written to that directory from all servers, the
access goes still through memory channel and the imkdec47.
The only thing that might influence this could be that the current server
and the imkdec46 have an additional hop over a second FC switch to the
target controller, all others are directly connected over the main switch
(16 Port EL). Access pathes are explicitly set to all servers on the HSG80.
Did I miss something in the setup for the disks ? Is this a bug ?
Of course, I will summarize.
-- 
Dr. Udo Grabowski                           email: udo.grabowski_at_imk.fzk.de
Institut f. Meteorologie und Klimaforschung II, Forschungszentrum Karslruhe
Postfach 3640, D-76021 Karlsruhe, Germany           Tel: (+49) 7247 82-6026
http://www.fzk.de/imk/imk2/ame/grabowski/           Fax:         "    -6141

Received on Tue Aug 07 2001 - 16:30:22 NZST