Document revision date: 30 March 2001

## Guidelines for OpenVMS Cluster Configurations

Previous | Contents | Index

### 6.7.2 Invalid Multipath Configuration

Figure 6-20 shows an invalid multipath configuration. The configuration is invalid because, if multiple hosts in a cluster are connected to an HSZ or HSG, they must all have connections to the same controller modules (see Table 6-1). In this configuration, each host is connected to a different controller module.

Figure 6-20 Invalid Multipath Configuration

### 6.7.3 Multipath System Parameters

Multipath support is enabled and qualified by the use of four system parameters, as described in Table 6-3. (A fifth multipath system parameter is reserved for the operating system.)

Table 6-3 Multipath System Parameters
Parameter | Description
--- | ---
MPDEV_ENABLE | Enables the formation of multipath sets when set to ON (1). If set to OFF (0), the formation of additional multipath sets is disabled. However, existing multipath sets remain in effect. The default is ON.
MPDEV_LCRETRIES | Controls the number of times the system retries direct paths to the controller that the logical unit is online to, before moving on to direct paths to the other controller, or to an MSCP served path to the device. The valid range for retries is 1 through 256. The default is 1.
MPDEV_POLLER | Enables polling of the paths to multipath set members when set to ON (1). Polling allows early detection of errors on inactive paths. If a path becomes unavailable or returns to service, the system manager is notified with an OPCOM message. If set to OFF (0), multipath polling is disabled. The default is ON.
MPDEV_REMOTE | Enables MSCP served disks to become members of a multipath set when set to ON (1). If set to OFF (0), only local paths to a SCSI or Fibre Channel device will be used in the formation of additional multipath sets. However, setting this parameter to OFF will not have any effect on existing multipath sets that have remote paths. This parameter must remain set to the default value of OFF. This restriction will be removed in a future release of OpenVMS.
MPDEV_D1 | Reserved for use by the operating system.

If a node has MPDEV_ENABLE or MPDEV_REMOTE set to OFF, and the node has one or more direct paths and served paths to its system disk, then the node must only boot on the direct paths.

### 6.7.4 Path Identifiers

The system management commands described in the following sections allow you to monitor and control the operation of multipath failover. These commands provide a path identifier to uniquely specify each path in a multipath set.

Direct Fibre Channel paths are identified by the local host adapter name and the remote Fibre Channel port WWID --- that is, the initiator and the target. For example, in Figure 6-21, the path identifier for the path from the host adapter on the left to the HSG storage controller on the left is PGB0.5000-1FE1-0000-0200. (The second port on each HSG is omitted for convenience.) You can obtain the WWID for a storage controller from its console.

Figure 6-21 Fibre Channel Path Naming

Direct parallel SCSI paths are identified by the local host adapter name and the remote SCSI bus ID --- that is, the initiator and the target. For example, in Figure 6-22, the path identifiers for node Edgar's two direct paths to the disk would be named PKB0.5 and PKC0.5.

The path identifier for MSCP served paths is `MSCP` .

Figure 6-22 Configuration With Multiple Direct Paths

### 6.7.5 Displaying Paths

When multipath support is enabled, you can display the multiple paths to a device using either of the following variants of the SHOW DEVICE DCL command:

    SHOW DEVICE/FULL device-name

    SHOW DEVICE/MULTIPATH_SET device-name

The SHOW DEVICE/FULL device-name command displays the traditional information about the device first and then lists all the paths to a device by their path identifiers (described in Section 6.7.4).

The SHOW DEVICE/MULTIPATH_SET device-name command lists only the multiple paths to a device.

Multipath information is displayed only on nodes that are directly connected to the multipath device.

#### 6.7.5.1 Displaying Paths With SHOW DEVICE/FULL

The following example shows the output of a SHOW DEVICE/FULL device-name command. Note that the use of multiple paths is shown at the beginning of the display ( `device has multiple I/O paths` ), and the multiple path descriptions are shown toward the end of the display, beneath `I/O paths to device` . Note, too, that the values for `Error count` and `Operations completed` shown at the beginning of the display are the sums of the counts for each path.

    $ SHOW DEVICE/FULL $70$DKA100:

    Disk $70$DKA100: (FLAM10), device type DEC HSZ70, is online, mounted, file-
        oriented device, shareable, device has multiple I/O paths, served to cluster
        via MSCP Server, error logging is enabled.

        Error count                    2    Operations completed             123471
        Owner process                 ""    Owner UIC                      [SYSTEM]
        Owner process ID        00000000    Dev Prot            S:RWPL,O:RWPL,G:R,W
        Reference count                1    Default buffer size                 512
        Total blocks             4109470    Sectors per track                    85
        Total cylinders             3022    Tracks per cylinder                  16
        Allocation class              70

        Volume label        "X6MV_SYS_1"    Relative volume number                0
        Cluster size                   9    Transaction count                     1
        Free blocks              2040228    Maximum files allowed            205524
        Extend quantity                5    Mount count                           4
        Mount status              System    Cache name        "_$600$DKA0:XQPCACHE"
        Extent cache size             64    Maximum blocks in extent cache   204022
        File ID cache size            64    Blocks currently in extent cache      0
        Quota cache size               0    Maximum buffers in FCP cache       2594
        Volume owner UIC           [1,1]    Vol Prot    S:RWCD,O:RWCD,G:RWCD,W:RWCD

      Volume Status:  ODS-2, subject to mount verification, file high-water marking,
          write-back caching enabled.
      Volume is also mounted on FIBRE3, COB2, FLAM10.

      I/O paths to device              2
      Path PKC0.1  (SISKO), primary path, current path.
        Error count                    1    Operations completed             121001
      Path PKB0.1  (SISKO).
        Error count                    1    Operations completed               2470

For each path of the multipath device, the path identifier, the host name associated with that path, the path status, the error count, and the operations count are displayed.

The terms that may appear in the multiple paths portion of the display are described in Table 6-4.

Table 6-4 SHOW DEVICE/FULL Multipath Terms
Term | Description
--- | ---
Primary path | This was the first path to the device found by the operating system.
Current path | This path is currently used for I/O.
User disabled | The DCL command SET DEVICE/NOENABLE has been executed for this path.
Polling disabled | The DCL command SET DEVICE/NOPOLL has been executed for this path.
Unavailable | The path is unavailable because the software driver has disconnected from the path.
MSCP host unavailable | The path is unavailable because the connection to the MSCP server on the remote host has been broken.

#### 6.7.5.2 Displaying Paths With SHOW DEVICE/MULTIPATH_SET

You can obtain a brief listing of multiple paths for a specific device, for all the devices in an allocation class, or for all devices with the DCL command:

    SHOW DEVICE/MULTIPATH_SET [device-name]

The device name is optional; when omitted, all devices that have formed multipath sets are shown. For each multipath device found, the device name, host name, device status, error count, number of accessible paths, total number of paths, and the current path's path identifier are displayed.

The host name displayed is the host name of the current path. For direct paths, this will be the local system's host name. For MSCP served paths, this will be the host name of the remote system which is serving access to the device.

The following example shows the output of a SHOW DEVICE /MULTIPATH command.

    $ SHO DEV /MULTIPATH
    Device                  Device           Error          Current
     Name                   Status           Count   Paths    path
    $70$DKA0:     (FIBRE3)  Online               0   3/ 3   PKB0.0
    $70$DKA100:   (FIBRE3)  MountVerify         11   2/ 3   PKB0.1
    $70$DKA200:   (FIBRE3)  Mounted              0   3/ 3   PKC0.2
    $70$DKA300:   (FIBRE3)  Mounted              2   1/ 3   PKC0.3
    $70$DKA400:   (FIBRE3)  Mounted              4   3/ 3   PKC0.4
    $70$DKA500:   (FIBRE3)  Mounted              7   3/ 3   PKC0.5
    $70$DKA1400:  (FIBRE3)  Mounted              0   3/ 3   PKC0.14
    $70$DKA1500:  (FIBRE3)  Mounted              4   3/ 3   PKB0.15
    $100$DKA100:  (FIBRE3)  Online               0   2/ 2   PKA0.1
    $100$DKA200:  (FIBRE3)  Mounted              3   2/ 2   PKA0.2
    $100$DKA300:  (FIBRE3)  Online               0   2/ 2   PKA0.3
    $100$DKA400:  (FIBRE3)  Online wrtlck        0   2/ 2   PKA0.4

If you choose to specify a partial device name, such as $70$DKA, the display shows all devices with multiple paths whose names begin with $70$DKA.

### 6.7.6 Path Polling

When SCSI multipath support is in effect, the system periodically polls all the I/O paths from each host adapter to each HSZ or HSG controller to determine the status of each I/O path. If the system detects any changes to a path, it outputs a message, similar to the following messages, to the console and to the operator's log:

    All multipath devices on path PKB0.5 are either disabled or not reachable.

or

    At least one multipath device on path PKB0.5 is enabled and reachable.

If all the devices on a path are removed, a path failure is reported. The path from the host to the HSx controller may still function, but this cannot be determined when there are no devices to poll.

You can turn polling on or off with the following command:

    SET DEVICE device/[NO]POLL/PATH=path-identifier

Turning off polling for a path that will be out of service for a prolonged period is useful because it can reduce system overhead.

### 6.7.7 Switching Current Paths Manually

You can switch a device's current path manually using the SET DEVICE command with the /SWITCH qualifier. The most common reason for doing this is to balance the aggregate I/O load across multiple HSx controller modules and buses.

The command syntax for switching the current path is:

    SET DEVICE device-name/SWITCH/PATH=path-identifier

The following command switches the path of device $2$DKA502 to an MSCP served path.

    $ SET DEVICE $2$DKA502/SWITCH/PATH=MSCP

Note that this command initiates the process of switching the path and then returns to the DCL prompt immediately. A delay may occur between when the DCL prompt reappears and when the path switch is complete.

A manual path switch is accompanied by mount verification messages and by a status message indicating the result of the path switch request. This is shown in Example 6-1:

Example 6-1 Messages Resulting from Manual Path Switch

    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:48:56.66
    Device $1$DGA50: (FCNOD1 PGA, FCNOD2) is offline.
    Mount verification is in progress.

    _$
    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:49:12.30
    Mount verification has completed for device $1$DGA50: (FCNOD1 PGA, FCNOD2)

    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:49:12.31
    Multipath access to device $1$DGA50: has been manually switched from
     path PGB0.5000-1FE1-0000-0D13 to path PGA0.5000-1FE1-0000-0D12

You can check for completion of a path switch by issuing the SHOW DEVICE/FULL command, or the SHOW DEVICE/MULTIPATH command.

Note that if the path that is designated in a manual path switch fails during the switch operation, then automatic path switching takes over. This can result in a switch to a path different from the one designated in the command.

If a manual path switch causes a virtual unit to switch from one HSG80 controller to another controller, then the command can affect other nodes in the cluster. These nodes will experience a mount verification on their current path. This will cause an automatic switch to a path on the other HSG80 controller. The messages indicating this event are shown in Example 6-2:

Example 6-2 Messages Displayed When Other Nodes Detect a Path Switch

    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:50:56.66
    Device $1$DGA50: (FCNOD1 PGA, FCNOD2) is offline.
    Mount verification is in progress.

    _$
    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:51:12.30
    Mount verification has completed for device $1$DGA50: (FCNOD1 PGA, FCNOD2)

    %%%%%%%%%%%  OPCOM  17-MAR-1999 20:51:12.31
    Multipath access to device $1$DGA50: has been automatically switched from
     path PGB0.5000-1FE1-0000-0D13 to path PGA0.5000-1FE1-0000-0D11

### 6.7.8 Path Selection by OpenVMS

In a multipath configuration, the first path to a device that OpenVMS configures is chosen as the initial current path. Note that this overrides the selection of a preferred path through HSx console commands such as the following:

    HSG> SET UNIT PREFERRED_PATH=THIS_CONTROLLER

or

    HSG> SET UNIT PREFERRED_PATH=OTHER_CONTROLLER

For this reason, it is not recommended that the OpenVMS system manager set preferred paths at the HSx console. Instead the manual path switching commands, described in Section 6.7.7 should be used.

In case of failover, OpenVMS chooses an alternate path according to the following order of precedence:

1. The path specified by the system manager in a manual switch command
2. A direct path where failover from one HSx controller module to another is not required
3. A direct path, where HSx failover is required
4. MSCP served path

OpenVMS avoids unnecessary failover from one HSx controller module to another because:

- Failover from one HSx controller module to another causes a delay of approximately 1 to 15 seconds, depending on the amount of cached data that needs to be synchronized.
- Other nodes that share access to the device must reestablish communication using an alternate path.

### 6.7.9 How OpenVMS Performs Multipath Failover

When an I/O operation to a multipath disk fails and the failure suggests that a retry (on the current path or an alternate path) might succeed, then failover proceeds as follows:

1. Mount verification is invoked.
2. Mount verification attempts to communicate with the device on the current path and to validate that the volume is still correct.
3. If mount verification fails on the current path, then a new path is sought according to the following order:
   - Direct paths that do not require an HSx failover are attempted a few times to maximize their chance of selection (the number of retries is controlled by the SYSGEN parameter MPDEV_LCRETRIES).
   - Any direct path is attempted.
   - Any path, including the served path is attempted.
   These steps execute in a loop, until a working path is located, or mount verification times out.
4. A successful mount verification restarts all failed I/Os and allows new ones to proceed.

### 6.7.10 Enabling or Disabling Paths as Path Switch Candidates

By default, all paths are candidates for path switching. You can disable or re-enable a path as a switch candidate by using the SET DEVICE command with the /[NO]ENABLE qualifier. The reasons you might want to do this include the following:

- You know a specific path is broken, or that a failover to that path will cause some members of the cluster to lose access.
- To prevent automatic switching to a selected path while it is being serviced.

Note that the current path cannot be disabled.

The command syntax for enabling a disabled path is:

    $ SET DEVICE device-name/[NO]ENABLE/PATH=path-identifier

The following command enables the MSCP served path of device $2$DKA502.

    $ SET DEVICE $2$DKA502/ENABLE/PATH=MSCP

The following command disables a local path of device $2$DKA502.

    $ SET DEVICE $2$DKA502/ENABLE/PATH=PKC0.5

Be careful when disabling paths. Avoid creating an invalid configuration, such as the one shown in Figure 6-20

### 6.7.11 Console Considerations

This section describes how to use the console with parallel SCSI multipath devices. Refer to Section 7.6 for information on using the console with FC multipath devices.

The console uses traditional, path-dependent, SCSI device names. For example, the device name format for disks is DK, followed by a letter indicating the host adapter, followed by the SCSI target ID, and the LUN.

This means that a multipath device will have multiple names, one for each host adapter it is accessible through. In the following sample output of a console show device command, the console device name is in the left column. The middle column and the right column provide additional information, specific to the device type.

Notice, for example, that the devices `dkb100` and `dkc100` are really two paths to the same device. The name `dkb100` is for the path through adapter PKB0, and the name `dkc100` is for the path through adapter PKC0. This can be determined by referring to the middle column, where the informational name includes the HSZ allocation class. The HSZ allocation class allows you to determine which console "devices" are really paths to the same HSZ device.

Note The console may not recognize a change in the HSZ allocation class value until after you issue a console INIT command.

    >>>sho dev
    dkb0.0.0.12.0              $55$DKB0                       HSZ70CCL  XB26
    dkb100.1.0.12.0            $55$DKB100                        HSZ70  XB26
    dkb104.1.0.12.0            $55$DKB104                        HSZ70  XB26
    dkb1300.13.0.12.0          $55$DKB1300                       HSZ70  XB26
    dkb1307.13.0.12.0          $55$DKB1307                       HSZ70  XB26
    dkb1400.14.0.12.0          $55$DKB1400                       HSZ70  XB26
    dkb1500.15.0.12.0          $55$DKB1500                       HSZ70  XB26
    dkb200.2.0.12.0            $55$DKB200                        HSZ70  XB26
    dkb205.2.0.12.0            $55$DKB205                        HSZ70  XB26
    dkb300.3.0.12.0            $55$DKB300                        HSZ70  XB26
    dkb400.4.0.12.0            $55$DKB400                        HSZ70  XB26
    dkc0.0.0.13.0              $55$DKC0                       HSZ70CCL  XB26
    dkc100.1.0.13.0            $55$DKC100                        HSZ70  XB26
    dkc104.1.0.13.0            $55$DKC104                        HSZ70  XB26
    dkc1300.13.0.13.0          $55$DKC1300                       HSZ70  XB26
    dkc1307.13.0.13.0          $55$DKC1307                       HSZ70  XB26
    dkc1400.14.0.13.0          $55$DKC1400                       HSZ70  XB26
    dkc1500.15.0.13.0          $55$DKC1500                       HSZ70  XB26
    dkc200.2.0.13.0            $55$DKC200                        HSZ70  XB26
    dkc205.2.0.13.0            $55$DKC205                        HSZ70  XB26
    dkc300.3.0.13.0            $55$DKC300                        HSZ70  XB26
    dkc400.4.0.13.0            $55$DKC400                        HSZ70  XB26
    dva0.0.0.1000.0            DVA0
    ewa0.0.0.11.0              EWA0              08-00-2B-E4-CF-0B
    pka0.7.0.6.0               PKA0                  SCSI Bus ID 7
    pkb0.7.0.12.0              PKB0                  SCSI Bus ID 7  5.54
    pkc0.7.0.13.0              PKC0                  SCSI Bus ID 7  5.54

The console does not automatically attempt to use an alternate path to a device if I/O fails on the current path. For many console commands, however, it is possible to specify a list of devices that the console will attempt to access in order. In a multipath configuration, you can specify a list of console device names that correspond to the multiple paths of a device. For example, a boot command, such as the following, will cause the console to attempt to boot the multipath device through the DKB100 path first, and if that fails, it will attempt to boot through the DKC100 path:

    BOOT DKB100, DKC100

Previous | Next | Contents | Index

| privacy and legal statement |
6318PRO_007.HTML