# hp Reliable Transaction Router
System Manager's Manual

Previous | Contents | Index

## 7.2.26 Monitor Netbytes

Displays a list of the links to other nodes. For each link, the total number of bytes received and sent on that link and the number of bytes received and sent per second are displayed. Derived from the NIO_BYTES_RCVD and NIO_BYTES_SENT counters. The `Max` field represents the maximum rate since the link started.

Example 7-25 Monitor Netbytes

    LINK TRAFFIC IN BYTES Fri Apr 16 1999 17:41:12, NODE: nodea

                              Bytes received                  Bytes sent
                       --------------------------    --------------------------
                          Count      Rate     Max        Count     Rate     Max
    Total              59201466    6776.0      -    1002579480 113857.0      -

    nodea->nodea        3072248     336.0   336.0      3072248    336.0    336.0
    nodea->nodeb       42569496    4974.0  4974.0    717457678  84438.0  84438.0
    nodea->nodec       13559722    1466.0  1466.0    282049554  29083.0  29083.0

## 7.2.27 Monitor Netstat

Displays the link status for connected links in detail and the fail code for any links on which a connection has failed. Unconnected links where connections have been lost are highlighted. Link aborts, rejects, loss, gain, restarts, state and architecture of the remote node are also displayed. Displays more detail than MONITOR CONNECTS.

Example 7-26 Monitor Netstat

                  C o n n e c t i o n   S t a t u s   D e t a i l

                  Node: NODEA                    Mon March 15 1999 09:50:28

                   Ini Cnf Acc Abo Rej Loss Gain Ctmo Rstr AFO(1) TFO(2) State  Type FailCode
    Node and Link   12   0   2  12   12   1    3 2074 2073 863 864

    NODEA ->nodeb    0   0   0   0    0   0    1    0    0   0   0 up alpha
    NODEA ->nodec    6   0   0   6    6   0    0    0    0   0   0 down     ? 76490676
    NODEA ->noded    6   0   0   6    6   0    0    0    0   0   0 down     ? 76490676
    NODEA ->nodee 2591   0   0 863 2590   0    1 2074 2073 863 864 cinit    ? Remote node
                                                                              unreachable

1. Address failover
2. Transport failover

## 7.2.28 Monitor Ortr

Displays lists of server and client transaction controllers.

Example 7-27 Monitor Ortr

                          oRTR Transaction Controller List on node : rtrdoc
                        Image: -ALL-                                     at 15:42:55

    Server Controllers                      Client Controller
    STC000                                  CTC000
    STC001                                  CTC001
    STC002                                  CTC002
    STC003                                  CTC003
    STC004                                  CTC004

## 7.2.29 Monitor Partit

Displays partitions as `partition name` if a partition name has been specified using the SET PARTITION command. The number of servers and key segments are shown for each partition. The least-significant byte of the partition's low and high bound is also shown, and callout type (if any). Table 7-3 shows partition state meanings.

Example 7-28 Monitor Partit

    PARTITION DEFINITIONS  Tue Apr 6 1999 10:40:31, NODE: NODEA

    Partition name                             #    #        Bounds      Callout
                                     State     Svrs Segs   lo      hi    type
    RTR$DEFAULT_PARTITION_16777218   active       1    1  "A"     "A"     -
    RTR$DEFAULT_PARTITION_16777221   active       1    1  "B"     "B"     -
    RTR$DEFAULT_PARTITION_16777217   active       1    1   0     429496   -

Table 7-3 Monitor Partition States
State | Meaning
--- | ---
wt_tr_ok | Server is waiting for routers to accept it
wt_quorum | Server is waiting for backend to be quorate
lcl_rec | Local recovery
lcl_rec_fail | Primary server waiting for access to a restart journal
lcl_rec_icpl | Getting next journal to recover from
lcl_rec_cpl | Processed all journals for local recovery
shd_rec | Shadow recovery
shd_rec_fail | Shadow server waiting for access to a restart journal
shd_rec_icpl | Shadow getting next journal to recover from
shd_rec_cpl | Processed all journals for shadow recovery
catchup | Secondary is catching up with primary
standby | Server is declared as standby
active | Server is active
pri_act | Server is active as primary shadow
sec_act | Server is active as secondary shadow
remember | Primary is running without shadow secondary

## 7.2.30 Monitor Queues

Shows transaction queues by partition. Uses counters from Transaction Manager (TM) and the Requester/Server configurator (RSC). A delay is the delay in seconds for the most recent transaction handed to a server, which on average will be the same for all servers.

Example 7-29 Monitor Queues

    TRANSACTION QUEUES BY PARTITION Fri Sep 08 2000 12:42:53, NODE: NODEA

    Partition name                            Processed          Queued     #
                                         Txns    Msgs  Rplys    Txn  Msg   Svrs Dly
    RTR$DEFAULT_PARTITION_16842753       5792    5794      0      2    6      3   0

## 7.2.31 Monitor Quorum

Displays for each node and facility, the state and the number of roles RTR sees as configured (CNF), reachable (RCH) and quorate (QRT). The State column shows quorum states for router (TR) and backend (BE) roles.

Example 7-30 Monitor Quorum

    QUORUM STATUS BY NODE AND FACILITY Tue Dec 04 2001 10:50:24, NODE: NODEA
    (Counts can be inaccurate for incorrectly configured facilities)
    States: bad configuration,not connected,minority,uncertain,quorate
    Node & Facility                       State                          CNF RCH QRT
    NODEA    RTR$DEFAULT_FACILITY         TR:quorate,BE:quorate            2   2   2
    NODEA    shadow                       TR:quorate,BE:minority(node12)   4   2   1
    NODEA    payroll                      TR:bad_config(NODEB)             1   0   0

## 7.2.32 Monitor Recovery

Shows the progress of transaction recovery. Last recovery backend is the last backend accessed to recover transactions. If the server state is `lcl_rec_fail` or `shd_rec_fail` , the Last Recovery Backend is the backend that could not be accessed. Journal scans is the number of journal files searched. Transactions recovered is the number of transactions found for this partition.

Table 7-4 shows the meaning of entries in the Server State column.

Example 7-31 Monitor Recovery

    RECOVERY INFORMATION at Tue Apr  6 1999 10:54:50, on NODEA

                               Last        Restart-Recovery     Shadow-Recovery
                  Server       Recovery    Journal  Txns        Journal  Txns
    Partition     State        Backend     Scans    Recovered   Scans    Recovered
    ------------  ------       -------     -------  ---------   -------  ---------
    RTR_PARTITA   active       NODEA             1          0         0          0
    RTR_PARTITB   active       NODEA             1          0         0          0
    RTR_PARTITC   active       NODEA             1          0         0          0

Table 7-4 Monitor Recovery States
State | Meaning
--- | ---
wt_tr_ok | Server is waiting for routers to accept it
wt_quorum | Server is waiting for backend to be quorate
lcl_rec | Local recovery
lcl_rec_fail | Primary server waiting for access to a restart journal
lcl_rec_icpl | Getting next journal to recover from
lcl_rec_cpl | Processed all journals for local recovery
shd_rec | Shadow recovery
shd_rec_fail | Shadow server waiting for access to a restart journal
shd_rec_icpl | Shadow getting next journal to recover from
shd_rec_cpl | Processed all journals for shadow recovery
catchup | Secondary is catching up with primary
standby | Server is declared as standby
active | Server is active
pri_act | Server is active as primary shadow
sec_act | Server is active as secondary shadow
remember | Primary is running without shadow secondary

## 7.2.33 Monitor Rejects

Displays the last `rtr_mt_rejected` message received by each running process. Table 7-5 gives the meaning of each column.

Example 7-32 Monitor Rejects

                             Rejected Transaction Summary
    NODE: NODEA                    PROCESS: 20413894    Fri Apr 9 1999 10:26:14

            Time           Pid     Chan    Reason   Status Text
    -------------------   ------  ------  --------  -----------------------------
    Fri Apr  9 10:18:43  20417266 client         0  No server available to handle
    Fri Apr  9 10:17:47  20417274 server         0  Client aborted tx

Table 7-5 Monitor Rejects Fields
Field | Meaning
--- | ---
Time | Date and time when the `rtr_mt_rejected` message was received
Pid | ID of the process that received the message
Chan | Type of channel (client or server) that received the message
Reason | Reason field returned in the `rtr_status_data_t` buffer, if any
Status Text | Text describing the reject reason

## 7.2.34 Monitor Rejhist

Displays the last 10 `rtr_mt_rejected` messages received by the selected process. Always invoke this picture with the /ID qualifier. The transaction identifier associated with the rejected transaction can be displayed with the `SHOW PROCESS` < `id>/COUNTER=api_reject*` command. Table 7-6 shows the meaning of Rejhist columns.

Example 7-33 Monitor Rejhist

                             Rejected Transaction History
    NODE: NODEA                    PROCESS: 38009A8B      Mon Mar  9 1999 10:26:14

            Time           Chan    Reason   Status Text
    -------------------   ------   -------  --------------------------------------
    Mon Mar 15 18:06:06   client         0  Client aborted tx
    Mon Mar 15 18:06:41   server         0  Normal successful completion
    Mon Mar 15 18:06:41   client         0  Server aborted tx

    ------------------------------------------------------------------------------

                            number of reject messages: 3
                            number of accept messages: 0
                            rejected / total txns (%): 100%

                This display should be invoked with the /ID qualifier.

Table 7-6 Monitor Rejhist Fields
Field | Meaning
--- | ---
Time | Date and time when the `rtr_mt_rejected` message was generated
Chan | Type of channel (client or server) that received the message
Reason | Reason field returned in the `rtr_status_data_t` buffer, if any
Status Text | Text describing the reject reason

## 7.2.35 Monitor Response

Displays the elapsed time that a transaction has been active on the opened channels of a process. On the client, transaction duration is measured between the `rtr_start_tx` or `rtr_send_tx` call and the receipt of the final `rtr_mt_accepted` or `rtr_mt_rejected` message. A call to `rtr_reject_tx` also marks the end of a transaction. On the server, transaction duration is measured between receipt of an `rtr_mt_msg1` or `rtr_mt_msg1_uncertain` message and the receipt of the final `rtr_mt_rejected` message or `rtr_reject_tx` call. Accepted transaction end times are recorded when the server issues an `rtr_receive_message` call to request a new transaction for processing.

Example 7-34 Monitor Response

                    TRANSACTION DURATION AT 10:24:51 Fri Apr  9 1999

    Process ID, Image Name      Client Response Time        Server Response Time
                             seconds 0---1---2---3---4   seconds 0---1---2---3---4
    20413894   SERVER.EXE;4   0.000                       3.670
    20417266   RTR.EXE;75     2.200                       3.440
    20417274   SERVER.EXE;4   0.000                       1.160

## 7.2.36 Monitor Rolequor

Displays for each facility the number of roles RTR sees as configured (CNF), reachable (RCH), and quorate (QRT), from the point of view of the routers and the backends. Compare Section 7.2.31.

Example 7-35 Monitor Rolequor

    QUORUM COUNTS BY FACILITY  Tue Dec 04 2001 14:32:48, NODE:  NODEA

    Facility name                  Router View of           Backend View of
                                Backends    Routers       Backends     Routers
                              CNF RCH QRT  CNF RCH QRT  CNF RCH QRT  CNF RCH QRT

    RTR$DEFAULT_FACILITY        1   1   1    1   1   1    1   1   1    1   1   1
    funds_transfer              1   0   0    0   0   0

## 7.2.37 Monitor Routers

Displays information on a router node. It gives an indication of the utilization of the router in terms of transactions and broadcasts routed through this node. Useful to monitor performance or locate problems. Uses counters in the Transaction Manager (TM) subsystem.

Example 7-36 Monitor Routers

    ROUTER TRANSACTION COUNTERS AT 14:33:29  Fri Sep 08 2000

    Node:     NODEA
    Facility: -ALL-

                  Count    Rate         10  20  30  40  50  60  70  80  90 100
    Starts:      116641    25.7
    Enqueues:    116641    25.7
    Commits:     116641    25.6
    Aborts:           0     0.0

## 7.2.38 Monitor Routing

Displays statistics of transaction and broadcast traffic by facility. Rate is the number of transactions or broadcasts per second within the monitoring interval. A value in the No KR column indicates the absence of an appropriate key range server. A value in the NoDst column indicates the absence of an appropriate broadcast destination. Counts in these columns probably mean that a required server application is missing or perhaps that there is a problem with application configuration.

Example 7-37 Monitor Routing

    ROUTING STATISTICS BY FACILITY  Tue Dec 04 2001 14:34:20, NODE: NODEA

    Facility name                              Transactions        Broadcasts
                                              Count    Rate NoKR Count    Rate NoDst
    Total                                    118489    39.2    0 68444   994.0     1

    RTR$DEFAULT_FACILITY                     118489    39.2    0 68444   994.0     1

## 7.2.39 Monitor Rscbe

Displays, by partition, the history of the most recent request-to-server messages issued to a backend from each router. The RSC calls column shows the message type, the Router column gives the router node name, the State column shows the partition state for messages to a router and the target state for messages from a router. T-delta is the elapsed time in seconds since the partition was established, and Seq_nr gives the partition sequence number or priority.

Example 7-38 Monitor Rscbe

    Most Recent RSC Dclsrv Calls History on Backend NODEA Thu Mar 4 1999,15:19:41

    Partition name: RTR$DEFAULT   Partition Start Time: THU MAR  4 15:18:22 1999
    Image Name:   RTR.EXE

    T-delta  RSC calls           Router           State                 Seq_nr
          0  send_dcl_to_master  nodez            wait_for_response          0
          1  recv_status_ok      nodez            rstart_rvy                 1
          1  send_dcl_to_master  nodez            rstart_rvy_incomp          1
          1  recv_status_ok      nodez            rstart_rvy                 1
          1  send_dcl_to_master  nodez            rstart_rvy_incomp          1
          1  recv_status_ok      nodez            active                     1
          1  send_dcl_to_other   node10           active                     1
          1  recv_status_ok      node10           active                     1

## 7.2.40 Monitor Stalls

Displays in real time any network links that are currently stalling (that is, waiting to transmit outbound traffic) and provides a history of the stalls encountered by the various links during their lifetime. The display shows:

- Node-to-node path of link analyzed
- The outgoing network message count per link
- The outgoing network message rate per link
- The number of bytes sent per link
- The number of link disconnects
- Total accumulated network write-completion wait time, in seconds
- Count of stalls between 1 and 3 seconds
- Count of stalls between 3 and 10 seconds
- Count of stalls between 10 and 30 seconds
- Count of stalls longer than 30 seconds
- Count of stalled network write completions

Example 7-39 Monitor Stalls

    NETWORK STALLS AT Fri Sep 08 2000 15:35:03, ON NODE: TR1
                       Messages    Bytes  Link             Stalls
                     Issued Rate    Sent Drops Secs  <3s  <10s  <30s  >30s Totals
    Total              5467  0.0  327148     2   33   23     1     0     0  24
    TR1 -> TR1           29  0.0    3718     0    0    0     0     0     0   0
    TR1 -> FE2          509  0.0   20707     0    4    4     0     0     0   4
    TR1 -> BE1          303  0.0   13707     0    3    3     0     0     0   3
    TR2 -> TR2          111  0.0   11682     0    0    0     0     0     0   0
    TR2 -> BE1          504  0.0   22743     0   18    8     1     0     0   9
    FE1 -> FE1           64  0.0    6645     0    0    0     0     0     0   0
    FE1 -> FE2          373  0.0   18890     0    2    2     0     0     0   2
    FE1 -> BE1          310  0.0   24487     0    0    0     0     0     0   0
    FE2 -> FE2          231  0.0   18900     0    0    0     0     0     0   0
    FE2 -> BE1          284  0.0   22503     1    1    1     0     0     0   1
    FE2 -> TR1          536  0.0   28166     0    0    0     0     0     0   0
    FE2 -> FE1          396  0.0   23643     0    0    0     0     0     0   0
    BE1 -> BE1          355  0.0   28121     0    0    0     0     0     0   0
    BE1 -> FE2          284  0.0   13014     1    0    0     0     0     0   0
    BE1 -> TR2          515  0.0   27502     0    2    2     0     0     0   2
    BE1 -> TR1          328  0.0   25698     0    1    1     0     0     0   1
    BE1 -> FE1          335  0.0   17022     0    2    2     0     0     0   2

Previous | Next | Contents | Index