# Encryption for OpenVMS Installation and Reference Manual

Previous | Contents | Index

# Chapter 4
Programming with Encryption for OpenVMS Routines

To program encryption operations into applications, use the Encryption for OpenVMS callable routines. Encryption provides the following routines, listed by function:

- Defining, generating, and deleting keys ( Section 4.2):
  - ENCRYPT$DEFINE_KEY
  - ENCRYPT$GENERATE_KEY
  - ENCRYPT$DELETE_KEY
- Encrypting and decrypting files ( Section 4.3):
  - ENCRYPT$ENCRYPT
  - ENCRYPT$ENCRYPT_FILE
  - ENCRYPT$DECRYPT
- Encrypting and decrypting records ( Section 4.4):
  - ENCRYPT$DECRYPT_ONE_RECORD
  - ENCRYPT$ENCRYPT_ONE_RECORD
- Intializing and terminating the context area ( Section 4.4):
  - ENCRYPT$INIT
  - ENCRYPT$FINI
- Returning statistics ( Section 4.4):
  - ENCRYPT$STATISTICS

Installation Note To use the programming interface from an image that was installed with privileges, ensure that the system startup procedure installs the ENCRYPSHR shareable image as a known image (see Section 2.4).

# 4.1 How the Routines Work

You can call the Encryption for OpenVMS routines from any language that supports the OpenVMS Procedure Calling Standard. After it is called, each routine:

- Performs its function
- Returns a 32-bit status code value for the calling program to determine success or failure
- Returns control to the calling program

The callable routines do not provide all the options of the file selection qualifiers available with the DCL ENCRYPT and DECRYPT commands. The functions of /BACKUP, /BEFORE, /BY_OWNER, /CONFIRM, /EXCLUDE, /EXPIRED, /SINCE, and /SHOW are supported only at the DCL-interface level. For more information, see the Guide to Creating OpenVMS Modular Procedures.

## 4.1.1 DES Key and Data Semantics

The NBS document FIPS-PUB-46 describes the operation of the DES algorithm in detail. The bit-numbering conventions in the NBS document are different from OpenVMS numbering conventions.

If you are using Encryption for OpenVMS routines in conjunction with an independently developed DES encryption system, ensure that you are familiar with the relationship between the NBS and OpenVMS numbering conventions. Table 4-1 highlights the differences.

Table 4-1 Comparison of NBS and OpenVMS Numbering Conventions
NBS | Encryption for OpenVMS
--- | ---
Numbers bits from left to right. | Numbers bits from right to left.
Displays bytes in memory from left to right. | Displays bytes in memory from right to left.
Handles keys and data in 8-byte blocks (see Figure 4-1). | Handles 8-byte blocks in OpenVMS display order (see Figure 4-2).
Treats keys and data as byte strings. | Treats keys and data as character strings.
The "most significant byte" is byte 1. | Same.
In DES keys, the parity bits are DES bits 8, 16, 24, and so forth. | In DES keys, the parity bits are OpenVMS bits 0, 8, 16, and so forth.
DES keys, when expressed as strings of hexadecimal digits, are given starting with the high digit of byte 1, then the low digit of byte 1, then the high digit of byte 2, and so forth, through the low digit of byte 8. | Same.

To convert a hexadecimal key string into the 8-byte binary key, convert from hex to binary one byte at a time. For example, a quadword hex-to-binary conversion, using the library subroutine OTS$CVT_TZ_L, yields an incorrect, byte-reversed key.

Figure 4-1 OpenVMS Numbering Overlay on FIPS-46 Numbering

Figure 4-2 NBS Numbering Overlay on an OpenVMS Quadword

# 4.2 Maintaining Keys

When you use the Encryption routines, first define the key that will be used in the encryption operation. Similarly, to decrypt a file specify the same key. Table 4-2 describes the callable routines that maintain keys.

Table 4-2 Routines for Maintaining Keys
Routine | Description
--- | ---
ENCRYPT$DEFINE_KEY | Creates a key definition with a key name and a key value. Puts the definition into a key storage table. Similar to the ENCRYPT /CREATE_KEY command.
ENCRYPT$DELETE_KEY | Removes a key definition from a key storage table. Uses the key name to identify the key to be removed. Similar to the ENCRYPT /REMOVE_KEY command.
ENCRYPT$GENERATE_KEY | Generates random key values.

When you call these routines, use the following arguments:

- With ENCRYPT$DEFINE_KEY
  - To pass the values for the key name and key value, use the key-name and the key-value arguments.
  - To specify a key storage table, use the key-flags argument.
  - To specify other key options, use the key-flags argument.
  - To override key compression, use the key-flags argument.
- With ENCRYPT$DELETE_KEY
  - To pass the key name, use the key-name argument.
  - To specify the key storage table in which the key resides, use the key-flags argument.
- With ENCRYPT$GENERATE_KEY
  - To define the length of the key, use the key-length argument.
  - To specify the buffer into which the generated key is to be placed, use the key-buffer argument.
  - To specify the algorithm that will use the key, use the algorithm-name argument.
  - To optionally pass three arbitrary values for added security, use the factor-a, factor-b, and factor-c arguments. These values are randomizing factors when the routine generates a key value. For example, the factors might be:
    - Time an operation started
    - Size of a certain stack
    - Copy of the last command line

# 4.3 Operations on Files

The ENCRYPT$ENCRYPT_FILE routine is similar to the DCL ENCRYPT and DECRYPT commands in that you use this routine with entire files.

The ENCRYPT$ENCRYPT_FILE routine specifies the key, the input file specification, the output file specification, and other file operation information.

Specify the type of operation, either encryption or decryption, with the file-flags argument.

ENCRYPT$ENCRYPT_FILE does not require a prior call to ENCRYPT$INIT.

# 4.4 Operations on Records and Blocks

To operate on small records or blocks of data, use the following routines:

- ENCRYPT$ENCRYPT_ONE_RECORD
- ENCRYPT$DECRYPT_ONE_RECORD

These routines are a shorthand form of the ENCRYPT$INIT, ENCRYPT$ENCRYPT, ENCRYPT$DECRYPT, ENCRYPT$FINI sequence of calls.

Do not use these routines for data larger than a few records.

# 4.5 Routine Descriptions

This section describes the syntax of each callable routine. The routines are listed alphabetically.

## 4.5.1 Specifying Arguments

Each routine's argument list shows the mandatory arguments first, followed by the optional arguments. Brackets ( [ ] ) identify optional arguments in the argument list.

For example, this format line shows that the required arguments are context, input, and output, and that the optional arguments are output-length and p1:

ENCRYPT$DECRYPT context ,input ,output [,output-length] [,p1]

When you specify arguments, follow these guidelines:

- The order is important. Specify arguments in the order in which they appear in the argument list.
- Separate each argument with a comma.
- Pass a zero value for each optional argument that you omit.

## 4.5.2 Bitmasks

Constants are associated with the symbolic names of the bitmasks used by the Encryption routines. These constants are defined in the ENCRYPT_STRUCTURES files that are provided with the kit.

The examples directory, ENCRYPT$EXAMPLES, has a copy of the ENCRYPT_STRUCTURES file in each supported programming language.

## 4.5.3 Error Handling

By default, Encryption signals error conditions with messages. To intercept a message that is inappropriate for your application, supply a condition handler.

For information about implementing condition handlers, see your programming language reference manual. For additional documentation, see:

- The OpenVMS Programming Interfaces: Calling a System Routine --- provides full information about data types, access type, and passing mechanisms
- ENCRYPT$EXAMPLES:ENCRYPT$EXAMPLES.TXT --- online file that contains descriptions of files with sample application programs

# ENCRYPT$DECRYPT

> Decrypts the next record of ciphertext according to the algorithm specified in the ENCRYPT$INIT call.

## Format

> ENCRYPT$DECRYPT context ,input ,output [,output-length] [,p1]

## Arguments

> ### context

> VMS usage: |
> type: | longword integer (signed)
> access: | write only
> mechanism: | by reference

> Context area initialized when ENCRYPT$INIT completes execution. The context argument is the address of a longword of unspecified interpretation that is used to convey context between encryption operations.

> ### input

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Ciphertext record that ENCRYPT$DECRYPT is to decrypt. The input argument is the address of a descriptor pointing to a byte-aligned buffer containing the input record to the decryption operation.

> ### output

> VMS usage: |
> type: | char_string
> access: | write only
> mechanism: | by descriptor

> Plaintext record that results when ENCRYPT$DECRYPT completes execution. The output argument is the address of a descriptor pointing to a byte-aligned buffer that will contain the output record from the decryption operation.

> If the descriptor is dynamic and insufficient space is allocated to contain the output record, storage will be allocated from dynamic memory. If insufficient space exists to contain the output of the operation, then an error status is returned.

> The ENCRYPT$DECRYPT routine adjusts the length of the output descriptor, if possible, to reflect the actual length of the output string. If the descriptor type is not DSC$K_DTYPE_VS (varying string), DSC$K_DTYPE_V (varying), or DSC$K_DTYPE_D (dynamic), the routine takes the actual output count from the output-length argument.

> ### output-length

> VMS usage: |
> type: | word integer
> access: | write only
> mechanism: | by reference

> Optional argument.

> Number of bytes that ENCRYPT$DECRYPT wrote to the output buffer. The output-length argument is the address of a word containing the number of bytes written to the output buffer, including any bytes of pad characters generated by the selected algorithm to meet length requirements of the input buffer, if any. Output length does not count padding in the case of a fixed-length string.

> Some encryption algorithms have specific requirements for the length of the input and output strings. In particular, DESECB and DESCBC pad input data with from 1 to 7 bytes to form complete 64-bit blocks for operation. The values of the pad characters are indeterminate.

> When you decrypt fewer than 8 bytes, present the full 8 bytes resulting from the ENCRYPT$ENCRYPT to ENCRYPT$DECRYPT. Retain the byte count of the input data in order to strip trailing pad bytes after a subsequent decryption operation.

> ### p1

> VMS usage: |
> type: | quadword
> access: | read only
> mechanism: | by reference

> Optional argument. The p1 argument is the address of a quadword initialization vector used to seed the two modes of the DES algorithm for which it is applicable (DESECB and DESCFB). If this argument is omitted, the initialization vector used is the residue of the previous use of the specified context block. ENCRYPT$INIT initializes the context block with an initialization vector of zero.

## Description

> The ENCRYPT$DECRYPT routine decrypts the next record of ciphertext according to the algorithm specified in the ENCRYPT$INIT call. Any errors encountered in the operation are returned as status values. The message authentication mode (DESMAC) is not supported by ENCRYPT$DECRYPT.

> The ENCRYPT$DECRYPT routine returns a 32-bit status code indicating the success or failure of the routine's operation.

## Condition Values Returned

> SS$_NORMAL | Record successfully decrypted.
> ENCRYPT$ xyz | An error reported by the Encryption software. xyz identifies the message (see Appendix B).
> SS$_ xyz | A return status from a called system service. xyz identifies the return status.

# ENCRYPT$DECRYPT_ONE_RECORD

> Decrypts a small amount of data on a decrypt stream.

## Format

> ENCRYPT$DECRYPT_ONE_RECORD input ,output ,key-name ,algorithm

## Arguments

> ### input

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Ciphertext record to be decrypted. The input argument is the address of a string descriptor pointing to a byte-aligned buffer containing the input record to be decrypted.

> ### output

> VMS usage: |
> type: | char_string
> access: | write only
> mechanism: | by descriptor

> Plaintext record resulting when ENCRYPT$DECRYPT_ONE_RECORD completes execution. The output argument is the address of a string descriptor pointing to a byte-aligned buffer that will contain the plaintext record.

> If the descriptor is dynamic and insufficient space is allocated to contain the output record, storage is allocated from dynamic memory. If insufficient space exists to contain the output of the operation, an error is returned.

> The ENCRYPT$DECRYPT_ONE_RECORD routine adjusts the length of the output descriptor, if possible, to reflect the actual length of the output string.

> ### key-name

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Key used to initialize the decrypt stream. The key-name argument is the address of a string descriptor pointing to the name of the previously defined user key to be used.

> ### algorithm

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Algorithm used for the decryption operation. The algorithm argument is the address of a string descriptor pointing to a code for the selected algorithm. The algorithm code is an ASCII string. Specify the descriptor type value as one of the following:
> - DSC$K_DTYPE_T (text)
> - DSC$K_DTYPE_VT (varying text)
> - DSC$K_DTYPE_Z (unspecified)

> The following algorithms are valid:

> - DESCBC (default)
> - DESECB
> - DESCFB

## Description

> In some applications, only a small amount of data needs to be decrypted on a particular decrypt stream. The ENCRYPT$DECRYPT_ONE_RECORD routine allows you to perform such a decryption operation.

> The ENCRYPT$DECRYPT_ONE_RECORD routine is a shorthand form of the ENCRYPT$INIT, ENCRYPT$DECRYPT, and ENCRYPT$FINI sequence of calls. However, using ENCRYPT$DECRYPT_ONE_RECORD repeatedly to decrypt records of a file is extremely inefficient.

> The ENCRYPT$DECRYPT_ONE_RECORD routine returns a 32-bit status code indicating the success or failure of the routine's operation.

## Condition Values Returned

> SS$_NORMAL | Operation performed.
> ENCRYPT$ xyz | An error reported by the Encryption software. xyz identifies the message (see Appendix B).
> SS$_ xyz | A return status from a called system service. xyz identifies the return status.

# ENCRYPT$DEFINE_KEY

> Places a key definition into the process, group, job, or system key storage table.

## Format

> ENCRYPT$DEFINE_KEY key-name ,key-value ,key-flags

## Arguments

> ### key-name

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Name of the key defined when ENCRYPT$DEFINE_KEY completes execution. The key-name argument is the address of a string descriptor pointing to a char_string that is interpreted as the name of the key to be defined. A maximum of 243 characters is permitted.

> Note Key names beginning with ENCRYPT$ are reserved for Compaq.

> ### key-value

> VMS usage: |
> type: | char_string
> access: | read only
> mechanism: | by descriptor

> Key value defined when ENCRYPT$DEFINE_KEY completes execution. The key-value argument is the address of a string descriptor pointing to a vector of unsigned byte values that are assigned to the named key. A maximum of 240 bytes may be assigned.

> ### key-flags

> VMS usage: |
> type: | longword
> access: | read only
> mechanism: | by reference

> Flags that ENCRYPT$DEFINE_KEY uses when defining a key. The key-flags argument is the address of a longword containing flags that control the key definition process.

> Each flag has a symbolic name. The constants associated with these names are defined in the ENCRYPT$EXAMPLES:ENCRYPT_STRUCTURES files in various programming languages. Table 4-3 defines the function of each flag.

> Table 4-3 ENCRYPT$DEFINE_KEY Flags
> Flag | Function
> --- | ---
> Symbolic Name | Function
> ENCRYPT$M_KEY_PROCESS | Places definition in process table
> ENCRYPT$M_KEY_GROUP | Places definition in group table
> ENCRYPT$M_KEY_JOB | Places definition in job table
> ENCRYPT$M_KEY_SYSTEM | Places definition in system table
> ENCRYPT$M_KEY_LITERAL | Stores key without compressing

## Description

> The ENCRYPT$DEFINE_KEY routine places a key definition into the process, group, job, or system key storage table. The key value supplied with the routine is processed as specified and placed in the key storage table under the indicated name. The ENCRYPT$DEFINE_KEY routine does not interpret the key value.

> By default, keys are treated as char_string keys, using the Digital Multinational Character Set and are compressed before being inserted into the key storage table. The compression proceeds as follows:

> 1. The string is converted to uppercase characters.
> 2. The digits 0 through 9 are left unchanged.
> 3. All characters except letters, digits, dollar signs, periods, and underscores are converted to spaces.
> 4. All sequences of multiple spaces (or characters that have been converted into spaces) are converted into single spaces.

> When a char_string key is retrieved from key storage for use as a DES key, it is folded into an 8-byte key by exclusive OR-ing 8-byte segments of the key string together, and then applying odd parity to each byte by modifying the sign bit (bit 7).

> The key flag ENCRYPT$M_KEY_LITERAL specifies that the key string supplied is a binary key. A binary key is not compressed, but is placed into key storage as is. When a binary key is used as a DES key, it is likewise folded into an 8-byte key by exclusive OR-ing 8-byte segments together. Odd parity is then applied by modifying the low bit (bit 0) of each byte.

> The ENCRYPT$DEFINE_KEY routine returns a 32-bit status code indicating the success or failure of the routine's operation.

## Condition Values Returned

> SS$_NORMAL | Key has been defined.
> ENCRYPT$ xyz | An error reported by the Encryption software. xyz identifies the message (see Appendix B).
> SS$_ xyz | A return status from a called system service. xyz identifies the return status.

Previous | Next | Contents | Index