IP Security (IPsec)

IP security (IPsec) defines security authentication and encryption services that can be applied to network traffic at the IP layer. The IPsec architecture is defined in RFC 4301, and other supplemental information is defined in RFCs 4302, 4303, and 4835.

When IPsec is enabled, all IPv6 traffic must conform to policies that describe the processing performed on each packet. On some system types, IPv4 traffic is also supported. In the MCP environment, these IPsec policies are defined by Security Center (through the MCP Security Policy Management module) and kept in the Security Center database. This list of policies is an ordered list (the IPsec module picks the first policy for which the packet matches the traffic description of the policy). An IPsec policy can be fine-grained or coarse-grained, depending on the description of the traffic that the policy governs (the traffic selector).

IPsec policies are one-directional (either incoming or outgoing), so IPsec policies need to be created for both inbound and outbound IP communications. For outbound packets, policies define rules under which packets can be transmitted to the network. For inbound packets, the Security Center database dictates the circumstances under which the packet can be accepted by the host for possible IPsec processing.

Note: If you enable IPsec by using the NW TCPIP OPTION command, then you must have at least one policy defined for each direction; otherwise, all traffic is discarded by default.

Each IPsec policy uses local IP and remote IP address ranges, upper layer protocol identifiers, and optionally, port numbers to describe a range of packets to which it applies. Each policy also specifies a disposition, which describes the processing that should be done. The dispositions are

  • BYPASS

    Specifies that each packet is allowed to bypass IPsec protection (that is, to be transmitted or received unencrypted).

  • DISCARD

    Specifies that each packet is discarded (that is, not allowed to be transmitted or received unencrypted).

  • PROTECT

    Specifies that each packet is protected as defined in the matching policy using IPsec services. These services are authentication (integrity checking) and encryption (confidentiality).

When using the PROTECT disposition, you can also specify one of the two IPsec security services to be performed. The services are

  • AH (Authentication Header)

    AH is a strong form of authentication, which covers the IP header as well as the data. AH is defined by RFC 4302.

    The Upper Layer Protocol (ULP) number (the protocol identifier in the IP header) is 51.

  • ESP (Encrypted Security Payload)

    ESP provides both authentication services (called ESP Integrity) and encryption services (called ESP Confidentiality), but differently than AH. ESP Integrity just covers the data (also called the payload) but does not cover the IP header. ESP Confidentiality provides for encryption of the payload data. ESP is defined by RFC 4303.

    The ULP number is 50.

Each IP packet in the MCP implementation can have either an AH or ESP IPsec policy defined, but not both. An ESP IPsec policy can define both ESP Integrity and ESP Confidentiality services.

MCP IPsec supports

  • Manual keying, which requires you to manually create the keys on each system and exchange keys between systems manually before IPsec can be used

  • Point-to-point IPsec policies, which require one policy to be defined between each set of systems that wish to use IPsec services

  • Transport mode (Tunnel mode is not supported.)

  • HMAC-SHA1-96 algorithm for authentication (both AH and ESP Integrity services)

    Note: Some systems also support HMAC-SHA-256 and HMAC-SHA-512.

  • AES (AES-CBC) algorithms for encryption (ESP Confidentiality service)

    Note: Some systems also support AES-CBC with 256-bit keys.

  • IPsec for IPv6 traffic for all MCP systems. The following systems support IPsec for IPv4 traffic:

    • ClearPath MCP Software Series machines running MCP 20.0 or later and ClearPath MCP Software Series Release 4.0 (any model)

    • Libra 4500/6500/8500, Libra 4501/6501/8501, FS800, and later systems

Configuring IPsec

To use IPsec between hosts, perform the following tasks. These tasks are described in more detail in the following topics.

  1. Create IPsec keys through the MCP Cryptographic Services Manager module of Security Center. These keys can be generated, defined by the user, or imported from other systems. When using user-defined keys, both hexadecimal and ASCII input is accepted.

  2. Create IPsec policies through the MCP Security Policy Management module of Security Center, or import the policies from other systems.

  3. Test the IPsec policies to ensure that the desired processing adheres to the intended policies.

  4. Commit the policies to the MCP environment.

  5. Enable IPsec in the MCP environment.

    Examine the SUMLOG for any discarded packets to ensure that the policies are performing as intended.

Creating IPsec Keys

IPsec uses symmetric keys as part of the input into the cryptographic algorithms used for protecting the data. The type of the key must agree with the IPsec algorithm to be used for protecting data. For example, a SHA-1 key must be used when using HMAC_SHA1_96 for authentication. The key (name and value) must be exactly the same on both the sending and receiving sides of the IPsec policy.

Use the following procedure to create an IPsec key:

  1. Initiate Security Center and select the MCP Cryptographic Services Manager module. This process assumes that the Security Center database has already been initialized.

  2. Select Trusted Keys, and then select IPsec Keys.

  3. Right-click and choose Create a Key.

    The Create IPsec Key dialog box is displayed.

  4. Enter the key name to be used for this key. (IPsec policies use key names rather than actual values like other types of systems.)

  5. Enter the algorithm to be used for the key (for example SHA-1, TDES, or AES).

  6. Do one of the following:

    • If this key is to be used between MCP systems, choose Generate.

      This action generates a random key for IPsec, which can be exported and imported to the other system.

    • If this key is to be used with other system types (such as Linux), choose User-Defined, which allows the user to enter the key in either hexadecimal or ASCII format. The User-Defined option is preferred for other system types because other systems do not support the importation of IPsec keys. Instead, most use a command-line interface.

  7. If an IPsec key is to be used on other MCP systems, export the key to a file and import that file on the other MCP systems. For more information about importing and exporting keys, refer to the Security Center Help.

Creating IPsec Policies

IPsec policies are created through the Security Policy Management module of Security Center. These policies are created through the Update IPsec Policies wizard, which enables you to create, modify, and import IPsec policies.

If you import IPsec policies from other systems, modify them before committing the policies to the MCP host. For example, swap local and remote IP addresses or modify the direction of traffic. Ensure that the source and destination IP addresses are correct for this MCP environment.

Note: ICMPv6 traffic (at least Neighbor Solicitation and Advertisement messages and Router Solicitation and Advertisement messages) should always bypass IPsec processing to allow for normal IPv6 operation. If these traffic types are not allowed through, the IPv6 node cannot be reached from the network. It is not recommended that you apply IPsec processing to these traffic types, since they are not usually protected.ICMPv6 traffic

If you use IPsec for IPv4, ICMP traffic should always bypass IPsec processing.

Example Policies

Adapt the following example policies to your local security policy.

Policies 1 and 2

Purpose: Allow ICMPv6 traffic to bypass IPsec processing.

In these examples, the optional local and remote selectors can be used to describe the IP addresses that are used as names by the policy. 

Policy Name = ALLOWICMPV6OUT
Local IP address = ANY
Remote IP address = ANY
Next Layer Protocol = ICMPv6
Local Types/Codes = ANY
Direction = OUTBOUND
Disposition = BYPASS
Policy Name = ALLOWICMPV6IN
Local IP address = ANY
Remote IP address = ANY
Next Layer Protocol = ICMPv6
Local Types/Codes = ANY
Direction = INBOUND
Disposition = BYPASS

Policies 3 and 4

Purpose: Use AH for all BNA (BIP) traffic between this host and a backup host.

These policies

  • Use the same key for both inbound and outbound policies, that is, the corresponding host has the same configuration. Inbound and outbound policies could use different keys if the corresponding policies on the remote host also use different keys.

  • Require that the SPI values must be unique on each MCP host.

  • Require that the remote IP address only specify one remote system because the MCP only supports point-to-point policies. 

Policy Name = PROTECTBNAOUT
Local IP selector = PRODUCTION
Local IP address = ANY
Remote IP selector = DRBACKUP
Remote IP address = FEC0::2222:A00:BFF:FE3A:127
Next Layer Protocol = BIP
Direction = OUTBOUND
Disposition = PROTECT
SPI = 2001
Protocol = AH
Mode = TRANSPORT
Integrity Key Name = BNAAHKEY
Integrity Algorithm = HMAC_SHA1_96
Policy Name = PROTECTBNAIN
Local IP selector = PRODUCTION
Local IP address = ANY
Remote IP selector = DRBACKUP
Remote IP address = FEC0::2222:A00:BFF:FE3A:127
Next Layer Protocol = BIP
Direction = INBOUND
Disposition = PROTECT
SPI = 2002
Protocol = AH
Mode = TRANSPORT
Integrity Key Name = BNAAHKEY
Integrity Algorithm = HMAC_SHA1_96

Policies 5 and 6

Purpose: Use ESP Confidentiality for FTP traffic between this host (FTP client) and the backup host (FTP server).

These policies

  • Show an example Security Policy Index (SPI) that uses different keys for the outbound and inbound directions. The corresponding policies on the DRSYSTEM must match, and the keys must be present on both systems.

  • Require that the values between source and destination systems match, especially if the destination is another system type. Some systems assume a hexadecimal format, while the MCP assumes a decimal format.

  • Assume that FTP has been configured to also be the server on the data connection (known as PASV mode). Otherwise, the data connection on port 20 is in the incoming direction. 

Policy Name = PROTECTFTPOUT
Local IP Address = ANY
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Next Layer Protocol = TCP
Local Ports = ANY
Remote Ports = 20-21
Direction = OUTBOUND
IPSec Action = PROTECT
Security Policy Index = 24577
Protocol = ESP Confidentiality
IPsec Mode = TRANSPORT
Confidentiality Key Name = PRODUCTIONKEY
Confidentiality Algorithm = AES_CBC
Policy Name = PROTECTFTPIN
Local IP Address = ANY
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Next Layer Protocol = TCP
Local Ports = ANY
Remote Ports = 20, 21
Direction = INBOUND
IPSec Action = PROTECT
Security Policy Index = 24578
Protocol = ESP Confidentiality
IPsec Mode = TRANSPORT
Confidentiality Key Name = DRSYSTEMKEY
Confidentiality Algorithm = AES_CBC

Policies 7 and 8

Purpose: Allow all other traffic to bypass IPsec processing for normal operation.

If the other traffic is required to be discarded rather than bypassed, the last two policies can be deleted.

From the main screen of the Update IPsec Policies wizard, you can create, modify, and import policies that have been saved from other systems. 

Upper Layer Provider = ANY
Direction = OUT
Disposition = BYPASS
Local IP address = ANY
Remote IP address = ANY
Upper Layer Provider = ANY
Direction = IN
Disposition = BYPASS
Local IP Address = ANY
Remote IP Address = ANY

Testing IPsec Policies

You must test the IPsec policies before deployment. Testing is necessary to ensure that the IPsec policies do not affect network access as the default action of IPsec is to discard any pockets that do not explicitly match a policy.

IPsec Testing is part of the Update and View IPsec Policies wizards.

Use the following procedure to create an IPsec policy test:

  1. Right-click the IPsec Policy node in the scope pane, and then click Update or View IPsec Policies.

    The IPsec Policy Wizard Welcome screen appears.

  2. Click Next.

    The IPsec Policies dialog box appears.

  3. Click Next.

    The Test IPsec Policies dialog box appears.

  4. Enter sample values for the following network parameters to test the IPsec policies list:

    • Unique Test Name

    • Upper Layer Protocol (TCP, UDP, ICMP, ICMPv6, or BIP)

    • Direction (IN or OUT)

    • Local IP address

    • Remote IP address

    • Local Port or ICMP Type/Code Range

    • Remote Port Range

    Note: Only one port type range (either local or remote) is supported.

    For example, enter the following test values:

    Test Name = TEST_PROTECTFTP
ULP = TCP
Direction = IN
Local IP Address = FEC0::2222:A00:BFF:FE3A:100
Local Port or ICMP Type/Code = 500
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Remote Port = ANY

  5. After specifying the values, click Test.

    The IPsec Test Results dialog box appears.

    For example, using the Example Policies 1 through 8 from the previous section and the test example from step 4, the IPsec Test Wizard displays the following three test result entries in the table on the IPsec Test Results dialog box:

    • The first result is for local port 500: the remote ports 1 to 19 are set to BYPASS

    • The second result is for local port 500: the remote ports 20 to 21are set to PROTECT

    • The third result is for local port 500: remote ports 22 to 65535 are set to BYPASS

    The test results are displayed as follows:

    Test Result for Test = TEST_PROTECTFTP
ULP = TCP
Direction = IN
Test Result = BYPASS
Local IP Address = FEC0::2222:A00:BFF:FE3A:100
Local Port or ICMP Type/Code = 500
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Remote Port = 1 - 19
First Matched Policy = BYPASSALLIN
Matching Bypass Polices = BYPASSALLIN
Matching Discard Polices = None
Matching Protect Polices = None
Test Result for Test = TEST_PROTECTFTP
ULP = TCP
Direction = IN
Test Result = PROTECT
Local IP Address = FEC0::2222:A00:BFF:FE3A:100
Local Port or ICMP Type/Code = 500
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Remote Port = 20 - 21
First Matched Policy = PROTECTFTPIN
Matching Bypass Polices = BYPASSALLIN
Matching Discard Polices = None
Matching Protect Polices = PROTECTFTPIN
Test Result for Test = TEST_PROTECTFTP
ULP = TCP
Direction = IN
Test Result = BYPASS
Local IP Address = FEC0::2222:A00:BFF:FE3A:100
Local Port or ICMP Type/Code = 500
Remote IP Address = FEC0::2222:A00:BFF:FE3A:127
Remote Port = 22 - 65535
First Matched Policy = BYPASSALLIN
Matching Bypass Polices = BYPASSALLIN
Matching Discard Polices = None
Matching Protect Polices = None

    These IPsec test results indicate that the FTP traffic is allowed to bypass IPsec protection for the remote ports 1 - 19 and 22 - 65535. While the FTP traffic for the remote ports 20 - 21 matched bypass policy, it was protected as defined in the first matching PROTECTFTPIN policy.

    Note: As the order of the IPsec policies in the list is crucial to set the desired traffic control, the results of the IPsec test show all the matching policies as well as the one that will be used.

  6. Double-click on the specific test result entry to display matching IPsec policies lists.

    The Matching Policies dialog box appears.

    This dialog box displays all matching IPsec policies for one test result entry in three separate lists (DISCARD, PROTECT and BYPASS). The first matching policy is marked as <FIRST MATCH>.

  7. Double-click on any policy name to display additional information about the specific IPsec policy and test.

    The IPsec Property page appears.

    You can switch between property pages in Update IPsec Policies Wizards or go back to add, modify, clone, delete, or reorder IPsec policies. You can thus test the updated IPsec policies list before it is committed to the ClearPath MCP database.

    The IPsec tests and tests results are discarded upon the completion of the View & Update IPsec Policies Wizards. However, for later reference, you can store the IPsec test summary locally on a Windows drive and on the MCP server in a text file.

Committing the Policies to the MCP Environment

Once you have tested the IPsec policies, click Finish in the Update IPsec Policies wizard to commit the policies to the Security Center database and into the TCP/IP Networking Provider. Any errors found in the IPsec policies are displayed.

If there are no errors in the commit process, enable IPsec using the following command: 

NW TCPIP OPT + IPSEC

This command enables the policies and enforces IPsec processing.

On systems that support IPsec for both IPv4 and IPv6, you should verify the setting of the IPSECDefault using the following command: 

NW TCPIP OPT

The default setting for IPSECDefault is IPv6Only. IPv6Only controls the behavior of TCPIP when processing inbound frames when no IPsec policy applies. For more information about the other available settings for IPSECDefault, such as BYPASS, DENY, and DENYALL, refer to the Networking Commands and Inquires Help.

Inquiring About IPsec Policies

Display the list of IPsec policies by using the NW TCPIP STATUS IPSEC ALL command on the ODT. Refer to “IPsec” under Troubleshooting the Security System Layers for the syntax of the NW TCPIP STATUS command. You can also use the Security Center to view IPsec policies.

IPsec Best Practices

Follow these recommendations for enabling and using IPsec:

  • Enabling IPsec causes more processor overhead for traffic subject to PROTECT policies. If the performance impact of protecting all traffic too high, you should consider creating policies that apply BYPASS mode to traffic that does not contain sensitive data.

  • Ensure that the ports to be protected with IPsec are not also protected by SSL/TLS or another network security protocol (for example, port 443 for HTTPS). If certain traffic is secured by SSL/TLS, one approach would be to allow this traffic to bypass IPsec processing and then allow all other traffic to be protected by a later policy.

  • Ensure that ICMPv6 messages are set to BYPASS. Or, at minimum, set to BYPASS the messages required for IPv6 normal operation. If ICMPv6 messages are not allowed to be bypassed, the MCP host is unreachable from and to the IPv6 network.

  • Ensure that ICMP messages are set to BYPASS for systems that use IPsec for IPv4.

  • Order is important. The first policy that the IP packet matches is used. Testing is very important to ensure correct operation. If there is no match, the IP packet is discarded.

  • If you are experiencing problems communicating with your new IPsec policies, use a packet analyzer to help identify the issue.

Prev  Up  Next
Setting Up a Trusted Store  Home  Secure Shell (SSH)