VMware ESXi and vSphere Cluster Management

OSPF MD5 Authentication Configuration and Verification

Learn how OSPF MD5 authentication works, configure it on Cisco IOS, verify Full adjacencies, and troubleshoot key, area, and authentication mismatches.

What OSPF Authentication Protects

OSPF (Open Shortest Path First) is a link-state interior gateway routing protocol. Routers use OSPF control-plane packets to discover neighbors and exchange link-state information.

OSPF authentication helps protect those exchanges from unauthorized participation and altered OSPF packets. A router can use authentication to validate that packets came from a peer using the expected method and credentials.

Authentication is not encryption. It validates participation and packet integrity, but it does not make OSPF routing information confidential. Someone who can observe the link may still be able to read OSPF information.

Adjacent routers must use compatible authentication settings. If one router expects MD5 and the other sends unauthenticated or clear-text-authenticated packets, the OSPF adjacency will not form or will drop.

How OSPF MD5 Message-Digest Authentication Works

MD5 message-digest authentication uses a shared secret and a numeric key ID. The sender calculates a message digest from the OSPF packet and the shared secret, then includes authentication information with the packet. The receiving router uses its matching secret to calculate and validate the expected digest.

If the calculated values agree, the packet can pass authentication. Packets fail authentication when the secret, key ID, authentication mode, or another required neighbor setting is incompatible.

• Shared secret: The password value configured on both neighbors.
• Message-digest key: The interface-level configuration containing a key ID, the MD5 algorithm, and the secret.
• Key ID: A numeric identifier associated with the key. Compatible peers must use an acceptable matching key configuration.
• OSPF adjacency: A fully established neighbor relationship through which routers exchange link-state information.

MD5 is a legacy cryptographic algorithm. It may still be required in older networks, labs, or interoperability scenarios, but it should not be selected as a new general-purpose security design when stronger options are supported. Consider HMAC-SHA authentication for OSPFv2 or an appropriate OSPFv3 authentication mechanism when the OSPF version, platform, software release, and organizational policy support it.

The commands in this lesson use Cisco IOS-style syntax. Exact commands and available authentication options can vary by platform and software release.

OSPF Authentication Methods Compared

Interface-Level MD5 Configuration

With interface-level authentication, the authentication type is enabled directly on each OSPF-enabled interface. The interface must already participate in the intended OSPF process and area.

First configure the message-digest key:

interface <interface-id>
 ip ospf message-digest-key 1 md5 secret

The number 1 is the key ID. The word secret is a lab example of the shared secret. Use a stronger, locally managed secret in a real deployment.

Then enable MD5 authentication on the interface:

interface <interface-id>
 ip ospf authentication message-digest

Configure compatible key and authentication settings on both sides of every OSPF adjacency that should use MD5.

Area-Level OSPF MD5 Authentication

Area-level authentication selects the authentication type for interfaces belonging to a specified OSPF area. Configure it from OSPF router configuration mode:

router ospf <process-id>
 area <area-id> authentication message-digest

This command selects MD5 authentication for the area, but the message-digest key remains associated with each participating interface. Configure a valid key and secret on every OSPF interface in the area that must form authenticated adjacencies.

Without the message-digest keyword, the area command selects clear-text/simple authentication instead:

router ospf <process-id>
 area <area-id> authentication

Avoid designing conflicting authentication policies. For example, applying an area-level MD5 policy while leaving affected interfaces without compatible message-digest keys can prevent multiple adjacencies from forming. Plan area-wide changes and update all affected peers in a coordinated manner.

Two-Router Interface-Level Example

In this example, R1 and R2 are directly connected. Their connected interfaces are addressed, participate in the same OSPF process and area 0, and use the same OSPF network parameters.

The objective is to use MD5 key ID 1 and the shared lab secret secret.

R1 configuration

interface GigabitEthernet0/0
 ip ospf message-digest-key 1 md5 secret
 ip ospf authentication message-digest

router ospf 1
 network 192.0.2.0 0.0.0.255 area 0

R2 configuration

interface GigabitEthernet0/0
 ip ospf message-digest-key 1 md5 secret
 ip ospf authentication message-digest

router ospf 1
 network 192.0.2.0 0.0.0.255 area 0

The interface names and network statements are examples; use the actual connected interfaces and addresses in your topology. The important requirements are that both interfaces participate in the intended OSPF process and area, both enable MD5, and both use compatible key ID and secret values.

After both routers have matching configuration, the adjacency should form or return. A healthy directly connected adjacency normally reaches the Full state when the network type and design require a full adjacency.

Area-Level Example

For a small area with multiple OSPF interfaces, select MD5 authentication for area 0 under the OSPF process:

router ospf 1
 area 0 authentication message-digest

Then configure a compatible key on every relevant interface:

interface GigabitEthernet0/0
 ip ospf message-digest-key 1 md5 secret

interface GigabitEthernet0/1
 ip ospf message-digest-key 1 md5 secret

Repeat compatible configuration on each neighboring router. Verify all affected relationships rather than checking only one interface.

Verification Commands

Verify interface authentication and area

show ip ospf interface GigabitEthernet0/0

Review the output for the interface's OSPF process, area assignment, network type, timers, and authentication type. Look for output indicating message-digest authentication. Output indicating no authentication or simple authentication means the interface is not using the intended MD5 mode.

Verify the neighbor state

show ip ospf neighbor

Confirm that the expected neighbor is present and reaches Full. A different state or a missing neighbor indicates that the adjacency is not fully established.

Inspect interface and process configuration

show running-config interface GigabitEthernet0/0
show running-config | section router ospf

Compare the interface message-digest key, interface-level authentication command, OSPF process, area assignment, and area-level authentication policy on both routers.

Cisco IOS OSPF MD5 Configuration Components

Troubleshooting OSPF Authentication Failures

Start with neighbor and interface status, then compare the complete configuration on both ends. Authentication problems often appear as a missing neighbor or an adjacency that fails to progress to Full.

1. Run show ip ospf neighbor on both routers and record the neighbor state.
2. Run show ip ospf interface <interface-id> on both connected interfaces.
3. Confirm both routers use MD5 rather than mixing MD5, clear-text/simple authentication, and no authentication.
4. Compare the key ID and shared secret. The secret must match exactly, including capitalization.
5. Confirm the interfaces belong to the same OSPF area and intended process.
6. Check baseline OSPF requirements, including IP connectivity, compatible hello and dead timers where applicable, compatible network parameters, and non-passive interfaces.
7. After correcting the configuration, verify that the neighbor reaches Full.

Changing authentication on only one router can immediately interrupt an established adjacency. An area-level change can affect several interfaces and neighbors at once. Coordinate the change so peers receive compatible settings.

Targeted debugging

In a lab or controlled maintenance window, use targeted adjacency debugging to collect evidence:

debug ip ospf adj
undebug all

Debugging can generate substantial output and consume device resources. Use it carefully, collect the relevant evidence, and disable it immediately after troubleshooting.

Authentication Failure Checklist

Example: Mismatched Password

Suppose R1 and R2 both use key ID 1 and area 0, but R1 uses secret while R2 uses different-secret. The routers calculate different digests, so the OSPF packets fail authentication and the adjacency does not form.

Inspect both sides with:

show ip ospf interface GigabitEthernet0/0
show running-config interface GigabitEthernet0/0
show ip ospf neighbor

Correct the inconsistent secret on one router, using the approved shared secret, and then verify the neighbor state again. The adjacency should recover after both sides use matching compatible authentication.

Exam and Operations Notes

• MD5 authentication validates OSPF packets; it does not encrypt routing information.
• The authentication mode must be compatible on adjacent routers.
• The key ID and shared secret are configured at the interface level.
• area <area-id> authentication message-digest selects MD5 for an area.
• area <area-id> authentication, without message-digest, selects clear-text/simple authentication.
• An interface must participate in the intended OSPF process and area before its authentication settings can protect the desired adjacency.
• Use stronger supported authentication options for new designs when platform and policy permit.

For related study, see OSPF MD5 authentication.