SD-WAN tunnels drop and require config revert to restore

Summary

SD-WAN IPsec tunnels on FortiGate 60F units are randomly dropping without any clear trigger. The tunnels appear as "up" in the GUI and CLI, but traffic stops passing through. The only reliable workaround is reverting to a previous configuration snapshot via FortiManager.

This issue is affecting multiple branch locations and causing significant business disruption. Sites lose connectivity to corporate resources until manual intervention is performed.

Environment

Symptoms

• SD-WAN health checks pass but application traffic fails
• IPsec tunnel shows Phase 1 and Phase 2 as up
• Routing table shows correct routes via SD-WAN interface
• Packet capture shows traffic entering tunnel but not exiting at hub
• No relevant error messages in FortiAnalyzer logs
• Issue occurs randomly, typically 2-5 days after last config push

Steps to Reproduce

1. Deploy standard SD-WAN configuration via FortiManager
2. Verify tunnels establish and traffic flows correctly
3. Wait 2-5 days with normal traffic patterns
4. Issue manifests without any configuration changes
5. Traffic stops flowing despite tunnels showing as up

Diagnostic Commands

The following commands were run during the outage to gather diagnostic information:

Check SD-WAN member status:

FGT60F # diagnose sys sdwan member
Member(1): interface: wan1, gateway: 203.0.113.1, priority: 0, weight: 1
    Status: alive, vdom: root
    Seq_num: 1, seq_num_oipro: 1
Member(2): interface: wan2, gateway: 198.51.100.1, priority: 0, weight: 1
    Status: alive, vdom: root
    Seq_num: 2, seq_num_oipro: 2

Check IPsec tunnel status:

FGT60F # diagnose vpn ike gateway list name HQ-VPN-01
vd: root/0
name: HQ-VPN-01
version: 2
interface: wan1 5
addr: 203.0.113.50:500 -> 10.1.1.1:500
tun_id: 10.1.1.1/::10.1.1.1
remote_location: 0.0.0.0
network-id: 0
created: 432000s ago
IKE SA: created 1/1  established 1/1  time 0/0/0 ms
IPsec SA: created 1/1  established 1/1  time 0/0/0 ms
  id/spi: 1234 abcd1234efgh5678/ijkl9012mnop3456
  direction: initiator
  status: established 432000-432000s ago = 0ms

Check SD-WAN service rules:

FGT60F # diagnose sys sdwan service
Service(1): Name: Corporate-Apps
    Addr mode: 4
    Protocol: 0
    Quality link: 1
    Member: 0x3 (1,2)
    Alive: 0x3 (1,2)
    selected: 0x1

Check routing table for SD-WAN routes:

FGT60F # get router info routing-table all | grep SD-WAN
S*      0.0.0.0/0 [1/0] via 203.0.113.1, wan1, [1/0]
                  [1/0] via 198.51.100.1, wan2, [1/0]
S       10.0.0.0/8 [10/0] via HQ-VPN-01 tunnel 10.255.1.1, [10/0]
S       172.16.0.0/12 [10/0] via HQ-VPN-01 tunnel 10.255.1.1, [10/0]

Session table shows sessions but no bytes transferred:

FGT60F # diagnose sys session list | grep 10.0.0.50
session info: proto=6 proto_state=01 duration=45 expire=3555
    org=192.168.1.100:52341 -> 10.0.0.50:443
    reply=10.0.0.50:443 -> 192.168.1.100:52341
    bytes=0 packets=0  # <-- No traffic passing!
    policy_id=5 tunnel=HQ-VPN-01

Workaround

The only reliable workaround is reverting the configuration to a previous revision via FortiManager. Simply restarting the VPN or flushing sessions does not resolve the issue.

FortiManager revision revert process:

# From FortiManager CLI:
FMG # execute fmconfig revlist FGT60F-Branch01 | head -10
Revision    Date                 Admin          Comment
--------    ----                 -----          -------
45          2025-12-12 08:30:15  admin          Policy update
44          2025-12-10 14:22:33  admin          Working config
43          2025-12-08 09:15:00  system         Auto-backup

FMG # execute fmconfig revert FGT60F-Branch01 44
This will revert FGT60F-Branch01 to revision 44.
Do you want to continue? (y/n) y
Reverting configuration...
Configuration reverted successfully.

FMG # execute fmconfig install FGT60F-Branch01
Installing configuration to FGT60F-Branch01...
Installation completed successfully.

Alternative: Restart IPsec tunnel (does NOT fix the issue, documented for reference):

FGT60F # diagnose vpn ike gateway flush name HQ-VPN-01
flushed 1 IKE SAs

FGT60F # diagnose vpn tunnel up HQ-VPN-01
tunnel HQ-VPN-01 going up...
tunnel HQ-VPN-01 is up

# Tunnel reconnects but traffic still does not pass
# Issue persists - config revert required

Root Cause Analysis

Investigation is ongoing. Current suspicions based on TAC case #12345678:

• Possible memory corruption in SD-WAN daemon after extended uptime
• Race condition between SD-WAN health checks and IPsec rekey events
• NPU offload table becoming out of sync with routing table
• Known bug in 7.4.3 related to SD-WAN rule evaluation order

Debug output showing potential NPU desync:

FGT60F # diagnose npu np6xlite session-offload list | grep HQ-VPN
index=0x1a2b3c4d ifname=HQ-VPN-01 ptype=1 hw_flag=0x0
    org_src=192.168.1.0/24 org_dst=10.0.0.0/8
    npd_flag=0x80000000  # <-- Invalid flag state
    last_used=0 packets=0 bytes=0

# Clearing NPU offload does not fix the issue:
FGT60F # diagnose npu np6xlite session-offload clear
Clearing all offloaded sessions...
# Traffic still does not pass after clearing

TAC Case Information

Related Issues

• #008 - FGFM tunnel flapping between FortiManager and remote FortiGates
• #002 - Policy package install fails silently on device groups
• #007 - Memory leak in WAD process causes unit to become unresponsive

Temporary Mitigation

Until a permanent fix is available, the following scheduled task can be configured to proactively reboot affected units during maintenance windows:

# Schedule weekly reboot on Sunday at 3:00 AM
FGT60F # config system global
FGT60F (global) # set admin-maintainer enable
FGT60F (global) # end

FGT60F # config system auto-script
FGT60F (auto-script) # edit "weekly-reboot"
FGT60F (weekly-reboot) # set interval 604800
FGT60F (weekly-reboot) # set repeat 0
FGT60F (weekly-reboot) # set start auto
FGT60F (weekly-reboot) # set script "execute reboot"
FGT60F (weekly-reboot) # set output-size 10
FGT60F (weekly-reboot) # next
FGT60F (auto-script) # end