Established BOVPN - One side only sends, Another side only receives #7506
Hi guys, i'm dealing with a really odd situation where an established BOVPN ceases to work, apparently when there's a external link failure in any of three external interfaces on the "Main side". Any blink that would cause any of the three external interfaces to Failed or same on the remote side makes this tunnel stop working.
When this failure happens, main side re-established BOVPN with remote site, but it doesn't send traffic. It does receive it.
On the remote side, it does send traffic, but it doesn't receive. The only workarround that seems to work it's causing an interface failure (unplug ethernet cable, set wrong gateway address or ICMP not responding on multi-wan target) on interface ETH0 at main side.
When we provoke this failure on ETH0, traffic on the tunnels goes up, i mean, sending and receiving. If we stop this simulate failure on ETH0, tunnel keeps working. THIS IS THE ONLY ACTION that make tunnel work again.
I have an XTM25W 12.1.3 OS on the remote side and i had an XTM505 11.10.7 OS on main side that had its whole configuration built from scratch, then we changed the appliance itself to another XTM505 11.12.4 and later to an M200 12.5.11.
Rebooting firewall on remote or main side doesn't make this tunnel to go operational, rekeying tunnel doesn't make tunnel go operational, removing tunnel and/or removing VPN gateway on each side per time or on both sides doesn't make tunnel go operational, rebooting ISP modem on remote side doesn't help.
I don't have configured routes on remote or main side firewalls that would cause any route loop. I tried changing from regular BOVPN to BOVPN Virtual Interface, same stuff happens. Important note that when i set loopback addresses inside BOVPN Virtual Interface for both sides, the Ping between these addresses from firewall worked out. Regular actual tunnel traffic NOT.
I have tried changing the remote side subnet, which is 192.168.9.0/24 to 192.168.10.0/24, but even with a different subnet main side would not be able to send traffic.
When still using XTM5 on main side, i tried making VPN changes on Web Ui, to overcome any glitch on WSM/PM, but that also didn't bring any positive result.
Remote side have other VPN tunnels that work and remain fine.
Main side have other VPN tunnels that work and remain fine.
I tried with TWO MORE different locations, create a VPN HUB
Remote side <> VPN HUB <> Main side
Traffic from networks on VPN HUB and Main side would work, traffic from networks on VPN HUB and remote side would work, traffic from remote side to main side would NOT.
Example
Remote side: 192.168.9.0/24
VPN HUB: 10.10.0.0/24 or 192.168.12.0/24 (SECOND LOCATION for VPN HUB)
Main side: 192.168.0.0/23
I have tried removing all VPN tunnels on remote side and on main side, this tunnel between these two would persist its problem. I know we can set as many VPN Gateways/tunnels we might like, but the active ones will be restricted by license, but by far we are below remote and main side license limitations.
Setting a VPN on remote side to a DIFFERENT location, we are able to re-establish this connection - using same subnets from source and destine.
Setting a VPN on main side to a DIFFERENT location, we are NOT able to re-establish this connection - using same subnets from source and destine. Same issue - doesn't not send, but receive traffic happens.
The global Multi-WAN is set to Failover, gradual failback and it's using interface ETH2 as primary link, then ETH3 and at last ETH0. I don't recall now if i tried changing it to see if traffic would go up or not.
Recently, a new ISP was installed on remote side, same issue happens with this new ISP on remote side. This issue happens whatever gateway pair we set, like:
Scenario A - primary and and failover options
Remote Link A <> Main side Link A
Remote Link A <> Main side Link B
Remote Link A <> Main side Link C
Scenario B - No failover options
Remote Link A <> Main side Link A
Scenario C - No failover options
Remote Link A <> Main side Link B
Scenario D - No failover options
Remote Link B <> Main side Link A
Scenario E - No failover options
Remote Link B <> Main side Link B
In short, any combination of gateway pair doesn't make tunnel to work or stop to making it going failure.
I tried using host-range on VPN tunnel instead of slash anotation, same issue.
Since, setting a simulated remote side on a DIFFERENT firewall to the main side, issue persisted, we narrowed down to something in main side.
We had two main suppositions 1) Crappy old XML 2) VPN BUG.
From XTM5 to M200, i didn't create a new XML file from scratch, i've justed "ported" it, only adapting PBR actions from SD-WAN.
I have checked unused interfaces to see if there was any subnet that would conflict. Example 10.0.2.0/24 for ETH2, although it's a disabled interface, bringing that subnet as default from appliance could cause route issues if you would send traffic to same subnet somewhere else, so when this happened in the past in other WG installs, we'd enable the interface, change its subnet and disable it again.
We have on our office PRTG installed pinging external interfaces for both these sites among others there are from same environment as well as remote probes to ping inside VPN, that's the reason we are sure, other remote places connecting to remote side or main side keeps working when we have a failure between "remote" and "main side".
Important note: This tunnel went working for more than six months straight without problem and the issue started, as far as we know without any network change at the period.
Regards,
Rafael da Costa
Comments
Hi @rafaeldacosta
The first thing I'd try with these older end of life firewalls is to restart the IKE process to see if that makes any change to the issue. You can do that without rebooting the firewall.
See:
(Use the CLI to restart the IKE process)
https://techsearch.watchguard.com/KB/WGKnowledgeBase?lang=en_US&SFDCID=kA10H000000g2wFSAQ&type=Article
The most common reason we see one way data transmission for your ESP traffic is being filtered in one direction.
-If your ISP requires a rule on the upstream device to pass ESP, please ensure that's enabled.
-If your ISP's equipment is using an ESP or IPSEC ALG, please ensure that is disabled.
-James Carson
WatchGuard Customer Support