Tracert results
I hope someone can shed some light on this one....
I upgraded my M470 to latest firmware from the cloud and it stopped responding.
I did this to 8 other watchguards and none of them had an issue.
I arrived on site the next day to find no internet but things looked fine otherwise (BOVPN, Vlans etc all worked as expected). I rolled WG back and restored from backup and still same issue. I had my service provider flush the ARP on his Cisco (internet router) and things came up so i upgraded WG and thought that was weird and off I went.
Now Here is my new problem. I cannot email a particular server (rcmp 199.212.150.8).
Everything is showing deferred on my barracuda, so i started to investigate.
Doing a tracert to 199.212.150.8 over 30 hops shows my WG as hop1 and then nothing. It should be going to next hop in this case the gateway of my external interface. I can't figure out what the heck is going on. I even went so far as to add a static route on the WG dumping 199.212.150.8 to the external gateway but tracert stays same.
I have a ticket in to WG, but if anyone has any insight I would appreciate it!
Comments
Sounds like an ISP issue to me.
Contact them
I had thought that as well, but shouldn't the firewall at the very least know the next hop is the external gateway? Even if it tried and failed....
The 2nd hop tracert response should be an IP addr outside your firewall, usually an ISP device
Yes, that should be the gateway of the External connection (Bell). This is exactly what I would expect to see, but when I do a tracert from the firewall it fails immediately. The IP of the gateway does not come up as 1st hop at all.
You can use TCP DUMP on the firewall to capture your tracert packets to that IP addr.
See what that shows.
You can do this using FSM or the Web UI.
Here is the link for FSM:
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/fsm/log_message_learn_more_wsm.html
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:27:10.682004 IP X.X.X.X.48016 > X.X.X.X.25: Flags [S], seq 2784157583, win 29200, options [mss 1460,nop,nop,TS val 35525194 ecr 0,nop,wscale 7], length 0
15:27:10.721701 IP X.X.X.X.48018 > X.X.X.X.25: Flags [S], seq 2915600310, win 29200, options [mss 1460,nop,nop,TS val 35525203 ecr 0,nop,wscale 7], length 0
15:27:10.788794 IP X.X.X.X.48020 > X.X.X.X.25: Flags [S], seq 1780422151, win 29200, options [mss 1460,nop,nop,TS val 35525220 ecr 0,nop,wscale 7], length 0
15:27:10.905714 IP X.X.X.X.48022 > X.X.X.X.25: Flags [S], seq 3908309551, win 29200, options [mss 1460,nop,nop,TS val 35525249 ecr 0,nop,wscale 7], length 0
15:27:11.698085 IP X.X.X.X.48016 > X.X.X.X.25: Flags [S], seq 2784157583, win 29200, options [mss 1460,nop,nop,TS val 35525448 ecr 0,nop,wscale 7], length 0
15:27:11.729888 IP X.X.X.X.48018 > X.X.X.X.25: Flags [S], seq 2915600310, win 29200, options [mss 1460,nop,nop,TS val 35525456 ecr 0,nop,wscale 7], length 0
15:27:11.793808 IP X.X.X.X.48020 > X.X.X.X.25: Flags [S], seq 1780422151, win 29200, options [mss 1460,nop,nop,TS val 35525472 ecr 0,nop,wscale 7], length 0
15:27:11.921885 IP X.X.X.X.48022 > X.X.X.X.25: Flags [S], seq 3908309551, win 29200, options [mss 1460,nop,nop,TS val 35525504 ecr 0,nop,wscale 7], length 0
15:27:13.713862 IP X.X.X.X.48016 > X.X.X.X.25: Flags [S], seq 2784157583, win 29200, options [mss 1460,nop,nop,TS val 35525952 ecr 0,nop,wscale 7], length 0
15:27:13.745859 IP X.X.X.X.48018 > X.X.X.X.25: Flags [S], seq 2915600310, win 29200, options [mss 1460,nop,nop,TS val 35525960 ecr 0,nop,wscale 7], length 0
15:27:13.809860 IP X.X.X.X.48020 > X.X.X.X.25: Flags [S], seq 1780422151, win 29200, options [mss 1460,nop,nop,TS val 35525976 ecr 0,nop,wscale 7], length 0
15:27:13.937852 IP X.X.X.X.48022 > X.X.X.X.25: Flags [S], seq 3908309551, win 29200, options [mss 1460,nop,nop,TS val 35526008 ecr 0,nop,wscale 7], length 0
15:27:17.809727 IP X.X.X.X.48018 > X.X.X.X.25: Flags [S], seq 2915600310, win 29200, options [mss 1460,nop,nop,TS val 35526976 ecr 0,nop,wscale 7], length 0
15:27:17.809768 IP X.X.X.X.48016 > X.X.X.X.25: Flags [S], seq 2784157583, win 29200, options [mss 1460,nop,nop,TS val 35526976 ecr 0,nop,wscale 7], length 0
15:27:18.065716 IP X.X.X.X.48022 > X.X.X.X.25: Flags [S], seq 3908309551, win 29200, options [mss 1460,nop,nop,TS val 35527040 ecr 0,nop,wscale 7], length 0
15:27:18.065720 IP X.X.X.X.48020 > X.X.X.X.25: Flags [S], seq 1780422151, win 29200, options [mss 1460,nop,nop,TS val 35527040 ecr 0,nop,wscale 7], length 0
15:27:18.998171 IP X.X.X.X.48100 > X.X.X.X.25: Flags [S], seq 2418123142, win 29200, options [mss 1460,nop,nop,TS val 35527273 ecr 0,nop,wscale 7], length 0
15:27:20.017732 IP X.X.X.X.48100 > X.X.X.X.25: Flags [S], seq 2418123142, win 29200, options [mss 1460,nop,nop,TS val 35527528 ecr 0,nop,wscale 7], length 0
15:27:22.033684 IP X.X.X.X.48100 > X.X.X.X.25: Flags [S], seq 2418123142, win 29200, options [mss 1460,nop,nop,TS val 35528032 ecr 0,nop,wscale 7], length 0
15:27:26.257704 IP X.X.X.X.48100 > X.X.X.X.25: Flags [S], seq 2418123142, win 29200, options [mss 1460,nop,nop,TS val 35529088 ecr 0,nop,wscale 7], length 0
15:30:08.993469 IP X.X.X.X.26944 > X.X.X.X.25: Flags [S], seq 1105149267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 369173580 ecr 0], length 0
15:30:08.993531 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:10.017803 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:12.067673 IP X.X.X.X.26944 > X.X.X.X.25: Flags [S], seq 1105149267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 369176630 ecr 0], length 0
15:30:12.067685 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:14.097803 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:15.268868 IP X.X.X.X.26944 > X.X.X.X.25: Flags [S], seq 1105149267, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 369179860 ecr 0], length 0
15:30:15.268898 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:19.431136 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
15:30:27.537801 IP X.X.X.X.25 > X.X.X.X.26944: Flags [S.], seq 3736888810, ack 1105149268, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
30 packets captured
30 packets received by filter
0 packets dropped by kernel
What do the x.x.x.x represent?
What is the 25 in this x.x.x.x.25 mean? Dest port?
Perhaps this is SMTP?
The following will capture ICMP packets - ping, tracert, replies etc. on eth0 - using the Advanced Option Arguments:
-i eth0 icmp
The following will capture packets for 1.1.1.1 on eth0
-i eth0 host 1.1.1.1
Dumped traffic and inspected with wireshark to find the MAC of the next hop was a Juniper device. I went back to ISP and told them what i found and they investigated logs to find some irregularities. They issued an emergency release order early this morning for a device reboot and issue is now resolved. They gave me this long spiel about affecting 85 corporate customers but in the end the issues were only getting worse as I documented another domain that was having routing issues just prior to the reboot.
All is well now, thanks for your help and Watchguards support.
I hadn't even realized I could do a tcp dump from FSM.
Awesome. Thanks.