Firebox tcp invalid connection state (Internal Policy) issue with smart thermostat
Have a Bosch smart thermostat that isn't working behind a WatchGuard.
Works fine directly connected to the isp modem but when behind a WatchGuard (M500 12.7.2) when the thermostat is attempting the xmpp-client/tcp connection to bosch cloud service the first connection succeeds but following connections have a tcp invalid connection state error with the destination as the Firebox rather than bosch ip and this cause the thermostat to fail to communicate with the bosch cloud service.
Firewall logs:
021-11-23 21:33:13 Allow 192.168.0.15 192.168.0.1 dns/udp 60807 53 Trusted Firebox DNS Forwarding 73 64 (Internal Policy) proc_id="firewall" rc="100" msg_id="3000-0148" record_type="A" question="xmpp.rrcng.ticx.boschtt.net"
2021-11-23 21:33:13 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 50644 5222 Trusted External Allowed 48 127 (TCP-UDP-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 611630185 win 11395"
2021-11-23 21:33:15 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 50644 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 2843393129 win 11395"
2021-11-23 21:33:15 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 50644 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 2843393129 win 11395"
2021-11-23 21:33:15 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 50644 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 2843393129 win 11395"
2021-11-23 21:33:23 Allow 192.168.0.15 139.15.227.124 https/tcp 61990 443 Trusted External Allowed 48 127 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 800259643 win 11395"
2021-11-23 21:33:24 Allow 192.168.0.15 139.15.227.124 https/tcp 61990 443 Trusted External HTTPS Request (HTTPS-proxy-00) proc_id="https-proxy" rc="548" msg_id="2CFF-0000" app_id="0" app_cat_id="0" proxy_act="Default-HTTPS-Client" action="allow" sent_bytes="1855" rcvd_bytes="7939" tls_version="TLS_V12" tls_profile="TLS-Client-HTTPS.Standard" sni="oscar.ticx.boschtt.net" cn="oscar.ticx.boschtt.net" cert_issuer="CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US" cert_subject="CN=oscar.ticx.boschtt.net,OU=TT/EIS5,O=Robert Bosch GmbH,L=Gerlingen,ST=Baden-W\x5cC3\x5cBCrttemberg,C=DE" sig_vers="18.060"
2021-11-23 21:33:24 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 50122 5222 Trusted External IP Request (TCP-UDP-proxy-00) proc_id="tcp-udp-proxy" rc="542" msg_id="2DFF-0000" proxy_act="TCP-UDP-Proxy.Standard" sent_bytes="1668" rcvd_bytes="7581"
2021-11-23 21:33:25 Allow 192.168.0.15 139.15.227.124 https/tcp 61656 443 Trusted External Allowed 48 127 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 1457307596 win 11395"
2021-11-23 21:33:27 Allow 192.168.0.15 139.15.227.124 https/tcp 61656 443 Trusted External HTTPS Request (HTTPS-proxy-00) proc_id="https-proxy" rc="548" msg_id="2CFF-0000" app_id="0" app_cat_id="0" proxy_act="Default-HTTPS-Client" action="allow" sent_bytes="1966" rcvd_bytes="7210" tls_version="TLS_V12" tls_profile="TLS-Client-HTTPS.Standard" sni="http.rrcng.ticx.boschtt.net" cn="http.rrcng.ticx.boschtt.net" cert_issuer="CN=RRCNG_Server,OU=Bosch Thermotechnik GmbH,O=Robert Bosch GmbH,L=Wetzlar,ST=Hessen,C=DE" cert_subject="CN=http.rrcng.ticx.boschtt.net,OU=Bosch Thermotechnik GmbH,O=Robert Bosch GmbH,L=Wetzlar,ST=Hessen,C=DE" sig_vers="18.060"
Comments
To remove the possibility that the TCP-UDP-proxy policy is somehow the cause of this, add a Custom Packet Filter for TCP port 5222.
Then add it as a policy, From: 192.168.0.15 To: Any-external
You can turn on Logging on this policy to see packets allowed by it in Traffic Monitor.
Let us know if this helps or not.
Interestingly, this line shows a completed session 9 seconds after the "tcp invalid connection state" for a different session from 192.168.0.15.
The different source ports indicate different sessions.
2021-11-23 21:33:24 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 50122 5222 Trusted External IP Request (TCP-UDP-proxy-00) proc_id="tcp-udp-proxy" rc="542" msg_id="2DFF-0000" proxy_act="TCP-UDP-Proxy.Standard" sent_bytes="1668" rcvd_bytes="7581"
Hi Bruce
Have previously tried a packet filter on port 5222.
It is odd, it does the initial connection via the port 5222 packet filter then jumps back to the TCP-UDP proxy skipping the 5222 packet filter and then failing to internal policy
2021-11-24 22:22:21 Allow 192.168.0.15 192.168.0.1 dns/udp 58610 53 Trusted Firebox DNS Forwarding 73 64 (Internal Policy) proc_id="firewall" rc="100" msg_id="3000-0148" record_type="A" question="xmpp.rrcng.ticx.boschtt.net"
2021-11-24 22:22:21 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 60849 5222 Trusted External Allowed 48 127 (Port 5222-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 3974930481 win 11395"
2021-11-24 22:22:22 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 55879 5222 Trusted External IP Request (TCP-UDP-proxy-00) proc_id="tcp-udp-proxy" rc="542" msg_id="2DFF-0000" proxy_act="TCP-UDP-Proxy.Standard" sent_bytes="1668" rcvd_bytes="7379"
2021-11-24 22:22:22 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 60849 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 1911791665 win 11395"
2021-11-24 22:22:23 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 60849 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 1911791665 win 11395"
2021-11-24 22:22:23 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 60849 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 1911791665 win 11395"
2021-11-24 22:22:30 Allow 192.168.0.15 139.15.227.124 https/tcp 57319 443 Trusted External Allowed 48 127 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 2907924449 win 11395"
If I disable the TCP-UDP policy and just let it go via Outgoing it does this:
2021-11-24 22:26:29 Allow 192.168.0.15 192.168.0.1 dns/udp 53047 53 Trusted Firebox DNS Forwarding 73 64 (Internal Policy) proc_id="firewall" rc="100" msg_id="3000-0148" record_type="A" question="xmpp.rrcng.ticx.boschtt.net"
2021-11-24 22:26:29 Allow 192.168.0.15 139.15.179.139 xmpp-client/tcp 50137 5222 Trusted External Allowed 48 127 (Port 5222-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 3955949439 win 11395"
2021-11-24 22:26:36 Deny 192.168.0.15 139.15.179.139 xmpp-client/tcp 50137 5222 Trusted Firebox tcp invalid connection state 188 128 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 1892810623 win 11395"
2021-11-24 22:26:39 Allow 192.168.0.15 139.15.227.124 https/tcp 50046 443 Trusted External Allowed 48 127 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="MYEXTERNALIP" tcp_info="offset 7 S 316413883 win 11395"
The only way to understand this is to do packet captures on the firewall, to see what the TCP settings are on the "tcp invalid connection state" packets.
One can do this using TCP Dump. With the Advanced settings, one can specify the source IP addr to capture.
See the TCP Dump section, here:
Run Diagnostic Tasks to Learn More About Log Messages
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/fsm/log_message_learn_more_wsm.html
Video:
https://www.watchguard.com/help/video-tutorials/TCP_Dump/index.html
If you have an active support license on your firewall, you should open a support incident to get help from WG on this.
To me, "tcp invalid connection state" means "Ignore this line!" because I see that all the time on all of my networks and coming from various LAN or VLAN computers, servers, wireless access points, and printers that all are functioning normally. WatchGuard never really has explained why they always fail and why they are trying to hit the Firebox when there is NO policy pointing them at the Firebox.
Gregg Hill
However, the stated problem is that the thermostat doesn’t work, and the only clue in the logs is the error.
Any thoughts on how to get the thermostat to work?
What model thermostat is this?
Exactly what doesn't work when it is behind the firewall?
Also, try adding a HTTPS packet filter From: 192.168.0.15 To: Any-external, and see if the HTTPS proxy is really the issue
Adding a policy for HTTPS packet filter From: 192.168.0.15 To: Any-external still has the same issue.
Even tried a top level Any packet filter From: 192.168.0.15 To: Any-external same issue.
Thermostat is a Bosch EasyControl, one of the annoying ones that needs an internet connection to work as it used a cloud server to allow remote access via an app on phone and it provides no local connection to enable to manage locally with or without internet.
The https just seems to be the initial check in with bosch as it shows up in the bosch portal, but the functionality of controlling it is 5222 via their cloud server.
The iPhone app to cloud server seems to connect successfully via https pulling in the register thermostat id and does 5222 but then the cloud server can't see the thermostat, this does not suffer from the internal policy issue.
My point is that the "this cause the thermostat to fail to communicate" may be just an assumption and not a log of the actual cause. I get the same "tcp invalid connection state" errors when devices work perfectly, so it may be a spurious message that actually has nothing to do with the cause of the failure.
Gregg Hill
Regarding "Even tried a top level Any packet filter From: 192.168.0.15 To: Any-external", I assume that you mean that rule was manually moved above all other policies vs. using auto-ordering of policies. Is that correct?
Did you enable logging of allowed packets on that policy?
A packet filter without Application Control, Geolocation, or IPS should behave like a plain old NAT router with zero outbound filtering.
Speaking of NAT, when you say, "Works fine directly connected to the isp modem", is the modem doing NAT, i.e., you have a dual-NAT setup here where the ISP has a private IP on the inside of its modem and the WatchGuard is on that private network as its WAN side? Or does the WatchGuard have a public IP on its WAN?
Gregg Hill
As an example of why I ignore those messages, the wireless AP at 192.168.16.212 has zero issues communicating with its controller at 10.0.6.101, yet the logs are full of the tcp connection state "error" messages.
I see the same behavior from desktop computers, servers, printers, and time clocks.
2021-11-25 12:14:33 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35186 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 640702557 win 4210" Traffic
2021-11-25 12:15:48 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35188 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 3280652196 win 4210" Traffic
2021-11-25 12:15:49 Deny src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35186 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AF 3628226653 win 25607" Traffic
2021-11-25 12:15:58 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35190 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 3634166933 win 4210" Traffic
2021-11-25 12:17:26 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35192 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 391000492 win 4210" Traffic
2021-11-25 12:17:27 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35194 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 653889591 win 4210" Traffic
2021-11-25 12:17:29 Deny src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35190 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AF 2578447509 win 25607" Traffic
2021-11-25 12:17:53 Deny src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35190 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AF 2578447509 win 25607" Traffic
2021-11-25 12:18:20 Deny src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35190 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AF 2578447509 win 25607" Traffic
2021-11-25 12:18:57 Deny src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35192 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AF 1750151596 win 25607" Traffic
2021-11-25 12:18:57 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35196 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 2345842926 win 4210" Traffic
2021-11-25 12:19:04 Allow src_ip=192.168.16.212 dst_ip=10.0.6.101 pr=webcache/tcp src_port=35198 dst_port=8080 src_intf=VLAN1-PrivateLAN dst_intf=VLAN6-CloudKey msg=Allowed pckt_len=60 ttl=63 policy=(UniFi-ControllerPlusMgmt.In-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" tcp_info="offset 10 S 1097882663 win 4210" Traffic
Gregg Hill