12.5.1 is GA

Thought I would post on the community that 12.5.1 is now available for download.
Proper Product and Support blog post will come out later this week.

George Grinnell
WatchGuard Representative

Comments

  • Install of WSM 12.5.1 on top of 12.5 does not ask what components one wants to be installed.
    Is this expected?

  • I just witnessed a MIRACLE!!!!

    A client's T10 upgraded from 12.5 U1 to 12.5.1 ON THE FIRST TRY without getting the dreaded low-memory error it usually gets.

    I am STUNNED.

    Gregg

    Gregg Hill

  • EGAD! I just upgraded my T35-W, and now the AuthPoint push for logging into the support site fails to show up on my phone when its on my wireless (UniFi access point). It worked fine before on 12.5 U1 firmware.

    UPDATE: It took about 10 minutes after the upgrade, and now the AuthPoint push works again with my phone on my wireless. You had me scared for a bit!

    Gregg

    Gregg Hill

  • Ever since installing 12.15.1 last night, I have multiple internal systems that show Deny in FSM traffic monitor with "msg=tcp invalid connection state" in the line. One is a UniFi wireless access point, one is a Windows 2019 server, one is my Moto G7 cell phone on Android 9, and one is a Roku on its own VLAN. I never saw these in my logs before, and I do not have TCP SYN Checking enabled.

    Restarting the UniFi AP or the Roku, or disabling/re-enabling wireless on my cell phone, results in the denies going away...for a while. After a few minutes, they start showing up again.

    Is anyone else seeing these "tcp invalid connection state" in your logs?

    Gregg

    Gregg Hill

  • edited September 2019

    Looks like a new "reason" message.
    I'm seeing them for what looks like mostly slow HTTPS replies - source port = 443, TCP
    All are Internal Policy denies from External to Firebox.

    Over the last number of releases I have seen lots of TCP source port = 443 denied in my logs. I have never found a reasonable explanation for them

  • Mine are internal devices: the UniFi AP on my LAN reporting back to my Cloud Key controller on its own VLAN on TCP 8080, or internal going to various external IPs on TCP 443 (several AWS IPs, plus others). I just noticed some incoming with the same message, with random high-numbered ports such as 35246, 10983, 36145, etc.

    Gregg Hill

  • The vast majority are the UniFi AP to the Cloud Key UniFi controller. Everything works fine on the wireless and in the UniFi controller; it's just the odd messages.

    Sometimes 8080 hits fine with an Allow, then there will be either one, two, three, four, or five times it shows Deny, followed by another Allow, rinse and repeat.

    Gregg Hill

  • Could be related to this "fix" in the V12.5.1 Release Notes:
    • The Firebox now generates meaningful log messages when it drops packets for a connection it is not aware of. [FBX-16424]

  • Hi, Bruce! It's not just that it a different message; it's a new problem with denying the packets that used to always go through. For example, I just looked, and there were seven lines in FSM traffic monitor from the IP of the UniFi AP. Two denies, one allow, one deny, one allow, two denies. Previously, these all would be Allow lines.

    I restarted the UniFi AP (I am testing with it only because this device is the easiest to test, although other devices have the same issue...Roku, etc.) and its first 8080 line was an Allow at 11:56:27, then Allows through 11:57:43, then the first Deny at 11:59:09, at which point it started the cycle of Allow/Deny again.

    Gregg Hill

  • without logging allowed packets or better yet doing a packet capture, one can't really say that this is a new problem or not.
    Perhaps the packets now shown with this new message were dropped in previous releases and there just was no log message to indicate it.

  • "Previously, these all would be Allow lines." I was already logging allowed traffic.

    The number of lines from this AP and the number for UniFi APs I have at client sites that phone home to my controller are nearly identical, expect that now the majority from this one AP are deny messages.

    I never saw the same ports get denied from the Roku, my server, or others I noted until this build.

    Gregg Hill

  • It is not clear from FBX-16424 if previously the log messages were not meaningful, or if XTM was not always displaying a log message when a packet got dropped, or both.

    The ones that I see are mostly for incoming packets, and mostly for what seems like replies to outgoing HTTPS sessions.
    There is 1 example:
    2019-09-12 16:11:54 Deny 17.248.137.141 xxx.xxx.xxx.xxx 48734/tcp 443 48734 0-External Firebox Denied 40 53 (Unhandled-External-Packets-00)
    2019-09-12 16:11:54 Deny 17.248.137.141 xxx.xxx.xxx.xxx 48734/tcp 443 48734 0-External Firebox tcp invalid connection state 40 53 (Internal Policy)
    2019-09-12 16:11:54 Deny 17.248.137.141 xxx.xxx.xxx.xxx 48734/tcp 443 48734 0-External Firebox tcp invalid connection state 40 53 (Internal Policy)

  • Sorry, Bruce, I seriously need some sleep. Something you mentioned a few posts back FINALLY clicked in my head...the destination interface is "Firebox" on each of the Deny messages, whether an internal source IP or external source IP!

    Each of the internal "tcp invalid connection state" Deny messages is when a device on my network tries to reach an IP on a different VLAN or an external IP that they normally communicate with just fine, and suddenly they pop in a few of those Deny lines to the Firebox, and then go off to the same targeted IP that just got denied and get the expected Allow line.

    Here are two lines from my Roku:

    2019-09-12 13:46:33 Deny src_ip=10.0.3.101 dst_ip=52.40.80.32 pr=https/tcp src_port=38224 dst_port=443 src_intf=3-VLAN3-CellPhonesTabletsRoku dst_intf=Firebox msg=tcp invalid connection state pckt_len=52 ttl=64 policy=(Internal Policy) proxy_action= proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AR 4082266954 win 176" Traffic

    2019-09-12 13:46:33 Allow src_ip=10.0.3.101 dst_ip=52.40.80.32 pr=https/tcp src_port=38224 dst_port=443 src_intf=3-VLAN3-CellPhonesTabletsRoku dst_intf=0-External msg=Allowed pckt_len=83 ttl=63 policy=(UnrestrictedAccess.Out-00) proxy_action= proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="172.112.192.232" tcp_info="offset 8 A 4082266923 win 176" geo_dst="USA" Traffic

    In each case, it tries to reach dst_ip=52.40.80.32, BUT the interface it tries to reach is "Firebox" on the Deny, and it's "0-External" on the Allow lines.

    For the UniFi AP, its destination interface is normally "6-VLAN6-CloudKey" but it's "Firebox" on the Deny lines.

    Too much in my brain now. Need Starbucks!

    Gregg Hill

  • What I don't get is why the Firebox suddenly becomes the target interface, when the packets are clearly sent to a particular IP address either external or on a VLAN. Also, why would it get an Allow to a particular IP address either external or on a VLAN and then Deny some to Firebox, then go back to Allow, then Deny again, rinse and repeat, all day long?

    Gregg Hill

  • Firebox is an alias for any firewall interface.
    My guess that this is hard coded as Firebox for the generation of the deny message.
    The log messages do not show any of the TCP flags on the packet and without that info it is hard to tell if the deny is justified and if the packet is different from the allowed packet.

  • Looks like a job for something like WireShark..

    Adrian from Australia

  • Looks like a job for Superman!

    Gregg Hill

  • I am opening a case for it now.

    Gregg Hill

  • Case - 01282162 has been opened, verified, and escalated.

    Gregg Hill

  • And?

    Adrian from Australia

  • And...crickets. It's a low priority to fix the utterly useless "tcp invalid connection state" messages that take up space in my FSM traffic monitor log. If I see a deny message, it should mean something.

    Gregg Hill

  • Adrian,

    In September 2019, James Carson created a feature request to disable showing those messages. No news since then.

    Gregg Hill

  • So the bottom line is that the messages are not important and you can turn them off? Where is the switch? I was going to say that I get these all the time, but after looking at the traffic monitor this morning I can't see any. :#

    Adrian from Australia

  • Here it is - it didn't take long and the only place I have been this morning is watcgguard.com and watchguard.centercode.com

    These are the original packets using port 52955:
    2020-05-25 10:11:40 Allow 10.0.10.7 142.250.66.234 https/tcp 52955 443 1-Trusted-WIRED 0-External HTTPS Request (HTTPS-proxy-00) proc_id="https-proxy" rc="548" msg_id="2CFF-0000" app_id="0" app_cat_id="0" proxy_act="HTTPS-Client.Standard.corp" action="allow" geo_dst="USA" src_user="xxxxx@xxxxx.xxxx.com.au" sent_bytes="594" rcvd_bytes="0" tls_version="SSL_0" tls_profile="TLS-Client.Standard.CORP" sni="fonts.googleapis.com"

    2020-05-25 10:11:40 Allow 10.0.10.7 142.250.66.234 https/tcp 52955 443 1-Trusted-WIRED 0-External Allowed 52 127 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" src_ip_nat="203.0.113.10" tcp_info="offset 8 S 198814100 win 64240" geo_dst="USA" src_user="xxxxx@xxxx.xxxx.com.au"

    Here is the error 15 seconds later:

    2020-05-25 10:11:55 Deny 142.250.66.234 203.0.113.10 52955/tcp 443 52955 0-External Firebox tcp **invalid connection state ** 52 122 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AS 4069553894 win 60720"

    2020-05-25 10:12:11 Deny 142.250.66.234 203.0.113.10 52955/tcp 443 52955 0-External Firebox tcp invalid connection state 52 122 (Internal Policy) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 8 AS 4069553894 win 60720"

    Any thoughts?

    Adrian from Australia

  • Two points:

    1) The messages are utterly useless because they show up for devices that are working perfectly and there are more of these than there are Allow messages from those working devices. For some working devices, I see four times more of these messages than I do the devices' Allow messages.

    2) There is no way to turn off these messages. That is why I opened the support case.

    ALL of these messages show "dst_intf=Firebox" when NONE of the policies for these devices has Firebox as the actual target, which is why they are useless clutter. Filtering FSM traffic for just the IP of my UniFi wireless point, there are 56 lines showing. Only 13 of those are green showing an Allow to my UniFi controller on VLAN6. The rest of a useless red smear on my screen.

    Gregg Hill

Sign In to comment.