fonts.googleapis.com misbehaving?
A few times, when I have opened www.watchguard.com or community.watchguard.com there has been a noticeable delay at the point where both sites load fonts.googleapis.com. If I do nothing it eventually shows the site, and every subsequent page loads very slowly.
A quick in investigation showed that the Firebox had blocked the IP for fonts.googleapis.com. Here are the relevant log entries, with user information redacted, after I cleared the blocked site entry and reopened community.watchguard.com:
2020-11-11 05:25:56 Allow 10.0.10.7 172.217.25.138 https/tcp 50166 443 Trusted-WIRED External ProxyAllow: HTTPS content inspection exception list match (HTTPS-proxy-00) proc_id="https-proxy" rc="590" msg_id="2CFF-000A" proxy_act="HTTPS-Client.Standard.corp" action="allow" geo_dst="USA" src_user="xxx" sni="fonts.googleapis.com" cn="upload.video.google.com" exception_rule="fonts.googleapis.com"
2020-11-11 05:25:56 Allow 10.0.10.7 172.217.25.138 https/tcp 50167 443 Trusted-WIRED External ProxyAllow: HTTPS content inspection exception list match (HTTPS-proxy-00) proc_id="https-proxy" rc="590" msg_id="2CFF-000A" proxy_act="HTTPS-Client.Standard.corp" action="allow" geo_dst="USA" src_user="xxx" sni="fonts.googleapis.com" cn="upload.video.google.com" exception_rule="fonts.googleapis.com"
2020-11-11 05:25:56 Allow 10.0.10.7 172.217.25.138 Google APIs(SSL) 50166 443 Trusted-WIRED External Application identified 557 128 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0149" src_ip_nat="203.0.113.10" tcp_info="offset 5 A 1094766545 win 516" app_name="Google APIs(SSL)" app_cat_name="Network protocols" app_id="698" app_cat_id="19" app_beh_name="Authentication" app_beh_id="1" action="Global.corp" geo_dst="USA" src_user="xxx" sig_vers="18.119"
2020-11-11 05:25:56 Allow 10.0.10.7 172.217.25.138 Google APIs(SSL) 50167 443 Trusted-WIRED External Application identified 557 128 (HTTPS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0149" src_ip_nat="203.0.113.10" tcp_info="offset 5 A 1598212204 win 516" app_name="Google APIs(SSL)" app_cat_name="Network protocols" app_id="698" app_cat_id="19" app_beh_name="Authentication" app_beh_id="1" action="Global.corp" geo_dst="USA" src_user="xxxx" sig_vers="18.119"
2020-11-11 05:25:56 Allow 10.0.10.7 172.217.25.138 https/tcp 50166 443 Trusted-WIRED External HTTPS Request (HTTPS-proxy-00) proc_id="https-proxy" rc="548" msg_id="2CFF-0000" app_id="0" app_cat_id="0" proxy_act="HTTPS-Client.Standard.corp" action="allow" geo_dst="USA" src_user="xxxx" sent_bytes="792" rcvd_bytes="6417" tls_version="TLS_V13" tls_profile="TLS-Client.Standard.CORP" sni="fonts.googleapis.com" cn="upload.video.google.com" cert_issuer="CN=GTS CA 1O1,O=Google Trust Services,C=US" cert_subject="CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US" sig_vers="18.119"
2020-11-11 05:25:57 Deny 172.217.25.138 203.0.113.10 **38045/tcp **443 38045 External Firebox Denied 71 57 (Unhandled External Packet-00) proc_id="firewall" rc="101" msg_id="3000-0148" tcp_info="offset 5 A 3663368292 win 2305" geo_src="USA"
2020-11-11 05:25:57 firewall Temporarily blocking host 172.217.25.138 (reason = autoblock by policy) msg_id="3001-1001"
2020-11-11 05:28:47 Allow 10.0.10.7 172.217.25.138 https/tcp 50167 443 Trusted-WIRED External HTTPS Request (HTTPS-proxy-00) proc_id="https-proxy" rc="548" msg_id="2CFF-0000" app_id="0" app_cat_id="0" proxy_act="HTTPS-Client.Standard.corp" action="allow" geo_dst="USA" src_user="xxxx" sent_bytes="1484" rcvd_bytes="10307" tls_version="TLS_V13" tls_profile="TLS-Client.Standard.CORP" sni="fonts.googleapis.com" cn="upload.video.google.com" cert_issuer="CN=GTS CA 1O1,O=Google Trust Services,C=US" cert_subject="CN=upload.video.google.com,O=Google LLC,L=Mountain View,ST=California,C=US" sig_vers="18.119"
In the third to last entry you can see that the Firebox blocked the site because it tried to connect using port 38045/tcp. Why is this so??
Adrian from Australia
Comments
I see lots of these, from various web sites - 443 as the source port.
Could be caused by many reasons - but for whatever reason the session tables no longer have an entry which matches this packet.
I haven't had "Auto-block source of packets not handled" enabled for 15 or more years on any of my firewalls because of the issues caused by "good" IP addrs ending up on the temp Blocked Sites list because of this setting.
The reason for my suspicion is that I searched Dimension for 38045 expecting to see a call to 172.217.25.138 with a source port of 38045, but the first entry is the "unhandled exception" log entry. In fact, all of my requests to this IP address today have source ports in the 50000 range. I feel that there is something rotten in Denmark..
More investigation is needed before I uncheck my Auto-block feature..
UPDATE: Also checked the Dimension logs for yesterday.. Also no mention of 38045... But yesterday's port of the day from google was:
FWDeny, Denied, pri=4, disp=Deny, policy=Unhandled-External-Packet-00, protocol=44455/tcp, src_ip=172.217.25.138, src_port=443, dst_ip=203.0.113.10, dst_port=44455, src_intf=External, dst_intf=Firebox, rc=101, pckt_len=71, ttl=59, pr_info=offset 5 A 1957479697 win 2305, 3000-0148, geo_src=USA
Adrian from Australia
Others, on the old Forum, have stated that Supports has said that XTM does not log all session packets.
That might be the case here.
No way to tell without packet captures to know if this is a rogue reply packet - hard to believe.
Auto-block for me, in the long ago past, has blocked Internet DNS server IP addrs because of slow responses.
And, I'm seeing 14 denies from techsearch.watchguard.com at the moment, with a source port of 443. Example:
2020-11-10 18:44:51 Deny 13.109.194.84 174.48.xxx.xxx 38875/tcp 443 38875 External Firebox Denied 40 245 (Unhandled-External-Packets-00) proc_id="firewall" rc="101" msg_id="3000-0148" fqdn_src_match="watchguard.com" tcp_info="offset 5 A 1698245211 win 22284" geo_src="USA" geo_dst="USA" Traffic
What is not obvious is if Dynamic NAT is changing the outgoing source port number, so that looking for the matching dest port shown on the deny in the logs is hopeless.
Sigh! That does make it hopeless to track down the potential problem..
Adrian from Australia