Google login issue

We are having an issue where users cannot log into Google, both personal and Workspace accounts. If I bypass the firewall, a user can connect. If I restart the firewall, users can connect again but by the next day the issue is back. We first had an issue with downloads from Google on the 23rd. The problem cleared up on its own, while I was troubleshooting the issue. I went ahead and updated the firewall to 12.11.8 anyways. Yesterday the login issue appeared. It doesn't matter what computer or browser we use. I've tried having the NAT connection go out of different IP addresses, I've tried switching from a proxy rule to just a packet filter rule.

Comments

  • Anything in Traffic Monitor to help understand this?

    Have you check your temporary Blocked Sites in FSM or the Web UI?

  • MeOMy
    edited March 27

    I've seen nothing in Traffic Monitor that would indicate that the firewall is blocking anything. Only thing in Blocked Sites was stuff I put there. AI seems to think it is something to do with how the Firewall is handling HTTP2, TLS, and NAT. I discovered that there is now a fireware OS 12.12 that just came out and I've updated to it. There was one other site that a person had trouble logging into but because we found a workaround I forgot to mention it. The blackbaud.com site wouldn't hand off to our SSO provider (Not Google). That issue only occurred when the Google issue accrued.

  • Chrome (and probably other web browsers) are now using HTTPS over UDP (QUIC) and DNS over UDP.
    These can hide what is going on from policies you may be using with logging enabled.

    I have both prevented in my policies - I like to know and possible control what is going out to the Internet.

    And you can turn these off in the settings of web browsers. Perhaps doing so on a test web browser can help understand if either of these relate to your issues.

    And consider opening a support case on this.

  • I did go into chrome;//flags and turn off QUIC but that didn't have any affect. Everything is working since the reboot and upgrade to 12.12 but usually doesn't surface until the next day. If it is back on Monday, I'll put in a support request. Hopefully the 12.12 update fixed it.

  • Same issue! Contacted support thought we found the solution but the problems back fresh today. About to update to 12.12 this afternoon hoping it will solve the problem.

  • For others, what did you change which did not end up helping at your site?

  • Well support suggested that it was our gateway AV reading packets that may have been causing the issue, and also suggested restarting the server if we hadn't already.

    We messed with the gateway AV some, ultimately toggling off the "When a scan error occurs -> drop" setting. After this our issues seemingly vanished.

    Until today that is, now after this post I am assuming it was not the Gateway AV change that solved our issue but the restart.

  • Posting to let you know we are having the same issue on OS 12.12. Rebooting fixes the issue for a while but then the problems with Google come back.

  • 12.12 didn't solve it for us either. I'm still working with support on this but we have found a workaround other than just restarting the firewall. If you look in System Manager -> Blocked Sites, you may see a Google address (142.250.x.x) that is being blocked with a reason of port scan. If you remove that address from the block IP list it works again.

  • You can set up email notifications for port scans, which could help get Google site access back quicker.

  • Support basically told me to just add the IP address to the Blocked Sites Exceptions list. Not really happy with that answer but I cannot keep going in and removing it from the blocked list. Hopefully enough other people are having the issue, it gets addressed in a future update.

  • I've found in the past that QUIC through the firewalls caused all sorts of issues with our Google Workspace. I have a custom policy defined for UDP 80 and UDP 443, which is used to block QUIC access from the internal and optional networks out to the external interfaces. This rule has been in place since around 2015 and we have not had any further Workspace issues caused by the firewalls.

  • Still to this day I'm experiecing issues with Classroom and Google sign-in services going down when WG catches a port scan attack from Google IPs. I've had to already put 4 Google IPs to the block exception list just to get these services back on, would be nice for a fix that is more than just adding them to the exception list. Anyone else seeing these port scan attacks?

  • Has anyone come up with a permanenet working solution to this. I work for a school district and this pops up from time to time on. Usually its random secretaries getting gmail and calendar disconnect issues.. but it also can be system wide with random students havine any google service stall out and not load. Then it goes away after awhile. About ready to just bite the bullet and go with a different firewall at this point. No issues in the summer though when the load is low. Any help appreciated.

  • For the record, what firewall model do you have and what Fireware version is it running?

    Have any of the above suggestions been tried, and if so, have any helped?

  • Sorry for the late reply..

    Watchguard M570, I've been keeping up with all the latest firmware updates. I will need to go back and dig through all the posts, but basically have tried everything that has been mentioned (increasing thresholds for when they lock a port or ip down etc..) I thought it was fixed, but I am thinking that was because summer hit.. and school was out. As soon as the Chromebooks got handed out the issue started popping up randomly again.

  • If this continues, consider opening a support case so that you can get help from a WG rep in understanding & resolving this.

    The rep can look at your logs & config - which we can't do here.

    Chromebooks do use QUIC - HTTP/HTTPS over UDP. And DNS queries over UDP. I block both which causes normal TCP port access for them.