Authpoint Gateway failing to connect
Opened a ticket case but also wanted to leave a post here since I see other people having very similar issues are doing so:
Gateway fails to connect to WG Cloud. We have alerts that its availability flapped from 2021-12-15 01:54:40 until finally going down and staying down at 2021-12-16 05:54:13. Restarting services doesn't appear to work (when it usually does), and the gateway service seems to keep stopping after a time even after multiple restarts. Updating the gateway via an up-to-date installer does not resolve the issue as well. Can anybody assist?
0
Sign In to comment.
Comments
Yes... I've got a customer who's been affected for over 10 hours now and is basically crippled... We called support and after an hour of waiting on hold to actually talk to someone, the call dropped. Called back and waited almost another hour. I also got the WG country manager involved. It's been over 7 hours now since we finally got a ticket opened and zero communication, despite asking for updates. I'm not overly impressed - WG's support team has always been way better than this!
Back to the affected customer - we've had no choice but to disable all MFA on their stuff because they are pretty much work from home now and no one was able to connect to anything.
dcc
I noticed while sitting on the gateway screen it goes from green to red over and over…had over 5k notifications.
Sounded like it’s going to be a day or two…pretty ridiculous.
Apparently this is residual of the Amazon-West outage that happened yesterday according to the ticket I opened with them... and confirm since several of other MFA providers that use AWS-west have been down for the count for the last few days.
Was able to discuss with WG Engineer using our platinum partner support line and they are currently working on it and are working on updating statuses via status.watchguard.com.
It seems to have impacted two components with different ranges of effects:
1st component (Currently installed AuthPoint Gateways): This had issues where authentication service wouldn't work, this (according to the tech I was working with) apparently has been resolved now.
2nd component (New AuthPoint Gateways): This caused issues with New Gateway Registration which, if you tried to reinstall yesterday to fix the issue, would cause new Gateways to fail registrations. This has NOT been resolved yet
Currently waiting on further update.... hopefully they will comment on this post with further detail than I can regurgitate with this post.
Also, there are some WG Cloud status issues (where the status says "Disconnected" but it is still logging... but the status causes AuthPoint to fail) with Fireboxes but they will work on that when the more severe AuthPoint gateway issue is fixed.
These issues do not appear to impact all clients but it is impacting a few.
If you have issues with a currently installed AuthPoint gateway that was not reinstalled yesterday or today then you should be able to restart the gateway service to fix the issue. Commands to do so is below:
net stop AuthPointGateway
net start AuthPointGateway
Thank you Amazon for giving all cloud vendors who use you a heart attack this week.
Well at least you guys are getting updates.. I'm only a 20+ year gold partner who only sells security products from WG, and I've heard squat... Err... Diddly squat... Over the past 11 hours, even though I've asked multiple times for updates... My customer doesn't care if it is Amazon's fault - they are paying WG for a security service, and right now, that service is useless and the customer has no security...
Trust me... our clients are just as peeved... I was just trying to provide the unbiased/opiniated facts so that other partners aren't scratching their heads the way we have been
Tristan - I've restarted the services multiple times today without any change...
The update I got from WG on that was around 12:55 PM PST... not sure if that helps. If that restart still doesn't work I advise contacting support and seeing if they can assist...
As long as Gateway hasn't been installed in last 48 hours the restart should work. The ones that haven't worked for me I have called WG on and they have assisted.
I am sorry to hear how this has impacted you. Hopefully AWS, and WG, are able to work a feasible resolution from this as well as provide an ROF at the end.
I would not be surprised if this caused clients to leave...
Thus far I have only had one client impacted by this so it appears to be region specific.
I'm not disputing that fact Tristan, and thanks for the updates (since WG isn't providing them)... Just frustrated at this point in the day and shit runs down hill, and I'm about half way down that hill right now with it being ankle deep. :-) I just turned off all the customer's infrastructure for the rest of the night and we'll see what morning brings. I also see the update on the status page that was supposed to be posted at 02:45 UTC is now at least 22m late too.
yeah I was looking there for their 6:45 PM PST update and it never came.
I have to Restart the gateways tonight myself to make sure that those who haven't been impacted don't become impacted. I just looked at the forums and figured I'd post since Misery loves company!
Good luck my friend, I will give updates here if I have any and will watch and see if you, other partners, or even WG themselves, have updates to share here.
Cheers
~T
I am actually surprised that the status.watchguard.com page still says that this issue is "Minor" given this issues scale and impact I feel the "minor" moniker is downplaying the severity of the issue grossly
So.. No change since last night, and no update from WG either. Ask me how impressed I am... Go ahead, ask me...
I decided to completely delete the existing gateway (since it wasn't working anyways), and I created a new one. Now WG cloud still shows it as "Not Installed" under Gateways, but when I look at the logs in C:\ProgramData\WatchGuard\AuthPoint, I can see it is up and communicating. I then went to my LDAP source in External Identities and selected "Check Connection". It says it's working. So then I did a "Start Synchronization" and it seemed to work, and I was able to verify that via the logs in C:\ProgramData\WatchGuard\AuthPoint. I re-enabled MFA on the Netscaler and I'm getting pushes and able to login. However, WG Cloud still shows the Gateway as not installed. So I guess we can take this as progress...
dcc
Support finally called me back. We stopped the services for a few minutes, and then restarted them, the status eventually updated on the gateways page that it was installed, and then I was able to add my secondary GWs.
I was also told there would be a statement from WG about this incident coming up.
WatchGuard sent announcement in my ticket that services should be restored by 6:00 AM PST. I was able to confirm this with my broken gateways this morning at 8:00 AM PST.