Dear R360 Cloud Hosting Customers,
We wanted to provide a recap on recent disconnect issues that have been occurring this past week.
Last weekend we installed Microsoft monthly updates as well as WatchGuard Router Firmware updates (come out every few months). These are required security updates that have to be installed in order for us to main compliancy with NSA Security guidelines as well as our cyber security insurance.
On Monday, disconnects started to occur, but they were sporadic as well as only affecting certain/different users, it wasn’t everyone. We ran full diagnostics on everything we manage and couldn’t find a culprit.
We noticed that our WatchGuard routers couldn’t communicate with WatchGuard AuthPoint’s cloud, so we rebooted our equipment. We opened a Support Ticket with WatchGuard and at the time they told us there weren’t any known issues (that they were willing to tell us about). We therefore chalked it up to rolling internet provider outages, which have been happening daily.
Today when the issue happened again, we called back on the Ticket with WG and they admitted that there was an actual outage with WG Cloud AuthPoint MFA and they didn’t know what was causing it but had been working on it since Tuesday. They also admitted to not updating their https://status.watchguard.com as they didn’t want to cause any concerns with customers. Today finally they did update their status page and also resolved the outage as of this morning (hopefully). Below is what is on their status page.
With the firmware updates mentioned above, we believe WG also pushed out a bad firmware update (similar to CrowdStrike a few months ago), but that issue is only affecting our backup router. We narrowed that down this morning and are working with WG hardware support on that to make sure the backup router is functioning properly when it is needed.
While our Servers, internal network aren’t having any issues, components we use from WatchGuard for EPDR, AuthPoint, WebBlocker, have all had outages this week and we’ve been doing everything we can internally to reset and resolve those issues when they arise.
We’d like to thank you for your patience on this as you’ve been wonderful customers. With all of the security in place to protect our networks, when issues like this arise we have to be thorough in troubleshooting them to make sure your financial data is protected. When the issues are with our software/vendor partners, we have to rely on them to fix their components.
Please let us know if you have any questions on the above.
R360 Cloud Hosting Support
Here are the logs on the https://status.watchguard.com page for Americas Region
Sep 27, 2024
2024-09-27 - AMER & EMEA - AuthPoint Authentication service partial outage.
Resolved - The service has been operating normally since our last update, there are no additional issues reported on AuthPoint Authentication methods and this incident is now resolved. Again, we apologize for the impact this may have had on you or your customers.
Sep 27 2024, 14:33 UTC
Monitoring - We were experiencing issues with AuthPoint Authentication methods, our team has already fixed the issue, users that experienced delays or timeouts in their authentication requests, should be able to authenticate successfully now, we will post our next message in 30 minutes if not sooner, we apologize for the impact this event has caused.
Sep 27 2024, 13:58 UTC
Sep 24, 2024
2024-09-24 - Global - WatchGuard Cloud Visibility Reports Issues
Resolved - We are no longer experiencing issues with the scheduled reports management and this incident is now resolved. The WatchGuard Cloud Visibility Reports service is operating normally at this time. Again, we apologize for any impact this may have had on you or your customers.
Sep 24 2024, 15:08 UTC
Monitoring - We have deployed a fix for the WatchGuard Cloud Visibility Reports service, and have identified the issue causing the scheduled reports management being down. All regions are showing signs of recovery and our teams are monitoring closely to guarantee normal operations, we apologize for the impact this event has caused. We'll post our next update in 30minutes if not sooner.
Sep 24 2024, 14:24 UTC
Investigating - We're currently investigating errors reported on the WatchGuard Cloud Visibility Reports service, once we determine the impact and scale of these errors we'll post our next update in 30 minutes, if not sooner.
Sep 24 2024, 13:42 UTC
Sep 21, 2024
2024-09-21 - GLOBAL - Watchguard portal login issues
Resolved - We are no longer experiencing alerts on login service to Watchguard portal and this incident is now resolved. The service is operating normally at this time.
Sep 21 2024, 10:19 UTC
Monitoring - Our systems show alerts on Watchguard portal service are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30 minutes, if not sooner.
Sep 21 2024, 09:47 UTC
Identified - We continue to work on login service to Watchguard portal and have identified the issue, only the users with MFA are impacted. Our teams are working to restore normal operations and we apologize for the impact this event has caused. We'll post our next update in 30 minutes, if not sooner.
Sep 21 2024, 09:34 UTC
Investigating - We're investigating alerts on login service to Watchguard portal. Once we determine the impact and scale of these errors we'll post our next update in 30 minutes, if not sooner.
Sep 21 2024, 09:28 UTC
Sep 16, 2024
2024-09-16 - GLOBAL - Firebox Webblocker Service not available
Postmortem -
Event Summary: The WatchGuard Firebox WebBlocker service was degraded impacting some of our WebBlocker customers in our Europe region between, approximately, September 16th 06:45 UTC and 07:10 UTC. Customers may have experienced 5XX errors. The event is now resolved, and the service is operating normally for all WatchGuard Firebox WebBlocker users.
Event Findings: On September 16th, at approximately 07:10 UTC, our engineers were alerted to a potential issue with the WebBlocker service. We reached out to our 3rd party provider of the service, and they were able to mitigate the issue by re-routing a subset of traffic to another region. Our 3rd party provider has performed an in-depth analysis with their cloud provider and has already taken action to mitigate the risk of recurrence.
We sincerely apologize for the impact on our affected customers, and we know the stability of our services is important to you and your business. At WatchGuard, we will never be satisfied with anything less than perfect operational performance and will continue to do everything we can to drive improvements across our services.
Invalid date UTC
Resolved - We are no longer experiencing alerts on the Webblocker service and this incident is now resolved. The service is operating normally at this time.
Sep 16 2024, 08:00 UTC
Monitoring - Our systems show alerts on the Webblocker service are returning to normal and we're monitoring to ensure system stability. We'll post our next update in 30 minutes, if not sooner.
Sep 16 2024, 07:39 UTC
Investigating - We're investigating alerts on the Webblocker service. Once we determine the impact and scale of these errors we'll post our next update in 30 minutes, if not sooner.
Sep 16 2024, 07:26 UTC
Sep 09, 2024
2024-09-09 - AMER & EMEA - Firebox WebBlocker service degraded
Resolved -
Event Summary: The WatchGuard Firebox WebBlocker service was degraded impacting some of our WebBlocker customers in our Americas and Europe regions between, approximately, September 9th 06:20 UTC and 08:34 UTC. Customers may have experienced 5XX errors. The event is now resolved, and the service is operating normally for all WatchGuard Firebox WebBlocker users.
Event Findings: On September 9th, at approximately 07:00 UTC, our engineers were alerted to a potential issue with the WebBlocker service. We reached out to our 3rd party provider of the service, and they were able to mitigate the issue by re-routing a percentage of traffic to another region.
We sincerely apologize for the impact on our affected customers, and we know the stability of our services is important to you and your business. At WatchGuard, we will never be satisfied with anything less than perfect operational performance and will continue to do everything we can to drive improvements across our services.
Sep 09 2024, 07:00 UTC
Sep 04, 2024
2024-09-04 - GLOBAL - WGC login Issues
Resolved - This incident has been resolved and the service is operating normally at this time. We apologize for any impact this may have had on you or your customers.
Sep 04 2024, 07:37 UTC
Monitoring - We are no longer experiencing issues with logins to WGC and main website, affected components are operational again and we're monitoring to ensure system stability. We'll post our next update in 30 minutes, if not sooner
Sep 04 2024, 07:05 UTC
Investigating - We are currently experiencing some issues with logins to WGC and main website not working properly. We will post our next update in 30 mins, if not sooner.
Sep 04 2024, 06:39 UTC