Duration: 29 minutes

Affected Components: ,

Sep 2, 21:03:54 GMT+0
Resolved - This incident has been resolved..

Sep 2, 21:02:46 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 2, 20:43:02 GMT+0
Identified - We have identified the issue are are applying a fix to each affected instance. .

Sep 2, 20:35:11 GMT+0
Investigating - We are currently investigating this incident..

Duration: 44 minutes

Affected Components: ,

Jul 16, 12:52:57 GMT+0
Resolved - This incident has been resolved. All instances are reporting back online. If you have any issues or concerns please contact customer.service@userscape.com..

Jul 16, 12:45:55 GMT+0
Monitoring - We are rolling out a fix to all affected instances and we are monitoring their recovery. We will update when the fix has rolled out to all customers..

Jul 16, 12:35:41 GMT+0
Identified - We have identified the issue and we are continuing to work on a fix for this incident..

Jul 16, 12:09:25 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 54 minutes

Affected Components: ,

Oct 17, 21:12:21 GMT+0
Resolved - This incident has been resolved..

Oct 17, 20:53:47 GMT+0
Identified - Most customers are restored, but some edge cases remain that we are still addressing regarding some secondary portals..

Oct 17, 19:18:42 GMT+0
Investigating - We have reports of some secondary portals still being offline for some customers from the larger outage earlier. We're working on addressing the issue. All other systems are operational..

Duration: 1 hour and 53 minutes

Affected Components: ,

Oct 17, 14:23:17 GMT+0
Postmortem - Our servers operating system (Ubuntu) updated Redis via its automated security update system. After the upgrade, Redis started with a permissions mismatch and could not write to its data directory. Because HelpSpot relies on Redis, this resulted in a service outage. **Remediation** We corrected directory ownership/permissions and verified Redis persistence before bringing services back online. Your system should now be stable again. **Next steps** We’re conducting a deeper review to analyze what additional controls and processes we need around the automated security update system for stateful services like Redis and will adjust our procedures as needed after that review is complete. This was our first system-wide outage in many years. Reliability is foundational to our service, and we will continue to invest in safeguards that keep your service consistently available. We apologize for the disruption. If you notice any lingering issues, please contact us and we’ll help right away..

Oct 17, 13:57:24 GMT+0
Resolved - This incident has been resolved..

Oct 17, 13:36:01 GMT+0
Monitoring - We implemented a fix that is currently rolling out to all customers. Customers should expect to see their systems come back online in the next 30 minutes..

Oct 17, 13:02:38 GMT+0
Identified - We have restore functionality to <https://support.helpspot.com> and we are testing a global fix for this issue..

Oct 17, 12:44:20 GMT+0
Identified - We have identified the root cause of the issue and we are working on remediation steps..

Oct 17, 12:04:14 GMT+0
Investigating - We are currently investigating this incident..

Duration: 7 minutes

Affected Components:

Jul 11, 14:22:29 GMT+0
Resolved - This incident has been resolved..

Jul 11, 14:17:51 GMT+0
Monitoring - We have implemented a fix and we are monitoring email connectivity.

Jul 11, 14:15:01 GMT+0
Investigating - We are currently investigating this incident. This only affects @[helpspotmail.com](http://helpspotmail.com) addresses..

Duration: 22 minutes

Affected Components:

Feb 24, 00:53:46 GMT+0
Resolved - All hosts are now reporting a recovered state we will continue monitoring this evening..

Feb 24, 00:45:48 GMT+0
Identified - This is affecting a small number of hosts (less that 3%). We are have a remediation that we are applying to those hosts that should restore responsiveness in the next 15 minutes..

Feb 24, 00:32:00 GMT+0
Investigating - We are currently working on remediating .