HelpSpot - Notice history

Our servers operating system (Ubuntu) updated Redis via its automated security update system. After the upgrade, Redis started with a permissions mismatch and could not write to its data directory. Because HelpSpot relies on Redis, this resulted in a service outage.

Remediation

We corrected directory ownership/permissions and verified Redis persistence before bringing services back online. Your system should now be stable again.

Next steps

We’re conducting a deeper review to analyze what additional controls and processes we need around the automated security update system for stateful services like Redis and will adjust our procedures as needed after that review is complete.

This was our first system-wide outage in many years. Reliability is foundational to our service, and we will continue to invest in safeguards that keep your service consistently available. We apologize for the disruption. If you notice any lingering issues, please contact us and we’ll help right away.

We implemented a fix that is currently rolling out to all customers. Customers should expect to see their systems come back online in the next 30 minutes.

We have restore functionality to https://support.helpspot.com and we are testing a global fix for this issue.

We have identified the root cause of the issue and we are working on remediation steps.