Connectivity Issues

Incident Report for Squarespace

Postmortem

On Wednesday November 6th, many Squarespace websites were unavailable for 102 minutes between 14:13 and 15:55 ET. Site visitors saw slow loads or “Service/Unavailable” (status 503) errors.

The incident was caused by an upgrade to our production database software the previous day. This change had been deployed to our internal testing environments and had been performing as expected for several days. Given this, we deployed the upgrade to our main application cluster on November 5th. The next day, just after 14:00 ET, we experienced a cascading failure of all hosts in our main application database cluster. This effectively disabled our main application for new requests. While we were still able to serve some traffic from our caches, no new requests could be completed.

Rollback of the upgrade began just after 15:00 ET. This allowed us to bring the main application database back online, at which point our service recovered.

We’re still investigating why the upgrade caused this behavior. We intend to reproduce the behavior in our lab and work with our vendor to ensure the bug is patched before we upgrade again.

We deeply apologize for this incident. It is of the utmost importance to us that Squarespace sites be up and available. Thank you for your patience.

Posted Nov 07, 2019 - 19:03 EST

Resolved

This incident has been resolved.

Posted Nov 06, 2019 - 18:46 EST

Update

We are continuing to monitor for any further issues.

Posted Nov 06, 2019 - 16:58 EST

Monitoring

A resolution has been implemented and we are monitoring closely before resolving the issue.

Posted Nov 06, 2019 - 16:18 EST

Update

We are continuing to confirm a root cause for this downtime. We will provide more information as soon as possible.

Posted Nov 06, 2019 - 16:06 EST

Update

We are investigating connectivity issues related to most Squarespace sites. Currently we are actively working to confirm a root cause for this downtime. We will provide more information as soon as possible.

Posted Nov 06, 2019 - 15:52 EST

Update

We are continuing to investigate connectivity issues related to most Squarespace sites. We have identified a malfunctioning system and are continuing to investigate its failure. This is taking longer than anticipated to confirm, and we will provide more information as soon as possible.

Posted Nov 06, 2019 - 15:33 EST

Update

We are continuing to investigate connectivity issues related to most Squarespace sites. We have identified a malfunctioning system and are continuing to investigate its failure. We will provide more information as soon as possible.

Posted Nov 06, 2019 - 15:09 EST

Investigating

We are investigating connectivity issues related to most Squarespace sites. We will provide more information as soon as possible.

Posted Nov 06, 2019 - 14:21 EST

This incident affected: Site Loading.