Squarespace System Status

A pump has been successfully installed that is carrying fuel to the roof fuel header, and it has been running for an hour. We are awaiting confirmation that the header and our tank are properly connected. The building is going to attempt to power up downed generators for other floors in 30 minutes.

We were able to temporarily resume carrying fuel for an hour, which means we have about 4-5 hours of fuel left on the roof. Datacenter engineers are on site attempting to finish a more permanent pump solution, which we should know more about within the hour.

Some bad news. We may not be able to deliver more fuel, as the building is shutdown mode. Last estimate is that we have about 3 hours left. Things change constantly, and we will keep you updated.

A short update to start the day…

We have plenty of fuel on hand. The PEER1 generator is running strong and is in the process of being filled again. The bucket brigade will start by 10AM to keep us running for another 8-10 hours before the next brigade begins.

A hopeful update from the building: This building has 2 basements and both are flooded. As of 7AM today, the first basement has been completely pumped out. A broken water main was discovered which was ADDING water to the basement as the pumps ran yesterday. By using 3x the number of pumps overnight, they were able to outpace the water main, clear the basement and turn off the running water main. We have no ETA on ConEd power for the building as they have another full basement to drain which will likely take another day. In parallel, the building is making progress on a separate solution to pump fuel to their 5000 gallon tank.

Finally, PEER1 is waiting for fuel pumps from NJ which would replace the human bucket brigade. Unfortunately, their trucks are waiting for gas in NJ.

Thanks again for all of your support.

The midnight bucket brigade has begun (photo below).

When tonight’s brigade is done, we estimate we have a conservative 10 hours of capacity, allowing our 8am shift to begin tomorrow with low stress and also a switch to 120 minute refill sessions every 8 hours, versus yesterday’s continuous huffing of fuel up the stairs. Yeah, we’re getting good at this.

The building has had crews pumping water out of the basement all day. Unfortunately, we have no ETA on the repairing of the basement fuel delivery systems until the building’s crews get down there. However, both the building and PEER1 have been putting together separate contingency fuel pumping systems. We hope one of these will be online in the next 24-36 hours so that fuel refills will involve very few people and simply take place via a truck at street level.

Our greatest risk at this point may be the generator itself. We’ve kept it fueled, and have a good system in place, but anything can happen. Fingers crossed. Have a good night, and again, thanks for all the great tweets and emails. They mean a lot!

Terrific progress this afternoon.

We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the night. On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. Special thanks to Mike Mazzei and his team at Peer1 NYC for unprecedented dedication these past days.

We have a big crew teaming up tonight for a midnight bucket brigade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.

We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after. Water is being successfully pumped from the basement at a rate of one foot per hour. Fifteen feet of water remain. We have increasing confidence that we may avoid downtime. Of course, there are no guarantees and we are working hour-by-hour.

Good news and bad news.

We were able to give the crew 90 minutes off because the tank is full. That’s about 4 hours of power given daytime usage. We’ll start back up at 1:30. Our awesome teammates are hoofing lunch over the Brooklyn Bridge for us. As we’ve said before, this situation is untenable. We can’t keep manpower going 24/7 for days.

The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.

The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.

We’ll keep you posted and thank you for your continued good wishes.

Bucket brigade going strong. We’ve gone through half of our morning fuel delivery and are expecting a truck with 5,000 gallons coming at noon. Potential issues are lack of bucket brigade manpower into the night, and our fuel pumps burning out. Peer1 is working to establish a more permanent alternative to fuel delivery. If successful, it may be online this afternoon. Spirits are strong and everyone from Peer1, Fog Creek, and Squarespace is working together. We’ve added some photos below in response to requests.