Success.
Within the past 30 minutes, we successfully transferred over to our street-level generator, ran for 20 minutes, then failed back to our building generator. Our building generator, street generator, and battery backups worked properly during this test.
We are confident to report that we now have redundant on-site power while we wait for our building to formally reconnect to the Manhattan power grid.
We have received word that the redundant street side generator has been connected, and a load transfer test will be performed tonight. Wednesday, November 7, 2012 from 8:00pm ET to 12:00am ET.
If this load transfer is successful, we will have 2 confirmed working power sources on site, and will be able to install a redundant fuel filter in our main generator tomorrow, further bolstering our resiliency.
We do not anticipate any impact from the Nor’easter blizzard/storm that is currently descending upon New York (though we certainly aren’t thrilled by it!)
Roll up generator for redundant power supply is in NY at facility. Electricians are almost done with wiring. No significant progress on restoring the street power for the building due to water damage.
We are still operating off of generator power and are awaiting the arrival of the redundant truck-mounted generator, which we expect to come this evening. We will update again when redundant power has been restored.
Squarespace office infrastructure in Soho is now completely restored, and we are ready to develop again and get rolling on Monday. We will continue to provide updates related to the Peer1 facility as more news emerges. The situation remains stable at this time.
We continue to operate sustainably via the building generators. Power has not been completely restored to the Wall St area, and we do not have an up to date report on the building’s mechanical state. We have received word that a street side generator will arrive by Monday to restore truly redundant power while we wait for the building to be repaired. We have systems staff on site at the datacenter 24/7 to monitor the situation until we are back to normal. Power to our office has been restored, and we are bringing development systems back online now.
We continue to successfully run via the connected pump.
We do not have additional infrastructure updates at this time, but major personnel support from Peer1 are on site. ConEdison is stating that power will be restored to lower Manhattan tonight, but we do not know how that will help with our situation immediately, as our building infrastructure is still damaged, but is obviously another step in the right direction.
We have confirmed that the fuel pump is now working. We can now operate off of fuel indefinitely and there are continuous deliveries scheduled.
Additional issues remain, the largest of which being that our generator will need to be taken offline at some point to clean the fuel filter. We can avoid that downtime if Peer1 can provide a street level generator. If this generator failure does occur, we expect the level of downtime to be on the order of an hour, instead of the multi-day outage we were facing at the onset of this crisis.
We remain in a difficult situation, but the most difficult challenge we were presented with is now behind us. As of this writing, Squarespace has miraculously avoided all downtime related to Sandy, but we still have many challenges ahead with respect to getting back to anything normal.
A pump has been successfully installed that is carrying fuel to the roof fuel header, and it has been running for an hour. We are awaiting confirmation that the header and our tank are properly connected. The building is going to attempt to power up downed generators for other floors in 30 minutes.
We were able to temporarily resume carrying fuel for an hour, which means we have about 4-5 hours of fuel left on the roof. Datacenter engineers are on site attempting to finish a more permanent pump solution, which we should know more about within the hour.
Some bad news. We may not be able to deliver more fuel, as the building is shutdown mode. Last estimate is that we have about 3 hours left. Things change constantly, and we will keep you updated.
A short update to start the day…
We have plenty of fuel on hand. The PEER1 generator is running strong and is in the process of being filled again. The bucket brigade will start by 10AM to keep us running for another 8-10 hours before the next brigade begins.
A hopeful update from the building: This building has 2 basements and both are flooded. As of 7AM today, the first basement has been completely pumped out. A broken water main was discovered which was ADDING water to the basement as the pumps ran yesterday. By using 3x the number of pumps overnight, they were able to outpace the water main, clear the basement and turn off the running water main. We have no ETA on ConEd power for the building as they have another full basement to drain which will likely take another day. In parallel, the building is making progress on a separate solution to pump fuel to their 5000 gallon tank.
Finally, PEER1 is waiting for fuel pumps from NJ which would replace the human bucket brigade. Unfortunately, their trucks are waiting for gas in NJ.
Thanks again for all of your support.
The midnight bucket brigade has begun (photo below).
When tonight’s brigade is done, we estimate we have a conservative 10 hours of capacity, allowing our 8am shift to begin tomorrow with low stress and also a switch to 120 minute refill sessions every 8 hours, versus yesterday’s continuous huffing of fuel up the stairs. Yeah, we’re getting good at this.
The building has had crews pumping water out of the basement all day. Unfortunately, we have no ETA on the repairing of the basement fuel delivery systems until the building’s crews get down there. However, both the building and PEER1 have been putting together separate contingency fuel pumping systems. We hope one of these will be online in the next 24-36 hours so that fuel refills will involve very few people and simply take place via a truck at street level.
Our greatest risk at this point may be the generator itself. We’ve kept it fueled, and have a good system in place, but anything can happen. Fingers crossed. Have a good night, and again, thanks for all the great tweets and emails. They mean a lot!
Terrific progress this afternoon.
We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the night. On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. Special thanks to Mike Mazzei and his team at Peer1 NYC for unprecedented dedication these past days.
We have a big crew teaming up tonight for a midnight bucket brigade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.
We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after. Water is being successfully pumped from the basement at a rate of one foot per hour. Fifteen feet of water remain. We have increasing confidence that we may avoid downtime. Of course, there are no guarantees and we are working hour-by-hour.
Good news and bad news.
We were able to give the crew 90 minutes off because the tank is full. That’s about 4 hours of power given daytime usage. We’ll start back up at 1:30. Our awesome teammates are hoofing lunch over the Brooklyn Bridge for us. As we’ve said before, this situation is untenable. We can’t keep manpower going 24/7 for days.
The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.
The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.
We’ll keep you posted and thank you for your continued good wishes.
Bucket brigade going strong. We’ve gone through half of our morning fuel delivery and are expecting a truck with 5,000 gallons coming at noon. Potential issues are lack of bucket brigade manpower into the night, and our fuel pumps burning out. Peer1 is working to establish a more permanent alternative to fuel delivery. If successful, it may be online this afternoon. Spirits are strong and everyone from Peer1, Fog Creek, and Squarespace is working together. We’ve added some photos below in response to requests.
