Hi all!
As Hurricane Sandy is about to hit the east coast US, and as Outbrain’s main Datacenter is located in downtown Manhattan, we are taking measures to make as little service interruption as possible for our partners and customers. Outbrain is normally serving from 3 data centers and in case of NY data center loss, we will supply the service from one the other data centers. On this page, below – we will update on any service interruption and ETAs for problem-solving. We assume all will go well and we will not have to update but… just in case 🙂
[UPDATE – Nov 3rd 3:45 pm EST] – At this time Utility power is back to all our datacenters and HQ office. It is now time to restore the service from NY and get the office back to work. This will take some time but systems will gradually be put back up over the next week or so. There should be no effect on users, publishers or clients.
Our HQ will also start working gradually depending on the availability of public transportation.
We are here closing this reporting post – if you see any issues, please report to am@outbrain.com or your rep.
I hope the storm of the century will be the last one for the next century (at least).
[UPDATE – Nov 1st 9:30 am EST] – Our HQ, located on 13th between 5th and 6th in downtown New York City is still without power and therefore closed. Thankfully, our NY-based team is safe and in dry locations, and will continue to try and work as best they can. We highly appreciate the concern and best wishes we received from our partners and clients across the globe; thank you!
We are doing our best to continue to provide the best in class service, one we hope you’ve come to expect from us. As an update, our datacenter in NY is still without power and we expect it to be down for a few more days. We will continue to serve from our other datacenters located in Chicago and Los Angeles. To reiterate, our service did not go down, and we are currently still serving across our client’s sites. As of this morning, we recovered and updated all our reporting capabilities, so we should be back to 100%.
If you are experiencing any difficulties or seeing different, please reach out to your respective contacts. We’ll also continue to operate under emergency mode until Monday, you can reach us 24/7 at am-emergency-support@outbrain.
[UPDATE – Oct 31st 6:46 am EST] – Serving still holds strong from our LA and Chicago data centers and we are not aware of any disruption to our service. We are working hard to recover our dashboard reporting capabilities, but it will probably take a couple more days before we’re able to get back to normal mode. Sorry for any inconvenience caused by this. Send us a note to am-emergency-support@outbrain.
[UPDATE – 6:51 pm EST] – Again, not much to update – All is stable with both LA and Chicago datacenters. It’s the end of the day here in Israel and we are trying to get some rest. Our teammates in the US are keeping an eye on the system and will alert us if there is anything wrong. Good night.
[UPDATE – 3:35 am EST] – Actually not much to update about the service. All is pretty much stable. we are safely serving from LA and Chicago. most back-end services are running in LA Datacenter and our tech team in Israel and NY are monitoring and handling issues as they raise. Our Datacenter vendors in NY are working with FDNY to pump the water from the flooded generator room so it will take a while to recover this datacenter 🙂
[UPDATE – 10:50 am EST] – The clients dashboard is back up.
[UPDATE – 10 am EST] – The client’s dashboard on our site is periodically down – we are handling the issues there and will update soon.
[UPDATE – 5 am EST] Our NY Datacenter went down. Our service is fully operational and we are serving through our Chicago and LA Datacenters. If you’re accessing your Outbrain dashboard you may experience some delays in data freshness. We are working to resolve this issue and will continue to update.
[UPDATE – 2am EST] – Our NY Datacenter went completely off – We are fully serving from our Chicago and LA Datacenters. External reports on our site are still down but we are working to fail overall services from the LA Datacenter. – we will follow with updates.
[Update – 12:50 am EST] – power just went all off in our NY Datacenter and the provider has evacuated the facility – we are taking our measures to move all functionality to other datacenters.
[UPDATE] – at 9 pm EST] commercial power went down in our NY Datacenter. Provider failed over to the generator and we continue to serve smoothly from this Datacenter. We continue to monitor the service closely and ready to take actions if needed.