Hazard Hosts LLC - NY Datacenter - Outage – Incident details

All systems operational

NY Datacenter - Outage

Resolved
Major outage
Started about 2 months agoLasted about 8 hours

Affected

NewYork VPS Hosting

Major outage from 8:10 PM to 3:39 AM, Partial outage from 3:39 AM to 3:59 AM

NY-VPS-AMD-2

Major outage from 8:10 PM to 3:39 AM, Partial outage from 3:39 AM to 3:59 AM

NY-VPS-AMD-3

Major outage from 8:10 PM to 3:39 AM, Partial outage from 3:39 AM to 3:59 AM

Locations

Major outage from 8:10 PM to 3:39 AM, Operational from 3:39 AM to 3:59 AM

New York

Major outage from 8:10 PM to 3:39 AM, Operational from 3:39 AM to 3:59 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We have confirmed networking infrastructure and cooling are restored. We will now begin working on servers.

  • Update
    UTC
    Update

    NYSEG has restored power and we will be first working on restoring cooling and network infrastructure, then continue with servers.

  • Update
    UTC
    Update

    NYSEG is now on site and investigating. We will provide an ETTR once they provide one.

  • Update
    UTC
    Update

    We have electricians on site and investigating.  We still have no specific ETR but optimistic that we will have a resolution in the coming hour.  

  • Update
    UTC
    Update

    Our electricians have determined that this is an issue due to the utility provider. An emergency dispatch to the utility provider has been created. Once the utility provider electricians arrive at the facility, then we can provide an ETTR.

  • Identified
    UTC
    Identified

    We have identified an issue with one of the main bus bars which is causing breakers to trip. All appropriate resources (NYSEG and our electricians) have been engaged and are enroute to the facility

  • Investigating
    UTC
    Investigating

    At approximately 1:02PM pacific time our NOC detected a cascading power issue within the Binghamton, NY datacenter. Onsite technicians are investigating the issue, we will post updates as soon as possible.