NOC: 1-843-410-8486
There are no Open Network Issues Currently
Chicago Network Outage - January 14th, 2024 (Resolved) Critical

Affecting Other - FACILITY

  • 14/01/2024 17:34 - 17/01/2024 17:59
  • Last Updated 14/01/2024 19:27

We are currently experiencing a network outage in our Chicago Datacenter at 600 South Federal.

The network at this facility backhauls from Crown Castle to 350 E Cermak (Equinix CH1).

Equinix CH1 is currently experiencing issues with their HVAC systems as a result of extreme weather.

They are supplying outside air to the 5th floor to counteract the heat load. More details for this outage can be found on Equinix's status page.

Once the issue is resolved and equipment is brought back online in Equinix CH1, connectivity to our cabinets should be restored in 600 South Federal.

Scheduled Maintenance: Lumen/Level3 - Mar 17, 2023 - 00:00 CDT (Resolved) High

Affecting Other - IL Fiber

  • 17/03/2023 00:00 - 07/05/2023 00:16
  • Last Updated 24/02/2023 08:13

Lumen/Level3, one of our upstream carriers has scheduled a maintenance window to repair and replace essential fiber which may impact our Chicago network. Ideally traffic should automatically reroute and begin traversing over GTT until the maintenance is completed, but there is the possibility of a lapse in connectivity, packet loss, or increased latency for clients that do not peer or have optimal routes with GTT.

Should you have any questions, please email [email protected].

Chicago Downtime RFO - February 5th, 2022, February 12, 2022 (Resolved) Critical

Affecting Other - Chicago Datacenter

  • 05/02/2022 07:55 - 05/02/2022 16:27
  • Last Updated 04/03/2022 15:27

This report is to provide some information in regards to the downtime which occurred on February 5, 2022.

 

An upstream carrier (Psychz) suffered an outage in their Chicago facility as a result of an unscheduled maintenance. Per their statement, an on-site engineer was replacing faulty aggregate switch, in the process the replacement switch pre-built image upgrade performed a system reboot, which led to the pre-configured cluster commit and as a result caused reconvergence, causing the old node to reboot and clear the network configurations.

 

Network configuration backups then had to be restored over console, resulting in an increase in the resolution time.

 

Upstream carrier has requested that vendors perform image upgrades before shipping equipment to prevent this from recurring in the future.

 

Total downtime (Feb. 5, 2022) was recorded internally as: 39 minutes and 4 seconds (2022-02-05 07:55:06 - 2022-02-05 08:34:10 CST)

 

---

 

February 12, 2022 - Psychz began unscheduled maintenance again on this faulty aggregate switch, we are currently working with them to establish an ETA on resolution.

Packet Loss (Resolved) Critical

Affecting Other - C1 DALLAS

  • 13/04/2021 20:00 - 15/04/2021 17:24
  • Last Updated 14/04/2021 03:33

We are investigating ingress packet loss at C1 Dallas. This is primarily effecting our 195.X subnets.

 

A team will continue work into the morning attempting to resolve this issue, upon resolution a report will be released as to the cause and remedy.

 

In the meantime, we appreciate your patience and apologize for any inconvenience caused by this issue. sShould you need to reach out to support, please email [email protected].

AT&T Chicago Routing Issues (Resolved) Medium

Affecting Other - DRT Chicago Datacenter

  • 15/01/2021 00:00 - 24/01/2021 00:00
  • Last Updated 24/01/2021 21:56

Over the past few weeks we have been receiving numerous reports of routing issues to our Chicago datacenter. We requested a number of MTRs and Traceroutes from customers to identify the routes customers were taking to reach the servers.

Nearly every traceroute we received with high latency or packet loss originated from customers who have AT&T provided internet service. As a result, we have diverted all AT&T subnets (AS7018) to GTT. This should result in a significant improvement for those customers.

If you have any questions, or would like to report a traceroute of connectivity/routing issues please email [email protected] so we can address it.

Los Angeles Outage (700 Wilshire) (Resolved) Critical

Affecting System - LACORE1

  • 18/03/2016 13:45 - 20/03/2016 12:47
  • Last Updated 09/02/2019 19:04

We are currently expirencing an outage in our Los Angeles #1 facility, we are working to resolve the issue now.

Chicago Network Intermittence (Resolved) High
  • 02/02/2018 15:26 - 02/02/2018 18:59
  • Last Updated 09/02/2019 18:59

We are currently investigating a network issue in Chicago.

More details to follow.

UPDATE: Connectivity has been restored, total downtime was 15 minutes 7 seconds.

The cause is being investigated.

UPDATE 2: Connectivity lost again after previous update, all traffic is now being routed through Psychz into Chicago. You may notice higher latency in the meantime while staff work the issue.

We are offering SLA credits for any game/voice server in our Chicago location.

Please submit a ticket to claim your SLA credit.

London Migration - June 2018 (Resolved) High

Affecting Other - DATACENTER

  • 01/05/2018 00:00 - 09/02/2019 17:59
  • Last Updated 13/06/2018 19:32

We will be migrating some servers to a new London facility beginning early in June.

Downtime should be expected to be greater than 7 hours.

The new facility will offer additional bandwidth options, and improved reliability for our UK location.

More updates to follow as we get closer to the deadline.

London Scheduled Maintenance (Resolved) Critical

Affecting Other - LONDON CORE

  • 11/09/2017 19:00 - 02/02/2018 15:27
  • Last Updated 05/09/2017 23:34

There will be a scheduled maintenance window where connectivity will be lost at the London facility. 

Start - 11/09/2017 @ 12:00 AM Wednesday, British Summer Time (BST) End - 12/09/2017 @ 12:30 AM Wednesday, British Summer Time (BST)

Expected downtime is less than 10 minutes, a 30 minute window has been allocated to ensure proper function and additional time to troubleshoot and resolve in the event of an unexpected delay.

If you have any questions regarding this downtime please submit a ticket and your account representative will assist you.

Game Server/Voice Server - Global Maintenance (Resolved) Critical

Affecting Other - INFRAS

  • 15/05/2017 10:00 - 05/09/2017 23:27
  • Last Updated 14/05/2017 20:03

We will begin maintenance on all game and voice server machines tomorrow to complete some security patches.

This will effect ALL game and voice server clients.

Dedicated server/VPS clients will not be effected.

France Location Shutdown - March 5th, 2017 (Resolved) Medium

Affecting System - FRA-DATACENTER

  • 10/03/2017 09:22 - 13/04/2017 11:59
  • Last Updated 05/03/2017 11:18

On March 5th we will begin moving clients from our Dunkirk, France location to our new London, UK location. The London location has much better hardware on-site, and will allow us to better manage network anomalies as they arise.

This move will result in all game servers in our France location being renumbered. But we will leave a copy of your old game servers up upon request on the old IP for 5 days after the game server has been transferred to the London location so you can inform your clients by changing the title, or setting redirects.

This will be our first major maintenance of 2017 and will result in some slight downtime for each game server as we transfer the files to the new location. The new London facility offers much larger DDoS Mitigation capacity, allows us to more closely monitor our internal network, and will provide much better server hardware in the coming years since there's an on-site OEM. The latency between our France location and the new London location is about 2ms. Additionally the London facility has better connectivity and access to more diverse fiber routes.

If you have any questions, please email support [AT] swiftnode.net, and we will try our best to be accomodating during the move.

Los Angeles Router Upgrades (Resolved) Critical

Affecting Other - junos

  • 19/06/2016 18:00 - 22/08/2016 19:50
  • Last Updated 19/06/2016 19:19

This is a scheduled maintenance to perform upgrades on our Juniper routers which will improve network stability in the region. We expect less than 10 minutes of impacted service.

This will effect our entire Los Angeles facility.

Thank you.

UPDATE: 6/19/2016: Maintenance has been completed.

Dallas Network Maintenance - March 2nd, 2016. (Resolved) Medium

Affecting Other - DALLAS CORE

  • 02/03/2016 09:05 - 15/06/2016 17:24
  • Last Updated 02/03/2016 09:05

An upcoming network maintenance that will cause a network outage of less than 10 minutes has been scheduled. The relatively short notice is due to the state of one of our network switches presently in a critical state. The Juniper switch will need to be rebooted to ensure all new configurations take effect and to prevent this problem from happening in the future. 

Date: March 2, 2016
Time: 11AM CST
Duration: 10 Minutes
Effect: Network Outage
Facilities: Dallas, Texas

If you have any questions about this outage or want more information on how it will relate to your specific services with us, please feel free to email [email protected]. The outage is not expected to last more than 10 minutes, with a 5-6 minute estimate duration.

Scheduled Maintenance 2/29/2016 (Resolved) Medium

Affecting Other - LOS ANGELES DC2 CORE

  • 29/02/2016 02:00 - 28/02/2016 15:06
  • Last Updated 28/02/2016 11:58

We will be taking our Los Angeles DC2 core offline for upgrades, we expect this to only take about 15-20 minutes but we are opening a 1 hour maintenance window for any issues that occur in the downtime.

This will effected the following IP blocks:

45.35.36.32/29

If you have any questions please email [email protected].

Network Maintenance (Resolved) Medium

Affecting Other - Los Angeles DC1

  • 13/02/2016 13:00 - 28/02/2016 11:59
  • Last Updated 12/02/2016 11:55

We will be performing network maintenance on Los Angeles DC1. Any clients utilizing our Internap network may expirence a brief network timeout. We expect a total of 5 minutes but have opened a maintenance window of 15 minutes for clearance.

This will not affect Los Angeles DC2.

Unscheduled Maintenance (Resolved) Medium

Affecting Other - DALLAS CORE

  • 03/08/2015 04:00 - 12/08/2015 01:16
  • Last Updated 03/08/2015 05:12

Dallas will be undergoing to upgrades late tonight, downtime is expected to be about 15-20 minutes. The window for the maintenance is 4AM-6AM EST. during this time you may expierence frequent disconnects/timeouts or increased latency.

We appreciate your patience and understanding during the maintenance and apologize for any inconvience.

TCAdmin Status Unknown (Resolved) Medium

Affecting Server - TCAdmin Master

  • 16/03/2015 18:25
  • Last Updated 16/03/2015 22:27

Today we expirenced a routing issue to the time.windows.com server from our entire network, globally disabling our tcadmin services across the board.

After reviewing the logs for the issue (posted below) we switched over to utilizting the time.nist.gov server.


03/17/2015 03:23:02    System.Web.Services    System.Net.WebException: The request failed with HTTP status 402: Please fix your server's date and time. (2).
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at TCAdmin.SDK.Web.References.ServiceManager.ServiceManager.GetStatus(Int32 serviceId)
   at TCAdmin.SDK.Objects.Service.get_Status()


For now this appears to be resolved, we are still unsure of why the routing issues to the window's servers arose, however we do not expect to have this problem again.

Services were shown as Unknown for a little while in TCAdmin but no servers were actually down, just the controls for each process was disabled.

Thank you.

Emergency Maintenance (Resolved) High

Affecting System - DALLAS CORE

  • 05/03/2015 01:52 - 05/03/2015 06:56
  • Last Updated 05/03/2015 06:52

We're working on resolving some intermittent connection issues, you may periodically lose connection in the meantime, we apologise for any inconvenience.

Fiber Cut - DALLAS (Resolved) Critical

Affecting System - Dallas Datacenter

  • 11/02/2015 19:29
  • Last Updated 12/02/2015 03:17

We greatly apologize for the inconvenience this is causing.

This morning around 11:20 AM the main fiber run to our Data Center 
was cut by a construction crew working along the path. A crew is currently
onsite repairing the connections. We estimate services will be back online by 4 PM Central time, February 11th. We will send out an update as they come in, for now none of your services
will be online due to this cut off.


Update: 4:19PM Central Time

It appears the cut is under a driveway and they are having to dig under to get
to the cut.  The fix will not happen until after 5:30 at the
earliest now , but until they can dig all the way through to the cut they will
not have a specific time.

Update: 9:00PM Central Time

Fiber cut has been repaired and all systems are now operational. Please contact
us for payment changes, we will apply credits to your accounts as quickly as possible.