Affecting Other - FACILITY
We are currently experiencing a network outage in our Chicago Datacenter at 600 South Federal.
The network at this facility backhauls from Crown Castle to 350 E Cermak (Equinix CH1).
Equinix CH1 is currently experiencing issues with their HVAC systems as a result of extreme weather.
They are supplying outside air to the 5th floor to counteract the heat load. More details for this outage can be found on Equinix's status page.
Once the issue is resolved and equipment is brought back online in Equinix CH1, connectivity to our cabinets should be restored in 600 South Federal.
Affecting Other - IL Fiber
Lumen/Level3, one of our upstream carriers has scheduled a maintenance window to repair and replace essential fiber which may impact our Chicago network. Ideally traffic should automatically reroute and begin traversing over GTT until the maintenance is completed, but there is the possibility of a lapse in connectivity, packet loss, or increased latency for clients that do not peer or have optimal routes with GTT.
Should you have any questions, please email [email protected].
Affecting Other - Chicago Datacenter
This report is to provide some information in regards to the downtime which occurred on February 5, 2022.
An upstream carrier (Psychz) suffered an outage in their Chicago facility as a result of an unscheduled maintenance. Per their statement, an on-site engineer was replacing faulty aggregate switch, in the process the replacement switch pre-built image upgrade performed a system reboot, which led to the pre-configured cluster commit and as a result caused reconvergence, causing the old node to reboot and clear the network configurations.
Network configuration backups then had to be restored over console, resulting in an increase in the resolution time.
Upstream carrier has requested that vendors perform image upgrades before shipping equipment to prevent this from recurring in the future.
Total downtime (Feb. 5, 2022) was recorded internally as: 39 minutes and 4 seconds (2022-02-05 07:55:06 - 2022-02-05 08:34:10 CST)
---
February 12, 2022 - Psychz began unscheduled maintenance again on this faulty aggregate switch, we are currently working with them to establish an ETA on resolution.
Affecting Other - C1 DALLAS
We are investigating ingress packet loss at C1 Dallas. This is primarily effecting our 195.X subnets.
A team will continue work into the morning attempting to resolve this issue, upon resolution a report will be released as to the cause and remedy.
In the meantime, we appreciate your patience and apologize for any inconvenience caused by this issue. sShould you need to reach out to support, please email [email protected].
Affecting Other - DRT Chicago Datacenter
Over the past few weeks we have been receiving numerous reports of routing issues to our Chicago datacenter. We requested a number of MTRs and Traceroutes from customers to identify the routes customers were taking to reach the servers.
Nearly every traceroute we received with high latency or packet loss originated from customers who have AT&T provided internet service. As a result, we have diverted all AT&T subnets (AS7018) to GTT. This should result in a significant improvement for those customers.
If you have any questions, or would like to report a traceroute of connectivity/routing issues please email [email protected] so we can address it.
Affecting System - LACORE1
We are currently expirencing an outage in our Los Angeles #1 facility, we are working to resolve the issue now.
We are currently investigating a network issue in Chicago.
More details to follow.
UPDATE: Connectivity has been restored, total downtime was 15 minutes 7 seconds.
The cause is being investigated.
UPDATE 2: Connectivity lost again after previous update, all traffic is now being routed through Psychz into Chicago. You may notice higher latency in the meantime while staff work the issue.
We are offering SLA credits for any game/voice server in our Chicago location.
Please submit a ticket to claim your SLA credit.
Affecting Other - DATACENTER
We will be migrating some servers to a new London facility beginning early in June.
Downtime should be expected to be greater than 7 hours.
The new facility will offer additional bandwidth options, and improved reliability for our UK location.
More updates to follow as we get closer to the deadline.
Affecting Other - LONDON CORE
There will be a scheduled maintenance window where connectivity will be lost at the London facility.
Start - 11/09/2017 @ 12:00 AM Wednesday, British Summer Time (BST) End - 12/09/2017 @ 12:30 AM Wednesday, British Summer Time (BST)
Expected downtime is less than 10 minutes, a 30 minute window has been allocated to ensure proper function and additional time to troubleshoot and resolve in the event of an unexpected delay.
If you have any questions regarding this downtime please submit a ticket and your account representative will assist you.
Affecting Other - INFRAS
We will begin maintenance on all game and voice server machines tomorrow to complete some security patches.
This will effect ALL game and voice server clients.
Dedicated server/VPS clients will not be effected.
Affecting System - FRA-DATACENTER
On March 5th we will begin moving clients from our Dunkirk, France location to our new London, UK location. The London location has much better hardware on-site, and will allow us to better manage network anomalies as they arise.
This move will result in all game servers in our France location being renumbered. But we will leave a copy of your old game servers up upon request on the old IP for 5 days after the game server has been transferred to the London location so you can inform your clients by changing the title, or setting redirects.
This will be our first major maintenance of 2017 and will result in some slight downtime for each game server as we transfer the files to the new location. The new London facility offers much larger DDoS Mitigation capacity, allows us to more closely monitor our internal network, and will provide much better server hardware in the coming years since there's an on-site OEM. The latency between our France location and the new London location is about 2ms. Additionally the London facility has better connectivity and access to more diverse fiber routes.
If you have any questions, please email support [AT] swiftnode.net, and we will try our best to be accomodating during the move.
Affecting Other - junos
This is a scheduled maintenance to perform upgrades on our Juniper routers which will improve network stability in the region. We expect less than 10 minutes of impacted service.
This will effect our entire Los Angeles facility.
Thank you.
UPDATE: 6/19/2016: Maintenance has been completed.
Affecting Other - DALLAS CORE
An upcoming network maintenance that will cause a network outage of less than 10 minutes has been scheduled. The relatively short notice is due to the state of one of our network switches presently in a critical state. The Juniper switch will need to be rebooted to ensure all new configurations take effect and to prevent this problem from happening in the future. Date: March 2, 2016 Time: 11AM CST Duration: 10 Minutes Effect: Network Outage Facilities: Dallas, Texas
If you have any questions about this outage or want more information on how it will relate to your specific services with us, please feel free to email [email protected]. The outage is not expected to last more than 10 minutes, with a 5-6 minute estimate duration.
Affecting Other - LOS ANGELES DC2 CORE
We will be taking our Los Angeles DC2 core offline for upgrades, we expect this to only take about 15-20 minutes but we are opening a 1 hour maintenance window for any issues that occur in the downtime.
This will effected the following IP blocks:
45.35.36.32/29
If you have any questions please email [email protected].
Affecting Other - Los Angeles DC1
We will be performing network maintenance on Los Angeles DC1. Any clients utilizing our Internap network may expirence a brief network timeout. We expect a total of 5 minutes but have opened a maintenance window of 15 minutes for clearance.
This will not affect Los Angeles DC2.
Affecting Other - DALLAS CORE
Dallas will be undergoing to upgrades late tonight, downtime is expected to be about 15-20 minutes. The window for the maintenance is 4AM-6AM EST. during this time you may expierence frequent disconnects/timeouts or increased latency.
We appreciate your patience and understanding during the maintenance and apologize for any inconvience.
Affecting Server - TCAdmin Master
Today we expirenced a routing issue to the time.windows.com server from our entire network, globally disabling our tcadmin services across the board.
After reviewing the logs for the issue (posted below) we switched over to utilizting the time.nist.gov server.
Affecting System - DALLAS CORE
We're working on resolving some intermittent connection issues, you may periodically lose connection in the meantime, we apologise for any inconvenience.
Affecting System - Dallas Datacenter
We greatly apologize for the inconvenience this is causing. This morning around 11:20 AM the main fiber run to our Data Center
was cut by a construction crew working along the path. A crew is currently
onsite repairing the connections. We estimate services will be back online by 4 PM Central time, February 11th. We will send out an update as they come in, for now none of your services
will be online due to this cut off.
Update: 4:19PM Central Time
It appears the cut is under a driveway and they are having to dig under to get
to the cut. The fix will not happen until after 5:30 at the
earliest now , but until they can dig all the way through to the cut they will
not have a specific time.
Update: 9:00PM Central Time
Fiber cut has been repaired and all systems are now operational. Please contact
us for payment changes, we will apply credits to your accounts as quickly as possible.
Our network, services, and support all work toward exceeding your needs.