This page includes all updates regarding the nationwide Bandwidth service outage that is affecting many local providers like CTS Cloud.
——
Update 10/3 1:04pm: Plan And Timing For Resolution
A few updates:
- 95% of the ports to the new carrier went through since Friday, with that last 5% currently being worked on.
- Toll Free Numbers are still in-progress.
- The original carrier (Bandwidth.com) continues to see sustained success after implementing new DDoS protection technology, but we’ll continue to monitor.
What all this means is that your phone service should be working as normal. If you still do have issues please email our support team at service@cts-companies.com and we will investigate.
On a personal note, I can’t imagine the disruption that this has caused your organization this week. We at CTS have succeeded in this business for 41 years because we take quality of service very seriously, so despite us largely having to rely on the carrier this week (including to leave them), we do apologize for the issues. We sincerely appreciate your patience and understanding and we will continue to ensure we fix any current and longterm issues.
Update 9/30 10:23pm: Plan And Timing For Resolution
We have quite a few updates this evening, a lot of it is the communication we’re getting from our technical partner and upstream carrier. The two biggest takeaways are as follows:
- 1. Numbers have started porting to the new carrier as planned, but may take a few days given the volume of numbers attempting to port away from Bandwidth.com (this is obviously longer than we were hoping for) – if we receive word of successful ports we will definitely pass that along.
- 2. Bandwidth.com (the current carrier) has been able to stabilize large portions of their network and cyber attacks on their service is at 20% of the peak due to successful mitigation techniques. So, while we’re waiting for the numbers to port, your phone service should have improved, but we (CTS) would still like to see a few days of sustained and improved performance before fully trusting the carriers communication.
Further details:
-
We received notification from the new carrier that the numbers that we provided have begun porting tonight and will continue until all numbers activated and complete, which may continue through the weekend based on the large backlog of queued porting.
- Toll Free is still in progress, having been scrubbed and we will update when confirmed that those Toll Free numbers will start porting, as well.
- Confirmation of individual porting is not possible, we will simply receive a “Completed” notice.
- Briefing from Bandwidth included the fact that they continue to have probing attacks. The original attacks were voluminous attacks to fill the Bandwidth network which were successful. As Bandwidth continued with additional mitigation and blocked the attacks, the hackers are now probing their network looking for an opening and their mitigation have successfully blocked these attempts.
- Bandwidth states that the attacks have significantly been reduced at 20% from peak.
-
Bandwidth was triaging one-way audio issues this afternoon, which they determined that half the issues were related to a small carrier that they disabled from their network, and another 25% were on other Bandwidth platforms.
Update 9/29 9:14pm: Plan And Timing For Resolution
We were able to confirm today that the new carrier accepted the port request (they informally confirmed yesterday evening and formally confirmed a few hours ago) and so our expectation is still that service will be ported over tomorrow (9/30) around 6pm. I’ve also included the previous updates below for those who have not received them. We will continue to work with partners towards the solution and keep you updated as things progress.
Update 9/29 8:09am: Plan And Timing For Resolution
For further clarification, we’re targeting a completed transition to the new carrier by 6p EST on Thursday 9/30. We will keep you updated as we learn more.
Update 9/28 6:07pm – Plan And Timing For Resolution
After a lack of timing expectations from Bandwidth and many conversations with our technical partner, we’ve put together a tentative plan to port customer numbers to a different carrier with a targeted completion date of end of day Thursday 9/30. The transition should not cause any further disruption to your service, and upon completion service should resume back to normal. While our primary focus right now is immediate restoration of service, we are also building plans to prevent future reliance on a single carrier and will share that at a later date. As always, we will do our best to keep you updated as things progress.
Update 9/28 2:22pm
We just got off a call with one of our partners that assists in managing the upstream carrier (Bandwidth). Given the continued cyber attacks on Bandwidth, they’re looking at the technical options to use other carriers and will let us know the feasibility of that in the next 4-6 hours, at which point we will send out another update with that information (ie hopefully a plan, timing, etc).
Update 9/28 10:46am
The upstream carrier has officially acknowledged a systemwide issue. We’ve been working with a number of partners over the last 24 hours to route as much outbound calling to a different carrier as possible. What this means is that calling out should work better than receiving calls. Please note that the carrier (Bandwidth) with the systemwide issues supports many organizations across America and Michigan and so you may be calling one of their other customers (so outbound will be better, but not perfect). Furthermore, you can look up your received/missed calls in your portal which will allow you to call your customers back (https://ctscloud.uccommportal.com). No current ETA on resolution, however as always we will keep you updated.
Update 9/28 10:08am
Even though the upstream carrier communicated they’ve resolved the issues it appears some customers are still experiencing issues. We’ve reached out to the carrier and they’ve communicated there may still be issues and they’re currently working on it. No current ETA, however we will keep you updated.
Update 9/28 8:05am
The intermittent issues with your phone service yesterday have been resolved (as of 8p EST last night). The issue was caused by a large DDoS (cyber) attack on a firm called Bandwidth.com, which is a major trunking company used by long list of carriers (CTS Cloud, Microsoft Teams, Google Voice, Ring Central, Zoom, and more). Bandwidth has not released any final report, but everything has been working and they continue to monitor the situation. We apologize for the inconvenience this caused you.
Update 9/27
There’s currently intermittent issues with CTS Cloud resulting from problems with carrier networks across the United States. We’re investigating and working with the carriers to resolve the issue (this includes Seats & SIP Trunks). We will keep you updated of the progress.