2022 Rogers Communications outage

Edit links
From Wikipedia, the free encyclopedia

2022 Rogers Communications outage
Chart of withdrawals from IP prefixes from Rogers Communications on July 8, 2022, between approximately 05:00 and 16:30 UTC, as measured by Cloudflare[1]
DateJuly 8, 2022 (2022-07-08)
Time4:30 a.m. (EDT)[2]
Duration15 hours to several days
TypeNetwork outage
Outcome
  • Disruption of internet and wireless phone service
  • Government services disrupted
  • 9-1-1 calls failing
  • Nationwide loss of Interac debit services

On July 8, 2022, Canadian telecom provider Rogers Communications experienced a major service outage affecting more than 12 million users of Rogers' cable internet and cellular networks,[3] including those of subsidiary brands Rogers Wireless, Fido, Cityfone, and Chatr. This followed another major national outage a year prior in April 2021.[4]

The 2022 outage impacted internet service providers with wholesale access to the Rogers network, such as TekSavvy,[5] as well as various other information systems nationwide that relied on the Rogers network, including Interac, OLG,[6] and some federal government services. Multiple international web monitoring organizations observed the outage.

The event spurred new government policies requiring all telecommunications providers to provide mutual assistance to each other in the event of an outage, offer emergency roaming service for rivals' affected customers, and establish plans for how to communicate to the public about outages.[3]

Rebates for customers are estimated to cost Rogers $28 million[4] to $70 million.[7] Splitting their physical and wireless lines for $261 million, and giving out $150 million in customer credits became part of a $10 billion plan for Rogers over a three-year span involving increased management and additionally the use of artificial intelligence to supervise potential future errors.[8] The economic toll to the Canadian economy for the outage period was projected at $142 million.[9]

Impact[edit]

The outage had a significant impact on various information systems nationwide, much of which extended past residential and commercial Rogers customers, due to their dependencies on Rogers infrastructure:

  • Around 25% of Canada lost internet connectivity, according to the UK cybersecurity organization Netblocks.[10][11] Over 12 million subscribers were affected.[3]
  • Interac was taken offline by the outage, preventing all businesses nationwide from being able to accept debit card transactions, regardless of their internet service provider. Some stores temporarily closed.[5]
  • The outage inhibited the ability to use 911 services from mobile phones on the Rogers network,[5] resulting in at least one reported case of a death. It is unclear as to whether or not the death could have been avoided had 911 contact been possible.[12]
  • Along with the inability mentioned above to accept debit, some public transport agencies reported other computer system issues tied to the outage.[5]
  • The outage affected some Government of Canada services, such as Service Canada, Canada Revenue Agency, and passport offices.[13]
  • The Canadian Federation of Independent Business (CFIB) reported small businesses lost anywhere from a few hundred to several thousands of dollars.[14]
  • In Toronto there was some dependency on Rogers. One quarter of all traffic signals relied on their cellular network for signal timing changes. The Rogers GSM network was also used to remotely monitor fire alarms and sprinklers in municipal buildings. Public parking payments and public bike services were also unavailable. It was also the sole provider of cable TV for the city.[15]
  • Medical appointments through the Niagara Health System for oncology patients with radiation therapy were rescheduled, with a plan to transfer patients requiring emergency care to Hamilton Health Sciences.[16]
  • Canadian singer-songwriter the Weeknd was forced to postpone a concert that night at Toronto's Rogers Centre on the After Hours til Dawn Tour, due to "service outages impacting venue operation".[17] The concert was later rescheduled for September 22, 2022. A second show at Rogers Centre was added for September 23.[18]

Recovery[edit]

Rogers had begun to slowly restore service that evening, but CEO Tony Staffieri stated there was no estimated time for when services would become fully operational again.[19] The next day, Rogers stated that it had restored service to the "vast majority" of its customers; however, not all service had been restored across the country.[20] On July 11, a spike of service issues was reported to the web traffic monitoring company Ookla's Downdetector, with some users reporting that service had not recovered for them.[21][22]

Causes[edit]

A report by Cloudflare suggested that the outage was due to internal, rather than external, causes. It identified spikes in BGP updates, as well as withdrawals of IP prefixes, noting that Rogers was not advertising its presence, causing other networks to not find the Rogers network.[1] As of the day after the outage, the cause remained unknown. Public Safety Canada stated that it was not a cyberattack.[23] The outage was later said to be caused by a maintenance upgrade that caused routers to malfunction.[24] A similar outage occurred in April 2021, which was attributed to a software update.[4]

Postmortem[edit]

In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. The deletion occurred during the sixth phase in a seven-phase update to its core IP network. The failure occurred at 4:45 a.m. and resulted in a nationwide loss of access to the company's phone and internet services, as well as internal access to systems such as the company's VPN to its core network nodes, hampering the ability of the company's employees to mobilize a team and identify the issue. Some employees were able to connect through “emergency SIMs” on alternate networks, which they had due to a practice established through reciprocal agreements made with other carriers in 2015. Others traveled to centralized locations to establish network access.

At 6 a.m., the company's CTO Jorge Fernandes, who has since been replaced, contacted leadership at rival companies Bell and Telus, advising them of the issues they were facing, as well as warning "to watch out for possible cyber-attacks." Both competitors offered assistance to Rogers, though the company determined that it was not possible to switch its users over to alternate networks, citing the inability to access its user database, as well as the excess load that the sudden addition of more than 10 million users would put on the other networks. A Memorandum of Understanding on how the carriers can work together in the future was submitted in September to the Minister of Innovation, Science and Economic Development.[25] Rogers plans to physically separate its wireless and wireline core networks to help mitigate the fallout of future outages.[26]

Reactions[edit]

Critics have cited the outage as an example of the impact of Canada's telecom oligopoly, and an argument against Rogers' proposed acquisition of Shaw Communications (which would further consolidate the industry).[27][19] NDP leader Jagmeet Singh accused the Liberal Party government of "protecting the profits of telecoms giants",[28] while Conservative MP Michelle Rempel Garner called for an investigation, arguing that "given the critical infrastructure that's affected, and that because the Canadian Radio-television and Telecommunications Commission (CRTC) itself is affected, the cause of the Rogers outage should be immediately explained."[28] A spokesperson for the CRTC promised that the organization would examine the outage and "put in place the necessary measures to prevent it from happening again."[10]

Critics also highlighted the importance of consumers and businesses diversifying their telecommunications services, such as businesses having backup service contracts with multiple providers and consumers reducing their dependence on bundled "all-in-one" cable, internet and phone service packages.[28]

Response[edit]

The Minister of Innovation, Science and Industry, François-Philippe Champagne, in 2022. Champagne met with the CEOs of Canada's major telecom companies on July 11, shortly after the outage began.

Rogers planned to credit customers for five days of lost service, which was to be credited August 1.[26][29] On July 10, 2022, Tony Staffieri, president and CEO of Rogers Communications, sent an email to all customers to officially apologize for the incident and inform them that Rogers would credit all customers' accounts for Friday.[30][31] The last time they credited consumers on such a scale was soon after their April 19, 2021 outage, when "wireless calls, SMS and data services were down across Canada for almost an entire day because of an issue with a software update".[32]

On July 8, the same day as the outage, an internal email was sent requesting that retail locations remove signage that contained the slogan “Canada’s Most Reliable 5G Network.”[33]

On July 11, the CEOs of Rogers, Bell, Telus, Shaw, Vidéotron, SaskTel, and EastLink were all invited to take part in a conference call with federal cabinet minister François-Philippe Champagne to discuss a new "mutual assistance" framework of cooperation between the companies in emergencies to prevent the recurrence of a similar incident in the future.[34] On the same day, Champagne announced new policies requiring all telecommunications providers to provide mutual assistance to each other in the event of an outage, offer emergency roaming service for rivals' affected customers, and establish plans for how to communicate to the public about said incidents. Champagne said that the companies would have 60 days to reach agreements with each other to that effect.[3]

The House of Commons of Canada were discussing requests to conduct a study of the outage involving members of parliament from the Liberal, Conservative, New Democratic Party, and the Bloc Québécois.[35]

A class action lawsuit was filed in the Superior Court of Quebec which requested all customers receive a $400 rebate, reasons cited included negligence of the CRTC regulations to make 9-1-1 calls available at all times.[36][33]

The CFIB requested Rogers to give a free month of service to compensate for losses to others.[7]

As a result of the outage, the city government of Toronto reviewed its legacy contract for sourcing with a variety of providers. Council meetings were planned to review the full cost to the economy and the ability to use local ConnectTO in the city's infrastructure instead in the future.[15]

On July 21, it was reported that the Chief Technology and Information Officer, Jorge Fernandes, was to be replaced by Ron McKenzie, who was the former president of Rogers for Business. The leadership team was updated the previous night on the Rogers Website to reflect this change.[37]

See also[edit]

References[edit]

  1. ^ a b Tomé, João; Strickx, Tom; Zhang, Mingwei (July 8, 2022). "Cloudflare's view of the Rogers Communications outage in Canada". The Cloudflare Blog. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  2. ^ "Rogers starts restoring service after outage, CEO apologizes to customers". Cornwall Seaway News. July 9, 2022. Retrieved July 10, 2022.[permanent dead link]
  3. ^ a b c d "Ottawa announces it will require telecoms to provide backup for each other during outages following Rogers system failure". thestar.com. July 11, 2022. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  4. ^ a b c Evans, Pete (July 8, 2022). "Rogers says services mostly restored after daylong outage left millions offline". CBC News. Archived from the original on July 11, 2022. Retrieved July 11, 2022.
  5. ^ a b c d da Silva, Michelle; Augustin, Mathilde (July 8, 2022). "The Rogers outage disrupted services across Canada. A list of what was affected". The Globe and Mail. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  6. ^ "Rogers says it has restored service for 'vast majority' of customers". thestar.com. July 8, 2022. Archived from the original on September 4, 2022. Retrieved September 4, 2022.
  7. ^ a b Alini, Erica; Hannay, Chris (July 12, 2022). "Consumers and businesses expected to take Rogers to court over network outage". The Globe and Mail. Archived from the original on July 13, 2022. Retrieved July 13, 2022 – via www.theglobeandmail.com.
  8. ^ "Rogers to spend $261M to split networks, but can't quantify economic impact of outage". CTVNews. August 25, 2022. Retrieved November 3, 2022.
  9. ^ Bloomberg, B. N. N. (July 12, 2022). "Rogers pledges five-day credits as Bay Street weighs outage impact - BNN Bloomberg". BNN.
  10. ^ a b "Rogers CEO apologizes for massive service outage, blames maintenance update". CBC News. July 9, 2022. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  11. ^ "Netblocks tweet on Rogers outage". Twitter. July 8, 2022. Archived from the original on July 8, 2022. Retrieved July 10, 2022.
  12. ^ Hristova, Bobby (July 12, 2022). "Hamilton man was unable to call 911 during Rogers outage as sister was dying". CBC News. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  13. ^ Lopez-Martinez, Melissa; Lee, Michael (July 8, 2022). "Rogers wireless services 'starting to recover,' company says". CTVNews. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  14. ^ "Rogers outage cost small businesses thousands of dollars: CFIB". July 11, 2022. Archived from the original on July 11, 2022. Retrieved July 13, 2022.
  15. ^ a b "City of Toronto eyeing ways to reduce reliance on Rogers after outage". Toronto Star. July 11, 2022. Archived from the original on July 14, 2022. Retrieved July 13, 2022.
  16. ^ Rosas, Aura; Hristova, Bobby. "Businesses, doctors, DARTS service in Hamilton and Niagara among those hit by massive Rogers outage". CBC News. Archived from the original on July 9, 2022. Retrieved July 10, 2022.
  17. ^ Morton, Michelle; Casaletto, Lucas (July 8, 2022). "Weeknd postpones concert in Toronto as Canadians face Rogers outage". CityNews. Archived from the original on July 9, 2022. Retrieved July 8, 2022.
  18. ^ Live Nation Ontario, (@LiveNationON) (July 18, 2022). ".@theweeknd show at Rogers Centre has been rescheduled to Thursday, September 22nd. Please hold onto your tickets as they will be valid for the new date". Twitter. Archived from the original on July 18, 2022. Retrieved July 18, 2022.
  19. ^ a b Karadeglija, Anja (July 8, 2022). "Critics call for Parliament, CRTC to immediately investigate Rogers outage". National Post. Retrieved July 9, 2022.
  20. ^ Roth, Emma (July 9, 2022). "Rogers restores service for 'vast majority' of customers after massive outage". The Verge. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  21. ^ Phan Nay, Isaac (July 11, 2022). "It's day 4 of the Rogers outage. Here's 4 things you need to know". thestar.com. Archived from the original on July 11, 2022. Retrieved July 12, 2022.
  22. ^ "Rogers down? Realtime status and problems overview". downdetector.ca. July 12, 2022. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  23. ^ "Rogers network resuming after major outage hits millions of Canadians". Reuters. July 8, 2022. Archived from the original on July 9, 2022.
  24. ^ Callan, Isaac (July 9, 2022). "Rogers CEO apologizes, says 'maintenance upgrade' behind major outage". Global News. Archived from the original on July 10, 2022. Retrieved July 10, 2022.
  25. ^ "Memorandum of Understanding on Telecommunications Reliability". Innovation, Science and Economic Development Canada. Archived from the original on September 7, 2022. Retrieved December 1, 2022.
  26. ^ a b Shecter, Barbara (July 24, 2022). "Rogers shares explanation after 'unprecedented' outage". Financial Post. Archived from the original on July 25, 2022. Retrieved July 24, 2022.
  27. ^ Deschamps, Tara (July 8, 2022). "Far-reaching implications of Rogers outage shows need for competition: Expert". CityNews Ottawa. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  28. ^ a b c Logan, Nick (July 8, 2022). "Rogers outage shows need for Plan B when wireless, internet services fail, analysts say". CBC News. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  29. ^ Paperny, Anna Mehler; Rajagopal, Divya (July 12, 2022). "Canada's Rogers to credit customers with 5 days service after massive outage". Reuters. Archived from the original on July 13, 2022. Retrieved July 13, 2022.
  30. ^ Staffieri, Tony (July 10, 2022). "A Message from Rogers President and CEO". Rogers. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  31. ^ Helena Hanson (April 21, 2021). "Here's Everything You Need To Know About Claiming Rogers Credit After The Outage". Narcity Media. Archived from the original on July 8, 2022. Retrieved July 10, 2022.
  32. ^ Lisa Belmonte (July 9, 2022). "The Last Canada-Wide Rogers Outage Resulted In Credits For Customers & This Is How It Worked". Narcity Media. Archived from the original on July 12, 2022. Retrieved July 10, 2022.
  33. ^ a b Zukra, Joey (July 11, 2022). "Application to Authorize the Bringing of A Class Action and to Appoint the Status of Representative Plaintiff" (PDF). LPC Avocat Inc. Archived (PDF) from the original on July 11, 2022. Retrieved July 24, 2022.
  34. ^ Alexandra Posadzki, "Rogers outage prompts meeting with telecom execs, Industry Minister to discuss ‘mutual assistance’ framework, sources say" Archived July 11, 2022, at the Wayback Machine. The Globe and Mail, July 11, 2022.
  35. ^ "Rogers to credit customers with five days of service as it faces scrutiny from CRTC after major outage". The Globe and Mail. July 12, 2022. Archived from the original on July 13, 2022. Retrieved July 13, 2022 – via www.theglobeandmail.com.
  36. ^ "Rogers customers in Toronto could get $400 rebate thanks to class action suit over outage". www.blogto.com. Archived from the original on July 13, 2022. Retrieved July 13, 2022.
  37. ^ Willis, Andrew (July 21, 2022). "Rogers replaces chief technology officer amid the fallout from nationwide outage". The Globe and Mail. Archived from the original on July 21, 2022. Retrieved July 21, 2022 – via www.theglobeandmail.com.
Retrieved from "https://en.wikipedia.org/w/index.php?title=2022_Rogers_Communications_outage&oldid=1217628418"