AWS outage highlights: Snapchat worst hit as disruption persists

This blog has been closed and will not be updated.

Amazon said it was making progress in the latest update about the AWS update.

“We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services,” the company said in an update shared on its website.

“Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely."

Starbucks has confirmed its app experienced disruptions earlier in the day but says it is now working again.

"I can confirm that our mobile order ahead app and pay features are all operating normally," a spokeswoman for the company told BBC.

"We experienced very limited interruption and were back up and running quickly."

Amazon said that it is seeing recovery after massive AWS disruption

“We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services,” the update said.

“We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches".

While just three tech giants account for more than half of the global cloud market, AWS leads the field by a significant margin.

According to the data firm Statista, AWS alone makes up 30 percent of the global market share, followed by 20 percent for Microsoft Azure and 13 percent for Google Cloud.

They are trailed by other firms such as Alibaba, Oracle, Salesforce, IBM, and Tencent, which make up between two and four percent each.

The web monitor tracking the effect of the AWS outage has released new figures of the number of people worldwide who have reported issues with their connectivity.

In a post on social media, it said more than 8 million people have reported disruptions, including nearly 1.9 million in the US and more than a million in the UK, while the Netherlands, Germany and Australia each saw about half a million people reporting disruptions.

Amazon said it had ‘narrowed down’ to the root cause of the connectivity issues plaguing the AWS systems.

“We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations,” the statement read.

Amazon is facing issues within the EC2 internal network.

“We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations,” the company said.

Amazon is now reporting "significant" errors for some services.

But it looks like most of the issues are affecting those in the US this time round. Thousands of people have reported issues using many services in the country, including the payment provider Venmo, rewards app Fetch and shopping tool Instacart.

Latest updates from Amazon suggest more issues with AWS.

“We have confirmed multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for the connectivity issues and are continuing to investigate the root cause. We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region. We are investigating and will provide further update in 30 minutes or soon if we have additional information,” the latest updates revealed.

A spokesperson for Lloyds Banking Group has confirmed that all its systems – including online banking for Halifax and Bank of Scotland – are now running as normal.

Speaking to Radical with Amol Rajan on BBC Radio 4, enterpreneur Matthew Prince said that although there are "amazing things" about relying on cloud sharing technology, "if you have an outage like this it can take down a lot of services we rely on.

“Some of the companies were able to stay online, but that comes with complexity and cost and not everyone makes those decisions," he added.

Ultimately though, Prince said, today's Amazon outage "is not a big deal".

Head of enterprise at Checkpoint Software Technologies Charlotte Wilson told the BBC that internet outages can trigger a rise in scams or phishing attempts.

“Artificial intelligence has been used to tailor scam emails to make them feel more personal in moments like this,” she said, referring to emails that may provide fake discounts apologising for the earlier disruption.

“Victims of critical infrastructure are usually only contacted directly if their personal information has been stolen,” Wilson continued, "but users should always go to the company directly before clicking on any links".

Wilson reiterated that the AWS outage is a reminder that individuals must save key information offline and back-up important data in more than one place, like on an external hard drive.

“Have multiple backups and don’t be reliant on singular service providers,” she said.

Amazon is the biggest cloud computing provider. Microsoft Azure is a big one used by businesses, particularly those already using Microsoft’s apps like Outlook and Teams.

Google, IBM and Oracle all also offer these services. There are also some smaller companies out there

Some NHS services were affected in the UK by the AWS outage, although so far any disruption appears to be limited.

Amazon has said that it is making progress in resolving the remaining issues as AWS services crawl back to normal.

“We are making progress on resolving the issue with new EC2 instance launches in the US-EAST-1 Region and are now able to successfully launch new instances in some Availability Zones. We are applying similar mitigations to the remaining impacted Availability Zones to restore new instance launches. As we continue to make progress, customers will see an increasing number of successful new EC2 launches. We continue to recommend that customers launch new EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ,” the latest update stated.

Amazon has said that it has resolved another issue as they continue working towards full recovery.

“We confirm that we have now recovered processing of SQS queues via Lambda Event Source Mappings. We are now working through processing the backlog of SQS messages in Lambda queues,” the company said in a update.

DownDetector has said that globally so far today, more than 6.5 million outage reports have been recieved, affecting more than 1,000 companies.

BBC reported that Snapchat is up and running again - but some users said that they lost all their friends on the app.

Snapchat was among the platforms experiencing issues this morning amid the AWS outage.

Amazon has issued fresh recomendations as it continues full recovery of the AWS systems.
“We recommend EC2 Instance launches that are not targeted to a specific Availability Zone (AZ) so that EC2 has flexibility in selecting the appropriate AZ. The impairment in new EC2 launches also affects services such as RDS, ECS, and Glue. We also recommend that Auto Scaling Groups are configured to use multiple AZs so that Auto Scaling can manage EC2 instance launches automatically,” the statement said.

The British National Rail has said that it is working with suppliers to resolve the problem.

“Work is ongoing to the systems that are still being impacted following the earlier global IT issues. We are working with suppliers to do this as quickly as possible. Please note, until these systems are fully restored, some of the information displayed on the National Rail Enquiries website / app may still be out of date,” the statement said.

The British National Rail website and app have been affected by the AWS outage.

According to the BBC, the Rail Delivery group says: "Due to technical issues with our infrastructure provider this morning, the National Rail website and app have intermittently been impacted by slower speeds and difficulty loading."

As many other sites appear to be recovering from the Amazon Web Services issues, Reddit is curiously seeing a spike in reports of problems from users.

Downdetector saw more than 5,000 reports when other sites started getting fixed - with most flagging problems affecting its app.

The New York Times reported that most websites and apps that were experiencing disruption are working normally now, and the underlying issue has been mitigated, though there may be some lingering delays.

Amazon said in an earlier update that the issue "appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1."

DNS, which stands for Domain Name System, is often likened to a phone book for the internet.

It effectively translates the website names people use into IP address numeric equivalents that can be read and understood by computers.

This process basically underpins the way we use the Internet, and disruptions to it can cause web browsers to be unable to locate the content they are looking for.