As of 21:15 PDT, the services at Amazon Web Services have returned to normal levels. Concurrently, we observed a decrease in the number of service errors. Amazon Web Services considered the incident resolved at 21:15 and posted their update to the service health dashboard at 23:08 PDT. As of now, Poll Everywhere services are operating normally. If you have any questions about this incident, please reach out to [email protected].
Posted Sep 18, 2023 - 23:31 PDT
Update
Amazon Web Services is continuing to resolve the service issue. Their last update said the following: "While network mapping propagation times have remained stable, we have not yet seen the improvement in propagation latencies that we had hoped for. In parallel, we are working on several other updates to address the resource contention within the subsystem responsible for network mapping propagation. We will continue to keep you updated on our progress towards full recovery."
Currently, at Poll Everywhere, we have disabled certain automated systems that can generate the error conditions. We have observed approximately a 1/3 to 1/2 reduction in the number of errors from Amazon Web Services after applying our mitigation measures. We will continue to apply these mitigation measures until the issue is resolved on the Amazon Web Services side. Again, we estimate the error rate to be lower than 0.01% of all requests.
Because of the prolonged nature of the issue, we will continue to monitor the situation, but, at this time, we will be providing less frequent updates as the situation has not changed over the past several hours and is changing very slowly.
If you experience any problems related to this issue, please contact [email protected].
The underlying issue that affects us is on Amazon Web Services: https://health.aws.amazon.com/health/status. The issue title is "Operational issue - Multiple services (Oregon)". We will continue to track the status of this problem as time progresses.
Posted Sep 18, 2023 - 18:42 PDT
Update
We are continuing to monitor the situation. According to Amazon Web Services, they are making progress on the issue. There has been no change to the rate of errors at this time. If you experience any difficulty as a result of this, please reach out to [email protected]. Thanks!
Posted Sep 18, 2023 - 15:10 PDT
Update
Temporary service faults are still occurring due to the issue at Amazon Web Services. We have seen no change in the system performance (good or bad) since the start of the incident. Again, we estimate that less than 0.01% of requests encounter a problem. If you are impacted by one of these errors while presenting, please contact [email protected]. At this time, we are continuing to monitor the situation at Amazon Web Services and await final restoration.
Posted Sep 18, 2023 - 14:15 PDT
Update
We're still seeing instances of errors occurring. At this time, we're continuing to monitor the situation and will continue to provide updates as we have more information. At this point, we believe the error rate to be less than 0.01%. According to automated monitoring systems, the significant majority of requests are successful.
We're experiencing an issue with Amazon Web Services in the Oregon (US-West-2) region. This may cause some intermittent errors. Refreshing the page or retrying your action may help solve the problem while we await the upstream solution.
Posted Sep 18, 2023 - 12:12 PDT
This incident affected: Third-party service providers (AWS EC2, AWS Route53) and PollEverywhere.com, PollEv.com, Apps and integrations.