How EverFold Stayed Online During the AWS Outage: A Technical Deep Dive
When AWS went down on October 20th, 2025, the impact was immediate. Major platforms went dark, airlines struggled with bookings, banks had service disruptions. EverFold customers kept creating and ordering their colouring books without noticing anything was wrong.
This wasn't luck. We built the system this way from the start. Here's what happened and why it worked.
The Event-Driven Architecture Advantage
At the core of EverFold's resilience is event-driven architecture. Instead of a monolith where everything depends on everything else, we built loosely coupled services that talk to each other through events.
When a customer completes a purchase, the system doesn't try to do everything at once. It follows a sequence:
Immediate Response: The Stripe webhook responds to payment confirmation within milliseconds, acknowledging receipt before any heavy processing begins. This prevents timeout failures that plagued other services during the outage.
Asynchronous Processing: Using Next.js 15's after() function, we process orders in
the background. PDF generation, database updates, and print job creation happen independently of the initial
webhook response.
Graceful Degradation: If one component experiences delays (like S3 during the outage), other components continue functioning. Orders are recorded, customers are notified, and processing continues as services recover.
Idempotency Everywhere: Every operation is designed to be safely retried. Duplicate webhook calls, network retries, and service restarts can't create duplicate orders or charges.
Multi-Layer Idempotency Protection
During network instability, services often retry failed requests. Without proper safeguards, this could lead to customers being charged multiple times or orders being duplicated. EverFold implements multiple layers of idempotency protection.
We track every Stripe event ID in DynamoDB. If a webhook is delivered multiple times (common during outages), we detect and skip duplicate processing immediately. Each payment intent is recorded with conditional writes to DynamoDB. If two webhooks race to process the same payment, only one succeeds, the other gracefully acknowledges the existing record.
Using DynamoDB's conditional expressions, we ensure that critical state changes (like marking an order as "processing") happen atomically, preventing race conditions even under high concurrency.
Strategic Service Decoupling
EverFold's architecture separates concerns across multiple AWS services, each chosen for specific resilience characteristics. Our order and book data lives in DynamoDB, AWS's most resilient database service. With automatic multi-AZ replication and 99.999% availability SLA, it remained operational throughout the outage.
Customer images and generated PDFs are stored in S3 with presigned URLs. Even if S3 experienced brief delays, our system queued operations and retried automatically. Email confirmations are sent via Amazon SES, but failures don't block order processing. Emails are logged to S3 for audit trails and can be resent if delivery fails.
Our integration with a printing supplier's service includes comprehensive error handling. If the print job creation fails, PDFs are already safely stored and the job can be recreated manually or automatically.
The Power of Eventual Consistency
One of our key decisions was embracing eventual consistency instead of demanding immediate consistency. This helps during outages. When a customer completes checkout, they get immediate confirmation that their payment went through. Behind the scenes, the order moves through stages: PDF generation, print job creation, email notification. Each stage is independent.
If S3 is slow during an outage, PDF generation might take longer than usual. But the customer's order is already recorded, their payment is captured, and they've got their confirmation. The system finishes the remaining steps when services recover. Logging ensures nothing falls through the cracks.
Real-World Impact: October 20th, 2025
When the AWS outage hit on October 20th, the system did what it was built to do. While major platforms struggled, our architecture handled the pressure.
The Washington Region Challenge
The outage hit AWS's US-East-1 (Northern Virginia) region hardest. For EverFold, this meant US customer orders initially failed to process as our primary infrastructure in the Washington region became unavailable.
But this is exactly the scenario our architecture was built for. Rather than losing orders or making customers retry, the system handled it:
Stripe Webhook Resilience: Stripe's webhook system automatically queued failed delivery attempts. When our services couldn't respond due to the regional outage, Stripe held onto the payment confirmation events rather than discarding them.
Automatic Failover: Our infrastructure includes cross-region failover capabilities. When the Washington region experienced issues, traffic was automatically routed to our secondary region in Europe (eu-west-1), ensuring the application remained accessible.
Idempotent Recovery: As the Washington region came back online, Stripe began redelivering the queued webhooks. Our idempotency protection ensured that each order was processed exactly once, despite multiple delivery attempts during the recovery period.
Zero Data Loss: Every payment captured by Stripe was safely recorded. No customer was charged without receiving their order, and no order was duplicated despite the chaos of the recovery process.
The result: US customers who ordered during the peak of the outage saw a brief delay in processing. Once services came back, their orders processed automatically in sequence. The idempotency keys we track (Stripe event IDs and payment intent IDs) handled deduplication as webhooks were retried.
Global Resilience in Action
While US orders faced temporary delays, customers in other regions continued to experience normal service:
- UK and European customers: Uninterrupted service throughout the outage
- Payment processing: 100% success rate via Stripe (which remained operational)
- Order records: Successfully written to DynamoDB once regions recovered
- PDF generation: Completed with only minor latency increases during recovery
- Email confirmations: Delivered successfully via SES
- Print jobs: Created successfully with a printing supplier's external API
No manual intervention was needed. The event-driven architecture, Stripe's webhook retry mechanism, and our idempotency protection handled the recovery automatically. When we checked our logs afterwards, every single order had been processed correctly.
Lessons for Modern Web Applications
The October 2025 AWS outage reinforced some basic principles. Assume every external service will fail. Build retry logic, timeouts, and fallbacks from the start. Don't make users wait for operations that can happen in the background. Every operation should be safely retryable - non-negotiable for payment systems. Loosely coupled services fail independently without taking everything down. And you can't fix what you can't see, so log everything.
Looking Forward
The AWS outage was a reminder that even the most reliable infrastructure can experience disruptions. While our architecture performed admirably, the Washington region outage highlighted areas for continued improvement. At EverFold, we're committed to enhancing our resilience:
- Enhanced multi-region failover with active-active deployment across US and EU regions
- Expanded monitoring and alerting capabilities with region-specific health checks
- Implementing additional redundancy for critical operations
- Regular chaos engineering exercises to test regional failover scenarios
- Documenting and sharing our learnings with the broader community
Building resilient systems isn't about preventing all failures. It's about handling them when they happen. The October 2025 AWS outage showed that EverFold's event-driven architecture, idempotency protection, and service decoupling work when it matters.
We're proud of how the system performed, but we're more focused on making sure customers can always create and order their colouring books, whatever the infrastructure throws at us.
About the Author
Dr Jon Baker is the CEO and founder of EverFold. With a PhD in Electronics Engineering and 15 years of experience building software, Jon has combined his technical expertise with a passion for creating innovative products. He created EverFold to make it easy for anyone to transform their photos into beautiful, professionally printed colouring books.
