Game Day
Jerome Kelly

Game Day

It’s crucial to remember that the primary goal is to uncover the blind spots in the project.

On December 13th, the team working on a large-scale mobile application project gathered in our Montreal office to tackle a terrible catastrophe (fortunately, a fictitious one). The scenario was as follows:

In 5 hours, our client is set to launch its biggest campaign of the decade—no less! But disaster strikes: AWS services are down across Canada. After conducting an urgent PIA (don’t forget the Law 25 ), the decision is made to migrate the entire infrastructure to the United States.

Although this scenario is far-fetched, the exercise itself is incredibly valuable. In fact, such simulations are increasingly common in the industry. In the rest of this article, we’ll discuss the various benefits and share some tips to help you get the most out of these exercises.


Choosing the Right Disaster

Every system is unique, and the scenario you choose should be tailored to its specific characteristics. A scenario that is too challenging can be demoralizing, while one that is too simple might add little value. Beyond the technological maturity of the project, a critical factor in selecting a disaster scenario is the team composition. If the original creators of the system are no longer part of the team, it’s often necessary to lower the difficulty of the scenario.

It’s crucial to remember that the primary goal is to uncover the blind spots in the project.


Taking Notes

Many organizations have disaster recovery policies that have never been tested. The “game day” presents an ideal opportunity to put them to the test. Throughout the exercise, it’s essential to document pain points and necessary fixes. The aim of the exercise isn’t to solve all the problems immediately but to identify, document, and address them afterward.

Finally, taking notes only adds value if it leads to concrete actions. It’s vital to schedule a post-mortem session in the days following the exercise to calmly analyze the results of the simulation. Ideally, tangible actions related to the project and its policies should be incorporated into the next sprint.


Balancing Speed and Learning

Even if the scenario includes a time constraint, it’s important to make time for learning. For many team members, such an exercise is a unique opportunity to understand how certain system components work or to familiarize themselves with less common concepts in a developer’s daily routine. For instance, during our last exercise, a team member had the chance to grasp the full process of configuring DNS records.

Some moderation may be needed if someone becomes overly competitive and prioritizes speed alone. Conversely, adding some pressure can be beneficial if the team isn’t taking the exercise seriously enough. One of the main objectives remains testing the team’s resilience in difficult situations.


Conclusion

For a relatively low cost, disaster recovery exercises are an extremely valuable tool for any organization that values operational excellence. They help test the team’s resilience under stress, uncover and address blind spots in the project and internal policies, and preserve institutional knowledge that might otherwise fade over time. For these reasons, we’ve decided to include these exercises as an option in the maintenance plans we offer to our clients.

For those curious, the team managed to recreate the environment in 3 hours and 41 minutes, which (barring another catastrophe this year ) keeps us compliant with our 99.95% SLO. This success was made possible through the use of modern technologies such as ECS/Fargate and CloudFormation, along with a robust database backup strategy.

The exercise was still highly relevant! We identified poorly documented Lambda functions, an outdated SQS queue, and several other opportunities for improvement, which will be implemented in the coming weeks.


share this article

Other articles

Jan 20, 2025

Thirdbridge in the spotlight: L'Arrière-Scène's digital partner

Thirdbridge is proud to announce that it is the official digital partner of JA Hypothèques and their latest project: L'Arrière-scène.

Jan 6, 2025

25 Key Trends to Optimize Your Mobile App in 2025

The Thirdbridge team has compiled this article outlining 25 trends to consider for mobile app development or strategy, or any other type of digital product in 2025.

Oct 24, 2024

Leverage Mobile to Optimize the Online Shopping Experience

The holiday season is a strategic time and a crucial opportunity for businesses to maximize their sales through mobile while enhancing the online shopping experience.

Oct 15, 2024

Enhancing Product Management: Key to Success in Software Development

The distinction between product management and project management is essential for ensuring optimal productivity. It’s not enough to treat them as interchangeable concepts; it’s crucial to adopt a proactive approach to place the right resources in the right places.

Sep 27, 2024

Thirdbridge in La Presse: Vision and Growth

Our President and Co-Founder, Pierre-Étienne Bousquet, was recently the guest of Camille Dauphinais-Pelletier of La Presse, where he shared his thoughts on Thirdbridge's journey.

Sep 12, 2024

Maximizing Engagement with User-Generated Content

The emergence of user-generated content (UGC) is revolutionizing co-creation. As a key tool in brands' marketing strategies, UGC is changing the way content is created and consumed.

Aug 9, 2024

PWAs: Test the Potential of Mobile Apps

With the advent of mobile applications, our daily lives have been transformed: these simple tools have become essential facilitators of daily tasks and catalysts for professional and personal interactions.

Jun 26, 2024

Thirdbridge in La Presse

As a business leader, one must ask themselves, "What am I trying to accomplish with my project?" and answer with a vision that extends beyond one's own interests.

Jun 14, 2024

Recruiting an In-House Team or Hiring an Agency for Developing Your Application?

When embarking on a project as significant and important as developing an application, a crucial dilemma quickly arises: choosing between a specialized agency or recruiting your own in-house team to accomplish the work. One thing is certain, both options present distinct advantages and constraints.

Jun 14, 2024

Mastering App Development: A 5-Step Guide to Success

Developing an application isn't something you can just wing. To succeed in this coveted domain, being well-prepared is essential. Unfortunately, a vast majority of large-scale digital projects fail due to inadequate preparation.

May 30, 2024

Do You Really Need an Application?

Don't furrow your brows! This is a genuinely good question. Just observe people on the subway, for example, or in a waiting room: almost everyone has a phone in hand, whether to read, text, play, get information, meet a soulmate, order food, or shop...

May 22, 2024

Optimizing Synergy with Your Software Development Partner

The digital realm, especially that of custom digital solution development, is constantly evolving—between fast technological advancements and changing consumer needs, it's quite challenging to predict what the future holds for web players.

May 14, 2024

Succeeding in Your Updates in 5 Steps

Did you know that at least 20% of the development time of an application should be allocated to testing and quality assurance?

May 3, 2024

Simplified Infrastructures for Enhanced Agility

At Thirdbridge, we believe that project-oriented teams deliver superior quality results, and do so more quickly. Given that they are responsible for the entire value creation flow, these teams can increase their velocity by eliminating bottlenecks themselves. Moreover, entrusting end-to-end flow responsibility to our developer teams makes their work even more engaging and motivating.

Apr 12, 2024

The Thirdbridge Entrepreneurial Scholarship

Thirdbridge is more than proud to be able to support a project and individuals full of promise.

Jan 14, 2025

Modern Password Management

At Thirdbridge, providing peace of mind to our clients is at the heart of our priorities, and we believe it all starts with the reliability of our internal practices and processes.

Oct 29, 2024

AI driving innovation: A new Era for Mobile Apps and User Experience

Artificial intelligence (AI) represents a digital transformation that impacts us all. This rapidly advancing technology, fueled by data analysis, not only enables informed decision-making and reliable forecasting but also allows for the completion of many tasks at a faster pace.

Oct 17, 2024

How to Gauge your Mobile App's performance?

A mobile application is the extension of a brand's customer experience.

Oct 15, 2024

Pierre-Étienne Bousquet guest of "Les Affaires"

Our president and co-founder, Pierre-Étienne Bousquet, discussed with Jean-François Venne from Les Affaires the significant growth of digital technology in the retail industry and its impact on online sales, which are becoming increasingly crucial for revenue.

Sep 24, 2024

Cybersecurity and Mobile Applications: Choosing the Right Authentication Method

Mobile applications are essential tools that handle personal data, access sensitive information, and are part of our daily lives. However, in an age where the term cybersecurity is on everyone's lips, ensuring the security of these applications and the information they contain is crucial.

Sep 4, 2024

The impact of UX research

Integrating user experience (UX) principles and practices into the software or application development process has become crucial.

Jul 15, 2024

The Phygital: Rethinking the Retail

Businesses have always had to innovate and rethink their approaches to remain relevant, and this is even more true in the digital age.

Jun 18, 2024

Hybrid vs. Native: Making the Right Choice

At Thirdbridge, the preferred development approach is hybrid. But let's delve deeper by comparing hybrid and native development across key stages of application development: costs, performance, security, and maintenance.

Jun 14, 2024

Funding Your Digital Project

It's no secret that realizing your wildest dreams regarding digital innovation within your company brings many benefits.

Jun 13, 2024

Launching Your Application: The Key to a Well-Planned Budget

Very few digital projects end within their initial budgets and timelines.

May 23, 2024

Tips and Tricks for Sustainable Software Design

When we think about reducing our ecological footprint, our first instinct is to consider the means of transportation we use or our recycling and consumption habits.

May 17, 2024

Maximizing Your App's Profitability: Our Advice

Whether you're looking to save time for your users, retain them, or enhance their shopping experience, we're sharing here the three key elements to consider to maximize your return on investment (ROI).

May 6, 2024

Couche-Tard Connecté: The Cashierless Convenience Store

Congratulations to our mobile development team, who gave their all in recent weeks to ensure a smooth launch of the Couche-Tard Connecté project.

Apr 25, 2024

Beyond Launch: Ensuring the Longevity of Your Application

You've diligently followed the development stages of your application and are about to launch it: congratulations! But even though this is a great accomplishment, your job is far from over...

Mar 22, 2024

Our 12 tips for succeeding in a software project after 12 years in the industry

Thirdbridge celebrates its 12th anniversary!