AWS RDS Blue/Green Deployment: A Safe and Efficient Strategy

Many businesses can be hesitant to update their databases due to several understandable challenges. There could be concerns about potential downtime that may disrupt their operations, the perceived complexity of the project and fear of encountering unforeseen bugs or data loss add to the hesitation.

In this article, one of our Data Engineers, Eduardo Donizeti, will walk you through a safe and efficient strategy using the AWS RDS Blue/Green Deployment that will keep your database optimised for performance while reducing risks and downtimes.

Upgrading Your Database: Why is it important?

Optimising your database is super important for any business that wants to run smoothly. If you don’t keep your database up to date, you could face cyber threats, lose data, and deal with slower processes, among other issues. Plus, as your business grows, making sure your database is optimised becomes even more crucial.

As your business expands, your database will have to handle more work and store more data. The more complex it gets, the more chances there are for things to go wrong. Losing data or getting hit by a cyber attack can be disastrous for a growing business.

There are several reasons why upgrading the database version is important. Some of them are:

• End of Life (EOL) for Database Versions: When a database version reaches its end of life, the vendor stops providing technical support, security updates, and bug fixes for that version.
• Security Enhancements: Version upgrades often include security patches that fix vulnerabilities discovered in previous versions. Keeping the database updated helps protect sensitive data from cyber threats.
• Performance Improvements: New database versions frequently include performance optimisations that can enhance the speed and efficiency of queries and transactions.
• New Features and Functionalities: Updates can introduce new features and functionalities not available in older versions. This might include support for new data types, improvements in replication and disaster recovery, and enhanced tools for database administration and monitoring.
• Regulatory Compliance: In some industries, keeping software up to date is necessary for compliance with data security regulations and standards.

Overview of Amazon RDS Blue/Green Deployments

To keep downtime low and reduce risks during database updates, we need a way to make sure everything stays up and running smoothly. We should be able to update without messing up production traffic, easily roll back if there are problems, and do full testing, scaling, and continuous delivery. This will make updates quick and low-risk, helping us release new features faster.

The RDS Blue/Green Deployment is a strategy that involves creating two separate but identical RDS instances. AWS introduced this feature for Amazon RDS in November 2022. It aims to make database updates safer, simpler, and faster by reducing risks and downtime.

A blue/green deployment essentially duplicates the production database environment into a synchronised staging environment. By using Amazon RDS Blue/Green Deployments, users can make changes to the database in the staging environment without impacting the production environment.

For example, users can upgrade the major or minor DB engine version, modify database parameters, or make schema changes in the staging environment (green instance) without affecting the production workloads (blue instance). When ready, they can switch over the environments, promoting the green environment to the new production environment. This switchover usually takes under a minute, with no data loss and no need for application changes.

Since the green DB instance is a replica of the production DB instance, it includes all features used by the production instance, such as read replicas, storage configuration, automated backups, Performance Insights, and Enhanced Monitoring. If the production DB instance is a Multi-AZ deployment, the green DB instance will also be a Multi-AZ deployment.

This feature was introduced to address the need for safer database updates, reducing the complexity and costs associated with manual management. It ensures updates can be carried out without negatively impacting the production environment and allows easy rollback in case of issues during the update.

At the time of this post, Blue/Green Deployments feature is available in all AWS Regions and supports the following engines:

• RDS for MariaDB version 10.2 and higher
• RDS for MySQL version 5.7 and higher
• RDS for PostgreSQL version 11.21 and higher

How does the AWS RDS Blue/Green deployment work?

The image demonstrates a blue/green deployment. During this process, RDS duplicates the entire topology and configuration of the primary DB instance to form the green environment. The copied DB instances are named with a “-green-random-characters” suffix. In the image, the staging environment includes a Multi-AZ DB instance (database-1-green-abc123) and a read replica (database-2-green-abc123).

When creating a blue/green deployment, you can upgrade your DB engine version and choose a different DB parameter group for the instances in the green environment. RDS also sets up logical replication from the primary DB instance in the blue environment to the primary DB instance in the green environment.

When you’re ready, switch over to promote the staging environment to the new production environment. This switchover will cause downtime, typically less than a minute, but it can be longer depending on your workload.

The image below shows the DB instances after the switch over.

After the switchover, the DB instances from the green environment become the new production instances. The names and endpoints of the current production environment are reassigned to the newly promoted environment, so no application changes are needed. Your production traffic will now flow to the new production environment. The DB instances in the previous blue environment are renamed by adding “-old1” to their current. The following changes occur during the switchover:

• The green environment Multi-AZ DB instance deployment named “database-1-green-abc123” becomes the production Multi-AZ DB instance deployment named “database-1.”
• The green environment read replica named “database-2-green-abc123” becomes the production read replica “database-2.”
• The blue environment Multi-AZ DB instance deployment named “database-1” becomes “database-1-old1.”
• The blue environment read replica named “database-2” becomes “database-2-old1.”

If you no longer need a blue/green deployment, you can delete it.

Considerations for AWS RDS Blue/Green Deployments

• Unique Identifiers: Amazon RDS uses unique, immutable DbiResourceId identifiers to track resources in blue/green deployments, distinct from the DB instance IDs. During a switchover, the instance ID changes, but the DbiResourceId remains constant. For example, an instance ID database-1 in the blue environment might change to database-1-old1 after switchover.

• DB Snapshots: When restoring DB snapshots, ensure the correct snapshot based on the time is used. Note that the DB instance name changes, affecting operations such as DescribeDBInstanceAutomatedBackups or RestoreDBInstanceToPointInTime.
• Read Replicas: New read replicas added in the green environment will not replace the blue environment replicas after switchover. Additionally, DB instances deleted in the green environment cannot be recreated within the same deployment.
• DDL Changes: Creation or modification of large objects or data definition language (DDL) statements (e.g., CREATE TABLE, CREATE SCHEMA) in the blue environment are not replicated to the green environment. If Amazon RDS detects a DDL change, the green databases will enter a “Replication degraded” state.
• Encryption: It is not possible to change an unencrypted DB instance into an encrypted DB instance or vice versa during the switchover process.

By carefully managing these aspects, AWS RDS Blue/Green Deployment ensures a smooth and reliable transition with minimal disruption to the production environment.

AWS RDS: Let’s do this

These are the steps to use this feature effectively, but you can also follow the steps in the AWS Management Console.

Automated Backups

Before you create a blue/green deployment for an RDS for MySQL DB instance, you must enable automated backups. For instructions, see Enabling automated backups.

Parameter Group

If you are using PostgreSQL, you may need to make some changes to the database parameter group. If you’re using the default parameter group, you should create a custom parameter group and associate it with your database.

Change the parameter rds.logical_replication to 1 (on). Logical Replication is required for replication from the blue environment to the green environment.

Because blue/green deployments need at least one background worker per database, ensure to adjust the following configuration settings based on your workload. For guidance on adjusting each setting, refer to Configuration Settings in the PostgreSQL documentation.

• max_replication_slots
• max_wal_senders
• max_logical_replication_workers
• max_worker_processes

Once you’ve enabled logical replication and configured all options, remember to restart the DB instance to apply your changes. Blue/green deployments require the DB instance to be in sync with the DB parameter group, otherwise, creation will fail.

If you are using MySQL, you need to ensure you have a sufficiently large binlog retention. I recommend a minimum of 24 hours. You can check your binlog retention by running the following:

If this is unset or has a lower value, you can correct that with the following:

Creating the Blue/Green Environment

After enabling automated backups and configuring all parameters, the next step is to create the blue/green deployment. You can create it using the AWS Management Console, the AWS CLI, or the RDS API.

In this example, we will use the AWS Management Console, which will guide us to a form where we can name our Blue/Green Deployment, specify the green engine version, and the parameter groups.

After clicking the ‘Create’ button, you will be redirected to the RDS dashboard, where you can see that a replica of your database is being created. In this example, we chose to upgrade a MySQL instance from version 5.7 to version 8.0, so after the replica is created, it will be upgraded to MySQL version 8.0.

At the end of the process, you’ll have an endpoint to connect to the green database, where you can run any query.

Testing

Once the green environment is up and running, several tests should be conducted before performing the switch over, such as:

Running integration, regression, and load tests on the Green environment.
Verifying if all integrated systems are functioning correctly with the new version.

Testing failover from the Blue to the Green environment to ensure it works as expected.

Switch Over

After finishing the tests, backing up the databases and scheduling a maintenance window, you are prepared to proceed. Although the expected downtime was only supposed to be a few minutes, I recommend that you initiate the change after business hours to allow time for addressing any potential issues.

Using the console, you can initiate the switchover, and a couple of minutes later, the process will be ‘complete’, meaning that the application is now pointing to what was previously the staging database.

You can compare the differences between the two environments (Blue and Green) and set a Timeout.

Post-Switchover Actions

To ensure your application continues to function correctly after the database update, follow these steps:

• Check the Health of the Environment: Monitor the environment to ensure it is stable.
• Ensure Monitoring Metrics: Verify that monitoring metrics are configured and functioning correctly.
• Verify Connections and Integrations: Ensure all connections and integrations with the application are operational.
• Monitor Performance: Keep an eye on the performance of the new environment to detect any issues.
• Document the Process: Record the deployment process, any problems encountered, and any changes that may affect application usage for future reference.

For integrated features and services, after the switchover, update the resource IDs for all integrated features and services, such as:

• RDS API: Adjust filtering parameters to reflect the new resource IDs.
• CloudTrail: Update consumer configurations to track the new resource IDs.
• Performance Insights API: Modify API calls to use the new resource IDs.
• IAM Policies: Include the new resource IDs to maintain access permissions.
• IAM Roles: Reassociate IAM roles with the new resource IDs, as they are not automatically transferred.
• IAM Database Authentication: Ensure that both blue and green databases are listed under the policy’s Resource element.
• AWS Backup: Adjust the resource IDs used by AWS Backup to point to the new instances.

If everything is running smoothly with your application and no issues were found during monitoring, you can go ahead and delete both the Blue/Green deployment and the old database instance. Just make sure to take a snapshot of the old instance before deleting it, just to be safe.

Roll Back (if necessary)

Even after extensive testing using the green instance, it’s possible that your database may face issues after the switch, such as instability and slow performance, since it will now be handling the production workload.. Unfortunately, once the switch over is done, it’s not possible to switch back because the button will be disabled.

One way to roll back to the old instance is to delete the Blue/Green Deployment (database-1-bgd) and rename the databases so that the endpoints are switched back.

example:

• database-1 to database-1-fail
• database-1-old to database-1

Note: Ensure the “read_only” parameter on the old production instance is configured correctly. In some cases, you may need to restart the old instance for it to accept write commands.

Remember to document, analyse, and fix any issues that occurred during the promotion (if any).

Conclusion

AWS RDS Blue/Green Deployment offers a robust and efficient strategy for database updates and migrations. By leveraging a synchronised staging environment that mirrors the production environment, this approach minimises risks and downtime. It ensures that updates, whether they are engine version upgrades, parameter changes, or schema modifications, can be thoroughly tested before being applied to the production database. This method significantly enhances security, reliability, and performance while reducing the complexity and costs associated with manual updates. The capability to swiftly switch environments with minimal downtime and no data loss makes RDS Blue/Green Deployment an invaluable tool for maintaining the integrity and availability of databases. Despite the potential additional costs of maintaining duplicate environments, the benefits in terms of safety, efficiency, and compliance make it a compelling choice for organisations looking to streamline their database management processes.