Multi-Cloud Disaster Recovery Architecture with Terraform

Why Disaster Recovery Architecture Matters

Every organization depends on the availability of its digital services, yet few adequately prepare for the catastrophic failure scenarios that can take entire cloud regions offline. Regional outages, while rare, do happen. AWS us-east-1 has experienced multiple significant incidents, Azure has seen region-level failures, and even Google Cloud has had global service disruptions. The question is not whether a disaster will occur but whether your organization will recover in minutes or in days.

Disaster recovery is fundamentally a business decision, not a technical one. The acceptable level of data loss and downtime determines the architecture, and the architecture determines the cost. A system that tolerates 24 hours of downtime can use simple backup-and-restore strategies costing hundreds of dollars per month. A system that demands zero data loss and sub-minute recovery requires multi-region active-active architectures costing tens of thousands per month. The role of the architect is to map business requirements to the right DR tier and implement it reliably through infrastructure as code.

Terraform is uniquely suited for DR architecture because it can declaratively define infrastructure across multiple regions and cloud providers from a single codebase. By expressing your DR strategy as Terraform configurations, you ensure that the recovery environment is always in sync with production, can be tested automatically, and can be activated through a simple pipeline rather than through error-prone manual procedures. The AWS Disaster Recovery whitepaper provides the foundational framework for these strategies.

Understanding RPO and RTO

Recovery Point Objective (RPO) and Recovery Time Objective (RTO) are the two metrics that define every disaster recovery requirement. RPO measures the maximum acceptable data loss in time: an RPO of one hour means you can tolerate losing up to one hour of data. RTO measures the maximum acceptable downtime: an RTO of 15 minutes means the service must be restored within 15 minutes of a failure being declared.

Setting RPO and RTO Targets

RPO and RTO targets should be set per workload, not per organization. A payment processing system might require an RPO of zero (no data loss) and an RTO of one minute, while a reporting dashboard might tolerate an RPO of four hours and an RTO of two hours. Setting targets too aggressively wastes money; setting them too loosely risks the business. Conduct a Business Impact Analysis (BIA) to determine the financial and operational cost of data loss and downtime for each system, and use those costs to justify the DR investment.

The Cost-Recovery Tradeoff

There is a direct and steep relationship between RPO/RTO targets and infrastructure cost. Achieving near-zero RPO requires synchronous replication, which doubles (or more) your database costs. Achieving near-zero RTO requires running a full copy of your infrastructure in the DR region, which doubles your compute costs. The art of DR architecture is finding the strategy that meets your business requirements at the lowest sustainable cost.

The Four DR Strategies

The industry recognizes four primary DR strategies, each representing a different point on the cost-recovery spectrum. Understanding these strategies is essential for selecting the right approach for each workload.

Backup and Restore

The simplest and lowest-cost strategy. Data is backed up regularly (database snapshots, file system backups) and stored in a separate region or cloud. During a disaster, infrastructure is provisioned from scratch using Terraform, and data is restored from backups. This strategy suits non-critical systems where hours of downtime and potentially significant data loss are acceptable. My terraform-aws-s3-bucket module supports cross-region replication for backup storage, ensuring backups survive a regional failure.

Pilot Light

A step up from backup-and-restore. The core infrastructure (databases, core networking, DNS) runs continuously in the DR region with data replication, but compute resources (application servers, worker nodes) are either stopped or not provisioned. During a disaster, you start the compute resources and update DNS to point to the DR region. RTO is typically 30 minutes to a few hours, depending on how much infrastructure needs to be started.

Warm Standby

The DR region runs a fully functional but scaled-down version of the production environment. All components are running (databases, application servers, load balancers), but at reduced capacity. During a disaster, you scale up the DR environment to match production capacity and redirect traffic. RTO is typically minutes, as the infrastructure is already running and only needs to be scaled.

Multi-Site Active/Active

Both regions actively serve traffic simultaneously. Data is replicated bidirectionally, and a global load balancer distributes requests across regions. During a disaster, the failed region is removed from the load balancer, and the surviving region absorbs all traffic (potentially with auto-scaling). RTO is near-zero as there is no failover action; the system simply continues operating with one fewer region. This is the most expensive strategy but provides the highest availability.

Aurora Global Database for Multi-Region DR

Amazon Aurora Global Database is the gold standard for multi-region relational database DR on AWS. It provides storage-level replication from a primary region to up to five secondary regions with typical replication lag under one second. During a regional failure, a secondary cluster can be promoted to become the new primary in under one minute.

Terraform Implementation

The following Terraform configuration deploys an Aurora Global Database spanning two regions with automated monitoring. This pattern is used in my terraform-aws-rds-aurora module:

# Primary region provider
provider "aws" {
  alias  = "primary"
  region = "us-east-1"
}

# DR region provider
provider "aws" {
  alias  = "secondary"
  region = "eu-west-1"
}

# Aurora Global Cluster
resource "aws_rds_global_cluster" "main" {
  global_cluster_identifier = "${var.project}-global"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
  database_name             = var.database_name
  storage_encrypted         = true
  deletion_protection       = true
}

# Primary cluster
resource "aws_rds_cluster" "primary" {
  provider = aws.primary

  cluster_identifier          = "${var.project}-primary"
  global_cluster_identifier   = aws_rds_global_cluster.main.id
  engine                      = aws_rds_global_cluster.main.engine
  engine_version              = aws_rds_global_cluster.main.engine_version
  database_name               = var.database_name
  master_username             = var.master_username
  manage_master_user_password = true
  storage_encrypted           = true
  kms_key_id                  = aws_kms_key.primary_rds.arn

  db_subnet_group_name   = aws_db_subnet_group.primary.name
  vpc_security_group_ids = [aws_security_group.primary_aurora.id]

  backup_retention_period      = 35
  preferred_backup_window      = "03:00-04:00"
  preferred_maintenance_window = "sun:04:00-sun:05:00"

  enabled_cloudwatch_logs_exports = ["postgresql"]
  deletion_protection             = true
  skip_final_snapshot             = false
  final_snapshot_identifier       = "${var.project}-primary-final"

  tags = merge(var.tags, { Role = "primary" })
}

# Primary cluster instances
resource "aws_rds_cluster_instance" "primary" {
  provider = aws.primary
  count    = var.primary_instance_count

  identifier         = "${var.project}-primary-${count.index + 1}"
  cluster_identifier = aws_rds_cluster.primary.id
  instance_class     = var.primary_instance_class
  engine             = aws_rds_global_cluster.main.engine
  engine_version     = aws_rds_global_cluster.main.engine_version

  performance_insights_enabled    = true
  performance_insights_kms_key_id = aws_kms_key.primary_rds.arn
  monitoring_interval             = 15
  monitoring_role_arn             = aws_iam_role.rds_monitoring.arn

  tags = merge(var.tags, { Role = "primary" })
}

# Secondary (DR) cluster
resource "aws_rds_cluster" "secondary" {
  provider = aws.secondary

  cluster_identifier        = "${var.project}-secondary"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  engine                    = aws_rds_global_cluster.main.engine
  engine_version            = aws_rds_global_cluster.main.engine_version
  storage_encrypted         = true
  kms_key_id                = aws_kms_key.secondary_rds.arn

  db_subnet_group_name   = aws_db_subnet_group.secondary.name
  vpc_security_group_ids = [aws_security_group.secondary_aurora.id]

  backup_retention_period = 35
  deletion_protection     = true
  skip_final_snapshot     = false
  final_snapshot_identifier = "${var.project}-secondary-final"

  enabled_cloudwatch_logs_exports = ["postgresql"]

  depends_on = [aws_rds_cluster.primary]

  tags = merge(var.tags, { Role = "secondary" })

  lifecycle {
    ignore_changes = [
      replication_source_identifier,
    ]
  }
}

# Secondary cluster instances
resource "aws_rds_cluster_instance" "secondary" {
  provider = aws.secondary
  count    = var.secondary_instance_count

  identifier         = "${var.project}-secondary-${count.index + 1}"
  cluster_identifier = aws_rds_cluster.secondary.id
  instance_class     = var.secondary_instance_class
  engine             = aws_rds_global_cluster.main.engine
  engine_version     = aws_rds_global_cluster.main.engine_version

  performance_insights_enabled    = true
  performance_insights_kms_key_id = aws_kms_key.secondary_rds.arn
  monitoring_interval             = 15
  monitoring_role_arn             = aws_iam_role.rds_monitoring_secondary.arn

  tags = merge(var.tags, { Role = "secondary" })
}

# CloudWatch alarm for replication lag
resource "aws_cloudwatch_metric_alarm" "replication_lag" {
  provider = aws.secondary

  alarm_name          = "${var.project}-aurora-replication-lag"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "AuroraGlobalDBReplicationLag"
  namespace           = "AWS/RDS"
  period              = 60
  statistic           = "Average"
  threshold           = 5000  # 5 seconds in milliseconds
  alarm_actions       = [var.sns_alert_topic_arn_secondary]

  dimensions = {
    DBClusterIdentifier = aws_rds_cluster.secondary.cluster_identifier
  }

  tags = var.tags
}

This configuration deploys a fully functional Aurora Global Database with encryption at rest using customer-managed KMS keys, Performance Insights for query analysis, enhanced monitoring at 15-second intervals, automated backups with 35-day retention, and a CloudWatch alarm that triggers when replication lag exceeds 5 seconds. The lifecycle block on the secondary cluster prevents Terraform from attempting to modify the replication source, which is managed by the Aurora Global Database service.

S3 Cross-Region Replication

For object storage disaster recovery, S3 Cross-Region Replication (CRR) asynchronously copies objects from a source bucket to a destination bucket in a different region. CRR supports replication of new objects, existing objects (via S3 Batch Replication), delete markers, and encrypted objects. Combined with S3 Versioning, CRR provides a comprehensive data protection strategy for unstructured data, static assets, backups, and data lake contents.

The terraform-aws-s3-bucket module includes built-in CRR configuration with encryption, lifecycle policies, and access logging, providing a turnkey solution for S3-based disaster recovery.

Route53 Failover Routing

Route53 health checks and failover routing policies are the traffic management layer of your DR architecture. Route53 continuously monitors the health of your primary endpoints and automatically redirects traffic to DR endpoints when a health check fails. My terraform-aws-route53 module configures health checks, failover records, and latency-based routing for multi-region deployments.

Health Check Configuration

Configure health checks to monitor the most critical indicator of service availability. For web applications, check a dedicated health endpoint that verifies database connectivity, cache availability, and upstream service health. Set the health check interval to 10 seconds (fast checks) with a failure threshold of 2-3, giving you a detection time of 20-30 seconds. Use calculated health checks to combine multiple individual checks into a single composite health status.

Failover Record Sets

Create failover record sets with a primary record pointing to the production region and a secondary record pointing to the DR region. When the primary health check fails, Route53 automatically serves the secondary record. For active-active configurations, use weighted or latency-based routing with health checks enabled on each record, so unhealthy endpoints are automatically removed from the rotation.

Azure Traffic Manager and Cosmos DB

On the Azure side, Traffic Manager provides DNS-based traffic routing with health probing, analogous to Route53 failover routing. Traffic Manager supports priority routing (active-passive failover), weighted routing (traffic splitting), performance routing (latency-based), and geographic routing (geo-fencing). For DR, priority routing directs all traffic to the primary endpoint and fails over to the secondary when the primary becomes unhealthy.

Azure Cosmos DB Multi-Region

Azure Cosmos DB provides turnkey multi-region distribution with configurable consistency levels. Unlike Aurora Global Database, which replicates at the storage layer, Cosmos DB replicates at the data layer and supports multi-master writes across regions. With automatic failover enabled, Cosmos DB promotes a secondary region to primary within seconds of detecting a regional failure. My terraform-azure-cosmos-db module configures multi-region Cosmos DB accounts with automatic failover, consistency policies, and diagnostic settings. Refer to the Azure business continuity documentation for detailed guidance on Azure DR patterns.

Cross-Cloud Disaster Recovery

Cross-cloud DR eliminates the single-cloud-provider risk entirely. If AWS experiences a catastrophic, prolonged outage, your workloads fail over to Azure (or vice versa). This is the most complex DR strategy but provides the highest level of resilience. Terraform's multi-provider capability makes it possible to define both the AWS and Azure infrastructure in a single codebase, ensuring consistency across clouds.

Key Design Principles

Cross-cloud DR requires cloud-agnostic application design. Containerize workloads with Kubernetes for portability across EKS, AKS, and GKE. Use cloud-agnostic data formats and replication mechanisms (application-level replication rather than cloud-native replication). Use a multi-cloud DNS provider (Cloudflare, NS1) or a combination of Route53 and Azure Traffic Manager with external health checks. Accept that cross-cloud DR will have higher RPO than within-cloud DR due to the complexity of data replication across providers.

Data Synchronization Challenges

The hardest problem in cross-cloud DR is data synchronization. Cloud-native replication services (Aurora Global Database, Cosmos DB geo-replication) do not work across clouds. Options include application-level CDC (Change Data Capture) pipelines using tools like Debezium, scheduled database dumps with cross-cloud transfer, event-driven replication through message queues, and eventual consistency models where each cloud maintains its own data store and reconciles asynchronously.

DR Testing and Validation

A disaster recovery plan that has never been tested is not a disaster recovery plan; it is a disaster recovery wish. Testing validates that your failover procedures work, measures actual RPO and RTO against targets, identifies gaps in runbooks and automation, and builds team confidence and muscle memory for real incidents.

Testing Cadence

Implement a layered testing approach. Monthly tabletop exercises walk through failure scenarios with the team, identifying procedural gaps without touching infrastructure. Quarterly partial failover tests exercise specific components (database failover, DNS switchover) in isolation. Semi-annual full failover tests simulate a complete regional failure and execute the full DR runbook. Use AWS Fault Injection Simulator (FIS) and Azure Chaos Studio to inject controlled failures and validate automated recovery.

Measuring Success

Every DR test should measure four outcomes: actual RPO (how much data was lost), actual RTO (how long recovery took), runbook accuracy (how many undocumented steps were needed), and team readiness (how many people could execute the failover). Track these metrics over time and set improvement targets for each test cycle.

Top 10 Disaster Recovery Best Practices

DR Strategies Comparison

Frequently Asked Questions