AWS Lambda Serverless Patterns with Terraform: Event-Driven Architecture at Scale

The Serverless Landscape in 2026

Serverless computing has matured from a niche deployment model into a foundational architecture pattern for cloud-native applications. AWS Lambda, as the pioneer and market leader, now supports execution times up to 15 minutes, memory up to 10 GB, container image deployments, ARM64 (Graviton2) processors, and advanced features like Lambda SnapStart, Provisioned Concurrency, and function URLs. These capabilities have eliminated many of the historical limitations that pushed workloads toward containers.

The real power of Lambda emerges not from individual functions but from the composition of functions into event-driven architectures. When Lambda is wired to SQS queues, DynamoDB streams, S3 event notifications, EventBridge rules, and API Gateway endpoints, it becomes the compute fabric of a fully reactive system that scales from zero to millions of concurrent executions without any capacity planning.

At Citadel Cloud Management, I deploy serverless architectures entirely through Terraform, treating Lambda functions and their event sources as infrastructure alongside VPCs, databases, and DNS records. This approach ensures that the entire system, from the API Gateway endpoint to the DynamoDB table to the dead letter queue, is version-controlled, peer-reviewed, and reproducible. My terraform-aws-lambda module encapsulates the Lambda function, IAM role, CloudWatch log group, and event source mappings into a single reusable module.

Lambda Fundamentals for Infrastructure Engineers

Understanding Lambda's execution model is essential for designing reliable serverless systems. When a Lambda function is invoked, AWS provisions a lightweight execution environment (a microVM based on Firecracker), downloads the function code, initializes the runtime, runs the initialization code outside the handler, and then executes the handler function. This entire process is the cold start.

Invocation Models

Lambda supports three invocation models. Synchronous invocation (API Gateway, Application Load Balancer) blocks the caller until the function completes and returns a response. Asynchronous invocation (S3 events, SNS, EventBridge) queues the event and returns immediately; Lambda handles retries (twice by default) and can route failures to a destination. Event Source Mapping (SQS, DynamoDB Streams, Kinesis) has Lambda poll the source, retrieve batches of records, and invoke the function synchronously with each batch.

Each invocation model has different failure handling semantics, and choosing the right model is critical for building reliable systems. The AWS Lambda best practices documentation provides detailed guidance on invocation patterns and error handling.

Concurrency and Scaling

Lambda scales automatically by creating new execution environments for each concurrent invocation. The default account-level concurrency limit is 1,000 concurrent executions (can be increased). Reserved Concurrency allocates a fixed pool of concurrency for a specific function, preventing one function from consuming all available concurrency. Provisioned Concurrency pre-warms a specified number of execution environments, eliminating cold starts for latency-sensitive functions.

SQS-Lambda Pattern with Dead Letter Queues

The SQS-Lambda pattern is the workhorse of event-driven architecture. Producers send messages to an SQS queue, and Lambda automatically polls the queue, retrieves batches of messages, and processes them. This pattern decouples producers from consumers, absorbs traffic spikes through the queue buffer, and provides built-in retry and dead-letter handling.

Terraform Implementation

The following Terraform configuration deploys a complete SQS-Lambda pipeline with a dead letter queue, IAM roles, and CloudWatch monitoring. This is a production-ready pattern used in my terraform-aws-sns-sqs module:

# Dead Letter Queue for failed messages
resource "aws_sqs_queue" "dlq" {
  name                      = "${var.project}-dlq"
  message_retention_seconds = 1209600  # 14 days
  kms_master_key_id         = aws_kms_key.sqs.arn

  tags = var.tags
}

# Main processing queue
resource "aws_sqs_queue" "main" {
  name                       = "${var.project}-processing"
  visibility_timeout_seconds = 360  # 6x Lambda timeout
  message_retention_seconds  = 86400
  kms_master_key_id          = aws_kms_key.sqs.arn
  receive_wait_time_seconds  = 20   # Long polling

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 3
  })

  tags = var.tags
}

# Lambda execution role
resource "aws_iam_role" "lambda" {
  name = "${var.project}-lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

# Lambda permissions for SQS and CloudWatch Logs
resource "aws_iam_role_policy" "lambda" {
  name = "${var.project}-lambda-policy"
  role = aws_iam_role.lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "sqs:ReceiveMessage",
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes"
        ]
        Resource = aws_sqs_queue.main.arn
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      },
      {
        Effect   = "Allow"
        Action   = ["kms:Decrypt"]
        Resource = aws_kms_key.sqs.arn
      }
    ]
  })
}

# Lambda function
resource "aws_lambda_function" "processor" {
  function_name = "${var.project}-processor"
  role          = aws_iam_role.lambda.arn
  handler       = "index.handler"
  runtime       = "python3.12"
  timeout       = 60
  memory_size   = 256
  architectures = ["arm64"]  # Graviton2 for cost savings

  filename         = data.archive_file.lambda.output_path
  source_code_hash = data.archive_file.lambda.output_base64sha256

  environment {
    variables = {
      ENVIRONMENT   = var.environment
      LOG_LEVEL     = "INFO"
      DYNAMODB_TABLE = aws_dynamodb_table.results.name
    }
  }

  dead_letter_config {
    target_arn = aws_sqs_queue.dlq.arn
  }

  tracing_config {
    mode = "Active"  # X-Ray tracing
  }

  tags = var.tags
}

# CloudWatch Log Group with retention
resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/${aws_lambda_function.processor.function_name}"
  retention_in_days = 30
  kms_key_id        = aws_kms_key.logs.arn
}

# SQS Event Source Mapping
resource "aws_lambda_event_source_mapping" "sqs" {
  event_source_arn                   = aws_sqs_queue.main.arn
  function_name                      = aws_lambda_function.processor.arn
  batch_size                         = 10
  maximum_batching_window_in_seconds = 5
  function_response_types            = ["ReportBatchItemFailures"]

  scaling_config {
    maximum_concurrency = 50
  }
}

# CloudWatch Alarm for DLQ messages
resource "aws_cloudwatch_metric_alarm" "dlq_messages" {
  alarm_name          = "${var.project}-dlq-messages"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_actions       = [var.sns_alert_topic_arn]

  dimensions = {
    QueueName = aws_sqs_queue.dlq.name
  }
}

# KMS key for encryption
resource "aws_kms_key" "sqs" {
  description             = "KMS key for SQS queue encryption"
  deletion_window_in_days = 7
  enable_key_rotation     = true
  tags                    = var.tags
}

Several design decisions in this configuration deserve attention. The visibility timeout is set to 6 times the Lambda timeout, which is the AWS recommended minimum to prevent messages from becoming visible again while Lambda is still processing them. Long polling (receive_wait_time_seconds = 20) reduces empty responses and API costs. The ReportBatchItemFailures response type enables partial batch failure reporting, so a single failed message does not cause the entire batch to retry. The maximum_concurrency setting on the event source mapping limits how many concurrent Lambda instances the SQS trigger creates, preventing downstream services from being overwhelmed.

API Gateway to Lambda Integration

API Gateway provides a fully managed HTTP endpoint for Lambda functions, handling TLS termination, request validation, authorization, throttling, and response transformation. HTTP APIs (API Gateway V2) offer lower latency and cost compared to REST APIs (V1) and are the recommended choice for most Lambda integrations.

Proxy Integration Pattern

The proxy integration pattern sends the entire HTTP request to Lambda and returns the Lambda response directly to the client. This gives the Lambda function full control over request routing, headers, and response formatting. It is the simplest pattern and works well with web frameworks like FastAPI, Express, or Flask running inside Lambda.

The terraform-aws-api-gateway-v2 module provides a complete HTTP API implementation with Lambda integration, custom domain names, and CloudWatch logging pre-configured.

Authorization Patterns

API Gateway supports multiple authorization mechanisms: IAM authorization (for service-to-service calls), Cognito authorizers (for user authentication), Lambda authorizers (for custom token validation), and JWT authorizers (for OIDC/OAuth2 providers). Choose JWT authorizers with HTTP APIs for the simplest integration with modern identity providers, and Lambda authorizers when you need custom logic such as API key validation or IP whitelisting.

DynamoDB Streams and Event Processing

DynamoDB Streams capture item-level changes (inserts, updates, deletes) in a DynamoDB table and make them available as an ordered stream of records. When paired with Lambda, DynamoDB Streams enable powerful patterns like change data capture, materialized views, cross-region replication, and event sourcing.

My terraform-aws-dynamodb module configures DynamoDB tables with streams enabled, point-in-time recovery, auto-scaling, and encryption using customer-managed KMS keys.

Stream Processing Considerations

DynamoDB Streams are ordered per partition key, meaning events for the same item are always processed in order. However, events for different items may be processed concurrently across multiple shards. Design your stream processors to be idempotent, as Lambda may retry records on failure. Use the bisectBatchOnFunctionError option to automatically split a failed batch in half, isolating the problematic record more quickly.

Step Functions for Workflow Orchestration

Step Functions orchestrate multiple Lambda functions into complex workflows with branching logic, parallel execution, error handling, and retries. They solve the problem of coordinating long-running, multi-step processes that exceed Lambda's 15-minute timeout or require human approval steps.

Common Patterns

The Sequential Processing pattern chains Lambda functions where each step's output feeds into the next. The Fan-Out/Fan-In pattern uses a Map state to process a list of items in parallel, then aggregates the results. The Saga Pattern implements distributed transactions with compensating actions, where each step has a corresponding rollback step that executes if a later step fails. The Human Approval pattern pauses the workflow using a task token and resumes when an external system (Slack bot, email link) sends the token back to Step Functions.

Express vs Standard Workflows

Standard Workflows are durable, support execution times up to one year, guarantee exactly-once processing, and cost per state transition. Express Workflows are designed for high-volume, short-duration workloads (up to 5 minutes), support at-least-once processing, and cost per execution and duration. Use Express Workflows for data processing pipelines and Standard Workflows for business-critical orchestration.

Lambda Layers and Shared Dependencies

Lambda Layers provide a mechanism for sharing code and dependencies across multiple functions. A layer is a ZIP archive that is extracted into the /opt directory of the Lambda execution environment. Common use cases include shared libraries (AWS SDK extensions, database drivers), custom runtimes, monitoring agents (Datadog, New Relic), and shared business logic.

Layer Management with Terraform

Manage layers as versioned Terraform resources. Each update creates a new layer version (layers are immutable), and functions reference specific versions. Use a separate Terraform module or workspace for layers so they can be updated independently from the functions that consume them. Store layer artifacts in S3 with versioning enabled for audit trails.

Cold Start Optimization Strategies

Cold starts remain the primary challenge for latency-sensitive Lambda workloads. A cold start occurs when Lambda creates a new execution environment, which adds latency ranging from 100ms (Python/Node.js) to several seconds (Java/.NET with large dependencies). Multiple strategies can minimize cold start impact.

Runtime Selection

Python and Node.js have the fastest cold starts due to their lightweight runtimes. Java and .NET have traditionally had the slowest cold starts, but Lambda SnapStart (for Java) reduces cold start latency by up to 90% by pre-initializing and caching a snapshot of the execution environment. If you must use Java, enable SnapStart. For new projects, Python 3.12 on ARM64 offers the best balance of performance, cost, and cold start latency.

Package Size Optimization

The deployment package size directly impacts cold start duration. Use layers for large dependencies that do not change frequently, tree-shake unused code in Node.js builds, avoid including test files and documentation, and use container images only when the deployment package exceeds the 250MB limit. For Python, use pip install --target . --no-cache-dir and exclude unnecessary files.

Provisioned Concurrency

For functions where cold starts are unacceptable (payment processing, real-time APIs), use Provisioned Concurrency. This pre-warms a specified number of execution environments, ensuring they are always ready to handle requests. Combine with Application Auto Scaling to adjust provisioned concurrency based on schedules or utilization metrics. The cost is approximately 40% of on-demand Lambda pricing for the provisioned capacity.

Lambda Security Patterns

Lambda security follows the principle of least privilege at every layer. Each function should have its own IAM role with the minimum permissions required. Never use wildcards in resource ARNs. Use environment variables for configuration but store secrets in AWS Secrets Manager or Parameter Store (SecureString). Enable X-Ray tracing for observability and use VPC placement only when the function needs to access private resources.

VPC Considerations

Lambda functions placed in a VPC lose access to the public internet by default. If the function needs to call AWS APIs or external services, you must either configure a NAT Gateway (adding cost and a single point of failure) or use VPC endpoints for AWS services. VPC-placed Lambda functions also have slightly longer cold starts due to ENI attachment. Only place functions in a VPC when they need to access private resources like RDS databases or ElastiCache clusters. Refer to the AWS serverless patterns documentation for detailed VPC networking guidance.

Top 10 Lambda Best Practices

Lambda vs Fargate vs EC2 Comparison

Frequently Asked Questions