DEV Community

Cover image for Building a Serverless Event-Driven Processing Pipeline with Amazon S3, Amazon SQS, and AWS Lambda
Narendra | SRE
Narendra | SRE

Posted on Edited on Originally published at builder.aws.com

Building a Serverless Event-Driven Processing Pipeline with Amazon S3, Amazon SQS, and AWS Lambda

Building a Serverless Event-Driven Processing Pipeline with Amazon S3, Amazon SQS, and AWS Lambda

Level: Intermediate

Focus: Event-driven architecture, asynchronous processing, reliability, security, and observability

Example code: aws-community-builders repository

Implementation status: This reference implementation has been deployed and validated in an AWS learning environment in the Asia Pacific (Hyderabad) Region ( ap-south-2 ).

The implementation was provisioned using Terraform and tested with real S3 uploads, SQS message delivery, Lambda processing, failure handling, retries, dead-letter queue routing, and
CloudWatch monitoring. Validation evidence is available in the accompanying GitHub repository.

This is a learning reference implementation, not a production-ready solution. Review the security, reliability, and cost considerations before adapting it for production.

Originally published on AWS Community Builder

Introduction

Imagine you operate an application where users upload invoices, images, reports, or CSV files. Each upload needs to trigger some work: validate metadata, extract information, transform data, or notify another system.

A tightly coupled implementation can make the upload path slower and harder to operate. If processing is temporarily unavailable, the upload workflow may also be affected. A more resilient approach is to separate the act of accepting a file from the work that follows it.

In this article, we'll explore an event-driven pattern built with Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda. We'll also cover a dead-letter queue, IAM permissions, encryption, monitoring, retries, and idempotency.


Figure 1. A file-created event is queued, processed asynchronously by Lambda, and observed through CloudWatch. Repeatedly failing messages can be moved to a dead-letter queue.

1. The problem we're solving

A simple file upload workflow often starts with one question: what should happen after a file arrives?

For a small prototype, an application might upload a file and then call a processing function directly. This can be acceptable, but it creates a tighter dependency between the upload path and the processing operation. Slow processing, transient errors, or sudden bursts of uploads can complicate the user experience and recovery process.

The design goal is to make file storage and file processing separate responsibilities. We want to:

  • Accept files without requiring the processing job to finish immediately.
  • Buffer events when uploads arrive faster than they can be processed.
  • Retry temporary failures.
  • Isolate repeatedly failing messages for investigation.
  • Limit permissions to what each component needs.
  • Make activity and failures visible to operators.

2. Architecture overview

The main components are:

Component Responsibility
Amazon S3 Stores uploaded objects and emits object-created notifications.
Amazon SQS processing queue Buffers notifications so processing can happen asynchronously.
AWS Lambda Polls the queue through an event source mapping and handles messages in batches.
Amazon SQS dead-letter queue (DLQ) Holds messages that could not be processed after the configured retry limit.
AWS CloudWatch Logs Stores Lambda execution logs for troubleshooting.
AWS IAM Defines which actions the Lambda execution role can perform.

The flow is deliberately asynchronous. S3 publishes an event notification to SQS; Lambda polls SQS using an event source mapping. This differs from the model where S3 invokes Lambda directly.

Why put SQS between S3 and Lambda?

SQS acts as a buffer. If a burst of uploads creates more work than Lambda can process immediately, messages can wait in the queue. The queue also gives the consumer a retry mechanism and a place to route messages that repeatedly fail.

A queue does not make processing exactly-once. Standard SQS queues provide at-least-once delivery, so a message may be delivered more than once. The consumer should be designed to tolerate duplicates.

3. End-to-end event flow



Figure 2. Upload, event notification, queueing, Lambda polling, processing, and logging in sequence.

Let's walk through the request lifecycle:

  1. A user uploads an object to S3. The bucket stores the object. The bucket can be configured to send object-created notifications to a designated SQS queue.
  2. S3 publishes a notification. The queue policy must allow the expected S3 bucket to send messages. Restrict the policy to the intended source bucket and account where applicable.
  3. SQS retains the message. The message stays in the queue until the consumer handles it successfully or the redrive policy sends it to the DLQ.
  4. Lambda polls the queue. The Lambda event source mapping retrieves batches of messages and invokes the function.
  5. The function validates and processes each record. In a real implementation, processing might read object metadata, parse a file, or submit a downstream task. The example handler logs object details as a small demonstration rather than performing expensive file transformations.
  6. Successful records are acknowledged. Lambda/SQS integration deletes successfully processed messages after invocation handling.
  7. Failures are retried. With partial batch response support enabled, the function can report individual failed message identifiers so successful records in the same batch do not have to be retried unnecessarily.
  8. Logs support investigation. Operators can review Lambda logs and queue metrics to understand processing volume and failures.

A practical note about payloads

S3 event notifications normally contain bucket and object-key details. Object keys in event records are URL-encoded; decode them before using them as literal keys. If the application needs to read the object body, the Lambda execution role also needs narrowly scoped S3 read access. Don't grant object access merely because the event contains a bucket and key.

4. Handling failures: retries and a dead-letter queue

Failures are part of normal distributed-system operation. A downstream dependency may time out, a file may be malformed, or the function may encounter a bug.


Figure 3. Failed messages can be retried by SQS/Lambda integration and moved to a dead-letter queue after the configured receive limit.

A dead-letter queue helps separate messages that need attention from the main processing backlog. The queue's redrive policy defines the maximum receive count before a message is moved to the DLQ.

When a message lands in the DLQ:

  • Inspect the body and message attributes to understand the failure.
  • Correlate the message with Lambda logs using timestamps and application identifiers.
  • Determine whether the issue was transient, data-specific, or caused by a code defect.
  • Fix the underlying problem before redriving messages to the source queue.
  • Avoid blindly replaying a large DLQ into an unhealthy downstream service.

Partial batch failures

Lambda can receive multiple SQS messages in one invocation. Without partial batch response handling, one failed record can cause the whole batch to be retried. When ReportBatchItemFailures is enabled, the function returns the message identifiers for records that failed, allowing successfully processed records to be acknowledged separately.

The handler must return the correct SQS message identifiers for failures. This improves retry efficiency but does not remove the need for idempotency.

5. Idempotency: preparing for duplicate delivery

For standard queues, at-least-once delivery means duplicate processing is possible. For example, a function might complete a side effect but fail before the message is acknowledged. The same message can then be delivered again.

An idempotent operation produces the same effective result when repeated with the same input. Useful techniques include:

  • Use a stable business identifier or a combination of bucket, object key, object version, and event metadata to identify the work item.
  • Store processing status in a durable data store when the business operation requires deduplication.
  • Use conditional writes or unique constraints to prevent duplicate side effects.
  • Make downstream APIs idempotent when they support idempotency keys.
  • Consider S3 versioning if the processing semantics depend on a particular object version.

The simple example in the repository demonstrates event handling and partial batch failure structure. It is not a complete business-level deduplication system.

6. Security design: least privilege and protected storage


Figure 4. Each service has a specific permission boundary: S3 may publish to the queue, Lambda may poll the queue and write logs, and bucket access remains private by default.

Security should be designed into the workflow rather than added after it works.

S3 bucket controls

The Terraform example configures S3 Block Public Access, server-side encryption, and versioning. These are useful baseline controls, but they do not replace a complete data-protection policy. For production, consider retention requirements, lifecycle rules, access logging where appropriate, and whether a customer-managed KMS key is required.

Queue policy

The queue policy should permit s3.amazonaws.com to send messages only from the intended bucket and account context. Avoid a broad policy that allows arbitrary sources to publish messages to the queue.

Lambda execution role

The Lambda role should contain only the permissions required for the function to operate: write its CloudWatch logs, receive/delete messages, inspect queue attributes, and access other resources only when the application requires them. The example should be reviewed carefully before production use, including the exact actions and resource ARNs.

Encryption

S3 server-side encryption protects stored objects. SQS encryption can be configured separately; if using a customer-managed KMS key, ensure the relevant principals have the necessary key permissions. Encryption configuration does not, by itself, determine who is authorized to read a resource.

Keep credentials out of code

Use an appropriate AWS credential provider or role-based access rather than embedding access keys in source files. Never commit real credentials, account identifiers that you intend to keep private, customer data, or secrets into the repository.

7. Observability and operational signals

CloudWatch Logs are the starting point for inspecting function execution, but logs alone are not enough to operate a queue-based workflow.

Monitor signals such as:

  • ApproximateNumberOfMessagesVisible: messages waiting to be processed.
  • ApproximateAgeOfOldestMessage: how long the oldest queued message has been waiting.
  • NumberOfMessagesSent / NumberOfMessagesReceived: useful for understanding queue activity, with the usual caveat that these are approximate queue metrics.
  • Lambda errors and throttles: indicators that processing is failing or capacity is constrained.
  • DLQ visible message count: a useful alert signal for messages that need investigation.

For a production deployment, define alarms using the service metrics that matter to the workload, set retention for logs, and document the response procedure when alerts fire. Avoid logging full file contents or sensitive values.

8. Cost considerations

The architecture avoids managing long-running servers, but serverless does not mean free. Charges may arise from S3 storage and requests, SQS requests, Lambda invocations and duration, CloudWatch log ingestion and storage, data transfer, and optional encryption or monitoring features.

Before running the example:

  1. Check the current AWS Free Tier terms and eligibility for your account. Offerings and eligibility rules can change.
  2. Use a dedicated test bucket and a small number of test objects.
  3. Configure log retention rather than keeping logs indefinitely by default.
  4. Delete test resources after validating the workflow.
  5. Review AWS Billing and Cost Management and consider a budget alert. A budget alert is not a hard spending cap and may arrive after charges have accrued.

This article intentionally does not promise a zero-cost deployment. Actual costs depend on your account eligibility, Region, request volume, payload sizes, logging, and usage duration.

9. Implementation and Validation

The companion repository contains the Terraform implementation, Python Lambda handler, and supporting validation evidence.

The infrastructure was deployed in AWS using Terraform.
The deployment created 14 resources successfully.

Deployment validation

The following activities were performed:

  1. Provisioned the infrastructure using Terraform.
  2. Uploaded a test object to the S3 bucket.
  3. Verified S3 event notification delivery through SQS.
  4. Confirmed successful Lambda execution.
  5. Inspected Lambda execution logs in CloudWatch.
  6. Tested intentional processing failures.
  7. Verified SQS retry behaviour.
  8. Verified failed-message delivery to the DLQ.
  9. Inspected CloudWatch DLQ alarm configuration and state.

Evidence

The repository includes:

  • Terraform deployment outputs.
  • Terraform infrastructure state.
  • Successful Lambda processing logs.
  • Failed processing logs.
  • SQS processing queue attributes.
  • DLQ attributes.
  • Lambda event source mapping configuration.
  • CloudWatch alarm information.
  • AWS Console screenshots.

See the  evidence/  directory for the corresponding outputs.

Cleanup

After completing validation and collecting the required evidence,
the test infrastructure was removed using Terraform.

This prevents unnecessary ongoing resource charges.

The example should be deployed in a controlled AWS account.
Always review the Terraform plan and estimated costs before applying it.

10. When would I use this architecture?

This pattern is useful when a storage event should start background work and the system benefits from buffering and retry handling. Examples include:

  • Validating uploaded CSV files.
  • Starting image or document processing.
  • Running metadata extraction jobs.
  • Kicking off asynchronous data-import workflows.
  • Separating ingestion from downstream processing in internal tools.

It may be unnecessary for a tiny workflow where a direct S3-to-Lambda integration is sufficient. It also isn't a universal solution for long-running orchestration, strict ordering, or exactly-once business effects. Those requirements need additional design choices.

Conclusion

Combining Amazon S3, Amazon SQS, and AWS Lambda provides a practical way to build asynchronous, event-driven processing systems.

This architecture separates object storage from processing, provides message buffering, supports retries, and isolates repeatedly failing messages through a dead-letter queue.

The implementation also demonstrates why security, observability, and operational validation are important parts of cloud architecture.

By deploying and testing this architecture with Terraform, we can move beyond understanding individual AWS services and learn how they work together in a real cloud environment.

The complete implementation and validation evidence are available
in the companion GitHub repository.

References


Author note before publication: Add your own short paragraph about why you chose this pattern and what you learned while reviewing the implementation. Once you've actually tested it, update the implementation-status note with the specific checks you completed and any limitations you observed. Do not claim deployment or test results before doing them.

Top comments (0)