Skip to content

Terraform Level 1, Task 34: Managing Data with aws_s3_bucket_object

Today's task was a fascinating and practical use of Terraform. Instead of just creating empty infrastructure, my objective was to use Terraform to manage data by uploading a local file into an S3 bucket.

This was a key lesson in understanding that Terraform can manage not just the "servers" and "buckets," but also the "objects" (files) that live inside them. I learned how to use the aws_s3_bucket_object resource to create a link between a local file and a file in S3. This document is my detailed, first-person guide to that entire process.

Table of Contents

The Task

My objective was to use Terraform to upload a local file into an existing S3 bucket. The requirements were: 1. All code had to be in a single main.tf file. 2. The target S3 bucket (nautilus-cp-5711) was already defined in the main.tf file. 3. The source file to be uploaded was /tmp/nautilus.txt on the local machine. 4. I had to update the main.tf file to add the new resource that would perform this copy.

My Step-by-Step Solution

The process involved updating the existing main.tf file to add a new aws_s3_bucket_object resource, which would then be created during the apply phase.

Phase 1: Writing the Code

In the /home/bob/terraform directory, I opened the main.tf file provided by the lab. It already contained the aws_s3_bucket resource. I added the new aws_s3_bucket_object resource block to the end of the file.

# (Provider block was in a separate provider.tf file)

# This block was already provided in main.tf
resource "aws_s3_bucket" "my_bucket" {
  bucket = "nautilus-cp-5711"
  acl    = "private"

  tags = {
    Name = "nautilus-cp-5711"
  }
}

# This is the new resource block I added to solve the task.
# It creates the S3 object (the file) inside the bucket.
resource "aws_s3_bucket_object" "file_upload" {
  # This links the object to the bucket, creating an implicit dependency.
  bucket = aws_s3_bucket.my_bucket.id

  # 'key' is the name of the file as it will appear inside the S3 bucket.
  key    = "nautilus.txt"

  # 'source' is the path to the local file on the machine running Terraform.
  source = "/tmp/nautilus.txt"

  # 'etag' is a hash of the file. This is how Terraform knows if the file
  # has changed. If the local file's hash no longer matches this value,
  # Terraform will plan to re-upload it.
  etag = filemd5("/tmp/nautilus.txt")
}

Phase 2: The Terraform Workflow

From my terminal in the same directory, I executed the three core commands. 1. Initialize: terraform init. 2. Plan: terraform plan. The output was very clear. It showed Plan: 2 to add, 0 to change, 0 to destroy.. This confirmed Terraform would create the bucket and the new object. 3. Apply: terraform apply. After I confirmed with yes, Terraform executed the plan. It created the bucket first, and then, once the bucket was available, it uploaded the file into it. The success message confirmed the task was done.

Why Did I Do This? (The "What & Why")

- S3 (Simple Storage Service): This is AWS's object storage service, like a limitless hard drive in the cloud. A bucket is the root-level container. An object is the data itself (a file) that is stored in the bucket. - aws_s3_bucket_object Resource: This is the Terraform resource that manages a single object (a file) within an S3 bucket. This is different from the aws_s3_bucket resource, which manages the bucket itself. - Terraform for Data Management: This task showed me that Terraform isn't just for provisioning empty infrastructure; it can also manage the content inside that infrastructure. By defining the file as an aws_s3_bucket_object resource, I am telling Terraform: "I want this file to exist in this bucket, and I want its content to match this local file. If it doesn't, fix it." - etag and filemd5(): This was a key part of the solution. The etag argument is the "entity tag," which is a hash (like an MD5 checksum) of the file's content. By setting etag = filemd5("/tmp/nautilus.txt"), I am telling Terraform to calculate the hash of the local file. When I run terraform plan in the future, Terraform will compare this local hash to the ETag of the file in S3. If they are different, Terraform will know the file has changed and will plan to re-upload it.

Deep Dive: A Line-by-Line Explanation of My main.tf Script

This script shows how to build and link a bucket and the object inside it.

[Image of a file being uploaded to an S3 bucket]

# (Provider block)

# This resource block defines the S3 Bucket itself.
resource "aws_s3_bucket" "my_bucket" {
  bucket = "nautilus-cp-5711"
  acl    = "private"
  # ...
}

# This is the resource block I added. It defines the file (object).
# "aws_s3_bucket_object" is the Resource TYPE.
# "file_upload" is the local NAME I use to refer to this object.
resource "aws_s3_bucket_object" "file_upload" {

  # This argument requires the name of the bucket.
  # I'm dynamically pulling the 'id' (which is the bucket name) from the
  # 'aws_s3_bucket' resource I named "my_bucket". This creates an
  # implicit dependency: Terraform will create the bucket FIRST.
  bucket = aws_s3_bucket.my_bucket.id

  # 'key' is the name and "path" of the file as it will be stored in S3.
  # I'm naming it 'nautilus.txt' at the root of the bucket.
  key    = "nautilus.txt"

  # 'source' is the path to the file on my local filesystem
  # (the machine running 'terraform apply').
  source = "/tmp/nautilus.txt"

  # 'etag' is a hash of the file. The 'filemd5()' function is a built-in
  # Terraform function that calculates the MD5 hash of a local file.
  # This is how Terraform will detect if the file's content changes.
  etag = filemd5("/tmp/nautilus.txt")
}

Common Pitfalls

- Source File Not Found: If the file at /tmp/nautilus.txt did not exist on the machine running terraform apply, the command would fail with a "file not found" error. - Permissions: The IAM user or role running Terraform needs s3:PutObject permission on the bucket to be able to upload the file. - Managing Large or Binary Files: Using aws_s3_bucket_object is great for small configuration files. For large binary files, this isn't always the best practice, as the file's content can get embedded in the Terraform state file, making it very large and slow. - Forgetting etag: If I had omitted the etag argument, Terraform would upload the file on the first apply, but it would have no way of knowing if the local file's content changed later. It would only see that the source argument (the file path) was the same and would not re-upload it.

Exploring the Essential Terraform Commands

- terraform init: Prepared my working directory and downloaded the aws provider plugin. - terraform plan: The "dry run" command. It showed me a plan to add 2 new resources (the bucket and the object) and correctly identified the implicit dependency between them. - terraform apply: Executed the plan and created both the bucket and the object in the correct order after I confirmed with yes. - terraform state list: After the apply, this command would list both resources: aws_s3_bucket.my_bucket and aws_s3_bucket_object.file_upload.