Skip to content

Terraform Level 1, Task 23: Provisioning an AWS OpenSearch Domain

Today's task was an exciting step into the world of big data and log analytics. I used Terraform to provision an Amazon OpenSearch Service domain. This is a powerful, fully managed service used for everything from real-time application monitoring and log analysis to full-text search.

This exercise was a great lesson in provisioning more complex, managed services with Terraform. Unlike a simple EC2 instance, an OpenSearch domain is a complete cluster of servers managed by AWS. I learned how to define this cluster in code and deploy it using the standard Terraform workflow. This document is my very detailed, first-person guide to that entire process.

Table of Contents

The Task

My objective was to use Terraform to create a new AWS OpenSearch Service domain. The specific requirements were: 1. All code had to be in a single main.tf file. 2. The domain's name had to be devops-es. 3. The resource had to be created in the us-east-1 region. 4. The final state of my infrastructure had to match the configuration, verified by terraform plan showing "No changes."

My Step-by-Step Solution

The process involved writing a Terraform file that defined the OpenSearch domain and then running the standard three-step workflow.

Phase 1: Writing the Code

In the /home/bob/terraform directory, I created my main.tf file. I wrote the following declarative code to define my OpenSearch domain with a minimal, lab-friendly configuration. 

# 1. Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

# 2. Define the OpenSearch Domain Resource
resource "aws_opensearch_domain" "devops_es_domain" {
  domain_name    = "devops-es"
  engine_version = "OpenSearch_2.11"

  cluster_config {
    instance_type = "t2.small.search"
  }

  tags = {
    Name = "devops-es"
  }
}

Phase 2: The Terraform Workflow

From my terminal in the same directory, I executed the core commands.

  1. Initialize: terraform init (to download the AWS provider).
  2. Plan: terraform plan. The output showed me that Terraform would create one aws_opensearch_domain resource.
  3. Apply: terraform apply. After I confirmed with yes, Terraform began creating the OpenSearch domain. The prompt noted that this can take several minutes, so I was patient. The final success message confirmed the task was done.
  4. Final Verification: As required, I ran terraform plan one last time. The output was: No changes. Your infrastructure matches the configuration. This was the definitive proof of success.

Why Did I Do This? (The "What & Why")

- Amazon OpenSearch Service: This is a fully managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. OpenSearch is a powerful, open-source search and analytics engine, forked from Elasticsearch. - What is it for? I learned to think of it as a specialized database designed for two main things: 1. Log Analytics: You can stream all your application and server logs into an OpenSearch cluster. It indexes this data and provides a powerful query language and a visualization tool (OpenSearch Dashboards) to search, analyze, and create dashboards from your logs in real-time. This is the heart of an "ELK Stack" (Elasticsearch, Logstash, Kibana) or "OpenSearch Stack". 2. Full-Text Search: It's the engine that powers the search functionality on many websites, from e-commerce product catalogs to documentation sites. - Fully Managed Service: The biggest benefit is that AWS handles all the hard work of managing the cluster for me. I don't have to worry about provisioning servers, installing the software, patching, or handling node failures. I just define the cluster I want, and AWS keeps it running.

Deep Dive: A Line-by-Line Explanation of My main.tf Script

The code for this task defines a minimal but functional OpenSearch cluster.

[Image of an AWS OpenSearch domain architecture]

# Standard provider configuration block.
provider "aws" {
  region = "us-east-1"
}

# This is the resource block that defines my OpenSearch Domain.
# "aws_opensearch_domain" is the Resource TYPE.
# "devops_es_domain" is the local NAME I use to refer to this domain.
resource "aws_opensearch_domain" "devops_es_domain" {

  # The 'domain_name' argument sets the unique name for the cluster.
  domain_name = "devops-es"

  # The 'engine_version' argument specifies which version of OpenSearch to run.
  # It's a good practice to pin this to a specific version.
  engine_version = "OpenSearch_2.11"

  # The 'cluster_config' block defines the size and type of the servers (nodes)
  # that will make up the cluster.
  cluster_config {
    # 'instance_type' specifies the EC2 instance type for the data nodes.
    # 't2.small.search' is a small, older-generation type suitable for labs,
    # but for production, you would use a more modern, search-optimized type.
    instance_type = "t2.small.search"
  }

  # Standard tagging to give the domain a recognizable name.
  tags = {
    Name = "devops-es"
  }
}
A real production setup would be much more complex, including settings for instance_count, ebs_options to define storage, vpc_options to place the cluster in a private network, and a restrictive access_policies JSON document to control security.

Common Pitfalls

- Long Creation Time: As the note mentioned, creating an OpenSearch domain is not instant. It can take 10-15 minutes or more. It's important to be patient and let terraform apply finish without interrupting it. - Choosing the Wrong Instance Type: Not all EC2 instance types are supported by the OpenSearch service. Using an invalid type would cause the apply command to fail. - Security Configuration: My minimal example creates a domain that is open to the public internet (with some default protections). In a real-world scenario, this is a major security risk. A production domain must have a restrictive access_policies block and should almost always be placed inside a private VPC. - Forgetting to Verify with plan: The final verification step, running terraform plan and seeing "No changes," is a crucial best practice. It confirms that the state of the real-world infrastructure (what AWS has) perfectly matches the desired state declared in my code.

Exploring the Essential Terraform Commands

- terraform init: Prepared my working directory by downloading the aws provider plugin. - terraform validate: Checks the syntax of Terraform files. - terraform fmt: Auto-formats code to the standard style. - terraform plan: Showed me a "dry run" plan of the aws_opensearch_domain resource to be created. - terraform apply: Executed the plan and created the OpenSearch domain after I confirmed with yes. - terraform show: Shows the current state of my managed infrastructure. - terraform state list: Lists all the resources that Terraform is currently managing. - terraform destroy: The command to destroy all the infrastructure managed by the current configuration. This is crucial for avoiding costs after finishing a lab.