Terraform Ep 5: Querying Existing Cloud Infrastructure with Data Sources

1. The Core Concept: What is a Data Source?

#

In enterprise environments, responsibilities are often split across teams. The Networking Team might create the Core VPC and Subnets. The Application Team then needs to deploy EC2 instances into those Subnets.

How does the Application Team get the ID of the Subnet? They could ask the Networking Team on Slack and hardcode the ID (subnet-01234abcd) into their main.tf. However, if the Networking Team ever recreates the Subnet (resulting in a new ID), the Application Team’s code will break instantly.

The robust, automated solution is to use a Data Source. The Application Team can write a query in their Terraform code: “Hey AWS, find me the Subnet that has the tag Name = Core-App-Subnet and return its ID.”

Anatomy of a Data Source

#

The syntax is identical to a resource block, but uses the data keyword:

data "provider_data_type" "local_logical_name" {
  filter_argument = "value"
}

2. Practice: Fetching the Latest Ubuntu AMI

#

When creating an EC2 instance, you must provide an Amazon Machine Image (AMI) ID (e.g., ami-0c55b159cbfafe1f0). AWS constantly updates these AMIs with security patches, meaning the ID changes every few weeks. Hardcoding it guarantees your code will rot.

Let’s use a Data Source to dynamically fetch the most recent Ubuntu 22.04 AMI ID directly from Canonical (the creators of Ubuntu).

Create a new directory named terraform-data-sources:

mkdir terraform-data-sources
cd terraform-data-sources

Create a main.tf file and paste the following:

# main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# The Data Source query
data "aws_ami" "latest_ubuntu" {
  most_recent = true
  owners      = ["099720109477"] # Canonical's official AWS Account ID

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# Output the dynamically fetched ID to the terminal
output "ubuntu_ami_id" {
  value = data.aws_ami.latest_ubuntu.id
}

Execution and Observation

#

Initialize the directory:

terraform init

Now, run the plan phase. Watch closely what happens:

terraform plan

Expected Terminal Output:

Changes to Outputs:
  + ubuntu_ami_id = "ami-0fc5d935ebf8bc3bc"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Did you see that? During the plan phase, Terraform executed the API query to AWS, filtered the massive list of thousands of AMIs, found the exact match for Ubuntu 22.04, and returned the ID (ami-0fc5d935ebf8bc3bc).

You can now pass this dynamically fetched ID directly into an EC2 resource block like this:

resource "aws_instance" "web_server" {
  # Dynamically referencing the Data Source!
  ami           = data.aws_ami.latest_ubuntu.id
  instance_type = "t3.micro"
}

3. Practice: Querying an Existing VPC

#

Let’s look at the cross-team collaboration scenario discussed earlier. Suppose the Networking Team created a VPC tagged with Environment = Production.

You can query it like this:

data "aws_vpc" "production_vpc" {
  tags = {
    Environment = "Production"
  }
}

# Now you can use it to create a security group inside that VPC
resource "aws_security_group" "web_sg" {
  name        = "web-server-sg"
  description = "Allow HTTP traffic"
  
  # Injecting the dynamically fetched VPC ID
  vpc_id      = data.aws_vpc.production_vpc.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

This guarantees your code is completely portable. If the Networking Team destroys and recreates the VPC, your code will automatically find the new ID on the next terraform plan.

Troubleshooting & Common Errors

#

1. Your query returned no results.

   • Root Cause: The filters you provided in the data block are too strict, or the resource simply does not exist in the AWS region you are targeting.
   • Solution: Verify the resource exists in the AWS Console. Double-check your tags and spelling. AWS tags are case-sensitive (Production is not equal to production).

2. Your query returned more than one result.

   • Root Cause: The API query must return exactly ONE unique match. If your filter (e.g., Environment = Production) matches two different VPCs, Terraform panics because it does not know which one you want.
   • Solution: Make your filters more specific (add more tags), or use a plural data source (like data "aws_vpcs") if you intentionally want a list of IDs.

Conclusion & Next Steps

#

You have conquered the final pillar of Terraform Fundamentals. You understand how to declare desired state (resource), parameterize your code (variables), extract information (outputs), and query existing reality (data sources).

You are no longer a beginner.

In the next section, we enter Tier 2: Intermediate Constructs. We will leave simple linear configurations behind and learn how to programmatically control Terraform using built-in functions, for_each loops, and dynamic blocks in Episode 6: Local Values, Built-in Functions, and Interpolation.