Pulumi Ep 11: The Automation API (Embedded Pulumi)

1. The Limitations of the CLI

#

Throughout the Fundamental and Intermediate tiers, our workflow has relied entirely on the Pulumi CLI:

1. Write TypeScript in index.ts.
2. Run pulumi up in the terminal.
3. Watch the interactive progress bar.

This is excellent for human operators. However, orchestrating this via software is a nightmare. If you want a Jenkins pipeline or an Express.js web server to run Pulumi, you would have to use child_process.exec("pulumi up --yes"), parse the raw stdout text, and try to extract the outputs via regex. This is extremely brittle.

2. Enter the Automation API

#

The Automation API is a strongly-typed SDK that exposes the core Pulumi Engine directly to your application code.

Instead of shelling out to a CLI, you call standard async functions like workspace.createStack() and stack.up(). The outputs are returned as native JavaScript objects.

Use Cases for the Automation API

#

• Internal Developer Portals (IDPs): Building a Next.js frontend (like Backstage) where developers can click “Provision Database”, triggering your backend to run Pulumi.
• SaaS Provisioning: Automatically spinning up dedicated, isolated tenant infrastructure when a customer enters their credit card.
• Custom CI/CD: Building sophisticated integration testing pipelines that dynamically create a stack, run Jest tests against the live URLs, and then tear down the stack.

3. Practice: Building an Express.js Provisioning Server

#

Let’s build a minimalist Internal Developer Portal. We will write a standard Express.js web server. When a developer sends a POST request to /api/create-bucket, our server will dynamically provision a Pulumi Stack and an S3 Bucket on the fly.

Step 1: The Infrastructure Logic (Inline Programs)

#

In standard Pulumi, you write infrastructure in an index.ts file on disk. With the Automation API, you can write the infrastructure logic as a standard TypeScript function directly inside your web server code! This is called an Inline Program.

// server.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import { LocalWorkspace } from "@pulumi/pulumi/automation";
import express from "express";

const app = express();
app.use(express.json());

// 1. Define the Infrastructure as a standard Async Function!
const createInfrastructure = async () => {
    // We can read context dynamically!
    const projectName = "SaaS-Platform";
    
    // Provision the bucket
    const bucket = new aws.s3.Bucket("tenant-bucket", {
        forceDestroy: true
    });

    // We must return any outputs we want our Express server to access
    return {
        bucketName: bucket.id,
        websiteUrl: bucket.bucketDomainName,
    };
};

Step 2: The Express.js Route Handler

#

Now, let’s write the POST route that triggers the Pulumi Engine.

// 2. The API Endpoint
app.post("/api/create-tenant", async (req, res) => {
    const tenantId = req.body.tenantId; // e.g., "customer-acme-corp"

    try {
        console.log(`Initializing Stack for ${tenantId}...`);

        // 3. Create a unique Stack for this specific customer
        const stack = await LocalWorkspace.createOrSelectStack({
            stackName: tenantId,
            projectName: "SaaS-Platform",
            // Pass the Inline Program function!
            program: createInfrastructure 
        });

        console.log("Configuring AWS Region...");
        await stack.setConfig("aws:region", { value: "us-east-1" });

        console.log("Running Pulumi Up...");
        // 4. Trigger the Pulumi Engine programmaticallly!
        const upResult = await stack.up({ onOutput: console.log });

        // 5. Extract the strongly-typed outputs
        const bucketUrl = upResult.outputs.websiteUrl.value;

        // 6. Return the physical infrastructure data to the user!
        res.status(200).json({
            message: "Tenant provisioned successfully!",
            stackName: tenantId,
            url: bucketUrl
        });

    } catch (error) {
        console.error(error);
        res.status(500).json({ error: "Provisioning failed" });
    }
});

app.listen(3000, () => console.log("Provisioning Engine running on port 3000"));

If you start this Express server and hit the endpoint with Postman:

curl -X POST http://localhost:3000/api/create-tenant \
  -H "Content-Type: application/json" \
  -d '{"tenantId": "customer-123"}'

Your web server will boot up the Pulumi Engine, communicate with AWS, provision the bucket, and return the live URL to the user in the JSON response.

You have just built the foundation of a modern SaaS architecture!

4. Local vs Remote Workspaces

#

In the example above, we used LocalWorkspace.createOrSelectStack(). This tells the Automation API to execute the Pulumi Engine on the local machine (the physical server running your Express app).

This means your Express server must have the pulumi CLI binary installed on its host OS, and valid AWS credentials in its environment variables.

For advanced enterprise architectures, you can use Remote Workspaces. This allows your Express server to instruct the Pulumi SaaS Cloud (or Pulumi Kubernetes Operator) to run the pulumi up command remotely, completely offloading the heavy lifting from your web server.

Troubleshooting & Common Errors

#

1. Command failed: pulumi exited with code 255

   • Root Cause: The Automation API is a wrapper around the Pulumi CLI. If the CLI crashes (e.g., due to missing AWS credentials on the host machine), the Node.js SDK will throw this generic error.
   • Solution: Check the error.stdout and error.stderr properties of the caught exception to read the actual Pulumi CLI output.

2. Concurrency Issues in Express

   • Root Cause: If 50 users hit your /api/create-tenant endpoint simultaneously, your Express server will spawn 50 concurrent Pulumi Engine processes. This will consume massive amounts of RAM and likely crash the server.
   • Solution: Do not run stack.up() synchronously in the HTTP request loop for heavy workloads. Instead, push the tenantId to a message queue (like AWS SQS or RabbitMQ), and have dedicated background worker nodes execute the Automation API logic.

Conclusion & Next Steps

#

The Automation API is Pulumi’s ultimate weapon. By embedding the IaC engine directly into standard software, you bridge the gap between traditional DevOps scripts and true Platform Engineering applications.

However, even with the Automation API, we are still relying on AWS resources that exist in the @pulumi/aws package. What if you need to automate a REST API for a proprietary internal company tool that doesn’t have a Pulumi provider?

In Episode 12: Dynamic Providers, we will learn how to write our own Custom Providers in pure TypeScript, allowing Pulumi to orchestrate absolutely anything with an API.