Crossplane Ep 13: Provider Families and Performance

1. The Monolith Problem

#

In the early days of Crossplane (v1.0 - v1.10), Upbound distributed providers as monoliths.

If you wanted to provision an S3 bucket, you installed xpkg.upbound.io/upbound/provider-aws.

When you ran that Helm chart, Crossplane did two things:

1. CRD Injection: It installed over 1,000 Custom Resource Definitions into the cluster (EC2, S3, RDS, SageMaker, GroundStation, etc.).
2. Controller Boot: It booted a single Go pod containing the reconciliation loops for all 1,000 of those resources.

The Impact on Kubernetes

#

Kubernetes was designed to manage a few dozen CRDs. Injecting 1,000 CRDs severely impacts the performance of the Kubernetes API Server and inflates the size of the etcd database.

Furthermore, the monolithic Go controller pod required over 2GB to 4GB of RAM just to idle, because it had to load the AWS SDK for every single AWS service into memory, even if you were only using S3!

2. Enter Provider Families

#

To solve this, Upbound re-architected their official providers into Provider Families.

Instead of one massive provider-aws package, they split it into 50+ micro-packages based on the AWS service namespace.

For example:

• provider-aws-s3 (Only contains S3 CRDs)
• provider-aws-ec2 (Only contains EC2/VPC CRDs)
• provider-aws-rds (Only contains Database CRDs)

By installing only the specific families you need, you reduce the CRD count in your cluster from 1,000 down to 20 or 30. Your API server remains lightning fast.

The Family Architecture

#

If you install provider-aws-s3 and provider-aws-ec2, you will notice that Crossplane actually boots three pods:

1. provider-aws-s3
2. provider-aws-ec2
3. provider-family-aws

What is that third pod? It is the shared authentication layer! Instead of forcing the S3 pod and the EC2 pod to independently authenticate with AWS and parse your ProviderConfig, the provider-family-aws pod handles the IAM authentication globally, and shares the secure token with the micro-providers via a local gRPC socket.

This drastically reduces AWS API rate-limiting issues.

3. Scaling the Reconciliation Loop

#

Even with Provider Families, you must monitor the performance of your Crossplane controllers.

By default, Crossplane limits the concurrency of its reconciliation loops. If your company suddenly merges an ArgoCD PR that requests 500 new S3 buckets, the provider-aws-s3 pod might take an hour to provision them because it is only processing 5 requests at a time.

You can tune the performance of the Provider by passing custom flags to the controller during installation.

Tuning the Provider

#

Create a ControllerConfig to override the Go binary arguments:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: high-performance-config
spec:
  args:
    # Increase the maximum number of concurrent reconciles!
    - --max-reconcile-rate=50
  
  # Ensure the Pod has enough RAM to handle the concurrency
  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 500m
      memory: 512Mi

Then, attach it to your Provider installation:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.14.0
  # Bind the tuning configuration!
  controllerConfigRef:
    name: high-performance-config

4. Managing AWS Rate Limits (Throttling)

#

When you increase --max-reconcile-rate, you run into a new problem: AWS API Throttling (HTTP 429 Too Many Requests).

If Crossplane tries to create 50 buckets simultaneously, the AWS API will actively block your IP address to protect its own infrastructure.

Crossplane handles this gracefully using Exponential Backoff. If a physical AWS API call fails due to a 429 error, the Crossplane controller will wait 1 second before retrying. If it fails again, it waits 2 seconds, then 4 seconds, up to a maximum delay (usually 60 seconds).

Because of this, you should expect to see ReconcileError events in your kubectl describe logs during massive burst deployments. This is normal behavior for a distributed control plane.

Troubleshooting & Common Errors

#

1. cannot create ProviderConfig: CRD not found

   • Root Cause: You installed a micro-provider (like provider-aws-s3), but the cluster doesn’t understand the ProviderConfig API yet because the provider-family-aws pod hasn’t finished booting.
   • Solution: Always wait for the Provider Family to be HEALTHY before applying ProviderConfig objects.

2. The Kubernetes API Server is extremely slow (kubectl commands take 5 seconds)

   • Root Cause: You likely installed the monolithic legacy provider instead of the Provider Families. etcd is struggling to index 1,000 massive Custom Resource Definitions.
   • Solution: Uninstall the monolith. Migrate your Compositions to use the modern xpkg.upbound.io/upbound/provider-aws-* family packages.

Conclusion & Next Steps

#

Optimizing a Crossplane cluster is a balancing act between Kubernetes RAM, Controller Concurrency, and AWS Rate Limits. By utilizing Provider Families and tuning your ControllerConfig, you can build an IDP capable of managing tens of thousands of physical cloud resources.

But what about Security? If a developer requests a 500GB database in their Claim, how do you mathematically prevent them from requesting a 5,000GB database and bankrupting the company?

In Episode 14: Custom Composition Webhooks, we will learn how to write a Kubernetes Validating Webhook to intercept and reject non-compliant Developer Claims before Crossplane even sees them!