Speed Up Terraform Pipelines by Pre-Caching Providers the Smart Way

If your CI/CD environment runs Terraform dozens—or even hundreds—of times a day, chances are you’re wasting time and bandwidth repeatedly downloading the same provider packages. Pre-caching Terraform providers in your pipeline images is a simple yet massively underutilized optimization that can drastically reduce runtime and improve reliability. In this post, I’ll walk you through a new project I’ve built that automates the entire pre-caching process using a Docker-based setup and a YAML config. Whether you want faster pipelines, fewer external dependencies, or more control over your Terraform environments, this guide will show you how to make it happen—step-by-step.

In a busy platform-as-a-service environment within a larger organization, it's common to see hundreds—if not thousands—of pipelines run daily for various Terraform provisioning tasks. This creates a significant amount of redundant network activity and wasted time as the same Terraform providers are downloaded repeatedly.

Fortunately, Terraform includes a local plugin cache that avoids re-downloading provisioners that already exist. Pre-populating this cache inside your CI/CD pipeline images offers two key benefits:

• Reduced Provisioner Run Time
  Eliminates the near-constant re-downloading of external binary packages from the Terraform Registry, leading to faster pipeline execution.

• Reduced External Dependencies
  The Terraform Registry has experienced outages in the past. When that happens, pipelines relying on live downloads can fail. Pre-caching ensures provisioning remains uninterrupted, even during external outages.

To simplify this process, I've created a project that includes a Dockerfile and supporting scripts that process a yaml file that contains target git repos (and any subpaths) that the image would be used within. When built, this image will:

• Pre-cache providers for the defined target git projects/folders
• Install multiple versions of the terraform and other binaries via mise

Start by cloning this repo into your organization then make updates as needed:

Update the config/provisioners.yml file with all of your downstream terraform provisioning projects, their branches, and target folders that will be processed.

1. Update the mise.toml file to include terraform and other binary versions you wish to have included.
2. Add CICD pipeline code for your organization to build and push your image.

NOTE The order of versions in mise.toml matter. The first one in the list will be used by default. See the configuration of mise for more details on this wonderful tool.

If you need to include latest versions of a provider or have a need to manually define one, you can easily do this as well. Edit the local config/provisioners.yml file and add a local path that contains a terraform version.tf file within the local config directory. Examples are provided in this project (that can be removed if you do not need them)

To see how this will work, you can run everything locally using the included taskfile tasks within.

task providers

This should produce a local tempproviders folder with all of the plugins for your downstream terraform provisioners.

Additionally, helper tasks for building and shelling into the container image are included.

task docker:build docker:shell

Shaving off 10 seconds per pipeline might sound trivial, but across hundreds of daily runs, the cumulative time savings and resilience improvements are substantial. Optimizing your CI/CD process by eliminating external dependencies and speeding up execution is a small engineering win with big operational impact.