Cloud Infrastructure Pipelines – Part 1

Terraform is a good tool for cloud infrastructure, especially more recent versions with backends for locking state files in Amazon, Azure, and Google clouds. It provides a level of independence that is still somewhat cloud specific but better than having to learn the templating languages of each cloud platform.  But where cloud infrastructure automation falls down is scalability and implementing best practices for development.

Imagine having to write Terraform code for hundreds of applications and environments during a major cloud migration. How many teams would have to learn and maintain Terraform code? There would be “fill-in-the-blanks” templates but still a manual and error-prone process. Using Jinja templates could address some of the manual process by generating the Terraform code with a YAML file for input variables. Or even skip Terraform and generate Cloud Formation or ARM templates if the focus is on a single cloud provider.

Which language to use for running the Jinja templates? Ansible seems the natural choice but Python or Salt are also options. And why use Terraform at all? Ansible has modules similar to Terraform for all the major cloud providers.  But unlike Terraform or the native template engines of the cloud providers, Ansible does not keep track of state. Instead it discovers state each time the playbook is run. Typically the playbook itself or companion YAML files represent the state of the cloud infrastructure. But without an external state representation, we cannot “plan” our changes before we “apply” them or check configuration drift from manual changes made via the console.

The Jinja templates would generate consistent terraform code, perhaps two files, a main.tf and variables.tf, to follow best practices. But creating the YAML file would be still a manual process. To address that, perhaps the YAML file could be generated by Jinja templates as well, making this a two-stage templating process. But where do inputs come to create the YAML file, if not by manual edits?  Find out in Part 2.