4 Tools to Automate Kubernetes Cluster Deployments
Kubernetes is hard
That is what you usually see voiced by community members when digging through obscure Github forums for the arcane k8s incantations to get your cluster nodes working smoothly.
This is probably why managed Kubernetes cluster services are such a hot topic, they promise to eliminate much of the deployment and management hassles, which can allow you to focus on the workloads you need to run. But if you are still going roll your own clusters, I’d first look through a few of the options covered in this article to see if one of them will save you some headaches.
Before we look at the tools, lets define a few deployment targets when it comes to Kubernetes. At the bare minimum we are looking for an application that consumes declarative manifests of some sort and bootstraps fully functioning Kubernetes clusters. We will also want to ensure that a cluster is highly available.
Goal 1: Highly available
A highly available K8s cluster is more than just having 3 nodes. The nodes must be configured with the right roles to be considered fault tolerant and highly available.
The bill of materials for an HA deployment include, at the minimum:
- 2 controller (master) nodes
- 3 etcd (config store) instances
- 2 worker nodes
This diagram shows HA components on nodes of a 3-node cluster (notice the included load balanced services).
NOTE: This does not do justice to the complications around truly HA K8s clusters. Read a great discussion on true HA here and an insightful article here.
Goal 2: Not highly available
A Kubernetes cluster is either highly available or it isn’t. Anything that is not highly available in Kubernetes may as well be a single node local deployment and is considerably easier to get working for the most part.
Single node K8s clusters are useful for development, deployment smoke testing, and treading the Kubernetes waters a bit. We will not be going over these options and strongly recommend simply using Docker CE (edge) single node Kubernetes option, minikube, or other purpose-built projects that make it easy to bring up a single node cluster on just about any desktop OS.
Target platforms
Most of these tools target AWS or Azure VMs to create Kubernetes clustering services. Some might also target other cloud providers or even local infrastructure as well. The general idea of all these solutions is to templatize the k8s cluster deployment process.
Some of the following are almost certainly required for your k8s cluster deployments regardless of the target platform:
- Shared/Clustered/HA backend persistent storage
- External Load Balancer Service/IP
- DNS zone for <clustername>.<zone>.<zoneroot> (Ideally external)
- Ability to create host VM instances and other IaaS for K8s cluster itself.
Going with a cloud-based target makes deployment a bit easier because you can cherry-pick from IaaS/PaaS that the provider offers to add geo-redundancy and other other services that make k8s useful.
Standard tools
The following tools I consider ‘standard’ and may be used by and of the deployment tools to ‘get the job done’.
Kubeadm/kubectl – These are known by most and can be used to deploy an entire cluster once the base hosts/networking is setup and available.
Ansible – configuration deployment and management platform. Ansible is a swiss army knife of configuration tools and can be used for every step of a k8s deployment. In fact some tools are simply complex Ansible playbooks. Ansible playbooks themselves can be authored to be declarative and idempotent as well.
Terraform – A declarative infrastructure deployment tool that can greatly simplify declaration and deployment of your infrastructure to multiple platforms.
Cloudformation Templates – JSON templating language for cloud deployment to AWS.
ARM Templates – JSON templating language for cloud deployment to Azure.
Deployment tools
Each of these tools can be used to deploy Kubernetes clusters to various platforms.
KubeSpray
Ansible driven Kubernetes cluster deployment playbook with deep customization and control. From their site you can deploy a production ready kubernetes cluster and has the following features:
- Can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Oracle Cloud Infrastructure (Experimental), or Baremetal
- Highly available cluster
- Composable (Choice of the network plugin for instance)
- Supports most popular Linux distributions
- Continuous integration tests
Website: https://github.com/kubernetes-incubator/kubespray
Thoughts: I listed this one first as I consider it to be a good starting point. It has an included Vagrantfile definition for quick testing (it can take quite a while and may require you to run vagrant provision more than a few times to complete.). The entire kubespray project is really an Ansible playbook that, you likely already know, can be customized to no end. That makes this project very flexible. As such it is also a bit more complex to get working properly in my experiences.
Conjure-up
From their site:
conjure-up lets you summon up a big-software stack as a “spell” — a model of the stack, combined with extra know-how to get you from an installed stack to a fully usable one. Start using your big software instead of learning how to deploy it.
Website: https://conjure-up.io/
Thoughts: This one is relatively new to me. The spells listed in their registry are more difficult infrastructure deployments like OpenStack, Kubernetes, and Sparc stacks. This particular tool will deploy the Canonical distribution of Kubernetes across several cloud providers or locally. That makes sense as the tool itself is a wrapper around Canonical’s juju framework. conjure-up gives you super pretty console prompts along the way to help guide the deployment but you can also easily automate the process with a ‘Conjourfile’.
While the whole process is easy and all, be careful as I somehow ended up with a cluster out in China when I had thought the process had failed against us-east-1. A Chinese east region k8s cluster built from conjure-up with mostly default values for a AWS will cost about 50 USD a day btw 🙂
All joking aside, conjure-up is well thought out, has native AWS integration, and is totally worth keeping an eye on.
Kops
From their site:
Kubernetes Operations (kops) – Production Grade K8s Installation, Upgrades, and Management
- Automates the provisioning of Kubernetes clusters in AWS and GCE
- Deploys Highly Available (HA) Kubernetes Masters
- Built on a state-sync model for dry-runs and automatic idempotency
- Ability to generate Terraform
- Supports custom Kubernetes add-ons
- Command line autocompletion
- YAML Manifest Based API Configuration
- Templating and dry-run modes for creating Manifests
- Choose from eight different CNI Networking providers out-of-the-box
- Supports upgrading from kube-up
- Capability to add containers, as hooks, and files to nodes via a cluster manifest
Website: https://github.com/kubernetes/kops
Thoughts: I put this one on the list because they have an almost 800 line Makefile. Honestly, that is kind of impressive! Aside from that, I’ve actually used this to templatize and deploy, then destruct, a Kubernetes cluster in AWS flawlessly. It was super easy to setup and get working and I believe it is likely what I’ll be employing for rapid cluster deployments in future projects.
kube-aws
From their site:
kube-aws is a command-line tool to create/update/destroy Kubernetes clusters on AWS.
- Create, update and destroy Kubernetes clusters on AWS
- Highly available and scalable Kubernetes clusters backed by multi-AZ deployment and Node Pools
- Deployment to an existing VPC
- Powered by various AWS services including CloudFormation, KMS, Auto Scaling, Spot Fleet, EC2, ELB, S3, etc
Website: https://kubernetes-incubator.github.io/kube-aws/
Thoughts: I’ve not had the chance to deploy using this tool but the documentation is very well thought out around what the requirements are for existing VPCs, subnets, and other AWS cloud services and what will be required to get things running.
Fortunately, the project name clearly states its only drawback, that it only targets AWS.
Even More Projects
Before going further, it would be wise for the reader to pop over to the Cloud Native Computing Foundation and explore the many projects under this umbrella site. All of these projects are likely to draw more attention and activity being backed by the CNCF.
Next, there are several other Kubernetes cluster deployment tools you may want to investigate as well. Here are very short descriptions and links to a few worth keeping on the radar:
BootKube – a tool for launching self-hosted Kubernetes clusters.
Kustomize – Template-free configuration customization for Kubernetes (Note: This is not about deploying a cluster but is extremely interesting as an arbitrary Kubernetes workload/configuration transform management tool).
Spread – a command line tool that makes it easy to version Kubernetes clusters, deploy to Kubernetes clusters in one command, and set up a local Kubernetes cluster.
Kubicorn – Very interesting deployment and lifecycle management tool for K8s. Goes beyond initial deployment with ‘cluster snapshot’.
Bonus! Here is a longer list of interesting Kubernetes projects of all kinds as well.
Conclusion
Kubernetes is hard but deploying it does not need to be with so many great tools being created to address the complexity concerns. I’ve listed a few purpose driven tools to get a production ready Kubernetes cluster up quickly. Each of these tools can be used to deploy Kubernetes clusters to various platforms. I recommend reviewing them all to see which will fit your own cluster deployment needs.
About the Author
Zachary Loeber is a Sr. Consultant with the Cloud practice within SPR. He is a passionate IT industry veteran in cloud automation, security, and design. Zach enjoys coding in PowerShell but is not adverse to Python, bash, groovy, or any form of declarative language.
