This site uses cookies and by using the site you are consenting to this. We utilize cookies to optimize our brand’s web presence and website experience. To learn more about cookies, click here to read our privacy statement.

How Developers, DevOps Teams can Navigate the Istio Rabbithole

Author: Zachary Loeber Posted In: Cloud, DevOps

Istio is an open platform that helps developers and DevOps professionals manage and secure their microservice-based applications. It can be overwhelmingly complex; but using the istio operator, helmfile, and local kubernetes in docker clusters, we can simplify the learning journey.

I’ve been meaning to dive into Istio for a while now. In a way, it was a purposeful delay of gratification as I knew Istio would be a riotously good learning experience that would be worth spending some time on. I recently found a reason to install it and was able to vet out some of what it can do. And now that I have, I’m glad I waited for the Istio hype-dust to settle. The recent efforts in the project to simplify its complexity via an operator has definitely worked out in my favor.

In this article I’ll go over what I used to plumb the depths of Istio (okay, let’s be honest, skim the surface). This article is more about the layout of tooling and framework I created to rapidly vet out new technologies on Kubernetes locally and how you can use it in your own learning journey.


Setting Profiles

Before starting, I ran into my first real conundrum: Which Kubernetes distribution should I use for local testing? As a docker runtime will be the common element across most testing environments I’ll be using (Ubuntu, WSL, OSX), I quickly determined that Kubernetes on docker was to be used. Toward this end, I wanted to use either k3d (k3s in docker) or kind (kubernetes in docker). Both are fast, have strong communities, and quite simply rock. And because I want it all, I created a project to use either at will. I call this little scaffolding project CICDHelper as it is meant to be a starting point for developing solutions locally before transposing them to a CICD pipeline.

So I broke down the tasks for using k3d or kind into generic commands sourced from individual Makefiles that get loaded based on a single top level profile. The top level profile gets loaded by a single environment variable called `PROFILE`. I originally was also going to include minikube, libvirt, and vagrant, but it felt like overkill. This hacky plugin system would still allow me to do so if required though!

For istio testing we may use a profile of ‘/config/istio.env’ with contents like the following to default to a kind-based cluster.
This tells us that:
  • we are using a kind-based 1.18.2 Kubernetes cluster,
  • with a name of cicd-istio,
  • that uses the helmfile environment of istio and an additional custom taskset for istio specific operations.

NOTE: I purposefully used istio as an example for this article as it is one of the more involved deployments and necessitated a custom taskset. The CICDHelper project started as a simple framework for gluing together various kinds of helm charts into helmfiles and deploying them rapidly to various environments. Most basic scenarios do not require custom tasksets at all.

# Set our working profile so we don't have to pass it to each make command manually
export PROFILE=istio

# Download any dependant binaries
make deps

# Start the cluster defined in our profile
make cluster/start

To ensure that your local kubectl profile is pointing to the newly created cluster update your current session’s `KUBECONFIG` env var. I also alias ‘kube’ to the locally downloaded kubectl binary.

export KUBECONFIG=`make kube/config/file`
alias kube=$(pwd)/.local/bin/kubectl
kube get ns

Deploying Istio

Istio can be deployed several ways. I’ve opted to use istioctl to do the deployment of the operator that then consumes a declarative manifest for configuring the istio services (also known as profiles), as that seemed to be the most declarative and straightforward. I made minor changes to the default profile to make the deployment more usable for the example bookinfo application and store it in ‘./deploy/istio/profile.yaml’. The only real difference in my profile versus the demo profile is that I disabled the egress controller.
# Install istio operator via istioctl using the ISTIO_PROFILE of demo
make istio/deploy

Deploy BookInfo

The bookinfo application is an example microservice to deploy into Istio. If you are just learning Istio then you should probably do this part manually and walk through one of the several learning exercises that Istio documentation provides.

What I’ve done is turned the deployment into a single command that uses this framework, helmfile, and the raw helm chart. This is the meat of what I initially set this little framework up to do anyway. Each helmfile can be a combination of several charts and even other helmfiles. As such, I like to refer to what gets deployed simply as a `STACK`. In this case we will deploy the bookinfo stack.
# Deploy the bookinfo helmfile for the full deployment along with the gateway and virtualroutes it requires
make helmfile/sync STACK=bookinfo
If you want to deploy additional ingress for kiali:
make helmfile/sync STACK=istiodashboards
You can also quickly start up any dashboard istioctl can find via kubectl proxy. Kiali is the one most people like to look at and show off (use admin/admin to login):
make istio/dashboard/kiali
Very shortly, we will setup dns forwarding with istio ingress to remove the need for a proxy connection. But first, let’s address some technical difficulties around local ingress deployments.

Hurdle – The LoadBalancer

When setting up a bare metal Kubernetes cluster, some components are lacking entirely from your typical cloud-based deployments. One in particular is sorely missed, the LoadBalancer resource. Without a loadbalancer we are left with using kubectl proxy commands to access exposed services like peasants! To work around this k3d includes its own implementation of a software loadbalancer and works out of the box. For kind we end up simply deploying MetalLB to the clusters and have it use a range of IPs that should be available and bridged in most docker deployments (172.17.0.x). I’ve set it up so that metallb gets installed automatically when running the `cluster/start` operation when deploying a kind cluster.
NOTE: kind will use the existing bridge docker network. k3d creates its own bridge network.

Hurdle – DNS

The loadbalancer allows us to access the cluster running in docker via IP address easily enough. But we also want to be able to test out real DNS queries and deployment of ingress in a manner which we can float up to a production level deployment in the future. In order to do this, we can manually modify hosts files on our workstations like button-mashers. Or, we can try to manipulate how DNS works on our workstation in craftier ways.

Many moons ago I once setup dnsmasq to forward to specific DNS servers based on the zone being queried and simply set my workstation to use dnsmasq so that I could administer multiple disparate forests at once. I started looking to do this again via a docker image instead and found another excellent project already out there for such DNS trickery called dns-proxy-server.

It is deceptively simple to use dns-proxy-server as it does not require you to make any system changes manually and suddenly, your resolution for custom domains will just ‘start working’. Behind the scenes there is a simple json configuration file that we can use to modify zone and record lookups and when we run the image we are also going to volume mount our resolve.conf file for it to modify so that lookups for our system first flow through the container.

Here is how you would enable this using CICDHelper.
# Start another pod to do dns forwarding of the url for this and the default domain (int.micro.svc) to your cluster loadbalancer 
make istio/start/dnsforward
This little line actually does quite a bit. First, we need to have some zone definition to forward requests to and for any future ingress definitions we wish to create. Because I’m a fan of helmfile, I put this in a helmfile environment definition yaml file. I store a parameterized template for dns-proxy-server in `deploy/dnsproxy/config.json`. This gets rendered via gomplate with the yaml environment file as a datasource and stored in a temporary directory before being volume mounted into the container.

This is all done with the following code.
tmpdir=$$(mktemp -d) && echo "$${tmpdir}" && \
export STACK_INGRESS_INTERNALIP=`$(kubecmd) -n istio-system get svc istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` && \
$(gomplate) \
-d config=$(CONFIG_PATH)/environment.$(ENVIRONMENT).yaml \
--file $(DEPLOY_PATH)/dnsproxy/config.json \
--out "$${tmpdir}/config.json" && \
docker run --rm -d \
--name dns-proxy-server \
-p 5380:5380 \
-v $${tmpdir}:/app/conf \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/resolv.conf:/etc/resolv.conf \
@echo ""
@echo "View dnsforwarding admin page at: http://localhost:5380"
I do this specifically in the istio taskset as retrieving the ingress IP is specific to the ingress deployment Istio creates. Were we to deploy some other ingress controller or do this in production, it would be wise to statically set the ingress loadbalancer IP upon creation instead of dynamically sourcing it like this. There is room for improvement here, I suppose.

In any case, if you have started up the DNS proxy and now visit the default product page for bookinfo, http://bookinfo.int.micro.svc/productpage, it should render for you.

Additional Helmfiles

This is technically a fully working application. But if you want to see other dashboards (such as kiali) then more virtual routes need to be created to handle the requests. I’ve gone ahead and created another helmfile called ‘istiodashboards’ that publishes kiali specifically. It is nominal to setup additional virtual routes for prometheus or grafana as well, simply update the existing helmfile or create new one in the helmfile directory.

## Enable the kaili dashboard
make helmfile/sync STACK=istiodashboards

## Disable it when you are done (as an example)
make helmfile/remove STACK=istiodashboards

Istio BookInfo Kaili Dashboard

Shutting Things Down

This is the easiest part:

make cluster/stop istio/stop/dnsforward


Now Use k3d

Prefer k3d? Ok, using k3d instead of kind for the cluster is easy. I’ve already created the profile for it (./config/istio-k3d.env).
make deps cluster/start istio/deploy helmfile/sync istio/start/dnsforward STACK=bookinfo PROFILE=istio-k3d
make helmfile/sync STACK=istiodashboards PROFILE=istio-k3d


That was fun, right? In the next article I’ll cover more in depth how we can turn that raw helm chart mess into a usable part of an archtype helm chart so that we can deploy and test multiple ingress controllers and expose resources using the same set of base definitions.

Just as I was finishing this article I ran across the “Istio Succinctly” ebook which dives into the various components of Istio. I highly recommend grabbing it and giving it a once over. Then with that knowledge and this little project, setting up additional components like citedel and galley for local testing will be a cake walk.

Here are some links in no particular order:

CICDHelper – My little abstraction framework

Istio Succinctly – An excellent free ebook on Istio

Gomplate – The king of devops file templating utilities

Helmfile – Like helm, only 10x better

dns-proxy-server – Intercept local dns resolution like a boss