Let Kubernetes Automate Your Configs and Policies with Anthos Config Management

With the widespread adoption of containers among organizations, Kubernetes has become the de facto standard to deploy and operate containerized applications. As you expand your Kubernetes footprint, creating and enforcing consistent configurations and security policies across your growing fleet can create friction. Anthos Config Management (ACM) tackles this by enabling you to set and enforce consistent configurations and policies for your Kubernetes resources — wherever you build and run them — and manage Google Cloud services the same way. Anthos Config Management makes it even easier to declaratively specify resources with YAML or JSON by adding pre-built, opinionated config and policy automations, such as creating a secure landing zone and provisioning a GKE cluster from a blueprint. 

As the famous adage goes “show, don’t tell.” In this post, We’ll show an example of how you can leverage a GitOps methodology using Config Sync to apply configuration consistently across clusters and environments with an auditable, transactional, and version-controlled deployment process. And we’ll use Policy Controller for the enforcement of programmable policies that represent constraints on the desired state. 

First, follow the setup process in the Git Repo to build out the infrastructure and then deploy the services as per the directions.

Now imagine that you’re a platform administrator, responsible for not just one of the services inside your organization, but for the entire Kubernetes environment, including the development, staging, and production clusters. An app developer may care most about testing their code and getting features into production with minimal friction, but your concerns are probably different. You care about consistency across the whole platform — that certain baseline configuration and policy are always deployed and in sync across all the clusters. (You do not want a developer to kubectl apply -f one of those resources by mistake, and you especially don’t want that to happen without knowing about it) You also care about compliance with your industry regulations, and you might work directly with your security team to make sure the necessary policies are in place.

In other words, as a platform admin, you really care about two things with the Kubernetes Resource Model (KRM) : 1) Consistency, and 2) Protecting the clusters from unsafe configuration. Let’s take a look at how using Anthos Config Management can help you achieve those two goals. 

​​In this example, we will deploy a mock banking application called Cymbal-Bank.

For the purpose of this exercise, let’s assume that you have already installed Config Sync and Policy Controller in your clusters.

  (foo) arunkasi$ gcloud beta container hub config-management status \
>--project=arun-krm
Name            Status          Last_Synced_Token  Sync_Branch  Last_Synced_Time      Policy_Controller                          
cymbal-dev      SYNCED          720476f            main         2021-12-02T12:31:37Z  INSTALLED                                  
cymbal-prod     SYNCED          720476f            main         2021-12-02T12:32:34Z  INSTALLED                                  
cymbal-staging  SYNCED          720476f            main         2021-12-02T12:33:11Z  INSTALLED

  (foo) arunkasi$ kubectl get pods -n config-management-system
NAME                                          READY   STATUS    RESTARTS   AGE
admission-webhook-64948475d7-74j29            1/1     Running   3          32m
admission-webhook-64948475d7-ggwt2            1/1     Running   2          32m
config-management-operator-7d5f54c74c-pbxbs   1/1     Running   0          32m
reconciler-manager-7c6dccbb5f-6phml           2/2     Running   0          32m
root-reconciler-66c788cc97-cfl4s              4/4     Running   0          2m15s

(foo) arunkasi$ kubectl get pods -n gatekeeper-system
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-8479c4fd8f-jmvcv                1/1     Running   0          2m48s
gatekeeper-controller-manager-7f778d8b94-vxsq9   1/1     Running   0          33m

The first set of pods, in the config-management-system namespace, run Config Sync. These workloads periodically check your GitHub policy repo for any updates to the KRM source of truth stored there, and deploy those updated resources to the cluster. These workloads help ensure that any resources entering the cluster — both through CI/CD or through Config Sync — adheres with any policies we set.

The second set of workloads, in the gatekeeper-system namespace, run Policy Controller,  which is based on the open-source Gatekeeper project, which in turn grew out of the OpenPolicyAgent project, part of the Cloud-Native Computing Foundation (the same foundation that hosts Kubernetes). 

Policy Controller is a Kubernetes dynamic admission controller that checks, audits, and enforces your clusters’ compliance with policies related to security, regulations, or arbitrary business rules. Policy Controller policies are broken up into two separate objects: Constraints and ConstraintTemplates. Having two distinct objects allows for separation of the policy definition (ConstraintTemplate) and policy enforcement (Constraint). 

Let’s unpack that. An admission controller is a pod that sits at the “gate” of a Kubernetes cluster, watching what’s coming into the API server and doing some type of operation on that resource, before it’s “allowed inside” the cluster and persisted in etcd. That “operation on the resource” could include modifying the resource in-flight (MutatingAdmissionWebhook) or rejecting the resource entirely (ValidatingAdmissionWebhook). Policy Controller uses the second kind of webhook to validate incoming config against the policies it knows about, allowing policies in, or rejecting them. 

Policy Controller enforces constraints, but what do those constraints actually look like? What kinds of Kubernetes resources can we use PolicyController to accept or reject?

Here’s an example constraint:

  (foo) arunkasi$ cat constraint-ext-services/constraint.yaml
# Blocks the creation of Ingress and Service type=LoadBalancer resources 
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoExternalServices
metadata:
  name: dev-no-ext-services
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-dev
spec:
  parameters:
    internalCIDRs: [ ]

Here, “kind: K8sNoExternalServices” refers to the k8snoexternalservices constraint template already installed on the cluster. And notice how we’re using Config Sync’s cluster-name-selector annotation to scope this resource to the cymbal-dev cluster only.

This Constraint implements the K8sNoExternalServices Constraint Template with concrete information about our environment.

Expected output:

  (foo) arunkasi$ kubectl get constraint 
NAME                                                                  AGE
k8snoexternalservices.constraints.gatekeeper.sh/dev-no-ext-services   2m15s

(foo) arunkasi$ kubectl apply -f constraint-ext-services/contacts-svc-lb.yaml
Error from server ([dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed)Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service" Name: "contacts", Namespace: "contacts"for: "constraint-ext-services/contacts-svc-lb.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed

With the above command, you just deployed a Policy Controller policy via Config Sync! When we went to deploy an application that violated the policy constraint, ACM correctly denied the request. If you would like to test drive this lab please follow the setup on this Git Repo developed by our Staff Developer Relations Engineer, Megan O’Keefe. If you’re a GKE customer, you can now use Anthos Config Management at a low incremental per-cluster cost. To learn more about Anthos Config Management and explore best practices, quickstarts, and tutorials, visit Google Cloud documentation page.

Take the next step

Start building on Google Cloud with $500 in free credits and 20+ always free products.