In the first two articles, we have seen that although simplicity is a key aspect of Kustomize, it is still a powerful tool to manage client-side Kubernetes resources. Nevertheless, there are still some features lacking that we would probably like to have:

Kustomize has always had some kind of means to add functionality by building plugins. From Kustomize’s point of view, the generators and transformers are already kind of plugins anyway. In the past, however, the plugins were proprietary and complicated to build in the form of executable scripts or go plugins. This way is becoming deprecated and will be replaced by a more KRM-driven approach. One of the first steps is to introduce KRM functions as an alpha feature.

What is KRM?

KRM stands for Kubernetes Resource Model and is a definition of the API-focus design that defines the Kubernetes core. This is probably one of the most important reasons for the success of Kubernetes as it defines more or less a common API to request all kinds of operational needs in the form of resources. Every requirement such as load-balancing, auto-scaling, routing, placement of pods, and gateway is defined as a resource. Additionally, Kubernetes supports the extension of known resources in the form of CRDs (Custom Resource Definition). So in short, KRM defines the way we interact with Kubernetes via resources.

What are KRM Functions?

The KRM function specification defines how to manipulate a list of KRM resources. Simplified, it describes how a containerized application (the function) shall receive a list of resources and additional, optional, configurations. It also describes how it provides a new list of resources and optionally a result:

KRM function definition
KRM function definition

The function uses the resources and results as follows:

You can roughly group functions into Transformers and Generators on the one side and Validators on the other.

Let’s have a look at an example input for a function described in the specification

apiVersion: config.kubernetes.io/v1
kind: ResourceList
functionConfig:
  apiVersion: foo-corp.com/v1
  kind: FulfillmentCenter
  metadata:
    name: staging
  spec:
    address: "100 Main St."
items:
  - apiVersion: v1
    kind: Service
    metadata:
      name: wordpress
      labels:
        app: wordpress
      annotations:
        internal.config.kubernetes.io/index: "0"
        internal.config.kubernetes.io/path: "service.yaml"
    spec: # Example comment
      type: LoadBalancer
      selector:
        app: wordpress
        tier: frontend
      ports:
        - protocol: TCP
          port: 80

The input is read from stdin as a yaml structure named ResourceList with two parts.

functionConfig This describes the configuration for this function. We will see later how we build our configuration and function

items This is a list of items our function will work on.

After the function runs, an example output looks like this:

apiVersion: config.kubernetes.io/v1
kind: ResourceList
items:
  - apiVersion: v1
    kind: Service
    metadata:
      name: wordpress
      labels:
        app: wordpress
      annotations:
        internal.config.kubernetes.io/index: "0"
        internal.config.kubernetes.io/path: "service.yaml"
    spec: # Example comment
      type: LoadBalancer
      selector:
        app: wordpress
        tier: frontend
      ports:
        - protocol: TCP
          port: 80
results:
  - message: "Invalid type. Expected: integer, given: string"
    severity: error
    resourceRef:
      apiVersion: v1
      kind: Service
      name: wordpress
    field:
      path: spec.ports.0.port
    file:
      path: service.yaml

It has two parts, again.

items The new list of items is used for further processing. Deleted items will no longer be processed by the next step.

results A list of structural validation results. If the function has an unstructured result, e.g. it can not parse the input, it would return with a status code other than 0.

What is Kustomize in the KRM Function Context?

Besides the function definition, the specification defines an orchestrator for functions. In the end, this is a tool that would prepare the correct input for a function and will take the output to either stop further processing (in case of a failure) or to forward it to the next function in the chain.

orchestrator functionality
orchestrator functionality

Kustomize is an orchestrator from the perspective of a function but the specification is not limited to Kustomize. There are other implementations of an orchestrator available.

Build a KRM Function

Let’s create our function to see how it works with Kustomize. The goal is that we want to reduce the errors when creating typical resources defining an application. Our application consists of a deployment and a service. It exposes an endpoint. When we do this manually, we have to create a Deployment resource and a Service resource which are correctly configured so that the Service endpoints are mapped to the correct port in the Deployment.

Therefore, what we want to build is a generator that can generate these resources for us based on a simple configuration with a few fields. The configuration we are aiming for is as follows:

apiVersion: app.innoq.com/v1
kind: App
metadata:
  name: hello
spec:
  size: medium
  image: test:123
  port: 8080
app.yaml

As you can see, a KRM function configuration is also expressed in the form of a Kubernetes resource. This configuration is, however, only used on the client side. Therefore, Kubernetes does not need to know this structure, only Kustomize. The 3 values we want to make configurable are the port, the image, and the size. The size is a T-shirt size value to define how much CPU and memory the service gets. The values are small, medium, and large to make it easier for developers to define.

Whilst this is a very simple definition of an app that ignores many aspects such as auto-scaling or configurations and secrets, based on this setup we could extend the function later with everything we want to have.

So, first we have to create a new golang project containing the code for our function.

mkdir app-creator
cd app-creator
go mod init app-creator
touch main.go
touch templates.go

The next step is to create templates for the deployment and the service.

package main

const SERVICE_TEMPLATE = `
apiVersion: v1
kind: Service
metadata:
 name: {{ .Metadata.Name }}
spec:
 selector:
   app: {{ .Metadata.Name }}
 ports:
 - port: {{ .Spec.Port }}
   targetPort: {{ .Spec.Port }}`

const DEPLOYMENT_TEMPLATE = `
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Metadata.Name }}
spec:
  selector:
    matchLabels:
      app: {{ .Metadata.Name }}
  template:
    metadata:
      labels:
        app: {{ .Metadata.Name }}
    spec:
      containers:
      - name: {{ .Metadata.Name }}
        image: {{ .Spec.Image }}
        resources:
          limits:
            {{ if eq .Spec.Size "large" }}
            memory: "1024"
            cpu: "1"
            {{ else if eq .Spec.Size "medium" }}
            memory: "512Mi"
            cpu: "500m"
            {{ else }}
            memory: "256Mi"
            cpu: "250m"
            {{ end }}
        ports:
        - containerPort: {{ .Spec.Port }}`
templates.go

These are inline golang templates defined as variables we can use in the main function. We can already see the if-else block to map the T-shirt size to concrete CPU and memory limits.

Next, we need to install some golang packages which help to implement the function.

go get sigs.k8s.io/kustomize/kyaml/fn/framework \
       sigs.k8s.io/kustomize/kyaml/fn/framework/command \
       sigs.k8s.io/kustomize/kyaml/fn/framework/parser

go: added github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: added github.com/davecgh/go-spew v1.1.1
go: added github.com/go-errors/errors v1.4.2
go: added github.com/go-openapi/jsonpointer v0.19.6
go: added github.com/go-openapi/jsonreference v0.20.1
go: added github.com/go-openapi/swag v0.22.3
go: added github.com/golang/protobuf v1.5.2
go: added github.com/google/gnostic v0.5.7-v3refs
go: added github.com/google/go-cmp v0.5.5
go: added github.com/google/gofuzz v1.1.0
go: added github.com/inconshreveable/mousetrap v1.0.0
go: added github.com/josharian/intern v1.0.0
go: added github.com/mailru/easyjson v0.7.7
go: added github.com/mitchellh/mapstructure v1.4.1
go: added github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: added github.com/spf13/cobra v1.4.0
go: added github.com/spf13/pflag v1.0.5
go: added github.com/xlab/treeprint v1.1.0
go: added golang.org/x/sys v0.3.0
go: added google.golang.org/protobuf v1.28.0
go: added gopkg.in/yaml.v2 v2.4.0
go: added gopkg.in/yaml.v3 v3.0.1
go: added k8s.io/kube-openapi v0.0.0-20230109183929-3758b55a6596
go: added k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
go: added sigs.k8s.io/kustomize/kyaml v0.14.0
go: added sigs.k8s.io/yaml v1.3.0

Kustomize provides some helper structs and functions to simplify the implementation.

The next step is to implement the function.

package main

import (
	"os"

	"sigs.k8s.io/kustomize/kyaml/fn/framework"
	"sigs.k8s.io/kustomize/kyaml/fn/framework/command"
	"sigs.k8s.io/kustomize/kyaml/fn/framework/parser"
	"sigs.k8s.io/kustomize/kyaml/kio"
	"sigs.k8s.io/kustomize/kyaml/yaml"
)

type Metadata struct {
	Name string `yaml:"name"`
}

type AppSpec struct {
	Image string `yaml:"image"`
	Port  int32  `yaml:"port"`
	Size  string `yaml:"size,omitempty"`
}

type App struct {
	Metadata Metadata `yaml:"metadata"`
	Spec     AppSpec  `yaml:"spec"`
}

func main() {
	config := &App{}
	fn := framework.TemplateProcessor{
		TemplateData:       config,
		PostProcessFilters: []kio.Filter{kio.FilterFunc(filterAppFromResources)},
		ResourceTemplates: []framework.ResourceTemplate{{
			Templates: parser.TemplateStrings(DEPLOYMENT_TEMPLATE, SERVICE_TEMPLATE),
		}},
	}
	cmd := command.Build(fn, command.StandaloneDisabled, false)
	command.AddGenerateDockerfile(cmd)
	if err := cmd.Execute(); err != nil {
		os.Exit(1)
	}
}

func filterAppFromResources(items []*yaml.RNode) ([]*yaml.RNode, error) {
	var newNodes []*yaml.RNode
	for i := range items {
		meta, err := items[i].GetMeta()
		if err != nil {
			return nil, err
		}
		// remove resources with the kind App from the resource list
		if meta.Kind == "App" && meta.APIVersion == "app.innoq.com/v1" {
			continue
		}
		newNodes = append(newNodes, items[i])
	}
	items = newNodes
	return items, nil
}
main.go

The three structs map to our configuration above. Kustomize provides some helper functions to build functions for some typical use cases, in our case a template-based processor (see framework.TemplateProcessor). We provide some templates and the configuration. Then, the framework fills the configuration with the real values from the configuration before merging the configuration with the templates.

The filter function filterAppFromResources is just a precaution in case the function configuration is also present in the resource list. This function filters out the App resource before delivering the result.

There are other helper functions for other use cases. For more examples have a look at the examples in the package documentation.

With command.AddGenerateDockerfile, we can automatically generate a Dockerfile for our function.

go run *.go gen .

The corresponding Dockerfile looks like this:

FROM golang:1.19-alpine as builder
ENV CGO_ENABLED=0
WORKDIR /go/src/
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -ldflags '-w -s' -v -o /usr/local/bin/function ./

FROM alpine:latest
COPY --from=builder /usr/local/bin/function /usr/local/bin/function
ENTRYPOINT ["function"]
Dockerfile

Now, we can generate our containerized function

docker build -t app-creator:1.0.0 .

Testing the New Function

We can provide a test file that simulates the input as it would come from an orchestrator. The test file looks like this:

apiVersion: config.kubernetes.io/v1
kind: ResourceList
items:
- apiVersion: app.innoq.com/v1
  kind: App
  metadata:
    name: hello
  spec:
    size: medium
    image: test:123
    port: 8080
functionConfig:
  apiVersion: app.innoq.com/v1
  kind: App
  metadata:
    name: hello
  spec:
    size: medium
    image: test:123
    port: 8080
test.yaml

If we run our function against this, we would see the following:

cat test.yaml | docker run -i app-creator:1.0.0

apiVersion: config.kubernetes.io/v1
kind: ResourceList
items:
- apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: hello
  spec:
    selector:
      matchLabels:
        app: hello
    template:
      metadata:
        labels:
          app: hello
      spec:
        containers:
        - name: hello
          image: test:123
          resources:
            limits:
              memory: "512Mi"
              cpu: "500m"
          ports:
          - containerPort: 8080
- apiVersion: v1
  kind: Service
  metadata:
    name: hello
  spec:
    selector:
      app: hello
    ports:
    - port: 8080
      targetPort: 8080
functionConfig:
  apiVersion: app.innoq.com/v1
  kind: App
  metadata:
    name: hello
  spec:
    size: medium
    image: test:123
    port: 8080

In the output, we see the two new resources (Deployment, Service) based on our templates. We also see the limits set based on the T-shirt size as we defined them in the function. If we would change the size to large, other limits would be set. Additionally, the resource of kind App is removed from items (as said Kustomize would not provide this in the items list, but other orchestrators could).

Use your Function with Kustomize

So how can we run this function with Kustomize?

First, we create a new Kustomize project:

kustomize init

Then we define the configuration for the function:

apiVersion: app.innoq.com/v1
kind: App
metadata:
  name: hello
  annotations:
    config.kubernetes.io/function: |
      container:
        image: app-creator:1.0.0
spec:
  size: medium
  image: test:123
  port: 8080

This is the same configuration as above in the functionConfig but we have an additional annotation. The annotation is needed for Kustomize to know the link between this configuration and the function to be called.

Next, we have to link this configuration to our kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
- app.yaml
kustomization.yaml

You can add functions under the generators, transformers or validators block. Depending on the block, the function will work slightly differently. If we put our function configuration into the validators block, Kustomize will ignore the new items list and will only use the results. On the other hand, generator functions will always run before transformer functions.

That’s all. Now we can run Kustomize.

kustomize build --enable-alpha-plugins

apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: hello
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - image: test:123
        name: hello
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m
            memory: 512Mi

We have to run Kustomize with the --enable-alpha-plugins flag as it is still quite new and under active development. The good thing is the functions can be used in parallel with all other known Kustomize features like, for example, the name prefix. Let’s add a name prefix:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namePrefix: test-
generators:
- app.yaml
kustomization.yaml

Then we build the resources:

kustomize build --enable-alpha-plugins

apiVersion: v1
kind: Service
metadata:
  name: test-hello
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: hello
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-hello
spec:
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - image: test:123
        name: hello
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m
            memory: 512Mi

We see that the resources generated by our function are also correctly modified with the name prefix.

Other KRM Function Orchestrators

Kustomize mixes KRM functions with their traditional built-in features to provide further possibilities to adapt to our needs. Nevetheless, it is quite difficult to understand in which order the KRM functions and built-in features run and this can only be influenced to a certain extent. We are not completely free to run generator, transformer, or validator functions in any order.

Another tool supporting KRM functions is kpt. In fact, KRM functions in kpt are first-class citizens. Kpt provides a possibility to define a pipeline of KRM functions running on resources. For our example above, a corresponding kpt file could look like this.

apiVersion: kpt.dev/v1
kind: Kptfile
metadata:
  name: app
pipeline:
  mutators:
  - image: localhost:3000/app-creator:1.0.0
    configPath: app.yaml
Kptfile

Here, we see a pipeline definition with mutators. We have to rename the image as kpt expects a registry otherwise it uses gcr.io/kpt-fn as default.

docker tag app-creator:1.0.0 localhost:3000/app-creator:1.0.0

We also have to define a .krmignore file to ignore the Kptfile itself in the final render output. It works similarly to .gitignore or .dockerignore.

Kptfile
.krmignore

Now we can run the same function with kpt.

kpt fn render -o unwrap

Package "app-creator":
[RUNNING] "localhost:3000/app-creator:1.0.0"
[PASS] "localhost:3000/app-creator:1.0.0" in 300ms

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: test:123
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  selector:
    app: hello
  ports:
  - port: 8080
    targetPort: 8080

We see the same resources in the output here as with Kustomize.

This is just a glimpse into the kpt features but we can see that we are able to use the same KRM function with Kustomize and kpt. Both are following the KRM function specification to prepare the input for the KRM function and to run it.

Verdict and Alternatives

KRM functions are a great extension for Kustomize for client-side validation and the generation of new resources. We can combine all the features we already know from Kustomize and introduce best practices by providing a simplified interface to the user/developer via client-side CRDs (here in the form of the App resource).

Another tool solving similar issues is Crossplane as it provides so-called XRDs (Composite Resource Definitions) as an interface for developers.

Both fulfill the same need for a simplified interface.

On the Kustomize side, KRM functions can do more than just generate resources. They can modify existing resources and validate them, all on the client side before applying them to the cluster. It lacks on the other hand the possibility to replace the best practice in the background without telling the developers that there is a new version of the KRM function.

Crossplane is more of a meta-framework for handling all kinds of resources. It also includes CRDs to generate external resources like databases directly on the provider platform like AWS or GCP. It focuses more on the platform team’s needs and limits the debugging possibilities for developers.

So which tool fits best depends on your current needs. Additionally, these tools are not mutually exclusive, so you can mix them as needed. But always be aware that every additional tool increases the complexity and makes testing and debugging more difficult.

TAGS