Using viper to merge yaml configuration files

viper

As a small company, our infrastructure is not complete enough, the project manager informed us in the mid-autumn festival that our system will be on the second-to-last stage environment and production environment in the near future, so from the consideration of the deployment efficiency of the operation and maintenance staff, we urgently developed a one-click installation script generation tool, so that the operation and maintenance staff can use the tool combined with the actual target environment to generate a one-click installation script, the principle of this tool is very simple, as shown in the following diagram.

tool

As you can see from the image above, our tool is based on a template to customize the final configuration and installation script.

templates/conf directory for the service configuration.
templates/manifests directory holds the k8s yaml scripts for the service
custom/configure file stores customized configuration data for the service configuration under templates/conf
custom/manifests file stores customized configuration data for the k8s yaml under templates/manifests.
templates/install.sh is the installation script.

The two files in the custom directory that store the customized configuration are closely related to the target environment .

When it comes to templates, the first thing that comes to Gopher’s mind is the Go text/template technique, which uses template syntax to write template configuration files in the templates directory above. However, text/template requires us to identify all the variables that need to be customized in advance, which is a bit too much and not flexible enough.

So what other technical solutions can we use? I finally chose the yaml file merging (including overwriting and appending) scheme , which is schematically shown below.

yaml file merging

This example contains both override and (append) merge cases. Let’s first look at override.

The configuration in custom/manifests.yml overrides the configuration in templates/manifests/*.yaml

Take templates/manifests/a.yml as an example. The default value of metadata.name in this template is default, but the operations staff customizes (customizing) the custom/manifests.yml file according to the target environment. In this file, a.yml file name as the value of the key, and then the full path of the configuration items to be overridden configured into the file (here the full path is metadata.name).

1
2
3

a.yml:
  metadata:
    name: foo

The change to namespace name in the custom/manifests.yml file, foo, will override default in the original template, which will be reflected in the final xx_install/manifests/a.yml.

// a.yml
apiVersion: v1
kind: Namespace
metadata:
  name: foo

Append the configuration from custom/manifests.yml to templates/manifests/*.yaml configuration

For the configuration that is not in the original template file but added in custom, it will be appended to the final generated configuration file, take b.yml as an example. The contents of b.yml in the original template directory are as follows.

// templates/manifests/b.yml
log:
  type: file
  level: 0
  compress: true

Here there are only three sub-configurations under log: type, level and compress.

And the operation and maintenance staff in custom/manifests.yml for log added several other kinds of configuration, such as access_log, error_log, etc..

// custom/manifests.yml
b.yml:
  log:
    level: 1
    compress: false
    access_log: "access.log"
    error_log: "error.log"
    max_age: 3
    maxbackups: 7
    maxsize: 100

Thus, except for level and compress which will overwrite the values in the original template, the rest of the added configuration will be appended to the generated xx_install/manifests/b.yml will be reflected.

// b.yml
log:
  type: file
  level: 1
  compress: false
  access_log: "access.log"
  error_log: "error.log"
  max_age: 3
  maxbackups: 7
  maxsize: 100

OK! The solution is determined, so how to implement the merging of yaml files?

The Go community’s yaml package is best known as https://github.com/go-yaml/yaml. This package implements the YAML 1.2 specification, which facilitates marshal and unmarshal between Yaml and go struct.

However, the interface provided by the yaml package is rather rudimentary, and to merge yaml files, you need to do more extra work yourself, which we don’t have time for anymore. So is there a ready-to-use tool? The answer is yes, and it’s the Go community’s famous viper!

viper is an open source Go application configuration framework developed by Steve Francia, author of gohugo and former product manager of the Go language project team. viper not only supports command line arguments to pass in configuration, but also supports getting configuration from various types of configuration files, environment variables, remote configuration systems (etc.) and so on. In addition , viper also supports configuration file merge and write operations to the configuration file .

Can we just use viper’s Merge series of operations? The answer is no! Why? Because it has to do with our design above. We put all the environment-related configuration into the custom/manifests.yml file, which will cause the configuration data in custom/manifests.yml to appear in each of the final templates/xx.yml configuration files once we merge.

Then we’ll implement a set of merge (overwrite and append) operations ourselves!

Let’s start by looking at the main function that drives merge.

// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go

var (
    sourceDir string
    dstDir    string
)

func init() {
    flag.StringVar(&sourceDir, "s", "./", "template directory path")
    flag.StringVar(&dstDir, "d", "./k8s-install", "the target directory path in which the generated files are put")
}

func main() {
    var err error
    flag.Parse()

    // create target directory if not exist
    err = os.MkdirAll(dstDir+"/conf", 0775)
    if err != nil {
        fmt.Printf("create %s error: %s\n", dstDir+"/conf", err)
        return
    }

    err = os.MkdirAll(dstDir+"/manifests", 0775)
    if err != nil {
        fmt.Printf("create %s error: %s\n", dstDir+"/manifests", err)
        return
    }

    // override manifests files with same config item in custom/manifests.yml,
    // store the final result to the target directory
    err = mergeManifestsFiles()
    if err != nil {
        fmt.Printf("override and generate manifests files error: %s\n", err)
        return
    }
    fmt.Printf("override and generate manifests files ok\n")
}

We see that the main package uses the standard library flag package to create two command line arguments -s and -d, which represent the source path where templates/custom is stored and the target path where the generated files are stored, respectively. After entering the main function, we first create manifests and conf directories under the target path to store the relevant configuration files respectively (in this case, no files are generated under the conf directory).

Then the main function calls mergeManifestsFiles to merge the yml files in templates/manifests under the source path with custom/manifests.yml.

// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go

var (
    manifestFiles = []string{
        "a.yml",
        "b.yml",
    }
)

func mergeManifestsFiles() error {
    for _, file := range manifestFiles {
        // check whether the file exist
        srcFile := sourceDir + "/templates/manifests/" + file
        _, err := os.Stat(srcFile)
        if os.IsNotExist(err) {
            fmt.Printf("%s not exist, ignore it\n", srcFile)
            continue
        }

        err = mergeConfig("yml", sourceDir+"/templates/manifests", strings.TrimSuffix(file, ".yml"),
            sourceDir+"/custom", "manifests", dstDir+"/manifests/"+file)
        if err != nil {
            fmt.Println("mergeConfig error: ", err)
            return err
        }
        fmt.Printf("mergeConfig %s ok\n", file)

    }
    return nil
}

We see that mergeManifestsFiles iterates through the template files and calls mergeConfig, the function that actually does the yml file merge, once for each file.

// github.com/bigwhite/experiments/tree/master/yml-merge-using-viper/main.go

func mergeConfig(configType, srcPath, srcFile, overridePath, overrideFile, target string) error {
    v1 := viper.New()
    v1.SetConfigType(configType) // e.g. "yml"
    v1.AddConfigPath(srcPath)    // file directory
    v1.SetConfigName(srcFile)    // filename(without postfix)
    err := v1.ReadInConfig()
    if err != nil {
        return err
    }

    v2 := viper.New()
    v2.SetConfigType(configType)
    v2.AddConfigPath(overridePath)
    v2.SetConfigName(overrideFile)
    err = v2.ReadInConfig()
    if err != nil {
        return err
    }

    overrideKeys := v2.AllKeys()

    // override special keys
    prefixKey := srcFile + "." + configType + "." // e.g "a.yml."
    for _, key := range overrideKeys {
        if !strings.HasPrefix(key, prefixKey) {
            continue
        }

        stripKey := strings.TrimPrefix(key, prefixKey)
        val := v2.Get(key)
        v1.Set(stripKey, val)
    }

    // write the final result after overriding
    return v1.WriteConfigAs(target)
}

We see that the mergeConfig function creates two viper instances (viper.New()) for the file under templates/manifests and the manifests.yml file under custom and loads the configuration data for each of them. Then we iterate through the keys in manifests.yml under custom and set the values of the matching configuration items to the viper instance representing the file under templates/manifests, and finally we write the data of the merged viper instance to the target file.

Compile and run the generator tool.

$make
go build
$./generator
mergeConfig a.yml ok
mergeConfig b.yml ok
override and generate manifests files ok

With the default command line arguments, the files are generated in the k8s-install path, so let’s take a look at the generated files.

$cat k8s-install/manifests/a.yml
apiversion: v1
kind: Namespace
metadata:
    name: foo

$cat k8s-install/manifests/b.yml
log:
    access_log: access.log
    compress: false
    error_log: error.log
    level: 1
    max_age: 3
    maxbackups: 7
    maxsize: 100
    type: file

We see that the results of merge are consistent with our expectations (it does not matter if the order of the fields is inconsistent, this is related to the use of go map when storing key-value inside viper, the traversal order of go map is random).

But careful people may find a problem: that is, the original apiVersion in a.yml in the results file became lowercase apiversion, which will cause a.yml in the submission to k8s when the verification failed!

Why is this so? The official explanation given by viper is as follows.

Viper merges configuration from various sources, many of which are either case insensitive or uses different casing than the rest of the sources (eg. env vars). In order to provide the best experience when using multiple sources, the decision has been made to make all keys case insensitive.

There has been several attempts to implement case sensitivity, but unfortunately it’s not that trivial. We might take a stab at implementing it in Viper v2, but despite the initial noise, it does not seem to be requested that much.

Well, since it’s official that it might be supported in v2, but v2 is out of reach, let’s use the fork version of viper to solve the problem! The developer lnashier has forked a viper code and fixed the problem due to this case problem, although it is rather old (and may not be comprehensive), but it can meet our requirements! Let’s try replacing spf13/viper with lnashier/viper, and rebuild and execute generator.

$go mod tidy
go: finding module for package github.com/lnashier/viper
go: found github.com/lnashier/viper in github.com/lnashier/viper v0.0.0-20180730210402-cc7336125d12

$make clean
rm -fr generator k8s-install

$make
go build 

$./generator
mergeConfig a.yml ok
mergeConfig b.yml ok
override and generate manifests files ok

$cat k8s-install/manifests/a.yml
apiVersion: v1
kind: Namespace
metadata:
  name: foo

$cat k8s-install/manifests/b.yml
log:
  access_log: access.log
  compress: false
  error_log: error.log
  level: 1
  max_age: 3
  maxbackups: 7
  maxsize: 100
  type: file

We see that after replacing it with lnashier/viper, the key apiVersion in a.yml is not changed to lowercase anymore.

The tool basically works now. But is this tool problem-free? Unfortunately not! The generator generates the wrong file when it faces the following two forms of configuration files.

//c.yml

apiVersion: v1
data:
  .dockerconfigjson: xxxxyyyyyzzz==
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: mysecret
  namespace: foo

and.

//d.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
  namespace: foo
data:
  my-nginx.conf: |
    server {
          listen 80;
          client_body_timeout 60000;
          client_max_body_size 1024m;
          send_timeout 60000;
          proxy_headers_hash_bucket_size 1024;
          proxy_headers_hash_max_size 4096;
          proxy_read_timeout 60000;
          location /dashboard {
             proxy_pass http://localhost:8081;
          }
    }

These two problems are a bit tricky, and lnashier/viper can’t solve them. I can only fork lnashier/viper to bigwhite/viper to solve this problem by myself, and the configuration form like d.yml is very specific and not universal, so bigwhite/viper is not universal, so I won’t go into details here, you can read the code (commit diff) by yourself if you are interested to see how to solve the above problem.

The code covered in this article can be downloaded from here.

Some other stuff.

kustomize

kustomize is an official k8s tool that allows you to customize YAML files for multiple purposes based on k8s resource template YAML files and in combination with kustomization.yaml, the original YAML will not be changed in any way.

However, it targets only k8s-related yaml files and may not be able to do anything for our business service configuration.
CUE

CUE is a powerful declarative configuration language that has gained popularity in the past two years, created by former Go core team member Marcel van Lohuizen, who co-founded the Borg Configuration Language (BCL) - the language used to deploy all applications at Google. cUE is the result of Google’s years of experience writing configuration languages designed to improve the developer experience while avoiding some of the pitfalls. It is a superset of JSON with additional features, and is heavily used by Solomon Hykes, the father of Docker, for his new startup dagger, and by kubevela, the enterprise cloud native application management platform promoted by Ali.

An in-depth study on how to use CUE instead of the scheme I described above is yet to follow.

Reference

https://tonybai.com/2022/09/20/use-viper-to-do-merge-of-yml-configuration-files/