When using helmfile, we first need to understand the use of helm and how to develop a helm chart. helm is a package management tool for kubernetes. In real life scenarios we need to deploy multiple charts at the same time, distinguish between different deployment environments, version control, etc. For this purpose, the helmfile tool can be used. helmfile helps users to manage and maintain multiple helm charts by means of helmfile files, which can be used to differentiate environments and implement version control.
Scenario Description
In our public cloud scenario or privatization scenario, the same product may involve multiple sets of environment configurations, for example, each set of environment deployment depends on different environment, the address and account password of the database, message queue middleware and other instances used are different. Therefore, for different environments we need to maintain the deployment files and secret key files of the development environment, test environment, pre-production environment, production environment and even multiple environments, and each small change will involve the modification of the configuration of multiple environments, which adds a great burden to the operation and maintenance staff, and how to maintain the uniformity of the configuration of multiple environments, which also greatly tests the meticulousness of the operation and maintenance staff and greatly increases the complexity. The storage of account passwords for the database middleware instances involved also adds a huge security risk to the O&M process. Based on the above, we can transform the service files of the business deployment into a helm chart, distinguishing between multiple environments and version control, and we use helmfile to unify the deployment management. We can use helm secrets to encrypt and decrypt the account passwords involved in the instance, as well as to ensure the security of operations and maintenance, thus greatly reducing the complexity of operations and maintenance. The use of helm secrets is described in detail in other articles.
Installation
helmfile offers a variety of installation options, see https://github.com/helmfile/helmfile/releases helmfile also supports running in containers and can be easily integrated into CICD’s pipeline.
1
2
3
4
5
|
# helm 2
$ docker run --rm --net=host -v "${HOME}/.kube:/root/.kube" -v "${HOME}/.helm:/root/.helm" -v "${PWD}:/wd" --workdir /wd quay.io/roboll/helmfile:v0.135.0 helmfile sync
# helm 3
$ docker run --rm --net=host -v "${HOME}/.kube:/root/.kube" -v "${HOME}/.config/helm:/root/.config/helm" -v "${PWD}:/wd" --workdir /wd quay.io/roboll/helmfile:helm3-v0.135.0 helmfile sync
|
Introduction to helmfile.yaml
helmfile.yaml is the core file of helmfile and is used to declare all the configuration. It will be briefly described below, and the official documentation can be consulted for specific instructions: https://github.com/roboll/helmfile#configuration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
|
# Declare repo configuration
repositories:
- name: <repo-name>
# url: repo url
# Basic configuration or tls authentication can be set
# certFile: certificate file
# keyFile: key file
# username: username
# password: password
# path to helm binary file
helmBinary: path/to/helm3
# These are the same as `helm SUBCOMMAND` and can be used to declare some, default, configurations
helmDefaults:
tillerNamespace: tiller-namespace #dedicated default key for tiller-namespace
tillerless: false #dedicated default key for tillerless
kubeContext: kube-context #dedicated default key for kube-context (--kube-context)
cleanupOnFail: false #dedicated default key for helm flag --cleanup-on-fail
# additional and global args passed to helm (default "")
args:
- "--set k=v"
# verify the chart before upgrading (only works with packaged charts not directories) (default false)
verify: true
# wait for k8s resources via --wait. (default false)
wait: true
# time in seconds to wait for any individual Kubernetes operation (like Jobs for hooks, and waits on pod/pvc/svc/deployment readiness) (default 300)
timeout: 600
# performs pods restart for the resource if applicable (default false)
recreatePods: true
# forces resource update through delete/recreate if needed (default false)
force: false
# when using helm 3.2+, automatically create release namespaces if they do not exist (default true)
createNamespace: true
...
# Set the same label for all releases in the helmfile, which can be used to mark all releases with the same version
commonLabels:
hello: world
# Set release configuration (multiple releases supported)
releases:
# Remote chart example (chart already uploaded to remote repository)
- name: vault # name of this release
namespace: vault # target namespace
createNamespace: true # helm 3.2+ automatically create release namespace (default true)
labels: # Arbitrary key value pairs for filtering releases
foo: bar
chart: roboll/vault-secret-manager # the chart being installed to create this release, referenced by `repository/chart` syntax
version: ~1.24.1 # the semver of the chart. range constraint is supported
condition: vault.enabled # The values lookup key for filtering releases. Corresponds to the boolean value of `vault.enabled`, where `vault` is an arbitrary value
missingFileHandler: Warn # set to either "Error" or "Warn". "Error" instructs helmfile to fail when unable to find a values or secrets file. When "Warn", it prints the file and continues.
# Values files used for rendering the chart
values:
# Value files passed via --values
- vault.yaml
# Inline values, passed via a temporary values file and --values, so that it doesn't suffer from type issues like --set
- address: https://vault.example.com
# Go template available in inline values and values files.
- image:
# The end result is more or less YAML. So do `quote` to prevent number-like strings from accidentally parsed into numbers!
# See https://github.com/roboll/helmfile/issues/608
tag: {{ requiredEnv "IMAGE_TAG" | quote }}
# Otherwise:
# tag: "{{ requiredEnv "IMAGE_TAG" }}"
# tag: !!string {{ requiredEnv "IMAGE_TAG" }}
db:
username: {{ requiredEnv "DB_USERNAME" }}
# value taken from environment variable. Quotes are necessary. Will throw an error if the environment variable is not set. $DB_PASSWORD needs to be set in the calling environment ex: export DB_PASSWORD='password1'
password: {{ requiredEnv "DB_PASSWORD" }}
proxy:
# Interpolate environment variable with a fixed string
domain: {{ requiredEnv "PLATFORM_ID" }}.my-domain.com
scheme: {{ env "SCHEME" | default "https" }}
# Use `values` whenever possible!
# `set` translates to helm's `--set key=val`, that is known to suffer from type issues like https://github.com/roboll/helmfile/issues/608
set:
# single value loaded from a local file, translates to --set-file foo.config=path/to/file
- name: foo.config
file: path/to/file
# set a single array value in an array, translates to --set bar[0]={1,2}
- name: bar[0]
values:
- 1
- 2
# set a templated value
- name: namespace
value: {{ .Namespace }}
# will attempt to decrypt it using helm-secrets plugin
# Example of a local chart (chart is saved locally)
- name: grafana # name of this release
namespace: another # target namespace
chart: ../my-charts/grafana # the chart being installed to create this release, referenced by relative path to local helmfile
values:
- "../../my-values/grafana/values.yaml" # Values file (relative path to manifest)
- ./values/{{ requiredEnv "PLATFORM_ENV" }}/config.yaml # Values file taken from path with environment variable. $PLATFORM_ENV must be set in the calling environment.
wait: true
# Additional helmfiles can be nested to support pulling helmfiles from local and remote locations
helmfiles:
- path: path/to/subhelmfile.yaml
# The label selector can filter the release to be overridden
selectors:
- name=prometheus
# Override value
values:
# Overwriting with files
- additional.values.yaml
# Override separate key
- key1: val1
- # Remote pull configuration
path: git::https://github.com/cloudposse/helmfiles.git@releases/kiam.yaml?ref=0.40.0
# If it points to a path that does not exist, an alert error is printed
missingFileHandler: Error
# Multi-environmental management
environments:
# When `--environment NAME` is not set, use default
default:
values:
# The content can be a file path or key:value
- environments/default/values.yaml
- myChartVer: 1.0.0-dev
# "production" environment, when `helmfile --environment production sync` is set
production:
values:
- environment/production/values.yaml
- myChartVer: 1.0.0
# disable vault release processing
- vault:
enabled: false
## `secrets.yaml` is decrypted by `helm-secrets` and available via `{{ .Environment.Values.KEY }}`
secrets:
- environment/production/secrets.yaml
# When `environments.NAME.values` is not found, it can be set to "Error", "Warn", "Info", "Debug", the default is "Error"
missingFileHandler: Error
# Hierarchical management, where all files can be merged in the following order: environments.yaml < - defaults.yaml < - templates.yaml < - helmfile.yaml
bases:
- environments.yaml
- defaults.yaml
- templates.yaml
# API functions
apiVersions:
- example/v1
|
helmfile debugging
Here, after compiling the relevant helmfile, we can use the following command to debug.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
# View the catalogue structure
$ ls
README.org environments helm helmfile helmfile.yaml releases
# View helmfile.yaml
$ cat helmfile.yaml
environments:
# If no environment is specified, the default test environment is used by default
default:
values:
- environments/test/config.yaml
- environments/test/versions.yaml
- environments/test//namespaces.yaml
secrets:
- environments/test/secrets.yaml
test:
values:
- environments/test/config.yaml
- environments/test/versions.yaml
- environments/test/namespaces.yaml
secrets:
- environments/test/secrets.yaml
helmDefaults:
createNamespace: true
releases:
- name: password-secrets
kubeContext: {{ .Values.kubeContext.service }}
namespace: {{ .Values.namespaces.service }}
chart: helm/charts/secrets
values:
- releases/secrets.yaml.gotmpl
labels:
app: secrets
- name: web
kubeContext: {{ .Values.kubeContext.business }}
namespace: {{ .Values.namespaces.business }}
chart: helm/charts/web
values:
- releases/web.yaml.gotmpl
labels:
app: web
# helmfile debugging
$ helmfile -e test template
|
Install chart
helmfile updates or removes a chart
Here you can specify the label to be updated or deleted by --selector.
1
2
3
4
|
# Updating web service
helmfile -e test --selector app=web sync
# Delete web service
helmfile -e test --selector app=web delete
|
View changes
1
2
3
4
|
# View the changes in the document.
helmfile -e test --selector app=web diff
# View only the parts of the document that have changed.
helmfile -e test --selector app=web diff --context 4
|