As we all know, in the K8s permission management system, you can bind RoleBinding to ServiceAccount, User and Group to achieve permission assignment.

We often use ServiceAccount to restrict the permissions of a pod; for User and Group, there are no specific resources to correspond to them except for some special system groups, which is very unfriendly to the user management in traditional projects.

This article talks about how to unify user management in K8s clusters.

Preparation

First we need an Identity Provider to unify the management of users in K8s and provide OIDC protocol services, this article uses KeyCloak as the Identity Provider.

Configuration in KeyCloak

To implement user management, we need to use the concept of group in K8s to assign privileges to a group of users, which requires the use of the Claim concept in the OIDC protocol to implement the grouping of users in K8s.

Claim is the information carried in the ID Token, which refers to the range of information requested by the client, such as user name, email address, etc., and these can be extended to carry some information about the group to which the user belongs, etc.

So the first step is to extend the Claim in KeyCloak, as follows.

Configuration in KeyCloak

We added a “User Attribute” to the Client and added it to the ID Token; Multivalued must be set to ON to ensure that the value of the “groups” Claim is a String array, where each value represents a group to which the User A User can belong to multiple groups at the same time, and each value is separated by a comma.

The second step is to set the “groups” property for the user.

Configuration in KeyCloak

Once everything is set up, you can see the “groups” attribute in the ID Token of the user “admin”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
    "jti":"9259af9c-8a3d-45ff-94f6-6780f2a79580",
    "exp":1564739637,
    "nbf":0,
    "iat":1564739337,
    "iss":"https://172.16.105.1:8082/auth/realms/hdls",
    "aud":"kubernetes",
    "sub":"f846ddb1-4435-429f-9ce5-faba7a791d43",
    "typ":"ID",
    "azp":"kubernetes",
    "auth_time":0,
    "session_state":"37b1a2ca-1b3b-4c61-ae2c-f8c14818ca6e",
    "acr":"1",
    "email_verified":false,
    "groups":[
        "manager"
    ],
    "preferred_username":"admin"
}

Configuration of ApiServer

There are several configurable environment variables left in ApiServer to support the OIDC plugin, official link.

oidc-issuer-url : URL of OIDC Server, only accepts https protocol. oidc-client-id : The client_id configured in OIDC Server, which is unique. oidc-username-claim : Specify the claim used to identify the user name in the ID Token. oidc-username-prefix : Username prefix, “-” means no prefix. oidc-groups-claim : A claim in the ID Token that identifies the user’s group. The claim must be in the form of an array, so the user can belong to multiple groups. oidc-groups-prefix : group prefix. oidc-ca-file: the path where the CA certificate of Identity Provider is located.

The configuration parameters of this article are as follows.

1
2
3
4
5
6

    - --oidc-issuer-url=https://172.16.105.1:8082/auth/realms/hdls
    - --oidc-client-id=kubernetes
    - --oidc-username-claim=preferred_username
    - --oidc-username-prefix=-
    - --oidc-groups-claim=groups
    - --oidc-ca-file=/etc/kubernetes/pki/ca.crt

KubeConfig configuration

As a user, we need to access the API Server through the Client Application. kubectl is obviously the preferred Client, so that kubectl can access Kubernetes as the user we created, “admin”, and pass the authentication, and this requires a configuration of KubeConfig to complete the following processes.

  1. create a kubeconfig user: “admin”.
  2. Configure client-id, client credential, id-token, refresh-token, certficaite, etc. for “admin”.
  3. Create a user context for “admin”.
  4. Set it to the current context.

The following command generates the configuration with one click.

1
2
3
4
5
6
7
8

kubectl config set-credentials USER_NAME \
   --auth-provider=oidc \
   --auth-provider-arg=idp-issuer-url=( issuer url ) \
   --auth-provider-arg=client-id=( your client id ) \
   --auth-provider-arg=client-secret=( your client secret ) \
   --auth-provider-arg=refresh-token=( your refresh token ) \
   --auth-provider-arg=idp-certificate-authority=( path to your ca certificate ) \
   --auth-provider-arg=id-token=( your id_token )

How to get Token

There are various ways to generate ID Token and Refresh Token, the easiest way is to use curl to authenticate with Password Grant to get the desired ID Token and Refresh Token.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

$ curl -k 'https://172.16.105.1:8082/auth/realms/hdls/protocol/openid-connect/token' -d "client_id=kubernetes" -d "client_secret=40dc1fef...c3eeec6" -d "response_type=code token" -d "grant_type=password" -d "username=test" -d "password=dangerous" -d "scope=openid"
{
    "access_token":"eyJhbGciOiJSU...0CMPw",
    "expires_in":300,
    "refresh_expires_in":1800,
    "refresh_token":"eyJhbGciOiJ...W1VUA",
    "token_type":"bearer",
    "id_token":"eyJhbGc...z3TaGJGQ",
    "not-before-policy":0,
    "session_state":"2845e...92ff2",
    "scope":"openid profile email"
}

However, this requires manually generating a Token and filling it into KubeConfig each time, which is a pain. The good news is that there are already many tools in the community that can automatically write Token into KubeConfig for you that work very well, such as

  1. kubelogin
  2. k8s-auth-client
  3. k8s-keycloak-oidc-helper
  4. kuberos
  5. k8s-oidc-helper

User Management

After getting everything configured successfully, let’s look at assigning permissions to users. This takes into account K8s’ RBAC system.

RBAC

For users with group manager, we assign them the “cluster-admin” role which comes with the system, i.e. the administrator privileges of the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: keycloak-admin-group
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: Group
  name: manager
  apiGroup: rbac.authorization.k8s.io

We “add” the admin user to the “manager” group in keyCloak.

manager group in keyCloak

Then use this user to access APIServer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[root@172-16-105-1 ~]# kubelogin --username=admin --password=dangerous
You got a valid token until 2019-08-03 15:32:42 +0800 CST
[root@172-16-105-1 ~]#
[root@172-16-105-1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}
[root@172-16-105-1 ~]# kubectl get no
NAME           STATUS   ROLES    AGE   VERSION
172-16-105-1   Ready    master   54d   v1.14.1
172-16-105-2   Ready    <none>   54d   v1.14.1
[root@172-16-105-1 ~]# kubectl get po
No resources found.
[root@172-16-105-1 ~]# kubectl get all
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP        54d

You can see that the “admin” user has access to all resources.

Then, we create a new role called “hdls-role” for the user whose group is developer, and only give them access to the pod.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: hdls-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: hdls-rolebinding
roleRef:
  kind: ClusterRole
  name: hdls-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: developer
  apiGroup: rbac.authorization.k8s.io

“Add” the test user to the “developer” group in keyCloak.

developer group in keyCloak

Then use that user to access APIServer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

[root@172-16-105-1 ~]# kubelogin --username=test --password=dangerous
You got a valid token until 2019-08-03 15:40:21 +0800 CST
[root@172-16-105-1 ~]#
[root@172-16-105-1 ~]# kubectl get po
No resources found.
[root@172-16-105-1 ~]# kubectl get no
Error from server (Forbidden): nodes is forbidden: User "test" cannot list resource "nodes" in API group "" at the cluster scope
[root@172-16-105-1 ~]#
[root@172-16-105-1 ~]# kubectl get cs
Error from server (Forbidden): componentstatuses is forbidden: User "test" cannot list resource "componentstatuses" in API group "" at the cluster scope
[root@172-16-105-1 ~]#
[root@172-16-105-1 ~]# kubectl get all
Error from server (Forbidden): replicationcontrollers is forbidden: User "test" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): services is forbidden: User "test" cannot list resource "services" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "test" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "test" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "test" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "test" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "test" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "test" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "test" cannot list resource "cronjobs" in API group "batch" in the namespace "default"

The test user is restricted from accessing all resources except for pod information.

Summary

This article only introduced how to manage users in K8s through KeyCloak and kubectl, accordingly, if your own user center implements the OIDC protocol, and the client accesses APIServer as “bearer token” through ID Token, you can really connect the permission system of K8s with the superstructure.