User and Authentication Authorization in Kubernetes

This chapter briefly describes the principles related to kubernetes authentication, and ends with an experiment to illustrate the idea of implementing the kubernetes user system.

The main content is as follows.

Understanding the principles of various kubernetes authentication mechanisms
Understanding the concept of kubernetes users
Understanding kubernetes authentication webhook
Complete experiments with an idea of how to get other user systems into kubernetes

Kubernetes Authentication

As described in the Kubernetes apiserver for authentication section, all users accessing the Kubernetes API (via any client, client library, kubectl, etc.) go through three stages of Authentication, Authorization, and Admission control to complete the authorization of the “user”, as shown in the following diagram.

Request processing steps for Kubernetes API requests

In most tutorials, the work done on these three phases is roughly as follows.

Authentication phase refers to confirming that the user requesting access to the Kubernetes API is a legitimate user
Authorization phase will refer to whether or not the user has permission to operate on the resource
Admission control phase controls the requested resource, which is commonly known as a veto, even if the first two steps are completed.

Having learned here that the Kubernetes API actually does the work of a “human user” with a kubernetes service account; this leads to the important concept of what a “user” is in Kubernetes, and what a user is in authentication, which is also the center of this chapter.

The concept of “user” is given in the official Kubernetes manual, and the users present in a Kubernetes cluster include “normal user” and “service account”, but Kubernetes does not manage normal users in the same way as normal users, except that users who use a valid certificate signed by the cluster’s certificate CA are considered legitimate users.

Then for making a Kubernetes cluster have a real user system, it is possible to divide Kubernetes users into “external users” and “internal users” based on the concept given above. How do you understand external vs. internal users? In fact, users that are managed by Kubernetes, i.e., in the data model that defines users in kubernetes, are “internal users”, as in service account; conversely, users that are not hosted by Kubernetes are “external users”. This concept is also a better articulation of kubernetes users.

For external users, Kubernetes actually gives a variety of user concepts, such as

Users with kubernetes cluster credentials
Users with Kubernetes cluster token (static token specified by --token-auth-file)
users from external user systems, such as OpenID, LDAP, QQ connect, google identity platform, etc.

Example of granting access to a cluster to an external user

Scenario 1: Requesting k8s via certificate

In this scenario kubernetes will use cn as the user and ou as the group in the certificate, and the request is legitimate if the corresponding rolebinding/clusterrolebinding gives permissions to the user.

$ curl https://hostname:6443/api/v1/pods \
 --cert ./client.pem \
 --key ./client-key.pem \
 --cacert ./ca.pem

Next, we analyze the related source code.

Authenticating users is what apiserver does in the Authentication phase, and the corresponding code is under pkg/kubeapiserver/authenticator, the whole file is a series of authenticators, and the x.509 certificate is one of them.


// Create an authenticator that returns a request or a standard error
func (config Config) New() (authenticator.Request, *spec.SecurityDefinitions, error) {

...

 // X509 methods
    // You can see that this is where the x509 certificate is parsed into the user
 if config.ClientCAContentProvider != nil {
  certAuth := x509.NewDynamic(config.ClientCAContentProvider.VerifyOptions, x509.CommonNameUserConversion)
  authenticators = append(authenticators, certAuth)
 }
...

Next, see the implementation principle, the NewDynamic function is located in the code http://k8s.io/apiserver/pkg/authentication/request/x509/x509.go

As you can see by the code, it is an authentication function with the user to resolve to an Authenticator

// NewDynamic returns a request.Authenticator that verifies client certificates using the provided
// VerifyOptionFunc (which may be dynamic), and converts valid certificate chains into user.Info using the provided UserConversion
func NewDynamic(verifyOptionsFn VerifyOptionFunc, user UserConversion) *Authenticator {
 return &Authenticator{verifyOptionsFn, user}
}

The verification function is a method of CAContentProvider, and the x509 part is implemented as http://k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go.VerifyOptions.

You can see that the return is an x509.VerifyOptions and an authenticated status

// VerifyOptions provides verifyoptions compatible with authenticators
func (c *DynamicFileCAContent) VerifyOptions() (x509.VerifyOptions, bool) {
 uncastObj := c.caBundle.Load()
 if uncastObj == nil {
  return x509.VerifyOptions{}, false
 }

 return uncastObj.(*caBundleAndVerifier).verifyOptions, true
}

And the user’s get is located at http://k8s.io/apiserver/pkg/authentication/request/x509/x509.go; you can see that the user is exactly the CN of the certificate, while the group is the OU of the certificate

// CommonNameUserConversion builds user info from a certificate chain using the subject's CommonName
var CommonNameUserConversion = UserConversionFunc(func(chain []*x509.Certificate) (*authenticator.Response, bool, error) {
 if len(chain[0].Subject.CommonName) == 0 {
  return nil, false, nil
 }
 return &authenticator.Response{
  User: &user.DefaultInfo{
   Name:   chain[0].Subject.CommonName,
   Groups: chain[0].Subject.Organization,
  },
 }, true, nil
})

Since authorization is out of the scope of this chapter, it is directly ignored until the inbound phase. The inbound phase is implemented by RESTStorageProvider, where each Provider provides an Authenticator which contains the allowed requests that will be written to the repository by the corresponding REST client.

type RESTStorageProvider struct {
 Authenticator authenticator.Request
 APIAudiences  authenticator.Audiences
}
// RESTStorageProvider is a factory type for REST storage.
type RESTStorageProvider interface {
 GroupName() string
 NewRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (genericapiserver.APIGroupInfo, error)
}

Scenario 2: via token

In this scenario, when kube-apiserver has --enable-bootstrap-token-auth enabled, you can use Bootstrap Token for authentication, usually with the following command, adding Authorization: Bearer <token> to the request header.

1
2
3

$ curl https://hostname:6443/api/v1/pods \
  --cacert ${CACERT} \
  --header "Authorization: Bearer <token>" \

As you can see, in the code pkg/kubeapiserver/authenticator.New() when kube-apiserver specifies the parameter --token-auth-file=/etc/kubernetes/token.csv this authentication will be activated.

if len(config.TokenAuthFile) > 0 {
    tokenAuth, err := newAuthenticatorFromTokenFile(config.TokenAuthFile)
    if err != nil {
        return nil, nil, err
    }
    tokenAuthenticators = append(tokenAuthenticators, authenticator.WrapAudienceAgnosticToken(config.APIAudiences, tokenAuth))
}

At this point, open token.csv to see what the token looks like.

1
2

$ cat /etc/kubernetes/token.csv
12ba4f.d82a57a4433b2359,"system:bootstrapper",10001,"system:bootstrappers"

Back to the code here http://k8s.io/apiserver/pkg/authentication/token/tokenfile/tokenfile.go.NewCSV, where you can see that it is reading the --token-auth-file= parameter specified by the tokenfile, and then parses it to the user, record[1] as the username, and record[2] as the UID.

// NewCSV returns a TokenAuthenticator, populated from a CSV file.
// The CSV file must contain records in the format "token,username,useruid"
func NewCSV(path string) (*TokenAuthenticator, error) {
 file, err := os.Open(path)
 if err != nil {
  return nil, err
 }
 defer file.Close()

 recordNum := 0
 tokens := make(map[string]*user.DefaultInfo)
 reader := csv.NewReader(file)
 reader.FieldsPerRecord = -1
 for {
  record, err := reader.Read()
  if err == io.EOF {
   break
  }
  if err != nil {
   return nil, err
  }
  if len(record) < 3 {
   return nil, fmt.Errorf("token file '%s' must have at least 3 columns (token, user name, user uid), found %d", path, len(record))
  }

  recordNum++
  if record[0] == "" {
   klog.Warningf("empty token has been found in token file '%s', record number '%d'", path, recordNum)
   continue
  }

  obj := &user.DefaultInfo{
   Name: record[1],
   UID:  record[2],
  }
  if _, exist := tokens[record[0]]; exist {
   klog.Warningf("duplicate token has been found in token file '%s', record number '%d'", path, recordNum)
  }
  tokens[record[0]] = obj

  if len(record) >= 4 {
   obj.Groups = strings.Split(record[3], ",")
  }
 }

 return &TokenAuthenticator{
  tokens: tokens,
 }, nil
}

The format configured in the token file is a comma-separated set of strings.

type DefaultInfo struct {
 Name   string
 UID    string
 Groups []string
 Extra  map[string][]string
}

The most common way such a user is kubelet will usually authenticate to the control plane as such a user, such as the following configuration.

KUBELET_ARGS="--v=0 \
    --logtostderr=true \
    --config=/etc/kubernetes/kubelet-config.yaml \
    --kubeconfig=/etc/kubernetes/auth/kubelet.conf \
    --network-plugin=cni \
    --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 \
    --bootstrap-kubeconfig=/etc/kubernetes/auth/bootstrap.conf"

The contents of /etc/kubernetes/auth/bootstrap.conf, where the kube-apiserver configuration of --token-auth-file= username and the group must be system:bootstrappers, are used.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ......
    server: https://10.0.0.4:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:bootstrapper
  name: system:bootstrapper@kubernetes
current-context: system:bootstrapper@kubernetes
kind: Config
preferences: {}
users:
- name: system:bootstrapper

The problems that usually occur in binary deployments, such as the following errors.

`1`	`Unable to register node "hostname" with API server: nodes is forbidden: User "system:anonymous" cannot create resource "nodes" in API group "" at the cluster scope`

And the usual solution is to execute the following command, here is the user authorization when communicating kubelet with kube-apiserver, because the official condition given by kubernetes is that the user group must be system:bootstrappers.

`1`	`$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers`

The generated clusterrolebinding is as follows

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2022-08-14T22:26:51Z"
  managedFields:
  - apiVersion: rbac.authorization.k8s.io/v1
    fieldsType: FieldsV1
   ...
    time: "2022-08-14T22:26:51Z"
  name: kubelet-bootstrap
  resourceVersion: "158"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubelet-bootstrap
  uid: b4d70f4f-4ae0-468f-86b7-55e9351e4719
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-bootstrapper
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers

Such a user does not exist within kubernetes and can be considered an external user, but is present in the authentication mechanism and bound to the highest privilege, which can also be used for authentication for other accesses.

Scenario 3: serviceaccount

serviceaccount is usually created automatically for the API, but in the user, there are actually two directions for authentication, one is --service-account-key-file This parameter can specify more than one, specifying the corresponding certificate file public or private key for issuing the token of the sa.

First, a token is generated based on the specified public or private key file.

if len(config.ServiceAccountKeyFiles) > 0 {
    serviceAccountAuth, err := newLegacyServiceAccountAuthenticator(config.ServiceAccountKeyFiles, config.ServiceAccountLookup, config.APIAudiences, config.ServiceAccountTokenGetter)
    if err != nil {
        return nil, nil, err
    }
    tokenAuthenticators = append(tokenAuthenticators, serviceAccountAuth)
}
if len(config.ServiceAccountIssuers) > 0 {
    serviceAccountAuth, err := newServiceAccountAuthenticator(config.ServiceAccountIssuers, config.ServiceAccountKeyFiles, config.APIAudiences, config.ServiceAccountTokenGetter)
    if err != nil {
        return nil, nil, err
    }
    tokenAuthenticators = append(tokenAuthenticators, serviceAccountAuth)
}

For --service-account-key-file he generates the user as “kubernetes/serviceaccount” , while for --service-account-issuer just provides a title for the sa issuer to identify who it is instead of the uniform “kubernetes/serviceaccount” , here you can see from the code that the two are exactly the same, just with different titles.

// newLegacyServiceAccountAuthenticator returns an authenticator.Token or an error
func newLegacyServiceAccountAuthenticator(keyfiles []string, lookup bool, apiAudiences authenticator.Audiences, serviceAccountGetter serviceaccount.ServiceAccountTokenGetter) (authenticator.Token, error) {
 allPublicKeys := []interface{}{}
 for _, keyfile := range keyfiles {
  publicKeys, err := keyutil.PublicKeysFromFile(keyfile)
  if err != nil {
   return nil, err
  }
  allPublicKeys = append(allPublicKeys, publicKeys...)
 }
// 唯一的区别 这里使用了常量 serviceaccount.LegacyIssuer
 tokenAuthenticator := serviceaccount.JWTTokenAuthenticator([]string{serviceaccount.LegacyIssuer}, allPublicKeys, apiAudiences, serviceaccount.NewLegacyValidator(lookup, serviceAccountGetter))
 return tokenAuthenticator, nil
}

// newServiceAccountAuthenticator returns an authenticator.Token or an error
func newServiceAccountAuthenticator(issuers []string, keyfiles []string, apiAudiences authenticator.Audiences, serviceAccountGetter serviceaccount.ServiceAccountTokenGetter) (authenticator.Token, error) {
 allPublicKeys := []interface{}{}
 for _, keyfile := range keyfiles {
  publicKeys, err := keyutil.PublicKeysFromFile(keyfile)
  if err != nil {
   return nil, err
  }
  allPublicKeys = append(allPublicKeys, publicKeys...)
 }
// 唯一的区别 这里根据kube-apiserver提供的称号指定名称
 tokenAuthenticator := serviceaccount.JWTTokenAuthenticator(issuers, allPublicKeys, apiAudiences, serviceaccount.NewValidator(serviceAccountGetter))
 return tokenAuthenticator, nil
}

Finally, a token is issued based on the values of ServiceAccounts, Secrets, etc., which is the value obtained by the following command.

`1`	`$ kubectl get secret multus-token-v6bfg -n kube-system -o jsonpath={".data.token"}`

Scenario 4: openid

OpenID Connect is OAuth2 style and allows users to authorize a three-party site to access their information stored on another service provider without giving the username and password to the third-party site or sharing all the contents of their data, here is a logical diagram of kubernetes using OID authentication.

Kubernetes OID Authentication

Scenario 5: webhook

The webhook is one of the custom authentication options provided by kubernetes, and is primarily a hook for authenticating “bearer tokens” that will be created by the authentication service. Access control is triggered when a user accesses kubernetes, and when the authenticaion webhook is registered for a kubernetes cluster, a token is generated for you when authenticating using the method provided by the webhook.

As shown in the code pkg/kubeapiserver/authenticator.New() the newWebhookTokenAuthenticator is created with the provided config (--authentication-token-webhook-config-file) create a WebhookTokenAuthenticator

if len(config.WebhookTokenAuthnConfigFile) > 0 {
    webhookTokenAuth, err := newWebhookTokenAuthenticator(config)
    if err != nil {
        return nil, nil, err
    }

    tokenAuthenticators = append(tokenAuthenticators, webhookTokenAuth)
}

The following diagram shows how WebhookToken authentication works in kubernetes.

kubernetes WebhookToken Authentication Principle

Finally, the authHandler in the token loops through all the Handlers and then runs the AuthenticateToken to get the user’s information.

func (authHandler *unionAuthTokenHandler) AuthenticateToken(ctx context.Context, token string) (*authenticator.Response, bool, error) {
   var errlist []error
   for _, currAuthRequestHandler := range authHandler.Handlers {
      info, ok, err := currAuthRequestHandler.AuthenticateToken(ctx, token)
      if err != nil {
         if authHandler.FailOnError {
            return info, ok, err
         }
         errlist = append(errlist, err)
         continue
      }

      if ok {
         return info, ok, err
      }
   }

   return nil, false, utilerrors.NewAggregate(errlist)
}

The webhook plugin also implements this method AuthenticateToken , where the injected webhook is called via a POST request that carries a JSON formatted TokenReview object containing the token to be authenticated.

func (w *WebhookTokenAuthenticator) AuthenticateToken(ctx context.Context, token string) (*authenticator.Response, bool, error) {

    ....

  start := time.Now()
  result, statusCode, tokenReviewErr = w.tokenReview.Create(ctx, r, metav1.CreateOptions{})
  latency := time.Since(start)
...
}

The webhook token authentication service returns the user’s identity, which is the data structure mentioned in the token section above (webhook to decide whether to accept or reject the user).

type DefaultInfo struct {
 Name   string
 UID    string
 Groups []string
 Extra  map[string][]string
}

Scenario 6: Proxy authentication

Experiment: LDAP-based Authentication

The above exposition gives a general idea of how users are classified in the kubernetes authentication framework and what the authentication policies consist of. The purpose of the experiment is also to illustrate the result that using OIDC/webhook is a better way to protect and manage kubernetes clusters than other ways. First, in terms of security, assuming that the network environment is insecure, then any node node that misses the bootstrap token file means that it has the highest privileges in the cluster; second, in terms of management, the larger the team, the more people, it is impossible to provide a separate certificate or token for each user, knowing that the traditional tutorials talk about token in kubernetes The token is permanently valid in the kubernetes cluster unless you delete the secret/sa, and the plugins provided by Kubernetes solve these problems very well.

Experimental environment

A kubernetes cluster
An openldap service, it is recommended that it can be external to the cluster, because webhook does not have a caching mechanism like SSSD, and the cluster is not available, then authentication is not available, and when authentication is not available will cause the cluster to be unavailable, so that the scope of the impact of the accident can be controlled, also called the minimization radius.
Understanding of ldap related technologies and go ldap client

The experiment is roughly divided into the following steps.

Set up an HTTP server to return to the kubernetes Authenticaion service
Query ldap to see if the user is legitimate
- Query the user to see if they are legitimate
- Query if the group the user belongs to has privileges

Start of experiment

Initialize user data

First prepare the openldap initialization data, create three posixGroup groups with 5 users admin, admin1, admin11, searchUser, syncUser all with password 111, the group is associated with the user using memberUid

$ cat << EOF | ldapdelete -r  -H ldap://10.0.0.3 -D "cn=admin,dc=test,dc=com" -w 111
dn: dc=test,dc=com
objectClass: top
objectClass: organizationalUnit
objectClass: extensibleObject
description: US Organization
ou: people

dn: ou=tvb,dc=test,dc=com
objectClass: organizationalUnit
description: Television Broadcasts Limited
ou: tvb

dn: cn=admin,ou=tvb,dc=test,dc=com
objectClass: posixGroup
gidNumber: 10000
cn: admin

dn: cn=conf,ou=tvb,dc=test,dc=com
objectClass: posixGroup
gidNumber: 10001
cn: conf

dn: cn=dir,ou=tvb,dc=test,dc=com
objectClass: posixGroup
gidNumber: 10002
cn: dir

dn: uid=syncUser,ou=tvb,dc=test,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: pwdPolicy
pwdAttribute: userPassword
uid: syncUser
cn: syncUser
uidNumber: 10006
gidNumber: 10002
homeDirectory: /home/syncUser
loginShell: /bin/bash
sn: syncUser
givenName: syncUser
memberOf: cn=confGroup,ou=tvb,dc=test,dc=com

dn: uid=searchUser,ou=tvb,dc=test,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: pwdPolicy
pwdAttribute: userPassword
uid: searchUser
cn: searchUser
uidNumber: 10005
gidNumber: 10001
homeDirectory: /home/searchUser
loginShell: /bin/bash
sn: searchUser
givenName: searchUser
memberOf: cn=dirGroup,ou=tvb,dc=test,dc=com

dn: uid=admin1,ou=tvb,dc=test,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: pwdPolicy
pwdAttribute: userPassword
uid: admin1
sn: admin1
cn: admin
uidNumber: 10010
gidNumber: 10000
homeDirectory: /home/admin
loginShell: /bin/bash
givenName: admin
memberOf: cn=adminGroup,ou=tvb,dc=test,dc=com

dn: uid=admin11,ou=tvb,dc=test,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: pwdPolicy
sn: admin11
pwdAttribute: userPassword
uid: admin11
cn: admin11
uidNumber: 10011
gidNumber: 10000
homeDirectory: /home/admin
loginShell: /bin/bash
givenName: admin11
memberOf: cn=adminGroup,ou=tvb,dc=test,dc=com

dn: uid=admin,ou=tvb,dc=test,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: pwdPolicy
pwdAttribute: userPassword
uid: admin
cn: admin
uidNumber: 10009
gidNumber: 10000
homeDirectory: /home/admin
loginShell: /bin/bash
sn: admin
givenName: admin
memberOf: cn=adminGroup,ou=tvb,dc=test,dc=com
EOF

Next, you need to determine how to authenticate successful users, as mentioned above for kubernetes in the user format of v1.UserInfo, that is, to get the user, that is, the user group, assuming that the need to find the user for admin, then the query filter in openldap as follows.

`1`	`"(\|(&(objectClass=posixAccount)(uid=admin))(&(objectClass=posixGroup)(memberUid=admin)))"`

The above statement means to find the information of entries with objectClass=posixAccount and uid=admin or objectClass=posixGroup and memberUid=admin, where “|” and “&” are used to get these two results.

Write webhook to query user part

Here because the openldap configuration password format is not explicit, if you use “=” directly to verify the content is not queried, so directly use an additional login to verify whether the user is legitimate.

func ldapSearch(username, password string) (*v1.UserInfo, error) {
 ldapconn, err := ldap.DialURL(ldapURL)
 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }
 defer ldapconn.Close()

 // Authenticate as LDAP admin user
 err = ldapconn.Bind("uid=searchUser,ou=tvb,dc=test,dc=com", "111")
 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 // Execute LDAP Search request
 result, err := ldapconn.Search(ldap.NewSearchRequest(
  "ou=tvb,dc=test,dc=com",
  ldap.ScopeWholeSubtree,
  ldap.NeverDerefAliases,
  0,
  0,
  false,
  fmt.Sprintf("(&(objectClass=posixGroup)(memberUid=%s))", username), // Filter
  nil,
  nil,
 ))

 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 userResult, err := ldapconn.Search(ldap.NewSearchRequest(
  "ou=tvb,dc=test,dc=com",
  ldap.ScopeWholeSubtree,
  ldap.NeverDerefAliases,
  0,
  0,
  false,
  fmt.Sprintf("(&(objectClass=posixAccount)(uid=%s))", username), // Filter
  nil,
  nil,
 ))

 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 if len(result.Entries) == 0 {
  klog.V(3).Info("User does not exist")
  return nil, errors.New("User does not exist")
 } else {
  // Verify that the username and password are correct
  if err := ldapconn.Bind(userResult.Entries[0].DN, password); err != nil {
   e := fmt.Sprintf("Failed to auth. %s\n", err)
   klog.V(3).Info(e)
   return nil, errors.New(e)
  } else {
   klog.V(3).Info(fmt.Sprintf("User %s Authenticated successfuly!", username))
  }
  //The user format for splicing into kubernetes authentication
  user := new(v1.UserInfo)
  for _, v := range result.Entries {
   attrubute := v.GetAttributeValue("objectClass")
   if strings.Contains(attrubute, "posixGroup") {
    user.Groups = append(user.Groups, v.GetAttributeValue("cn"))
   }
  }

  u := userResult.Entries[0].GetAttributeValue("uid")
  user.UID = u
  user.Username = u
  return user, nil
 }
}

Writing the HTTP part

There are a few things to note here, namely the definition of the user or token to be authenticated, here the username@password format is used as the user identification, i.e. when logging into kubernetes you need to directly enter username@password as the login credentials.

The second part is the return value, which must be returned to Kubernetes in the api/authentication/v1.TokenReview format, with Status.Authenticated indicating the user authentication result, and if the user is legitimate, then set tokenReview. Authenticated = true and vice versa. If authentication is successful you also need Status.User which is in ldapSearch.

func serve(w http.ResponseWriter, r *http.Request) {
 b, err := ioutil.ReadAll(r.Body)
 if err != nil {
  httpError(w, err)
  return
 }
 klog.V(4).Info("Receiving: %s\n", string(b))

 var tokenReview v1.TokenReview
 err = json.Unmarshal(b, &tokenReview)
 if err != nil {
  klog.V(3).Info("Json convert err: ", err)
  httpError(w, err)
  return
 }

 // Extract username and password
 s := strings.SplitN(tokenReview.Spec.Token, "@", 2)
 if len(s) != 2 {
  klog.V(3).Info(fmt.Errorf("badly formatted token: %s", tokenReview.Spec.Token))
  httpError(w, fmt.Errorf("badly formatted token: %s", tokenReview.Spec.Token))
  return
 }
 username, password := s[0], s[1]
 // Query ldap to verify that the user is legitimate
 userInfo, err := ldapSearch(username, password)
 if err != nil {
  // The reason the log is not printed here is that it was printed in ldapSearch
  return
 }

 // Set the returned tokenReview
 if userInfo == nil {
  tokenReview.Status.Authenticated = false
 } else {
  tokenReview.Status.Authenticated = true
  tokenReview.Status.User = *userInfo
 }

 b, err = json.Marshal(tokenReview)
 if err != nil {
  klog.V(3).Info("Json convert err: ", err)
  httpError(w, err)
  return
 }
 w.Write(b)
 klog.V(3).Info("Returning: ", string(b))
}

func httpError(w http.ResponseWriter, err error) {
 err = fmt.Errorf("Error: %v", err)
 w.WriteHeader(http.StatusInternalServerError) // 500
 fmt.Fprintln(w, err)
 klog.V(4).Info("httpcode 500: ", err)
}

Here is the complete code.

package main

import (
 "encoding/json"
 "errors"
 "flag"
 "fmt"
 "io/ioutil"
 "net/http"
 "strings"

 "github.com/go-ldap/ldap"
 "k8s.io/api/authentication/v1"
 "k8s.io/klog/v2"
)

var ldapURL string

func main() {
 klog.InitFlags(nil)
 flag.Parse()
 http.HandleFunc("/authenticate", serve)
 klog.V(4).Info("Listening on port 443 waiting for requests...")
 klog.V(4).Info(http.ListenAndServe(":443", nil))
 ldapURL = "ldap://10.0.0.10:389"
 ldapSearch("admin", "1111")
}

func serve(w http.ResponseWriter, r *http.Request) {
 b, err := ioutil.ReadAll(r.Body)
 if err != nil {
  httpError(w, err)
  return
 }
 klog.V(4).Info("Receiving: %s\n", string(b))

 var tokenReview v1.TokenReview
 err = json.Unmarshal(b, &tokenReview)
 if err != nil {
  klog.V(3).Info("Json convert err: ", err)
  httpError(w, err)
  return
 }

 s := strings.SplitN(tokenReview.Spec.Token, "@", 2)
 if len(s) != 2 {
  klog.V(3).Info(fmt.Errorf("badly formatted token: %s", tokenReview.Spec.Token))
  httpError(w, fmt.Errorf("badly formatted token: %s", tokenReview.Spec.Token))
  return
 }
 username, password := s[0], s[1]
 userInfo, err := ldapSearch(username, password)
 if err != nil {
  return
 }

 if userInfo == nil {
  tokenReview.Status.Authenticated = false
 } else {
  tokenReview.Status.Authenticated = true
  tokenReview.Status.User = *userInfo
 }

 b, err = json.Marshal(tokenReview)
 if err != nil {
  klog.V(3).Info("Json convert err: ", err)
  httpError(w, err)
  return
 }
 w.Write(b)
 klog.V(3).Info("Returning: ", string(b))
}

func httpError(w http.ResponseWriter, err error) {
 err = fmt.Errorf("Error: %v", err)
 w.WriteHeader(http.StatusInternalServerError) // 500
 fmt.Fprintln(w, err)
 klog.V(4).Info("httpcode 500: ", err)
}

func ldapSearch(username, password string) (*v1.UserInfo, error) {

 ldapconn, err := ldap.DialURL(ldapURL)
 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }
 defer ldapconn.Close()

 // Authenticate as LDAP admin user
 err = ldapconn.Bind("cn=admin,dc=test,dc=com", "111")
 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 // Execute LDAP Search request
 result, err := ldapconn.Search(ldap.NewSearchRequest(
  "ou=tvb,dc=test,dc=com",
  ldap.ScopeWholeSubtree,
  ldap.NeverDerefAliases,
  0,
  0,
  false,
  fmt.Sprintf("(&(objectClass=posixGroup)(memberUid=%s))", username), // Filter
  nil,
  nil,
 ))

 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 userResult, err := ldapconn.Search(ldap.NewSearchRequest(
  "ou=tvb,dc=test,dc=com",
  ldap.ScopeWholeSubtree,
  ldap.NeverDerefAliases,
  0,
  0,
  false,
  fmt.Sprintf("(&(objectClass=posixAccount)(uid=%s))", username), // Filter
  nil,
  nil,
 ))

 if err != nil {
  klog.V(3).Info(err)
  return nil, err
 }

 if len(result.Entries) == 0 {
  klog.V(3).Info("User does not exist")
  return nil, errors.New("User does not exist")
 } else {
  if err := ldapconn.Bind(userResult.Entries[0].DN, password); err != nil {
   e := fmt.Sprintf("Failed to auth. %s\n", err)
   klog.V(3).Info(e)
   return nil, errors.New(e)
  } else {
   klog.V(3).Info(fmt.Sprintf("User %s Authenticated successfuly!", username))
  }
  user := new(v1.UserInfo)
  for _, v := range result.Entries {
   attrubute := v.GetAttributeValue("objectClass")
   if strings.Contains(attrubute, "posixGroup") {
    user.Groups = append(user.Groups, v.GetAttributeValue("cn"))
   }
  }

  u := userResult.Entries[0].GetAttributeValue("uid")
  user.UID = u
  user.Username = u
  return user, nil
 }
}

Deploying webhook

The official kubernetes manual states that the flag to enable webhook authentication is to specify the parameter --authentication-token-webhook-config-file in kube-apiserver . And this configuration file is a kubeconfig type file format.

The following is the configuration for deploying outside of a kubernetes cluster.

Create a configuration file /etc/kubernetes/auth/authentication-webhook.conf for use by kube-apiserver.

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: http://10.0.0.1:88/authenticate
  name: authenticator
users:
- name: webhook-authenticator
current-context: webhook-authenticator@authenticator
contexts:
- context:
    cluster: authenticator
    user: webhook-authenticator
  name: webhook-authenticator@authenticator

Modify the kube-apiserver parameter.

# Points to the corresponding profile
--authentication-token-webhook-config-file=/etc/kubernetes/auth/authentication-webhook.conf
# This is the token cache time, which means that the user does not need to request the webhook for authentication for a certain period of time after the authentication is passed when accessing the API.
--authentication-token-webhook-cache-ttl=30m
# Version specifies which version of the API to use, authentication.k8s.io/v1 or v1beta1
--authentication-token-webhook-version=v1

After starting the service, create a user in kubeconfig to verify the results.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 
    server: https://10.0.0.4:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: k8s-admin
  name: k8s-admin@kubernetes
current-context: k8s-admin@kubernetes
kind: Config
preferences: {}
users:
- name: admin
  user: 
    token: admin@111

Authentication results

When the password is incorrect, use the user admin to request the cluster.

1
2

$ kubectl get pods --user=admin
error: You must be logged in to the server (Unauthorized)

When the password is correct, use the user admin to request the cluster.

1
2

$ kubectl get pods --user=admin
Error from server (Forbidden): pods is forbidden: User "admin" cannot list resource "pods" in API group "" in the namespace "default"

You can see that the admin user is a non-existent user in the cluster, and prompted no permission to operate the corresponding resources, at this time the admin user and the cluster-admin in the cluster binding, test results:

1
2
3

$ kubectl create clusterrolebinding admin \
 --clusterrole=cluster-admin \
 --group=admin

At this point, try again to access the cluster using the admin user.

$ kubectl get pods --user=admin
NAME                      READY   STATUS    RESTARTS   AGE
netbox-85865d5556-hfg6v   1/1     Running   0          91d
netbox-85865d5556-vlgr4   1/1     Running   0          91d

Summary

The kubernetes authentication plugin provides the ability to inject an authentication system that perfectly solves the problem of users in kubernetes who do not exist in kubernetes and can be authenticated without having to prepare a large number of serviceaccounts or certificates for multiple users. The The first return value criterion is as follows: if the kubernetes cluster has Groups that are available on other user systems and clusterrolebinding or rolebinding is established, then all users in the group will have those privileges. The administrator only needs to maintain as many clusterrole and clusterrolebinding as there are groups in the company user system

type DefaultInfo struct {
 Name   string
 UID    string
 Groups []string
 Extra  map[string][]string
}

Table of Contents