tl;dr If all went well, you should see three nginx pods running on your cluster

But, what is going on under the hood

The Beginning

A really good thing about K8s is that it hides its complexities by providing simple abstractions and user-friendly APIs. I think that to fully understand the values it offers, it's good to know a bit about its internals. In this post, I'll trace the path a K8s deployment object takes from the time we enter kubectl create deploy nginx --image=nginx --replicas=3 to when we finally see the three running nginx pods.

k8s API

The Kubernetes API is an HTTP REST API. This API is the real Kubernetes user interface. Kubernetes is fully controlled through this API. This means that every Kubernetes operation is exposed as an API endpoint and can be executed by an HTTP request to this endpoint.

Kubernetes is a fully resource-centered system. That means, Kubernetes maintains an internal state of resources, and all Kubernetes operations are CRUD operations on these resources. Kubernetes is controlled by manipulating these resources (and Kubernetes figures out what to do based on the current state of resources).

The kubeconfig file

K8s api-server talks HTTP(S) and exposes some REST endpoints. The kubeconfig file in your directory contains information about your clusters' endpoints, users' auth info, and a map of clusters and users. Almost all k8s related components/clients make use of this file to connect to the cluster. Default kubeconfig file resides in users home dir at ~/.kube/config

The kubectl

We entered

kubectl create deploy nginx --image=nginx --replicas=3

Now, what?

While the server-side validation is favored and is de-facto. The first thing that kubectl will do is perform some trivial client-side validations. This ensures that the requests that should always fail (like invalid spec key, wrong indentation, etc) will fail fast without unnecessary server load.

At this point kubectl will serialize the request and start forming the HTTP request it will send to kube-apiserver.

client-auth

User creds are almost always stored in the kubeconfig file. To locate the config file, kubectl does the following

• If a --kubeconfig flag is provided, use that

• If the KUBECONFIG env var is set, use that

• otherwise, look at the default (~/.kube/config) location

After parsing the file, it then determines the current context to use, the current cluster to point to, and any auth information associated with the current user.

API groups and version negotiation

K8s uses a versioned API that is categorized into "API groups". An API group is meant to categorize similar resources so that they're easier to reason about. It also provides a better alternative to a singular monolithic API. The API group of a Deployment is named apps, and its most recent version is v1

After kubectl generates the object, it starts to find the appropriate API group and version for it. This discovery stage is called version negotiation and involves kubectl scanning the /apis path on the remote API to retrieve all possible API groups. Since kube-apiserver exposes its schema document (in OpenAPI format) at this path, it's easy for clients to perform their own discovery. kubectl caches the discovery result at ~/.kube/cache location.

After the API Group and version discovery, kubectl will validate if the resource we are going to create is actually supported by the server. And then it'll POST to the appropriate API. Generally the API path will be /apis/<api-group>/<api-version>/namespaces/<namespace>/<kind>

So, in our case request will be sent to /apis/apps/v1/namespaces/default/deployments with POST data being the object data. You can check this with kubectl's -v flag

kubectl create deploy nginx --image=nginx --replicas=2 -v=8
I0203 12:34:48.915446   25098 loader.go:379] Config loaded from file:  /Users/xxxxxxx/.kube/config
I0203 12:34:48.916475   25098 cert_rotation.go:137] Starting client certificate rotation controller
I0203 12:34:48.918611   25098 request.go:1107] Request Body: {"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"nginx","creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"nginx"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx"}},"spec":{"containers":[{"name":"nginx","image":"nginx","resources":{}}]}},"strategy":{}},"status":{}}
I0203 12:34:48.918676   25098 round_trippers.go:422] POST https://192.168.64.3:8443/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-create
I0203 12:34:48.918682   25098 round_trippers.go:429] Request Headers:
I0203 12:34:48.918686   25098 round_trippers.go:433]     Accept: application/json, */*
I0203 12:34:48.918690   25098 round_trippers.go:433]     Content-Type: application/json
I0203 12:34:48.918694   25098 round_trippers.go:433]     User-Agent: kubectl/v1.20.6 (darwin/amd64) kubernetes/8a62859
I0203 12:34:48.933429   25098 round_trippers.go:448] Response Status: 201 Created in 14 milliseconds
I0203 12:34:48.933466   25098 round_trippers.go:451] Response Headers:
I0203 12:34:48.933470   25098 round_trippers.go:454]     Content-Type: application/json
I0203 12:34:48.933474   25098 round_trippers.go:454]     X-Kubernetes-Pf-Flowschema-Uid: c504618b-a382-437b-9bef-25ad5118629f
I0203 12:34:48.933479   25098 round_trippers.go:454]     X-Kubernetes-Pf-Prioritylevel-Uid: 3d1e0523-1c81-495b-885f-7056ed97da3d
I0203 12:34:48.933482   25098 round_trippers.go:454]     Content-Length: 1645
I0203 12:34:48.933485   25098 round_trippers.go:454]     Date: Thu, 03 Feb 2022 03:34:49 GMT
I0203 12:34:48.933488   25098 round_trippers.go:454]     Cache-Control: no-cache, private
I0203 12:34:48.933521   25098 request.go:1107] Response Body: {"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"nginx","namespace":"default","uid":"ecddc09b-f322-4a04-8332-aa00484867c0","resourceVersion":"641059","generation":1,"creationTimestamp":"2022-02-03T03:34:49Z","labels":{"app":"nginx"},"managedFields":[{"manager":"kubectl-create","operation":"Update","apiVersion":"apps/v1","time":"2022-02-03T03:34:49Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{},"f:strategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}},"f:template":{"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"nginx\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}}}}]},"spec":{ [truncated 621 chars]
deployment.apps/nginx created

The API-Server

So our request has been sent, woaah! What now? We have already talked about the kube-apiserver briefly but this is where it takes the central stage.

Authentication

All K8s clusters have two categories of users: service accounts (managed by K8s) and normal users. For normal users, when the apiserver starts up, it looks at all the CLI flags the user-provided and assembles a list of suitable authenticators. For example, if a --client-ca-file has been passed in, it appends the x509 authenticator; if it sees --token-auth-file provided, it appends the token authenticator to the list. Additionally webhook-token-authentication, authenticating-proxy can be used for custom authenticator plugins.

As HTTP requests are made to the API server, plugins attempt to associate the following attributes with the request:

• Username: a string that identifies the end-user. Common values might be kube-admin or jane@example.com.

• UID: a string that identifies the end-user and attempts to be more consistent and unique than username.

• Groups: a set of strings, each of which indicates the user's membership in a named logical collection of users. Common values might be system:masters or devops-team.

All values are opaque to the authentication system and only hold significance when interpreted by an authorizer.

You can enable multiple authentication methods at once. You should usually use at least two methods:

• service account tokens for service accounts

• at least one other method for normal user authentication.

When multiple authenticator modules are enabled, the first module to successfully authenticate the request short-circuits evaluation. The API server does not guarantee the order authenticators run in.

Let's talk about a couple of authenticators next

X509 Client Certs

Client certificate authentication is enabled by passing the --client-ca-file=SOMEFILE option to API server. If a client certificate is presented and verified against above CA, the common name(CN) of the subject is used as the user name for the request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships using the certificate's organization(O) fields.

For example, following request would create a CSR for the username "nakam", belonging to two groups, "devops" and "devs".

openssl req -new -key key.pem -out csr.pem -subj "/CN=nakam/O=devops/O=devs"

Static Token File

The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option on the command line. Currently, tokens last indefinitely, and the token list cannot be changed without restarting the API server.

The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names.

When authenticating from an http client, the API server expects an Authorization header with a value of Bearer <TOKEN>.

Authorization: Bearer 31ada4fd-adec-460c-809a-9e56ceb75269

Service Account Tokens

A service account is an automatically enabled authenticator that uses signed bearer tokens to verify requests. The plugin takes two optional flags:

• --service-account-key-file A file containing a PEM encoded key for signing bearer tokens. If unspecified, the API server's TLS private key will be used.

• --service-account-lookup If enabled, tokens which are deleted from the API will be revoked. Service accounts are usually created automatically by the API server and associated with pods running in the cluster through the ServiceAccount Admission Controller. Bearer tokens are mounted into pods at well-known locations but are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API.

Service accounts authenticate with the username system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT), and are assigned to the groups system:serviceaccounts and system:serviceaccounts:(NAMESPACE).

➜ kubectl get sa default -oyaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2021-12-10T11:19:21Z"
  name: default
  namespace: default
secrets:
- name: default-token-w8dkr

➜ kubectl get secret default-token-w8dkr -oyaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwdGFXNXAKYTNWaVpVTkJNQjRYRFRJeE1USXdOakF4TURreU4xb1hEVE14TVRJd05UQXhNRGt5TjFvd0ZURVRNQkVHQTFVRQpBeE1LYldsdWFXdDFZbVZEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTTlOCjQzbnV0dmJucXdkVlNEOWFsSmJJZWdacHdoOERGT2JDL2RUYjY0eGpzUHAxQklpenFJTnVvNnJLaXF3RDZDdkgKT1psMzhnNWNjcmtCVEw1OUhDa0JBdTh4U1I5VTczdG0weWZHeDdJQWp6a09BcDQ2Z1dGSjBodDJTOE85ZU9yYgo3aGIvTGUvVUl5NVErWm82ellNWUE3K0NmQUNxZm0rd0syVVJYUzVndjkrc1k0cFJGYWp4NFJWV05NTnpKdDExCllHdFQvYWV3WFVhRk9iSWxraUlEWHVNbVFLVGRyVlAvZktrNDhEM21JOGFiUlBYU2RvRU1PNnR0Mkl3Y01XaXIKZVR0OHdWdlVUSkdsa0dRL2ZTQm5EUFpzMElOREZoenl6MkJvK1pLVU1sK1FvR1VtdmFpTmJyUjBQRWlYOGNxQQpPQmZMcHZHc2U4VTZWcXVIeHUwQ0F3RUFBYU5oTUY4d0RnWURWUjBQQVFIL0JBUURBZ0trTUIwR0ExVWRKUVFXCk1CUUdDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVcKQkJUVU80U0dla243Uld6WTB1VDdMc2NBaGpoWjd6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFHeExoRjFDYwplQXp2ZG9kQ1RKKzdJNk91M2Ira1h4Nkx6Q2VtMGJkMXRIQUhvWTdiaWhxaHRlalQ4STFheDNNNm55RG1zdk00Ckd0MUtQT3BJRm01ZHU1emt3RXZrZE4vbFFJWTBoK1h4TERySUc0S21aYmZyQWRtWVlZWHo5M3dsbitVdm9LZkYKeEFuTU8xY1F4OWlHRTVFeGhScm9xb0J1dFJIemVSMFczTjB6VFczaGpVZEp3RTA2andEYlV0QUNHS0FkQURtRgpia1NRYXFud0pIY3lNckhnL2JOTTN0K3k4S2dQL1Rtd1o1aXVPVWJjT1g3WjBuSXVmWlU2T3hLSEpINkJTR2w0CnJUMzFsZXYwc3RSSklRTVcwZXYrTENOZXpYZ2VlUDZNOTc1Z0h4QldSZEY1K25yVzczNmVUbGVWcWI0S2JWcGEKRWltMmFuT0N5MnhxOEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  namespace: ZGVmYXVsdA==
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrOUJhRWd0V0hoeVZESktXSHAxVlRKbmMwTTJabk01Vmt0eFRrdHZOalJwZFV0cVMwdDZiekpvYVRnaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dGR6aGthM0lpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJamc1T1dZd1lUaG1MVFUxTXpjdE5EbG1OeTA1TW1KaExUYzJZMlZpTmpFM09ETmtOaUlzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuR0cyaWFwTV9IQlNOQTE2Vm5fNTN5NFRER1o0NUltUHBYYzVfUlV3OEJKbUxpZ3VSN004X1NhM1A3YUZnQnIzT19uZk1rWDZqcFkxNXBzRlpuVU1XS0JJZy1KRVJTb21NbGJhZk0zdFZTekZYWVVza1FQYWtPb1FIOWV1V2JuV3k5Q1RROW5hQnd5YWRJSldfa1ZYS0piNFJaZVdnMnpveElIamI3VFVMQk9QUzY2aGZwU3Y3N3IzREh0T1RMeXEyVlpOSlpDdXJYT1hZTExaQ1VtUWtLdUtETUZ5TG91NzJFQ2Q1WkVsZmY5Zk0tZ2plWjd6LWhiWEhJVG9DcVFqb2JaNWM2dzYwSnVBMHR2TkxwMjgxUm14WlhmTzVjMG5WSEdyYS1hY2k3SGZ4X2d1anBLcWRUZHVENnRvQ3FXSFF1T25WYkRCdEtHSllZZjNWOW94OHFB
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
    manager: kube-controller-manager
    operation: Update
    time: "2021-12-10T11:19:21Z"
  name: default-token-w8dkr
  namespace: default
  resourceVersion: "410"
  uid: 3d7ac7b4-633e-4591-8c34-8d5d0f84470f
type: kubernetes.io/service-account-token

➜ TOKEN="ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrOUJhRWd0V0hoeVZESktXSHAxVlRKbmMwTTJabk01Vmt0eFRrdHZOalJwZFV0cVMwdDZiekpvYVRnaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dGR6aGthM0lpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJamc1T1dZd1lUaG1MVFUxTXpjdE5EbG1OeTA1TW1KaExUYzJZMlZpTmpFM09ETmtOaUlzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuR0cyaWFwTV9IQlNOQTE2Vm5fNTN5NFRER1o0NUltUHBYYzVfUlV3OEJKbUxpZ3VSN004X1NhM1A3YUZnQnIzT19uZk1rWDZqcFkxNXBzRlpuVU1XS0JJZy1KRVJTb21NbGJhZk0zdFZTekZYWVVza1FQYWtPb1FIOWV1V2JuV3k5Q1RROW5hQnd5YWRJSldfa1ZYS0piNFJaZVdnMnpveElIamI3VFVMQk9QUzY2aGZwU3Y3N3IzREh0T1RMeXEyVlpOSlpDdXJYT1hZTExaQ1VtUWtLdUtETUZ5TG91NzJFQ2Q1WkVsZmY5Zk0tZ2plWjd6LWhiWEhJVG9DcVFqb2JaNWM2dzYwSnVBMHR2TkxwMjgxUm14WlhmTzVjMG5WSEdyYS1hY2k3SGZ4X2d1anBLcWRUZHVENnRvQ3FXSFF1T25WYkRCdEtHSllZZjNWOW94OHFB"

➜ echo -n $TOKEN | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik9BaEgtWHhyVDJKWHp1VTJnc0M2ZnM5VktxTktvNjRpdUtqS0t6bzJoaTgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdzhka3IiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg5OWYwYThmLTU1MzctNDlmNy05MmJhLTc2Y2ViNjE3ODNkNiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.GG2iapM_HBSNA16Vn_53y4TDGZ45ImPpXc5_RUw8BJmLiguR7M8_Sa3P7aFgBr3O_nfMkX6jpY15psFZnUMWKBIg-JERSomMlbafM3tVSzFXYUskQPakOoQH9euWbnWy9CTQ9naBwyadIJW_kVXKJb4RZeWg2zoxIHjb7TULBOPS66hfpSv77r3DHtOTLyq2VZNJZCurXOXYLLZCUmQkKuKDMFyLou72ECd5ZElff9fM-gjeZ7z-hbXHIToCqQjobZ5c6w60JuA0tvNLp281RmxZXfO5c0nVHGra-aci7Hfx_gujpKqdTduD6toCqWHQuOnVbDBtKGJYYf3V9ox8qA

➜ echo -n "eyJhbGciOiJSUzI1NiIsImtpZCI6Ik9BaEgtWHhyVDJKWHp1VTJnc0M2ZnM5VktxTktvNjRpdUtqS0t6bzJoaTgifQ" | base64 -d
{"alg":"RS256","kid":"OAhH-XxrT2JXzuU2gsC6fs9VKqNKo64iuKjKKzo2hi8"

➜ echo -n "eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdzhka3IiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6Ijg5OWYwYThmLTU1MzctNDlmNy05MmJhLTc2Y2ViNjE3ODNkNiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ" | base64 -d
{"iss":"kubernetes/serviceaccount","kubernetes.io/serviceaccount/namespace":"default","kubernetes.io/serviceaccount/secret.name":"default-token-w8dkr","kubernetes.io/serviceaccount/service-account.name":"default","kubernetes.io/serviceaccount/service-account.uid":"899f0a8f-5537-49f7-92ba-76ceb61783d6","sub":"system:serviceaccount:default:default"

Authenticating Proxy

Custom authenticator plugins can be specified as Webhooks. This is how AWS IAM users are authenticated to EKS clusters using aws-iam-authenticator, the authenticator gets its configuration information from the aws-auth ConfigMap in kube-system namespace.

Once the authentication is done, the authenticator can send extra information to the api-server to set the client identity. The API server can be configured to identify users from request header values, such as X-Remote-User. An authenticating proxy, can set the request header value.

• --requestheader-username-headers Required, case-insensitive. Header names to check, in order, for the user identity. The first header containing a value is used as the username.

• --requestheader-group-headers 1.6+. Optional, case-insensitive. "X-Remote-Group" is suggested. Header names to check, in order, for the user's groups. All values in all specified headers are used as group names.

For example, with this configuration:

--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group

this request:

GET / HTTP/1.1
X-Remote-User: fido
X-Remote-Group: dogs
X-Remote-Group: dachshunds

would result in this user info:

name: fido
groups:
- dogs
- dachshunds

In order to prevent header spoofing, the authenticating proxy is required to present a valid client certificate to the API server for validation against the specified CA before the request headers are checked.

• --requestheader-client-ca-file Required. PEM-encoded certificate bundle. A valid client certificate must be presented and validated against the certificate authorities in the specified file before the request headers are checked for user names.

• --requestheader-allowed-names Optional. List of Common Name values (CNs). If set, a valid client certificate with a CN in the specified list must be presented before the request headers are checked for user names. If empty, any CN is allowed.

If every authenticator fails, the request fails and an error is returned. If authentication succeeds, the Authorization header is removed from the request, and user information is added to its context. This gives future steps (such as authorization and admission controllers) the ability to access the previously established identity of the user.

Authorization

Okay, so. Request has been sent and we have successfully defended to kube-apiserver that we are who we say we are, what a relief 😌 !!

But we might not have permissions to do what we might want to do (after all, identity and permissions are not the same thing).

The kube-apiserver handles authorization the same way it did authentication: based on flag inputs, it will assemble a chain of authorizers that will be run against every incoming request. When multiple authorization modules are configured, each is checked in sequence. If any authorizer approves or denies a request, that decision is immediately returned and no other authorizer is consulted. If all modules have no opinion on the request, then the request is denied. A deny returns an HTTP status code 403.

Request Attributes

Kubernetes reviews only the following API request attributes:

• user - The user string provided during authentication.

• group - The list of group names to which the authenticated user belongs.

• extra - A map of arbitrary string keys to string values, provided by the authentication layer.

• API - Indicates whether the request is for an API resource.

• Request path - Path to miscellaneous non-resource endpoints like /api or /healthz.

• API request verb - API verbs like get, list, create, update, patch, watch, delete, and deletecollection are used for resource requests.

• HTTP request verb - Lowercased HTTP methods like get, post, put, and delete are used for non-resource (requests to endpoints other than /api/v1/... or /apis/<group>/<version>/... are considered "non-resource requests") requests.

• Resource - The ID or name of the resource that is being accessed (for resource requests only)

• Subresource - The subresource that is being accessed (for resource requests only).

• Namespace - The namespace of the object that is being accessed (for namespaced resource requests only).

• API group - The API Group being accessed (for resource requests only). An empty string designates the core API group.

Authorization Modes

The Kubernetes API server may authorize a request using one of several authorization modes:

Node

A special-purpose authorization mode that grants permissions to kubelets based on the pods they are scheduled to run.

RBAC

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In this context, access is the ability of an individual user to perform a specific task, such as view, create, or modify a file.

When specified, RBAC (Role-Based Access Control) uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing admins to dynamically configure permission policies through the Kubernetes API. To enable RBAC, start the apiserver with --authorization-mode=RBAC.

Webhook

A WebHook is an HTTP callback: an HTTP POST that occurs when something happens.

Check API Access

You can check if you are authorised to perform an action using kubectl's auth can-i subcommand. The command uses the SelfSubjectAccessReview API in authorization.k8s.io group to determine if the current user can perform a given action, and works regardless of the authorization mode used.

kubectl auth can-i create deployments --namespace dev

Administrators can combine this with user impersonation to determine what action other users can perform.

kubectl auth can-i list secrets --namespace dev --as dave

kubectl auth can-i list pods --namespace target  --as system:serviceaccount:dev:default

Admission control

At this point, from kuber-apiserver's perspective it believes we are who we say we are and are permitted to continue. But with Kubernetes, other parts of the system have strong opinions about what should and should not be permitted to happen. This is where admission controllers enter the picture.

Admission controllers intercept the request to ensure that it matches the wider expectations and rules of the cluster. They are the last control before an object is persisted to the etcd. Admission controllers may be "validating", "mutating", or both. Mutating controllers may modify related objects to the requests they admit; validating controllers may not. They limit requests to create, delete, modify objects. They do not limit requests to read objects.

The way admission controllers work is similar to the way authenticators and authorizers work, but with one difference: unlike authenticator and authorizers chains, if a single admission controller fails, the entire request is rejected immediately and an error is returned to the end-user.

There are a ton of admission controllers and each is stored as a plugin in plugin/pkg/admission directory and is then compiled into main K8s binary. Some examples are

• AlwaysPullImages

• CertificateApproval

• CertificateSigning

• DefaultIngressClass

• DefaultStorageClass

• NamespaceAutoProvision

Dynamic Admission Controls

In addition to compiled-in admission plugins (which we saw above), admission plugins can be developed as extensions and run as webhooks configured at runtime. These webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating and mutating admission webhook. Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies.

etcd

By this point K8s has fully vetted our request and permitted it to forth by storing it in etcd. K8s uses etcd as storage backend to store all cluster data.

In our case (deploying 3 nginx replicas - in case you forgot about that 😉).

• A create handler was created

• Resource was decoded/deserialized

• Auditing was done, admission controllers were run

• If everything went well, api-server saves the resource to etcd by delegating to the storage provider.

• Any create errors are caught and, finally, the storage provider performs a get call to ensure the object was actually created.

• The HTTP response is constructed and sent back.

At this point, end-user will get deployment/nginx created message.

kubectl create deploy nginx --image=nginx --replicas=2
deployment.apps/nginx created

But, it just means that a deployment object is persisted to the storage. There are no nginx pods running yet.

Control loops

K8s makes use of the Control Loops or the Controllers to bring the current state closer to the desired state. A controller tracks at least one K8s resource type, these objects have spec field that represents the desired state. Controllers are run in parallel by the kube-controller-manager component.

In our case, deployment and replicaset controllers will be used.

Deployment

After a Deployment record is stored to etcd and initialized, it is made visible via kube-apiserver. When this new resource is available, it is detected by the Deployment controller, whose job it is to listen out for changes to Deployment records. In our case, the controller registers a specific callback for create events via an informer (An informer is a pattern that allows controllers to subscribe to storage events and easily list resources they're interested in. blog post).

This handler will be executed when our Deployment first becomes available and will start by adding the object to an internal work queue. By the time it gets around to processing our object, the controller will inspect our Deployment and realise that there are no ReplicaSet records associated with it. It does this by querying kube-apiserver with label selectors.

After realising none exist, it will begin a scaling process to start resolving state. It does this by rolling out (i.e. creating) a ReplicaSet resource(note: all the steps starting with auth, authz to persisting object in etcd would happen for ReplicaSet as well), assigning it a label selector, and giving it the revision number of 1. The ReplicaSet's PodSpec is copied from the Deployment's manifest, as well as other relevant metadata.

The status is then updated and it then re-enters the same reconciliation loop waiting for the deployment to match a desired, completed state. Since the Deployment controller is only concerned about creating ReplicaSets, this reconciliation stage needs to be continued by the next controller, the ReplicaSet controller.

ReplicaSet

Now we have a deployment object, a replicaset object but still no pods. This is where ReplicaSet controller comes in. It monitors the lifecycle of ReplicaSets and their dependent resources (Pods). Like most other controllers, it does this by triggering handlers on certain events(event we're interested in is creation).

When a ReplicaSet is created (courtesy of the Deployments controller) the RS controller inspects the state of the new ReplicaSet and realizes there is a skew between what exists and what is required. It then seeks to reconcile this state by bumping the number of pods that belong to the ReplicaSet.

Kubernetes enforces object hierarchies through Owner References. Not only does this ensure that child resources are garbage-collected once a resource managed by the controller is deleted (cascading deletion), it also provides an effective way for parent resources to not fight over their children. So, the pods will have owner ref to RS and RS would have ref to Deployment.

Scheduler

Now, we have a Deployment, a RS and three pods objects in etcd. Our pods, however, are stuck in Pending state because they haven't been scheduled to a node. Scheduler controller to rescue!

The scheduler runs as a standalone component and operates in the same way as other controllers. It filters pods that have an empty NodeName field in their PodSpec and attempts to find a suitable Node that the pod can reside on. The way the default scheduling algorithm works is the following:

• filtering - finds the set of Nodes where it's feasible to schedule the Pod. For example, the PodFitsResources filter checks whether a candidate Node has enough available resource to meet a Pod's specific resource requests.

• scoring - the scheduler ranks the remaining nodes to choose the most suitable Pod placement. The scheduler assigns a score to each Node that survived filtering, basing this score on the active scoring rules. The highest ranked node is then selected for scheduling.

Once the algorithm finds a node, the scheduler sends a bind object with the pod and the node ref to the apiserver. API-server sets the NodeName to the one in the bind object and sets its PodScheduled status condition to True.

Custom Scheduler

Filtering and scoring behavior of the scheduler can be configured using

• Scheduling Policies: allow you to configure Predicates for filtering and Priorities for scoring.

• Scheduling Profiles: allow you to configure Plugins that implement different scheduling stages.

OR you can write your own custom scheduler and use it by specifying schedulerName field in podSpec.

kubelet

Okay, let's see what has been done so far: the HTTP request passed authentication, authorization, and admission control stages; a Deployment, ReplicaSet, and three Pod resources were persisted to etcd; and, finally, each Pod was scheduled to a suitable node. However, the state we are talking about exists purely in etcd. The next steps involve distributing this state across the worker nodes, which is the whole point of a distributed system like Kubernetes! The way this happens is through a component called the kubelet.

Kubelet runs on every node on the cluster and is responsible for managing the lifecycle of pods. This means it handles all of the translation logic between the abstraction of a "Pod" (which is really just a Kubernetes concept) and its building blocks, containers. It also handles all of the associated logic around mounting volumes, container logging, and many more important things.

Pod sync

Like a controller, kubelet periodically queries the apiserver for pods whose NodeName matches the name of the node kubelet is running on. Once it has that list, it detects changes by comparing against its own internal cache and begins to synchronise the state. Let's take a look at what that synchronization process looks like when the pod is being created(we are creating nginx pods, remember?):

1. Record pod worker start latency since kubelet first saw the pod.

2. Call generateAPIPodStatus to prepare an v1.PodStatus for the pod, which represents the state of a Pod's current Phase. The Phase of a Pod is a high-level summary of where the Pod is in its lifecycle. Examples include Pending, Running, Succeeded, Failed and Unknown.

3. Update the status of the pod in the status manager, which is tasked with asynchronously updating the etcd record via the apiserver.

4. Next, it is checked whether pod can be run. A series of softAdmitHandlers are checked, these include enforcing AppArmor profiles and [NO_NEW_PRIVS (github.com/kubernetes/kubernetes/blob/v1.23..) (pod-security-policies)[kubernetes.io/docs/concepts/policy/pod-secu... Pods denied at this stage will stay in the Pending state indefinitely.

5. Create Cgroups for the pod and apply resource parameters. This is to enable better Quality of Service (QoS) handling for pods.

6. Create a mirror pod if the pod is a static pod, and does not already have a mirror pod

7. Data directories are created for the pod. These include the pod directory (usually /var/run/kubelet/pods/<podID>), its volumes directory (<podDir>/volumes) and its plugins directory (<podDir>/plugins).

8. Wait for volumes to attach/mount defined in Spec.Volumes. Depending on the type of volume being mounted, some pods will need to wait longer (e.g. cloud or NFS volumes).

9. Fetch the pull secrets for the pod defined in Spec.ImagePullSecrets.

10. Call the container runtime's SyncPod callback, the container runtime then runs the container

CRI

Now, our containers (not pod anymore) are ready to be launched. The software that does this launching is called the Container Runtime (docker, rkt, containerd etc)

Since v1.5.0, the kubelet has been using a concept called CRI (Container Runtime Interface) for interacting with concrete container runtimes. The CRI is a plugin interface which enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components. CRI is the main protocol for the communication between the kubelet and Container Runtime.

The Kubernetes Container Runtime Interface (CRI) defines the main gRPC protocol for the communication between the cluster components kubelet and container runtime.

During, Container Runtime(CR) syncPod call, kubectl makes following gRPCs to the underlying CR following the CRI

1. Compute sandbox and container changes.

2. Kill pod sandbox if necessary - when sandbox changes or containers in it are dead.

3. Create ephemeral containers - These are used to debug actual application containers.

4. Create init containers.

5. Create normal containers.

In Docker (container runtime) creating a sandbox involves creating a pause container. The pause container provides a way to create cgroups and various other linux namespaces (network, IPC, PID etc) and to share them among all the containers of the pod. That's why we can reach all the containers of a pod at localhost from one another. Pod networking is setup using a CNI plugin during sandbox creation.

Pod networking and CNI

Our Pod is now created: a pause container which hosts all of the namespaces to allow inter-pod communication and our nginx container. But how does networking work and how is it set up?

Kubelet delegates the task of networking setup to a CNI plugin. CNI stands for Container Network Interface and operates in a similar way to the Container Runtime Interface. CNI is an abstraction that allows different network providers to use different networking implementations for containers.

During, the pod sandbox creation kubelet calls the registered CNI plugin with the pod specific data (its name, network namespace etc). Once the plugin finishes its job, the pod should be able to reach other pods on the same host.

Inter-host Networking

By this time pods on same node can communicate with each other. But, what if two pods on different machines want to communicate.

There are several ways to accomplish this. For instance, AWS EKS does it using native VPC networking while solutions like flannel do it using overlay networking.

Note: Inter-host networking isn't configured with every pod creation. It's usually done during cluster creation.

Container startup

As we talked earlier, once the PodSandbox is created, kubelet will

1. Kill the sandbox if necessary

2. Create ephemeral containers - These are used to debug actual application containers.

3. Create init containers.

4. Create normal containers.

ephemeral, init and normal containers are created in the same way

1. Pull container image

2. Create Container via CRI. podSandboxID, containerConfig, podSandboxConfig are passed to the CR.

3. Container is then registered with cpu and memory manager. This allows containers in guaranteed QoS pods with integer CPU requests access to exclusive CPUs on the node.

4. The container is then started

5. If there are any post-start container lifecycle hooks, they are run

Done!

Is this it? I guess so.

After all this, we should have three nginx pods running in our cluster.

kubectl create deploy nginx --image=nginx --replicas=3
deployment.apps/nginx created

kubectl get deploy nginx --show-labels
NAME    READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
nginx   3/3     3            3           80s   app=nginx

kubectl get pods -lapp=nginx
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-4qcft   1/1     Running   0          89s
nginx-6799fc88d8-dlngw   1/1     Running   0          89s
nginx-6799fc88d8-gntj5   1/1     Running   0          89s

Update: I'm publishing blogs as github issues now. You can find them here https://github.com/nakamorg/blogs/issues

Refs:

• main idea and content

• what-is-kubectl

• k8s-schema-validation

• authn-authz

• aws-eks-cluster-auth

• aws-eks-cluster-user

• k8s-auth

• k8s-custom-auth

• more-on-k8s-auth

• k8s-scheduler

• cri-arch

• ephemeral-containers debug-pods

• almighty-pause-container

• cpu-management-policies