K8s

DevOps K8s Overview 1

• Kubernetes:
  1. Manages containerized workloads and services.
  2. A Kubernetes deployment gives you a cluster.
• Service: an abstraction that exposes an application and provides a cluster IP, a DNS name, and a port for a set of pods.
• Pods:
  1. The basic components of an application workload.
  2. The smallest deployable units of computing that Kubernetes can create and manage.
  3. Groups of containers that share namespaces, filesystem volumes, and network resources.
  4. Containers within the same logical host are tightly coupled. Outside a cloud environment, the logical host is usually the same physical machine or VM.
• Node: a VM or physical machine that hosts pods for the application workload.
• Control plane: in production, the control plane usually runs across multiple machines.
• Namespace: a logical boundary for resource names, commonly used for environments such as dev, uat, and prod.
• Target port: the port on a pod that a service maps incoming traffic to.

DevOps K8s Overview 2

• Ingress controller: users access the ingress controller via domain name, which then routes to different services (music, streaming, shopping, etc.)
• Managed Kubernetes options:
  1. Azure K8s service
  2. GCP Kubernetes service
  3. AWS EKS
• Control plane: Azure Kubernetes Service cluster management, including auto-scaling.
• Cosmos DB: Azure NoSQL database.
• Grafana: visualizes logs and metrics from Azure data sources; it may pull log data from NoSQL storage depending on the architecture.
• Relationships:
  1. One Kubernetes cluster has many nodes or VMs for fault tolerance and high availability.
  2. One service can target many pods.
  3. One node has at least one pod; worker nodes host the pods.
  4. One pod has at least one container.

DevOps K8s Deep 1

K8s is designed to be more powerful, more flexible, and more robust; refer to earlier posts for basic info.

Docker on a single EC2 instance creates a single point of failure.

Control plane == orchestrator.

Kubernetes provides:

1. Service discovery and load balancing: Kubernetes can discover services and expose containers through DNS.
2. Storage orchestration across local and cloud storage.
3. Automated rollouts and rollbacks for code and configuration across multiple containers.
4. Automatic bin packing: Kubernetes places workloads efficiently, scales up or down, and can enforce custom resource allocation. A common target is keeping node CPU below about 60%.
5. Self-healing: Kubernetes detects health problems, replaces broken instances, and waits for new instances to become healthy before routing traffic.
6. Secret and configuration management for environment variables, passwords, and centralized configuration.

Control panel:

1. Decides which pods run on which nodes, maximizing resource utilization.
2. Creates pods.
3. Can run on one node in a small setup, but should be highly available in production.
4. Kube-apiserver: exposes the Kubernetes API and is used for service communication.
5. etcd: stores secrets & configuration
6. kube-scheduler: assign new pod to available node, better space management
7. Kube-controller-manager: logic handler
8. Cloud-controller-manager: integrates with cloud provider APIs at the cluster level

DevOps K8s Deep 2

Node:

1. == EC2
2. A node is managed by the control plane.
3. It does the actual work and is also called a worker node.
4. Each node runs at least one pod.
5. at least one kubelet: a middleman between the brain (control plane) and the worker
6. at least one container runtime: responsible for running containers (anything that implements the container runtime interface works)
7. Kube-proxy: network proxy, responsible for managing IP communication between pods
8. Docker here is responsible for managing containers within each pod

K8s cluster:

1. at least one worker node
2. Cluster services run across multiple servers and control multiple nodes.

Swarm: creates a swarm using the Docker CLI, think of it as a lightweight, Docker-native Kubernetes

1. swarm == Kubernetes cluster

• Kind: a tool for running local Kubernetes clusters using docker container "nodes". Primarily used for testing Kubernetes itself
• namespace == isolated logical project
• ingress controller == load balancer

Swarm troubleshooting tips:

1. Version, version, version! Try to match your instructor's version as closely as possible, otherwise there will be many pitfalls
2. If using Kubernetes 1.21, remember to use apiVersion: batch/v1beta1 in demojob; otherwise use apiVersion: batch/v1 (I succeeded by upgrading to the latest Kubernetes 1.25)
3. Similarly, for kubectl -n create token admin-user, if using an older version, the command will not exist and you cannot directly generate an admin user token

Kubernetes basic components 1

• Kubernetes benefits: high availability, high scalability, and disaster recovery.
• Pods
  1. abstraction over container
  2. each pod gets its own private IP address
  3. smallest unit
• service:
  1. provide the permanent IP address
  2. lifecycle of pod and service not connected; so when one fails, another one can connect directly quickly
  3. load balancer; send traffic to more pods
• internal service vs external service(public ip address with port in the node level): private IP vs public IP
• Ingress: a proxy(provides Public IP address) for external service → connect to service
• ConfigMap: external configuration of your application (the Pod actually gets the data that ConfigMap contains) such as url link (ENV)
• secret: used to store secret data without a plain text (ENV)
• Volumes: preserve data because Kubernetes does not manage application data persistence by itself.
• deployment:
  1. A blueprint for application pods, including how many replicas should run and how they scale up or down.
  2. is a layer of abstraction on top of pods
• deployment for stateless Apps vs StatefulSet for stateful apps or DB

K8s Basic components 2

• one node(slave nodes) must have: container runtime, kubelet, Kube proxy
  1. kubelet: interacts with both container and node; starting a pod with a container inside and assign the resource from node to the container (CPU, RAM)
  2. Kube Proxy: forwarding requests from services to pods (avoid network overhead of sending the request to another machine)
• Master nodes(control plane):
  1. API server: like a cluster gateway which gets the initial request of any UPDATE or QUERY; acts as a gatekeeper for authentication.
  2. scheduler: schedule new pod → api server → scheduler → where to put the pod (like packing luggage, stuffing bags of different sizes into different containers); scheduler just decides on which Node new Pod should be scheduled. (the actual execution is done by kubelet on each node)
  3. controller management: detect any pods die in any nodes and reschedule those pods as soon as possible; (detects cluster state changes); controller manager → scheduler → kubelet; ensures proper STATE of cluster components.
  4. etcd: key-value store of a cluster state; cluster brain! dedicated to recording state; this storage cross all the master replica nodes. A typical small cluster has 2 masters and 3 workers; masters require less compute resources, workers require more; masters and workers can be scaled infinitely based on demand
• Worker nodes:
  1. container runtime engine: Docker, containerd, CRI-O, frakti…
  2. kubelet: it is an agent that runs on each node in the cluster — ensures each instance is running correctly and reports back to master; scheduler also uses this to create pods
  3. kube-proxy(network proxy): allows each node communicate with each other (allow network communications)

K8s Services

• A service links to pods through selectors.
• Kubernetes creates an Endpoint object with the same name as the service and tracks which pods are members of that service.
• Headless service: only communicate with 1 specific Pod directly (ClusterIP: None)
• load balancer service is an extension of the NodePort Service
• NodePort Service is an extension of ClusterIP service

K8s Common Imperative Commands

1. kubectl edit pod redis: edit the YAML configuration of an already-created pod, which in turn modifies the image used to build the pod
2. kubectl run <tagName, your own name> --image=<docker image name> e.g. kubectl run redis --image=redis meaning: create a new pod
3. kubectl delete pods <tagName> — delete a specific pod
4. kubectl get pods -o wide — get a list of pods with extended output (wide)
5. kubectl describe pod <tagName>
6. kubectl get pods
7. kubectl run redis --images=redis123 --dry-run=client -o YAML > redis.yaml — output configuration to YAML file
8. kubectl apply -f <filename> — apply a file to create pods
9. kubectl create -f <filename> — create from file
10. kubectl get replicaset: get command to see a list of replica sets created
11. kubectl delete replicaset <tagName>
12. kubectl replace -f replicaset <fileName> — replace configuration
13. kubectl scale --replicas=6 -f <fileName>
14. kubectl edit replicaset myapp-replicaset — edit replicaset settings
15. kubectl scale rs new-replica-set --replicas=2 — rs is short for replicaset
16. kubectl get all — get full details including deployments, replicas and pods.

Deployment strategy: create, get, update, status, rollback

1. kubectl rollout undo deployment/myapp-deployment — rollback deployed code

-- record=true will record the command execution history

1. kubectl set image deployment myapp-deployment Nginx=Nginx:1.18 — update image

Service

1. NodePort: allows external access to internal services, split into target port, port, and nodeport

kubectl get pods,svc — view both running services

debugging:

kubectl logs {pod-name}

kubectl exec -it <pod name> --bin/bash

kubectl get deployment nginx-deployment -o YAML > nginx-deployment-result.yaml — copy configuration info to YAML file

'kubectl delete --all deployments --namespace=foo' — delete all deployments, services, etc.

When starting a service with type=loadbalancer, use minikube service mongo-express-service to expose the external IP address

kubectl create namespace uat-petlover

Volume

• PersistentVolume (PV): not tied to any namespace; it is available to the whole cluster.
• PersistentVolumeClaim (PVC): must exist in the same namespace as the pod that uses it.
• StorageClass (SC): provisions persistent volumes dynamically when a PVC requests storage.
• ConfigMaps and Secrets are local volume sources managed by Kubernetes; they are not created through PVs or PVCs.

Usage flow:

1. Old approach: admins configure storage (Manually) => create PV => developer claim PV using PVC
2. New approach: developer claim PV using PVC => Storage class SC allocate resources (PV) to developer

Helm - Package Manager of K8s

• Helm Charts:
  1. bundle of Yaml files
  2. create your own Helm charts with Helm
  3. push them to Helm repository
  4. Use charts created by others.
  5. Helm hub, Helm repository, Helm pages
• templating engine
  1. define a common blueprint
  2. dynamic values are replaced by placeholders
• dev - staging - prod
• release management: state management in tiller server (V2, V3 now tiller no longer exists)
• Helm command:
  1. Add a Helm repository locally: helm repo add bitnami https://charts.bitnami.com/bitnami
  2. Search for needed application from your added repository: helm search repo Jenkins
  3. Pull down a Helm chart: helm pull bitnami/<image_name> --untar=true
  4. Install a Helm chart: helm install <give-a-name> bitnami/nginx
  5. Delete a Helm chart's deployed service from the node: helm delete <give-a-name> (note: at this point, the service inside the pod will also be deleted)

Namespaces in system

• Arch knowledge:
  1. deployment manage a replica set
  2. replica set manage all replica of its pod
  3. the pod is an abstraction of a container
• Kubernetes namespaces
  1. a virtual cluster inside a cluster
  2. kubernetes-dashboard(namespace) only with minikube
  3. kube-system: system processes; master and kubectl processes; do not create or modify in kube-system
  4. kube-public: publicly accessible data; a configmap, which contains cluster information
  5. kube-node-lease: heartbeats of nodes; each node has associated lease object in namespace; determines the availability of a node
  6. default: resources you create are located here
  7. kubectl create namespace my-namespace
  8. create a namespace with a configuration file
• When should you use multiple namespaces?
  1. To group resources such as databases, monitoring, logging, Elastic Stack, and Nginx-ingress.
  2. To let multiple teams work on the same application without affecting each other.
  3. resource sharing: staging and development
  4. blue/green deployment (reuse those components in both envs)
  5. To enforce resource limits in constrained environments. Each namespace should define its own ConfigMaps; services can communicate across namespaces.
  6. Volumes and nodes are not namespaced; they are shared cluster resources.
  7. Add --namespace=my-namespace to isolate commands to a namespace.
  8. Switch namespace:
     1. kubectl config set-context $(kubectl config current-context) --namespace=test-ns — switch to test-ns
     2. kubectl config set-context $(kubectl config current-context) --namespace=default — switch back to default

1. kube-node-lease: improves the performance of our node heartbeats (benefit for cluster autoscaling)
2. kube-public: readable for cluster
3. kube-system:
   1. kube-dns
   2. kube-proxy
   3. ebs-csi-node

Probes

• liveness Probe
  1. kubelet uses liveness probes to know when to restart a container
  2. liveness probes could catch a deadlock, restart container " livenessProbe: exec: command: - /bin/sh - -c - nc -z localhost 8095 initialDelaySeconds: 60 periodSeconds: 10 " 'nc -z' will check if someone is listening on the other side
• readiness probe
  1. to know when a container is ready to accept traffic
  2. When the container is not ready, remove it from load balancers based on the readiness probe signal.
• startup probe:
  1. when a container application has started
  2. it will disable liveness & readiness checks first to avoid restarting container repeatedly

OIDC and IRSA with IAM identity federation

• To enable and use AWS IAM roles for Kubernetes service accounts on our EKS cluster, we must create & associate OIDC identity provider. Allow it to make calls to AWS APIs on your behalf.
  1. IRSA: IAM role service account — can associate an IAM role with a Kubernetes service account, so that service account can provide AWS permissions to any pods using that service account
  • To create and associate an OIDC identity provider with our EKS cluster, we need to follow these steps:
    1. Create an IAM OIDC provider in the IAM console.
    2. Associate the IAM OIDC provider with our EKS cluster.
    3. Create an IAM policy that allows our Kubernetes service account to assume the IAM role.
    4. Create a Kubernetes service account that uses the IAM role.
    5. Verify that the Kubernetes service account can assume the IAM role by running a test pod.
  2. OpenID connect provider: EKS acts as a provider, you can use the AWS EKS OpenID Connect provider to access AWS services using the IAM identity federation https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html