← all posts

EKS

April 10, 2023
EKS

EKS Prerequisites

  • Create the cluster.
eksctl create cluster --name=petlover-back \
                      --region=ap-southeast-2 \
                      --zones=ap-southeast-2a,ap-southeast-2b \
                      --without-nodegroup
  • Create the OIDC provider and node group. With --approve, eksctl replaces the service account with a new one.

eksctl utils associate-iam-oidc-provider \
    --region ap-southeast-2 \
    --cluster petlover-back \
    --approve
  • Create the node group.
    1. These add-ons create the required IAM policies automatically within the node group role.
# Create Public Node Group
eksctl create nodegroup --cluster=petlover-back \
                       --region=ap-southeast-2 \
                       --name=petlover-back-ng-public1 \
                       --node-type=t3.medium \
                       --nodes=2 \
                       --nodes-min=2 \
                       --nodes-max=4 \
                       --node-volume-size=20 \
                       --ssh-access \
                       --ssh-public-key=my_key \
                       --managed \
                       --asg-access \
                       --external-dns-access \
                       --full-ecr-access \
                       --appmesh-access \
                       --alb-ingress-access \
                       --node-private-networking # for private VPC
  • Delete the node group and cluster.
eksctl delete nodegroup --cluster=petlover --name=petlover-back-ng-public1

eksctl delete cluster petlover-back
  • If you cannot access AWS EKS, update your kubeconfig:
aws eks --region ap-southeast-2 update-kubeconfig --name petlover-uat

Pull from remote private ECR locally (Secret)

  1. https://skryvets.com/blog/2021/03/15/kubernetes-pull-image-from-private-ecr-registry/

  2. https://medium.com/@danieltse/pull-the-docker-image-from-aws-ecr-in-kubernetes-dc7280d74904

    1. Note: you can use the default namespace here
    2. Note: check your local environment variables
    kubectl create secret docker-registry regcred \
--docker-server=[046381260578.dkr.ecr.ap-southeast-2.amazonaws.com](http://046381260578.dkr.ecr.ap-southeast-2.amazonaws.com/) \
--docker-username=AWS \
--docker-password=$(aws ecr get-login-password --region ap-southeast-2) \
--namespace=default

    The common key name is regcred, used to store the credentials (the key is stored in .dockerconfigjson).

  3. Get current context: kubectl config current-context

    1. arn:aws:eks:ap-southeast-2:0463812XXXX:cluster/petlover-uat — AWS cluster name
    2. Petlover-Prod — Azure AKS cluster name

  4. minikube service <service-name> --url — start a local service server

  5. kubectl get secret regcred --output=yaml — output the newly created secret in YAML format

  6. kubectl delete secret regcred — delete the secret

  7. sudo service docker start — start the Docker daemon

  8. echo -n '<content-to-encode>' | base64 — then paste the result back into the secret file

EKS EBS

  • CSI: container storage interface
  • EBS: elastic block storage (for persistent volumes); provides block level storage for use with EC2 & container instance
  • EBS volumes exposed as storage volumes that persist independently from the life of the EC2 or container instance
  • SC (storage class) and PV (persistence volume) are not namespace-based
  • PVC (persistence volume claim) works with SC to dynamically assign storage. No need to pre-purchase storage — the developer (PVC) simply claims available storage as needed
  • The older approach was: purchase PV first, then submit a PVC, then allocate storage

Demo:

  1. Add the IAM policy that allows EC2 to access EBS to the node group, then apply the CSI driver: https://github.com/stacksimplify/aws-eks-kubernetes-masterclass/tree/master/04-EKS-Storage-with-EBS-ElasticBlockStore/04-01-Install-EBS-CSI-Driver

EKS Cluster

  • EKS control plane (master node)
    1. Is not shared across clusters or AWS accounts.
    2. Consists of at least two API server nodes and three ETCD nodes running across three AZs within a region.
    3. EKS handles unhealthy control plane instances by restarting them across availability zones within the region.
  • Worker nodes and node groups
    1. Connect to the cluster control plane through the cluster API server endpoint.
    2. Additional instances in the node group are deployed through an EC2 Auto Scaling group.
    3. All instances in a group must use the same instance type, AMI, and worker node IAM role.
  • Fargate profiles
    1. AWS provides Fargate controllers that recognize pods assigned to Fargate and schedule them onto Fargate profiles.
  • RBAC policies
    1. RBAC policies authorize Kubernetes actions and can support cross-cluster or cross-account access patterns when combined with AWS identity controls.
  • OIDC: Open ID Connect provider

EKS eksctl

55767f52e49cd59144143a2ca098761.png

Untitled

Untitled

Untitled

  • AMI update: updating the master node is fine — it's always managed by AWS, so you won't notice any difference. When updating the worker (slave) nodes and AMI, the desired count was originally 4, but it scaled up to 10 running nodes and actually 11 total. Micro instances probably couldn't handle the load. The pod count remained stable at the 3 I defined, with a rolling update happening internally.
  • The Kubernetes Metrics Server: is an aggregator of resource usage data in your cluster. Server is commonly used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard

Security: RBAC, IRSA, ClusterRole, ClusterRoleBinding

RBAC: Role based access control

IRSA: IAM role service account

  • A pod may need an IAM role to access AWS resources. Because pod lifecycles are short and IAM is outside Kubernetes, EKS uses a cloud-specific bridge: a Kubernetes service account mapped to an IAM role.
  • The service account is inside EKS and is verified through the cluster-level IAM OIDC provider.

Untitled

Untitled

  • cluster role: grants the cluster access to resources such as nodes and pods for communication; cluster roles are not namespace-limited (equivalent to an AWS policy)
  • A service account (SA) is also a form of cluster role, since it has permissions to operate resources within the cluster, but cannot create new pods — e.g. the alb-ingress-controller

Untitled

Untitled

Untitled

  • How to create a cluster role and SA? Create them separately, then combine them together.
  • role vs cluster role?
    1. Role sets permissions within a particular namespace (must specify namespace) — a role defined when you define a namespace is a namespaced Role
    2. ClusterRole is non-namespaced
  • Granting user permissions:

Untitled

Untitled

Ingress Load balancer controller 1

ALB?

Untitled

Untitled

Untitled

Untitled

Untitled

  • Inside Kubernetes we call it an Ingress Service; in AWS we call it an AWS ALB — but they are equivalent. We refer to them together as a single AWS Application Load Balancer object.

Untitled

Ingress Load Balancer Controller 2

  • prerequisite: install cluster + node groups + OIDC provider

  • Download IAM policy JSON file

    curl -o iam_policy_latest.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
    1. Create the policy
    aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy_latest.json
    1. Note the ARN
    2. Create an IAM service account, binding the policy to the role
    eksctl create iamserviceaccount \
  --cluster=petlover-back \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::046381260578:policy/AWSLoadBalancerControllerIAMPolicy \
  --override-existing-serviceaccounts \
  --approve
    1. Install the AWS Load Balancer Controller with Helm.
    helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=petlover-back \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

// omit it now
  --set region=us-east-1 \
  --set vpcId=vpc-0165a396e41e292a3 \
  --set image.repository=602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon/aws-load-balancer-controller
    1. Optional useful command:
    helm uninstall aws-load-balancer-controller -n kube-system

  • Ingress class? If multiple ingress controllers are running in a Kubernetes cluster, how do you identify which ingress controller an Ingress resource should be associated with? → Use a K8s IngressClass object to associate with the ALB ingress controller (defined at the cluster level to set the default controller)

  • The controller deployment contains only two things: a metrics server and a webhook-server (exposed by the controller service, triggered on resource events)

External DNS

Untitled

;