
The cleanest way to think about Bedrock AgentCore is this: Bedrock lets you call models; AgentCore helps you operate agents.

That sounds small, but it is actually the difference between a feature and a system.

A feature can get away with inference plus some glue code. A system that reasons, calls tools, keeps state, and acts over time needs stronger answers to harder questions:

1. Where does execution live?
2. Where does state live?
3. How is outbound access controlled?
4. How is identity propagated?
5. How do you debug a bad tool decision after the fact?

## Runtime

Runtime is the first place where teams often misunderstand the product. They hear "runtime" and assume persistence. That is the wrong mental model.

The better model is session-scoped managed execution. It gives you an environment where the agent can run, not a permanent home where its truth should live.

The useful concrete detail is that the isolation model is closer to a managed micro-VM world than to "just another function call." That gives stronger session separation, but it still does not turn local runtime disk into durable application state.

That distinction matters because one of the fastest ways to build a fragile agent is to let local runtime state become business state by accident.

If the session expires, scales down, or restarts, the system should still know what task was in progress and what the next safe step is. That means durable state belongs outside the runtime.

This matters even more for longer-running agents. If the platform lets a container run for hours, that is helpful for workflows. It is not permission to blur execution state and system-of-record state.

## Gateway

Gateway is more important than it first appears because it is not just a convenience layer for tool calls. It is where you begin to recover control over what the agent is allowed to touch.

Many weak agent implementations hand raw tool access directly to the orchestration layer and hope prompt wording will keep behavior reasonable. That breaks down as soon as the tool surface grows.

Gateway helps by making tool exposure a platform decision rather than a prompt improvisation problem.

That becomes more valuable as you add browser automation, code interpreters, MCP-style tools, or internal APIs. The problem stops being "can the agent call tools?" and becomes "which tools should exist for which caller under which identity?"

## Memory

Memory is where teams need the most discipline.

Not every transcript should become long-term memory. A lot of conversational data is just noise, retries, or temporary reasoning that should never be promoted into durable context.

The hard question is not "can we store memory?" It is "what deserves to become memory?"

If you store every intermediate step, memory turns into a junk drawer. If you store curated preferences, summaries, and durable facts, memory becomes operationally useful.

This is also where cost judgment matters. Managed memory is not automatically the cheapest place to store raw chat history. For simple transcripts or session artifacts, DynamoDB or S3 may still be the cleaner answer.

## Identity

Identity is where production agent systems stop being a toy.

There are two identity problems:

1. Inbound identity: who is allowed to call the agent.
2. Outbound identity: who the agent represents when calling external systems.

If every outbound call uses a generic service account, you lose attribution. If every outbound call impersonates the human without policy boundaries, you widen blast radius.

This is why "tool calling works" is not the same thing as "the agent is production-ready."

## Observability And Evaluation

This is the part most demos skip and the part operations teams care about most.

When the agent makes a wrong decision, you need to know:

1. What context it saw.
2. Which tools were available.
3. Which tool it chose.
4. What came back.
5. Why the final answer looked plausible anyway.

Without traces and evaluation, the only debugging method is reading chat-like logs and guessing.

That is the real production gap. Not model quality in isolation. Operational evidence.

The teams that benefit most from AgentCore are usually the ones already feeling this pain. They have moved beyond "does the demo work?" and into "how do we version, trace, evaluate, and govern this thing over time?"

## When AgentCore Fits

AgentCore makes sense when an agent is stateful, integrated, security-sensitive, or expected to survive beyond a hackathon.

It is overkill when the application is still one prompt plus retrieval.

My test is simple: if the main risk has shifted from answer quality to system behavior, you are entering AgentCore territory.

That is when runtime boundaries, memory policy, gateway control, identity propagation, and observability stop being optional architecture diagrams and start becoming the product.


Runtime, Gateway, Memory, Identity, observability, and the production gap for agentic applications

AWS Bedrock AgentCore For Production Agents


The confusing part of AWS AI is not the number of services. It is that AWS now has products at several altitudes, and teams often choose by branding instead of by control boundary.

The clean way to read the portfolio is this:

1. Managed AI products.
2. Bedrock as the model platform.
3. AgentCore as the operating layer for agents.
4. SageMaker and lower infrastructure for custom ML work.

The mistake I see most is skipping straight to the bottom because it feels "more serious." In practice, many teams should start higher.

## Start Higher Than Your Instinct Tells You

If a managed product already matches the job, use it. A lot of companies waste time rebuilding assistants for search, incident triage, or code help when the differentiator is not the model orchestration. The differentiator is the business process around it.

The more code you own, the more operational debt you own. So the first decision is not "which model?" It is "how much of this stack do we actually need to own?"

## Bedrock Is The Default App Layer

Bedrock is the right center for most application teams because it gives model access, inference controls, guardrails, knowledge bases, and evaluation features without forcing you into model training or infrastructure ownership.

That matters because AWS is no longer asking you to bet on one model family. Bedrock gives access to a broad model catalog across vendors, plus operational features around evaluation, prompt caching, routing, and inference tier selection.

That means it is a good fit for:

1. RAG over internal documents.
2. Summarization.
3. Content generation.
4. Classification and extraction.
5. Document processing.

In other words: if your product still looks like "input, retrieve, infer, return," Bedrock is usually enough.

Most teams should exhaust this layer before inventing an agent platform.

In practice, this is where a lot of useful product work lives: RAG, summarization, extraction, content generation, document workflows, and policy-governed internal assistants.

## AgentCore Is About The Production Gap

AgentCore matters when the system is no longer just a model endpoint. It matters when the system has to:

1. Plan across multiple steps.
2. Call tools with identity and policy boundaries.
3. Persist memory or task state.
4. Resume work.
5. Emit traces and evaluations that operations teams can trust.

That is the production gap between a demo agent and a service someone is willing to own.

If your team still cannot clearly define tool boundaries, memory semantics, or evaluation criteria, AgentCore will not save you. It gives you operating surfaces. It does not give you application clarity.

This is why I think of Bedrock and AgentCore as answering two different questions:

1. Bedrock: how do I call and govern models?
2. AgentCore: how do I run and operate agents safely?

## SageMaker Is For Teams With A Real ML Reason

SageMaker still matters, but not for the reasons many application teams think.

You go there when you need:

1. Fine-tuning or custom training.
2. ML pipelines and feature engineering.
3. Domain-specific hosting decisions.
4. Stronger control over model lifecycle and deployment.

You do not go there just because Bedrock feels "too managed."

Owning more infrastructure is only a win when that control directly improves the business outcome.

Below SageMaker, the infrastructure layer starts to matter: vector search, databases, storage, Trainium, Inferentia, GPUs, and data foundations. That is real power, but it is also real ownership.

## A Simple Selection Rule

1. Need an out-of-the-box workflow? Start with the managed product.
2. Need a custom GenAI application? Start with Bedrock.
3. Need a production-grade agent that acts over time? Evaluate AgentCore.
4. Need custom model work or ML platform control? Use SageMaker.

The expensive mistake is descending layers before the product proves it needs that freedom.

AWS gives you a staircase. Most teams try to jump to the basement.


Bedrock, AgentCore, managed agents, SageMaker, and where each layer fits

The AWS AI Portfolio For Builders


The agent ecosystem is noisy because different tools solve different layers of the problem. Before choosing a framework, separate three concerns: how the agent runs, how developers build it, and how much of the product experience is already packaged.

## Runtime, Framework, Harness

A runtime answers: how does this keep running, recover, stream, persist state, and support human review?

A framework answers: how do I compose prompts, tools, memory, and agent loops without writing everything from scratch?

A harness answers: can I start with an opinionated, ready-to-use agent environment that already includes tools, defaults, and workflow conventions?

This distinction matters because most teams choose a framework too early and for the wrong reason. They are impressed by the demo loop, not by the operational shape of the problem.

My rule is blunt: if you cannot explain why a plain script plus model API is insufficient, you probably do not need a framework yet.

## Where Teams Usually Go Wrong

The first mistake is buying "agentic" complexity before proving the workflow. A lot of internal tools are still one of these three things:

1. Retrieval plus answer generation.
2. Structured extraction.
3. Human-in-the-loop automation.

None of those require a swarm of agents.

The second mistake is confusing orchestration with product value. A system with planner, executor, reviewer, memory, and tool router sounds impressive. It is also expensive to debug when the real requirement was "read three documents, draft an answer, and ask for approval."

## My Actual Selection Rule

I choose from the bottom up, not the top down:

1. Start with plain API calls if the path is deterministic.
2. Add a lightweight framework if tool wiring or prompt composition becomes repetitive.
3. Add graph-style orchestration only when state, branching, retries, or human checkpoints are truly part of the workflow.
4. Add multi-agent roles only when role separation changes quality, not just aesthetics.
5. Add a durable runtime only when the work must survive long sessions, restarts, audits, or asynchronous handoffs.

That sequence avoids a common trap: adopting a platform before understanding the failure mode.

## LangChain: Good Glue, Weak Governor

LangChain is useful when you need connectors, prompt templates, retrievers, and tool wrappers quickly. It is strong as assembly glue.

It is weak as a governing layer. If you need auditability, resumability, or precise control over execution boundaries, LangChain alone is usually not enough. Teams that overuse it end up writing custom state management beside it anyway.

Use it when:

1. You are prototyping quickly.
2. The value is mostly in integrations.
3. The loop is still short and understandable.

Avoid it when the workflow needs hard operational guarantees.

## LangGraph: Better For Real Workflows

LangGraph becomes useful when the workflow has durable state, explicit branches, retry logic, and human approval gates. In practice, that is where many serious agent systems eventually land.

The benefit is not "more agentic behavior." The benefit is explicit control. You can see where the system is, why it moved, and where it can safely resume.

The cost is that you now have to design the workflow honestly. If the graph is unclear, the business process is probably unclear too.

## CrewAI And AutoGen: Use Sparingly

Multi-agent frameworks are attractive because they mirror how engineering teams think about roles. Researcher. Planner. Reviewer. Implementer.

That mapping is sometimes real. Often it is theater.

If the same model could do the task in one pass with tool access and a verification step, splitting it into five role prompts usually adds latency and confusion, not quality.

I only reach for multi-agent patterns when:

1. The task decomposes into truly independent lanes.
2. Different tools or permissions belong to different roles.
3. A review pass needs isolation from the generation pass.

If every agent reads the same context and edits the same files, you built ceremony, not leverage.

## Lightweight Harnesses Are Underrated

For a lot of engineering automation, the best answer is still a small local harness: one model call loop, a repo, a few files for state, and strong verification.

That approach is boring. It is also easier to debug than a framework stack where failures can come from the model, the framework, the tool adapter, the runtime, or the orchestrator.

Small harnesses win when:

1. The operator is technical.
2. The workflow is local to one repo or task.
3. Human oversight already exists.
4. Durability can be handled with files, tests, and git.

## The Hard Question To Ask First

Before evaluating frameworks, ask: what is the most expensive failure here?

If the failure is "the answer is mediocre," you probably need better prompts and evaluation.

If the failure is "the system edited production data with the wrong identity," you need permissions, audit, and stronger orchestration boundaries.

If the failure is "the work disappeared halfway through a long task," you need durable state.

That question usually narrows the framework choice faster than any feature matrix.

## My Practical Recommendation

For most teams building their first useful agent system:

1. Do not start with multi-agent.
2. Do not start with memory.
3. Do not start with autonomous planning.
4. Start with one agent, one tool boundary, one verification path.

Then add complexity only when the current version fails for a specific reason you can name.

The strongest agent architecture is rarely the most sophisticated one. It is the one whose mistakes you can reproduce on Tuesday morning without guessing.


When to use LangChain, LangGraph, CrewAI, AutoGen, smolagents, or a simple harness

Choosing An Agent Framework In 2026


A lot of people think the hard part of an agent harness is tool calling. It is not. The hard part is stopping the system from lying about completion.

Once an agent can edit files and run commands, the real engineering problem becomes operational control:

1. What exactly is the task?
2. What state survives the session?
3. What counts as done?
4. What evidence is required before the system says it is done?

If those answers are weak, the agent may look productive while quietly drifting.

## The Failure Pattern I See Most

Most long-running agent failures are not model failures in the narrow sense. They are harness failures:

1. The task is underspecified, so the agent rewrites the goal into something easier.
2. Session context compresses away important requirements.
3. The agent edits a lot, verifies little, and reports confidently.
4. Progress is visible in chat but not durable in the repo.

That is why I prefer a plain, inspectable harness over a "smart" one. The job of the harness is not to look autonomous. The job is to make cheating difficult.

## Use A Requirement Ledger, Not A Vibe

Keep a machine-readable feature list or requirement ledger in the repo:

```json
[
  { "id": 1, "feature": "User can sign in", "passes": false },
  { "id": 2, "feature": "User can export CSV", "passes": false }
]
```

The important rule is strict: the agent may change `passes: false` to `passes: true`, but it must not delete requirements just because they are inconvenient.

That sounds obvious, but it blocks a real behavior: agents silently narrowing scope until the remaining task looks done.

## Make Every Session Rebuild State

Every session should begin by rebuilding context from durable sources:

1. Check the working directory.
2. Read the progress note.
3. Inspect recent git history.
4. Read the feature list.
5. Run a baseline test.

This is the engineering equivalent of a shift handover. If a new session cannot reconstruct the work from repo state, the system is not durable yet.

## Work One Verifiable Unit At A Time

Long sessions encourage fake progress. A better rhythm is:

1. Pick one unchecked feature.
2. Implement it.
3. Verify it.
4. Commit it.
5. Update progress.

This keeps rollback cheap, review understandable, and failure attribution possible.

## Separate Claim From Proof

An agent saying "implemented" is not evidence. At minimum, the harness should require one of:

1. A passing test.
2. A build result.
3. A reproducible manual verification note.
4. A screenshot or trace for UI work.

This sounds procedural, but it is the line between automation and theater.

## Verify Like A User

Unit tests are useful, but agents can pass unit tests while the product is unusable. For user-facing work, add end-to-end checks. A browser test that logs in, fills a form, and submits it is stronger evidence than a mocked function test.

## Store Memory Outside The Model

Do not rely on conversation compression as the system of record. Compression is lossy, invisible, and impossible to review after the fact.

Use files, git commits, test output, and progress notes instead. The clean mental model is the same as distributed systems: treat model context like working memory, not durable storage.

## A Better Harness Has Friction In The Right Places

The best harnesses are not "frictionless." They add friction exactly where agents are most dangerous:

1. Before changing task scope.
2. Before claiming success.
3. Before taking risky actions.
4. Before writing irreversible external state.

If a harness makes it easier to edit than to verify, it is biased toward hallucinated completion.

## A Minimal Harness Shape

```text
while work remains:
  read durable state
  choose one feature
  implement
  test
  commit
  update progress
```

This is not glamorous, but it is robust. The goal is not to impress someone with autonomy. The goal is to keep the work recoverable, inspectable, and difficult to fake.

That is the standard I use when evaluating an agent harness: if the current session vanished right now, would the next session know what happened and what still needs proof?


File-backed state, small loops, and practical completion control for long-running agents

Durable AI Agent Harness Design


### EKS Prerequisites

- Create the cluster.

```bash
eksctl create cluster --name=petlover-back \
                      --region=ap-southeast-2 \
                      --zones=ap-southeast-2a,ap-southeast-2b \
                      --without-nodegroup
```

- Create the OIDC provider and node group. With `--approve`, `eksctl` replaces the service account with a new one.

```bash

eksctl utils associate-iam-oidc-provider \
    --region ap-southeast-2 \
    --cluster petlover-back \
    --approve
```

- Create the node group.
  1. These add-ons create the required IAM policies automatically within the node group role.

```bash
# Create Public Node Group
eksctl create nodegroup --cluster=petlover-back \
                       --region=ap-southeast-2 \
                       --name=petlover-back-ng-public1 \
                       --node-type=t3.medium \
                       --nodes=2 \
                       --nodes-min=2 \
                       --nodes-max=4 \
                       --node-volume-size=20 \
                       --ssh-access \
                       --ssh-public-key=my_key \
                       --managed \
                       --asg-access \
                       --external-dns-access \
                       --full-ecr-access \
                       --appmesh-access \
                       --alb-ingress-access \
                       --node-private-networking # for private VPC
```

- Delete the node group and cluster.

```bash
eksctl delete nodegroup --cluster=petlover --name=petlover-back-ng-public1

eksctl delete cluster petlover-back
```

- If you cannot access AWS EKS, update your kubeconfig:

```bash
aws eks --region ap-southeast-2 update-kubeconfig --name petlover-uat
```

### Pull from remote private ECR locally (Secret)

1. [https://skryvets.com/blog/2021/03/15/kubernetes-pull-image-from-private-ecr-registry/](https://skryvets.com/blog/2021/03/15/kubernetes-pull-image-from-private-ecr-registry/)
2. [https://medium.com/@danieltse/pull-the-docker-image-from-aws-ecr-in-kubernetes-dc7280d74904](https://medium.com/@danieltse/pull-the-docker-image-from-aws-ecr-in-kubernetes-dc7280d74904)

   1. Note: you can use the `default` namespace here
   2. Note: check your local environment variables

   ```bash
   kubectl create secret docker-registry regcred \
   --docker-server=[046381260578.dkr.ecr.ap-southeast-2.amazonaws.com](http://046381260578.dkr.ecr.ap-southeast-2.amazonaws.com/) \
   --docker-username=AWS \
   --docker-password=$(aws ecr get-login-password --region ap-southeast-2) \
   --namespace=default
   ```

   The common key name is `regcred`, used to store the credentials (the key is stored in `.dockerconfigjson`).

3. Get current context: `kubectl config current-context`
   1. `arn:aws:eks:ap-southeast-2:0463812XXXX:cluster/petlover-uat` — AWS cluster name
   2. `Petlover-Prod` — Azure AKS cluster name
4. `minikube service <service-name> --url` — start a local service server
5. `kubectl get secret regcred --output=yaml` — output the newly created secret in YAML format
6. `kubectl delete secret regcred` — delete the secret
7. `sudo service docker start` — start the Docker daemon
8. `echo -n '<content-to-encode>' | base64` — then paste the result back into the secret file

### EKS EBS

- CSI: container storage interface
- EBS: elastic block storage (for persistent volumes); provides block level storage for use with EC2 & container instance
- EBS volumes exposed as storage volumes that persist independently from the life of the EC2 or container instance
- SC (storage class) and PV (persistence volume) are not namespace-based
- PVC (persistence volume claim) works with SC to dynamically assign storage. No need to pre-purchase storage — the developer (PVC) simply claims available storage as needed
- The older approach was: purchase PV first, then submit a PVC, then allocate storage

Demo:

1. Add the IAM policy that allows EC2 to access EBS to the node group, then apply the CSI driver: [https://github.com/stacksimplify/aws-eks-kubernetes-masterclass/tree/master/04-EKS-Storage-with-EBS-ElasticBlockStore/04-01-Install-EBS-CSI-Driver](https://github.com/stacksimplify/aws-eks-kubernetes-masterclass/tree/master/04-EKS-Storage-with-EBS-ElasticBlockStore/04-01-Install-EBS-CSI-Driver)

### EKS Cluster

- EKS control plane (master node)
  1. Is not shared across clusters or AWS accounts.
  2. Consists of at least two API server nodes and three ETCD nodes running across three AZs within a region.
  3. EKS handles unhealthy control plane instances by restarting them across availability zones within the region.
- Worker nodes and node groups
  1. Connect to the cluster control plane through the cluster API server endpoint.
  2. Additional instances in the node group are deployed through an EC2 Auto Scaling group.
  3. All instances in a group must use the same instance type, AMI, and worker node IAM role.
- Fargate profiles
  1. AWS provides Fargate controllers that recognize pods assigned to Fargate and schedule them onto Fargate profiles.
- RBAC policies
  1. RBAC policies authorize Kubernetes actions and can support cross-cluster or cross-account access patterns when combined with AWS identity controls.
- OIDC: Open ID Connect provider

### EKS eksctl

![55767f52e49cd59144143a2ca098761.png](55767f52e49cd59144143a2ca098761.png)

![Untitled](<k8s_eks%20(26).png>)

![Untitled](<k8s_eks%20(27).png>)

![Untitled](<k8s_eks%20(28).png>)

- AMI update: updating the master node is fine — it's always managed by AWS, so you won't notice any difference. When updating the worker (slave) nodes and AMI, the desired count was originally 4, but it scaled up to 10 running nodes and actually 11 total. Micro instances probably couldn't handle the load. The pod count remained stable at the 3 I defined, with a rolling update happening internally.
- The Kubernetes Metrics Server: is an aggregator of resource usage data in your cluster. Server is commonly used by other Kubernetes add ons, such as the [Horizontal Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html) or the [Kubernetes Dashboard](https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html)

### Security: RBAC, IRSA, ClusterRole, ClusterRoleBinding

RBAC: Role based access control

IRSA: IAM role service account

- A pod may need an IAM role to access AWS resources. Because pod lifecycles are short and IAM is outside Kubernetes, EKS uses a cloud-specific bridge: a Kubernetes service account mapped to an IAM role.
- The service account is inside EKS and is verified through the cluster-level IAM OIDC provider.

![Untitled](<k8s_eks%20(29).png>)

![Untitled](<k8s_eks%20(30).png>)

- cluster role: grants the cluster access to resources such as nodes and pods for communication; cluster roles are not namespace-limited (equivalent to an AWS policy)
- A service account (SA) is also a form of cluster role, since it has permissions to operate resources within the cluster, but cannot create new pods — e.g. the alb-ingress-controller

![Untitled](<k8s_eks%20(31).png>)

![Untitled](<k8s_eks%20(32).png>)

![Untitled](<k8s_eks%20(33).png>)

- How to create a cluster role and SA? Create them separately, then combine them together.
- role vs cluster role?
  1. Role sets permissions within a particular namespace (must specify namespace) — a role defined when you define a namespace is a namespaced Role
  2. ClusterRole is non-namespaced
- Granting user permissions:

![Untitled](<k8s_eks%20(34).png>)

![Untitled](<k8s_eks%20(35).png>)

### Ingress Load balancer controller 1

ALB?

![Untitled](<k8s_eks%20(36).png>)

![Untitled](<k8s_eks%20(37).png>)

![Untitled](<k8s_eks%20(38).png>)

![Untitled](<k8s_eks%20(39).png>)

![Untitled](<k8s_eks%20(40).png>)

- Inside Kubernetes we call it an Ingress Service; in AWS we call it an AWS ALB — but they are equivalent. We refer to them together as a single AWS Application Load Balancer object.

![Untitled](<k8s_eks%20(41).png>)

### Ingress Load Balancer Controller 2

- prerequisite: install cluster + node groups + OIDC provider
- Download IAM policy JSON file

  1.

  ```bash
  curl -o iam_policy_latest.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
  ```

  2. Create the policy

  ```bash
  aws iam create-policy \
      --policy-name AWSLoadBalancerControllerIAMPolicy \
      --policy-document file://iam_policy_latest.json
  ```

  3. Note the ARN
  4. Create an IAM service account, binding the policy to the role

  ```bash
  eksctl create iamserviceaccount \
    --cluster=petlover-back \
    --namespace=kube-system \
    --name=aws-load-balancer-controller \
    --attach-policy-arn=arn:aws:iam::046381260578:policy/AWSLoadBalancerControllerIAMPolicy \
    --override-existing-serviceaccounts \
    --approve
  ```

  5. Install the AWS Load Balancer Controller with Helm.

  ```bash
  helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
    -n kube-system \
    --set clusterName=petlover-back \
    --set serviceAccount.create=false \
    --set serviceAccount.name=aws-load-balancer-controller

  // omit it now
    --set region=us-east-1 \
    --set vpcId=vpc-0165a396e41e292a3 \
    --set image.repository=602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon/aws-load-balancer-controller
  ```

  6. Optional useful command:

  ```bash
  helm uninstall aws-load-balancer-controller -n kube-system
  ```

- Ingress class? If multiple ingress controllers are running in a Kubernetes cluster, how do you identify which ingress controller an Ingress resource should be associated with? → Use a K8s IngressClass object to associate with the ALB ingress controller (defined at the cluster level to set the default controller)
- The controller deployment contains only two things: a metrics server and a webhook-server (exposed by the controller service, triggered on resource events)

### External DNS

![Untitled](<k8s_eks%20(42).png>)



**Note:** all commands in this guide were tested from Windows PowerShell.

### Prerequisite

1. Install the Azure CLI (`azure-cli`).
2. Run `az login`, copy the code, open it in your local browser, and paste the code.
3. Install the Azure AKS CLI.

```bash
# Install Azure AKS CLI
az aks install-cli
```

4. Create the AKS cluster, node pool, and VNet manually.
5. Configure the cluster credentials and kubeconfig.

```bash
# Configure Cluster Creds (kube configuration)
az aks get-credentials --resource-group petlover-AKS --name Petlover-Prod
```

6. Optional: create a PowerShell alias so `kubectl` can be called with a shorter command such as `k`.

[Set Alias for Kubectl](https://www.techtarget.com/searchitoperations/tutorial/Manage-Kubernetes-clusters-with-PowerShell-and-kubectl#:~:text=Kubectl aliasing in PowerShell&text=Aliasing in PowerShell is the,name to reference a command.&text=After setting up this alias,make a call to Kubernetes).

### Ingress Nginx Load Balancer Controller

![Untitled](<aks%20(46).png>)

![Untitled](<aks%20(47).png>)

1. Create a static IP address.

```bash
# Get the resource group name of the AKS cluster (resource name -> aks name)
az aks show --resource-group petlover-AKS --name Petlover-Prod --query nodeResourceGroup -o tsv

# record resource group name: MC_petlover-AKS_Petlover-Prod_eastus

# TEMPLATE - Create a public IP address with the static allocation
az network public-ip create --resource-group MC_petlover-AKS_Petlover-Prod_eastus \
                            --name myAKSPublicIPForIngress \
                            --sku Standard \
                            --allocation-method static \
                            --query publicIp.ipAddress \
                            -o tsv

# record public IP address
```

2. Install Helm in PowerShell. Helm also needs to be installed on the Jenkins server.

Run PowerShell as administrator, then run `choco install kubernetes-helm`.

3. Add the official stable repository.

```bash
# Create a namespace for your ingress resources
kubectl create namespace ingress-basic

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

#  Customizing the Chart Before Installing.
helm show values ingress-nginx/ingress-nginx

# Use Helm to deploy an NGINX ingress controller
helm install ingress-nginx ingress-nginx/ingress-nginx \
    --namespace ingress-basic \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."kubernetes\.io/os"=linux \
    --set controller.service.externalTrafficPolicy=Local \
    --set controller.service.loadBalancerIP="20.124.104.64"
```

### External DNS - By Managed Service Identity (MSI)

![Untitled](<aks%20(48).png>)

1. Get the tenant ID and subscription ID.

```bash
# To get Azure Tenant ID
az account show --query "tenantId"# To get Azure Subscription ID
az account show --query "id"
```

2. **Create MSI - Managed Service Identity for External DNS to access Azure DNS Zones**

![Untitled](<aks%20(49).png>)

Record the user-assigned identity ID.

3. Create the JSON file.

   ```json
   {
     "tenantId": "b985d900-a4c4-4ea6-9e09-8d1167bac273",
     "subscriptionId": "c4c2e681-0156-457e-b1f2-88020d2efe5c",
     "resourceGroup": "petlover-AKS",
     "useManagedIdentityExtension": true,
     "userAssignedIdentityID": "cc69d3df-0680-4e72-9a58-33bc57b09587"
   }
   ```

4. Associate the MSI with the AKS cluster VMSS.

![Untitled](<aks%20(50).png>)

5. Create the Kubernetes secret and deploy ExternalDNS.

'kubectl create secret generic azure-config-file --from-file=azure.json'

Apply `external-dns.yaml`.

### Ingress - SSL

![Untitled](<aks%20(51).png>)

- Label the ingress-basic namespace to disable resource validation

  ```bash
  kubectl label namespace ingress-basic [cert-manager.io/disable-validation=true](http://cert-manager.io/disable-validation=true)
  ```

- Add the Jetstack Helm repository

  ```bash
  helm repo add jetstack [https://charts.jetstack.io](https://charts.jetstack.io/)
  ```

- Update your local Helm chart repository cache

  helm repo update

- Install the cert-manager Helm chart
  ```bash
  helm install \
  cert-manager jetstack/cert-manager \
  --namespace ingress-basic \
  --version v1.8.2 \
  --set installCRDs=true
  ```
  ```powershell
  helm install cert-manager jetstack/cert-manager --namespace ingress-basic --version v1.8.2 --set installCRDs=true
  ```

### ACR Connection - By Service Principal

![Untitled](<aks%20(52).png>)

1. Create the ACR.
2. Go to access keys and enable the admin user.
3. Use this username and password to log in to ACR and push images

```bash
# Export Command
export ACR_REGISTRY=acrdemo2ss.azurecr.io
export ACR_NAMESPACE=app2
export ACR_IMAGE_NAME=acr-app2
export ACR_IMAGE_TAG=v1
echo $ACR_REGISTRY, $ACR_NAMESPACE, $ACR_IMAGE_NAME, $ACR_IMAGE_TAG

# Login to ACR
docker login $ACR_REGISTRY

# Tag
docker tag acr-app2:v1  $ACR_REGISTRY/$ACR_NAMESPACE/$ACR_IMAGE_NAME:$ACR_IMAGE_TAG
It replaces as below
docker tag acr-app2:v1 acrdemo2ss.azurecr.io/app2/acr-app2:v1

# List Docker Images to verify
docker images acr-app2:v1
docker images $ACR_REGISTRY/$ACR_NAMESPACE/$ACR_IMAGE_NAME:$ACR_IMAGE_TAG

# Push Docker Images
docker push $ACR_REGISTRY/$ACR_NAMESPACE/$ACR_IMAGE_NAME:$ACR_IMAGE_TAG
```

```powershell
docker tag petlover-back:latest petloverprod[.azurecr.io/back/petlover-back:](http://demopublicpetlover.azurecr.io/app1/demo:v1)latest
docker push petloverprod.azurecr.io/back/petlover-back:latest
```

4. Create a service principal.

```bash
#!/bin/bash
# This script requires Azure CLI version 2.25.0 or later. Check version with `az --version`.

# Modify for your environment.
# ACR_NAME: The name of your Azure Container Registry
# SERVICE_PRINCIPAL_NAME: Must be unique within your AD tenant
ACR_NAME=acrdemo2ss
SERVICE_PRINCIPAL_NAME=acr-sp-demo

# Obtain the full registry ID for subsequent command args
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)# Create the service principal with rights scoped to the registry.
# Default permissions are for Docker pull access. Modify the '--role'
# argument value as desired:
# acrpull:     pull only
# acrpush:     push and pull
# owner:       push, pull, and assign roles
SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpull --query password --output tsv)
SP_APP_ID=$(az ad sp list --display-name $SERVICE_PRINCIPAL_NAME --query [].appId --output tsv)# Output the service principal's credentials; use these in your services and
# applications to authenticate to the container registry.
echo "Service principal ID: $SP_APP_ID"echo "Service principal password: $SP_PASSWD"
```

Or

```powershell
$ACR_NAME='petloverprod'
$SERVICE_PRINCIPAL_NAME='acr-petlover-demo'
$ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)

$SP_PASSWD=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpull --query password --output tsv)

$SP_APP_ID=$(az ad app list --display-name http://$SERVICE_PRINCIPAL_NAME --query [].appId --output tsv)

# Note (explanation, no action needed):
az ad sp create-for-rbac --name <service_principal_name> --role Contributor --scopes /subscriptions/<subscription_id>
# Equivalent to:
az ad sp create-for-rbac --name myAKSClusterServicePrincipal
az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --service-principal <appId> \
    --client-secret <password>
az role assignment create --assignee <appId> --scope <resourceScope> --role Contributor

# Steps:
# 1. Create a principal identity
# 2. Create a resource group and assign the principal identity (normally done via Terraform)
# 3. Assign a role to the resource group by scope
```

5. create image pull secret

```bash
kubectl create secret docker-registry <named_secret> \
    --namespace default \
    --docker-server=acrdemo.azurecr.io \
    --docker-username=<$SP_APP_ID> \
    --docker-password=<$SP_PASSWD>
```

```powershell
kubectl create secret docker-registry my-secret --namespace default --docker-server=$ACR_NAME[.azurecr.io](http://demopublicpetlover.azurecr.io/) --docker-username=$SP_APP_ID --docker-password=$SP_PASSWD
```

Note: `kubectl delete secret my-secret` deletes the created secret.

### SSH Connection to Remote Server

- Establishing a connection:
  1. Send your public key to the target node
  2. Exchange public keys and confirm mutual trust
  3. You can then log in



### DevOps K8s Overview 1

- Kubernetes:
  1. Manages containerized workloads and services.
  2. A Kubernetes deployment gives you a cluster.
- Service: an abstraction that exposes an application and provides a cluster IP, a DNS name, and a port for a set of pods.
- Pods:
  1. The basic components of an application workload.
  2. The smallest deployable units of computing that Kubernetes can create and manage.
  3. Groups of containers that share namespaces, filesystem volumes, and network resources.
  4. Containers within the same logical host are tightly coupled. Outside a cloud environment, the logical host is usually the same physical machine or VM.
- Node: a VM or physical machine that hosts pods for the application workload.
- Control plane: in production, the control plane usually runs across multiple machines.
- Namespace: a logical boundary for resource names, commonly used for environments such as `dev`, `uat`, and `prod`.
- Target port: the port on a pod that a service maps incoming traffic to.

## DevOps K8s Overview 2

- Ingress controller: users access the ingress controller via domain name, which then routes to different services (music, streaming, shopping, etc.)
- Managed Kubernetes options:
  1. Azure K8s service
  2. GCP Kubernetes service
  3. AWS EKS
- Control plane: Azure Kubernetes Service cluster management, including auto-scaling.
- Cosmos DB: Azure NoSQL database.
- Grafana: visualizes logs and metrics from Azure data sources; it may pull log data from NoSQL storage depending on the architecture.
- Relationships:
  1. One Kubernetes cluster has many nodes or VMs for fault tolerance and high availability.
  2. One service can target many pods.
  3. One node has at least one pod; worker nodes host the pods.
  4. One pod has at least one container.

### DevOps K8s Deep 1

![Untitled](Untitled%2054.png)

![IMG_0408.PNG](IMG_0408.png)

![IMG_0409.PNG](IMG_0409.png)

K8s is designed to be more powerful, more flexible, and more robust; refer to earlier posts for basic info.

Docker on a single EC2 instance creates a single point of failure.

Control plane == orchestrator.

Kubernetes provides:

1. Service discovery and load balancing: Kubernetes can discover services and expose containers through DNS.
2. Storage orchestration across local and cloud storage.
3. Automated rollouts and rollbacks for code and configuration across multiple containers.
4. Automatic bin packing: Kubernetes places workloads efficiently, scales up or down, and can enforce custom resource allocation. A common target is keeping node CPU below about 60%.
5. Self-healing: Kubernetes detects health problems, replaces broken instances, and waits for new instances to become healthy before routing traffic.
6. Secret and configuration management for environment variables, passwords, and centralized configuration.

Control panel:

1. Decides which pods run on which nodes, maximizing resource utilization.
2. Creates pods.
3. Can run on one node in a small setup, but should be highly available in production.
4. Kube-apiserver: exposes the Kubernetes API and is used for service communication.
5. etcd: stores secrets & configuration
6. kube-scheduler: assign new pod to available node, better space management
7. Kube-controller-manager: logic handler
8. Cloud-controller-manager: integrates with cloud provider APIs at the cluster level

### DevOps K8s Deep 2

Node:

1. == EC2
2. A node is managed by the control plane.
3. It does the actual work and is also called a worker node.
4. Each node runs at least one pod.
5. at least one kubelet: a middleman between the brain (control plane) and the worker
6. at least one container runtime: responsible for running containers (anything that implements the container runtime interface works)
7. Kube-proxy: network proxy, responsible for managing IP communication between pods
8. Docker here is responsible for managing containers within each pod

K8s cluster:

1. at least one worker node
2. Cluster services run across multiple servers and control multiple nodes.

Swarm: creates a swarm using the Docker CLI, think of it as a lightweight, Docker-native Kubernetes

1. swarm == Kubernetes cluster

- Kind: a tool for running local Kubernetes clusters using docker container "nodes". Primarily used for testing Kubernetes itself
- namespace == isolated logical project
- ingress controller == load balancer

Swarm troubleshooting tips:

1. Version, version, version! Try to match your instructor's version as closely as possible, otherwise there will be many pitfalls
2. If using Kubernetes 1.21, remember to use apiVersion: batch/v1beta1 in demojob; otherwise use apiVersion: batch/v1 (I succeeded by upgrading to the latest Kubernetes 1.25)
3. Similarly, for kubectl -n create token admin-user, if using an older version, the command will not exist and you cannot directly generate an admin user token

### Kubernetes basic components 1

![Untitled](<k8s_eks%20(5).png>)

![Untitled](<k8s_eks%20(6).png>)

![Untitled](<k8s_eks%20(7).png>)

![Untitled](<k8s_eks%20(8).png>)

- Kubernetes benefits: high availability, high scalability, and disaster recovery.
- Pods
  1. abstraction over container
  2. each pod gets its own private IP address
  3. smallest unit
- service:
  1. provide the permanent IP address
  2. lifecycle of pod and service not connected; so when one fails, another one can connect directly quickly
  3. load balancer; send traffic to more pods
- internal service vs external service(public ip address with port in the node level): private IP vs public IP
- Ingress: a proxy(provides Public IP address) for external service → connect to service
- ConfigMap: external configuration of your application (the Pod actually gets the data that ConfigMap contains) such as url link (ENV)
- secret: used to store secret data without a plain text (ENV)
- Volumes: preserve data because Kubernetes does not manage application data persistence by itself.
- deployment:
  1. A blueprint for application pods, including how many replicas should run and how they scale up or down.
  2. is a layer of abstraction on top of pods
- deployment for stateless Apps vs StatefulSet for stateful apps or DB

### K8s Basic components 2

![Untitled](<k8s_eks%20(9).png>)

![Untitled](<k8s_eks%20(10).png>)

- one node(slave nodes) must have: container runtime, kubelet, Kube proxy
  1. kubelet: interacts with both container and node; starting a pod with a container inside and assign the resource from node to the container (CPU, RAM)
  2. Kube Proxy: forwarding requests from services to pods (avoid network overhead of sending the request to another machine)
- Master nodes(control plane):
  1. API server: like a cluster gateway which gets the initial request of any UPDATE or QUERY; acts as a gatekeeper for authentication.
  2. scheduler: schedule new pod → api server → scheduler → where to put the pod (like packing luggage, stuffing bags of different sizes into different containers); scheduler just decides on which Node new Pod should be scheduled. (the actual execution is done by kubelet on each node)
  3. controller management: detect any pods die in any nodes and reschedule those pods as soon as possible; (detects cluster state changes); controller manager → scheduler → kubelet; ensures proper STATE of cluster components.
  4. etcd: key-value store of a cluster state; cluster brain! dedicated to recording state; this storage cross all the master replica nodes.
     A typical small cluster has 2 masters and 3 workers;
     masters require less compute resources, workers require more;
     masters and workers can be scaled infinitely based on demand
- Worker nodes:
  1. container runtime engine: Docker, containerd, CRI-O, frakti…
  2. kubelet: it is an agent that runs on each node in the cluster — ensures each instance is running correctly and reports back to master; scheduler also uses this to create pods
  3. kube-proxy(network proxy): allows each node communicate with each other (allow network communications)

### K8s Services

![Untitled](<k8s_eks%20(11).png>)

![Untitled](<k8s_eks%20(12).png>)

![Untitled](<k8s_eks%20(13).png>)

![Untitled](<k8s_eks%20(14).png>)

- A service links to pods through selectors.
- Kubernetes creates an Endpoint object with the same name as the service and tracks which pods are members of that service.
- Headless service: only communicate with 1 specific Pod directly (ClusterIP: None)
- load balancer service is an extension of the NodePort Service
- NodePort Service is an extension of ClusterIP service

### K8s Common Imperative Commands

1. kubectl edit pod redis: edit the YAML configuration of an already-created pod, which in turn modifies the image used to build the pod
2. kubectl run <tagName, your own name> --image=<docker image name> e.g. kubectl run redis --image=redis meaning: create a new pod
3. kubectl delete pods <tagName> — delete a specific pod
4. kubectl get pods -o wide — get a list of pods with extended output (wide)
5. kubectl describe pod <tagName>
6. kubectl get pods
7. kubectl run redis --images=redis123 --dry-run=client -o YAML > redis.yaml — output configuration to YAML file
8. kubectl apply -f <filename> — apply a file to create pods
9. kubectl create -f <filename> — create from file
10. kubectl get replicaset: get command to see a list of replica sets created
11. kubectl delete replicaset <tagName>
12. kubectl replace -f replicaset <fileName> — replace configuration
13. kubectl scale --replicas=6 -f <fileName>
14. kubectl edit replicaset myapp-replicaset — edit replicaset settings
15. kubectl scale rs new-replica-set --replicas=2 — rs is short for replicaset
16. kubectl get all — get full details including deployments, replicas and pods.

Deployment strategy: create, get, update, status, rollback

![Untitled](<k8s_eks%20(15).png>)

1. kubectl rollout undo deployment/myapp-deployment — rollback deployed code

-- record=true will record the command execution history

1. kubectl set image deployment myapp-deployment Nginx=Nginx:1.18 — update image

Service

1. NodePort: allows external access to internal services, split into target port, port, and nodeport
2.

![Untitled](<k8s_eks%20(16).png>)

kubectl get pods,svc — view both running services

debugging:

kubectl logs {pod-name}

kubectl exec -it <pod name> --bin/bash

kubectl get deployment nginx-deployment -o YAML > nginx-deployment-result.yaml — copy configuration info to YAML file

'kubectl delete --all deployments --namespace=foo' — delete all deployments, services, etc.

When starting a service with type=loadbalancer, use minikube service mongo-express-service to expose the external IP address

kubectl create namespace uat-petlover

### Volume

![Untitled](<k8s_eks%20(17).png>)

![Untitled](<k8s_eks%20(18).png>)

![Untitled](<k8s_eks%20(19).png>)

- PersistentVolume (PV): not tied to any namespace; it is available to the whole cluster.
- PersistentVolumeClaim (PVC): must exist in the same namespace as the pod that uses it.
- StorageClass (SC): provisions persistent volumes dynamically when a PVC requests storage.
- ConfigMaps and Secrets are local volume sources managed by Kubernetes; they are not created through PVs or PVCs.

Usage flow:

1. Old approach: admins configure storage (Manually) => create PV => developer claim PV using PVC
2. New approach: developer claim PV using PVC => Storage class SC allocate resources (PV) to developer

![Untitled](<k8s_eks%20(20).png>)

![Untitled](<k8s_eks%20(21).png>)

### Helm - Package Manager of K8s

![Untitled](<k8s_eks%20(22).png>)

![Untitled](<k8s_eks%20(23).png>)

![Untitled](<k8s_eks%20(24).png>)

![Untitled](<k8s_eks%20(25).png>)

- Helm Charts:
  1. bundle of Yaml files
  2. create your own Helm charts with Helm
  3. push them to Helm repository
  4. Use charts created by others.
  5. Helm hub, Helm repository, Helm pages
- templating engine
  1. define a common blueprint
  2. dynamic values are replaced by placeholders
- dev - staging - prod
- release management: state management in tiller server (V2, V3 now tiller no longer exists)
- Helm command:
  1. Add a Helm repository locally: helm repo add bitnami https://charts.bitnami.com/bitnami
  2. Search for needed application from your added repository: helm search repo Jenkins
  3. Pull down a Helm chart: helm pull bitnami/<image_name> --untar=true
  4. Install a Helm chart: helm install <give-a-name> bitnami/nginx
  5. Delete a Helm chart's deployed service from the node: helm delete <give-a-name> (note: at this point, the service inside the pod will also be deleted)

### Namespaces in system

![Untitled](<k8s_eks%20(1).png>)

- Arch knowledge:
  1. deployment manage a replica set
  2. replica set manage all replica of its pod
  3. the pod is an abstraction of a container
- Kubernetes namespaces
  1. a virtual cluster inside a cluster
  2. kubernetes-dashboard(namespace) only with minikube
  3. kube-system: system processes; master and kubectl processes; do not create or modify in kube-system
  4. kube-public: publicly accessible data; a configmap, which contains cluster information
  5. kube-node-lease: heartbeats of nodes; each node has associated lease object in namespace; determines the availability of a node
  6. default: resources you create are located here
  7. kubectl create namespace my-namespace
  8. create a namespace with a configuration file
     ![Untitled](<k8s_eks%20(2).png>)
- When should you use multiple namespaces?
  1. To group resources such as databases, monitoring, logging, Elastic Stack, and Nginx-ingress.
  2. To let multiple teams work on the same application without affecting each other.
  3. resource sharing: staging and development
     ![Untitled](<k8s_eks%20(3).png>)
  4. blue/green deployment (reuse those components in both envs)
  5. To enforce resource limits in constrained environments. Each namespace should define its own ConfigMaps; services can communicate across namespaces.
     ![Untitled](<k8s_eks%20(4).png>)
  6. Volumes and nodes are not namespaced; they are shared cluster resources.
  7. Add `--namespace=my-namespace` to isolate commands to a namespace.
  8. Switch namespace:
     1. kubectl config set-context $(kubectl config current-context) --namespace=test-ns — switch to test-ns
     2. kubectl config set-context $(kubectl config current-context) --namespace=default — switch back to default

1. kube-node-lease: improves the performance of our node heartbeats (benefit for cluster autoscaling)
2. kube-public: readable for cluster
3. kube-system:
   1. kube-dns
   2. kube-proxy
   3. ebs-csi-node

### Probes

- liveness Probe
  1. kubelet uses liveness probes to know when to restart a container
  2. liveness probes could catch a deadlock, restart container
     "
     livenessProbe:
     exec:
     command: - /bin/sh - -c - nc -z localhost 8095
     initialDelaySeconds: 60
     periodSeconds: 10
     "
     'nc -z' will check if someone is listening on the other side
- readiness probe
  1. to know when a container is ready to accept traffic
  2. When the container is not ready, remove it from load balancers based on the readiness probe signal.
- startup probe:
  1. when a container application has started
  2. it will disable liveness & readiness checks first to avoid restarting container repeatedly

### OIDC and IRSA with IAM identity federation

- To enable and use AWS IAM roles for Kubernetes service accounts on our EKS cluster, we must create & associate OIDC identity provider. Allow it to make calls to AWS APIs on your behalf.
  1. IRSA: IAM role service account — can associate an IAM role with a Kubernetes service account, so that service account can provide AWS permissions to any pods using that service account
  - To create and associate an OIDC identity provider with our EKS cluster, we need to follow these steps:
    1. Create an IAM OIDC provider in the IAM console.
    2. Associate the IAM OIDC provider with our EKS cluster.
    3. Create an IAM policy that allows our Kubernetes service account to assume the IAM role.
    4. Create a Kubernetes service account that uses the IAM role.
    5. Verify that the Kubernetes service account can assume the IAM role by running a test pod.
  2. OpenID connect provider: EKS acts as a provider, you can use the AWS EKS OpenID Connect provider to access AWS services using the IAM identity federation
     ![Untitled](<k8s_eks%20(53).png>)
     [https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)



### DevOps CDN&DNS

![Untitled](Untitled%208.png)

![3693b95f250de356d9af95d8789e589.png](3693b95f250de356d9af95d8789e589.png)

![Untitled](Untitled%209.png)

CDN(Content Delivery Network) == AWS CloudFront:

1. low latency: distributed content to a globally-distributed network of proxy servers/locations local server by caching, such as S3 static resources, which copies content from the original server.
2. high availability:
   - avoid single point of failure: if one server node failed, all the services will be down.
   - avoid DDoS(Distributed Denial of Service) Attack
3. The application needs to CDNify via file versioning. Otherwise, it needs time to update (data propagation delay)

so it can reduce bandwidth consumption, secure the original server

DNS(Domain Name System) == AWS Route 53:

The **Domain Name System(DNS)** is the hierarchical and decentralized naming system used to identify computers reachable through the Internet or other Internet Protocol networks.

(Multi-level queries, ultimately returning the IP address of the target server)

DNS Host: acts like a local phone book. When we access a new domain name, the computer first looks up the corresponding IP address in its own hosts file. If not found, it uses DNS to resolve the domain name to an IP address.

ps: When attempting to connect via HTTP without a certificate, it will throw an error (as shown in figure 4) — only HTTPS is supported.

### Common Interview Questions

![structure.png](structure.png)

TLD (Top-Level Domain)

SLD (Secondary-Level Domain)

Classic interview question: What happens after you type a domain name and press Enter?

1. The browser checks its own DNS cache.
2. If not found, check the local client (local LAN/zone) hosts file for the corresponding IP address.
3. If not found, query the DNS server provided by the ISP (root name server) for a cached DNS entry.
4. If not found, query the root server recursively (starting from the TLD, e.g. .edu => washington => ... narrowing down step by step until the client obtains the IP address).
5. Once the IP address is obtained, make an HTTP/HTTPS request; the client performs a three-way handshake with the server.
6. The HTTP request packet contains a header and a body. The header includes metadata about the requested file operation; the body contains the payload/data.
7. After the server receives the HTTP request, it performs load balancing and other backend processing.
8. Server returns header + response.
9. Client (browser) renders the data.

### DevOps DNS Record

![Untitled](Untitled%2010.png)

### DNS Record Types

- **A**: Points a domain name to an IPv4 address (e.g. 11.11.11.11).
- **CNAME**: Points a domain name to another domain name, producing the same access result as the target domain. The target is usually a domain provided by a hosting provider.
- **MX**: Mail exchange record — points to a mail server address for email delivery.
- **NS**: Name server record — specifies which DNS server is responsible for resolving a subdomain.
- **TXT**: Optional. Typically used for verification purposes or security configurations, such as SPF (anti-spam) records.
- **AAAA**: Points a hostname (or domain name) to an IPv6 address (e.g. ff03:0:0:0:0:0:0:c1).
- **SRV**: Service record — specifies which server provides which service. Format: service-name.protocol-type (e.g. \_example-server.\_tcp).
- **SOA**: Start of Authority record. While NS records identify multiple name servers, the SOA record indicates which one is the primary server.
- **PTR**: The reverse of an A record — also called an IP reverse lookup record or pointer record. Resolves an IP address back to a domain name.
- **Explicit URL redirect**: Points a domain to an HTTP(S) address; visitors are automatically redirected and the browser address bar shows the target URL.
- **Implicit URL redirect**: Points a domain to an HTTP(S) address; visitors are automatically redirected but the browser address bar continues to show the original domain.

Zone: A Zone is a collection of nodes (subdomains) under the main domain. The server maintains a database called a zone file for every zone.

CDN reference: [https://cloud.tencent.com/developer/article/1537894](https://cloud.tencent.com/developer/article/1537894)

AWS documentation: [https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ResourceRecordTypes.html](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/ResourceRecordTypes.html)

Route 53 Alias: [https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html)


CDN&DNS


### DevOps Linux System 01

- Azure's resource group is equivalent to GCP's project: a container for project-related resources (based on business logic)
- RedHat vs CentOS vs Debian vs Ubuntu
  1. Ubuntu: comes with a GUI
  2. Debian: more stable than Ubuntu, smaller size, rock solid as a base OS
  3. RedHat: the gold enterprise edition, quite expensive
  4. CentOS: the open-source counterpart of RedHat
- apt acts as an index connecting to different version repositories; `apt update` refreshes the package index
- `~/.jenkins` can archive user files separately; JENKINS_HOME defaults to `/var/lib/jenkins`
- keep the same number of CPUs as executors (total number of concurrent jobs)
- SCM: source code management — pulls code from version control

### DevOps Linux file struct 02

![d0c50-linux2bfile2bsystem2bhierarchy.jpg](d0c50-linux2bfile2bsystem2bhierarchy.jpg)

- **/bin, /sbin, /usr/bin, /usr/sbin**: The default directories for system executables — for example, `ls` lives at `/bin/ls`.
- **/bin** and **/usr/bin** are for general users (non-root), while `/sbin` and `/usr/sbin` are for root.
- **/var**: Stores logs and other data that programs generate at runtime. Log files go under `/var/log`; default mail storage is also here.
- `/var` contains data that changes during normal system operation. Its subdirectories grow and shrink frequently. Content that used to live in `/usr` was moved here to keep `/usr` relatively stable. This directory is system-specific and not shared over the network.
- `/etc` contains system configuration files, including many network configuration files.

### DevOps Linux sudo vs su 03

![95e72e18d4c703f711599198dea084c0.webp](95e72e18d4c703f711599198dea084c0.webp)

- Difference between `su` and `su -`: `su` switches the user identity to root but keeps the current shell environment (the original user's PATH); `su -` switches both the identity and the shell environment to root's. Only `su -` avoids PATH errors. After `su`, `pwd` still shows the original user's working directory; after `su -`, it switches to root's home directory.
- sudo vs su: `sudo` = super user do; `su` = switch user
  1. `su` switches to root by default
  2. `su - -c` temporarily borrows root privileges to run a single command, then returns to the current user
  3. `sudo su -` == `sudo -i`: switches to root as a login shell. `sudo su -` requires the current user's password; `su -` requires the root password.
  4. Whether a user can use `sudo` depends on whether root has granted authorization in `/etc/sudoers`. Only root can edit that file with `visudo`.
- `[root@localhost ~]` means:
  1. Username: root
  2. Hostname: localhost
  3. Current path: `~` (the current user's home directory)
  4. Privilege indicator: `#` for root, `$` for regular users
- Difference between `~` and `/` in `[root@localhost ~]`:
  1. `~` is root's home directory (equivalent to `C:\Users\Administrator` on Windows); `/` is the Linux root partition (equivalent to `C:\`).
  2. Under `/` there are `/root` (root's home) and `/home` (home directories for other users, typically `/home/username`)

### DevOps Linux Cron Job

![Untitled](Untitled%2038.png)

![Untitled](Untitled%2039.png)

- `set -x` enables verbose logging (prints each command before executing); `set +x` disables it. It's an alternative to scattering `echo` statements throughout shell scripts.
- `echo $?`: checks whether the last command succeeded. Returns 0 on success, non-zero on failure.
- Searching files and content
  1. `grep "search_string" filename` — **find lines in a file matching a string**
  2. `grep -e "regex" filename` — find lines matching a regular expression
  3. `grep -i "search_string" filename` — case-insensitive search
  4. `find / -name "*.log" | xargs grep "ERROR"` — find all `.log` files from the root directory and search them for "ERROR"
- Pipes
  1. `|`: combine multiple commands when one cannot complete the task alone
  2. `| xargs`: use when the piped output is a list of filenames
- Scheduled tasks (cron)
  1. `crontab -e` — edit scheduled tasks
  2. `crontab -l` — list scheduled tasks
- Copy a file from local machine to a remote server (securely)

  1. `scp -i ~/.ssh/key.pem <local_file> ec2XX@<public_IP>:<remote_path>`

     e.g. `scp -i ~/.ssh/my-key.pem newfile ec2@53.33.33.33:/home/ec2-user/newfile`
     To copy from remote to local, reverse the arguments. FileZilla can also be used for transfers.

- Set file owner: `sudo chown <username>:<group> <fileName>`
- Watch a live log file: `tail -f filename` (`-f` follows the file as it grows)
- PID: process identification number — a unique ID automatically assigned to each process when it is created by the OS
- An executable file launched in the OS creates a running copy of the application called a process
- `less` is a handy command for reading file contents page by page

#AustraliaIT #DevOps #JobHunting #Programming

[https://crontab.guru/](https://crontab.guru/)

### DevOps Regex Cheatsheet

Regular expressions are one of those things you learn and immediately forget — here's a quick reference:

- `^`: matches the start of a string
- `$`: matches the end of a string
- `*`: 0 or more occurrences
- `+`: 1 or more occurrences
- `?`: 0 or 1 occurrence
- `x{n}`: exactly n occurrences of x
- `x{n,}`: n or more occurrences of x
- `x{n,m}`: between n and m occurrences of x
- `()`: treat as a single group
- `[]`: character set — matches any one character inside
- `\d`: a single digit
- `\w`: a single word character (letter, number, or underscore)
- `\s`: a single whitespace character (space, tab, or line break)
- `\1`: back-reference to the content captured by the first group, e.g. `/(x)(y)\1/` matches `(x)(y)(x)`

`\D`, `\W`, `\S` are the negations of the above (non-digit, non-word, non-whitespace)

- Flags (placed after the closing `/` of a pattern)
  1. `g`: global search — returns all matches, not just the first
  2. `m`: multiline matching
  3. `i`: case-insensitive matching

### DevOps Mac && Linux Environment Variables

![Untitled](Untitled%2047.png)

![Untitled](Untitled%2048.png)

- `sh` generally refers to the Linux shell, but on Mac `sh` behaves differently (generally considered more compatible)
- `bash` is an extension of `sh`; on Mac both `sh` and `bash` refer to bash
- Environment configuration load order:
  1. `/etc/profile` — loaded first, applies to all users, lowest priority (not recommended to modify)
  2. `/etc/paths` — global environment variable configuration file
  3. `~/.bash_profile` → `~/.bash_login` → `~/.profile` → `~/.bashrc` — loaded after system files; if `bash_profile` is found, the others are skipped. Order may differ on Linux.
  4. `~/.zshrc` — used when oh-my-zsh is installed; the system reads `~/.zshrc` on zsh startup and skips `~/.bash_profile`. Add `source ~/.bash_profile` at the end of `.zshrc` to also load bash configuration.
- `source ~/.bash_profile` — reload the configuration file to apply environment variable changes immediately
- `export ZSH=$HOME/.oh-my-zsh` — export an environment variable; view it with `echo $ZSH`
- `PATH=$PATH:$HOME/bin` — append to the existing PATH using `:` as a separator; the shell searches each directory in order when looking for executables
- Environment variables set directly in the terminal (without adding to a configuration file) only apply to the current session and are lost on restart
- `alias ll='ls -lt'` — define a command shortcut; run `alias` alone to list all defined aliases
- `env` — display all environment variables
- `echo $PATH` — display the value of a specific environment variable

- Linux command chaining operators
  1. `&` — run a task in the background
  2. `&&` — run the next command only if the previous one succeeded
  3. `|` — pipe the output of one command as the input to the next
  4. `||` — run the next command only if the previous one failed
- Linux IO streams: `stdin`, `stdout`, `stderr` — equivalent to Python's `input()`, `print()`, and exceptions
- Trim whitespace from both ends: `sed -e 's/[[:space:]]*$//' | sed -e 's/^[[:space:]]*//'`


AWS Identity and Access Management(IAM), Elastic Compute Cloud(EC2), S3, VPC, NAT Gateway, NAT Internet

Linux


### Jenkins VM Server Installation Guide (to be automated with Ansible later)

- Install Jenkins on a VM
  1. Update apt
  2. Install Java environment
  3. Import the GPG key for the Jenkins repository
  4. Add the Jenkins repository to the system
  5. Once the Jenkins repository is added and enabled, update the package list and install the latest version of Jenkins
  6. apt install Jenkins: `sudo apt install jenkins -y`
  7. Check Jenkins status: `systemctl status jenkins`
  8. Open port 8080 (on GCP this is more involved — you need to create a new firewall rule in GCP Firewalls, then attach it to the GCP VM via a network tag)
  9. Copy the key from the VM: `sudo cat /var/lib/jenkins/secrets/initialAdminPassword`
  10. `sudo -i` => switch to root => `gpasswd -a jenkins root` (add the Jenkins account to the root group)
  11. `groups jenkins` should show root
  12. `visudo` => add `jenkins ALL=NOPASSWD:ALL` so Jenkins can run system-level commands (e.g. Docker) without entering a password
  13. `sudo usermod -aG docker $USER` — fixes "dial unix /var/run/docker.sock: connect: permission denied" when Jenkins has no permission to run Docker (adds Jenkins to the Docker group)
      On Linux, the default home directory is /home/user, but Jenkins' home directory is /var/lib/jenkins, so any commands Jenkins needs to run must be installed under that path — for example, GitHub SSH keys should go there.
  14. After switching to the Jenkins user, run `ssh-keygen` to generate a GitHub access key
  - For master-node coordination:
  1. `ssh-keyscan -H <ip-address> >> /var/lib/jenkins/.ssh/known_hosts` — skips the yes/no prompt, optional — adds the host record to known_hosts
  2. Give the public key to master, place it in authorized_keys: `echo "content" >> ~/.ssh/authorized_keys`

### Jenkins Credential Types

- Secret text — tokens such as API tokens (e.g. GitHub personal access tokens)
- Username and password — can be stored as separate fields or as a colon-separated string: username:password
- Secret file — encrypted content stored in a file
- SSH Username with private key — SSH public/private key pair
- Certificate — a PKCS#12 certificate file and optional password
- Docker Host Certificate Authentication credentials

### Jenkins Plugins Required

- AnsiColor: colorizes Console Output
- Blue Ocean
- pipeline: AWS steps
- **CloudBees AWS Credentials: stores access key and secret key**
- CloudBees Docker AWS Credentials
- Role-Based Authorization Strategy
- Email Extension Plugin
- Docker
- Docker pipeline
- Bitbucket Plugin
- Slack Notification Plugin
- Azure Credentials plugin

### DevOps Jenkins 01

![IMG_0394.PNG](IMG_0394.png)

![IMG_0395.PNG](IMG_0395.png)

- Jenkins
  - An open-source application that enables CI/CD, integrating with a wide range of testing and deployment technologies.
  - Improves efficiency: CD with a web project => web farm (web server farm), with deployment status easily tracked from the dashboard.
- CI
  - Automatically verifies the build using unit tests and packages the solution for each new code change submitted.
  - Compile == Build == CI (source code/text files => binary) => .exe, lib, pkg
- CD
  - Takes executable files (validated code packages) from the build process and deploys them into staging or production environments.
  - Developers can track which deployments succeeded or failed and narrow down issues to specific package versions.
  - Two server sets alternate between staging and production (see diagram) — blue/green deployment.
    [https://blog.inedo.com/blue-green-deployment-best-practices](https://blog.inedo.com/blue-green-deployment-best-practices)
    Jenkins can also be used to orchestrate other cloud services such as AWS CodeBuild/CodeDeploy — a useful complement to existing infrastructure.
    Note: M1 Macs use the ARM architecture (RISC), so running Docker likely goes through Rosetta2 translation. Running Jenkins inside Docker on M1 feels extremely slow — roughly 10x slower than on Windows.

### DevOps Jenkins & Intranet Tunneling 02

![Untitled](Untitled%2022.png)

- Install Jenkins locally
  - On Windows, run `sudo service docker start` to start the Ubuntu Docker engine (can connect Docker engine to Ubuntu as shown in diagram 2)
  - Run Jenkins container
  - Get admin password to complete Jenkins setup
- ngrok (reverse proxy / intranet tunneling)
  - Visit ngrok website to get an auth token
  - Local installation:
    - Windows: change permissions → extract → configure token → open port 80
    - macOS: use `brew install --cask ngrok` (no manual PATH configuration needed), rest is the same as Windows
    - Copy the tunnel URL to Jenkins configuration => Jenkins URL
- Connect GitHub with Jenkins
  - Add new item
  - Freestyle project
  - Configure with Git + Repo URL with .git connection
  - Build triggers => GitHub hook trigger for GITScm polling
  - Branches to build: choose `*/master` or leave empty to trigger on all branches
  - Add webhook in GitHub => go to git repository
  - Payload URL must end with `github-webhook/` e.g. `http://XXXX.ngrok.io/github-webhook/`
  - Select "Just the push event" as the trigger
  - Check "Active" to enable the webhook

Windows/Ubuntu users need to prepend `sudo` to most commands.

Installing ngrok on macOS:

[https://gist.github.com/wosephjeber/aa174fb851dfe87e644e?permalink_comment_id=3621858](https://gist.github.com/wosephjeber/aa174fb851dfe87e644e?permalink_comment_id=3621858)

### DevOps Azure Docker Jenkins 1

- Create VM
  1. Check connection settings in the server
  2. Open ports 22, 80, and 443
  3. Allocate 64GB SSD; check "delete on termination" for both disk and public IP
  4. Open port 8080 for demo
- SSH into server => install Jenkins via Docker
  1. `sudo apt update`
  2. `sudo apt install docker-compose`
  3. `sudo docker run XXX`
- NS delegation from AWS to Azure (subdomain)
  1. `nslookup -type=NS shawnwong.click` — check DNS name servers
  2. Add the domain name in Azure DNS (e.g. sure.XXX.XXX)
  3. Copy NS records from Azure DNS to the domain registrar (NS delegation)
  4. `nslookup -type=NS nz.shawnwong.click` — verify DNS name servers
  5. Add a new record set in Azure DNS with type A and alias
- Install Nginx on Azure VM
  1. `apt install nginx`
  2. `systemctl enable nginx` (enable auto-start on boot)
- Add SSL
  1. Modify Nginx configuration files
  2. Restart Nginx: `sudo systemctl restart nginx`
  3. `systemctl status nginx` — check service status

Gotchas:

- WSL Ubuntu cannot modify file permissions for Windows files
  1. Copy the file into the Linux directory using `cp`
  2. Directly modify file permissions via `/etc/wsl.conf`

### DevOps with Private GitHub 2 & Nginx Config

![Untitled](Untitled%2027.png)

![Untitled](Untitled%2028.png)

![Untitled](Untitled%2029.png)

![Untitled](Untitled%2030.png)

Configure DNS for a specific domain (from AWS to Azure):

- Create in Azure DNS; Name can be set to `XXX.<yourdomainname>`
- Copy NS records to AWS Route53 as delegation
- Create a record set => use an alias record set to point directly to a VM resource
- Or use an A record pointing to the public IP address

Install Nginx & add certificate to Jenkins server:

- `sudo apt install nginx`
- `sudo systemctl enable nginx` — enable auto-start on boot
- Add certificate
- `sudo systemctl status nginx` — check service status
- `sudo systemctl restart nginx` — restart Nginx

Narrow way to add SSH public key:

- Add SSH key in specific GitHub repository => Settings => Deploy Keys to restrict access to that repository only

Troubleshooting guide:

1. If Jenkins is running inside Docker, add the `.ssh` key inside the container (use `sudo docker exec -it <cont_id> sh` to enter the container, then run `ssh-keygen`)
2. No need to add credentials in Jenkins dashboard => pipeline => Source Code Management; set credentials to "None"
3. Restart container with: `sudo docker restart <cont_id>`
4. When modifying Nginx configuration, remember to comment out the 404 redirect

### DevOps Jenkins Blue Ocean

![Untitled](Untitled%2031.png)

- Mainly covers CI and CD (content delivery) as a multibranch pipeline; does not cover content deployment
- Content deployment involves production data (user data, databases, live workloads)
- `git push origin <new-branch>` or `git push origin HEAD` — push a new branch to GitHub
- `git pull . master` — pull files from master into the current branch (syncs locally)
- Difference between a Plugin and Docker Engine?
  - One is a tool that uses Docker; the other is Docker itself — Arno
- Blue Ocean workflow:
  1. Connect GitHub => GitHub Settings => Developer Settings => Personal access tokens => grant permissions (user, write, etc.) => create a pipeline
  2. Install Docker Engine on Linux
  3. Run `which docker` => find the Docker Engine path, then add it to global configuration
  4. Configure Docker path in Jenkins and add the Docker Pipeline and Docker plugins
  5. Add `HOME '.'` in Jenkinsfile to run files in the current user's directory (`/bin/sh`) rather than the root directory

### DevOps Jenkins Master-Node Coordination

![Untitled](Untitled%2040.png)

AZ-GCP-Local Jenkins

- Start AZ, GCP VM + local machine
  1. Add Docker environment
  2. Add JDK 11+ environment
- Add local node: Manage Jenkins => Manage Nodes and Clouds => Add a new node
  1. Note: in Configure Global Security, add a fixed TCP port 5003
  2. Download the agent jar to the local machine via wget
  3. Open inbound port 5003 on the Azure VM
  4. Since this is local, the agent may go offline — configure it to start only when a job matching its label is triggered
  5. Choose "Launch agent by connecting it to the controller" (Azure VM proactively calls the local machine to execute tasks)
- Add GCP node
  1. `visudo` — used to edit `/etc/sudoers` to grant sudo permissions to a regular user without a password. Add: `jenkins ALL=NOPASSWD:ALL`
  2. Use this node as much as possible
  3. `/var/lib/jenkins` is equivalent to the Jenkins user's home directory, similar to `/root` or `/home/<username>`
  4. Generate an SSH key for the Jenkins user: `/var/lib/jenkins/.ssh/id_rsa`
  5. `ssh-keyscan -H 34.129.185.248 >> /var/lib/jenkins/.ssh/known_hosts` — add the GCP node host to the trusted list
  6. Add the `id_rsa.pub` key to GCP for passwordless access: `echo 'keyxxx' >> ~/.ssh/authorized_keys`
  7. Add the `id_rsa` private key to Jenkins credentials to complete authentication with the GCP VM

### Jenkinsfile Generator

![Untitled](Untitled%2041.png)

![Untitled](Untitled%2042.png)

![Untitled](Untitled%2043.png)

Jenkinsfile code can be generated using Jenkins' built-in pipeline code generator, which includes a Snippet Generator and a Declarative Directive Generator.

- `pipeline` — starts the code pipeline
- `agent`
  1. `any` — any available agent
  2. `label` — run on an agent matching a label
  3. `docker` — run inside a Docker container
  4. `dockerfile` — build from an image matching a label
- `stages` — pipeline entry point
- `post` — execute logic after all stages complete (useful for cleanup or notifications based on build outcome)
- `parameters`

Snippet Generator:

- `cleanWs()` — delete workspace when build is done
- `git` — git credentials
- `timeout + input`

Troubleshooting guide:

1. When using `withCredentials` from the Snippet Generator, select "Username and Password (separate)", link to your Jenkins credentials. The variable name you define (e.g. `dockerPwd`) will be referenced in the Jenkinsfile and mapped to the actual password at runtime — providing credential masking in the console output.
2. `credentialsId` is the name you gave the credential in Jenkins.
3. When `parameters` is configured, a "Build with Parameters" option appears in Jenkins, allowing the pipeline to be triggered with custom parameter values.


Jenkins CI/CD

Jenkins


![Untitled](Untitled%2072.png)

### DevOps Terraform Basic 1

- infrastructure as code?
  1.  Use code to provision hardware resources (Create, Update/Change, Destroy — CUD) (Terraform)
  2.  deploy/update applications on top of the infrastructure
  3.  manage the configurations used by the applications (Ansible)
- Terraform vs Ansible
  1. Terraform: responsible for the infrastructure
  2. Ansible: responsible for configuring that infrastructure
- Terraform: core component one → input sources
  1. core: TF-configuration — what to create/configure?
  2. state: snapshot of the current infrastructure state
     core compares both → plan: what needs to be CUD? Modifies the current state based on the desired configuration
- Terraform: core component two → main components
  1. providers: various clouds, Kubernetes, services
- declarative vs imperative?
  1. declarative: "give me a cheeseburger" 🤓 Terraform
  2. imperative: step-by-step instructions on how to do it (toast the bun, grill the patty, add cheese...)
- acl: access control list — manages permissions
- Uses HCL (Hashicorp Configuration Language)
- init → plan → apply → destroy

### DevOps Terraform 2

![1605496195438-image.png](1605496195438-image.png)

- VPC spans all AZs (subnets) in the region — one region has many VPCs, each AZ has a corresponding subnet
- subnet level control ⇒ NACL
- instance level control ⇒ security group
- Terraform
  1. source: create new cloud resource
  2. data: get known resources already from the cloud
  3. provider == import library
  4. resource/data == function
  5. tags = {Name: XXX} — note: only when Name is capitalized does it modify the VPC's Name attribute
  6. terraform destroy -target aws_subnet.dev-subnet-2 — delete a specific resource
  7. terraform apply -auto-approve — automatically approve changes
- Terraform state
  1. list: list all resources in the current state
  2. show <resource>
- Terraform variables:
  1. as parameters
  2. input via command line
  3. terraform.tfvars file to record variables
  4. export TF_VAR_<variable_name>="Oldiron666"
- Terraform multi-environment deployment?
  1. Create different Terraform.vars files, e.g. terraform-dev.tfvars
  2. terraform apply -var-file="terraform-dev.tfvars"
  3. Switch working directory/environment first: terraform workspace new uat
  4. terraform apply -var-file=uat.tfvars or terraform apply -var-file=.tfvars
  5. S3 will generate a .environment environment and automatically help organize multi-environment setups

### DevOps Terraform 3

- Do you know why Sydney has three sub-VPCs? One per AZ!
- NACL: firewall at the subnet level, open by default
- Security Group: restricts at the server level, closed by default
- route table: like roads, controls traffic within a private VPC
  1. local = within the VPC
- Both the internet gateway and AWS route table are virtual!
  1. Internet gateway == virtual modem
  2. route table == virtual router
- The procedure to connect the network for a VPC:
  1. vpc
  2. Internet gateway
  3. route table
  4. connect route table with the internet gateway
  5. connect route table with subnet (subnet association) — so traffic in the subnet is handled by the routing table
     Note: if no subnet association is specified, the default subnet is assigned automatically
- A note on SSH keys:
  1. With GitHub, you provide your public key to GitHub
  2. The key downloaded from AWS is a private key
  3. When connecting with your own SSH key, you first send the public key to AWS, then log in with your private key
     This is a classic asymmetric-then-symmetric encryption flow: public key for transmission, private key for decryption. SSH uses the private key by default — just run ssh <connectionString> and save time.
     Note: on first authentication you need to add the host to known_hosts — it will ask you to confirm with yes

### DevOps Terraform final

- Modules: like functions in programming — encapsulate code and improve reusability. A module should combine at least 3+ resources
  1. root module
  2. /modules = "child modules" called by another configuration
- input variables == function arguments
- output values == function return values
- external Terraform uses source to determine the module's origin (we use source to add path for our local module)
- configure a remote backend to safely store and share Terraform state via S3
  1. All state changes are uploaded directly to S3
  2. Terraform state list can connect remotely to S3 to view created and active resources

Then I did something dumb — I manually deleted the state file that had been uploaded to S3, which meant all the already-created resources had to be deleted manually...

- handle version and shared storage for state
  1. S3 for shared storage for state files
  2. DynamoDB for version locking
- Pitfalls:
  1. Terraform configuration is mainly set in the backend block — remember to run init after configuring it
  2. S3 and DynamoDB tables need to be created manually first, then referenced by name in the backend block — otherwise Terraform will try to find them from AWS and fail
  3. If you let Terraform auto-create S3 and DynamoDB, running destroy will wipe them too — a strange default behavior

Downside: Terraform still feels like an immature product, though the concept is solid. The cascading variable passing issue and the way state and lock files are stored both feel unreasonable.

### DevOps Terraform Frontend Pitfalls

- The frontend stack consists of S3, CDN, ACM, and Route53
  1. The tricky part of S3 is fetching the policy from aws_iam_policy_document — this policy only allows CDN to access S3's GetObject via OAI. It's fetched using data sources, and data source outputs cannot be returned from a module via outputs, so it must be written in the root main.tf.
  2. Route53: since my domain is in a different AWS account, I had to delegate it first, then fetch the zone ID via data source before proceeding. Route53 is also tightly coupled with CDN — the flow is S3 → CDN → Route53, so Route53 mainly pulls data from CDN using an alias. Additionally, Route53 creates a CNAME to validate the ACM certificate.
  3. CDN, while linked to Route53, also needs to validate the ACM certificate. In viewer_certificate, providing just the acm_certificate is not enough — per the docs, you must also set ssl_support_method (choose sni-only here), and using the default "TLSv1" protocol version lets you successfully select the certificate generated for your domain. This is a fairly deep pitfall.
  4. A key ACM pitfall is the creation region. To be linked with CloudFront, ACM must be created in Virginia (us-east-1). The entire frontend stack is largely region-agnostic since S3 data is distributed globally via CDN (though you can select coverage regions). If you need to create resources in multiple regions within a child module, you must declare a provider with a specific region and alias, then reference it in multi-region resources with provider = aws.<aliasName>. Note: the provider argument inside a resource block is undocumented — I discovered it works through trial and error.
  5. In the end, both my Route53 (acm_certificate_validation — yes, I put it in the Route53 module since Route53 is the one doing the validation) and ACM are created in Virginia.

### DevOps Terraform Multi-Environment + ECS Theory

- Terraform
  1. terraform fmt -recursive — format all files uniformly
  2. Deploy to different environments: terraform apply -var-file="<dev/prod>.tfvars"
  3. Manage multi-environment + multi-workspace ⇒ terraform workspace — create different workspaces for dev and prod
  4. provider "aws" { profile = "XXX" } — the profile matches credentials (access_key_id and secret_access_key) stored on your machine
  5. terraform init -backend-config="dev-backend.conf"
- ECS Fargate
  1. task definition: describes one or more containers through attributes, combining multiple container definitions
     1. Also includes container definition: container image and container-level settings (port, registry, environment variables)
  2. cluster: fully managed by AWS
  3. service: allows running and maintaining a specified number of simultaneous instances of a task definition in an ECS cluster — defines how many tasks to run
  4. task: instantiated from a task definition

### DevOps tf-backend Notes

![Untitled](Untitled%2076.png)

- Why does an Application Load Balancer need both security groups and listeners?
  My current theory: the load balancer is implemented using EC2, so it has the concept of SGs to control which ports allow traffic in. The server also runs software like Nginx to handle load balancing, which requires port configuration. Additionally, you typically set up both listeners and routing rules — receiving traffic and forwarding it to a destination. AWS abstracts the software configuration and gives customers a better UI to manage it.
- apt remove: removes the package but keeps its configuration files
- apt purge: removes both the package and its configuration files
- IAM role
  1. Similar to an IAM user, but unlike a user who is tied to one person, a role is intended to be assumed by a user, service, or application.
  2. A role does not have credentials such as a password or access key.
- IAM user: 1 user → many groups. 1 group → many users
- CloudFormation, Terraform, AWS CDK & Pulumi:
  1. CloudFormation and Terraform are both declarative — they just use YAML instead of HCL
  2. AWS CDK and Pulumi are both imperative — use any programming language to provision infrastructure
- Pitfalls:
  1. lb's enable_deletion_protection = true will block destroy on the LB — set it to false first in AWS for it to be auto-destroyed, though the tutorial recommends against allowing auto-destroy
  2. ecr_repo looks for images in private by default — it won't find public images
  3. ECR repository names must be all lowercase
  4. Keep ECR images in private whenever possible

### DevOps AWS Backend Notes

![1cf0c8732975a2d7c01779df01d7a97.jpg](1cf0c8732975a2d7c01779df01d7a97.jpg)

![41d57fdd2b81731fbeeb74d5cba6125.jpg](41d57fdd2b81731fbeeb74d5cba6125.jpg)

- Public knowledge:
  1. AWS CDK compiles down to CloudFormation
  2. VPC peering: allows resources in two subnets to communicate
  3. Least privilege model: only grant a service the ports it needs, even within the same private subnet — if one service is compromised, it can't access others
  4. each private subnet has one NAT
  5. a public subnet can share one NAT
  6. How does a security group demonstrate stateful behavior? If traffic can get in, it can get out. NACL is stateless.
  7. Why can the ELB target group port be set to anything? That port is the entry port from the ELB listener into the target group — it's the target group's ingress port, so any value works (just avoid reserved system ports)
  8. In Terraform, ECS containers can have assign_public_ip disabled — it's not useful anyway
  9. In Terraform SG ingress rules, from_port and to_port define a port range, e.g. 666–777. If both values are equal, only one port is opened.
  10. In SGs, generally only ingress ports are defined; egress is fully open, e.g. 1–65535.
  11. AWS is written in JavaScript — TypeScript is the best pairing (currently only studying Terraform, more later)
  12. What do metric_interval_upper_bound and lower_bound do in CloudWatch + Auto Scaling step scaling policy configuration? They work with CloudWatch to set upper and lower adjustment thresholds for multi-tier scaling. e.g. if CloudWatch alarm threshold is 50, and the metric delta is between 10–20, add one instance. If the delta is between 20–30, add two. The upper bound must always be greater than the lower bound.

### Docker Image — Multiple Tags / ALB Deep Dive

1. Tag the docker build with both a version tag and a latest tag:

- docker tag <name> <image-name>:${BUILD_NUMBER}
- docker tag <name> <image-name>:latest

2. Access multiple services (Jenkins, SonarQube, Vault) through a load balancer.
3. Use the ALB to route different paths to each service — no need to generate individual certificates per service, AWS handles certificate issuance.

### Terraform Advance

- for_each: iterate over a list or dict, one item at a time; use each to access values:

1. each.value
2. each.key

- lookup: retrieve a value from a dict by key

1. lookup(<map>, <key>)

- file()

1. e.g. public_key = file("~/.ssh/key.pub")

- template: templatefile(path, vars) — useful for generating lambda configs, task_definitions

1. e.g. generate 10+ clusters or ECS services from a template


AWS Terraform


- Docker:
  - Docker is a set of platform-as-a-service products that deliver software in isolated packages. Each package bundles its own software, libraries, and configuration files.
- Why not a virtual machine?
  - Unlike virtual machines, containers do not have a high overhead and hence enable more efficient usage of the underlying system and resources.
- Container:
  - Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data centre, the public cloud, or even a developer's personal laptop.
  - Benefits: predictable environments, isolation between applications, portability, more granular resource control, improved efficiency, and better compute utilization.
- Docker Image: The basis of a docker container. Represents a full application.
- Docker Container: The standard unit in which the application service resides and executes.
- Docker Engine: Creates, ships and runs Docker containers deployable on a physical or virtual, host locally, in a data centre or cloud service provider.
- Repo (Docker Hub(Public) or Docker Trusted private): Cloud or server-based storage.

My mental model from object-oriented programming:

- Java class == image
- new instances == container
- GitHub == Docker Hub (repository)

Docker works at the system level and helps keep runtime environments and dependency versions consistent.

- Users can download an image from Docker Hub and start a container with the same environment defined by that image.
- A modified container can be committed back into an image and shared with other users, which improves version control and reproducibility.

- sudo service docker start — start the Docker service
- docker build -t <imageName> .
  - Build an image from a Dockerfile.
  - `-t`: add a tag name for the image.
  - `-f`: specify the Dockerfile path.
- docker image rm <imageName>
  - delete an IMAGE from the local image store
- docker run -p 80:5000 —name <containerName> <imageName>
  - Run a container from an image in the foreground.
- docker run -d —rm -p 80:5000 —name <containerName> <imageName>
  - Run a container from an image in the background and remove it automatically when it stops.
  - `-p`: map a local port to a container port.
  - `-d`: detach, run the container in the background, and print the container ID.
  - `—rm`: remove the container when it stops.
  - `—name`: set the container name.
- docker stop <containerName>
  - graceful shutdown, can be resumed
- docker kill <containerName>
  - force kill (like pulling the power cord)
- docker ps -a
  - list all the running containers including stopped containers
- docker restart <containerName> or <containerID>
  - restart a container

Dockerfile: a text document that defines how to assemble an image through an automated build.

- FROM: initialize a new build stage with base IMAGE (from the local repository or public repository)
- ENV: key - value pair
- WORKDIR: working directory for any [RUN, CMD, ENTRYPOINT, COPY and ADD instructions] — sets the landing directory after logging into the terminal
- COPY: COPY <src> <dest>
- ADD: equivalent to COPY + automatically handles URLs and extracts tar archives
- RUN: run commands in the current image (Linux) — supports both shell command format and exec format with String array
- EXPOSE: the container expose a specific port to listen
- ENTRYPOINT: cannot be overridden by commands after docker run (CMD provides additional arguments to ENTRYPOINT) → <ENTRYPOINT> <CMD>
- CMD: specifies what to do after the container starts; when ENTRYPOINT is specified, CMD provides specific parameters; supports multiple instructions but only the last one takes effect (can be overridden by arguments after docker run, acts as default value)

PS: CMD runs at `docker run` time, RUN runs at `docker build` time

- docker exec -it <containerName> /bin/bash
  - operate inside an already-deployed container
  - -it: run the command in the container with bash, keep interactive
- docker container rm -f $(docker ps -aq)'
  - Delete all Docker containers.
  - -q → quiet: only display container ID
  - -a → all
  - -f → force
- docker ps -q | xargs docker kill
  - kill all running containers
  - -q: only show container IDs
  - -a all
  - `xargs`: pass piped output as command arguments.
- docker images -qa | xargs docker rmi -f
  - list all images with only container IDs and force delete them
- docker system prune
  - clean up unused data (-a, -f for deeper cleanup, see docs)
- docker inspect demo
  - inspect the container
- docker cp your_file demo:/
  - copy a file into the container
- docker <command> your_docker_account_name/image_name:<tag>
  - tag can be treated as a version number; defaults to `latest` if not specified

docker-compose.yml: a file for quickly orchestrating groups of Docker containers.
- service: individual application container instances, e.g. order microservice, inventory microservice, mysql container (one application container can include multiple running instances of the same container; each service has its own name, uses an image, and depends on other services)
- project: a complete business unit composed of a group of related containers. 1 project = multiple services (container application instances)
- Write the Dockerfile first, define each microservice and build the corresponding images
- Use docker-compose to define a complete project, arranging all containers in the application, their order, and interactions
- docker-compose up deploys everything in one command

- docker-compose build: build all the Docker files
- docker-compose up: run all the containers with dependencies
- docker-compose down: stop

### DevOps Docker Volume

- docker build -f Dockerfile-check -t
- docker volume create sortlog-back
- docker volume inspect sortlog-back
- docker run -d -p 4000:4000 -v "$(pwd)/docker-output:/app/node_modules" vol_try
- docker run -d -p 4000:4000 -v sortlog-back:/app/node_modules vol_try
- volume definition in Dockerfile


Docker, Docker compose, CLI

Docker/Docker Compose


### DevOps AWS IAM 1

![Untitled%2045.png](Untitled%2045.png)

![Untitled%2046.png](Untitled%2046.png)

- resources: the entities you create in AWS e.g. S3 bucket or EC2
- Policy Statement
  1. Sid: name of this policy statement
  2. actions: perform actions on resources. e.g. createBucket. type of access that is allowed or denied access
  3. implicit deny: anything not explicitly declared is denied; explicit deny also denies; if both allow and deny are given, deny takes precedence (AWS first processes the deny list, then the allow list — if neither applies, the result is deny)
  4. resource: limit the scope of this action — which resource this action applies to
  5. principal: from the resource's perspective, determines who can use this resource. Less commonly seen because we attach policies to principals (either a role or a user). Definition: the entity that is allowed or denied access (i.e., who this policy targets)
  6. condition: constraints that you can apply to your policy docs depending on context
- When a Policy is assigned to a user, that user can operate on resources via the AWS UI console, CLI, and SDK according to that policy
- Amazon Resource Name (ARN): uniquely identifies an AWS resource; some segments can be omitted and replaced with ::
  e.g. arn:partition:service:region:account-id:resource-type:resource-id
- trust relationships: account1 trusts account2, and account2 trusts account1, so accounts can consume roles from each other
- Least privileged model: grant only the permissions necessary to perform a task

IAM policies:

1. Who should have access to do what?
2. AWS Evaluation: can this person call this action on this resource

when you define access policies. You specify which IAM **principals** are allowed to perform which **actions** on specific AWS **resources** and under which **conditions**

### DevOps AWS IAM 2

![1411662087834_.pic.jpg](1411662087834_.pic.jpg)

- Policies are divided into identity-based policies and resource-based policies
  1. These two are completely independent of each other. A resource-based policy is viewed from the resource's perspective — it defines which principals can access this resource. It cannot be applied in bulk; each resource's policy is assigned individually based on context and cannot be reused across resources.
  2. An identity-based policy is viewed from the user, group, or role's perspective — it can be reused and applied in bulk.
- roles: used by non-login entities (not Group/User) to access AWS services, connecting AWS services to IAM. When you have an EC2 instance, you create a role so it can connect to S3.
- Each policy can have multiple statements, but each statement belongs to only one policy
- resource-based policies are mainly configured within each service's console — they define permissions between services, allowing a service to communicate with its corresponding resource
- Both roles and resource-based policies can enable resource-to-resource communication (provided the role contains the relevant resource-based policy)


Shawn's Blog