← all posts

Jenkins

November 15, 2022
Jenkins

Jenkins VM Server Installation Guide (to be automated with Ansible later)

  • Install Jenkins on a VM
    1. Update apt
    2. Install Java environment
    3. Import the GPG key for the Jenkins repository
    4. Add the Jenkins repository to the system
    5. Once the Jenkins repository is added and enabled, update the package list and install the latest version of Jenkins
    6. apt install Jenkins: sudo apt install jenkins -y
    7. Check Jenkins status: systemctl status jenkins
    8. Open port 8080 (on GCP this is more involved — you need to create a new firewall rule in GCP Firewalls, then attach it to the GCP VM via a network tag)
    9. Copy the key from the VM: sudo cat /var/lib/jenkins/secrets/initialAdminPassword
    10. sudo -i => switch to root => gpasswd -a jenkins root (add the Jenkins account to the root group)
    11. groups jenkins should show root
    12. visudo => add jenkins ALL=NOPASSWD:ALL so Jenkins can run system-level commands (e.g. Docker) without entering a password
    13. sudo usermod -aG docker $USER — fixes "dial unix /var/run/docker.sock: connect: permission denied" when Jenkins has no permission to run Docker (adds Jenkins to the Docker group) On Linux, the default home directory is /home/user, but Jenkins' home directory is /var/lib/jenkins, so any commands Jenkins needs to run must be installed under that path — for example, GitHub SSH keys should go there.
    14. After switching to the Jenkins user, run ssh-keygen to generate a GitHub access key
    • For master-node coordination:
    1. ssh-keyscan -H <ip-address> >> /var/lib/jenkins/.ssh/known_hosts — skips the yes/no prompt, optional — adds the host record to known_hosts
    2. Give the public key to master, place it in authorized_keys: echo "content" >> ~/.ssh/authorized_keys

Jenkins Credential Types

  • Secret text — tokens such as API tokens (e.g. GitHub personal access tokens)
  • Username and password — can be stored as separate fields or as a colon-separated string: username:password
  • Secret file — encrypted content stored in a file
  • SSH Username with private key — SSH public/private key pair
  • Certificate — a PKCS#12 certificate file and optional password
  • Docker Host Certificate Authentication credentials

Jenkins Plugins Required

  • AnsiColor: colorizes Console Output
  • Blue Ocean
  • pipeline: AWS steps
  • CloudBees AWS Credentials: stores access key and secret key
  • CloudBees Docker AWS Credentials
  • Role-Based Authorization Strategy
  • Email Extension Plugin
  • Docker
  • Docker pipeline
  • Bitbucket Plugin
  • Slack Notification Plugin
  • Azure Credentials plugin

DevOps Jenkins 01

IMG_0394.PNG

IMG_0395.PNG

  • Jenkins
    • An open-source application that enables CI/CD, integrating with a wide range of testing and deployment technologies.
    • Improves efficiency: CD with a web project => web farm (web server farm), with deployment status easily tracked from the dashboard.
  • CI
    • Automatically verifies the build using unit tests and packages the solution for each new code change submitted.
    • Compile == Build == CI (source code/text files => binary) => .exe, lib, pkg
  • CD
    • Takes executable files (validated code packages) from the build process and deploys them into staging or production environments.
    • Developers can track which deployments succeeded or failed and narrow down issues to specific package versions.
    • Two server sets alternate between staging and production (see diagram) — blue/green deployment. https://blog.inedo.com/blue-green-deployment-best-practices Jenkins can also be used to orchestrate other cloud services such as AWS CodeBuild/CodeDeploy — a useful complement to existing infrastructure. Note: M1 Macs use the ARM architecture (RISC), so running Docker likely goes through Rosetta2 translation. Running Jenkins inside Docker on M1 feels extremely slow — roughly 10x slower than on Windows.

DevOps Jenkins & Intranet Tunneling 02

Untitled

  • Install Jenkins locally
    • On Windows, run sudo service docker start to start the Ubuntu Docker engine (can connect Docker engine to Ubuntu as shown in diagram 2)
    • Run Jenkins container
    • Get admin password to complete Jenkins setup
  • ngrok (reverse proxy / intranet tunneling)
    • Visit ngrok website to get an auth token
    • Local installation:
      • Windows: change permissions → extract → configure token → open port 80
      • macOS: use brew install --cask ngrok (no manual PATH configuration needed), rest is the same as Windows
      • Copy the tunnel URL to Jenkins configuration => Jenkins URL
  • Connect GitHub with Jenkins
    • Add new item
    • Freestyle project
    • Configure with Git + Repo URL with .git connection
    • Build triggers => GitHub hook trigger for GITScm polling
    • Branches to build: choose */master or leave empty to trigger on all branches
    • Add webhook in GitHub => go to git repository
    • Payload URL must end with github-webhook/ e.g. http://XXXX.ngrok.io/github-webhook/
    • Select "Just the push event" as the trigger
    • Check "Active" to enable the webhook

Windows/Ubuntu users need to prepend sudo to most commands.

Installing ngrok on macOS:

https://gist.github.com/wosephjeber/aa174fb851dfe87e644e?permalink_comment_id=3621858

DevOps Azure Docker Jenkins 1

  • Create VM
    1. Check connection settings in the server
    2. Open ports 22, 80, and 443
    3. Allocate 64GB SSD; check "delete on termination" for both disk and public IP
    4. Open port 8080 for demo
  • SSH into server => install Jenkins via Docker
    1. sudo apt update
    2. sudo apt install docker-compose
    3. sudo docker run XXX
  • NS delegation from AWS to Azure (subdomain)
    1. nslookup -type=NS shawnwong.click — check DNS name servers
    2. Add the domain name in Azure DNS (e.g. sure.XXX.XXX)
    3. Copy NS records from Azure DNS to the domain registrar (NS delegation)
    4. nslookup -type=NS nz.shawnwong.click — verify DNS name servers
    5. Add a new record set in Azure DNS with type A and alias
  • Install Nginx on Azure VM
    1. apt install nginx
    2. systemctl enable nginx (enable auto-start on boot)
  • Add SSL
    1. Modify Nginx configuration files
    2. Restart Nginx: sudo systemctl restart nginx
    3. systemctl status nginx — check service status

Gotchas:

  • WSL Ubuntu cannot modify file permissions for Windows files
    1. Copy the file into the Linux directory using cp
    2. Directly modify file permissions via /etc/wsl.conf

DevOps with Private GitHub 2 & Nginx Config

Untitled

Untitled

Untitled

Untitled

Configure DNS for a specific domain (from AWS to Azure):

  • Create in Azure DNS; Name can be set to XXX.<yourdomainname>
  • Copy NS records to AWS Route53 as delegation
  • Create a record set => use an alias record set to point directly to a VM resource
  • Or use an A record pointing to the public IP address

Install Nginx & add certificate to Jenkins server:

  • sudo apt install nginx
  • sudo systemctl enable nginx — enable auto-start on boot
  • Add certificate
  • sudo systemctl status nginx — check service status
  • sudo systemctl restart nginx — restart Nginx

Narrow way to add SSH public key:

  • Add SSH key in specific GitHub repository => Settings => Deploy Keys to restrict access to that repository only

Troubleshooting guide:

  1. If Jenkins is running inside Docker, add the .ssh key inside the container (use sudo docker exec -it <cont_id> sh to enter the container, then run ssh-keygen)
  2. No need to add credentials in Jenkins dashboard => pipeline => Source Code Management; set credentials to "None"
  3. Restart container with: sudo docker restart <cont_id>
  4. When modifying Nginx configuration, remember to comment out the 404 redirect

DevOps Jenkins Blue Ocean

Untitled

  • Mainly covers CI and CD (content delivery) as a multibranch pipeline; does not cover content deployment
  • Content deployment involves production data (user data, databases, live workloads)
  • git push origin <new-branch> or git push origin HEAD — push a new branch to GitHub
  • git pull . master — pull files from master into the current branch (syncs locally)
  • Difference between a Plugin and Docker Engine?
    • One is a tool that uses Docker; the other is Docker itself — Arno
  • Blue Ocean workflow:
    1. Connect GitHub => GitHub Settings => Developer Settings => Personal access tokens => grant permissions (user, write, etc.) => create a pipeline
    2. Install Docker Engine on Linux
    3. Run which docker => find the Docker Engine path, then add it to global configuration
    4. Configure Docker path in Jenkins and add the Docker Pipeline and Docker plugins
    5. Add HOME '.' in Jenkinsfile to run files in the current user's directory (/bin/sh) rather than the root directory

DevOps Jenkins Master-Node Coordination

Untitled

AZ-GCP-Local Jenkins

  • Start AZ, GCP VM + local machine
    1. Add Docker environment
    2. Add JDK 11+ environment
  • Add local node: Manage Jenkins => Manage Nodes and Clouds => Add a new node
    1. Note: in Configure Global Security, add a fixed TCP port 5003
    2. Download the agent jar to the local machine via wget
    3. Open inbound port 5003 on the Azure VM
    4. Since this is local, the agent may go offline — configure it to start only when a job matching its label is triggered
    5. Choose "Launch agent by connecting it to the controller" (Azure VM proactively calls the local machine to execute tasks)
  • Add GCP node
    1. visudo — used to edit /etc/sudoers to grant sudo permissions to a regular user without a password. Add: jenkins ALL=NOPASSWD:ALL
    2. Use this node as much as possible
    3. /var/lib/jenkins is equivalent to the Jenkins user's home directory, similar to /root or /home/<username>
    4. Generate an SSH key for the Jenkins user: /var/lib/jenkins/.ssh/id_rsa
    5. ssh-keyscan -H 34.129.185.248 >> /var/lib/jenkins/.ssh/known_hosts — add the GCP node host to the trusted list
    6. Add the id_rsa.pub key to GCP for passwordless access: echo 'keyxxx' >> ~/.ssh/authorized_keys
    7. Add the id_rsa private key to Jenkins credentials to complete authentication with the GCP VM

Jenkinsfile Generator

Untitled

Untitled

Untitled

Jenkinsfile code can be generated using Jenkins' built-in pipeline code generator, which includes a Snippet Generator and a Declarative Directive Generator.

  • pipeline — starts the code pipeline
  • agent
    1. any — any available agent
    2. label — run on an agent matching a label
    3. docker — run inside a Docker container
    4. dockerfile — build from an image matching a label
  • stages — pipeline entry point
  • post — execute logic after all stages complete (useful for cleanup or notifications based on build outcome)
  • parameters

Snippet Generator:

  • cleanWs() — delete workspace when build is done
  • git — git credentials
  • timeout + input

Troubleshooting guide:

  1. When using withCredentials from the Snippet Generator, select "Username and Password (separate)", link to your Jenkins credentials. The variable name you define (e.g. dockerPwd) will be referenced in the Jenkinsfile and mapped to the actual password at runtime — providing credential masking in the console output.
  2. credentialsId is the name you gave the credential in Jenkins.
  3. When parameters is configured, a "Build with Parameters" option appears in Jenkins, allowing the pipeline to be triggered with custom parameter values.
;