awk is one of the most used commands in bash environment. I can’t possibly cover everything about awk but I will write about an example that can be applied to many situations.
When I list files and directories in a directory, I execute…
ls -lah
That shows outputs the list like below.
total 296
drwxr-xr-x+ 50 hiriumi staff 1.6K Aug 27 21:44 .
drwxr-xr-x 5 root admin 160B Dec 5 2019 ..
-r-------- 1 hiriumi staff 7B Aug 18 22:06 .CFUserTextEncoding
-rw-r--r--@ 1 hiriumi staff 20K Aug 26 20:47 .DS_Store
drwx------ 86 hiriumi staff 2.7K Aug 26 16:57 .Trash
-rw------- 1 hiriumi staff 36K Aug 29 17:44 .bash_history
-rw-r--r-- 1 hiriumi staff 788B Jul 19 17:31 .bash_profile
-rw-r--r-- 1 hiriumi staff 272B Mar 10 23:35 .bash_profile.backup
drwx------ 3 hiriumi staff 96B Apr 14 22:46 .cache
drwx------ 5 hiriumi staff 160B May 21 19:54 .config
drwx------ 3 hiriumi staff 96B Mar 7 13:40 .cups
drwxr-xr-x 14 hiriumi staff 448B Mar 6 22:17 .dropbox
drwxr-xr-x 16 hiriumi staff 512B Aug 1 14:25 .dvdcss
-rw-r--r-- 1 hiriumi staff 56B Mar 9 00:01 .gitconfig
drwxr-xr-x 15 hiriumi staff 480B May 10 09:09 .iterm2
-rwxr-xr-x 1 hiriumi staff 22K May 10 09:10 .iterm2_shell_integration.bash
drwxr-xr-x 6 hiriumi staff 192B Mar 9 00:26 .kivy
<SNIP>
As you can see, if it’s a file, the line starts with “-” and if it’s a directory, it starts with “d”. Let’s just list the files using grep. I will show an example in awk after that.
ls -lah | grep '^-'
The following command outputs exactly the same list.
If permanent slaves are required and hundreds of them, it’s very manual and tedious tasks. I have created a PowerShell script to create Jenkins nodes in bulk. CloudBees’ version of Jenkins seems to have nice API to create nodes but open source/free version doesn’t seem to have one though Jenkins CLI could be utilized to accomplish it. Please follow my previous blog article to make sure your Jenkins CLI works before moving on with this article. Please note that this script does not install Jenkins slaves on separate machines but I am planning to create a script to do that in the future.
Prerequisites
Windows (The script I am going to introduce utilizes PowerShell, so Windows is the safest choice of OS.)
The command above creates nodes in Jenkins master from winnode-001 to winnode-100. The red cross icon you see below only indicates that they are offline and no actual slaves is communicating with them yet.
You can change the naming pattern specifying -NodeNamePattern.
By specifying -OverwriteNodeIfExists parameter to $True, it can delete the node and recreate. It can be dangerous, so its default value is $False and it’s an optional parameter.
The PowerShell Code
I am pasting all the code I have written for this operation. But if you just want the file, please go to my GitHub repo. All the details of each parameters are documented in the code.
<#
.SYNOPSIS
Generates Jenkins nodes.
.DESCRIPTION
This script bulk generates Jenkins nodes on the master so that nodes will be ready for Jenkins slaves to talk to them.
It utilizes Jenkins CLI, so Java (https://java.com/en/download/) and downloaded jenkins-cli.jar file from Jenkins master is required.
.PARAMETER JenkinsServerUrl
Required. Base URL of the Jenkins master. e.g. https://myjenkins.com/
.PARAMETER Username
The username to use for this process.
.PARAMETER ApiToken
Generated API Token of the username.
.PARAMETER NodeCount
Specifies how many nodes to created.
.PARAMETER NodeNamePattern
Specify the pattern of node name here. If 20 nodes to be created and "foobar-###" is passed, foobar-001 to foobar-020 would be generated.
.PARAMETER Description
Optional. Specify the description of each node. Use this field to specify what kind of fleet you are creating.
.PARAMETER RemoteRootDir
Optional. Defaults to "C:\slave". Specify any directory where you expect slave bits to reside on each node.
.PARAMETER Mode
Optional. Defaults to NORMAL.
.PARAMETER NumExecutors
Optional. Defaults to 1.
.PARAMETER Labels
Optional. e.g. "WINDOWS 2019". Separate labels by space.
.PARAMETER JenkinsCliJarFilePath
Optional. When you download jenkins-cli.jar, the file is usually at ~/Download/jenkins-cli.jar. Downaload the file from https://your-jenkins-server/jnlpJars/jenkins-cli.jar
There is a download link on https://your-jenkins-server/cli
.PARAMETER OverwriteNodeIfExists
Optional. If the node to be created already exists on the Jenkins master, it deletes it and recreates it if $True is passed. If $False, it is skipped.
.PARAMETER KeepNodeXmlFiles
Optional. Defaults to $False. If $True is passed, the XML files created for the nodes will be kept in the same directory as this script. This option could be used to reuse the XML files in different environments.
#>
[CmdletBinding()]
param
(
[Parameter(Mandatory=$True)]
[string]$JenkinsServerUrl,
[Parameter(Mandatory=$True)]
[string]$Username,
[Parameter(Mandatory=$True)]
[string]$ApiToken,
[Parameter(Mandatory=$False)]
[int]$NodeCount = 5,
[Parameter(Mandatory=$False)]
[string]$NodeNamePattern = "winnode-###",
[Parameter(Mandatory=$False)]
[string]$Description = "Bulk gen'ed node",
[Parameter(Mandatory=$False)]
[string]$RemoteRootDir = "C:\slave",
[Parameter(Mandatory=$False)]
[string]$Mode = "NORMAL",
[Parameter(Mandatory=$False)]
[int]$NumExecutors = 1,
[Parameter(Mandatory=$False)]
[string]$Labels = "",
[Parameter(Mandatory=$False)]
[string]$JenkinsCliJarFilePath = "$env:HOMEPATH\Downloads\jenkins-cli.jar",
[Parameter(Mandatory=$False)]
[bool]$OverwriteNodeIfExists = $False,
[Parameter(Mandatory=$False)]
[bool]$KeepNodeXmlFiles = $False
)
$digitCount = ($NodeNamePattern.ToCharArray() | Where-Object {$_ -eq '#'} | Measure-Object).Count
ForEach ($index In 1..$NodeCount)
{
$number = "{0:D$digitCount}" -f $index
$nodeName = $NodeNamePattern -replace "#{$digitCount}", $number
$xmlSettings = New-Object System.Xml.XmlWriterSettings
$xmlSettings.Indent = $true
$xmlSettings.Encoding = [System.Text.Encoding]::UTF8
$xmlSettings.OmitXmlDeclaration = $True
$xmlSettings.ConformanceLevel = "Document"
$sw = New-Object System.IO.StringWriter
$xw = [System.Xml.XmlWriter]::Create($sw, $xmlSettings)
try
{
$xw.WriteStartElement("slave")
$xw.WriteElementString("name", $nodeName)
$xw.WriteElementString("description", $Description)
$xw.WriteElementString("remoteFS", $RemoteRootDir)
$xw.WriteElementString("numExecutors", $NumExecutors)
$xw.WriteElementString("mode", $Mode)
$xw.WriteStartElement("retentionStrategy")
$xw.WriteAttributeString("class", 'hudson.slaves.RetentionStrategy$Always')
$xw.WriteEndElement() # retentionStrategy
$xw.WriteStartElement("launcher")
$xw.WriteAttributeString("class", "hudson.slaves.JNLPLauncher")
$xw.WriteStartElement("workDirSettings")
$xw.WriteElementString("disabled", "false")
$xw.WriteElementString("internalDir", "remoting")
$xw.WriteElementString("failIfWorkDirIsMissing", "false")
$xw.WriteEndElement() # workDirSettings
$xw.WriteElementString("websocket", "false")
$xw.WriteEndElement() # launcher
$xw.WriteElementString("label", $Labels)
$xw.WriteElementString("nodeProperties", "")
$xw.WriteEndElement() #slave
$xw.WriteEndDocument()
$xw.Flush()
}
catch [System.Exception]
{
Write-Host $_
exit 1
}
finally
{
$xw.Close()
}
# Write out the node XML to file to be used for standard input below.
Set-Content -Path "$nodeName.xml" -Value $sw.ToString() -Force
# Check the existence of the node.
$processInfo = New-Object System.Diagnostics.ProcessStartInfo
$processInfo.FileName = "java"
$processInfo.RedirectStandardError = $True
$processInfo.RedirectStandardOutput = $True
$processInfo.RedirectStandardInput = $True
$processInfo.Arguments = "-jar $JenkinsCliJarFilePath -s $JenkinsServerUrl -auth $Username`:$ApiToken get-node $nodeName"
$processInfo.UseShellExecute = $False
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $processInfo
$process.Start() | Out-Null
$process.WaitForExit()
$stdo = $process.StandardOutput
If ($process.ExitCode -ne 0)
{
Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
Write-Host "$nodeName created ($(Get-Date))"
}
else
{
If ($OverwriteNodeIfExists)
{
Write-Host "Deleting $nodeName..."
Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","delete-node $nodeName"
Write-Host "Recreating $nodeName..."
Start-Process -FilePath java -NoNewWindow -Wait -ArgumentList "-jar $JenkinsCliJarFilePath","-s $JenkinsServerUrl","-auth $Username`:$ApiToken","create-node" -RedirectStandardInput "$nodeName.xml"
}
else
{
Write-Host "Node $nodeName exists."
}
If (!$KeepNodeXmlFiles)
{
Remove-Item -Path "$nodeName.xml" -Force
}
}
}
Next Step
The script I created can bulk create the “receivers” for the actual slaves to talk to the Jenkins master. I will write another PowerShell to script to automatically install and communicates with the Jenkins master.
Recap
Pull Requests, any suggestion or even forking is welcome. This should make Jenkins permanent node/slave administration a little easier.
I previously wrote How to Install Jenkins Slave as Windows Service in this blog. It has been one of the most accessed articles on this site. Though the article worked for people who visited here, I thought of taking it to the next level. What if I come up with a way to easily install Jenkins Slave as Windows Service by running scripts? It would save so much time and effort without mistakes.
Before working on the whole script, I want to make sure Jenkins CLI works. Jenkins CLI is different from REST API of Jenkins and it needs some preliminary preparation.
Jenkins CLI
Jenkins CLI is available from Manage Jenkins -> Tools and Actions -> Jenkins CLI.
When I click Jenkins CLI, there is a list of commands available.
I’m going to try to see if help works for sanity check. Before running the command java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ help , make sure to install Java and download jenkins-cli.jar from the Jenkins CLI page. When I ran it, I get the following error.
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
<SNIP>
at hudson.cli.FullDuplexHttpStream.(FullDuplexHttpStream.java:73)
at hudson.cli.CLI.plainHttpConnection(CLI.java:361)
at hudson.cli.CLI._main(CLI.java:299)
at hudson.cli.CLI.main(CLI.java:96)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
<SNIP>
… 20 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
… 25 more
This is because the SSL cert of the Jenkins master server is not trusted by Java. Let’s download the cert and get Java to trust it.
Download SSL Cert
Using openssl, we will download the SSL certificate as a file. Execute the following command.
Don’t mind some seemingly error message. Now you get a file jenkins.linux-mint.cer.
Trust the Cert
When you have Java on your system, you have a file called cacerts. Basically, you import the SSL cert you just downloaded into the cacerts file. Where is the file? Let’s find out. Execute the following command to locate cacerts.
sudo find /Library/Java -name cacerts
My system right now is a Mac and I happen to have the file at the following location.
If you want to remove the certificate, you can execute the following command. (Do not execute it if you want to avoid the error I talked about earlier.)
First, you need to generate API token for your user. Follow the steps below.
Login to Jenkins master.
Click on your username around the upper right corner.
Click Configure.
Click Add new Token button.
Copy the generated token in clipboard.
Now try to execute the following command.
java -jar jenkins-cli.jar -s https://jenkins.linux-mint.local/ -auth [Your User]:[Your Token] help
Now you don’t get the error and you will see the list of available commands.
add-job-to-view
Adds jobs to view.
build
Builds a job, and optionally waits until its completion.
cancel-quiet-down
Cancel the effect of the "quiet-down" command.
Resume using a node for performing builds, to cancel out the earlier "offline-node" command.
<SNIP>
wait-node-online
Wait for a node to become online.
who-am-i
Reports your credential and permissions.
Recap
I personally like using REST API of Jenkins better than Jenkins CLI but what I am planning to do may require Jenkins CLI. SSL protected Jenkins makes it harder to deal with it via its API but this makes it possible. Remember Java has its own keystore separate from the OS where it resides.
I previously wrote a blog article about changing Jenkins’ header color so that we can easily distinguish which instance of Jenkins you are dealing with. Some readers reported that the snippet to change the color was not working anymore after upgrading Jenkins. (Thank you for letting me know!)
Here are the complete steps again.
Navigate to Manage Jenkins.
Click Manage Plugins.
Click Available tab.
Install “Simple Theme” plugin by clicking Install without Restart button.
Go to Manage Jenkins -> Configure System.
Click Add under Theme section and select Extra CSS.
Enter the following simple CSS to change the background color of the header. This is what has changed from the previous article. #header { background:red; }
Save the change and when you refresh your browser, the header color has changed.
I don’t remember the exact HTML before but here is what the header looks like now and the CSS makes sense.
When I left my previous job, I didn’t have a Jenkins instance to test it out but now that I provisioned my own instance with Docker, I can test things with Jenkins again.
I sometimes visit the basic commands to refresh my memory. You work on so many different things in your real work, so it’s not possible to remember all the details of each command. I’ll go over sed in this blog article.
The description of sed in its man page says the following.
The sed utility reads the specified files, or the standard input if no files are specified, modifying the input as specified by a list of commands. The input is then written to the standard output.
The syntax looks like this.
sed [option] [script] [target file]
There are so many options and usages but I will just summarize the ones that are used often.
The command above prints from the first line to the 3rd line in sample.txt file. The output looks like the following.
Hello
Heeeeello
Helloooooo
It is possible to list the lines that match regular expression.
sed -n /^He.*llo/p sample.txt
The result looks like the following.
Hello
Heeeeello
Helloooooo
Remove Lines
sed 1,3d sample.txt
The command above removes the lines from 1 to 3 and outputs the rest of the text to the standard output. Please note that the command will not remove those lines from the file itself but it just outputs the result in the standard output. The sample output shows like the following.
foobar1
foobar2
foobar3
It’s also possible to remove lines using regular expression. Please see the example below.
sed /^Hello/d sample.txt
The command above removes lines that start with “Hello”. The output will show like this.
Heeeeello
foobar1
foobar2
foobar3
Replace Texts
sed can replace certain text with something else based on regular expression like the following.
I’ve wanted to provision Jenkins on NGINX reverse proxy for a long time. I would hit walls here and there, but I finally got things sorted out at least to the point where I can hit Jenkins master on NGINX reverse proxy on my browser both running on Docker container.
There is so much to cover, so I think this blog post is going to be quite long.
First of all, I need to make clear what I want to accomplish.
I want to have multiple instances of Jenkins hosted on a single Docker host. Depending on the host name the HTTP request came, NGINX reverse proxy routes the traffic to the appropriate Jenkins instance.
The diagram below represents the logical architecture.
Here is the list of things that will be covered in this blog entry.
Docker Network
Jenkins on Docker
NGINX on Docker as Reverse Proxy
Putting NGINX and Jenkins Together
Docker Network
As you can see, there are 3 containers in the diagram above. NGINX reverse proxy routes traffic to Jenkins instances. It means that NGINX needs to be able to communicate with Jenkins instance. The containers in the diagram exist within Docker network.
By default, Docker containers use bridge network meaning that Docker uses its own network though it is configurable. If you execute ip addr on your terminal, you’d see network interfaces like this.
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:67:fe:f4:d5 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
103: br-1418967f1f10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:63:60:8f:d1 brd ff:ff:ff:ff:ff:ff
inet 172.25.0.1/16 brd 172.25.255.255 scope global br-1418967f1f10
valid_lft forever preferred_lft forever
inet6 fe80::42:63ff:fe60:8fd1/64 scope link
valid_lft forever preferred_lft forever
When you spin up a Docker container, each container has its own IP address within the Docker network. I provisioned a Jenkins container and I can check its IP address with the steps below.
Check the container ID. docker ps -a
Run the following command to get the IP address. docker inspect [container ID] | grep IPAddress
I got 172.25.0.2 as the IP address of the container. This means that it’s within the range of the CIDR 172.25.0.1/16 listed in the br-1418967f1f10 bridge network above. Based on the CIDR, the network can host up to 65,536 containers which is way more than enough for normal operations.
Why is this important in this article. It’s because I am planning to provision NGINX as a Docker container and I want the NGINX container to communicate with Jenkins container within the Docker network without using the host network. It will increase the network security level to the next level. It is important to understand the network aspect of containerization.
Jenkins on Docker
First of all, I need to provision a Jenkins container. Here is the initial Docker Compose file (YAML) I created.
Please note that you should run mkdir jenkins_home (create the directory) beforehand to make sure you won’t get an error because as you can see it in docker-compose.yaml, /var/jenkins_home in the container is mapped to the host’s ./jenkins_home directory where all the data is persisted.
If you save this file as docker-compose.yaml and run docker-compose up -d and then access it on your browser http://foobar:8080/, you should see the initial set up UI. If you want to stop the container, you can just run docker-compose down.
It would be operational enough to use the Jenkins container as the way it is. However, it’s limited in a sense that if you want to have another instance of Jenkins on the same machine, you would have to use another port like http://foobar:8081 and http://foobar:8082 if you need another instance. It would mean you would have to open more ports as you have more instances of Jenkins on the same Docker host. Also, it’s kind of ugly. By having NGINX reverse proxy in place in front of the Jenkins instances on the Docker host, it solves the problem.
Moreover, NGINX can be the front man to service SSL communication. It’s the best practice nowadays that even internal web applications are protected by encrypted communication by SSL. You may want to have URLs that looks like the following.
https://jenkins1.foobar and https://jenkins2.foobar
This way the URL is easier to remember and also protected by SSL communication.
NGINX on Docker as Reverse Proxy
Here is the NGINX’s docker-compose.yaml file that I’ve got.
For the NGINX container, I am mapping the /etc/nginx/nginx.conf to the local ./data/nginx.conf file and /etc/nginx/conf.d directory to ./data/conf.d directory. Make sure that you have ./data and ./data/conf.d directories before you run docker-compose up -d.
I am also creating networks -> proxynet -> custom_network here in docker-compose.yaml because I want to have the Docker network where NGINX and Jenkins can communicate with each other.
Putting NGINX and Jenkins Containers Together
Here is the docker-compose.yaml file that can spin up NGINX and Jenkins within the same subnet within Docker. Notice that I commented out - 8080:8080 line. It means I stopped exposing the port 8080 of the Jenkins container to the host side because it is not necessary anymore. NGINX and Jenkins will communicate with each other within the Docker network subnet.
Also you need to have the following ssl.conf file under ./data/conf.d directory file. Notice that this file does 2 important things. One is to enable SSL communication between NGINX and the client side. If you wonder how to create SSL certificate, I have written how to just do it in my previous blog post. The second thing is to route the HTTPS traffic to the server jenkins:8080. This communication with done within the Docker network subnet, so no traffic goes out to the host’s network. You can also see that the SSL certificate files are under conf.d/ssl directory.
upstream jenkins_upstream {
server jenkins:8080;
}
server {
server_name jenkins.linux-mint.local;
listen 443 ssl;
ssl_certificate /etc/nginx/conf.d/ssl/jenkins.linux-mint.local.crt;
ssl_certificate_key /etc/nginx/conf.d/ssl/jenkins.linux-mint.local.key;
location / {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
resolver 127.0.0.11;
#proxy_redirect http:// https://;
proxy_pass http://jenkins_upstream;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_buffering off; # Required for HTTP-based CLI to work over SSL
# workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
add_header 'X-SSH-Endpoint' 'jenkins.linux-mint.local:50022' always;
}
}
Once it’s all done, just execute the command docker-compose up -d in the directory where you have docker-compose.yaml. If it’s successful and you access the URL like https://jenkins.linux-mint.local, you should see the initial screen where you can start to manage the instance of Jenkins.
If you execute cat jenkins_home/secrets/initialAdminPassword, you should see the initial password to enter on the screen. And you just have to click Continue button, you are on to setting up Jenkins.
Recap
I have covered the steps to spin up Jenkins behind NGINX as a reverse proxy. Based on my experience, this is the most flexible and safest way to provision Jenkins. There are steps to provision agents (slaves) for it. I will cover the topic some other time.