I’ve worked on Docker for a while but I have never written a blog about Docker as much as I’ve wanted. I will blog about how to install Docker CE on CentOS 7. I’m writing this blog based on the official documentation here.
Run the following command to test if Docker was installed correctly.
$ sudo docker run hello-world
You will see an output like the image below.
It will download the hello-world image and run it.
If you execute the command below, you will see the container that ran and exited.
$ sudo docker ps -a
The container is no longer useful, so let’s remove it. Replace the [CONTAINER ID] with what you see in the result of the previous command.
$ sudo docker rm [CONTAINER ID]
Run Docker Commands without sudo
It’s quite a bit more typing if you have to “sudo” for Docker commands on top of entering your password often by using it. There is a way to omit sudo when running Docker commands.
$ sudo usermod -aG docker $USER
$ exit
You have to exit the SSH session and log back in to be able to execute docker commands without sudo.
Removing Downloaded Image
Let’s see the list of images on the Docker host by executing the command below. I will omit sudo from here on.
$ docker images
Let’s remove it because we really don’t need it anymore.
$ docker rmi [IMAGE ID]
Recap
We went over the installation and the very basic test of Docker. What I’ve written here should get you going with Docker on CentOS 7.
Now I’m going to try to write about Jenkins installation and configuration on AWS. I obviously don’t want to just reinvent the wheel or completely do things wrong, so I did my search and I found a documentation here. I’m going to refer to it and work on an instance actually on my personal AWS account.
Create EC2 (VM) Instance
Let’s create a VM instance to host the Jenkins master. I’m going to assume that you’ve already set up an AWS account.
With your browser, navigate to aws.com and click on EC2 under All services –> Compute.
Make sure to select Oregon as the region at the upper right area.
Click Launch Instance button.
I’m going to select “Amazon Linux 2 AMI (HVM), SSD Volume Type”.
In the next step, I’m going to select the VM template of t2 micro of vCPUs 1 and 1 GB memory. I think it’s enough to run an instance of Jenkins master but I would not select this configuration in actual enterprise environments for its sheer size to scale. Click Review and Launch button.
Click Launch button in the next step.
You are now prompted to select an existing key pair or create a new one. I am creating a new one for this one and naming it jenkins_master. Make sure to download it to your desktop for later use for your SSH session. Finally click Launch instances button.
SSH into the Created Instance
Let’s SSH into the created instance. First, navigate to EC2 -> Instances and select the instance you just created. And click Connect button.
By clicking Connect button, you will see the instruction on how to SSH into the instance like the following.
Open terminal and navigate to the directory where you store the .pem file and copy and paste the command in the Example: section.
Make sure that the Jenkins daemon starts by itself after reboot.
$ sudo systemctl enable jenkins
See if the Jenkins process is up and running by executing the following command.
You should see something like the following image.
$ ps aux | grep jenkins
–httpPort=8080 indicates that Jenkins master is listening to port 8080 with plain HTTP. I will review the configurations
Configuring Jenkins
We know that Jenkins exposes its UI on port 8080/tcp. I would like to use the default port of 80 for HTTP traffic for this instance of Jenkins. So I’m going to change the listening port by modifying the configuration file at /etc/sysconfig/jenkins
$ sudo vi /etc/sysconfig/jenkins
Now let’s make sure that JENKINS_PORT=”8080″ is present in the file. This means that you will be able to access the Jenkins UI via HTTP at the port 8080. Restart Jenkins master just in case.
$ sudo systemctl restart jenkins
Opening a Port
To access the running instance of Jenkins master on your browser, the port 80/tcp has to be open on AWS side. Here is how you can do it.
Navigate to EC2 dashboard and click Security Groups under NETWORK & SECURITY on the left menu.
Click Create Security Group button and you will see a popup like the image below. Click Add Rule button and select Custom TCP on Type column. Enter 8080 in the Port Range column. Click Create button.
Now we are going to assign this security group to the instance we created previously. Navigate to Instances and make sure that the target instance is checked. Click Actions button and select Networking –> Change Security Groups.
Select the security group that was just created and click Assign Security Groups button.
This opens the port 8080 to the target instance. Let’s try to access it!
There are two ways to access it. If you navigate to the EC2 Instances page and select the target instance, you will see the detailed information about the instance. You can see the public DNS and its IP address so enter an URL like below.
http://[your public DNS]:8080
When it’s successful, you should see a UI asking you to unlock Jenkins.
I have already documented the way to configure Jenkins here in my previous post, so please refer to it from here on.
Recap
We have learned how to spin up a VM and actually install Jenkins master on it on AWS. There are a lot more to get to the point where Jenkins master may be secured and usable on AWS. The principle is pretty much the same as the one on Azure, such as enabling SSL and getting slaves communicate with it via the secured communication. I will talk more about it later in my blog.
I’ve wanted to come up with a code that can get the whole list of Jenkins jobs recursively. I was able to write it but I’m not really liking it… View it in GitHub.
def get_jobs(self, recursive=False):
r = None
if self.base_url.endswith("/"):
api_url = self.base_url + "api/json"
else:
api_url = self.base_url + "/api/json"
try:
urllib3.disable_warnings()
headers = {'Content-Type': 'application/json'}
r = requests.get(url=api_url, headers=headers, auth=(self.username, self.api_token), verify=ssl.CERT_NONE)
jenkins = json.loads(r.text)
result = {}
if recursive:
folders = []
while True:
self.process_result(jenkins['jobs'], folders, result)
if len(folders) == 0:
break
for index, folder in enumerate(folders):
r = requests.get(url=f"{folder}api/json", headers=headers,
auth=(self.username, self.api_token), verify=ssl.CERT_NONE)
jenkins = json.loads(r.text)
self.process_result(jenkins['jobs'], folders, result)
folders.pop(index)
if len(folders) == 0:
break
else:
for job in jenkins['jobs']:
if job['_class'] != 'com.cloudbees.hudson.plugins.folder.Folder':
result[job['url']] = job['name']
return result
finally:
if r is not None:
r.close()
def process_result(self, jobs, folders, result):
for job in jobs:
if job['_class'] == 'com.cloudbees.hudson.plugins.folder.Folder':
folders.append(job['url'])
else:
result[job['url']] = job['name']
The reason I’m not really liking it because I’m repeating process_result twice in the code. At least I was successful in what I wanted to do and I didn’t use recursive function for recursive result but there has to be a better way to accomplish it. I’m going to dig more into this.
Edit: I realized that the reason why I have repeat the process_result function twice in the code is because Python doesn’t have support do…while. I could be wrong but I’m thinking that’s the core of the issue but if anyone out there can suggest a better way, please let me know. 🙂
This technique looks kind of useless but it has a huge potential if you have 500 Jenkins legacy jobs that needs to be managed without DSL. Jenkins API allows us to update jobs by us uploading the XML files. It means you could make mass changes to the XML files and upload them.
The ideal way to manage many Jenkins jobs is to use Job DSL and there is no doubt about it, but for legacy Jenkins jobs, this technique will lead us to something really advantageous.
I previously wrote an article on setting up a DNS ANAME for prepare for CentOS 7 to join a Windows domain. It’s a prerequisite for the steps in this article.
Steps
Let’s ssh into the CentOS 7 as root. I’m planning to use the VM as a Docker host, so I named it as dockerhost01.
$ ssh root@dockerhost01.homenet.iriumi.ad
Now I’m going to install the prerequisite packages using yum.
Now using realm, we will join the domain. Make sure the credential you are using in the following command should be from the Windows domain. Make sure to replace [domain admin user] to an actual user.
You can alternatively add -v option to show verbose information.
Execute the following command to confirm that the OS is not a part of the Windows domain.
# realm list
Let’s see if I can check a user.
# id hiriumi@homenet.iriumi.ad
It shows the ID of the user and also which AD domain groups the user belongs to.
I would have to use [myusername]@homenet.iriumi.ad as my login name, which I would like to do. I’m going to make some changes in the configuration. Open up the configuration file executing the following command.
# vi /etc/sssd/sssd.conf
Change use_fully_qualified_names to False and I’m going to change fallback_homedir to /home/%u instead of /home/%u@%d
use_fully_qualified_names indicates whether you want to use [your username]@domain.foo.com as your username or not. fallback_homedir indicates how and where you want SSSD to create your home directory. My sssd.conf looks like the following.
If you enter pwd, it created /home/[your username] directory as your home directory.
Now when you execute a command that requires sudoer permission, you will get message like the following.
[your username] is not in the sudoers file. This incident will be reported.
We are going to logout and ssh back into it as root to fix this issue.
Add a new file at /etc/sudoers.d/sudoers
# vi /etc/sudoers.d/sudoers
Add a line like the following in the file. I’m going add a single user as a sudoer for now.
[your username] ALL=(ALL) ALL
You can also add AD groups as sudoers by adding a line like below.
%domain\ admins@homenet.iriumi.ad ALL=(ALL) ALL
Now that you added yourself as a sudoer, logout as root and login as the domain user. You should be able to execute sudo commands.
Recap
Having a centralized credential manager like Windows domain controller is quite essential to efficiently manage many servers. I have introduced a way to get CentOS 7 to join a Windows domain but this technique can be used for RedHat line of Linux distro. I’m not sure how it can be done for Debian/Ubuntu line of Linux distro and it might be a good topic for another blog article.
I’m getting ready to have an integrated environment on my server. I have a Windows domain controller up and running and I’m about to get my CentOS 7 join the domain.
Before I can go on, CentOS 7 needs to be able to communicate with the DNS server that I created on the Windows Server Core. The IP address of the DNS server in my network is 192.168.1.26. And the domain name is homenet.iriumi.ad.
A DNS server can have multiple zones. Let’s see what kind of zones I have by executing the following command.
It gives you the list of DNS entries in the zone. I have a CentOS 7 host that I have assigned a static IP address to and I’m going to make sure I can resolve it.
By executing the command above, as long as a machine can talk to the DNS server, it can resolve dockerhost01.homenet.iriumi.ad to 192.168.1.27. In other words, dockerhost01.homenet.iriumi.ad is mapped to 192.168.1.27.
If necessary, you can remove the DNS entry by executing the following command.
$ Remove-DnsServerResourceRecord -ZoneName "homenet.iriumi.ad" -RRType "A" -Name "dockerhost01"
Next, I’m going to ssh into my CentOS 7 VM and then configure it so that it asks the DNS server on Windows Server Core to resolve names.
# vi /etc/sysconfig/network-script/ifcfg-[your network interface]
In the text file, add the following entry. Change the IP address and the domain name to fit your environment, obviously.
There is a nice Python library that allows you to manipulate Jenkins master but having more dependencies can make it cumbersome to manage systems. For some simple things, I’m for writing my own script from scratch.
Jenkins has nice set of REST-like API interfaces. I’ve actually created an open source tool in WPF and C# around the mechanism before. I know there will be time when I will want some simple Python scripts to manipulate Jenkins jobs/builds, so I’m going to write some of it just for my reference. Hope this will help someone out there.
I have these 5 Jenkins jobs on my Jenkins instance on Azure. Let’s see if I can get the list using Python.
It may not be necessary to mention this but I’m writing the code in Python 3. It seems that they are really doing away with version 2, so… The following class lets you get the list of jobs. View code in GitHub.
import requests
import base64
import urllib3
import sys
import ssl
class jenkins_server:
def __init__(self, base_url, username, api_token):
self.base_url = base_url
self.username = username
self.api_token = api_token
self.crumb = None
def get_jobs(self):
r = None
api_url = None
if self.base_url.endswith("/"):
api_url = self.base_url + "api/json"
else:
api_url = self.base_url + "/api/json"
try:
urllib3.disable_warnings()
headers = {}
headers['Content-Type'] = 'application/json'
r = requests.get(url=api_url, headers=headers, auth=(self.username, self.api_token), verify=ssl.CERT_NONE)
return r.json()
except Exception as e:
print("Error occurred: ", str(e))
finally:
if r is not None:
r.close()
Please remember that requests is not in the standard library, so you may need to install it with pip. Another thing to note is that I could not figure out a proper way to verify the SSL cert. I tried many things but I was spending too much time so I’m setting verify=ssl.CERT_NONE. I’m hoping to figure it out later when I have more time.
Here is the sample test code.
def test_get_jobs(self):
jenkins = jenkins_server("https://jks.westus.cloudapp.azure.com", "yourusername", "api token here")
jobs = jenkins.get_jobs()
for job in jobs['jobs']:
print(f"name: {job['name']}")
