Migrated to ARM64 Oracle Linux 7

I have migrated my site to Oracle Linux 7 on ARM64 just because I was curious how it would work on ARM64 or if it would work on ARM64 at all.

As of this writing, I’ve got this WordPress site on Docker to work on Oracle Linux 7 on ARM64 processor. It was some struggle but I feel it’s snappier.

Here is the Docker Compose file that works for this site. As you can see, I commented out the mysql image and using Maria DB for ARM64 instead. Maria DB is a fork for MySQL, so it’s sufficient enough for my personal blog site.

version: "3.9"
services:
  db:
    image: arm64v8/mariadb # mysql/mysql-server:8.0.20-aarch64
    volumes:
      - ./mysql:/var/lib/mysql
    restart: always
    command: mysqld --default-authentication-plugin=mysql_native_password
    environment:
      MYSQL_ROOT_PASSWORD: YourPassword
      MYSQL_DATABASE: hayato_iriumi_db
      MYSQL_USER: blog_admin
      MYSQL_PASSWORD: AnotherPassword
    networks:
      proxynet:
  phpmyadmin:
    depends_on:
      - db
    image: arm64v8/phpmyadmin
    restart: always
    ports:
      - '8080:80'
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: YourPassword
    networks:
      proxynet:
  wordpress:
    image: arm64v8/wordpress
    container_name: wordpress
    depends_on:
      - db
        # ports:
        # - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: blog_admin
      WORDPRESS_DB_PASSWORD: AnotherPassword
      WORDPRESS_DB_NAME: hayato_iriumi_db
      WORDPRESS_DEBUG: 'true'
    volumes:
      - ./html:/var/www/html
      - ./wp-content:/var/www/html/wp-content
    networks:
      proxynet:
  reverse:
    image: arm64v8/nginx
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/conf.d:/etc/nginx/conf.d
    ports:
      - "80:80"
      - "443:443"
    restart: always
    networks:
      proxynet:
volumes:
  db_data: {}
  wordpress: {}
networks:
  proxynet:

Docker Compose 2 works well on ARM64 as well. It is definitely faster than Docker Compose 1. By just changing the platform, I feel I upgraded my blog site for a better performance.

How to Install Docker Compose 2 on ARM Processor Linux Host (OL7)

I have provisioned Oracle Linux 7 on ARM processor on OCI. In an attempt to get my blog site to work on the host, I have installed Docker engine and now I want to install Docker Compose 2. Here are the steps.

  1. Create a directory.
    mkdir -p ~/.docker/cli-plugins/
  2. Download the Docker Compose 2 binary for ARM64.
    curl -SL https://github.com/docker/compose/releases/download/v2.2.2/docker-compose-linux-armv7 -o ~/.docker/cli-plugins/docker-compose
  3. Give the binary file execute permission.
    chmod +x ~/.docker/cli-plugins/docker-compose
  4. Check if the binary has been installed successfully.
    docker compose version

Not Working on ARM Host

I spined up an ARM host on OCI and installed Docker Compose 2 and then downloaded my docker-compose.yaml file to start my blog engine on it, but I got the following error.

This means no image is available for ARM64. I don’t think there is anything I can do at the moment (unless I make the effort and create images myself), so I am giving it up for now. Oh well…

Edit: I did some research to see if those Docker images are available and they are. I will give it another try.

The Basic Linux Commands an Interviewer Want You to Know

I interview young candidates from time to time. At a company where Linux is prevalent, I’d expect candidates to know some basic Linux commands.

  • cat
  • grep
  • vim (know the basic operations such as open text file, edit and save it and close it)
  • top (htop) to monitor processes
  • traceroute
  • df -h
  • netstat
  • chmod
  • chown
  • kill
  • wget
  • mkdir
  • ls
  • find
  • lsblk
  • lscpu
  • fdisk
  • find

Ha, those are very basic commands! You may say, but if you are not exposed to Linux at all, you just don’t know. I didn’t know those when I was an engineer with only Windows experience but it’s good to prepare.

How to Generate Free SSL by Let’s Encrypt

I attempted to use Certbot by Let’s Encrypt for free SSL certificate for my subdomain using a Docker container but it was never successful, so I left it alone for a while. The DNS service that I use provides SSL certs but it charges me extra for subdomain. There are other services such as SSL for Free but they either limit the number of certificates or they charge for subdomains or they want me to pay like $10 per month for wildcard certificate. That’s pretty steep considering the host is being used only privately. The alternative is to create your own private CA authority and issue SSL cert and have the root CA cert on the machines that you use the hosts.

But this morning, I figured out a way to generate the free SSL cert for my subdomain using Certbot. I will write about what worked for me.

I followed this instruction to install snapd on Ubuntu.

First, remove certbot if installed by apt.

sudo apt-get remove certbot

Install Certbot.

sudo snap install --classic certbot

Prepare the Certbot command.

sudo ln -s /snap/bin/certbot /usr/bin/certbot

Now prepare your NGINX server to accept HTTP traffic for acme challenge.

Edit nginx.conf to accept HTTP (port 80).

   server {
      location / {
          root   /var/www/html;
          index  index.html index.htm;
      }
      listen       80 default_server;
      listen       [::]:80 default_server;
      server_name  _;

If you have the HTTP redirect to HTTPS, uncomment the line.

# return 301 https://jenkins.hayato-iriumi.net$request_uri;

Now back to Certbot, execute the following command to start to issue your ssl cert.

sudo certbot certonly -a manual --rsa-key-size 4096 --email hiriumi@gmail.com -d jenkins.hayato-iriumi.net

You will see an output like the following.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for jenkins.hayato-iriumi.net

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:

s8wH1u2z00ePejV4hyy4y3CTyW3pYvrFgxwxwsPVdd8.O3THIaz5tgLf8NuxfBYw8FZfrdQNf_Y_1U--J0PsgqQ

And make it available on your web server at this URL:

http://jenkins.hayato-iriumi.net/.well-known/acme-challenge/s8wH1u2z00ePejV4hyy4y3CTyW3pYvrFgxwxwsPVdd8

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Now you should create the file with the data specified in the output. Once you have that hit Enter to get the cert generated.

Lastly, when the cert generation is successful, you will see the output like the following.

Now switch to the root user by executing…

sudo -i

The cert files are at /etc/letsencrypt/archive/jenkins.hayato-iriumi.net

Copy cert1.pem and privkey1.pem to the directory where you would like to store your SSL files. In my ssl.conf file, I have specified the cert files like the following.

server {
    server_name jenkins.hayato-iriumi.net;
    listen 443 ssl;
    ssl_certificate /etc/nginx/conf.d/ssl/cert1.pem;
    ssl_certificate_key /etc/nginx/conf.d/ssl/privkey1.pem;
    client_max_body_size 3000m;

Now unc0mment the line in ssl.conf to redirect HTTP to HTTPS traffic. Once you restart your NGINX, NGINX starts to service the traffic in SSL.

I’m sure there are ways to automate this and I am thinking of exploring the way to do it but it works well for now.

How to Make Linux Jenkins Slave a Daemon

I wrote an article on creating a Jenkins slave on Linux. The method was to just create a bash script file that requires to be executed by hand. And it wouldn’t survive restarting the host, so what I need to do is to make the script a daemon (service).

Here is what I did before configuring the daemon.

  1. Provision a Ubuntu host on Azure (it doesn’t matter where you provision the host as long as your Jenkins master on the public Internet and secured).
  2. Update the system. (sudo apt update && sudo apt upgrade)
  3. Install OpenJDK. (sudo apt install openjdk-11-jre-headless)
  4. Open port 50000 (Inboud and Outbound) to the host. I am opening all protocols.

Creating a Daemon

We will create a script at home directory first. To contain everything for Jenkins slave, I am creating /home/azureuser/jenkins-slave directory. You can create jenkins-slave or whatever the directory name you would like anywhere.

Then create slave.sh in /home/azureuser/jenkins-slave directory with the following content. Change the URL and the secret acccording to the Jenkins node you have created on Jenkins master. Make the script executable by executing chmod +x slave.sh.

java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir "./slave"
exit 0

Also make sure you download agent.jar from Jenkins master to /home/azureuser/jenkins-slave directory. Also user opessl and keytool to trust the SSL cert. You can refer to the previous blog article on how to use keytool.

Now create /etc/systemd/system/jenkins-slave.service file with the following content. sudo vim /etc/systemd/system/jenkins-slave.service

[Unit]
Description=JenkinsSlave

[Service]
User=azureuser
WorkingDirectory=/home/azureuser/jenkins-slave
ExecStart=/bin/bash /home/azureuser/jenkins-slave/slave.sh
Restart=always

[Install]
WantedBy=multi-user.target

Enable the daemon with the following command.

sudo systemctl enable jenkins-slave.service

Now start the daemon with the following command.

sudo systemctl start jenkins-slave.service

Check the status of it.

sudo systemctl status jenkins-slave.service

Result:

● jenkins-slave.service - JenkinsSlave
     Loaded: loaded (/etc/systemd/system/jenkins-slave.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-22 22:10:03 UTC; 8min ago
   Main PID: 14698 (bash)
      Tasks: 19 (limit: 486)
     Memory: 103.3M
     CGroup: /system.slice/jenkins-slave.service
             ├─14698 /bin/bash /home/azureuser/jenkins-slave/slave.sh
             └─14699 java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc40>

Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Handshaking
Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Connecting to jenkins.hayato-iriumi.net:50000
Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Trying protocol: JNLP4-connect
Oct 22 22:10:06 jenkins-slave bash[14699]: Oct 22, 2021 10:10:06 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:06 jenkins-slave bash[14699]: INFO: Remote identity confirmed: da:ca:7a:5a:1e:ac:df:56:81:96:8a:d7:71:d9:5e:4c
Oct 22 22:10:06 jenkins-slave bash[14699]: Oct 22, 2021 10:10:06 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:06 jenkins-slave bash[14699]: INFO: Connected

When the connection is established to the Jenkins master, here is what it looks like.

With this method, it survives restarting the host.

If you try to list the services on the host with the following command, you will see the item in the list.

sudo systemctl list-units --type=service

How to Create Jenkins Slave on Linux

Most of the articles I find on creating a permanent Jenkins slave on Linux requires the slave node to be exposed to public Internet. I want the Linux slave to be pinging Jenkins master just like Windows service. Here is the way I came up with.

Install Prerequisites

  • Java (sudo dnf install java-11-openjdk.x86_64)
    • Check if the Java has been installed. (java -version)
      Result:
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

Add a Permanent Node

Login to Jenkins master and click Manage Jenkins -> Manage Nodes and Clouds. Click New Node. And then give the node a name (like linux-node), select Permanent Agent and click OK.

And then, click Save button. If you navigate to the node that you just created, you should see something like…

We will take copy this line.

java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir ""

Now, we’ll have to download agent.jar from the Jenkins master and upload the file to the slave machine. Just click on the agent.jar link to download it.

I have the agent.jar file in Downloads directory, so here is the command to upload the agent.jar file to the slave machine.

scp ./Downloads/agent.jar hiriumi@192.168.1.29:~

Now, ssh into the slave machine.

ssh hiriumi@192.169.1.29

Trust SSL Certificate

If your Jenkins master has SSL implemented, it’s a good practice to trust the SSL certificate. Here is how you can download the certificate on your slave machine.

openssl s_client -showcerts -connect jenkins.hayato-iriumi.net:443 < /dev/null | openssl x509 -outform DER > jenkins.hayato-iriumi.net.cer

Now use keytool to trust it.

sudo keytool -trustcacerts \
-keystore "/etc/java/java-11-openjdk/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/lib/security/cacerts" \
-storepass changeit -alias jenkins -import -file \
"/home/hiriumi/jenkins.hayato-iriumi.net.cer"

Create a Script File and Execute

Paste the Java command you get from the node page in to slave.sh

java -jar agent.jar \
-jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp \
-secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir "./slave"

Make sure slave.sh is executable by adding execute flag on the file.

chmod +x slave.sh

If you execute the slave.sh file, it starts to communicate with the Jenkins master and starts to serve as one of the Jenkins slave.

./slave.sh

Once the connection is successful, you will see something like the following.

Now, this method does not survive restarting the slave machine. Now that the communication is successful, I will look into making this script a daemon.