How to Generate Free SSL by Let’s Encrypt

I attempted to use Certbot by Let’s Encrypt for free SSL certificate for my subdomain using a Docker container but it was never successful, so I left it alone for a while. The DNS service that I use provides SSL certs but it charges me extra for subdomain. There are other services such as SSL for Free but they either limit the number of certificates or they charge for subdomains or they want me to pay like $10 per month for wildcard certificate. That’s pretty steep considering the host is being used only privately. The alternative is to create your own private CA authority and issue SSL cert and have the root CA cert on the machines that you use the hosts.

But this morning, I figured out a way to generate the free SSL cert for my subdomain using Certbot. I will write about what worked for me.

I followed this instruction to install snapd on Ubuntu.

First, remove certbot if installed by apt.

sudo apt-get remove certbot

Install Certbot.

sudo snap install --classic certbot

Prepare the Certbot command.

sudo ln -s /snap/bin/certbot /usr/bin/certbot

Now prepare your NGINX server to accept HTTP traffic for acme challenge.

Edit nginx.conf to accept HTTP (port 80).

   server {
      location / {
          root   /var/www/html;
          index  index.html index.htm;
      }
      listen       80 default_server;
      listen       [::]:80 default_server;
      server_name  _;

If you have the HTTP redirect to HTTPS, comment the line out.

# return 301 https://jenkins.hayato-iriumi.net$request_uri;

Now back to Certbot, execute the following command to start to issue your ssl cert.

sudo certbot certonly -a manual --rsa-key-size 4096 --email hiriumi@gmail.com -d jenkins.hayato-iriumi.net

You will see an output like the following.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for jenkins.hayato-iriumi.net

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:

s8wH1u2z00ePejV4hyy4y3CTyW3pYvrFgxwxwsPVdd8.O3THIaz5tgLf8NuxfBYw8FZfrdQNf_Y_1U--J0PsgqQ

And make it available on your web server at this URL:

http://jenkins.hayato-iriumi.net/.well-known/acme-challenge/s8wH1u2z00ePejV4hyy4y3CTyW3pYvrFgxwxwsPVdd8

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Now you should create the file with the data specified in the output. Once you have that hit Enter to get the cert generated.

Lastly, when the cert generation is successful, you will see the output like the following.

Now switch to the root user by executing…

sudo -i

The cert files are at /etc/letsencrypt/archive/jenkins.hayato-iriumi.net

Copy cert1.pem and privkey1.pem to the directory where you would like to store your SSL files. In my ssl.conf file, I have specified the cert files like the following.

server {
    server_name jenkins.hayato-iriumi.net;
    listen 443 ssl;
    ssl_certificate /etc/nginx/conf.d/ssl/cert1.pem;
    ssl_certificate_key /etc/nginx/conf.d/ssl/privkey1.pem;
    client_max_body_size 3000m;

Now unc0mment the line in ssl.conf to redirect HTTP to HTTPS traffic. Once you restart your NGINX, NGINX starts to service the traffic in SSL.

I’m sure there are ways to automate this and I am thinking of exploring the way to do it but it works well for now.

How to Make Linux Jenkins Slave a Daemon

I wrote an article on creating a Jenkins slave on Linux. The method was to just create a bash script file that requires to be executed by hand. And it wouldn’t survive restarting the host, so what I need to do is to make the script a daemon (service).

Here is what I did before configuring the daemon.

  1. Provision a Ubuntu host on Azure (it doesn’t matter where you provision the host as long as your Jenkins master on the public Internet and secured).
  2. Update the system. (sudo apt update && sudo apt upgrade)
  3. Install OpenJDK. (sudo apt install openjdk-11-jre-headless)
  4. Open port 50000 (Inboud and Outbound) to the host. I am opening all protocols.

Creating a Daemon

We will create a script at home directory first. To contain everything for Jenkins slave, I am creating /home/azureuser/jenkins-slave directory. You can create jenkins-slave or whatever the directory name you would like anywhere.

Then create slave.sh in /home/azureuser/jenkins-slave directory with the following content. Change the URL and the secret acccording to the Jenkins node you have created on Jenkins master. Make the script executable by executing chmod +x slave.sh.

java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir "./slave"
exit 0

Also make sure you download agent.jar from Jenkins master to /home/azureuser/jenkins-slave directory. Also user opessl and keytool to trust the SSL cert. You can refer to the previous blog article on how to use keytool.

Now create /etc/systemd/system/jenkins-slave.service file with the following content. sudo vim /etc/systemd/system/jenkins-slave.service

[Unit]
Description=JenkinsSlave

[Service]
User=azureuser
WorkingDirectory=/home/azureuser/jenkins-slave
ExecStart=/bin/bash /home/azureuser/jenkins-slave/slave.sh
Restart=always

[Install]
WantedBy=multi-user.target

Enable the daemon with the following command.

sudo systemctl enable jenkins-slave.service

Now start the daemon with the following command.

sudo systemctl start jenkins-slave.service

Check the status of it.

sudo systemctl status jenkins-slave.service

Result:

● jenkins-slave.service - JenkinsSlave
     Loaded: loaded (/etc/systemd/system/jenkins-slave.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-22 22:10:03 UTC; 8min ago
   Main PID: 14698 (bash)
      Tasks: 19 (limit: 486)
     Memory: 103.3M
     CGroup: /system.slice/jenkins-slave.service
             ├─14698 /bin/bash /home/azureuser/jenkins-slave/slave.sh
             └─14699 java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc40>

Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Handshaking
Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Connecting to jenkins.hayato-iriumi.net:50000
Oct 22 22:10:05 jenkins-slave bash[14699]: Oct 22, 2021 10:10:05 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:05 jenkins-slave bash[14699]: INFO: Trying protocol: JNLP4-connect
Oct 22 22:10:06 jenkins-slave bash[14699]: Oct 22, 2021 10:10:06 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:06 jenkins-slave bash[14699]: INFO: Remote identity confirmed: da:ca:7a:5a:1e:ac:df:56:81:96:8a:d7:71:d9:5e:4c
Oct 22 22:10:06 jenkins-slave bash[14699]: Oct 22, 2021 10:10:06 PM hudson.remoting.jnlp.Main$CuiListener status
Oct 22 22:10:06 jenkins-slave bash[14699]: INFO: Connected

When the connection is established to the Jenkins master, here is what it looks like.

With this method, it survives restarting the host.

If you try to list the services on the host with the following command, you will see the item in the list.

sudo systemctl list-units --type=service

How to Create Jenkins Slave on Linux

Most of the articles I find on creating a permanent Jenkins slave on Linux requires the slave node to be exposed to public Internet. I want the Linux slave to be pinging Jenkins master just like Windows service. Here is the way I came up with.

Install Prerequisites

  • Java (sudo dnf install java-11-openjdk.x86_64)
    • Check if the Java has been installed. (java -version)
      Result:
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

Add a Permanent Node

Login to Jenkins master and click Manage Jenkins -> Manage Nodes and Clouds. Click New Node. And then give the node a name (like linux-node), select Permanent Agent and click OK.

And then, click Save button. If you navigate to the node that you just created, you should see something like…

We will take copy this line.

java -jar agent.jar -jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp -secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir ""

Now, we’ll have to download agent.jar from the Jenkins master and upload the file to the slave machine. Just click on the agent.jar link to download it.

I have the agent.jar file in Downloads directory, so here is the command to upload the agent.jar file to the slave machine.

scp ./Downloads/agent.jar hiriumi@192.168.1.29:~

Now, ssh into the slave machine.

ssh hiriumi@192.169.1.29

Trust SSL Certificate

If your Jenkins master has SSL implemented, it’s a good practice to trust the SSL certificate. Here is how you can download the certificate on your slave machine.

openssl s_client -showcerts -connect jenkins.hayato-iriumi.net:443 < /dev/null | openssl x509 -outform DER > jenkins.hayato-iriumi.net.cer

Now use keytool to trust it.

sudo keytool -trustcacerts \
-keystore "/etc/java/java-11-openjdk/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/lib/security/cacerts" \
-storepass changeit -alias jenkins -import -file \
"/home/hiriumi/jenkins.hayato-iriumi.net.cer"

Create a Script File and Execute

Paste the Java command you get from the node page in to slave.sh

java -jar agent.jar \
-jnlpUrl https://jenkins.hayato-iriumi.net/computer/linux%2Dnode/jenkins-agent.jnlp \
-secret 136fa14dcc4013727e24c9f1a9b84127d7c7ca0cfa15e22c1e1d4e0140122529 -workDir "./slave"

Make sure slave.sh is executable by adding execute flag on the file.

chmod +x slave.sh

If you execute the slave.sh file, it starts to communicate with the Jenkins master and starts to serve as one of the Jenkins slave.

./slave.sh

Once the connection is successful, you will see something like the following.

Now, this method does not survive restarting the slave machine. Now that the communication is successful, I will look into making this script a daemon.

Installing JetBrains Products on Oracle Linux 8

I am trying to configure Oracle Linux 8 as my spare laptop. I need to install JetBrains products on it. I tried to install JetBrains Toolbox but it wouldn’t work. It’s packaged as AppImage file, so it should be pretty easy but when I execute it, a blank white window shows up and disappears.

So I looked for an alternative way to install JetBrains products. I installed snapd on it with the following command.

sudo dnf install snapd

Then, I searched for the JetBrains products like the following.

snap search jetbrains

Result:

Name                       Version   Publisher   Notes    Summary
pycharm-community          2021.2.2  jetbrains✓  classic  PyCharm Community Edition
phpstorm                   2021.2.3  jetbrains✓  classic  PhpStorm
pycharm-professional       2021.2.2  jetbrains✓  classic  PyCharm Professional Edition
intellij-idea-community    2021.2.3  jetbrains✓  classic  Capable & Ergonomic Java IDE
intellij-idea-ultimate     2021.2.3  jetbrains✓  classic  Capable & Ergonomic Java IDE for Enterprise, Web & Mobile Development
webstorm                   2021.2.2  jetbrains✓  classic  WebStorm
datagrip                   2021.2.4  jetbrains✓  classic  DataGrip
clion                      2021.2.3  jetbrains✓  classic  A cross-platform IDE for C and C++
pycharm-educational        2021.2.2  jetbrains✓  classic  Easy and Professional Tool to Learn & Teach Programming with Python
rubymine                   2021.2.3  jetbrains✓  classic  The Most Intelligent Ruby and Rails IDE
space                      2021.2.0  jetbrains✓  -        Desktop Application for JetBrains Space
rider                      2021.2.2  jetbrains✓  classic  A fast & powerful cross-platform .NET IDE
goland                     2021.2.3  jetbrains✓  classic  GoLand
intellij-idea-educational  2021.2.2  jetbrains✓  classic  IntelliJ IDEA Educational Edition
kotlin                     1.5.31    jetbrains✓  classic  Command line Kotlin compiler

The first application I want to install is PyCharm, so I ran the following command to install it.

snap install pycharm-professional

If you search pycharm in your GNOME UI, you will be able to start to use it.

I still would like to use JetBrains’ Toolbox so I posted my question in their support forum to resolve

Before I posted it, I did a fair bit of research. Toolbox is packaged as AppImage, so you can check the command options like the following.

./jetbrains-toolbox --appimage-help

I learned that you can even extract files from the image like the following.

./jetbrains-toolbox --appimage-extract

I did digging into the extracted files but I could not find a solution for it. Oh well, I can use JetBrains’ products anyway, so I’m happy for now.

How to Change Java Version on Oracle Linux 8

I had Java 8 on my Oracle Linux 8 machine but I wanted to change it to Open JDK 11. Here is what I did. The following instruction will probably work for RedHat descendants.

First, I had to find a way to install Open JDK. I ran the following command to get the right package information.

dnf search java-11

Here is the result.

java-11-openjdk.x86_64 : OpenJDK 11 Runtime Environment
java-11-openjdk.src : OpenJDK 11 Runtime Environment
java-11-openjdk-demo.x86_64 : OpenJDK 11 Demos
java-11-openjdk-devel.x86_64 : OpenJDK 11 Development Environment
java-11-openjdk-headless.x86_64 : OpenJDK 11 Headless Runtime
                                : Environment
java-11-openjdk-javadoc.x86_64 : OpenJDK 11 API documentation
java-11-openjdk-javadoc-zip.x86_64 : OpenJDK 11 API documentation
                                   : compressed in a single archive
java-11-openjdk-jmods.x86_64 : JMods for OpenJDK 11
java-11-openjdk-src.x86_64 : OpenJDK 11 Source Bundle
java-11-openjdk-static-libs.x86_64 : OpenJDK 11 libraries for static
                                   : linking

To install Open JDK 11, run the following command.

sudo dnf install java-11-openjdk.x86_64

Now I want to switch my default Java version to the one I just installed.

sudo alternatives --config java

alternatives lists the available Java installations and it gives you options to select from.

Once you make the selection, check the default Java version like the following.

java -version

Result:

openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7-LTS, mixed mode, sharing)

New Dell Laptop

I have received a new Dell laptop from the company I work for. This is for a project that has to do with Oracle Linux. I feel like Christmas already. I really appreciate the opportunity to work on it.

I will write about some of the things I can expose here in my blog. My goal is to create an Oracle Linux environment where I can do my dev work with Cinnamon DE on it!

The Right Ports

nmap is a very useful tool to check the open ports. Yeah, bad guys could use it too but you want to make sure the host you have exposed to the Internet has the minimal number of ports open. When I scan my own host that hosts this blog site like nmap hayato-iriumi.net, I get the following output.

Starting Nmap 7.80 ( https://nmap.org ) at 2021-09-23 18:27 PDT
Nmap scan report for hayato-iriumi.net (150.136.86.255)
Host is up (0.097s latency).
Not shown: 996 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
8080/tcp open  http-proxy

I have the 4 ports open intentionally for my own management of my site. This makes me think what I should actually do down the road. I should close 22 and use a bastion to SSH into the host for management.

8080 is open for another management reason. Obviously, 80 is open for HTTP connection which redirects traffic to 443 (SSL, HTTPS). If you do nmap google.com, you can see port 80 and 443 are open to public as well.

Starting Nmap 7.80 ( https://nmap.org ) at 2021-09-23 18:28 PDT
Nmap scan report for google.com (142.250.69.206)
Host is up (0.026s latency).
Other addresses for google.com (not scanned): 2607:f8b0:400a:805::200e
rDNS record for 142.250.69.206: sea30s08-in-f14.1e100.net
Not shown: 998 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Here is the first paragraph of nmap man page. It tells you what it’s supposed to do.

Nmap (“Network Mapper”) is an open source tool for network exploration and security auditing. It was designed to rapidly scan large networks, although it works fine against single hosts. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. While Nmap is commonly used for security audits, many systems and network administrators find it useful for routine tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime.

man page for nmap is pretty big so there must be a lot we can do with this tool.

How to Install htop on Oracle Linux 7

I wrote an article on how to install htop on Oracle Linux before. Thanks to Markus, I learned that installing htop is just a matter of enabling a repo on Oracle Linux 8. I have a Oracle Linux 7 host that I use for a customer and I wanted to install htop on it. I tried to look for epel repo in /etc/yum.repos.d/oracle-linux-ol7.repo but I could not find it. So the only option for me is to add the epel repo under /etc/yum.repo.d

I looked for EPEL repo for Oracle Linux 7 and added the following in /etc/yum.repos.d/oracle-epel-ol7.repo

[ol7_developer_EPEL]
name=Oracle Linux $releasever EPEL Packages for Development ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL7/developer_EPEL/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

Then run the following command.

sudo yum update
sudo yum install htop

Then, you get to install htop on Oracle Linux 7. 🙂

Troubleshooting Dockerized Blog

This blog is a dockerized WordPress blog. I noticed that my blog site was down this morning. I couldn’t even ssh into the host. I thought it was hacked somehow. After poking it around, I got it back up and running. Here is the things I did to get it back up.

  1. When I did ping hayato-iriumi.net, I got response back.
  2. After a while, I could hit the website but it wasn’t connecting to the database.
  3. I couldn’t even ssh into the host, so I restarted it.
  4. I was able to ssh into it now, so I checked the running containers with the following command.
    docker ps -a
  5. I noticed that NGINX container was failing because it could not start because port 80 was already in use.
  6. Checked which process was using port 80 with the following command.
    sudo netstat -pna | grep 80
  7. It turned out that another instance of NGINX was hogging the port. I stopped it and disabled it with the following command.
    sudo systemctl stop nginx
    sudo systemctl disable nginx
    sudo apt remove nginx
  8. I’m not sure what installed the instance of NGINX.
  9. Restarted the host.
  10. The site came back up.

I am seeing some errors in journalctl so something else may have caused the issue. This is a very common troubleshooting for Linux users but you should know where to look to troubleshoot Linux hosted service. I may rebuild this blog host again just in case it might have been hacked.