How to Install htop on Oracle Linux 7

I wrote an article on how to install htop on Oracle Linux before. Thanks to Markus, I learned that installing htop is just a matter of enabling a repo on Oracle Linux 8. I have a Oracle Linux 7 host that I use for a customer and I wanted to install htop on it. I tried to look for epel repo in /etc/yum.repos.d/oracle-linux-ol7.repo but I could not find it. So the only option for me is to add the epel repo under /etc/yum.repo.d

I looked for EPEL repo for Oracle Linux 7 and added the following in /etc/yum.repos.d/oracle-epel-ol7.repo

[ol7_developer_EPEL]
name=Oracle Linux $releasever EPEL Packages for Development ($basearch)
baseurl=https://yum$ociregion.$ocidomain/repo/OracleLinux/OL7/developer_EPEL/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

Then run the following command.

sudo yum update
sudo yum install htop

Then, you get to install htop on Oracle Linux 7. 🙂

Troubleshooting Dockerized Blog

This blog is a dockerized WordPress blog. I noticed that my blog site was down this morning. I couldn’t even ssh into the host. I thought it was hacked somehow. After poking it around, I got it back up and running. Here is the things I did to get it back up.

  1. When I did ping hayato-iriumi.net, I got response back.
  2. After a while, I could hit the website but it wasn’t connecting to the database.
  3. I couldn’t even ssh into the host, so I restarted it.
  4. I was able to ssh into it now, so I checked the running containers with the following command.
    docker ps -a
  5. I noticed that NGINX container was failing because it could not start because port 80 was already in use.
  6. Checked which process was using port 80 with the following command.
    sudo netstat -pna | grep 80
  7. It turned out that another instance of NGINX was hogging the port. I stopped it and disabled it with the following command.
    sudo systemctl stop nginx
    sudo systemctl disable nginx
    sudo apt remove nginx
  8. I’m not sure what installed the instance of NGINX.
  9. Restarted the host.
  10. The site came back up.

I am seeing some errors in journalctl so something else may have caused the issue. This is a very common troubleshooting for Linux users but you should know where to look to troubleshoot Linux hosted service. I may rebuild this blog host again just in case it might have been hacked.

Advanced map with Python

As I was writing Python code using map, I came across an issue. When I ran the following code, I came across an error.

import unittest


class test(unittest.TestCase):
    def func1(self, x, y):
        return x ** y

    def test_map(self):
        a = [1, 2, 3, 4]
        results = map(self.func1, a, 2)
        print(results)

I basically wanted to pass 2 for the y parameter of func1 instead of another list. Here is the error I got.

FAILED (errors=1)

Error
Traceback (most recent call last):
  File "C:\Users\hiriu\dev\hoge\test.py", line 10, in test_map
    results = map(self.func1, a, 2)
TypeError: 'int' object is not iterable

Right, the number 2 is not a collection and is not iterable. How do I solve this problem? I searched the web and I found the following solution.

import unittest
import functools

class test(unittest.TestCase):
    def func1(self, x, y):
        return x ** y

    def test_map(self):
        a = [1, 2, 3, 4]
        results = map(functools.partial(self.func1, y=2), a)
        print(list(results))

By using the functools.partial, you get to pass a fixed value to the portion of the function. Here is the output.

[1, 4, 9, 16]

Process finished with exit code 0

Update:

I found an easier way to map a fixed parameter. Here is the example. It’s more readable and maintainable.

import unittest
from itertools import repeat


class test(unittest.TestCase):
    def func1(self, x, y):
        return x ** y

    def test_map(self):
        a = [1, 2, 3, 4]
        results = map(self.func1, a, repeat(2))
        print(list(results))

map Function in Python

map function in Python is a convenient way to execute function for a collection. Let’s see an example that does not use map function.

class playground(unittest.TestCase):
    def pow(self, n):
        return n**n

    def test_pow(self):
        numbers = range(10)
        for number in numbers:
            result = self.pow(number)
            print(result)

Output:

1
1
4
27
256
3125
46656
823543
16777216
387420489

The example above just executes the pow function sequentially for every item in the integer list.

If you use a map function, the code becomes concise and easier to manage. It might be a little confusing but if you get used to it, it’s not too bad.

class playground(unittest.TestCase):
    def pow(self, n):
        return n**n

    def test_map(self):
        numbers = range(10)
        results = map(self.pow, numbers)
        print(list(results))

Output:

[1, 1, 4, 27, 256, 3125, 46656, 823543, 16777216, 387420489]

I didn’t know 0^0 was 1… I knew n^0 was always 1 but… Interesting. 🙂

Batch Processing with Python with Multithreading (Improved)

I wrote an article on how to do batch processing with multithreads in Python last week but there are things that my sample code wasn’t handling.

  • Handle results from the threaded function.
  • Handle exceptions from the threaded function.

With these 2 points in mind, I rewrote the sample code.

from concurrent.futures import ThreadPoolExecutor
from concurrent import futures
import time


def task(n):
    print(f"processing {n}")
    if n % 5 == 0:
        raise Exception("It is divisible by 5")
    time.sleep(1)
    return True


def main():
    print("Starting ThreadPoolExecutor")
    tasks = []
    fs = []
    for i in range(23):
        tasks.append(task)

    with ThreadPoolExecutor(max_workers=5) as executor:
        for i, t in enumerate(tasks):
            future = executor.submit(t, i)
            fs.append(future)
        results = futures.wait(fs)
    for result in results.done:
        if result.done():
            print(f"{result.done()}")
        if result.exception() is not None:
            print(f"Handle exception here: {result.exception()}")


if __name__ == '__main__':
    start_time = time.time()
    main()
    elapsed_time = time.time() - start_time
    print(f"Took {elapsed_time}")

Here is the output:

Starting ThreadPoolExecutor
processing 0
processing 1
processing 2
processing 3
processing 4
processing 5
processing 6
processing 7
processing 8
processing 9
processing 10
processing 11
processing 12
processing 13
processing 14
processing 15
processing 16
processing 17
processing 18
processing 19
processing 20
processing 21
processing 22
True
Handle exception here: It is divisible by 5
True
True
True
Handle exception here: It is divisible by 5
True
True
Handle exception here: It is divisible by 5
True
True
True
True
True
True
True
True
True
True
Handle exception here: It is divisible by 5
True
True
True
True
True
True
Handle exception here: It is divisible by 5
True
Took 4.017247915267944

This way, you can handle situations where you are expecting certain results from the threaded function and also exception situation. The previous sample did not have any of those, so this sample is a better one. Also it is easier to specify the number of concurrent threads.

Uploading Backup File to OCI’s Object Storage via Jenkins

I have had a need to upload a zip file for backup from a Windows agent to Oracle Cloud Infrastructure’s Object Storage. Here is what I did.

Installed OCI CLI for Windows. Please follow this link to install it on Windows. Then, Install Jenkins slave on the same machine. I have a step by step instruction on how to do it. Once you install it, make sure to change the account to run the slave as to the account you used to install OCI CLI. Otherwise, it won’t work.

On the Jenkins job, using Compress-Archive Cmdlet, you can zip up some directories into a zip file.

Compress-Archive -Path $zipPaths -DestinationPath $zipFile

Please note that Compress-Archive has a limitation of 2GB. I heard that it’s the limitation of the underlining API.

Now that you have the zip file, you can upload it to Object Storage like the following.

oci os object put -bn backup --file $zipFile -ns "yournamespace" `
	--parallel-upload-count 5 --part-size 20 --verify-checksum

I am recommending this method to a customer because Object Storage is a relatively cheap and secure storage on OCI. It also supports retention duration and also replication. Great feature for relatively reasonable service.

How to install Docker and Docker Compose on Oracle Linux 7

I have a need to install Docker and Docker Compose on Oracle Linux 7. Here is my note for future reference.

sudo yum -y update
sudo yum install -y docker-engine
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER

Now, logout and log back in and execute a docker command to see if you don’t need sudo to execute it.

docker ps -a

Now install Docker Compose.

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Now check if docker-compose was successfully installed.

docker-compose -v

How to Set Default User for WSL (Ubuntu)

I have Ubuntu for WSL (Windows Subsystem for Linux). I’m not sure how it happened, but when I started the terminal, it started to default to the root user. I wanted to default the user to the one that’s not the root user.

ubuntu config --default-user hiriumi

Make sure to run it either from the Windows command line or PowerShell. Next time when you open Ubuntu terminal, it goes straight to the user you specified.

Missed Demo for the Japanese Students I Gave Speech to

I had an opportunity to give a speech to students who are looking to become engineers in Japan last Friday. It was great to meet them though it was online. They seemed eager to learn what takes to be engineers. I was honored to give speech to them. They were also learning English, so this may be good for them.

Due to my unpreparedness on my side, I missed one demo. I was basically compiling Java code and decomplie it with IntelliJ. I would like to show how it can be done here in this blog for their view.

First, create a text file (helloworld.java) with the following code. It’s just a simple hello world program in Java.

public class helloworld {
    public static void main(String args[])
    {
        System.out.println("Hello World");
    }
}

Once you have the file, compile it like the following from your terminal (command line).

javac helloworld.java

The javac (Java compiler) compiles the text file to Java byte code (helloworld.class). You can execute the hello world program like the following.

java helloworld

Output:

Hello World

When you open helloworld.class file with IntelliJ, you can decompile it.

Decompilation is not really a useful technique anymore because of the current trend of open source but if the source code is closed but you want to learn how the Java program works, it’s still an interesting technique to use especially while you are learning how program works.

I believe JetBrain provides students with free license so you may be able to use the IDEs for free.

The reason why I could not find the helloworld.class file at the time of demo was because I was using WSL 2 on Windows. I had the file on the Linux side of the OS but I had forgotten to copy the file on the Windows side. I’m so sorry about it.

There was so much more I wanted to talk to everyone about but our time was limited. I wish all of you successful careers and bright future. 🙂

Batch Processing with Python Multithreading

I want to execute 5 threads at a time but I have 23 things I want to run in total. Here is the code I came up with.

import threading
from multiprocessing import Pool
import time


class MyThread(threading.Thread):
    def run(id):
        print(f"thread {id}")
        time.sleep(3)


if __name__ == '__main__':
    start_time = time.time()
    threads = []
    batch_size = 5
    for i in range(23):
        threads.append(MyThread.run)

    batch_index = 1
    thread_index = 1
    while len(threads) > 0:
        pool = Pool()
        print(f"Batch {batch_index}")
        for j in range(batch_size):
            if threads:
                t = threads.pop()
                pool.apply_async(t, (thread_index,))
                thread_index += 1
        pool.close()
        pool.join()
        batch_index += 1

    elapsed_time = time.time() - start_time
    print(f"Took {elapsed_time}")

Output:

Batch 1
thread 1
thread 2
thread 3
thread 4
thread 5
Batch 2
thread 6
thread 7
thread 8
thread 9
thread 10
Batch 3
thread 11
thread 12
thread 13
thread 14
thread 15
Batch 4
thread 16
thread 17
thread 18
thread 19
thread 20
Batch 5
thread 21
thread 22
thread 23
Took 15.523964166641235

Each thread takes 3 seconds. If I executed the function sequentially, it would take at least 60 seconds but with the 5 threads at a time, it ended up with only 12 seconds. This is a huge improvement.

Another thing to note is that I declared threads variable as list. List has pop() method in Python. This returns the item (thread object in this case) and removes it from the list. This way, you can use the list to keep track on the threads.

I also needed to add if threads: to check if the threads still has items in case the number of threads is not divisible by 5. If I had 23 threads I want to execute, it attempts to execute 20, 21, 22, 23, 24, 25. 24 and 25 do not exist in the list so it errors out. To prevent such a situation, the if statement is necessary.