Building Bash Muscle (6) sed

What if you want to replace text in a file? Use sed. I will give the simplest example. First, I’m creating a text file that has “hello world” in it.

echo 'hello world' > test.txt

I’m going to replace “hello” with “hi” and create a new file called test_new.txt.

cat test.txt | sed 's/hello/hi/g' > test_new.txt

You can accomplish the same operation without using pipe.

sed 's/hello/hi/g' test.txt > test_new.txt

If you want to just replace the text without creating a new file…

sed -i 's/hello/hi/g' test.txt

If you want to create a backup file, you can add .bak right after -i option like the following example.

sed -i.bak 's/hello/hi/g' test.txt

This way, the original content of test.txt is preserved in test.txt.bak.

Integrating Artifactory with Active Directory LDAP

In my previous post, I went over how to spin up Artifactory OSS on Docker. I will go over how to integrate it with LDAP this time. This was a bit of struggle because the examples that I found online were not so clear because they try to hide the information. I’m going to expose mine because it’s just my personal lab environment and I’m hoping that it will help some people out there.

Basic Data

  • Window Server 2019 already installed and configured.
  • Created UsersOU as the very basic organizational unit. I have all my users in it.
  • svcldap is the account I’m using for the binding account.
  • svcldap is configured to be able to read the OU information. (I have not gone over this one in my blog, but I will work on it later.)
  • OU (Organizational Unit) to use is UsersOU in this example. All users exists in this OU.
  • The domain is homenet.iriumi.ad.

LDAP Configuration in Artifactory

With the information above, follow the instruction below to have the users authenticate against LDAP on Windows Server 2019.

First, login to Artifactory as an admin and navigate to LDAP.

Here is the detailed information on the configuration with my actual lab environment.

LDAP Settings

    • Enabled: check
    • Settings name: homenet (or whatever you want. It’s just a label.)
    • LDAP URL: ldap://pdc01.homenet.iriumi.ad:389/dc=homenet,dc=iriumi,dc=ad
      SSL is not enabled in my environment, so 389. If it’s enabled, it would be 686.
    • Auto Create Artifactory Users: check
    • Allow Created Users Access To Profile Page: check
    • User DN Pattern: uid={0},ou=UsersOU
    • Email Attribute: mail
    • Search Filter: sAMAccountName={0}
    • Search Base: ou=UsersOU
    • Secure LDAP Search: check
    • Search Sub-tree: check
    • Manager DN: CN=svcldap,OU=UsersOU,DC=homenet,DC=iriumi,DC=ad 
    • Manager Password: The password for svcldap

Here is the actual image of the settings.

Once the Test LDAP Connection is successful, use your credential to login to Artifactory. Logging in to Artifactory creates the account in it. Once that’s done, logout and log back in as admin. Make your user an admin and then log back into it as the LDAP user. You’re all good to go!

Artifactory on Docker

Artifactory is a binary store solution in short. You could use it as a storage to store artifacts as a result of builds. There are paid editions but I’m going with open source edition for my experiments for now.

Prerequisites

I’m going to assume you already the following prerequisites.

Open Port

Execute the following commands to open port 80 for Artifactory UI.

$ sudo firewall-cmd --list-all --add-port=80/tcp --permanent --zone=public
$ sudo firewall-cmd --reload

If you want to accept any traffic trying to access HTTP to the VM, you can alternatively run the following command.

sudo firewall-cmd --add-service=http --permanent --zone=public
sudo firewall-cmd --reload

Create Directory and chown

We will persist the data at /data/artifactory so let’s create the directory.

sudo mkdir -p /data/artifactory

In this example, we will run the container as svcdocker that I have in my Active Directory, so I’m going to change the ownership of the directory. Let the user you want to use own the directory.

sudo chown -R svcdocker:'domain users' /data/artifactory

docker-compose

Copy and paste the text blow in docker-compose.yaml file and place it in your home directory or wherever. One thing to note is I have user as a number. I have “svcdocker” as a domain user but Docker doesn’t seem to be able to figure out “svcdocker” so I gave its id in docker-compose.yaml file. You can figure out the account’s id by executing “id svcdocker”.

version: '3'
services:
  artifactory:
    image: docker.bintray.io/jfrog/artifactory-oss:latest
    container_name: artifactory
    user: '1985601112'
    ports:
     - 80:8081
    volumes:
     - /data/artifactory:/var/opt/jfrog/artifactory
#    environment:
#     - EXTRA_JAVA_OPTIONS=-Xms512m -Xmx4g
    restart: always
    ulimits:
      nproc: 65535
      nofile:
        soft: 32000
        hard: 40000

Once the YAML data is saved in a file, just run the following command.

docker-compose up -d

After a while, Artifactory is ready for you to access its UI. Just browse to the host with your browser and you will be prompted to enter initial password and username. Artifactory’s initial username and password is “admin” and “password” respectively. Once you enter the credential, you will be welcomed with the image below.Lastly, make sure to change the admin password and do whatever to secure Artifactory.

How to Install docker compose

Switching the gear a little bit here to get ready to install Artifactory on Docker.

Docker compose is a mechanism to start Docker containers with the predefined parameters. It’s kind of like bash script file that executes another executable with parameters but it’s in yaml format. Let’s see how we can install it on Oracle Linux 7.7 which I’m using as my main distribution now. Oracle Linux is very similar to CentOS with more security enhancements.

I’m going to assume that you have already installed Docker. Docker installation instruction is in this blog entry.

Docker Compose Installation

  1. Open terminal and ssh into the machine where you have Docker installed.
  2. As of October of 2019, the latest version is 1.24.1. You can find out the docker compose releases here.
  3. Execute the following command to install it.
sudo curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose

Once it’s done, check if docker-compose works on your machine by executing the following command.

$ docker-compose -v

It should show the version of docker-compose you just installed.

docker-compose version 1.24.1, build 4667896b

I will cover how to create a docker-compose file in the next blog entry while I get Artifactory to work on Docker.

Commands

Just did find ./ -printf "%f\n" under /bin and got the list of available commands. There are 177 of them. There are commands I don’t know. To build bash muscle efficiently, I should investigate the ones I don’t know. This will be a bit of journey but I will try to remember the ones that are actually useful. At the same time, it’s good to go through all of them just in case.

kbd_mode
openvt
mt
su
systemctl
systemd-escape
ntfssecaudit
ping
plymouth
lesskey
lsmod
readlink
mkfs.btrfs
systemd-tty-ask-password-agent
chgrp
umount
loadkeys
stty
systemd-notify
systemd-machine-id-setup
bzless
btrfs-image
zcat
sh.distrib
bzdiff
unicode_start
which
nc.openbsd
echo
systemd-ask-password
lessfile
mkdir
zdiff
dmesg
setfacl
journalctl
ping6
efibootdump
btrfs-zero-log
uncompress
zfgrep
zforce
ntfstruncate
findmnt
wdctl
bzcmp
ed
ntfsfix
grep
lesspipe
mountpoint
udevadm
zless
chacl
mount
systemd-tmpfiles
lessecho
btrfs-find-root
tar
vdir
lsblk
run-parts
ntfsinfo
gzip
efibootmgr
ulockmgr_server
dir
true
uname
bash
keyctl
ps
zcmp
false
ypdomainname
btrfs-debug-tree
bunzip2
rm
bzfgrep
mknod
pidof
open
bzip2recover
nc
zegrep
bzexe
tempfile
zmore
mktemp
date
ntfsfallocate
znew
dnsdomainname
chown
fsck.btrfs
fusermount
ntfs-3g.probe
brltty
systemd-inhibit
netstat
ntfsusermap
gzexe
nano
setfont
bzcat
ntfswipe
sync
systemd
whiptail
dd
fgrep
mt-gnu
ln
less
more
rbash
egrep
bzmore
kmod
bzgrep
static-sh
zgrep
btrfs
ls
bzegrep
lowntfs-3g
ntfsrecover
ntfsmove
ntfscmp
cp
fuser
sh
ntfscluster
btrfsck
rnano
fgconsole
df
ping4
gunzip
dumpkeys
rmdir
ntfsls
ss
red
bzip2
systemd-sysusers
cat
login
hostname
dash
pwd
getfacl
sleep
domainname
busybox
kill
netcat
setupcon
btrfs-map-logical
chvt
cpio
chmod
archdetect
ntfscat
touch
btrfs-select-super
systemd-hwdb
ntfs-3g
loginctl
btrfstune
networkctl
mv
nisdomainname
hciconfig
sed
ip

Building Bash Muscle (5)

Let’s talk about scp this time. What is scp? It’s “secure copy (remote file copy program)” It lets you “copy files between hosts on a network.” I’m going to write an example to upload a file to a server.

$ scp file_to_upload user@remotehost:/path/on/remotedir/file_to_upload

You can upload multiple files like the following example.

$ scp file1 file2 user@remotehost:/path/on/remotedir

Alternatively, you could specify -r option to copy the entire directory recursively. Here is what the man page says. “Recursively copy entire directories. Note that scp follows symbolic links encountered in the tree traversal.”

scp works well when you are just dealing with Mac or Linux but you may find it not so easy from Windows. I personally used WinSCP to transfer files from Windows to Linux machines.

I just checked Git Bash on Windows and yes scp is available. Git Bash is actually giving us pretty good tools for Windows users. (I’m no longer full time Windows user anymore… as I use Linux Mint as my main desktop.)

That’s it for today. Happy scp’ing!

Building Bash Muscle (4)

I want to focus on file/directory related stuff today. I use these commands pretty much daily but I would like to get my brain organized and possibly learn some new techniques along the way. I’m just going to randomly come up with possibly useful commands.

Where am I?

$ pwd
output: /home/amaterasu48

That was too basic.

List only files in the current directory

$ ls -al | grep '^-'

If I explain what it means, ls is to list directory contents. -l means “use a long listing format”. -a means all entries including the ones that start with . The grep part filters the result. '^-' means the entries that starts with dash. The output looks like the following.

-rw-------  1 amaterasu48 amaterasu48  31956 Sep 30 21:11 .bash_history
-rw-r--r--  1 amaterasu48 amaterasu48    220 Apr  7 15:17 .bash_logout
-rw-r--r--  1 amaterasu48 amaterasu48   4284 May 19 19:56 .bashrc
-rw-r--r--  1 amaterasu48 amaterasu48     23 Sep 18 23:25 .dmrc
-rw-rw-r--  1 amaterasu48 amaterasu48     56 May 19 21:53 .gitconfig
-rw-r--r--  1 amaterasu48 amaterasu48     22 Apr  7 15:17 .gtkrc-2.0
-rw-r--r--  1 amaterasu48 amaterasu48    516 Apr  7 15:17 .gtkrc-xfce
-rw-------  1 amaterasu48 amaterasu48  32422 Sep 29 12:50 .ICEauthority
-rw-------  1 amaterasu48 amaterasu48    144 Oct  1 20:53 .lesshst
-rw-r--r--  1 amaterasu48 amaterasu48    258 Apr  7 17:16 .pam_environment
-rw-r--r--  1 amaterasu48 amaterasu48    807 Apr  7 15:17 .profile
-rw-------  1 amaterasu48 amaterasu48      7 Aug  2 10:53 .python_history
-rw-r--r--  1 amaterasu48 amaterasu48      0 Apr  7 15:30 .sudo_as_admin_successful
-rw-------  1 root        root          8504 Sep 30 21:22 .viminfo
-rw-rw-r--  1 amaterasu48 amaterasu48     14 Sep  7 22:09 .vimrc
-rw-rw-r--  1 amaterasu48 amaterasu48    344 Aug  4 19:01 .wget-hsts
-rw-------  1 amaterasu48 amaterasu48     55 Sep 29 12:50 .Xauthority
-rw-r--r--  1 amaterasu48 amaterasu48    132 Apr  7 17:18 .xinputrc
-rw-------  1 amaterasu48 amaterasu48 186154 Oct  1 20:52 .xsession-errors
-rw-------  1 amaterasu48 amaterasu48 432810 Sep 29 12:24 .xsession-errors.old

How many files are there in the current directory?

We can reuse the previous command the pipe it with wc -l. wc means “print newline, word, and byte counts for each file”. -l is an option that “prints the newline counts.” The command looks like this.

$ ls -al | grep '^-' | wc -l

How many directories are there in the current directory?

Again, applying the previous technique, this is quite easy.

$ ls -la | grep '^d' | wc -l

What is the size of the current directory?

$ du -sh

What if I want to check the size of a specific directory? Just add the full path at the end of the command.

$ du -sh /home

du is estimate file space usage. -s option “displays only a total for each argument.”

Sort by date

There are times when I think “What was modified most recently?” Here is the command for it.

$ ls -lat

The output looks like the following.

total 52
drwxrwxr-x 10 amaterasu48 amaterasu48 4096 Oct  1 23:27 .
-rw-r--r--  1 amaterasu48 amaterasu48 9709 Oct  1 23:27 ort
drwxr-xr-x 72 amaterasu48 amaterasu48 4096 Oct  1 23:26 ..
drwxrwxr-x  7 amaterasu48 amaterasu48 4096 Sep 29 22:42 control-repo
drwxrwxr-x  5 amaterasu48 amaterasu48 4096 Sep 24 21:03 playground
drwxrwxr-x  9 amaterasu48 amaterasu48 4096 Sep 22 09:35 rtl8814au
drwxrwxr-x  3 amaterasu48 amaterasu48 4096 Sep 18 22:28 cmdb-manager
drwxrwxr-x  6 amaterasu48 amaterasu48 4096 Sep 18 11:24 jenkins-python
drwxrwxr-x  5 amaterasu48 amaterasu48 4096 Sep 14 22:02 docker
drwxrwxr-x  5 amaterasu48 amaterasu48 4096 Jul 29 22:07 jenkins-lib
drwxrwxr-x  3 amaterasu48 amaterasu48 4096 Jul 17 08:53 bash-utilities

-t option means “sort by modification time, newest first.”

Find files that contains certain text

$ grep -rli 'test'

If you want to limit the file type, you can add –include like the following example.

sudo grep -rli 'test' --include=\*.txt

OK, that’s it for today. Keep on digging!

Building Bash Muscle (3)

List the files that contains “linux” in their file name. Also ignore the case.

$ ll | grep -i linux

By the way, ll is an alias. Type the following command to see what it really is.

$ type ll

The output shows the following.

ll is aliased to `ls -alF'

Some basic stuff but always nailing the basics is very important in not only software engineering but in anything.

Building Bash Muscle (2)

I’ve been writing quite a bit of PowerShell code. The huge difference between PowerShell and Bash is that when executing a command, PowerShell returns collections of objects not just texts. That said, I still need to build my Bash muscle because Bash is useful after all in Linux world.

Today, I’m going to see what process is listening to which port on the local machine. It’s really critical to know this when you are troubleshooting a service that’s having a problem. Execute the following command.

$ sudo netstat -tlup

-t means –tcp. -u means –udp. -l means –listening and -p means –program. Let me show you the actual screenshot of the result.

As you can see, it lists all the processes (programs) that are listening on the machine. If you grep this result, you will be able to narrow down the result more to find the process that’s listening to certain port. Let’s see what’s up with ssh.

$ netstat -tulp | grep ssh

It’s listening the port “ssh”. Well, ssh usually listens to the port 22 so translate that in your head. I wonder if there is a way to actually show 22… I don’t know right now.

When something is not working, the first thing you should do is if a process is even listening to the expected port. This command will help you troubleshoot any service that may be serving people.

CentOS 8 Installation

I downloaded CentOS 8 and I’ve installed the minimal install on my ESXi. There are some changes from CentOS 7 installation but there weren’t anything that got me confused.

I will paste some screenshot so that you get some feel for what to expect with CentOS 8 installation screen.

Initial screen
Selection
Software Selection. “Server with GUI” is the default option, so if you want to create a server, make sure to select Minimal Install.
Set at least root password here
This is something we never saw in CentOS 7. Just accept the license and move on.

Recap

There is nothing too special in CentOS 8 installation. Just make sure what you want to install in Software Selection before you “Begin Installation.”

I will cover these things later but nmtui, yum, systemctl, hostnamectl are still available. Restarting network got me kind of confused because you have to systemctl restart NetworkManager in CentOS 8. Otherwise, it’s been pretty much the same so far.