Parsing syslog with Bash

I’ve got /var/log/syslog that constantly gets written by my cron job. I want to list all the log that I had cron job write. Here is how grep can help.

# under /var/log
$ grep Windows10 syslog

Here is the result.

Jan 23 00:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 01:00:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 01:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 02:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 02:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 03:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 03:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 04:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 04:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 05:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 05:30:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 06:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 06:30:01 linux-mint amaterasu48: Windows10 VM is already running
<snip>
Jan 23 16:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 16:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 17:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 17:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 18:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 18:30:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 19:00:02 linux-mint amaterasu48: Starting Windows10 VM...
Jan 23 19:00:06 linux-mint amaterasu48: Windows10 VM started successfully
Jan 23 19:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 20:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 20:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 21:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 21:30:02 linux-mint amaterasu48: Windows10 VM is already running

Kinda boring. I want to exclude “Starting Windows 10 VM…” and “Windows10 VM started successfully” lines. Here is how.

grep Windows10 syslog | grep -v -e Starting -e started

Here is the result.

Jan 23 00:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 01:00:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 01:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 02:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 02:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 03:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 03:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 04:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 04:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 05:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 05:30:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 06:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 06:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 07:00:01 linux-mint amaterasu48: Windows10 VM is already running
<snip>
Jan 23 15:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 16:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 16:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 17:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 17:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 18:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 18:30:02 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 19:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 20:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 20:30:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 21:00:01 linux-mint amaterasu48: Windows10 VM is already running
Jan 23 21:30:02 linux-mint amaterasu48: Windows10 VM is already running

Now let’s see how many occurrence there are.

$ grep Windows10 syslog | grep -v -e Starting -e started | wc -l

It simply counts the number of lines of the stdout.

Let’s make it more useful for an engineer’s life. I often need to copy and paste the log in another system where markdown is allowed. This means I could format the stdout and use it to copy and paste it in my markdown documentation. Here is what I can do.

grep Windows10 syslog | grep -v -e Starting -e started | sed 's/^/* /g' | tee ~/tmp/hoge.txt

The output result looks like this.

* Jan 23 00:30:01 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 01:00:02 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 01:30:01 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 02:00:01 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 02:30:01 linux-mint amaterasu48: Windows10 VM is already running
<snip>
* Jan 23 20:30:01 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 21:00:01 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 21:30:02 linux-mint amaterasu48: Windows10 VM is already running
* Jan 23 22:00:01 linux-mint amaterasu48: Windows10 VM is already running

What’s happening here is you format the output using sed. It basically adds an asterisk and a space at the very first part of each line and it also writes the result out to the ~/tmp/hoge.txt using tee command. This way you could go back to the text file to get the formatted text later as well.

Since I have done so much PowerShell scripting in the past, I kind of still miss the concept of everything you get is an object. Bash depends heavily on stdout but it is very fast and efficient.

Author: admin

A software engineer in greater Seattle area

Leave a Reply

Your email address will not be published. Required fields are marked *