Tutorial: Using SSH to Grep Logs – The Dow Hurst Average

Editors Note:   Using SSH to Grep Logs!  What a great idea!  SwitchDoc Labs is pleased to welcome our new columnist Dow Hurst to the SDL family.   Dow has a great background and really likes showing people of all abilities to master the Raspberry Pi and various SwitchDoc Projects.  He will be posting here twice a month and is very active on our technical support forums at forum.switchdoc.com.

Many of us who bought the SkyWeather2 kit may not be familiar with the Linux OS that runs on the raspberry pi called Raspbian. We have followed the directions and gotten to the point where we could see the web pages for the dash_app, or even had our own personal weatherstem page for our weather station. However, if something goes wrong you will need a way to get to the SkyWeather2 log file and the raspbian log files to troubleshoot a software or hardware problem.

Here we will go over how to use the SSH server “sshd” that comes activated on the SwitchDoc SD Card that runs the SkyWeather2. After we learn how to access the pi’s sshd server via ssh, I’ll show you how to find the log files and get information from them. Most of us want to get pertinent info so we can post an answer or formulate our question. We will figure that out too! (Hey, if you haven’t had a problem, great! But you still might want to know where those files are and how to look at them.)

SSH is the protocol we will use to access the raspberry pi. You can learn all about what it is here:

SSH Academy (Lots of info, don’t get lost in there)

Fortunately, the SD Labs SD card that you might have ordered with your kit, or can order here:

SDL Preloaded SD card

The SD Card has the “sshd” daemon enabled and the openssh client program called “ssh” preinstalled on the pi. This means that as soon as the pi boots with the onboard wifi, it starts the sshd daemon and provides a local hotspot you can connect to on the network port 22. What you need is a client on your computer to connect to the pi. Go get one for Windows from the list below, or if you have a linux or mac os/x machine, just make sure openssh is installed/enabled. If you didn’t get the SDL preloaded SD card, then go to Enabling SSH on Raspbian and get sshd enabled on your pi now, otherwise proceed on!

Installation on your computer

Linux using the shell

sudo apt update

sudo apt install ssh (metapackage, installs all the ssh related components and starts the ssh daemon)

 

Mac using the Terminal

sudo systemsetup -setremotelogin on (enables ssh)

sudo systemsetup -getremotelogin  (will check if ssh enabled)

 

Windows 10, you only need to use one of the following programs:

Tunnelier Bitvise ssh client (you need the free client, not the server which isn’t free)

Putty (you would download the 64-bit x86 version for standard 64 bit Windows 10)

 

My advice is if you installed ssh on your own computer AND the process included the ssh server along with the client, reboot it before you try using ssh. You don’t have to have a ssh server on your PC, only a client program. Macs and linux machines will have both client and server. Okay, that’s enough about installing ssh, let’s start using it!

 

Using SSH

The SkyWeather2 AssemblyAndTestManual1.3.pdf currently has the procedure on page 18 and 19 to get your pi onboard wifi hotspot up and explains how to connect your computer to the hotspot. It also explains how to configure the pi to have it connect to your own wifi network and no longer run the hotspot. I took their recommendations and used Angry IP to scan my network. It’s a great tool. If you completed getting your pi on your own wifi network, we will then substitute the string “foobar” for your local wifi network IP address in the commands below. If you didn’t get the pi on your network, ask for help on the SDL forums. If your local router allows you assign a local DNS name and/or static IP address to device’s MAC address, then that is also helpful in the long run.

 

Linux in a shell

ssh pi@foobar

 

Mac in Terminal

ssh pi@foobar

 

Windows in Putty

open Putty, put “foobar” in the Host Name (or IP address) field. Enter “pi” as the username when it asks.

 

When you first connect to any system you will be asked to accept that server’s host key, so do that. Also, the password is the default for the SDL SD card or for Raspbian unless you have changed it.  I’ve put an example of where I logged into my SkyWeather2 system with the network name pi2ether below. The window title bar shows the hostname of the pi. Once you have logged in, take a deep breath and relax. Many people struggle to get to where you are now!

 

Finding your log files

Linux has a / character that separates subdirectories just like on mac OS/X. Get used to thinking in terms of / as the directory separator character and that all directory paths start from that first / character. In other words, all filesystems in linux are mounted to the root filesystem as subdirectories, and the root filesystem is designated by the / character. There is no C: or D: drive in linux. Macs use the same principals as linux since they both come from a unix background historically. The path /home/pi/SDL_Pi_SkyWeather2 is the directory where the default SkyWeather2 python script will create a log file named “nohup.out”. The SkyWeather2 python program will write to that log file based on the startup command given in the SkyWeather2 manual. If you use the dash_app you need to realize the startup command given in the manual also provides the same name nohup.out for the log file it will create and write to, but that log file is located inside the subdirectory /home/pi/SDL_Pi_SkyWeather2/dash_app.

To view those log files, at the ssh prompt in the pi’s shell type:

cd /home/pi/SDL_Pi_SkyWeather2

or

cd /home/pi/SDL_Pi_SkyWeather2/dash_app

As you type the above path, /home/pi/SDL_Pi_SkyWeather2, I want you to try using the Tab key to auto complete your directory path and filename. I find that after I type /home/pi/SDL and hit Tab, it will auto-complete out to the end of SDL_Pi_. Then I’ll have to add Sk, hit Tab, and add the number 2. The reason I want you to try this now is that it becomes a real time saver in the long run to learn to use the auto-complete shell Tab key functionality at the linux command line. Now, use the “less” command to view the nohup.out log file in whichever subdirectory you chose.

less ./nohup.out

The PgUp, PgDn, or up and down arrow keys will scroll through the file. One nice trick is to search for a particular word or search string with the / (forward slash) key while using the less command to look at a logfile. Use less to view the nohup.out file and then issue the following command without quitting less:

/Tick

and hit Enter. Each time you repeat just the / key and hit Enter, less will jump to the next time the string “Tick” appears in the log file. Use the “q” key to leave the program. Now, when you ssh’d to the pi as the user pi, ssh dropped you into the /home/pi directory by default because that is the pi user’s home directory. When moving around the filesystem with the cd, or change directory command, a useful shortcut is:

 

cd ~ (returns you to /home/pi since ~ represents /home/pi)

To find out what directory you are in if it isn’t obviously shown at the prompt in the shell is:

pwd (just displays the current working directory)

Other useful cd command variants are:

cd .. (shifts the current directory to the level above)

cd – (switches to the previous working directory)

try this little cd command exercise shown in the example below:

Now that is very useful for bouncing back and forth between two different directories where files are that you want to keep examining!

 

Do this now:

cd  (without any flags to the command returns you back to /home/pi)

cd SDL_Pi_SkyWeather2 (works because the current directory /home/pi is in your known path)

The path is a set of directories searched through for commands and is set on login in the PATH environment variable. If you didn’t have this set, then programs would not be found unless an explicit path to the program was provided! By default, the current working directory is included in the PATH variable. Okay, enough about that…

The linux system logs live in /var/log and are named in Raspbian:

syslog

daemon.log

messages

kern.log

 

They get rotated by the syslogd daemon, so they don’t grow too large. This means the pi can have many stored older log files. After the first stored log file, they are automatically compressed by the “gzip” program. For example, the kern.log file might have:

kern.log

kern.log.1

kern.log.2.gz

They can be large and have a lot of info that doesn’t pertain to what you might need to find. Using the “grep” command, in tandem with gunzip if needed for the older compressed versions, is the way to go.

cd /var/log

grep firmware kern.log

would search and display all lines containing the word “firmware.” I found this useful when troubleshooting a network problem.

gunzip -c ./kern.log.2.gz | grep firmware

has “gunzip” decompress the file to standard output, then pipe it into the standard input of grep, and finally allows grep to search the decompressed text for the string firmware, all in one command. Using this technique there is no need to decompress the file, search it, and recompress it manually. Personally, I found the messages to be the most useful log file for tracking “firmware transaction” errors that were related to network errors due to the configuration of my dual WAN router setup. If you want to look for a string that contains a space, use double quotes to protect the spaces from being misinterpreted by the shell. If there is a variable name or wild card character in your string, then use single quotes instead.

Here is a link to what I consider a normal messages log from the SkyWeather2 when there is no error, and it has just rebooted. The first few and last lines are reproduced below:

May  4 15:17:06 SwitchDocLabs kernel: [    0.000000] Booting Linux on physical CPU 0x0
May  4 15:17:06 SwitchDocLabs kernel: [    0.000000] Linux version 5.10.17-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1403 SMP Mon Feb 22 11:33:35 GMT 2021
May  4 15:17:06 SwitchDocLabs kernel: [    0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
May  4 15:17:06 SwitchDocLabs kernel: [    0.000000] CPU: div instructions available: patching division code
May  4 15:17:06 SwitchDocLabs kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
.
.
.
May  4 15:17:13 SwitchDocLabs kernel: [   15.188549] Bluetooth: Core ver 2.22
May  4 15:17:13 SwitchDocLabs kernel: [   15.188663] NET: Registered protocol family 31
May  4 15:17:13 SwitchDocLabs kernel: [   15.188680] Bluetooth: HCI device and connection manager initialized
May  4 15:17:13 SwitchDocLabs kernel: [   15.189313] Bluetooth: HCI socket layer initialized
May  4 15:17:13 SwitchDocLabs kernel: [   15.189337] Bluetooth: L2CAP socket layer initialized
May  4 15:17:13 SwitchDocLabs kernel: [   15.189373] Bluetooth: SCO socket layer initialized
May  4 15:17:13 SwitchDocLabs kernel: [   15.201480] Bluetooth: HCI UART driver ver 2.3
May  4 15:17:13 SwitchDocLabs kernel: [   15.201503] Bluetooth: HCI UART protocol H4 registered
May  4 15:17:13 SwitchDocLabs kernel: [   15.201610] Bluetooth: HCI UART protocol Three-wire (H5) registered
May  4 15:17:13 SwitchDocLabs kernel: [   15.202811] Bluetooth: HCI UART protocol Broadcom registered

Here is a link to a messages log file that has the firmware transaction error. The first few lines and the last few lines are shown below:

Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] Booting Linux on physical CPU 0x0
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] Linux version 5.10.17-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1403 SMP Mon Feb 22 11:33:35 GMT 2021
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] CPU: ARMv7 Processor [410fd083] revision 3 (ARMv7), cr=30c5383d
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] CPU: div instructions available: patching division code
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] OF: fdt: Machine model: Raspberry Pi 4 Model B Rev 1.2
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] Memory policy: Data cache writealloc
Apr 27 15:42:23 SwitchDocLabs kernel: [    0.000000] Reserved memory: created CMA memory pool at .
.
.
.
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714591] [] (dbs_work_handler) from [] (process_one_work+0x250/0x5a0)
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714603]  r9:00000000 r8:00000040 r7:eff29900 r6:eff26640 r5:c3f8b100 r4:c3f8bfb8
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714618] [] (process_one_work) from [] (worker_thread+0x60/0x5c4)
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714630]  r10:eff26640 r9:c1203d00 r8:eff26658 r7:00000008 r6:eff26640 r5:c3f8b114
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714638]  r4:c3f8b100
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714653] [] (worker_thread) from [] (kthread+0x170/0x174)
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714665]  r10:c3421e74 r9:c3f8b100 r8:c023df68 r7:c22a2000 r6:00000000 r5:c6042440
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714673]  r4:c493bec0
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714687] [] (kthread) from [] (ret_from_fork+0x14/0x28)
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714696] Exception stack(0xc22a3fb0 to 0xc22a3ff8)
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714707] 3fa0:                                     00000000 00000000 00000000 00000000
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714718] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714728] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714740]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c02458e0
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714748]  r4:c6042440
Apr 27 21:00:54 SwitchDocLabs kernel: [17724.714759] ---[ end trace 9578fc7736b43e21 ]---

Let’s say you don’t know what to look for. Comparing a normal log with a log where an error may have occurred could make a big difference to your troubleshooting efforts. See if you can find the beginning of the error I had. Look for the string firmware transaction in the log with the error and then try to find it in the log that didn’t have an error. Repeat with just looking for the string firmware. You should find that firmware shows up in both versions, but firmware transaction does not.

Exit from your ssh session with the pi by just typing “exit” at the shell prompt and pressing the Enter key.

Transferring files from the pi back to your computer

Two different programs that will help you transfer files back to your linux or mac are:

scp

rsync

 

In the PC ssh clients, there is a sftp client as part of the program, or as a separate program, that provides a graphical interface for dragging and dropping files from the pi back to PC.

On linux or mac os/x using scp requires a flag or two depending on whether you want a single file or a directory to transfer. To preserve the timestamp and permissions assigned to the logfile requires the “-p” flag and a recursive copy of a directory requires the “-r” flag.

Formal usage:

scp -p user@host:/path/filename user@host:/path/filename

or

scp -rp user@host:/path/subdirectory user@host:/path

Rsync is very similar to scp but has many more options dealing with syncing directories. We will use just a simple set of flags for transferring a file or a directory:

rsync -avP -e ssh user@host:/path/filename user@host:/path/filename

rsync -avP -e ssh user@host:/path/ user@host:/path

Practical examples to transfer a single file with scp:

scp -p pi@foobar:SDL_Pi_SkyWeather2/nohup.out ./

scp -p pi@foobar:/var/log/messages ./SkyWeather2-failure_messages.log

Transfer a directory with scp:

scp -rp pi@foobar:SDL_Pi_SkyWeather2/dash_app ./

ingle file example for rsync:

rsync -avP -e ssh pi@foobar:SDL_Pi_SkyWeather2/nohup.out ./

Transfer the changed or new contents of a remote directory to an identically named local directory:

rsync -avP -e ssh pi@foobar:SDL_Pi_SkyWeather2/ ./SDL_Pi_SkyWeather2

In the first example for scp and rsync I did not have to put the full path /home/pi/SDL_Pi_SkyWeather2/nohup.out.  The pi user’s home directory is /home/pi and these ssh based commands accept a relative path after the “:” character. Just remember that after the “:” you can have a full path like:

/home/pi/SDL_Pi_SkyWeather2

or a relative path to the home directory like:

SDL_Pi_SkyWeather2

and they will understand that. Try transferring a file back to your computer now. I introduced rsync since if you take the time to learn it, it will become more useful to you than scp. It does have tremendous functionality.

Additional commands for working with files

Now ssh back into your pi and try these commands:

cd SDL_Pi_SkyWeather2

head -n 100 ./nohup.out (shows you the first 100 lines of the nohup.out file)

 tail -n 30 ./nohup.out (shows the last 30 lines)

grep -n mySkyCameraText ~/SDL_Pi_SkyWeather2/nohup.out (finds all lines with “mySkyCameraTest”, plus shows the line number in the file)

Sometimes you know what you don’t want to look at, but not what you want to find. The following command removes the search string being searched on, pipes that output into another grep to remove more info and so on:

grep -v “:” ./nohup.out | grep -v HM3301 | grep -v “]”

I have one of the modifications in the SkyWeather2 scripts that exposes more info from the AQI sensor. I didn’t want to see that data. I didn’t want any line with the “:” symbol in it. Nor did I want to see lines with the right square bracket, “]”. So, you can see the sky is the limit with piping the output of one grep command into another to zero in on what you want to find, or in this case, not show. Try coming up with your own pipeline of grep commands.

cat /home/pi/SDL_Pi_SkyWeather2/SkyWeather2.JSON (displays the contents of the file to the screen)

ls -lp (lists the current directory to the screen in a long format)

ls -ltrp (same thing but reverses the order by time so the newest file is shown last)

ls -la (shows hidden files/directories, they start with a period)

Most linux programs have a manual page you can access with the “man” command.

man ls

provides all the flags available for the “ls” command and a brief explanation of what they are for.

Caveats to remember

Is it possible to ssh more than once simultaneously to the pi? Yes, it is! On mac os/x or linux just open another terminal or shell and issue another ssh pi@foobar command. In the PC ssh client, just start a new instance of Putty to create a new ssh session in a different terminal.

When posting log files, only post the section you think is pertinent. There are a LOT of lines in the log files that are the same between a healthy system and a system where problem has occurred. Likely, you can tell the difference by comparing a normal log file with a problem log file. I’ve given examples you can download and compare for yourself. I’ve found that SkyWeather2 exhibited problems in the log files only after all the normal messages logged during the boot process were finished and the pi had run for a while. You can either copy and paste relevant lines from log files to the forums or download and edit a copy for upload. Brevity is appreciated and you will get asked for more info if you didn’t provide enough.

If the ssh host key for a machine changes, you will get a warning and your ssh client won’t connect without your accepting the new host key. Likely you reinstalled raspbian or maybe used a different SD card that has a different install of raspbian. Just read the message that ssh provides. It will tell you how to remove the old hostkey and show you how to do it on linux and on a mac. The PC clients will allow you to replace the old hostkey you have stored with a new one, but it will be done through the graphical interface and not at a command line.

Don’t forget to look at the shell prompt to help you remember which machine you have command of. I’ve issued commands thinking I was on the pi when I was still on my linux laptop!

Conclusion

SSH is a powerful way to access and control your raspberry pi. Troubleshooting becomes much easier when you can access the log files for either the SkyWeather2 python scripts, or the linux OS system. And, if you have questions the SwitchDoc Lab forums are a great place to get answers and interact with other users.