Advanced SSH techniques and Screen command usage – The Dow Hurst
Average
Editors Note: More advanced SSH usage and the elusive “screen” command! SwitchDoc Labs is pleased to welcome our new columnist Dow Hurst to the SDL family. Dow has a great background and really likes showing people of all abilities to master the Raspberry Pi and various SwitchDoc Projects. He will be posting here twice a month and is very active on our technical support forums at forum.switchdoc.com.
In our tutorial on using SSH to grep logs, we did not cover more advanced ssh usage or the utility called screen. You don’t have to type your password every time you login to your Skyweather2! This is probably the most useful technique that you can apply when using ssh. We will have to go over some basics so you can understand what must be configured. To get to these first steps, I must recommend for those using the Windows Bitvise ssh client to install Cygwin on your PC. Cygwin provides a linux environment on top of the windows operating system. The Bitvise ssh client can access the openssh package from Cygwin, which provides passphrase-based authentication and the agent program we need. If you use Putty on Windows, instead of Bitvise, then you already have all the utilities you need and you don’t need Cygwin.
Configure a ssh passphrase and start an agent
We need:
- To generate a private/public key pair, where the private key stays on your local machine and the public key will get put on the Skyweather2 pi you want to login to without typing a password.
- To run an agent program that will remember our private key after we unlock it.
- To put our public key in the right file on our Skyweather2 pi.
My assumptions for the procedure below are that you have used ssh to connect to your Skyweather2 pi before now. If not, please do so after reading the last blog entry. Whenever you first use ssh to connect to a remote server, two things happen. First, the ~/.ssh subdirectory is created on your local machine, and secondly the remote server’s host key is put into the newly created ~/.ssh/known_hosts file.
Linux: In a terminal, or shell, issue the command below. Press Enter to take the default file name for the key. When you are asked to enter a passphrase, I recommend using your login password for your machine, if it is a good one. Do not leave the passphrase empty! Many tutorials recommend that option and it is not good practice. Good passwords are a combination of at least 8 characters with letters (both cases), numbers, and special characters all mixed up together. IMHO, a good password you don’t have to change is better than weaker ones you are changing all the time.
ssh-keygen -t rsa
Generating public/private rsa key pair. Enter file in which to save the key (/home/hurst/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hurst/.ssh/id_rsa Your public key has been saved in /home/hurst/.ssh/id_rsa.pub The key fingerprint is: SHA256:TZsbPgpgCMNBCme+rSEypZPacGbCsADodlRGNU8Agjs hurst@pop-os The key's randomart image is: +---[RSA 3072]----+ |=o+.o=o+.. | |*=..o + | |B.= .. | |+E = o o | |%.X + S = | |o@ + . . o | |. o . + | | . . . | | . | +----[SHA256]-----+
The ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub files are text files. The id_rsa is your personal secret key that you keep secure, while the id_rsa.pub is the public key that you will put on computers you want to ssh to, such as your Skyweather2 raspberry pi. The ssh-keygen program creates the appropriate permissions on those files when they are generated. Only you can access the id_rsa file, while anyone can access at the id_rsa.pub file.
In the command below, please change “skyweather2.local” to your Skyweather2 pi’s IP or hostname.
scp -p ~/.ssh/id_rsa.pub pi@skyweather2.local:/home/pi/.ssh/authorized_keys
This transfers the id_rsa.pub file to the right spot on your Skyweather2 raspberry pi and names the file “authorized_keys”. Mine looks like this after that command:
pi@SwitchDocLabs:~ $ cat .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEy1U7cq2LY+9Piuj3t4wqw+bT5XB9+sutzkegZgFb+kzwhQ2ertcQ3pKHJ5KbYT85mra8tLJsG4T2xhmFSjNIyyoGObku/nsUrRLV2pBnqZReyiV7LlkIxIDB5FgK/jq0oUmFiIR95kpRseZ9hHFLgxKh00Wvh6yKHaJcwrO5gW5njrnJpIbHQfM4xKDlZmFQ3QLFPX5yqJOrvreVXzoMuxYZ3X9hmphr5gJ2krWZtXuYkYHPAUiIu+tOagdz8vYutGZv7Brwl+kLL+Z3CGjqXjSUq/MfVjo95FgWSBhwP17VMXX1uuiJrPRIxmGEFFVrcYqaKS7EToBrZBKhreGt hurst@local pi@SwitchDocLabs:~ $
The ssh-rsa string before the key identifies the type of encryption used to create the key. The user@machine meta-data string at the very end of the long public key is there to help you identify the user and host who created the key. You might, like me, end up with many public keys that need managing down the road. We used the default 1024 bit encryption when we created our key. There are other options than RSA and 1024 you can use, so feel free to read the Skyweather2 ssh-keygen manual page (man ssh-keygen) if you are interested in learning about what is possible.
When you graphically login on modern linux distributions, such as Ubuntu, the ssh-agent program is automatically started and will attempt to use your login password to unlock your private key. This is the program that can hold unlocked private keys for usage when negotiating a login via ssh to the Skyweather2 pi. If this doesn’t just magically work for you, then check to see if you have the package ssh-askpass variant for your desktop, such as ssh-askpass-gnome installed. We will go over starting ssh-agent manually later in the context of using the program screen.
Windows: You can use the openssh package in Cygwin to generate a key with the same directions given for Linux above. Here is a great tutorial on installing cygwin and then openssh on windows (https://www.poftut.com/install-use-cygwin-terminal-ssh-examples/). Falko Timme provided a wonderful explanation of how to setup a passphrase and load it automagically using Putty. There are four linked pages in that tutorial and you need to go through each page in order. For Bitvise, you have two agent options: use the Putty Pageant program to store keys and provide them to Bitvise, or to use the Cygwin openssh ssh-agent program. A short explanation of both of those options by Bitvise is located here.
Mac OS/X: Luckily, you also use the same procedure given for Linux above to generate private/public ssh keys. Add these directives to your ~/.ssh/config file if they don’t exist. If the ~/.ssh/config text file doesn’t exist, create it and add those two directives.
UseKeychain yes
AddKeysToAgent yes
and add your passphrase to the os/x keychain with
ssh-add -K
I found those gems in this os/x ssh-agent tutorial by Danila Vershinin. You may have more than one raspberry pi or server you end up connecting to, so pay attention to the Caveat section on Multiple SSH keys in that webpage.
Using ssh conveniently
Now you have ssh configured with a passphrase and when you ssh to your Skyweather2 pi, you no longer need to type a password, you get put directly into the pi’s terminal. That should look like this on Linux or Mac OS/X:
ssh pi@pi2ether
Linux SwitchDocLabs 5.10.17-v7l+ #1403 SMP Mon Feb 22 11:33:35 GMT 2021 armv7l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sat May 29 20:18:20 2021 from 192.168.1.9 pi@SwitchDocLabs:~ $
A cool benefit, besides not having to type a password a million times a day, is that you can now run commands using ssh directly.
ssh pi@pi2ether ‘cat ~/.ssh/authorized_keys’
will execute the cat command and show you the contents of the authorized_keys file. No need to login and then run the command in two steps. I put single quotes around the command to protect the ~ from being interpreted locally and not on the Skyweather2 pi.
All the ssh based commands such as sftp, scp, rsync, and ssh make use of the agent-based passphrase authentication. This means that you will be able to use those commands without the shell-based interruption of entering a password, which leads to using commands in scripts where the agent is available to provide authentication.
The ~/.ssh/config file
On linux I use the following directives in the config file, located inside your local .ssh directory, to provide an easy way to access machines that are not directly reachable, but require a connection through an intermediary server. Your public key, id_rsa.pub, needs to be put inside the ~/.ssh/authorized_keys file on each machine specified in the config file. The intermediary server and the destination servers must all be able to use your public key if you want to avoid typing a password.
Host teller
User hurst
HostKeyAlias teller.foobar.foo
HostName teller.foobar.foo
Host zeus
User hurst
ProxyCommand ssh teller -W zeus:22
Host dio
User hurst
ProxyCommand ssh teller -W dio:22
I know this works on linux, and therefore raspbian, by asking teller.foobar.foo to forward the ssh connection to either zeus or dio. I have not been able to test this on os/x or Cygwin in Windows. The raspbian ssh manual page states,
“-W host:port Requests that standard input and output on the client be forwarded to host on port over the secure channel. Implies -N, -T, ExitOnForwardFailure and ClearAllForwardings, though these can be overridden in the configuration file or using -o command line options.”
There is an example tested on os/x given here that covers port forwarding for os/x very well. The -W flag for the ProxyCommand may be a linux implementation-only flag. You should put the names of your machines in place of my config file example. If you have a laptop that moves around to different locations, you can create a symbolic link named config to point to a location specific config file like so:
hurst@pop-os:~/.ssh$ ls -l -rw-r--r-- 1 hurst hurst 2560 Jul 23 2018 authorized_keys lrwxrwxrwx 1 hurst hurst 11 Feb 2 17:22 config -> home_config -rw-r--r-- 1 hurst hurst 1810 Apr 22 2018 coffeeshop_config -rw-r--r-- 1 hurst hurst 2351 Mar 5 11:17 home_config
Create a file name coffeeshop_config with the information needed, then create the symbolic link like this:
cd ~/.ssh
ln -s coffeeshop_config config
To switch the symbolic link between location specific config files, do this:
rm -f ./config
ln -s home_config config
For Putty, you setup the connection to the intermediary server as a normal ssh connection to a ssh server. But, you also select Connection > SSH > Tunnels in the left panel to be able to Add a local source port on the intermediary server to forward the ssh connection to a destination server on port 22, which is the port ssh servers provide by default. You can go back to the Session panel to Save those settings as a session too. The example provided shows the Tunnel window.
It is possible now to transfer files and run remote commands much more easily now that you have ssh configured as we have it.
Install and use Screen
What if you are using apt to update the pi’s software? Or you might be editing a python script on the pi in nano or vim and get an important phone call. What happens if your ssh connection to the Skyweather2 is interrupted or dies? There is a solution and that is the program called screen.
Screen is program that provides a virtual terminal that can be detached from standard input and output required by regular terminals. A great example is when editing a set of python scripts on the Skyweather2 pi with multiple ssh connections from your laptop, but you need to physically switch your location to a nearby coffee shop. You have four scripts open in vim or nano through four separate ssh connections and you are working back and forth editing two of those scripts and referencing the other two for information. If you are using screen, you would detach from each of the screen sessions where nano or vim was running. Now imagine you shutdown your laptop, drive to a coffee shop, and then instead of reopening those four files and finding where you were, you just attach back to each of the screen sessions. The editor is exactly where you left it in each of the files.
Another use case is when updating the OS on a pi. A slow pi can take a long time to download, install, and configure software, especially if it is compiling modules for a kernel using dkms. You do not want that to be interrupted by a network connection dying, so you can use screen to preserve the “apt” commands in case of a network disaster.
Screen isn’t installed by default on the SwitchDoc Labs SD card and isn’t in a default raspbian installation either. To install it do the following:
pi@SwitchDocLabs:~ $ screen -bash: screen: command not found pi@SwitchDocLabs:~ $ sudo apt install screen Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: libexiv2-14 libgfortran3 libgmime-2.6-0 libncurses5 libwebsockets8 lxplug-volume rpi-eeprom-images rtimucli uuid-dev Use 'sudo apt autoremove' to remove them. The following additional packages will be installed: libutempter0 Suggested packages: byobu | screenie | iselect The following NEW packages will be installed: libutempter0 screen 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 555 kB of archives. After this operation, 961 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://mirror.umd.edu/raspbian/raspbian buster/main armhf libutempter0 armhf 1.1.6-3 [7,736 B] Get:2 http://mirror.umd.edu/raspbian/raspbian buster/main armhf screen armhf 4.6.2-3+deb10u1 [547 kB] Fetched 555 kB in 1s (609 kB/s) Selecting previously unselected package libutempter0:armhf. (Reading database ... 225409 files and directories currently installed.) Preparing to unpack .../libutempter0_1.1.6-3_armhf.deb ... Unpacking libutempter0:armhf (1.1.6-3) ... Selecting previously unselected package screen. Preparing to unpack .../screen_4.6.2-3+deb10u1_armhf.deb ... Unpacking screen (4.6.2-3+deb10u1) ... Setting up libutempter0:armhf (1.1.6-3) ... Setting up screen (4.6.2-3+deb10u1) ... Processing triggers for man-db (2.8.5-2) ... Processing triggers for install-info (6.5.0.dfsg.1-4+b1) ... Processing triggers for libc-bin (2.28-10+rpi1) ... Processing triggers for systemd (241-7~deb10u6+rpi1) ... pi@SwitchDocLabs:~ $
To start screen just type:
screen
and then hit the Space Bar or Enter to get out of the entry message. You will see a normal shell prompt.
start top to have a program running that would normally die if the ssh connection was terminated.
top
issue a Cntrl-a and press the d key. That sequence, Cntrl-a then d, will detach the screen session and you will see the original shell prompt. The screen session is “detached” but top doesn’t know it. It is still running. Try the following command to prove it and you will see this output:
pi@SwitchDocLabs:~ $ ps aux | grep top | grep -v grep pi 23969 0.8 0.0 10440 3084 pts/1 S+ 01:27 0:00 top pi@SwitchDocLabs:~ $
ps aux lists all running processes on the pi, grep searches for the string “top” in that listing, the “grep -v grep” removes any line that contains the string “grep” from the output, and finally the pipe command “|” pushes the output of one command into the input of the next.
So top is still running because screen provides a virtual input and output top can use. To get back into the screen session we issue:
screen -r
and top re-appears. Use Cntrl-c to end top and “exit” to quit the screen session. If more than one screen session is present, you will be given a list of virtual terminals to choose from when trying to reconnect:
pi@SwitchDocLabs:~ $ screen ( I detached with Cntrl-a d) pi@SwitchDocLabs:~ $ screen ( I detached with Cntrl-a d) pi@SwitchDocLabs:~ $ screen -r There are several suitable screens on: 24011.pts-0.SwitchDocLabs (06/02/2021 01:37:42 AM) (Detached) 23950.pts-0.SwitchDocLabs (06/02/2021 01:22:42 AM) (Detached) Type "screen [-d] -r [pid.]tty.host" to resume one of them.
Press the up arrow, the space bar, and copy and paste one of the listed pts as shown above to reconnect to that session.
pi@SwitchDocLabs:~ $ screen -r 24011.pts-0.SwitchDocLabs
You can name the virtual terminals, or pts, with the -S flag when you start them.
pi@SwitchDocLabs:~ $ screen -S nano-JSON [detached from 24160.nano-JSON] pi@SwitchDocLabs:~ $ screen -S vim-dash-index.py [detached from 24167.vim-dash-index.py] pi@SwitchDocLabs:~ $ screen -r There are several suitable screens on: 24167.vim-dash-index.py (06/02/2021 01:48:00 AM) (Detached) 24160.nano-JSON (06/02/2021 01:47:33 AM) (Detached) Type "screen [-d] -r [pid.]tty.host" to resume one of them.
If you forget and leave a screen session attached to a ssh session from a machine at your office, but once you get home you want to reconnect to that same screen session to keep working, you can do that like with the command below. Just replace the X’s below with your screen session info:
ssh pi@pi2ether screen -list screen -d XXXXXX.pts-X.SwitchDocLabs screen -r XXXXXX.pts-X.SwitchDocLabs
The -d flag detaches that screen session and then allows it to be available for reattachment.
man screen
will show you many options for controlling screen. I recommend just trying out the options and exploring how screen can work for you.
Combining agent passphrase authentication with screen
After a screen session is started, the shell running inside does not have access to any running ssh-agent. To make one available to shell scripts that might have ssh based commands, you need to start an agent and unlock your key.
hurst@pop-os:~$ eval `ssh-agent` Agent pid 24968 hurst@pop-os:~$ ssh-add -l The agent has no identities. hurst@pop-os:~$ ssh-add Enter passphrase for /home/hurst/.ssh/id_rsa: Identity added: /home/hurst/.ssh/id_rsa (/home/hurst/.ssh/id_rsa) hurst@pop-os:~$
If you have more than one pi and want to have a pi run scripts on the others, then this is one way to securely do that. The eval command has back ticks, not single quotes, around the ssh-agent command. Those are found on an US based qwerty keyboard on the top left, usually next to the 1 key. Your keyboard may vary. The ssh-agent will continue to be available while the screen session lasts. Any new screen session would require starting a new ssh-agent the same way. I see the above procedure useful as given on linux and it may work in Cygwin on Windows. The eval command is a bit different on os/x:
eval $(ssh-agent -s)
If you have questions, don’t hesitate to contact me on the SwitchDoc Labs forum. I’d love to hear feedback or comments if the information was helpful or if you spotted a problem.
Caveats with advanced ssh and screen
The “ssh hostkey man in the middle attack” warning (seems scary): If you have connected previously to the Skyweather2 with ssh, then your client has registered the server host key. One day you may see a message like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:0xgFiU5j9W2WgyurDOgORf+qeFQoHf0YE6G92KnrduY. Please contact your system administrator. Add correct host key in C:\\Users\\JamesAlesi/.ssh/known_hosts to get rid of this message. Offending RSA key in C:\\Users\\JamesAlesi/.ssh/known_hosts:2 ECDSA host key for 192.168.1.123 has changed and you have requested strict checking. Host key verification failed.
What happened? Well, there is usually a simple explanation as outlined in this excellent answer. Most likely you replaced your router which then assigned a different IP via DHCP to the Skyweather2, or you upgraded your SD card. Rarely will your Skyweather2 have been hacked. The message tells you how to fix the problem. In this example the 2nd line in the known_hosts file contains the old host key, which doesn’t match the host key the Skyweather2 is presenting when the ssh connection is established. Open the known_hosts file in vim or nano, delete the line with the offending host key, and save the file. Retry your ssh connection and it will save the new host key.
Screen: You may have an open file in nano or vim in a screen session and you have forgotten about it. You might be about to reboot your Skyweather2 and could lose work, or damage a file. To find screen sessions, especially if there are more than just the user pi on your Skyweather you can use the ps command along with grep. However, a running screen command in the ps output is in all caps.
pi@SwitchDocLabs:~ $ ps aux | grep SCREEN pi 24510 0.1 0.0 8136 2216 ? Ss 02:40 0:00 SCREEN pi 24517 0.0 0.0 7348 556 pts/0 S+ 02:40 0:00 grep --color=auto SCREEN
will return all lines out of the ps command showing running screens. If you only have the pi user, then just using “screen -list” will show the pts. But, if you have other users on your pi, you won’t see those and could use the ps output as way to make sure you weren’t losing someone else’s work.
Conclusion
I hope you enjoyed reading about advanced SSH usage and the screen command. Next up is some measurements on the range of SwitchDoc Labs 433MHz signals with different antennas.
Cygwin? Windows now has SSH built in.