Remote Monitoring and Upgrading of Arduino/Teensy via Networked Raspberry Pi

This post covers how to remotely monitor / upgrade an Arduino or other MCU behind a firewall.

(Jump down to the Dec 2016 Update for Notes about the Duplicity Service that will easily and quickly allow remote access to an RPI)

Let’s look at an example:

sshTun

On the far right of the diagram is an Arduino ‘in the field’ and I want to access it from a local PC which is on the far left.

The first step to accessing the remote Arduino is to connect it serially to a networked computer. I propose using a Raspberry Pi for this purpose since they are cheap. I then wish to use putty and VNC to access remoteRPI to finally access the Arduino. But just adding remoteRPI doesn’t make either accessible.

There are a couple of problems with this solution. First, remoteRPI will most  probably be assigned a private IP address by DHCP. There is no direct access of remoteRPI due to the remote firewall, remoteFW. A hole is going to be needed in the firewall.

Putting holes in ‘customer’ firewalls is a difficult proposition, at best. Having the ‘customer’ create the hole is probably impossible. Gaining access and doing it yourself is equally impossible.

The usual way to get around the firewall access issue is to have the remoteRPI create the connection back to the local network. The remoteFW has no issues with outbound connections. We create and use that connection when necessary.

Using remoteRPI to create the connection is the access method I will implement here. This will be done using an SSH reverse tunnel.

At boot time, remoteRPI will open an SSH reverse tunnel to localRPI. I’m using a Raspberry Pi for localRPI, but any linux system will work. This tunnel remains open continuously. When you want to gain access to remotePRI, you make an SSH connection to localPRI which forwards to remotePRI.

I was unable to find any references that fully showed implementing the diagrammed solution from begin to end. This blog will cover all of the config changes and commands to implement this tunnel but not the whys. You can google all of the commands and learn the why easily enough so I won’t try to duplicate the effort.

Allocating TCP Ports

We are going to need to define 3 TCP ports for our project. I like to use a random number generator (google random number generator) to select a port between 10,000 and 32,767.

27045 – Local Firewall NAT Port

This is the port that will allow remoteRPI access to localRPI via SSH (port 22).

27046 – SSH forwarding port

We will connect to this port on localRPI when we want to access SSH on  remoteRPI.

27047 – VNC forwarding port

We will connect to this port on localRPI when we want to access VNC on  remoteRPI.

Create NAT Rule on localFW

remoteRPI needs to connect to localRPI via SSH (port 22). We will create a port forward rule on the localFW to allow this access; however, we will expect port 27045 to be the destination port on the firewall and we will translate that to port 22:

Application Protocol Source Net Port From IP Address Port To
sshRevTun TCP 27045 192.168.0.33 [localRPI] 22

This rule states any inbound packet with destination port of 27045 will be sent to IP address 192.168.0.33 (localRPI) using destination port 22.

Activate the firewall rule.

You can then do a simple test on the rule by using a remote port check utility such  as http://ping.eu/port-chk/ and enter the external IP address of your firewall and port 27045 to verify the port is open.

Enable Global Port Forwarding

By default, any reverse SSH tunnel built on localRPI will only be available to localRPI. My requirement is that I be able to use any PC on my local network to access remoteRPI, so we need to allow others the ability to use the reverse SSH tunnel.

On localRPI, in the file /etc/ssh/sshd_config, add the following line anywhere:

GatewayPorts yes

Save the file and restart SSH with

service ssh restart

Make Scripts to Create the Tunnels

We need two scripts to build the two SSH reverse tunnels – one for each protocol (SSH and VNC).

I create these using the standard pi user and simply place them in pi’s home directory.

Put these commands in a file named sshRevTunSsh:

#!/bin/bash

sleep 60

while [ true ] ; do
  ssh -p 27045 -N -R \*:27046:localhost:22 -o "ServerAliveInterval 30" \
    -o "ServerAliveCountMax 3"  pi@localfw.mydomain.com
  sleep 30
  done

And put these commands in a file named sshRevTunVnc:

#!/bin/bash

sleep 60

while [ true ] ; do
  ssh -p 27045 -N -R \*:27047:localhost:5900 -o "ServerAliveInterval 30" \
    -o "ServerAliveCountMax 3"  pi@localfw.mydomain.com
  sleep 30
  done

don’t forget to make them executable:

chmod 577 sshRevTunSsh sshRevTunVnc

These scripts will connect to localRPI by routing to the localFW (localfw.mydomain.com) and then being forwarded by the firewall to the localRPI system where they will attempt to log on as the pi user.

I am using a DNS name localfw.domain.com. You can use a DNS name or the external IP address of your router as well. If your router is assigned its WAN IP address using DHCP, you should seriously consider using a dynamic DNS service such as afraid.org. Otherwise, when your ISP finally forces the IP address on your router to change, the SSH reverse tunnel will fail, and it will be be difficult to fix.

Test Scripts

You might want to comment out the initial sleep 60 command as this is really only necessary when the script is executed at boot time.

To start the SSH reverse tunnel for SSH, type

./sshRevTunSsh

You will be asked for the pi user password for localRPI. Then there will be no further output, it just runs.

Now from a local PC, run putty. For the hostname/IP address, enter the hostname for localRPI and specify 27046 as the port:

	localRPI		27046

This will connect you to remoteRPI. Enter a user/password for the remoteRPI system.

Should the script fail, you can place -v in the SSH cmd of the script to get debugging output.

Now run the script for sshRevTunVnc. From a local PC, run VNC and login using

	localRPI:27047

This will connect you to VNC on remoteRPI.

Create SSH Reverse Tunnel without Requring a Password

The scripts to start the SSH reverse tunnels allow automation, but having to enter a password prevents us from actually automating the process.

There are many locations on the web explaining this process. I will do a very brief example of it.

On remoteRPI, logon to the pi user and type:

cd ~/.ssh
ssh-keygen -t rsa

This creates a public/private key pair for password-free logins.

Now copy the remote RPI’s public key to the local RPI

scp -P 27045 id_rsa.pub pi@localfw.mydomain.com:.ssh/temp.key

On the local RPI, you want to append that key (in the file temp.key) to the authorized_keys file:

cd ~/.ssh
cat temp.key >> authorized_keys

Rerun the scripts and they will no longer require passwords.

Dealing with IP Address / Key Changes

Since originally writing this, I’ve had to deal with changing the local RPI host. SSH knows the host has changed which causes a message along the lines of

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

To fix this problem, you need to use ssh-keygen to reset the key:

ssh-keygen -R <ipaddress>

Because a non standard port is being used, it appears you must specify that as well for this specific type of access, so specify that as

ssh-keygen -R [<dnsname>]:<port>

Once the old key is deleted, follow the procedure above to regen and redistribute keys.

Fully Automate the SSH Reverse Tunnels

Everything is now in place to allow the SSH reverse tunnels to be created at boot time. Should the tunnel go down, the script will automatically restart it.

We will run the scripts at boot time using cron:

sudo crontab -e

and add the lines

@reboot su pi -c "/home/pi/sshRevTunSsh " >/dev/null 2>&1
@reboot su pi -c "/home/pi/sshRevTunVnc " >/dev/null 2>&1

Don’t forget to restore the ‘sleep 60’ commands in the script files if you  commented them out.

Reboot the system. You can verify the scripts are running:

pstree | grep sshR
        |-cron-+-cron---sh---su---sshRevTunVnc---ssh
        |      `-cron---sh---su---sshRevTunSsh---ssh

Finally, again test that you can access remoteRPI from your local PCs.

Issue Programming Arudino Nano from Raspberry Pi

I’m sticking this note here because I don’t have a better place for it yet.

Now that I’ve got the remoteRPI operational, I’m starting to experiment with actually controlling / updating an Arduino.

I started with a Nano and have found there is a problem with Nanos. When you first connect the Nano to the RPI via USB, it will come to life and populate /dev/tty properly and you can access it via /dev/ttyUSB<n>.

But, when the RPI is rebooted, the Nano disappears. Nothing seems to bring it back short of unplugging the USB cable and reconnecting. Not acceptable.

This problem appears to only occur with Nanos due to a design mistake when implementing FTDI. This is covered here: https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=59420.

I have since moved on to an Arduino Uno for testing and it works fine – when the RPI is rebooted it is reconnected properly to the USB port.

Dec 2016 Update:

Reverse Tunnel Status:

I’ve been using the aforementioned scheme to access a remote RPI for 4 months now and in general it works great. The only issue I have is for some reason the local linux server where the ssh tunnels terminate periodically has an issue such that I can no longer SSH into the tunnel. The only solution has been to reboot the local system which then causes the RPI to reestablish the tunnels and all is fine.

Duplicity:

I recently learned about dataplicity.com, a 3rd party service to allow remote access to your RPI.

It is super easy to get your RPI up and running on this service. They give you a command to type at the RPI’s bash prompt and it fully installs and starts the duplicity service. Then you go to your devices at duplicity.com and can get to a shell prompt on their web page.

They also provide ‘wormholes’ that allow you web access and VNC access to the RPI. But this access requires manually installing all the necessary services, etc.

The service is free, at least at the moment. How they do this for free is puzzling. I’m not complaining, but if I were going to use this for anything ‘production’ I would want to know they aren’t going to go out of business because they are giving the service away. It might be a royal pain to recall devices and come up with a different mechanism if they were to vanish (or start charging more $$$ than I could justify).

At the moment I don’t see a way to transfer files TO the RPI. It may be this needs to be done by creating a webdav directory that allows write access. FTP/SSH access, like I can use with the reverse tunnels I create in my scheme is not possible as far as I can tell.

Still, for easy remote access of the shell, this is a fantastic alternative.

I decided to use dataplicity for an emergency backdoor to remote RPI in case the sshtunnels completely fail. I’ve also noticed there is no way to cut/paste in their terminal emulator which is makes it not quite as useful as one would hope.

Another update for Dataplicity:

My ‘puzzlement’ has been answered quickly. I just received an email that they will be charging between $2 – $3.50 / mo.

Oct 2017 Update

I have three RPIs that maintain a reverse SSH tunnel as outlined, but they have driven me crazy because if the local firewall router is rebooted, the tunnel is not re-established and I have to do back flips trying to restart the tunnels (either have the user reboot the RPI which is risky since they can only remove/reapply power OR use dataplicity).

I finally had time to work on this issue and I believe I have solved the problem. There is a utility called autossh that will monitor the tunnels and restart them if they go down. I would think the keep-alives would do that, but they don’t.

Install autossh. Then in your startup script, use:

autossh -M 0 <user>@<dnsName> -p <sshPort> -N -R <inputPort>:localhost:<outputPort> \
 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
 -o ServerAliveCountMax=2

Using my example above, this would be setup as:

autossh -M 0 pi@localfw.mydomain.com -p 27045 -N -R 27046:localhost:22 \
 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
 -o ServerAliveCountMax=2

So far, this has worked without problem.

Advertisements
This entry was posted in c-arduino, c-rpi, c-teensy and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s