Virtual USB flash drive for a non-networked CNC router

By Daniel Foote on

Recently, a business that we worked with purchased a brand new 1325 CNC router - that is a CNC router with a 1300mm x 2500mm working area, fitted with an automatic tool changer. While a fantastic machine, there were a few things missing from the machine.

The machine uses a Weihong NK105 G3 control system. It's a solid control system, but naturally has it's own quirks! One of the main issues we had with the machine is that the only way to get jobs onto the machine is via a USB drive. It has no networking options at all. This was known before the machine was purchased; to save costs, this controller was selected rather than an upgraded one that had networking functionality.

This CNC router is working on a wide variety of different jobs each day, which can involve several dozen plug-unplug cycles on both the CNC router and the computer used to generate the gcode for the machine. This adds wear and tear to all the components involved, not to mention the reaching up and down to where the drive is located on the machine.

But that doesn't mean we can't network it...

A quick bit of internet research shows that a Raspberry Pi Zero's USB ports have a gadget mode - such that you can actually use it to create a virtual USB drive. So while we can't fully network the device, we can at least stop the constant tumble of unplugging and re plugging in a USB drive all the time.

The goal of this project was simple - just set up a Raspberry Pi Zero to act as a USB flash drive, and allow it to accept file updates from a remote computer over the network.

Once the files are available to the CNC, jobs were started from the control handle on the CNC. Once the job is loaded into memory on the CNC controller, you can actually disconnect the USB flash drive, which means that we can reload new files while the current job is in progress.

Overview

In this specific case, the user of the CNC machine works on a Mac laptop, and they use Fusion 360 to generate toolpaths. This allowed me to take a few shortcuts with the development of this project!

Fusion 360 typically post processes files into a single folder; and it's often easier just to use a single folder for this. This gives us a single source of files to send to the CNC.

Then, as it was a Mac laptop, it already comes with rsync, which is a battle hardened way to transfer files via a network reliably, so why not make use of it, rather than reinventing the wheel in a less perfect way?

The user in question is tech savvy; we created a shortcut for them to be able to quickly sync the Fusion 360 folder over to the virtual drive.

For other users, I would deploy this a bit differently - I was originally going to find a simple web-based file manager that allowed uploading of files, and set that up on the Raspberry Pi, and modifying it a touch to allow you to manually mount or unmount the virtual drive. However, time is money, and this solution robustly met the requirements with minimal time, although it's not the most user friendly solution.

Hardware

The core hardware required is:

  • A Raspberry Pi Zero. I used a W model with built in 2.4GHz wireless. However, through other technical limitations in the destination workshop, we only had 5GHz wireless available, meaning we needed to Ethernet connect the Raspberry Pi Zero to another 5GHz wireless router to get everything to work.
  • A micro SD card for the Raspberry Pi. It'll need to be big enough for the OS and also extra space for the virtual flash drive. I used a 16GB one, with a 2GB virtual drive, which is more than adequate for our use. The largest gcode files we work with are only around 5MB, but they average much smaller.
  • Either: ** A USB A-to-A cable, and then a USB micro to female A cable, joined together; ** or a USB male A to Micro USB male cable.
  • A power supply for the Raspberry Pi (powering it from the CNC controller is possible, kinda, but when you virtually disconnected the flash drive, the CNC controller dropped power temporarily on the USB port causing the Raspberry Pi to reboot. So external power is necessary).

And optional hardware, which was needed due to our specific setup:

  • An external Ethernet board. We used an ENC28J60 based adapter which communicates via SPI with the Raspberry Pi. There are other boards available for the Raspberry Pi Zero that offer extra USB ports and an Ethernet port via a RealTek chip, however, these boards take over the USB port and add a USB hub, and prevents the gadget mode from working correctly. The ENC28J60 board uses SPI and thus doesn't take over the USB port. ENC28J60 modules are commonly available very cheaply in a variety of forms.
  • An external WiFi router in client mode if needed, to connect via Ethernet. In our case, we used a TP-LINK WA1201 AC1200, as it was readily available and was one of the cheapest that properly supported client mode.

Setup of the Raspberry Pi Zero

We flashed Raspbian Lite using the official imaging tool on a Windows machine. At time of writing this was 2022-04-04, and the 32bit version.

When using the imaging tool, we used the advanced configuration to set:

  • WiFi credentials (for setup and testing purposes, as everything was configured and tested in a different workshop from where it was ultimately installed)
  • Password for the "pi" user
  • Hostname (I used "cncdrive" making the hostname "cncdrive.local", but adjust for your setup)
  • Timezone
  • Enable SSH

After it was flashed, the card was inserted and the Pi was booted. I connected to it via SSH, and checked that it expanded the filesystem correctly to the full drive. You can also take this time to apply other standard setup steps for your environment, like installing monitoring tools and so forth.

Set up the gadget mode overlay

We need to enable the gadget overlay, and get it to insert the module at boot.

$ sudo nano /boot/config.txt
... at end add:
dtoverlay=dwc2

$ sudo nano /etc/modules
... at end add:
dwc2

Now reboot the Pi, and we'll create the filesystem.

Creating the filesystem

Firstly we create the file that will be the virtual block device:

$ sudo dd bs=1M if=/dev/zero of=/drive.bin count=2048

I then connected it to a Windows 10 machine and got it to partition the block device and format it. Yes, technically I could have done this all with Linux, but I was aiming for compatibility with the CNC router, which worked more reliably with drives formatted by a Windows machine.

To get it to expose the file as a block device over USB ("plugging in" the USB drive):

$ sudo modprobe g_mass_storage file=/piusb.bin stall=0

Then using a Windows machine, partition the drive and format it - we used FAT32 to get the compatibility we needed. When you're done, you can release the block device as follows ("unplugging" the USB drive):

$ sudo modprobe -r g_mass_storage

Now we'll set up the mount point on the Pi.

$ sudo mkdir /mnt/drive

Then comes the interesting part. Basically, we're using rsync as a destination, without a password or encryption. (Yes, this is insecure, but it's on a private network and we're aiming for simplicty in this instance). Before the transfer starts, rsync runs an "early exec" script, which unmounts the virtual drive from the CNC, and then remounts the filesystem locally, allowing the file transfer to take place. After the sync, it runs a "post-xfer exec" script, which umounts the drive from Linux and re-exports it to the CNC. As mentioned before, this was the quickest and most robust solution to meet this specific users requirements, although it's not super user friendly.

So let's set up Rsync. Turns out that systemd has solid additional default settings used to harden rsync installations. These settings are ideal for almost all installations of rsync - except for ours! So we had to relax the systemd rules slightly to allow our very unusual disc usage pattern.

Let's loosen the systemd rules for rsync slightly, to allow the exec scripts to be able to mount filesytems and run modprobes. Without this change, the scripts will fail to run. I did spend a good 30 minutes working this one out:

$ sudo vim /lib/systemd/system/rsync.service
... modify:
PrivateDevices=off

$ sudo systemctl daemon-reload

Now let's configure the rsync daemon:

$ sudo vim /etc/rsyncd.conf

... the entire contents of this file should be:

uid = root
gid = root
max connections = 10
socket options = SO_KEEPALIVE

[cnc]
path = /mnt/drive
comment = CNC Drive
read only = false
early exec = /home/pi/pre-copy.sh
post-xfer exec = /home/pi/post-copy.sh

Set up the early exec script. Note this can't be a "pre-xfer exec" as it's not early enough in the transfer; rsync already has aquired file handles for the folder, and doesn't see the mount point made inside the script. Yes, this took a few goes to work out...!

$ sudo vim /home/pi/pre-copy.sh
#!/bin/bash -x

set -e

echo "Removing from CNC..."
sudo modprobe -r g_mass_storage

sleep 1

echo "Mounting drive locally..."
sudo mount -o loop,offset=65536,sizelimit=2144337920 /drive.bin /mnt/drive

sleep 1

echo "Ready to receive files."

Wait a sec! What's happening here? For simplicity, I'm not setting up a loopback mount device for the file which would allow us to address the partions separately, as this was extra configuration and more complexity. Instead, I worked out the offset and partition size from the file image, and put them into the mount command, bypassing the need to set up a loopback device. The numbers below will be different for your device, but it gives you an idea. (Snippet originally from StackExchange) Note that the numbers are in sectors, so multiply them by 512 (in this case) to get the actual physical numbers for the mount command.

$ sudo fdisk -lu sda.img
...
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
...
  Device Boot      Start         End      Blocks   Id  System
sda.img1   *          56     6400000     3199972+   c  W95 FAT32 (LBA)

And the post copy script, to unmount the drive and expose it to the CNC again. Note that it uses the "ro=1" flag for g_mass_storage, meaning that the CNC has a read only view of the virtual drive. There isn't a specific need for this, but there also isn't a reason for the CNC controller to manage files on the drive.

$ sudo vim /home/pi/post-copy.sh
#!/bin/bash -x

set -e

echo "Unmounting drive..."
umount /mnt/drive

echo "Re-exporting mass storage..."
modprobe g_mass_storage file=/drive.bin stall=0 ro=1

echo "Completed."

And finally, a script to execute on boot, which exposes the drive to the CNC on boot.

$ sudo vim /home/pi/on-reboot.sh
#!/bin/bash -x

set -e

echo "Exporting mass storage..."
/usr/sbin/modprobe g_mass_storage file=/drive.bin stall=0 ro=1

echo "Completed."

Make all these scripts executable:

$ sudo chmod a+x /home/pi/*.sh

And start rsync:

$ sudo systemctl restart rsync

And we're going to use crontab to run the on-boot script to expose the drive. There are other ways to accomplish this, but this is the way I chose to do it:

$ sudo crontab -e
...
@reboot /home/pi/on-reboot.sh

And we're ready to test! At this stage, if you reboot the Pi and connect to it via USB, you should get a virtual flash drive. Now, if you're ready to send files to it, you can simply use this rsync command. This will work from a Linux or Mac host directly without any additional software to install in most cases. You'll see the drive vanish from your computer when you start the transfer, and then re-appear shortly afterwards:

$ rsync --progress --recursive --verbose --delete source/* rsync://cncdrive.local/cnc

It should give you progress as it transfers the files, and will delete on remote so that the remote folder looks exactly like your local source folder.

Adding Ethernet

As mentioned before, in our specific installation, we didn't have 2.4GHz wireless available at the installation location, but only 5GHz wireless. So we had to add an external ethernet device to the setup.

This was based on the Raspberry Pi Spy's instructions although our module was different to the one pictured, and had different markings on the pins. Also, the documentation for our module had some inconsistencies too which led to some confusion. But here is the lowdown:

  • The documentation for the module we had said it was to be powered by 5V, but the IO level was 3.3V (compatible with the Pi Zero), but this was in fact a lie, and we had to power the module from 3.3V to get it to work.
  • I originally had the MISO & MOSI pins swapped causing it not to work. On our module, the pins as labelled SI and SO, and they're opposite on the Pi - that is, connect Pi->MISO to Module->SI and Pi->MOSI to Module->SO. Read it as "serial in" and "serial out" and it starts to make sense!
  • Our module had a MAC address already assigned to it, and didn't need us to choose a MAC address or set this at boot.

Once wired up, setting up the Pi was very simple:

$ sudo nano /boot/config.txt

... add or alter:
dtparam=spi=on
dtoverlay=enc28j60

$ sudo reboot

On reboot, the ethernet port will appear as eth0, and automatically aqcuire an address via DHCP if it can. It's actually that simple...! I was surprised actually by just how easy the ethernet setup was.

Installation

For installation, I made a small panel out of acrylic that the Raspberry Pi Zero and the ENC28J60 modules were screwed to, to keep them together. Then we made a larger box to house the wireless router, USB power supply for the Pi, and a powerboard. This kept all the components safe and shielded them from the very dusty woodworking shop that they were installed into. This box was mounted on a wall near the CNC router, and the USB A-to-A cable was used to connect to the CNC. The A-to-A cable was 3 meters long allowing us to route the USB cable out of the way, and it just plugged into the front panel of the CNC router.

Should the Raspberry Pi Zero fail, we can easily unplug it from the CNC router and resume using the USB thumb drive as we used to do, allowing us to keep working even if it fails.

Results

A week after the installation, the user of the machine was very happy with this addition to their CNC router. It's saved them a lot of time swapping USB drives and has made for a much easier workflow for them, as they're often tweaking or adjusting gcodes for various jobs, so as to get the perfect results. Even though it's not that user friendly, everything is handed by a single click on the Mac, so it's more than easy enough to use for this application.

Future improvements

With more time, we'd make a few improvements to the system:

  • Use a password and/or encryption when talking to rsync. In our installation, this was basically the only device on this segment of the network, so we can work around not having additional security.
  • Add a web-based file manager with some extras onto the device. This would then allow use from a Windows computer via a web browser instead. A quick search shows a couple of simple open source projects which would be a good starting point for a basic web file manager, allowing uploads and deletes. We'd have to expand it slightly to allow manually "unplugging" the USB drive and "plugging" it back in, but this should be straight forward to add to an existing system.