Using Syncthing as a DevOps Tool Across Multiple Platforms
Are you ready to use a robust open source tool to streamline your cross desktop DevOps experience? Read on.
First things first
I’ve been an OS agnostic geek my entire career. I tend to lean more toward open source tools to get things done on any platform but in the end, I’ve never been able to settle with using a single OS to do all my work. I have a Macbook Pro for Docker/build automation testing, a Linux desktop/server running a Kubernetes cluster (and more), a Windows gaming desktop for when time permits, and a Windows 10 laptop I tote with me just about everywhere that runs a number of vagrant boxes on demand for specific scenarios. These are all connected via an always-on VPN SSL based darknet-style mesh network managed on a raspberry pi device on my network (which is always on for me to use as a quick shell in a pinch). This combination allows me to access my fleet of tools, data, and work systems from just about anywhere in the world with a network connection in a secure manner without any third party cloud providers being involved.
This layout works well for me, but working across all the various platforms can be a bit of a bummer. In the past I’ve installed Dropbox to help keep a limited number of the important project files I use synced (of course many of these are already safely in a git repo somewhere). But soon Dropbox will stop supporting Linux platforms.
So recently I’ve revisited my work configuration and have flipped over to using the OSS Syncthing app to effectively perform the same function as Dropbox but without the size restrictions, third party storage of my data, or sudden service outages. I’ve found this setup to be incredibly useful. This article goes more into this tool and how you might use it for cross platform project work.
Installation
Thus far I’ve installed Syncthing on my Mac, Windows, and Linux desktops. Syncthing has a handful of GUI clients to interface with its exposed API engine that is at the core of the project. The links to such GUIs are at the Syncthing website. For Windows I installed SyncTrayzor, for Mac I installed syncthing-macosx, and Syncthing-GTK for my Ubuntu based box.
Synced folders
Folders I’ve created:
Zloeber-Projects – big folder of data
Zloeber-Profile – shell profiles and such
The drive location, name, and path are irrelevant. Choose and use what works best for your needs. In my case I ended up with the following:
Windows: F:/Projects
OSX: /Users/zloeber/Projects Linux: /home/zloeber/Projects
WARNING: If the ‘Projects’ folder already exists between workstations that is generally ok as they will merge sync when everything is done being configured. As always, ensure you have appropriate backups of your important data before attempting to sync it though! Also remember that whatever folders you are syncing will require the same disk space on all devices you sync across.
Ignored folders
You will want to put in some sensible ignore patterns as well. Otherwise very large files you may only want to keep local, will needlessly thrash your network and possibly even cause you issues. For example, I run localized Vagrant files in some projects. This allows me to ‘vagrant up’ across three systems simultaneously without worrying about them interfering with one another. As such I always ensure that ‘.vagrant’ is ignored so only the local version will be used.
Ignore patterns can be manually defined in the included web interface (or sometimes in the included GUI wrapper tool) but they can also be dropped in the root of a synced folder in a special file called .stignore. Here is what I’m currently using for my ignore patterns, update to suit your particular needs:
(?d)**/__pycache__ (?d)*.pyc (?d)**/logs/ (?d)**/*.log (?d)**/*.log.* (?d)**/tmp/ (?d)**/temp/ (?d).vagrant (?d).DS_Store
The (?d) at the beginning of each line means that this pattern is allowed to be deleted if it is found within a sync delete operation (Otherwise you will see sync errors slowly start rising!). This makes things quite a bit easier to manage and will keep you from scratching your head around files/folders not syncing.
Caveats
I noticed that you have to be a bit careful with the folder settings on a per workstation basis. For example, on my Linux workstation the Syncthing daemon defaulted newly added shared folders to be ‘Receive Only’ instead of ‘Send & Receive’. This would cause sync issues were you to actively begin working on Linux workstation. Just ensure that all devices you will be making file changes from are locally configured with the correct folder type and you should be fine.
End result
For me the results work beautifully. I’m able to grab my Mac or Windows laptop, work on or offline, then know that the project files will be available for me on any other system when Syncthing gets its chance to connect.
Next steps
Sky’s the limit! That is if you have the storage on your devices. But there are other more interesting uses I can imagine. For instance, I see Syncthing being a phenomenal local supplement for cloud based backups. Simply setup your synced folder to point to an existing local cloud storage local folder (OneDrive or Dropbox) and you can keep your data synced locally and to the cloud, regardless if the OS supports the cloud storage client software.
