Tips For a Local Cloud
I've always liked the allure of the cloud as a place where I can store data and run services. However, I find the current offerings a bit lackluster, at least in terms of pricing, data-control, and privacy.
After thinking it through, I decided I wanted my own place to store and process data. This local cloud, so to speak, would not be a public-facing service, but rather a server on my local network I could link different devices up to.
To do this, I use a couple of raid'd 2TB disks as storage, mounted as a network file system (via the SMB client
samba), running through a cluster of Raspberry Pi 3s and 4s. These rpis act as one system that I can use to create and store data and run long-running processes—a local cloud if you will.
I ran into some issues, the primary one being network latency. I'm a big proponent of Rust. Compiling large Rust projects became a lot slower, as the the compiler had to do all of its reads and writes over a network.
To prevent this IO-over-network slowdown, I really had two choices:
- I could set the default compilation target to be a location on my local disk, so all further reads/writes are local.
- I could sync certain folders between the local cloud and the local machine.
The latter option is more appealing; it allows me to access certain files when I'm not using them. Additionally, I can still defer the compilation of large projects to the server as needed (i.e. CI/CD), while still retaining fast local compilation for quick iterative development.
Syncing Local Files
So, how to sync files across devices? At first, I though that Syncthing would be the best choice. Syncthing also works over the internet, so it's possible to access your local cloud from outside your local network.
However, I ran into some roadblocks pretty quickly. Syncthing was slow to propagate changes, crashed for seemingly random reasons, and had trouble detecting when files were changed.
Now I've used Syncthing in the past and it's worked pretty well. Not many people are stupid enough to use a cluster of RPI as an honest-to-goodness server, so this is more of a use-case issue on my end.
I decided to forfeit this methodology after I discovered the
rsync utility, which is built in to most *nix computers. With
rsync I could unplug a drive from my local cloud and carry it with me, having it sync whenever I plugged it back into my computer. The commands I use to sync the drives with rsync are loosely:
# Sync a computer to a disk rsync -aE ~/some/local/folder /Volumes/cloud/folder # Sync a computer to a local cloud rsync -aE ~/some/local/folder smb://email@example.com/folder
These commands start a daemon that worries about syncing the disks for you. That's it for now 😊.
An adventure awaits...
If it's been a long day, there's nothing wrong with giving up and going home.
I'm usually pretty permissive with what I create. However, my writing is my writing and my opinions are my own. Although this website's source code is released under the MIT License, all published content is © Isaac Clayton.