Asynchronous Syncing

2024-04-04

---

Recently I've been thinking about how I can reduce my computing and energy footprint. I'm trying all sorts of new workflows, from using low-power devices to removing GUIs on some of them. Where I feel comfortable doing it, I am also removing Syncthing and not have an always-on syncing program running.

I hope to not need Syncthing at all one day. It would be nice to have a store-and-forward or other asynchronous tool, similar to git or NNCP, that can look at the differences between two directories and resolve their divergences. That way I can keep only one or two small servers as file hosts, or even as dumb relays that simply connect machines to each other like croc.

The problem is that I have a lot of files I sync, and some of them change rapidly. I share my password database, my to-do list, several collections of notes, my camera photos, and even backups of video game save files. Not all directories are shared with every device, and not every device peers with every other device. The system is rather complicated, in all honesty.

I see several tools that work similarly to what I'd like to have. git comes close, for example: it requires no central server or relay, as long as the two nodes can connect to each other directly. It can bring diverging directories in line automatically, has a built-in conflict resolution mechanism, and it can even update asynchronously using `git bundle`. However, git keeps a full history of all the files in the directory; I just want to know which files need updating. It would also be nice if git had the ability to pull data from multiple up-to-date sources at once.

rsync is another tool that is similar to what I want. git's commit history makes it easy to see what files need to be updated; rsync, on the other hand, uses incremental file lists, shared in real time with the two endpoints to decide what to sync. rsync would actually come closest if it had the ability to write the incremental list to a file or data stream, and for the remote node to be able to read that file or stream. That would provide the asynchronous aspect I'm looking for. But I don't know of any way rsync can do it without additional scripting.

Other tools begin to diverge from my goals. BitTorrent and croc can use relays, and BitTorrent can even send or receive files from multiple peers at the same time. But neither tool can handle file updates--BitTorrent is specifically designed to preserve files statically. I know Resilio Sync can handle file updates while being based on BitTorrent, but that's a proprietary tool, and naturally I prefer FOSS programs.

There are a few other tools I still need to look into, like Unison. But a cursory glance at most of them tells me they're not exactly what I'm looking for.

Ideally, I'd like to have a workflow like the following:

I want to sync directory A on machine M with directory B on machine N using some on-demand syncing tool--let's call it "odsync".

I use odsync on M to create a file/data stream, which has information about all the contents of A.

I send the data from M to N using a tool of my choosing, such as bundling it into an NNCP packet or opening a TCP connection with N (either directly or over a relay).

odsync on N receives the data, either by being called directly with the data or after being called by a daemon listening for the data on the network.

odsync on N analyzes the information from M and determines which files and directories to send back from B in order to bring A up to date. It also determines which files and directories it needs to request from A in order to bring B up to date.

odsync gathers files and directories in B to send to A, as well as a list of requested files and directories from A.

I send the gathered data and request from N to M using a tool of my choosing. It does not have to be the same tool as earlier in the process.

odsync on M updates A with the files and directories from B.

odsync on M gathers the requested files and directories for B.

I send the gathered data from M to N using a tool of my choosing. It does not have to be same tool as earlier in the process.

odsync on N updates B with the files and directories from A.

It sounds like a lot, but many store-and-forward tools offer scheduling to automate tasks like this, if not outright transfer tools of their own, such as NNCP's call command. I'd especially like it if a listening daemon was decoupled from the tool that performs the syncing, as is the case with NNCP and is not the case with tools like Syncthing or Resilio Sync.

If anyone knows of any tools that work like this, I would be very interested to hear about them.

---

Up One Level

Home

[Last updated: 2024-04-04]