How it works

A star around one master. Clients never talk to each other: each pushes its changes to the master and pulls everyone else's from it. The server is stateless between runs apart from the index and the per-client bases it keeps on disk.

One session

Every run is a short-lived process. In outline:

1. The client checks its root and takes its lock

   The sync root must contain the .filesync-root marker that filesync init wrote, so an unmounted drive is reported rather than treated as a tree of deletions. A local lock keeps two runs of the same repo apart.

2. It opens the transport

   Over ssh, with BatchMode=yes and a forced command pinned in authorized_keys. The pinned command carries the repo, the client id and the role, so the session itself never chooses the server root. The local transport runs the same protocol over a pipe for testing and single-machine use.

3. It scans the tree and sends a manifest

   The walk is sorted, so manifests and logs are reproducible. A cache of size and mtime decides which files need hashing. Ignore patterns are applied on both sides. The manifest has one entry per file, and it is sent whole: there is no change journal.

4. The server plans against its index

   Holding the repo lock, the server compares the client's manifest with the master index and with that client's base, and answers with one plan: what to pull, what to push, what to delete, and which paths conflict.

5. Both sides move the files the plan names

   Pulls are requested in chunks and pushes are sent in chunks; each 256 KiB chunk waits for its acknowledgement, so one request is in flight at a time. Files of at least 64 KiB are sent as block deltas when the other side already holds an older copy.

6. Transfers are verified, then committed

   An upload is checked against its SHA-256 at the end and only then renamed into the master. A pull is checked against the hash in the plan; a mismatch fails that pull and the next run retries. Temporary names carry a .filesync-tmp- prefix, which the backup example excludes.

7. State is written and the run is logged

   The server rewrites the index and the client's base and appends to the repo journal. The client updates its cache, prunes its local trash unless --no-prune was given, and appends to ~/.filesync/logs/REPO.jsonl whatever the outcome. The summary reports pulled, pushed, deleted and conflicted counts, the payload bytes in each direction, and how many files moved as a delta.

Transfer and deltas

Each control message is one JSON object on one line. A message that announces a payload carries len, the bytes on the wire, and z when the payload was deflated; the payload follows the line directly. Only payload bytes are counted in the transfer report; headers and control messages are excluded.

Deflation is used when it makes the frame smaller. For a file the other side already has an older copy of, the sender builds a signature of the receiver's blocks and sends copy instructions plus literals, rsync-style. The rolling scan runs in pure Perl, so it covers at most the first 8 MiB after the unchanged prefix; later changes go as literals. Offsets and lengths are 32-bit on the wire, so a delta cannot refer beyond 4 GiB into the receiver's file; larger remainders fall back to whole-file transfer.

Writing is atomic between processes: every file is written to a temporary name and renamed into place. Nothing is fsynced, so a rename survives a crash as a name but not necessarily as durable data.

Conflicts

When both sides changed a path, the newer mtime wins, and the master wins ties within two seconds. The loser is not discarded: it is kept beside the winner as a conflict copy named before the extension, for example

report.conflict-laptop-1-20261022T101500Z.txt

The name is capped at 200 bytes so it fits common filesystems. The server keeps a conflict copy under the planned name whether or not the client names it, and refuses any other name; acknowledgements are accepted only for actions the client was told to apply.

Because the tie-break uses mtimes, a client with a wrong clock can pick the wrong winner. The losing content is still preserved, so the cost is a merge, not lost data.

Deletions and trash

A deletion is a plan action like any other, but it is bounded: by default a session may delete at most 200 files or 10% of the known files, whichever limit is reached first. A tree that suddenly looks empty therefore stops the run instead of being propagated. filesync sync --force proceeds when that is really what you want.

On the client, a removed file is moved into that repo's local trash, kept for 30 days by default. A second deletion of the same path on the same day is stored as PATH.1, then PATH.2, and so on. The server keeps its own trash, with the retention set per repo.

filesync restore notes/a.txt --list
filesync restore notes/a.txt
filesync restore notes/a.txt --to /tmp/recovered

Restores copy rather than move, so the trash entry survives and the operation is repeatable. The destination must not already exist.

Replica snapshots

A machine registered with --role replica may take a daily snapshot of its tree. Snapshots live in .filesync-snapshots inside the replica root and are made by hard link or by copy, chosen with --snapshot-mode. Fourteen are kept by default.

A hard-linked snapshot shares the file with the live tree. The sync replaces files by rename, which leaves the snapshot's link intact, but any other tool that edits a replica file in place also changes every snapshot that links it. Use snapshot_mode copy if anything but filesync writes there.

Snapshots guard against a bad sync, not against losing the disk: they sit on the same filesystem as the replica.

On-disk layout

On the client

On the server

The master is a normal tree, so backup is rsync and restore is rsync plus filesync-admin reindex. The index is a JSON file rewritten whole at the end of each session.

Locking and concurrency

Both the client and the server take locks, and a refusal is not an error: filesync exits 2 so cron can tell "a run was already in progress" from "a run failed". The plan is computed once, from a scan taken while the repo lock is held; something else writing into the master during a session is not seen until the next one.

flock is advisory and, on NFS, may not lock across machines at all. Keep the server root and the client's ~/.filesync on local disks.

Security model

• Authentication is ssh's. The client id is a name for bases, journals and conflict copies, not a credential. ssh runs with BatchMode=yes, so cron cannot sit at a password prompt.
• The server pins the session. sshd starts filesync-server with --repo, --client and --role from authorized_keys. An ssh server of the form -v@host, or a program name starting with -, is refused rather than passed to ssh as an option.
• Ids are plain. A repo or client name must match [A-Za-z0-9][A-Za-z0-9._-]* and be at most 64 characters, so ../x cannot become a path.
• Paths are validated on the wire. No absolute paths, drive letters, backslashes, NUL or control characters, empty or ./.. components; at most 4096 bytes, and no component over 255. Symlinks are not followed.
• Frames are bounded. A payload length must be a number no larger than 64 MiB, and the receiver refuses a frame that does not match it. An upload larger than announced is refused and discarded.
• Errors carry no file paths by default. Commands strip Perl's at FILE line N suffix; FILESYNC_TRACE=1 turns every error into a full stack trace when debugging.