Self-hosted Distributed Storage
- Thread starter HalfEatenPie
- Start date
Question for everyone tinkering with SyncThing - Does it sync files opened for writing?
I'm mostly curious if it syncs streaming data. Namely, you're WGET'ing a file to the FS,
is it syncing all of that on the fly or does it wait for the file to be complete before
blasting it off?
Francisco
I'm mostly curious if it syncs streaming data. Namely, you're WGET'ing a file to the FS,
is it syncing all of that on the fly or does it wait for the file to be complete before
blasting it off?
Francisco
drmike
100% Tier-1 Gogent
drmike
100% Tier-1 Gogent
So @Francisco SyncThing.....
Rescan Interval
30 s
Lowered that from 60s.
Ran Debian ISO of 4GB~ DVD. Slow downloading (BW + VPN lag).
SyncThing on the remote end (this in datacenter where I am replicating to):
Global Repository 237 items, ~1.19 GiB Local Repository 237 items, ~634 MiB Out Of Sync
On the local SyncThing master (here on my desk):
Global Repository 237 items, ~1.24 GiB Local Repository 237 items, ~1.24 GiB Out Of Sync 0 items, 0 B
The local repository continues to group in GiB size:
Global Repository 237 items, ~1.25 GiB Local Repository 237 items, ~1.25 GiB Out Of Sync 0 items, 0 B
On the local master repository here on my desk:
Download Rate 61 bps (261 KiB) Upload Rate 208 kbps (54.4 MiB) Address -:22000 Synchronization 74%
The upload rate = current throughput and total transfer to remote since this session was initiated. This will at times drop and reconnect and thus may be subject to reset of values.
So it *apears* the remote replication is happening while I download the file locally with wget.
I need to figure out where SyncThing stashes partial file transfers like this to confirm where we are.
Rescan Interval
30 s
Lowered that from 60s.
Ran Debian ISO of 4GB~ DVD. Slow downloading (BW + VPN lag).
SyncThing on the remote end (this in datacenter where I am replicating to):
Global Repository 237 items, ~1.19 GiB Local Repository 237 items, ~634 MiB Out Of Sync
On the local SyncThing master (here on my desk):
Global Repository 237 items, ~1.24 GiB Local Repository 237 items, ~1.24 GiB Out Of Sync 0 items, 0 B
The local repository continues to group in GiB size:
Global Repository 237 items, ~1.25 GiB Local Repository 237 items, ~1.25 GiB Out Of Sync 0 items, 0 B
On the local master repository here on my desk:
Download Rate 61 bps (261 KiB) Upload Rate 208 kbps (54.4 MiB) Address -:22000 Synchronization 74%
The upload rate = current throughput and total transfer to remote since this session was initiated. This will at times drop and reconnect and thus may be subject to reset of values.
So it *apears* the remote replication is happening while I download the file locally with wget.
I need to figure out where SyncThing stashes partial file transfers like this to confirm where we are.
Last edited by a moderator:
drmike
100% Tier-1 Gogent
The partial transfer file in SyncThing is given a period prefixed name:
.syncthing.debian-7.6.0-amd64-DVD-1.iso
= 70M
wget on the local server is up to about 750M currently....
Yes, my upload speeds are much slower than download speeds.
Will continue to watch and see how this does.
.syncthing.debian-7.6.0-amd64-DVD-1.iso
= 70M
wget on the local server is up to about 750M currently....
Yes, my upload speeds are much slower than download speeds.
Will continue to watch and see how this does.
drmike
100% Tier-1 Gogent
That ISO is still chugging, slowly along on the replication.
Gets mega confusing if you have multiple web interfaces open for the end points in your SyncThing setup. Cause on the master progress percentages aren't updated until the file is finished and complete on the other side. In this instance I have like 1.6GB uploaded to remote and another 2GB+ to go on mirroring the ISO.
It appears to make some attempt to start/restart the files out of sync. Waiting until done to see total bandwidth chewed up, but so far, looks right for file it is moving and not multiple retries or anything weird on full parts of files wasted/flushed...
I need to send up two ends on remote servers to pace this properly... This week
Gets mega confusing if you have multiple web interfaces open for the end points in your SyncThing setup. Cause on the master progress percentages aren't updated until the file is finished and complete on the other side. In this instance I have like 1.6GB uploaded to remote and another 2GB+ to go on mirroring the ISO.
It appears to make some attempt to start/restart the files out of sync. Waiting until done to see total bandwidth chewed up, but so far, looks right for file it is moving and not multiple retries or anything weird on full parts of files wasted/flushed...
I need to send up two ends on remote servers to pace this properly... This week