Synchronised Home Directory / Workspace

[splitice]

I want to create a synchronised / remote home directory setup, the idea would be to provide working space to staff members to use on remote nodes. This space would be in sync between all nodes, ideally not actually stored on the nodes themselves but on a remote server and accessed via something like NFS.

NFS is an option, however due to its semantics and being in kernel, it could be difficult to get setup (i.e would like it to be dismounted when not in use).

Preferably user-space (FUSE). Bonus points if it has block level support (i,e its possible to use to dump possibly large log files). I am not looking to run something like mysql on it, ideally it would be to serve as a script library + work space.

Ideas people?

 

[Francisco]

What about btsync? I'm positive I've seen people using it.

Francisco

 

[tchen]

s3ql  not as far as pure block-level, but it does split files and reads/writes only those parts required (effectively yes).

 

[howardsl2]

Have a look at my SSHFS tutorial for CentOS and Ubuntu (using FUSE). It is quick and easy.
https://blog.ls20.com/using-sshfs-to-share-folders-between-your-servers/

 

[mojeda]

What about btsync? I'm positive I've seen people using it.


Francisco

btsync syncs files to local computers.

What about Samba?

 

[Francisco]

btsync syncs files to local computers.

What about Samba?

I thought btsync kept everything in sync in all directions?

Francisco

 

[splitice]

Some interesting suggestions so far, good work.

Have a look at my SSHFS tutorial for CentOS and Ubuntu (using FUSE). It is quick and easy. https://blog.ls20.com/using-sshfs-to-share-folders-between-your-servers/

Interesting, as far as I know SFTP put transfers the entire file which sucks a little.

s3ql  not as far as pure block-level, but it does split files and reads/writes only those parts required (effectively yes).

Very interesting, entirely user space too. Neat.

What about btsync? I'm positive I've seen people using it.

Francisco

We already use btsync in our infrastructure.

Its not really suitable due to:

 - Frequent updates arent handled well (remote slave back over the top of master) if any nodes clock is out by more than 1-2 seconds.

 - Doesn't synchronise permissions, any unix specifics or symlinks

 - Requires a separate share / hash to be created for every user, and a seperate daemon to be constantly running due to the re-index being slow / intensive

Keeping permissions intact is very important for the setup I am considering with multiple staff members involved, not to mention how annoying / impossible it is to fix permissions on large scripts (with includes etc).

We use btsync to synchronise packages and it works well enough for that (even if it does struggle with the 200,000+ .git/* files). It also has other issues, such as dynamically choosing its ports and not supporting any form of bind address.

btsync syncs files to local computers.

What about Samba?

Given how slow it is on local networks I don't imagine it would work suitably, plus as a kernel mode module I would probably use nfs instead if came to that.

 

[mojeda]

Last I checked you can specify the ports that btsync uses as well as the IP.

 

[perennate]

Last I checked you can specify the ports that btsync uses as well as the IP.

And?

 

[drmike]

Seems like options are:

1. A distributed/replicated filesystem.

2. Btsync or similar P2P

3. SSHFS + ????

I use SSHFS quite a bit.   It's nice cause once up and perfected it just works.   Plus you can use many common tools with it.  My favorite, good old rsync.   I just run an aggressive CRON job with rsync.   But better ways to do that, say based on actual underlying disk activity.

Btsync is on my to-do list.  Sounds intriguing, but not preserving file permissions sucks.

Shortcoming of generic rsync implementation is lack of versioning and preservation of prior files/edits.  That can likely be worked around creatively also.  Lots of scripts out there with all sorts of wizardry and cooked up hackery.

 

[drmike]

and... of course...

#4, SSHFS  + Unison

 

[splitice]

Last I checked you can specify the ports that btsync uses as well as the IP.

The Web UI port can be specified and disabled, but ports for transfers and data communication can not be.

 

[splitice]

Currently s3ql is looking very promising, I just need to work out what an appropriate backend would be. I am not sure about the S3 datastore and Google Cloud Storage. Personally I would prefer to use a Backupsy instance or something akin.

 

[willie]

In my experience trying to work on an active remote disk is painful even over a LAN.  Better to work locally and sync on demand, using rsync or git depending on the data.

 

[mikho]

In my experience trying to work on an active remote disk is painful even over a LAN. Better to work locally and sync on demand, using rsync or git depending on the data.

Really depends on what you are working with.
Opening a gazillion files over lan/wan can be painful if you have bad latency.


It's all about the connection.

 

[mojeda]

The Web UI port can be specified and disabled, but ports for transfers and data communication can not be.

These are separate from the web ui. However I was wrong about being able to make btsync use a specific IP, not just the web ui.

 

[tchen]

Currently s3ql is looking very promising, I just need to work out what an appropriate backend would be. I am not sure about the S3 datastore and Google Cloud Storage. Personally I would prefer to use a Backupsy instance or something akin.

At the end of the backends docs, there's the bit about local storage.  Frankenstein a sshfs mount (silly as it may sound).

 

[splitice]

These are separate from the web ui. However I was wrong about being able to make btsync use a specific IP, not just the web ui.

It still creates a port for each active transfer. Yes the listen port can be set.

 

[Taronyu]

The problem I had with sshfs is that I first had to browse to my directory I needed and only then Sublime text was able to see the files. Might have been a unstable connection but ai never tried again.


Send from my Galaxy S3

 

[blergh]

I've had better luck with btsync than sshfs, should be the other way around, no?