Remote file uploader / sync / offloader

[drmike]

So, I have remote nodes here and there that do various things.  Some record audio and video content and that's the most aggravating to deal with, mainly due to size.   Historically I've done batch downloads of the data, as needed.  That gets silly on time and sometime bump into disk space issues.

All files need to head to a central location which is remote.  Think centralized storage.  The original file post upload needs purged.

Goal is to get these files off the end recorder nodes whenever the activity is done (i.e. after recording).  Things like rsync may or may not suffice and dealing with open files and potential for wasted bandwidth with partials.  Plus need to make sure entire file from origin reaches destination.


Is anyone doing something similar?  What tools are your using to accomplish this?

 

[Cloudrck]

The simplest method would be to create a shell script that rsync's to some a remote server (preferably one with an optimized file-system). Rsync will handle the bulk of the work with transferring and making sure it get's there for the most part.

The bulk of the logic you'd need to write would be determining when files are done recording. I'm not familiar with how A/V works in terms of recording data.

Of course there are a quite a few other options. You could always use a pre-made software like Bacula

 

[drmike]

Looks like I am going to experiment with grafting rsync onto the end of the recording script.  Heavily use rsync elsewhere, so not new to me.

Dealing with tmp and partial files is the puzzle piece.

Off to setup some small recording jobs and perfect rsync commandline to deal with this gracefully.

 

[kro]

You can have rsync trigger on file modification etc -> I use this for some real time syncing stuff.
Can get a bit heavily involved though, perhaps why its not so commonly mentioned. But I've been using this for yonks ahahha

http://www.csync.org/
 

Hope this helps ^_^

 

[drmike]

Figured out what I need a combo tool to support:

 Inotify and Rsync

Inotify for the for file close.... Spawn rsync on the directory.