Syncing files between two servers (mac mini)

[unplugme71]

I have two Mac Mini's with USB 3.0 RAID 10 storage.

I work with files from one Mac Mini through my MacBook Air. After I save what I'm working on, I remote in, and run the rsync script that copies from server 1 to server 2.

This is tedious and time consuming.

I'm considering making rsync run every 15 minutes. My concern is, if rsync needs more time to copy before the next 15 min, will another rsync session start? And if it does start, will it try and copy the same file twice because rsync does not yet see the large file copied over completely yet?

Also, I notice that even if I don't update some files, rsync will still copy them because their modified date changes or something.

Just trying to figure out a way to keep the two servers in sync. Most of it is a one way sync (Server 1 -> Server 2).

Server 3 backups Server 2.

Otherwise, I might just run two copies of Windows Server 2012 and use DFS/Replication but I'm concerned about Mac files being on a windows share as I heard issues with file forks, meta data, etc.

 

[northerngit]

I have two Mac Mini's with USB 3.0 RAID 10 storage.

I work with files from one Mac Mini through my MacBook Air. After I save what I'm working on, I remote in, and run the rsync script that copies from server 1 to server 2.

This is tedious and time consuming.

I'm considering making rsync run every 15 minutes. My concern is, if rsync needs more time to copy before the next 15 min, will another rsync session start? And if it does start, will it try and copy the same file twice because rsync does not yet see the large file copied over completely yet?

Also, I notice that even if I don't update some files, rsync will still copy them because their modified date changes or something.

Just trying to figure out a way to keep the two servers in sync. Most of it is a one way sync (Server 1 -> Server 2).

Server 3 backups Server 2.

Otherwise, I might just run two copies of Windows Server 2012 and use DFS/Replication but I'm concerned about Mac files being on a windows share as I heard issues with file forks, meta data, etc.

I'm unaware of native filesystem replication in OS X. Are you replicating for backup purposes, or to enable multiple users access to the files?

Could you paste your rsync command? Certainly date changes shouldn't cause a replication event. Are you rsync'ing the full file on change, or just the deltas?

There are many tools like ChronoSync and SyncMate that allow schedule to be set and agents run on local machines for seamless replication. ChronoSync is my personal favourite! Allowing bi-directional mirroring of content.

Of course, this won't solve multiple user access issues and maintaining file locks.

 

[unplugme71]

I'm unaware of native filesystem replication in OS X. Are you replicating for backup purposes, or to enable multiple users access to the files?

Could you paste your rsync command? Certainly date changes shouldn't cause a replication event. Are you rsync'ing the full file on change, or just the deltas?

There are many tools like ChronoSync and SyncMate that allow schedule to be set and agents run on local machines for seamless replication. ChronoSync is my personal favourite! Allowing bi-directional mirroring of content.

Of course, this won't solve multiple user access issues and maintaining file locks.

The replication i'm discussing is to maintain a spare 'live' file share. No one connects to the 2nd share unless the first one fails. I also have another drive that backs up every 4 hours (incremental) and then once a week (full).

How can I sync the deltas? I think its a full file on change which is causing things to be slow. I'm sure its something with iTunes. I'm guessing the play count is stored within the movie or song? Could be why my replication is slow.

If I can get just the changes to be copied over, that would be much quicker and save me a ton of time/resources!

 

[Alrescha]

I'm considering making rsync run every 15 minutes. My concern is, if rsync needs more time to copy before the next 15 min, will another rsync session start? And if it does start, will it try and copy the same file twice because rsync does not yet see the large file copied over completely yet?

You could create a script that never ends, sleeping for 15 minutes, running rsync, then going back to sleep.

Alternatively, you could create a script that writes a lock file that subsequent invocations check for before they start rsync.

A.

 

[jg900ss]

Syncing two servers, using SYNK

I have three MacMini servers (the Core 2 Duo family) that are positioned in different countries. I used SYNK, a simple scripting product, to keep certain directories on the servers identical so that members could log into any one of the three servers and count on finding identical up-to-date information. The product was not expensive, and my Mac consultant was able to write scripts that kept those files/directories in sync at great distance, different time-zones, and securely using VPN. Something to explore?

 

[unplugme71]

You could create a script that never ends, sleeping for 15 minutes, running rsync, then going back to sleep.

Alternatively, you could create a script that writes a lock file that subsequent invocations check for before they start rsync.

A.

I might try your second option first, if that seems too difficult to perform I will go to the first.

 

[Alrescha]

I might try your second option first, if that seems too difficult to perform I will go to the first.

# implement the lock file
ls sync.lock > /dev/null 2> /dev/null
if [ "$?" -eq "0" ]
then
exit
fi
#
touch sync.lock
#

<do stuff>

# remove the lock as we are done
rm -f sync.lock

A.

 

[mkelly]

Bittorrent Sync

I have a similar setup with two laptops and a Mac Mini server. I work primarily on one of the laptops but want the data available on both the secondary laptop and server.

My first solution was some custom rsync scripts that would sync things around (and utilized a lock file to prevent more than one rsync from running at a time). It worked, but it wasn't pretty ;-)

I've been experimenting with BitTorrent Sync for Mac, and I'm very happy with it. It seems to be keeping everything in sync nicely between the three machines, does so reasonably quickly, both on the LAN and remotely, and includes the option to keep a versioned history of changed files per directory (which you can enable/disable on a per-directory, per-machine basis). Also, it's free.

http://www.bittorrent.com/sync

 

[unplugme71]

# implement the lock file
ls sync.lock > /dev/null 2> /dev/null
if [ "$?" -eq "0" ]
then
exit
fi
#
touch sync.lock
#

<do stuff>

# remove the lock as we are done
rm -rf sync.lock

A.

Great! I'll give that a try.

Thanks