macOS rsync changing modification date on files not modified since last copied

[johngpt]

I've been reading about using the terminal command line input for updating folders/directories containing back up copies of photos stored on external hard drives.

I have back up copies of folder/directories on several external drives.

Copying a new folder/directory isn't a problem using cp -pvR, where -p preserves pretty much everything, -v verboses the output in the terminal window, and -R allows the directory to recursively be copied to the appropriate directory on the external HDD. Sometimes using Finder dragging and dropping the folder to be copied to the appropriate folder on the external HDD runs into snags so I've gotten into the habit of using Terminal.

So I've done a lot of reading in man pages and via googling. The rsync with appropriate flags looks like the answer, but each time I've used it to update the contents of a folder/directory on an external HDD, it's changed the modification dates/times of file/photos that haven't really been altered. Not all photos. Just those with a tif extension, and not even all of those. The date changes to the date of the synchronizing. Running rsync today changed the modification date to "today," as seen in a Finder window.

The command I've tried is

rsync -aEvP

Where -a preserves pretty much everything, -E preserves extended attributes (Apple specific for Darwin), -v verboses the output to the terminal window, and -P shows the progress under the verbose stuff's line.

I've tried rsync -aEvPt, but the -t didn't help.

Thinking that there might be a problem with the change from daylight time to standard time, I tried:

rsync -aEvP --modify-window=3601

Where the --modify-window=3601 causes rsync not to update the file unless the file has changed by longer than an hour.

That also didn't change the rsync behavior.

.

After using the rsync and seeing the changed modification dates, following a suggestion found at apple support forum, I ran:

ls -Tl for the folder/directory on my internal HDD in one Terminal window and the same command for the folder/directory on the external HDD in another Terminal window.

The output of ls -Tl shows what it should for the folder of photos on the internal HDD and it shows the modification dates for the folder of photos on the external HDD to be today's date and the time is the time of the synchronization. This mirrors what the Finder windows show.

So using the command rsync -aEvP is creating a change in some of the tif files that it analyzes and decides to rewrite at the target directory. It hasn't changed any of the previously copied jpg, dng, or raf files.

I've also tried putting a / (slash) at the end of the source folder/directory to just synchronize the contents and not create a new folder/directory at the target, but that hasn't made a difference either.

So I've been having to copy the whole folder/directory again using cp -pvR or using the command ditto. The ditto command like the cp command (with all its flags) copies the whole folder/directory's contents preserving all the appropriate attributes and modification dates/times. The command ditto seems slower than the cp command.

I'm hoping that someone in the MacRumors forum community might have some insight into the rsync behavior I'm seeing, and possibly have some suggestions that might cause it to accurately transfer files.

I'm using a late 2011 mbp running OS 10.7.5 and the Terminal is at its default settings, saying bash at the top of the window. Terminal is version 2.2.3.

The external HDDs are all HFS+ formatted. The connection is FW800 to the first HDD and FW400 to the other external HDDs, daisy chained. The behaviour of rsync occurs when attempting to synchronize between the mbp and the first HDD or between the first HDD and the second HDD.

 

[johngpt]

Okay, I think I've found the answer.

The problem seems to be in the -E flag.

When I've run rsync -avP only the files that are new to the source directory are copied over to the target directory.

Here is what the command that worked looked like:

John$ date ; rsync -avP /Volumes/2\ TB\ WiebeTech/Temporary_Photo_Backup/2014-09_Arizona_trip/AZ_xt1/ /Volumes/WiebeTech\ 750/Temp_Photo_Backup/2014-09_Arizona_trip/AZ_xt1/ ; date ;

I have the command date in there before and after the rsync command so that I can see how long it required to finish the process.

Just prior to running this, I again tried the rsync -aEvP with the above directories and once again it changed dates on quite a few files. I re-ran the same directories with rsync -avP and it fixed the target directory so that it matched the source directory. I was a very surprised. I totally had thought that I'd have to run the cp command to fix the target directory. I thought that the changed target directory from having used the -E flag would have made it unfixable.

So now I have a good way to run synchronization and not have to waste time checking and rechecking.

I'm leaving my original request for help intact along with this fix that I've found. There may be others who have come across the same problem and this might help.

Cheers,

 

[mfram]

Haven't tested your assertion to see if I see the same behavior, but I wanted to verify: what filesystem are you using on your external disks? If it isn't Mac HFS+, then I can understand why you might see weird behavior. Resource forks are a Mac-only concept, so it wouldn't surprise me if you saw strange behavior from non-Mac filesystems.

 

[johngpt]

Hi mfram, in original post I mentioned that all the external HDDs are HFS+.

I'll be interested in hearing if your system reproduces the behavior.

 

[Alrescha]

Just FYI, the version of rsync that comes with OS X appears to be getting a bit long in the tooth. I use a more recent self-compiled version for my backups. I believe that both BackupList+ and Carbon Copy Cloner include more up-to-date versions of rsync in their products.

A.

Addendum: I have a faint memory of a bug in rsync with regard to certain characters in the filename causing unnecessary file copies - could this be related to the problem you described?

 

[johngpt]

Hello Alrescha, I've read in my googling about newer versions of rsync. There are more flag options available it seems within those versions than are available in the version that's within my now comparatively aging OS. Googling for and reading online man pages for Darwin's latest rsync shows options unavailable when I read my man pages. And reading about rsync in Linux forums show more options also.

I'm happy that I've stumbled upon a way to accomplish what I've needed.

That -E flag seems to have been the culprit. I've since continued quite successfully to update directories with only those files that have been added.

What I need to check now is whether rsync -avP is able to update a previously existing file whose name hasn't changed.

For example I've created foo.tif and it exists on my internal HDD and over on an external HDD as back up. Then I decide to make changes to foo.tif, opening it in photoshop from the file on the internal HDD. I make my changes and save it. Then later I use rsync to synchronize the directory in which foo.tif resides on my internal HDD with the directory within which my back up foo.tif resides on my external HDD. Does rsync make those changes within the file on the external HDD?

I can see that rsync is copying over to the external HDD any new photos I've created, but I haven't yet checked to see if it's updating changes to pre-existing photos that are there.

I've very glad you posted as in responding to you, I've thought of an important aspect to check upon.

 

[Alrescha]

Then later I use rsync to synchronize the directory in which foo.tif resides on my internal HDD with the directory within which my back up foo.tif resides on my external HDD. Does rsync make those changes within the file on the external HDD?

The short answer is yes, if rsync can detect that a file has changed (size, modification date, etc.) then it will update the destination. There are several switches available to modify the behavior - for example you could tell rsync to checksum each file as a (very slow) detection mechanism.

I have not run into any problems with rsync failing to copy files, but rather failing to recognize that some files have already been copied. I attributed this to that character set bug I referred to earlier.

A.

 

[johngpt]

The short answer is yes, if rsync can detect that a file has changed (size, modification date, etc.) then it will update the destination.

A.

Thank you. I just checked and sure enough, rsync updated as it should.

From an external HDD I opened a photo into Ps and it did not show the changes I had made to the original on the internal HDD last night.

I used rsync to update the contents of the directory on the internal HDD to the directory on the external HDD, while watching what happened in a Finder window of the target directory.

And just as you said, that photo in the external HDD showed the newer modification date and time. I opened it in Ps and it displayed the changes that I had made to the original last night.

Thank you for your reassurance. It's very helpful to a Terminal newb like myself. Sometimes the sheer amount of information found when googling can be overwhelming.

I haven't yet googled for this, and since I have your ear, I'll ask:

Can I use rsync to update a directory from my internal HDD to two daisy chained external HDDs with one command string?

So far I've been writing the command to update to one external HDD and then when that's done, writing the command to update to the second external HDD.

To save typing and to avoid errors, I use a combination of typing at the command line and dragging from a Finder window.

I type in rsync -avP and then drag my source directory from a Finder window to the Terminal window. Add the slash to just update contents, then make sure there is a space. Then I drag the target directory from a Finder window to the Terminal window. This way all the spaces are appropriately escaped and the paths are correct.

Can I drag the target directory from a second external HDD Finder window right behind what I had dragged from the first external HDD Finder window?

Please excuse me if I'm abusing the privilege of picking your brain! Or if my question doesn't make sense.

 

[Alrescha]

Can I use rsync to update a directory from my internal HDD to two daisy chained external HDDs with one command string?

Not to my knowledge.

If you are concerned about making typos or just want to streamline the process, your next Terminal trick will be to create a small script that contains the commands you want to issue, and you run that instead of rsync, e.g.:

#!/bin/sh
# caffeinate prevents the Mac from sleeping while this is going on
/usr/bin/caffeinate rsync -avP thesourcedirectory thedestinationvolume1
/usr/bin/caffeinate rsync -avP thesourcedirectory thedestinationvolume2

If you create a file like that in your home directory, you can run it with:

sh ./mybackupfilename

Have fun,

A.

 

[chown33]

So far I've been writing the command to update to one external HDD and then when that's done, writing the command to update to the second external HDD.

To save typing and to avoid errors, I use a combination of typing at the command line and dragging from a Finder window.

You can make a shell script as previously suggested, or you can simply type the command lines once into a TextEdit plain-text file and save the file. Later, either copy and paste them in order into a Terminal window, or select, drag, and drop onto a Terminal window.

I do this all the time. It's usually because I'm running various tests, which needs a half-dozen or so different commands, but each command is separate and distinct (they don't all run in order). So I make a text file, build the commands there, then pick and paste whatever test I want to run next. It's probably been months since I actually typed any non-trivial command-line directly into a Terminal window.

 

[johngpt]

You can make a shell script as previously suggested, or you can simply type the command lines once into a TextEdit plain-text file and save the file. Later, either copy and paste them in order into a Terminal window, or select, drag, and drop onto a Terminal window.

I do this all the time. It's usually because I'm running various tests, which needs a half-dozen or so different commands, but each command is separate and distinct (they don't all run in order). So I make a text file, build the commands there, then pick and paste whatever test I want to run next. It's probably been months since I actually typed any non-trivial command-line directly into a Terminal window.

Thank you both very much. You two have given me much to consider.

chown33, had you used this same name over at the macosxhints forum?

I hadn't been to that forum for quite some time and then a couple (maybe a few) days ago went to see what I could find there about rsync. I was quite surprised to find that I was a day late! The previous day the forum had gone archive only.

 

[t0ltec]

just found this thread in all_the_cloud with exactly answer what to do with the same rsync issue ..
here is Jan 2025) using macOS Mojave: System Version: macOS 10.14.6 / Kernel Version: Darwin 18.7.0 I have the same time wasting problem with correctly time modifying when syncing dirs between internal SSD and external USB-HDD...
many option of rsync tried to test, but this one from here can help...
So,

johngpt​

thank you a lot you have marked it here