Merging Multiple Photos Databases (External + iCloud)

[XavierScott]

I am sure a variation of this request has come up in the past but since software and hardware change so fast, I thought I'd try to be concise and break it down as best as possible.

The problem is: I have 2 Photos databases - one is on an external HD and the other is on my Mac Studio/iPhone/iPad aka iCloud (set to optimized storage). What is the best way to merge these two? And once this is merged, is it possible for the merged Photos database to live on an external drive and have it sync across devices via iCloud?

If that all is possible, what HD/SSD would be best to support this? I am generally a hobbyist, wanting to make sure family memories are not lost or vacation photos are backed up so I don't need anything that would break the bank but something that can hold up for some time.

Thanks!

 

[Brian33]

And once this is merged, is it possible for the merged Photos database to live on an external drive and have it sync across devices via iCloud?

That part is easy. Yes, you can use iCloud Photos with the Photos library on an external drive. That's what I have set up.

I have 2 Photos databases - one is on an external HD and the other is on my Mac Studio/iPhone/iPad aka iCloud (set to optimized storage). What is the best way to merge these two?

I'm not sure of the best way.

You could export all photos from library1 and then import them into library2. But if you use Photos' "File-->Export" function, I believe you'd lose any keywords, albums, folder arrangement, faces (people) assignments, titles, captions, etc. These are all features of the Photos app where the data is stored outside the actual image files. You wouldn't lose geolocation, originating date/time, camera info, etc., as that is stored inside the image file. Also, I think File-->Export exports the edited version of the photo; so the original version would be lost in the process. If you haven't done much editing, keyword assignment, captioning, etc., this might be the most straightforward way.

Alternatively, to retain the Photos feature data, it could be done using a command-line tool called osxphotos (https://github.com/RhetTbull/osxphotos). I believe the tool can export the images AND keep the Photos-specific feature information, and then import that into the other Photos library. (I've only used the export function, so I'm not sure about how the import works with keywords, albums, etc.) There may be one or two features not supported; I'm not sure. I'm also not sure about original vs. edited versions of the photos (I know they can both be exported with osxphotos, but I'm not sure about importing again).

There is very good documentation for the tool, but you'd have to put some real time into it. So it can be a somewhat complicated process, depending upon what data you want to keep or ignore.

There are probably commercial products to do this, too.

 

[XavierScott]

Thank you so much! I do remember buying an app that merges Photos apps so I will look and see.

What's the best sort of HD in order to do this? I read SSD have obviously better read time but once they are dead, they are dead. Is there anything specific I should do (format for Mac OS Extended Journaled)?

 

[gilby101]

What's the best sort of HD in order to do this? I read SSD have obviously better read time but once they are dead, they are dead. Is there anything specific I should do (format for Mac OS Extended Journaled)?

Photos will be happier with the library on an SSD. USB 3.1 SSDs (1 or 2 TB) are pretty cheap and will last for a long time (likely much longer than a HDD). I use Samsung T5/T7.

You will need another disk (HDD is fine) for backup, but maybe you are using one already - just make sure it is big enough to backup both your internal disk and the Photos external.

I format all my disks (SSD and HDD) with APFS.

 

[RhetTbull]

I'm the author of the OSXPhotos tool @Brian33 mentioned. It can perform the export and the import while retaining all metadata with the exception of persons/faces (limitation of Photos). There is also a commercial product PowerPhotos that can merge libraries and retain metadata. Personally, I would start with PowerPhotos; while my tool is free it requires some level of comfort with complex command line tools.

> What's the best sort of HD in order to do this? I read SSD have obviously better read time but once they are dead, they are dead. Is there anything specific I should do (

Definitely use SSD, connected via Thunderbolt, and format as APFS. Anything else and you risk corruption to your Photos library. Photos uses a large SQLite database to store the metadata about the library and it very much does not like being on a slow external disk. Photos also uses APFS specific features. Also note that some features like face detection and AI image categorization (for example, the ability to type "dog" in the search bar and find all photos of dogs) does not always work on external disks and does require the disk to be formatted with APFS.

If you do want to perform the merge with OSXPhotos, the following commands should get you started. There is extensive documentation available and I suggest you peruse it before using OSXPhotos.

To perform the export, first open the library you want to export from, then in Terminal, run:

osxphotos export /path/to/export --export-by-date --sidecar xmp --export-aae --ramdb

Replace "/path/to/export" with the path to your export location. E.g. "/Volumes/ExternalDrive/Photos". The photos will be exported in "YYYY/MM/DD" folders and XMP metadata sidecars will be created for each photo. "--export-aae" causes .AAE files to be exported. These hold information to reconstruct the non-destructive edits performed in Photos.

Then, open the library you want to import the photos to and run:

osxphotos import /path/to/export --walk --exportdb /path/to/export --verbose --report import.csv

Replace "/path/to/export" with the path you used for the export command in step 1. This will import all the photos and read the export database created by OSXPhotos upon export to reconstruct all metadata (location, title, caption, keywords, albums). A CSV file (readable by Excel, Google Sheets, Numbers) will be created with details about each file that was imported. If you'd like to verify what the command can do before actually importing, you can add the "--dry-run" option and run it. No photos will actually be imported.

osxphotos import /path/to/export --walk --exportdb /path/to/export --verbose --report import.csv --dry-run

 

[XavierScott]

I'm the author of the OSXPhotos tool @Brian33 mentioned. It can perform the export and the import while retaining all metadata with the exception of persons/faces (limitation of Photos). There is also a commercial product PowerPhotos that can merge libraries and retain metadata. Personally, I would start with PowerPhotos; while my tool is free it requires some level of comfort with complex command line tools.

> What's the best sort of HD in order to do this? I read SSD have obviously better read time but once they are dead, they are dead. Is there anything specific I should do (

Definitely use SSD, connected via Thunderbolt, and format as APFS. Anything else and you risk corruption to your Photos library. Photos uses a large SQLite database to store the metadata about the library and it very much does not like being on a slow external disk. Photos also uses APFS specific features. Also note that some features like face detection and AI image categorization (for example, the ability to type "dog" in the search bar and find all photos of dogs) does not always work on external disks and does require the disk to be formatted with APFS.

If you do want to perform the merge with OSXPhotos, the following commands should get you started. There is extensive documentation available and I suggest you peruse it before using OSXPhotos.

To perform the export, first open the library you want to export from, then in Terminal, run:

osxphotos export /path/to/export --export-by-date --sidecar xmp --export-aae --ramdb

Replace "/path/to/export" with the path to your export location. E.g. "/Volumes/ExternalDrive/Photos". The photos will be exported in "YYYY/MM/DD" folders and XMP metadata sidecars will be created for each photo. "--export-aae" causes .AAE files to be exported. These hold information to reconstruct the non-destructive edits performed in Photos.

Then, open the library you want to import the photos to and run:

osxphotos import /path/to/export --walk --exportdb /path/to/export --verbose --report import.csv

Replace "/path/to/export" with the path you used for the export command in step 1. This will import all the photos and read the export database created by OSXPhotos upon export to reconstruct all metadata (location, title, caption, keywords, albums). A CSV file (readable by Excel, Google Sheets, Numbers) will be created with details about each file that was imported. If you'd like to verify what the command can do before actually importing, you can add the "--dry-run" option and run it. No photos will actually be imported.

osxphotos import /path/to/export --walk --exportdb /path/to/export --verbose --report import.csv --dry-run

Amazing, thank you so much! Is there a way to combine the two, given that one is on the iCloud (optimized) would I take that Photos database that sits in my Photos folder, move it to the external SSD (formatted to APFS) and then open and download files there? Once that is done then use the app to merge the two on the SSD?

 

[RhetTbull]

Is there a way to combine the two, given that one is on the iCloud (optimized) would I take that Photos database that sits in my Photos folder, move it to the external SSD (formatted to APFS) and then open and download files there? Once that is done then use the app to merge the two on the SSD?

You can move the library to the external disk using these instructions from Apple then open it in Photos and switch to "Download originals to this Mac" instead of using the optimized library. Once it syncs with iCloud and downloads everything, you could then merge the libraries.

 

[XavierScott]

You can move the library to the external disk using these instructions from Apple then open it in Photos and switch to "Download originals to this Mac" instead of using the optimized library. Once it syncs with iCloud and downloads everything, you could then merge the libraries.

Amazing, thank you! And should it be APFS Case Sensitive? Sorry for the questions, just want to make sure I do it correctly.

 

[RhetTbull]

I would use APFS case insensitive.

 

[XavierScott]

I feel like I ran into a hiccup @RhetTbull - I copied the Photos library over but when I go to open it and change it from Optimized to Downloaded, it says it can only be done when it's the System Photo Library BUT when I go to change it over to that it says anything not downloaded will be removed.

So should I try downloading them all to my Mac Studio Ultra and then move it over?

 

[RhetTbull]

I feel like I ran into a hiccup @RhetTbull - I copied the Photos library over but when I go to open it and change it from Optimized to Downloaded, it says it can only be done when it's the System Photo Library BUT when I go to change it over to that it says anything not downloaded will be removed.

So should I try downloading them all to my Mac Studio Ultra and then move it over?

I don't understand the bit about anything not downloaded will be removed. Would be helpful to see a screenshot of the dialog box. Another solution is to create a new library on the external drive (quite Photos, then restart it while holding down Option). Set the new library to be the system library and let it download everything from iCloud then do the merge.

 

[XavierScott]

I don't understand the bit about anything not downloaded will be removed. Would be helpful to see a screenshot of the dialog box. Another solution is to create a new library on the external drive (quite Photos, then restart it while holding down Option). Set the new library to be the system library and let it download everything from iCloud then do the merge.

Thank you so much!

I think I got it all set up... my only concern now is that I am trying to export a video and it just hangs when I'm in Photos... but then if I just drag it off it is immediate. I'm wondering if something is wrong (don't want to delete anything before I see it's working).

 

[RhetTbull]

> my only concern now is that I am trying to export a video and it just hangs when I'm in Photos... but then if I just drag it off it is immediate. I

How are you exporting the video?

 

[XavierScott]

> my only concern now is that I am trying to export a video and it just hangs when I'm in Photos... but then if I just drag it off it is immediate. I

How are you exporting the video?

The usual way - File > Export > Modified to 720P.

 

[RhetTbull]

In that case something doesn't seem right. Dragging it out isn't re-coding the video to a different resolution and the File > Export method might be but if the file is downloaded, this should happen relatively quickly. Do you have "Download originals to this Mac" in Photos > Settings > iCloud? Does Photos say that it is synced (at the bottom of the library view)?

 

[XavierScott]

It does, I have iCloud on along with "Download Originals". It was last synced this morning at 1AM. I tried to see if it was just an export issue but I tried to hide a photo and it won't hide but when I go to Edit in the tool bar, it has "Undo Hide" even though the photo is clearly not hidden. Along with that I checked to see if the permissions were right on the drive and they are.

 

[XavierScott]

I am also noticing that when I am trying to import I get "there is an error attempting to free up disk space..." even though I have 1.6 TB available.

 

[RhetTbull]

> my only concern now is that I am trying to export a video and it just hangs when I'm in Photos... but then if I just drag it off it is immediate. I

How are you exporting the video?

It does, I have iCloud on along with "Download Originals". It was last synced this morning at 1AM. I tried to see if it was just an export issue but I tried to hide a photo and it won't hide but when I go to Edit in the tool bar, it has "Undo Hide" even though the photo is clearly not hidden. Along with that I checked to see if the permissions were right on the drive and they are.

I'm not sure. Sounds like Photos isn't happy but I'm not sure why. Is the drive formatted with APFS? Is it directly connected or via network? Have you tried repairing the library?

 

[XavierScott]

I'm not sure. Sounds like Photos isn't happy but I'm not sure why. Is the drive formatted with APFS? Is it directly connected or via network? Have you tried repairing the library?

It is formatted with APFS and it's directly connected via the back of the Mac Studio Ultra.

Update: I repaired the Photos library and it is still happening and I triple checked, it's APFS Case-Sensitive.

 

[RhetTbull]

> APFS Case-Sensitive.

MacOS defaults to case-insensitive. I don't know if that's the problem but could be an issue. At this point I'd contact Apple.

 

[XavierScott]

> APFS Case-Sensitive.

MacOS defaults to case-insensitive. I don't know if that's the problem but could be an issue. At this point I'd contact Apple.

Sounds good, I appreciate the help. Which is the better APFS? I could copy the library and change it, I just made it that one as that's the one you mentioned.

 

[gilby101]

Which is the better APFS?

Unless you have a very strong reason, you should always format APFS case-insensitive.

 

[XavierScott]

Unless you have a very strong reason, you should always format APFS case-insensitive.

Yeah that's what it is so I'm not sure why it's being so weird.

 

[gilby101]

Yeah that's what it is so I'm not sure why it's being so weird.

Earlier you said yours was case-sensitive. Confused?

 

[XavierScott]

Earlier you said yours was case-sensitive. Confused?

Sorry, I read it too quickly - I will move the Photos drive to my other external and reformat for case-INsensitive.