On-Board and Off-Board iCloud Photos Storage?

[purdnost]

I use iCloud to sync my photos and videos across devices as well as serve as a short-term backup. I don’t want to use iCloud as a long-term backup, however. I like to keep my iCloud library small as I’ve found that Photos runs much more efficiently than hosting my entire vast library.

So, I move all of my media from my phone to an external hard drive every month or so. The Photos library on my external drive seems to be corrupted now, maybe because it’s too big, or because the hard drive is failing.

I don’t think it’s possible to use iCloud on my phone for just new captures, while keeping everything else off my devices. If there is some workaround, I’d like to hear it.

 

[lkalliance]

You’re correct that you couldn’t use your iPhone to just take pics if you wanted it to upload to iCloud automatically without also having your full library represented on the phone. If you wanted to, though, you could turn off iCloud Photo Library on your phone, and then sync manually to your Mac. Then your Mac could upload to iCloud.

You could even place a middle-man in there: connect your iPhone to your Mac, and instead of syncing right into Photos, use Image Capture to take the photos from your phone to a folder somewhere. Then you could go through those photos at your leisure, and drag the ones you want into Photos, where they’d be uploaded to iCloud.

 

[rajs]

... Photos runs much more efficiently than hosting my entire vast library.

... The Photos library on my external drive seems to be corrupted now, maybe because it’s too big, or because the hard drive is failing.

What machine is your primary library on (machine type and how much RAM)

How large is your vast library (in gigs or terabytes). How many photos and videos in total stored within it ?

Thx

 

[purdnost]

What machine is your primary library on (machine type and how much RAM)

How large is your vast library (in gigs or terabytes). How many photos and videos in total stored within it ?

Thx

2020 MacBook Pro, 2.3 GHz Quad-Core Intel Core i7, 16 GB RAM
Over 200,000 photos, around 20,000 videos

 

[lkalliance]

2020 MacBook Pro, 2.3 GHz Quad-Core Intel Core i7, 16 GB RAM
Over 200,000 photos, around 20,000 videos

Wow, that IS a lot of photos.

You don’t need to have any devices storing your full-res images. In fact I don’t think you literally NEED to have your iCloud Photo Library on ANY device: the full-resolution images live in iCloud. I might be wrong about that, though. But if you don’t have at least one device or computer synced with iCloud then adding photos to the library becomes a manual browser-based process.

 

[Kimcha]

Is your external drive a HDD or SSD?

On HDDs Apple Photos it's painfully slow even with smaller libraries. But on SSDs it's very fast. My library is half the size of yours (at "only" 97,000 items), but the performance is excellent.

 

[Kimcha]

Another thing you could do:

• Sync the library using iCloud.
• Use osxphotos to export all the photos from the library with all metadata to a folder structure.
• Delete the photos in the library

You can tell osxphotos to store all metadata in xmp files. And if you use APFS on your external drive, they will be cloned and won't use up any additional space.

 

[purdnost]

Is your external drive a HDD or SSD?

On HDDs Apple Photos it's painfully slow even with smaller libraries. But on SSDs it's very fast. My library is half the size of yours (at "only" 97,000 items), but the performance is excellent.

HDD (portable). I’ve thought about upgrading to a desktop drive for better speed and efficiency (and longevity, maybe). And importing photos has been taking extremely long, not sure why. I have also considered an SSD, but isn’t HDD better for long-term storage (and less risk of corruption)?

 

[Kimcha]

HDD (portable). I’ve thought about upgrading to a desktop drive for better speed and efficiency (and longevity, maybe). And importing photos has been taking extremely long, not sure why. I have also considered an SSD, but isn’t HDD better for long-term storage (and less risk of corruption)?

I think that's your problem.

HDDs are WAY too slow for Apple Photos. Even much smaller libraries than yours won't run well on an HDD.

But an SSD could probably handle even a huge library like yours.

I'm using the Samsung T7 and am happy with it. But it's not the fastest SSD. For the highest speed you can buy an NVME drive and a separate USB 4 or even thunderbolt case to get the highest performance.

I also think that external HDDs are at a higher risk of corruption, because they have mechanical parts. If you move or drop your HDD while it's connected, it can easily damage it and lead to data loss.

With SSDs that's not a risk.

Either way, I wouldn't recommend to rely on an SSD or HDD not breaking. The risk for both is too high and it's extremely important to have a solid backup strategy.

Personally, I create client-side-encrypted cloud backups using ArqBackup. I also create a non-proprietary backup by exporting all photos and metadata using osxphotos and then also backing it up to the cloud using Arq.

I just documented my entire photos backup workflow here:

https://forums.macrumors.com/threads/backup-strategy-optimized-icloud-photo-library-on-mac-and-full-one-on-external-ssd.2351777/post-31738760

 

[rajs]

x2 of what @Kimcha told you.

IMO you definitely need to:

1. move your entire library onto a SSD (External - 2TB or 4TB is going to be needed based on your # of photos and videos. I've got ~ 130K photos and only ~5K videos and I am around 1.3 TB. On an SSD - external one - it is pretty fast.
2. Definitely do a FULL backup on a regular basis of the above -- whether it be to another distinct and separate storage device at home or onto the cloud (ArqBackup, Backblaze, etc..)

 

[purdnost]

x2 of what @Kimcha told you.

IMO you definitely need to:

1. move your entire library onto a SSD (External - 2TB or 4TB is going to be needed based on your # of photos and videos. I've got ~ 130K photos and only ~5K videos and I am around 1.3 TB. On an SSD - external one - it is pretty fast.
2. Definitely do a FULL backup on a regular basis of the above -- whether it be to another distinct and separate storage device at home or onto the cloud (ArqBackup, Backblaze, etc..)

About to order a portable SSD. Wondering if I should create a brand new library and import the photos by year, rather than attempting to copy the entire library over at once. Or does it even matter?

 

[Kimcha]

About to order a portable SSD. Wondering if I should create a brand new library and import the photos by year, rather than attempting to copy the entire library over at once. Or does it even matter?

I would start with the entire library and see how it performs.

I think it will perform just fine.

If it doesn't, you can split it up into multiple libraries. But I don't think its necessary to do one per year. Splitting it at most into two libraries should be enough.

 

[purdnost]

I would start with the entire library and see how it performs.

I think it will perform just fine.

If it doesn't, you can split it up into multiple libraries. But I don't think its necessary to do one per year. Splitting it at most into two libraries should be enough.

I just picked up a SSD. I assume I should be formatting it to APFS since I’ll be using it with a Mac, right?

 

[BigMcGuire]

I just picked up a SSD. I assume I should be formatting it to APFS since I’ll be using it with a Mac, right?

Yep, that's what I format my external SSDs to (APFS Encrypted).

SSDs a must for photos - let us know how it goes! How big is that library (total GB)?

 

[Kimcha]

I just picked up a SSD. I assume I should be formatting it to APFS since I’ll be using it with a Mac, right?

Yes. Make sure it’s encrypted so that if someone steals or finds your SSD, they can’t see your private photos.

 

[purdnost]

I attempted to copy the library file over, but it seemed to be stuck on “preparing to copy”. So, after several minutes, I open the library package contents and navigate to originals and there’s nothing in there. Folder size is zero KB. Open the Photos library and there’s only 21,000 items, and they all reference the files on the source folder that I was copying from this morning. Talk about perfect timing… Do I need to do a library repair? Disk recovery? 😩

 

[Kimcha]

Did you just create this library this morning?

Were you using folders before that?

If that’s the case, I would copy all the folders to the SSD and then create a new library and import all the photos.

Be patient with the copying. I imagine you have over 1tb in photos. That’s going to take forever to copy.

Even the preparing step can take a long time.

 

[purdnost]

Did you just create this library this morning?

Were you using folders before that?

If that’s the case, I would copy all the folders to the SSD and then create a new library and import all the photos.

Be patient with the copying. I imagine you have over 1tb in photos. That’s going to take forever to copy.

Even the preparing step can take a long time.

The library was created on October 24, when I started importing all of my files that were living in just a folder hierarchy (no photo management software). There should be over 220k files, now it’s saying there’s 20k. The hard drive properties window is saying there’s only 395 GB used on the disk, which seems really low to me. It’s a 4 TB drive. What on Earth happened?!

 

[Kimcha]

I see. I don't know what happened, but I would start fresh if you still have the folder hierarchy. Copy it all to the SSD.

Then import the photos into the Photos Library. It won't use additional disk space. You can have the photos both in the photo hierarchy and in the Photos library and they will only use space "once".

But you have to make sure to copy the hierarchy to the SSD first for this to work.

 

[purdnost]

Good point, except that I would delete the folders after successfully importing them into Photos. So, not sure if the repair will resolve the issue or if I need to recover the files.

 

[Kimcha]

Good point, except that I would delete the folders after successfully importing them into Photos. So, not sure if the repair will resolve the issue or if I need to recover the files.

I’m really sorry to hear. I hope the repair works.

In Photos I think you can create a smart album and then select one of the criteria to show all referenced photos.

Referenced photos are photos that are not stored inside the library, but outside of it.

 

[purdnost]

I’m really sorry to hear. I hope the repair works.

In Photos I think you can create a smart album and then select one of the criteria to show all referenced photos.

Referenced photos are photos that are not stored inside the library, but outside of it.

I imported year 2017 into my Photos library this morning, now all of the prior 12 years are gone and replaced with just the 2017 files, but they’re referencing the original source when they should have been copied into the library. Somehow, things got really messed up. Worried that a repair won’t restore the originals folder (as it currently shows 0 KB), but we’ll see. I also ran a disk utility first aid repair on the drive earlier today to see if that would speed things up… but I can’t imagine it would have caused this mess…

 

[purdnost]

Something that came to mind is that I could upload copies of everything to iCloud, then I could access them from my devices through Safari. I would have to turn off iCloud syncing, and I would have to pay extra for more cloud storage, and I’m not sure how necessary it is for my photos to be accessible online.

 

[Kimcha]

How did it go? Were you able to recover your photos?

 

[purdnost]

How did it go? Were you able to recover your photos?

Thanks for asking. My Disk Drill scan is 89% complete (a going on nearly 49 hours). 😬 Confident I’ll get my photos and videos back. Will update later.