iPad with Apple SD card reader loses 1% of photos

[LibbyLA]

I have been using iPads to import and view photos from our game (trail) cameras for nearly two years. When we pull the cards from our cameras, we record the number of photos that the camera says are on the card and that number matches what Windows says when I view the card in Windows.

I'd noticed that the number of photos shown in "Last Import" was always fewer than the number I'd recorded as being on the card, but I wrote it off to an occasional glitchy photo file until earlier this week. I realized that the number of photos the iPad imported was about 1% less than the number of photos on the card.

Today, I decided to do a little testing to see if I could identify if there was a pattern to the missing photos. I created a list of file names from the SD card in Windows and a list of file names from the photos on the iPad then looked at only the numbers that were missing from the iPad. (Game cameras tend to write file names that are 8 characters, with the last four being sequential numbers.) (I'm a statistical programmer and I've worked with photo file data before so this was easy peasy for me to whip up.)

On a card with 1044 photos, the iPad imported 1034 of them. The missing photos were numbered:

0101
0202
0303
0404
0505
0606
0707
0808
0909
1010

On a card with 820 photos, the iPad imported 812 of them. The missing photos were 0101 through 0808 of the list above.

I have not tried this with photos from regular digital cameras so I don't know if the problem exists with those, but it's something to keep an eye on. I've reported it to Apple, although I had to tag it onto a case I already have going because online tech support didn't seem to be available.

 

[sparksd]

Interesting pattern on the file names. I have not seen this with RAW images from a Canon DSLR. Images are named with 8 characters, e.g. IMG_1234.CR2. (Using latest card reader on iPad Pro with iOS 11.x).

 

[LibbyLA]

That's the same form they are with game cameras. The two I tried are PICTxxxx.JPG and IMG_xxxx.JPG.

I have some cards that have over 4000 images so I'm going to try those at some point.

 

[LibbyLA]

Update. I just tested an SD card with 4467 photos on it in five folders. The first four folders had 999 photos, the last had 471. Based on my observations of the other two SD cards, I was pretty sure that the iPad would import 4427 photos, 990 from each of the first four folders and 467 from the last one and that's exactly what it did.

These pictures had as the first four character the two-digit month and two-digit date rather than PICT or IMG_ or something similar, but the last four digits of the missing photos were 0101, 0202, 0303, 0404, 0505, 0606, 0707, 0808, 0909.

I've tested SD cards with pictures from three different brands of cameras and the import behavior has been identical on all three.

Apple tech support is supposed to call me Friday.

 

[sparksd]

Really odd behavior. I've never imported very large numbers of photos so that may likely be why I haven't seen this. Keep us posted on their findings.

 

[akash.nu]

Just wondering if this could be down to the file system being used. I’ve never done this personally but keep us posted.

 

[Donka]

Have you tried from a different camera? Are the file sizes roughly the same with the images? I wonder if they are being skipped because they are being classed as duplicates - like how Mac will rename a duplicate file with the suffix _1, _2 etc. Perhaps the repeating sequence in the last 4 digits is being considered a duplicate or something along those lines? how are you importing - I'm assuming just tapping import all? What happens if you manually select these images if you can identify them?

 

[LibbyLA]

Have you tried from a different camera? Are the file sizes roughly the same with the images? I wonder if they are being skipped because they are being classed as duplicates - like how Mac will rename a duplicate file with the suffix _1, _2 etc. Perhaps the repeating sequence in the last 4 digits is being considered a duplicate or something along those lines? how are you importing - I'm assuming just tapping import all? What happens if you manually select these images if you can identify them?

I've tested using SD cards from three different brands of cameras.

It has nothing to do with file sizes. These cameras are set to take bursts of three pics per trigger, all files sizes are pretty similar, none larger than 5 MP. It has to be something to do with the import process because it's the same photo numbers for every camera.

There's no way to easily identify which photo is 0101, 0202, etc., to view that particular thumbnail.

Yes, I'm importing using Import All in Photos.

 

[Donka]

I mentioned file sizes because a burst of images of the same seen will likely yield a similar file size and this along with the file name may cause the import process to consider the file as a duplicate. It is hard to debug though given the Photos app doesn't provide the filenames on the import selection screen. All speculation though in my part.

 

[sparksd]

I mentioned file sizes because a burst of images of the same seen will likely yield a similar file size and this along with the file name may cause the import process to consider the file as a duplicate. It is hard to debug though given the Photos app doesn't provide the filenames on the import selection screen. All speculation though in my part.

It would be interesting to know to what degree the number of photos being imported plays. I tried a simple experiment with 10 photos named -

IMG_0100.jpg
IMG_0101.jpg
IMG_0200.jpg
IMG_0202.jpg
etc.

and didn't see a problem (latest card reader, iPP 10.5, iOS 11.4, images ~8MB each).

 

[Donka]

That would rule out the duplicate theory. Will be interesting to see if Apple come back with anything.

 

[LibbyLA]

I mentioned file sizes because a burst of images of the same seen will likely yield a similar file size and this along with the file name may cause the import process to consider the file as a duplicate. It is hard to debug though given the Photos app doesn't provide the filenames on the import selection screen. All speculation though in my part.

These cameras are set to take a burst of three photos every time they trigger. Two of the cameras take the three photos boom, boom, boom, one right after the other. The third is set for something called slow burst so there's one to two seconds between shots one and two and four to six seconds between shots two and three.

0101 is the middle shot of a three-shot burst (99 is 33 triggers, so 100, 101, 102 is the next burst). 0202 is the first shot of a burst. 0303 is the first shot.

This may explain why I've sometimes felt like I was missing an important shot from one of my license plate cameras. The very one in the sequence (5-shot burst, but it keeps firing if there's still something in front of the camera) that would have shown me the most readable plate wasn't there.

I'm disappointed because now I have to look at every card again on my desktop computer and manually search for those files that I know are missing to be sure that it's not a picture I need.
[doublepost=1528399291][/doublepost]

It would be interesting to know to what degree the number of photos being imported plays. I tried a simple experiment with 10 photos named -

IMG_0100.jpg
IMG_0101.jpg
IMG_0200.jpg
IMG_0202.jpg
etc.

and didn't see a problem (latest card reader, iPP 10.5, iOS 11.4, images ~8MB each).

It is missing every 101st photo. 101, then 202 is 101 + 101. 303 is 202+101, and so on.

I don't have an easy way to create files that are named with something like aaaa, aaab, aaac, etc. to see if it's the count (every 101st) rather than the actual number, but I can probably find a folder with a bunch of files with random names and try that. Thanks for giving me the idea!
[doublepost=1528401628][/doublepost]Update on this. I copied the 4427 pics that the iPad imported (from the original 4467) to an SD card and imported them again on the iPad. 4384 imported, so 43 were lost, which is what I'd expect (with 1% loss, should be 43 or 44 pics lost of 4427).

It's not the picture name/numbers, it's the number of photos. Thanks for giving me the idea to try this.

 

[sparksd]

These cameras are set to take a burst of three photos every time they trigger. Two of the cameras take the three photos boom, boom, boom, one right after the other. The third is set for something called slow burst so there's one to two seconds between shots one and two and four to six seconds between shots two and three.

0101 is the middle shot of a three-shot burst (99 is 33 triggers, so 100, 101, 102 is the next burst). 0202 is the first shot of a burst. 0303 is the first shot.

This may explain why I've sometimes felt like I was missing an important shot from one of my license plate cameras. The very one in the sequence (5-shot burst, but it keeps firing if there's still something in front of the camera) that would have shown me the most readable plate wasn't there.

I'm disappointed because now I have to look at every card again on my desktop computer and manually search for those files that I know are missing to be sure that it's not a picture I need.
[doublepost=1528399291][/doublepost]

It is missing every 101st photo. 101, then 202 is 101 + 101. 303 is 202+101, and so on.

I don't have an easy way to create files that are named with something like aaaa, aaab, aaac, etc. to see if it's the count (every 101st) rather than the actual number, but I can probably find a folder with a bunch of files with random names and try that. Thanks for giving me the idea!
[doublepost=1528401628][/doublepost]Update on this. I copied the 4427 pics that the iPad imported (from the original 4467) to an SD card and imported them again on the iPad. 4384 imported, so 43 were lost, which is what I'd expect (with 1% loss, should be 43 or 44 pics lost of 4427).

It's not the picture name/numbers, it's the number of photos. Thanks for giving me the idea to try this.

You could narrow down the count issue by reducing the number of imports by half. If there are still missing imports, reduce again by half, etc. If none are missing, split the difference between the current and last import.

 

[LibbyLA]

You could narrow down the count issue by reducing the number of imports by half. If there are still missing imports, reduce again by half, etc. If none are missing, split the difference between the current and last import.

I'm pretty sure that they are missing on every import except the rare one with very few pics. We check our cameras no more often than every two or four weeks; some go for months. I had noticed that every import seemed to be fewer than the number of photos recorded for each card when we took it from the camera.

It's every 101st picture.

 

[LibbyLA]

Just got off the phone with the support guy and we have established that the missing photos are not being displayed in the thumbnails before the import is actually done. On my desktop computer, I opened the first missing file (IMG_0101.JPG) and saved it as IMG_0000.JPG. Apparently the import process displays the files in date order rather than name order so the new photo showed up as the last thumbnail, which I was able to import. (The photos are sequences of 5 so they are labelled 1/5, 2/5, 3/5, etc., which makes it easy to determine which photo is which.)

The problem has to do with the system reading the card in the first place rather than displaying all thumbnails and just not importing everything. It's not even showing the missing photos in the first place.

 

[LibbyLA]

I spent over an hour and 15 minutes on the phone with Apple support yesterday afternoon. It’s confirmed that the photo import process misses every 101st photo. I don’t know if it affects file types other than JPG because my cameras only save JPG files. Apparently it is a bug in iOS, confirmed in both iOS 10.3 and 11.4.

Once we knew what the pattern was, it was easy to confirm which photos were missing on the iPad because all the photos have date and time stamp info on a strip at the top or bottom, depending on the camera brand.

The missing photos are not showman in the thumbnails. They go missing when iOS reads the files off the card and displays the thumbnails.

The engineer seemed really amazed, and not in a good way. It was nothing but luck that I caught it and was able to do a little programming to confirm something was wrong.

I am very disappointed. I wouldn’t have bought my first iPad (Pro 9.7), much less two more, if I had known the photo import process was defective. My main purpose for buying an iPad in the first place was to view pictures from my game cameras. I use them for other things, to be sure, but they fail in a very sneaky way at the main task for which they were purchased.

 

[sparksd]

I spent over an hour and 15 minutes on the phone with Apple support yesterday afternoon. It’s confirmed that the photo import process misses every 101st photo. I don’t know if it affects file types other than JPG because my cameras only save JPG files. Apparently it is a bug in iOS, confirmed in both iOS 10.3 and 11.4.

Once we knew what the pattern was, it was easy to confirm which photos were missing on the iPad because all the photos have date and time stamp info on a strip at the top or bottom, depending on the camera brand.

The missing photos are not showman in the thumbnails. They go missing when iOS reads the files off the card and displays the thumbnails.

The engineer seemed really amazed, and not in a good way. It was nothing but luck that I caught it and was able to do a little programming to confirm something was wrong.

I am very disappointed. I wouldn’t have bought my first iPad (Pro 9.7), much less two more, if I had known the photo import process was defective. My main purpose for buying an iPad in the first place was to view pictures from my game cameras. I use them for other things, to be sure, but they fail in a very sneaky way at the main task for which they were purchased.

Wow, I wonder how that got through their testing process. I would have thought that a simple test case of copying a few hundred photos and verifying the copy was good would be in their test suite. It's also surprising that this has not been reported as a problem by other users. I do a lot of photography but never use the iPad Photos app - I prefer to process and manage on a laptop or desktop and transfer a select few to the iPad using a 3rd party app, Photo Manager Pro

An option - albeit a slow one - would be to use a wireless device like a RAVPower FileHub to copy the files from the card. It works, but wireless transfers are very slow, particularly when compared to the USB 3 speeds of the card reader. The right answer is for Apple to fix this ASAP. Bet a lot of people have had this happen to them without knowing it. I'd be really unhappy if I lost photos because I formatted the card after thinking I had made a good download.

 

[LibbyLA]

Wow, I wonder how that got through their testing process. I would have thought that a simple test case of copying a few hundred photos and verifying the copy was good would be in their test suite. It's also surprising that this has not been reported as a problem by other users. I do a lot of photography but never use the iPad Photos app - I prefer to process and manage on a laptop or desktop and transfer a select few to the iPad using a 3rd party app, Photo Manager Pro

An option - albeit a slow one - would be to use a wireless device like a RAVPower FileHub to copy the files from the card. It works, but wireless transfers are very slow, particularly when compared to the USB 3 speeds of the card reader. The right answer is for Apple to fix this ASAP. Bet a lot of people have had this happen to them without knowing it. I'd be really unhappy if I lost photos because I formatted the card after thinking I had made a good download.

I typically look at 10-20K pictures every four weeks, somewhat fewer for card pulls in between. A card with over 4K pics on it takes 13 minutes to import on my iPP 10.5, 72 minutes on the iP Mini 4. Not sure it is worth the slowness and additional cost of an external device.

I'm not sure what I'm going to do now. Either I continue using the iPad but have to look at each SD card (sometimes as many as 30 or more) on my desktop computer to check the photo numbers that are missing (at least it's not random) to see if they are worth keeping (which means handling each card twice) or I switch to my Surface or an Android tablet for the interim. The photo viewing and keeping options aren't nearly as nice on those, although they don't loose pictures.

I'm actually not surprised that it has not been reported by anyone else. In the first place, who pays attention to exactly how many pictures are on an SD card when they take it out of the camera? The only reason I'm aware of it is that we use a run sheet when we run our cameras and we record the number of pics that the camera says is on the card, along with whether we change batteries, what kind of batteries we use, any oddball behavior or observations. Also, our cameras typically take 3-5 shots each time they trigger and they have the date/time stamp and some have 1/5, 2/5, etc., so it's a lot easier to identify that something is missing if you are paying attention to that. Photos from regular cameras wouldn't have that. I had noticed that sometimes it seemed that a photo was missing from a sequence that I wanted to keep but just wrote it off to a camera problem.

I don't usually have the run sheets in front of me when I import the photos, though, because I'm usually riding in a bouncy pickup truck. I remember some photo counts, but just wrote off the discrepancy to my bad memory or perhaps munged files on the card. This last time, though, I paid attention to the counts and noticed that the iPad was always low and then realized that it was predictably low (1%).

A relatively small percent of the population would have the programming knowledge to create a text file of the file names, dates/times, and file sizes from the SD card and from the iPad (files copied to the desktop computer) and then write a program to compare the two lists and kick out the names of the files that were missing from the iPad. It's a very geeky thing, but I already had experience working with getting the file names because I'd written a program to help me identify photos that I needed to look at when I got a camera that took photo after photo after photo because of "wind in the weeds."

At first I thought it was the file suffix (0101, 0202), but you tested those file names (thank you!) and it wasn't the names, so that's what made me do a little more thinking and realize that it was every 101st file. I confirmed that it wasn't the file names by copying the imported files to my desktop (40 pics short), then copying those files to an SD card and importing them again, and it being 40 pics more short.

 

[sparksd]

I typically look at 10-20K pictures every four weeks, somewhat fewer for card pulls in between. A card with over 4K pics on it takes 13 minutes to import on my iPP 10.5, 72 minutes on the iP Mini 4. Not sure it is worth the slowness and additional cost of an external device.

I'm not sure what I'm going to do now. Either I continue using the iPad but have to look at each SD card (sometimes as many as 30 or more) on my desktop computer to check the photo numbers that are missing (at least it's not random) to see if they are worth keeping (which means handling each card twice) or I switch to my Surface or an Android tablet for the interim. The photo viewing and keeping options aren't nearly as nice on those, although they don't loose pictures.

I'm actually not surprised that it has not been reported by anyone else. In the first place, who pays attention to exactly how many pictures are on an SD card when they take it out of the camera? The only reason I'm aware of it is that we use a run sheet when we run our cameras and we record the number of pics that the camera says is on the card, along with whether we change batteries, what kind of batteries we use, any oddball behavior or observations. Also, our cameras typically take 3-5 shots each time they trigger and they have the date/time stamp and some have 1/5, 2/5, etc., so it's a lot easier to identify that something is missing if you are paying attention to that. Photos from regular cameras wouldn't have that. I had noticed that sometimes it seemed that a photo was missing from a sequence that I wanted to keep but just wrote it off to a camera problem.

I don't usually have the run sheets in front of me when I import the photos, though, because I'm usually riding in a bouncy pickup truck. I remember some photo counts, but just wrote off the discrepancy to my bad memory or perhaps munged files on the card. This last time, though, I paid attention to the counts and noticed that the iPad was always low and then realized that it was predictably low (1%).

A relatively small percent of the population would have the programming knowledge to create a text file of the file names, dates/times, and file sizes from the SD card and from the iPad (files copied to the desktop computer) and then write a program to compare the two lists and kick out the names of the files that were missing from the iPad. It's a very geeky thing, but I already had experience working with getting the file names because I'd written a program to help me identify photos that I needed to look at when I got a camera that took photo after photo after photo because of "wind in the weeds."

At first I thought it was the file suffix (0101, 0202), but you tested those file names (thank you!) and it wasn't the names, so that's what made me do a little more thinking and realize that it was every 101st file. I confirmed that it wasn't the file names by copying the imported files to my desktop (40 pics short), then copying those files to an SD card and importing them again, and it being 40 pics more short.

With your volume of photos, any wireless transfer (or any transfer slower than USB3) would be painful. You could use an Android tablet - I have a Galaxy Tab S2 with a 256GB uSD card in it) - but you won't get USB3 over an OTG cable+card reader combo or a card reader with a micro USB interface. But it would work as I've gone that path in the past.

One other option would be a small laptop. I have a Dell XPS 13 that use on travel for my photo work.

 

[LibbyLA]

With your volume of photos, any wireless transfer (or any transfer slower than USB3) would be painful. You could use an Android tablet - I have a Galaxy Tab S2 with a 256GB uSD card in it) - but you won't get USB3 over an OTG cable+card reader combo or a card reader with a micro USB interface. But it would work as I've gone that path in the past.

One other option would be a small laptop. I have a Dell XPS 13 that use on travel for my photo work.

A 13" laptop is too big. I have one. Then bought a Surface Pro 3. Too big. Then Surface 3. Still too big. 8" Dell Windows tablet is too small. iPads with Smart Keyboard have the perfect small footprint with a usable keyboard and good photo viewing and selecting.

I have an OTG card reader with micro USB that I can use with various tablets. I'll have to experiment to figure out what I'm going to do because I'll probably be pulling cards Thursday or Friday.

I have a couple of Galaxy Tabs and a couple of cheapie 10" Android tablets, too, so I'm all set with devices. Just have to figure out what my battle plan is.

 

[Donka]

Well I would hope this fix is a high priority for Apple although I suspect it will be fixed in a iOS 12 beta given iOS 12 introduces a few improvements around the sd import process.

 

[LibbyLA]

Well I would hope this fix is a high priority for Apple although I suspect it will be fixed in a iOS 12 beta given iOS 12 introduces a few improvements around the sd import process.

I guess I could install the iOS 12 beta on one of my iPads and see if anything has changed.

 

[Donka]

I guess I could install the iOS 12 beta on one of my iPads and see if anything has changed.

Could be worth a try but I imagine the bug will still be there in beta 1. Hopefully will be fixed soon though.

 

[LibbyLA]

I bought this SD card reader:

https://smile.amazon.com/gp/product/B078M8DT85/ref=oh_aui_detailpage_o07_s00?ie=UTF8&psc=1

It uses its own app, the thumbnails are larger than with the Apple reader, and you can see the file names. It does not skip 0101, 0202, etc.

Viewing the photos isn't as convenient as with the Photos app, but you view the larger thumbnails then view anything you want to see better. When you save a photo, it saves to the Photos app, but just in the camera roll. I think I can work around that by selecting the new photos and putting them into an album.

I don't like that it only works in portrait mode.

 

[sparksd]

I bought this SD card reader:

https://smile.amazon.com/gp/product/B078M8DT85/ref=oh_aui_detailpage_o07_s00?ie=UTF8&psc=1

It uses its own app, the thumbnails are larger than with the Apple reader, and you can see the file names. It does not skip 0101, 0202, etc.

Viewing the photos isn't as convenient as with the Photos app, but you view the larger thumbnails then view anything you want to see better. When you save a photo, it saves to the Photos app, but just in the camera roll. I think I can work around that by selecting the new photos and putting them into an album.

I don't like that it only works in portrait mode.

Glad you found something. I have 3 or 4 different uSD card readers, some better than others, and I have tried and returned at least 3 others. I wanted something that could both efficiently display RAW images stored on the card (no copying) and stream large (>4GB) video files. I wanted an SD reader that could that but couldn't find one that did both and ended up with uSD readers. I switched to a uSD in an SD adapter in my DSLR so I could readily review the shots on my iPad Pro plus they can read uSD cards from my GoPro. Nothing is easy with external devices in the iOS world.