Resolved Unsearchable Folder

[SpeQ]

You can try moving the contents to another non-package folder.

Well isn't that what I have done by dragging a copy of the folder to my desktop?

 

[SpeQ]

Just to eliminate possibilities, I dragged the same files from the Originals folder on the external drive to a new folder I created on my desktop. The search does not find results.

 

[chown33]

Well isn't that what I have done by dragging a copy of the folder to my desktop?

It is. I was reading too quickly and misread what you'd done. I thought you'd copied the original package folder to your desktop.

I think my error rate is too high in my current under-caffeinated state. I'll take a longer look at everything presented so far, and write a more measured reply later.

 

[SpeQ]

It is. I was reading too quickly and misread what you'd done. I thought you'd copied the original package folder to your desktop.

I think my error rate is too high in my current under-caffeinated state. I'll take a longer look at everything presented so far, and write a more measured reply later.

lol. My caffeine has worn off too. Later.

 

[chown33]

Please paste the output of this command line:

xattr -h ; ditto -h

The output should be the concatenated usage summary for both cmds.

I want to know what features are available in the cmds on your system. I can already infer you're running Catalina or later, because that's when the com.apple.macl xattr was introduced.

 

[SpeQ]

11.2.2

usage: xattr [-l] [-r] [-s] [-v] [-x] file [file ...]
       xattr -p [-l] [-r] [-s] [-v] [-x] attr_name file [file ...]
       xattr -w [-r] [-s] [-x] attr_name attr_value file [file ...]
       xattr -d [-r] [-s] attr_name file [file ...]
       xattr -c [-r] [-s] file [file ...]

The first form lists the names of all xattrs on the given file(s).
The second form (-p) prints the value of the xattr attr_name.
The third form (-w) sets the value of the xattr attr_name to the string attr_value.
The fourth form (-d) deletes the xattr attr_name.
The fifth form (-c) deletes (clears) all xattrs.

options:
  -h: print this help
  -l: print long format (attr_name: attr_value and hex output has offsets and
      ascii representation)
  -r: act recursively
  -s: act on the symbolic link itself rather than what the link points to
  -v: also print filename (automatic with -r and with multiple files)
  -x: attr_value is represented as a hex string for input and output
Usage: ditto [ <options> ] src [ ... src ] dst

    <options> are any of:
    -h                         print full usage
    -v                         print a line of status for each source copied
    -V                         print a line of status for every file copied
    -X                         do not descend into directories with a different device ID

    -c                         create an archive at dst (by default CPIO format)
    -x                         src(s) are archives
    -z                         gzip compress CPIO archive
    -j                         bzip2 compress CPIO archive
    -k                         archives are PKZip
    --keepParent               parent directory name src is embedded in dst_archive
    --arch archVal             fat files will be thinned to archVal
                               multiple -arch options can be specified
                               archVal should be one of "ppc", "i386", etc
    --bom bomFile              only objects present in bomFile are copied
    --norsrc                   don't preserve resource data
    --noextattr                don't preserve extended attributes
    --noqtn                    don't preserve quarantine information
    --noacl                    don't preserve ACLs
    --sequesterRsrc            copy resources via polite directory (PKZip only)
    --nocache                  don't use filesystem cache for reads/writes
    --hfsCompression           compress files at destination if appropriate
    --nopreserveHFSCompression don't preserve HFS+ compression when copying files
    --zlibCompressionLevel num use compression level 'num' when creating a PKZip archive
    --password                 request password for reading from encrypted PKZip archive

 

[chown33]

Thanks for the info on xattr and ditto.

I probably won't get farther into this before Sat, but you can do some setup now that will help me out in the long run. What I'd like you to do is setup a folder with a few image files in it. I'm going to give it a specific name, in a specific place on your external disk. I'm doing this because it's much faster for me to just give you a command line to paste in, rather than doing a description with a partial command line and then instructions on what to drag and drop where.

Since you haven't given the name of your external disk, where the original photo library package folder resides, I'm going to give it the nickname "disko". So when you see a pathname like /Volumes/disko/something in a command, it refers to an item named "something" on the external disk. Also note that the leading '/' is significant. I'll generally give you things that should be directaly pasteable into a Terminal window. If I don't, I'll tell you first.

You can simplify things on your end by actually renaming your external drive to "disko". This can be temporary.


Setup 1
Make a folder on "disko" named "master". Please use that exact lower-case name.

The "master" folder should not be located within any other folder on "disko". It should be at the top-most level of the drive.

Test 1

ls -ldeO@ /Volumes/disko/master

This should show that you own the folder. It may show some other things, or it may not. Please paste the output of the command here.


Setup 2
Go into your photo library package folder, and select about a half-dozen image files from your "Originals" folder. These should be files that were not being found in your previous trials.

Copy and paste the selected image files into the "master" folder. I intend for these files to be copies made by Finder.

Test 2

ls -laeO@ /Volumes/disko/master

This should list each image file pasted in, followed by any xattrs, and also listing any ACLs (access control lists) that may be present. It will also list the dirs "." and "..".

The xattr's will list only the name followed by a size in bytes. An ACL will show a number, a group or user, then an allow or deny statement, with one ACL item per line. I don't know whether any ACLs will be shown or not.

Please copy and paste the output of the above command here.

If you want to see actual real-life ACLs, do this:

ls -ldeO@ ~

You should see an ACL on folders like Documents, Pictures, and several other dirs in your home dir. This is typical, but there may be others:

 0: group:everyone deny delete

Test 3
Use Finder to search "master" for images, with the same criteria you used originally (name starts with "P", etc.). Tell us whether it works or fails.

 

[SpeQ]

You can simplify things on your end by actually renaming your external drive to "disko".

External drive is partitioned, two volumes. So I just rename the relevant volume to disko, right?

 

[chown33]

External drive is partitioned, two volumes. So I just rename the relevant volume to disko, right?

Yes.

 

[SpeQ]

Got it. I'm not sure if I'll get into this tonight.

 

[SpeQ]

Note: "username" edited.

Test 1

drwxr-xr-x  2 username  staff  - 64 Mar 20 07:52 /Volumes/disko/master




Test 2

total 29096
drwxr-xr-x   8 username  staff  -     256 Mar 20 08:01 .
drwxr-xr-x@ 35 username  staff  -    1120 Mar 20 07:52 ..
    com.apple.FinderInfo         32 
-rw-r--r--@  1 username  staff  - 2318073 Apr  3  2009 P1050620.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      52889 
    com.apple.metadata:kMDItemSupportFileType         70 
-rw-r--r--@  1 username  staff  - 2570960 Apr  3  2009 P1050621.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      53586 
    com.apple.metadata:kMDItemSupportFileType         70 
-rw-r--r--@  1 username  staff  - 2476490 Apr  3  2009 P1050622.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      54021 
    com.apple.metadata:kMDItemSupportFileType         70 
-rw-r--r--@  1 username  staff  - 2427318 Apr  3  2009 P1050623.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      54039 
    com.apple.metadata:kMDItemSupportFileType         70 
-rw-r--r--@  1 username  staff  - 2403396 Apr  3  2009 P1050624.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      53837 
    com.apple.metadata:kMDItemSupportFileType         70 
-rw-r--r--@  1 username  staff  - 2355364 Apr  3  2009 P1050625.JPG
    com.apple.FinderInfo         32 
    com.apple.ResourceFork      52112 
    com.apple.metadata:kMDItemSupportFileType         70

Test 3
Failed.

 

[SpeQ]

Let me see if I can guess the next step:

attr -rc /Volumes/disko/master

Right?

 

[chown33]

Let me see if I can guess the next step:

attr -rc /Volumes/disko/master

Right?

I was going to do some other things first.

I also don't plan to modify "master", since it's the Known Working (or Known Malfunctioning) component in this test scenario.

Test 4

mdutil -s -a

This lists the Spotlight indexing status of all mounted volumes.

Example output from my Mac:

/Volumes/disko:
    Indexing enabled.
/Volumes/cross-Pi:
    Indexing disabled.
/:
    Indexing enabled.
/Volumes/ignored:
    Indexing and searching disabled.

The volume "cross-PI" is a read-only disk-image holding Raspberry-Pi development tools. It's read-only, so it's inherently unindexed (not updated), but when it was read-write it WAS indexed, so it's searchable (I confirmed that it works).

The "ignored" volume is a read-write disk-image I put a few test files on, then added to Spotlight's Privacy list.


Test 5

mkdir -p /Volumes/disko/test5
ditto --norsrc --noextattr --noqtn --noacl  \
  /Volumes/disko/master/*.JPG  /Volumes/disko/test5
ls -laeO@ /Volumes/disko/test5

It's important that you paste this one exactly as given. The command ending with \ is continued on the following line. This is a shell trick I often use when assembling longer cmds.

This copies the files to a "test5" folder adjacent to "master", stripping off resource forks, all xattrs, and any quarantine flags or ACLs. If the files had custom icons or tags before, the copies will not.

Please paste the output here. It should show the same JPG files as Test 2, but with no xattrs at all.


Test 6

cp -afR /Volumes/disko/master /Volumes/disko/test6
ls -laeO@ /Volumes/disko/test6

xattr -c -r /Volumes/disko/test6
ls -laeO@ /Volumes/disko/test6

This makes an exact copy of the "master" dir into an adjacent dir named "test6". It then lists the contents, including xattr's and ACLs.

Next, it uses 'xattr' to clear all xattr's from everything in "test6", and lists the result.

Please paste the output here (both the before and after listings). The after one should show the same files as Test 2, but with no xattrs at all. Any ACLs would be preserved, though.


Test 7
Repeat Test 3 on the dirs "test5" and "test6".

 

[SpeQ]

Test 4
Note:ExternalHD name edited.

/:
    Indexing enabled.
/System/Volumes/Data:
    Indexing enabled.
/System/Volumes/Data:
    Indexing enabled.
/Volumes/ExternalHD:
    Indexing enabled.
/Volumes/disko:
    Indexing enabled.

Test 5
Note: "username" edited.

total 28440
drwxr-xr-x   8 username  staff  -     256 Mar 20 18:10 .
drwxr-xr-x@ 36 username  staff  -    1152 Mar 20 18:10 ..
    com.apple.FinderInfo         32
-rw-r--r--   1 username  staff  - 2318073 Apr  3  2009 P1050620.JPG
-rw-r--r--   1 username  staff  - 2570960 Apr  3  2009 P1050621.JPG
-rw-r--r--   1 username  staff  - 2476490 Apr  3  2009 P1050622.JPG
-rw-r--r--   1 username  staff  - 2427318 Apr  3  2009 P1050623.JPG
-rw-r--r--   1 username  staff  - 2403396 Apr  3  2009 P1050624.JPG
-rw-r--r--   1 username  staff  - 2355364 Apr  3  2009 P1050625.JPG

 

[SpeQ]

I don’t want to make an incorrect assumption. For test 6, am I supposed to paste first two lines in Terminal, hit enter and copy/paste results, then proceed to next two lines and do the same, or c/p all five lines at the same time?

In Test 5, what does “total 28440” represent?

 

[chown33]

I don’t want to make an incorrect assumption. For test 6, am I supposed to paste first two lines in Terminal, hit enter and copy/paste results, then proceed to next two lines and do the same, or c/p all five lines at the same time?

Either way. It won't matter.

In Test 5, what does “total 28440” represent?

It's the total size of files in the dir, measured in 512-byte blocks.

An easy way to illustrate is to make a dir, make a 1-byte file in it, then use ls -l' on the dir. The "total" should be 8, because a 1-byte file gets a 4KB allocation (the default size of allocation blocks). The number of 512-byte blocks in a 4KB allocation is 8.

There's a lot of stuff like this that's a holdover from early days of Unix.

mkdir example
ls -la example

touch example/empty
ls -la example

echo >example/single
ls -la example

After the empty file is made, the total is still 0, because an empty file has no allocation blocks.

 

[SpeQ]

Test 6

total 29096
drwxr-xr-x   8 username  staff  -     256 Mar 20 08:01 .
drwxr-xr-x@ 37 username  staff  -    1184 Mar 20 23:28 ..
    com.apple.FinderInfo         32
-rw-r--r--@  1 username  staff  - 2318073 Apr  3  2009 P1050620.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      52889
    com.apple.metadata:kMDItemSupportFileType         70
-rw-r--r--@  1 username  staff  - 2570960 Apr  3  2009 P1050621.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      53586
    com.apple.metadata:kMDItemSupportFileType         70
-rw-r--r--@  1 username  staff  - 2476490 Apr  3  2009 P1050622.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      54021
    com.apple.metadata:kMDItemSupportFileType         70
-rw-r--r--@  1 username  staff  - 2427318 Apr  3  2009 P1050623.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      54039
    com.apple.metadata:kMDItemSupportFileType         70
-rw-r--r--@  1 username  staff  - 2403396 Apr  3  2009 P1050624.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      53837
    com.apple.metadata:kMDItemSupportFileType         70
-rw-r--r--@  1 username  staff  - 2355364 Apr  3  2009 P1050625.JPG
    com.apple.FinderInfo         32
    com.apple.ResourceFork      52112
    com.apple.metadata:kMDItemSupportFileType         70

total 28440
drwxr-xr-x   8 username  staff  -     256 Mar 20 08:01 .
drwxr-xr-x@ 37 username  staff  -    1184 Mar 20 23:28 ..
    com.apple.FinderInfo         32
-rw-r--r--   1 username  staff  - 2318073 Apr  3  2009 P1050620.JPG
-rw-r--r--   1 username  staff  - 2570960 Apr  3  2009 P1050621.JPG
-rw-r--r--   1 username  staff  - 2476490 Apr  3  2009 P1050622.JPG
-rw-r--r--   1 username  staff  - 2427318 Apr  3  2009 P1050623.JPG
-rw-r--r--   1 username  staff  - 2403396 Apr  3  2009 P1050624.JPG
-rw-r--r--   1 username  staff  - 2355364 Apr  3  2009 P1050625.JPG

Test 7
Both successful

 

[SpeQ]

I have to admit I’m confused here. Maybe because I’m not clear on the differences between Access Control Lists and Extended Attributes. Test 5&6 seem to end up with the same end product. Does that mean everything listed below is and extended attribute? Or did something go wrong with the command?

com.apple.FinderInfo         32
    com.apple.ResourceFork      53837
    com.apple.metadata:kMDItemSupportFileType         70

I'm also curious what the number 37 represents below. I've noticed that number keeps growing as we add folders, but it doesn't match the number of folders on the drive. Even after revealing hidden folders.

drwxr-xr-x@ 37 username  staff  -    1184 Mar 20 23:28 ..

 

[chown33]

There weren't any ACLs in your data. I was being thorough, in case they showed up unexpectedly. Since we don't know the cause, it's best to look at everything.

Test 5 & 6 should end up with the same result, but they get there in different ways. This may be significant or it may not. It depended on the result of Test 7. Since Test 7 was successful, the conclusion is that the route taken doesn't matter.

The "com.apple.*" things you listed are all extended attributes.

The number 37 for ".." is the number of hard-links to the root directory of the disk. On APFS, it might differ from the number of accessible directories at the root of the disk. I'm not entirely sure, because I don't know exactly how APFS magic like snapshots work. What's the output of this:

ls -al /Volumes/disko/

Since Test 7 is successful, the next step is to narrow down which xattr is causing the problem. I'll post more tests later, after adequate caffeination.

 

[SpeQ]

The number 37 for ".." is the number of hard-links to the root directory of the disk. On APFS, it might differ from the number of accessible directories at the root of the disk. I'm not entirely sure, because I don't know exactly how APFS magic like snapshots work. What's the output of this:

ls -al /Volumes/disko/

The difference between what I'm seeing with that output and what is shown in the volume is ".", "..", and ".DS_Store". Taking those into account, the total is 37.

That output shows a lot of other numbers that I'm curious about, but I'll hold off on that.

Take your time, I'll be out for a couple/few hours.

 

[chown33]

The man page for 'ls' describes all the items that appear on the '-l' line: man ls. Be warned, it's a pretty long man page.

EDIT
I was looking for a prettier way to view man pages, and came across this, using Preview.app:

man -t "ls" | open -f -a /Applications/Preview.app/

I think this might work on any recent macOS release, assuming Preview.app hasn't moved. I remembered reading about this kind of thing years ago, but never got around to trying it. Other ways I know of to get pretty man pages: a Mac app called Bwana, a Pi web-server on LAN.

The cmd line produces a PostScript man page, then pipes that to Preview.app, which shows it.

The web page showing this trick is here:

View Man Pages with Preview on Mac OS X

I am always looking for better ways to integrate the unix components into Mac OS X. I recently came across a great way to connect the `man` pages to a more visual mode in OS X.

www.devopsderek.com

 

[chown33]

Each of the following command-line sets makes a copy of the "master" dir, then removes one specific xattr from the files in that dir. The listing should show what was removed, as a check that it did the right thing.

You can run all the commands then check the results with a Finder search on each folder, or check each one with a Finder search before going to the next one.

cp -afR /Volumes/disko/master /Volumes/disko/test-no-support
xattr -d -r "com.apple.metadata:kMDItemSupportFileType"  \
  /Volumes/disko/test-no-support
ls -laeO@ /Volumes/disko/test-no-support

cp -afR /Volumes/disko/master /Volumes/disko/test-no-finfo
xattr -d -r "com.apple.FinderInfo"  \
  /Volumes/disko/test-no-finfo
ls -laeO@ /Volumes/disko/test-no-finfo

cp -afR /Volumes/disko/master /Volumes/disko/test-no-rsrc
xattr -d -r "com.apple.ResourceFork"  \
  /Volumes/disko/test-no-rsrc
ls -laeO@ /Volumes/disko/test-no-rsrc

If all the resulting "test-no-*" dirs fails a Finder search, then the problem is probably a combination of xattr's, not a single xattr. If that happens, I can post new cmds that delete xattr's in pairs, rather than singly.

If more than one works with Finder search, please post exactly which ones work.

 

[Brian33]

Just wanted to say that I’ve found the investigative techniques and incidental information enlightening. Good luck!

 

[SpeQ]

cp -afR /Volumes/disko/master /Volumes/disko/test-no-support
xattr -d -r "com.apple.metadata:kMDItemSupportFileType"  \
  /Volumes/disko/test-no-support
ls -laeO@ /Volumes/disko/test-no-support

This is the only one that was successful. So stripping the Support File Type attribute makes the files "findable". Or maybe changing that number from 70 to something else might work too(?), but we don't need to go there, unless you just want to for funsies. ?

 

[SpeQ]

I was looking for a prettier way to view man pages, and came across this, using Preview.app:

man -t "ls" | open -f -a /Applications/Preview.app/

I think this might work on any recent macOS release, assuming Preview.app hasn't moved.

The application /Applications/Preview.app cannot be opened for an unexpected reason, error=Error Domain=NSCocoaErrorDomain Code=260 "The file “Preview.app” couldn’t be opened because there is no such file." UserInfo={NSURL=file:///Applications/Preview.app/, NSFilePath=/Applications/Preview.app, NSUnderlyingError=0x14a70cd80 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}

Preview hasn't moved in Big Sur, I don't think that is the problem...

EDIT: The page you linked to talks about editing the bash profile before doing the command...