Invisible files and Apple's Finder, tar, rsync, and GNU tar

[solouki]

Preface:
I know most individuals do not use the tar utility, but
this problem affects not only tar backups but also the
Apple Finder's Drag-and-Drop, Copy-and-Paste, AirDrop,
and the Universal Control's Drag-and-Drop.

File: README.Visibility.txt

Title: Invisible files and Apple's Finder, tar, rsync, and GNU tar

Date: 2022/04/22

Purpose:
Test the Finder's Drag-and-Drop, AirDrop, Copy-and-Paste,
and Universal Control's Drag-and-Drop for copying visible
and invisible files.
Test the Apple tar, GNU tar, and rsync utilities for copying
visible and invisible files.

Précis:
The short and sweet version:
Apple's Finder and Apple's tar utility miss some invisible
files when copying/backing up/transferring!
The GNU tar and rsync utilities do NOT miss these
invisible files.


Machines:
2020 13" M1 MacBook Pro (MacBookPro17,1),
2021 16" M1 Max MacBook Pro (MacBookPro18,2)

OS: macOS Monterey 12.3.1 (latest)

Programs:
Apple's Finder: version 12.3,
Apple's /usr/bin/rsync: version 2.6.9 protocol version 29
Apple's /usr/bin/tar: bsdtar 3.5.1 - libarchive 3.5.1 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.8
GNU's /opt/homebrew/bin/gtar: tar (GNU tar) 1.34
(all of these are the latest versions available)

Below are the test files: some visible, some invisible, some hidden, some not hidden:
(I apologize for the formatting of the following listings. I do not know how to ensure
that the columns/spacings are obeyed in the following listings.)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
dirac:Test-visibility-files solouki$ ls -alOR@
total 43392
drwxr-xr-x 13 solouki staff - 416 Apr 22 22:09 .
drwxr-xr-x 57 solouki staff - 1824 Apr 23 00:38 ..
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 ._invisible-type-2.bin
-rw-r--r-- 1 solouki staff - 54 Apr 21 06:12 ._invisible-type-2.txt
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1-made-visible.txt
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 .invisible-type-1.bin
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1.txt
-rw-r--r--@ 1 solouki staff - 7209398 Apr 18 12:29 Screen Shot 2022-04-18 at 12.29.24 PM.png
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
com.apple.FinderInfo 32
com.apple.lastuseddate#PS 16
com.apple.metadata:kMDItemIsScreenCapture 42
com.apple.metadata:kMDItemScreenCaptureGlobalRect 76
com.apple.metadata:kMDItemScreenCaptureType 48
drwxr-xr-x 12 solouki staff - 384 Apr 22 22:11 Temp
-rw-r--r-- 1 solouki staff - 36 Apr 22 01:33 hidden-made-visible.txt
-rw-r--r-- 1 solouki staff hidden 23 Apr 22 01:27 hidden.txt
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
-rw-r--r-- 1 solouki staff - 24 Apr 18 09:40 visible.txt

./Temp:
total 43392
drwxr-xr-x 12 solouki staff - 384 Apr 22 22:11 .
drwxr-xr-x 13 solouki staff - 416 Apr 22 22:09 ..
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 ._invisible-type-2.bin
-rw-r--r-- 1 solouki staff - 54 Apr 21 06:12 ._invisible-type-2.txt
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1-made-visible.txt
-rwxr-xr-x 1 solouki staff - 187 Apr 12 2020 .invisible-type-1.bin
-rw-r--r-- 1 solouki staff - 43 Apr 18 09:31 .invisible-type-1.txt
-rw-r--r--@ 1 solouki staff - 7209398 Apr 18 12:29 Screen Shot 2022-04-18 at 12.29.24 PM.png
com.apple.FinderInfo 32
com.apple.lastuseddate#PS 16
com.apple.metadata:kMDItemIsScreenCapture 42
com.apple.metadata:kMDItemScreenCaptureGlobalRect 76
com.apple.metadata:kMDItemScreenCaptureType 48
-rw-r--r-- 1 solouki staff - 36 Apr 22 01:33 hidden-made-visible.txt
-rw-r--r-- 1 solouki staff hidden 23 Apr 22 01:27 hidden.txt
-rw-r--r-- 1 solouki staff - 14970880 Apr 21 12:00 test.tar
-rw-r--r-- 1 solouki staff - 24 Apr 18 09:40 visible.txt
dirac:Test-visibility-files solouki$
----------------------------------------------------------------------

As you will notice in the above listing, there are `dot' files
(normally invisible from the Finder but are visible if the AppleShowAllFiles flag
is set for the Finder, i.e., if you execute:

defaults write com.apple.finder AppleShowAllFiles -bool YES && killall Finder

then the Finder shows these files as `grayed-out' names).

You will also notice there are `hidden' files
(files whose hidden flag has been set with:

chflags hidden hidden.txt

then the Finder shows these files as `grayed-out' if the AppleShowAllFiles
flag is also set, otherwise these files are hidden
from the Finder).

There is also a screenshot file with metadata.

You will also notice that there are `dot-underscore' files.
These files are not only invisible to the Finder, but they are not
shown by setting the AppleShowAllFiles flag. These files are not
only invisible to the Finder, they are also not copied by any Finder
techniques! Thus if you attempt to copy these files using the
Finder's Drag-and-Drop, AirDrop, Universal Control's Drag-and-Drop,
or Copy-and-Paste, these files will NOT be copied or transferred.

In addition, Apple's tar utility will NOT backup the `dot-underscore'
files, even if they are listed directly in the tar include file!
However, the GNU tar utility will backup these `dot-underscore'
files! Both Apple's tar and the GNU tar utilities now, under macOS
12.3.1, have "Cannot stat: No such file or directory" errors when
pattern matching regular expressions in the `--files-from=' option.
This used to work in earlier macOS versions, but pattern matching in
tar includes does not work in macOS 12.3.1. Intriguingly, regular
expression pattern matching does work in the ``--exclude-from='
option -- so there is an inconsistency between the include and
exclude options for pattern matching!

(I tried multiple different ways of including these `dot-underscore' files in
the Apple tar backups, including directly naming them. None of these
attempts was successful.)

Apple's rsync utility does copy/backup/transfer the `dot-underscore' files.
It does not copy the `hidden' flag nor Apple metadata, however, just the file.
(Thus there is an inconsistency between Apple's tar and rsync
utilities, with Apple's tar missing the `dot-underscore' files while
Apple's rsync properly copying the `dot-underscore' files.)

For the Finder methods for copying files, Drag-and-Drop and
Copy-and-Paste both copy Apple's metadata and the hidden flag,
but both miss the `dot-underscore' files. Apple's AirDrop copies
Apple's metadata but misses Apple's hidden flag and misses the
`dot-underscore' files. Intriguingly, even though the AirDrop was
between the same user on both computers, AirDrop added the
Apple metadata "com.apple.quarantine" to every file transferred by
AirDrop. Finally, Universal Control's Drag-and-Drop misses not
only the `dot-underscore' files, but also misses Apple's metadata
and Apple's hidden flag. UC's Drag-and-Drop, however, does add
the Apple metadata "com.apple.mac1" to the Folder being dragged
and dropped, but not to the Folder's contents.


In Summary:
If you wish to transfer/backup/copy all files, especially including
any `dot-underscore' files, do NOT use Finder nor Apple's tar
utility which will miss these invisible files.
(In other words, Drag-and-Drop, Copy-and-Paste, AirDrop,
Universal Control, or Apple's tar will not transfer/backup/copy
all files.)
Instead, do use the GNU tar or Apple's rsync utilities to
transfer/backup/copy all files including the invisible
`dot-underscore' files.
In a nutshell, Apple's Finder and tar both miss `dot-underscore'
files, but they both do copy Apple's metadata and Apple's
hidden flag. On the other hand, Apple's rsync and GNU's tar
do not miss the `dot-underscore' files, but neither copies
Apple's metadata nor the hidden flag. All of these backup utilities
do, however, copy the exif metadata.


Addendum:
While Apple's tar utility allows the `--files-from=' and
`--exclude-from=' options to be in any order, the GNU tar utility
requires the `--exclude-from=' option to precede the `--files-from='
option on the commandline.

EDIT: If anyone is interested in performing their own tests, I have a
bash shell script that performs the Apple tar and GNU tar tests.

 

[Diskutant]

The dot underscore files are only needed on non Apple Filesystems
Using the finder the data in there will be saved with the file in the filesystem, those are not seperate files in Apple Filesystems. It's just some metadata.

The gnu tar version doesn't know about those, so it saves them as extra files, while the Apple version includes it in the file as it should. So using the gnu version you are actually losing the meta data as those are now seperated from the file.

Also to show hidden files in Finder just press cmd+shift+. (dot), there is no need to use the Terminal commands for that.

 

[solouki]

The dot underscore files are only needed on non Apple Filesystems
Using the finder the data in there will be saved with the file in the filesystem, those are not seperate files in Apple Filesystems. It's just some metadata.

The gnu tar version doesn't know about those, so it saves them as extra files, while the Apple version includes it in the file as it should. So using the gnu version you are actually losing the meta data as those are now seperated from the file.

Also to show hidden files in Finder just press cmd+shift+. (dot), there is no need to use the Terminal commands for that.

Diskutant, thanks for the information...

My tests were on separate files beginning with `dot-underscore'. These are not copied/backed up/transferred by Apple's Finder nor Apple's tar, but they are by Apple's rsync.

Also, in my hands, GNU tar does copy the included exif information (the hundred or so exif variables viewed with exiftool), say for the screenshot, but it does not copy the Apple "com.apple.metadata:*" information while Apple's tar does copy Apple's "com.apple.metadata:*" information as well as the exif information. But Apple's tar and Finder do not copy separate `dot-underscore' files while GNU tar does. I suppose, in some fashion, `dot-underscore' files are "illegal" under Apple's Finder and tar, but "legal" under Apple's rsync, zip, ls, stat, and file. In other words, while numerous Apple macOS utilities recognize `dot-underscore' files, I could not get Apple's tar to recognize these files. There seems to be an inconsistency between Apple's tar on one hand, and Apple's rsync, zip, ls, file, stat, and GNU tar on the other hand, at least in my hands. Interesting...

Thanks, Solouki

 

[nmt1900]

As you will notice in the above listing, there are `dot' files
(normally invisible from the Finder but are visible if the AppleShowAllFiles flag
is set for the Finder, i.e., if you execute:

defaults write com.apple.finder AppleShowAllFiles -bool YES && killall Finder

then the Finder shows these files as `grayed-out' names).

It was probably on 10.14 Mojave or 10.15 Catalina, when we got much easier way to toggle hidden files in Finder's view - by using key combination Command+Shift+. (period)...

 

[solouki]

It was probably on 10.14 Mojave or 10.15 Catalina, when we got much easier way to toggle hidden files in Finder's view - by using key combination Command+Shift+. (period)...

Yes, thanks. This does not make the `._' files visible, however, and also there is the inconsistency between tar and rsync, zip, ls, stat, etc. GNU tar does partially alleviate this inconsistency while introducing a different problem.

And Apple's rsync behaves like GNU's tar; both copy all files but both miss Apple's metadata and Apple's hidden flag.

In a nutshell, we have Apple's Finder and Apple's tar, which miss copying some `invisible' files but do copy Apple's metadata and hidden flag, or Apple's rsync and GNU's tar, which copy all files but miss Apple's metadata and hidden flag.

The fact that "rsync -avz" misses Apple's metadata and hidden flag makes me wonder how backup apps, like CCC, that employ rsync to perform their clones, capture these data? Is there some additional option for rsync that I'm missing?