"diff -s" bug ... (not too serious)

[solouki]

Hi all,

This is not too serious of a bug, since "diff" still makes an appropriate "patch" file that "patch -p0" will properly use to correctly patch a file.

However, the Terminal output from "diff -s" is incorrect! So if you are just reading the results of "diff -s" on the Terminal, it will inform you that two different files are in fact "identical" --- this is an error in reporting the results of "diff -s" on the Terminal, and NOT an actual error of "diff -u" preparing an incorrect "patch" file when the two files differ. In summary, "diff -u/patch -p0" works correctly, but "diff -s" will report that two different file "are identical" on the Terminal line.

I'm using "/bin/bash" instead of "bin/zsh" (because I've found that some of my bash scripts do not run property when executed under zsh even though my scripts specify that /bin/bash should be used).

This reporting error is on an M3 Max MBP running Sonoma 14.1.1. I also checked an Intel MBP running Sonoma 14.1.1 and found the same reporting problem; however, on an old Intel MBP running Big Sur 11.6.2 this reporting error does not occur --- that is, under Big Sur "diff -s" does the proper thing and does NOT report that two different files "are identical" the way Sonoma does.

Solouki

P.S. And yes, I have send a bug report to Apple. I use diff/patch all the time, so thank goodness this still works correctly -- I just didn't want anybody to get "caught" thinking two files were the same because "diff -s" reports that the files are identical when in fact they are different.

Edit: I also checked an old MBP running Mavericks 12.7.1 and found that "diff -s" works properly unlike under Sonoma 14.1.1.

Edit 2: I have now checked Sonoma 14.1.2, and this reporting error still exists in it.

 

[Brian33]

@solouki -- many thanks for reporting this. I use 'diff -qs' often, so I find it rather alarming that diff may report non-identical files as being the same!

Is this hard to reproduce? I ask because I'm running Monterey (12.7.1) and I want to be sure this bug is not present. My very simple test with two small text files doesn't show any problem on Monterey. I'm just wondering if the test files need to be a certain size, or contain binary data, etc.?

 

[solouki]

@solouki -- many thanks for reporting this. I use 'diff -qs' often, so I find it rather alarming that diff may report non-identical files as being the same!

Is this hard to reproduce? I ask because I'm running Monterey (12.7.1) and I want to be sure this bug is not present. My very simple test with two small text files doesn't show any problem on Monterey. I'm just wondering if the test files need to be a certain size, or contain binary data, etc.?

Hi Brian33,

"diff" worked properly under Monterey and Ventura, it is only under Sonoma that it reports that two different files are identical (the reporting error).

I use "diff" a lot to make "patch" files, and fortunately this operation still works correctly even though it reports that the files are "identical", "diff" makes a patch file of their actual differences. That's why I used the "(not too serious)" in the title of this posting.

I also do a fair amount of public key encryption/decryption, and my code always decrypts an encrypted file and compares the decrypted file with the original plaintext just to make certain that the original encryption was performed properly. Fortunately, I do this comparison myself by computing sha256 hash codes and comparing these hash codes. If the hash codes differ, I then do a "diff -s" to view what the differences are --- this is how I discovered this reporting bug, even though the files were different, "diff -s" reported that they were identical.

You can easily test "diff" yourself by making two different files and performing "diff -s" on them. Under Monterey "diff -s" will report that the files differ, but under Sonoma "diff -s" will report the files are identical.

Following are the bash Terminal commands to execute:

cat > test1.txt << EOF
test1.dat
test.dat
EOF
cat > test2.txt << EOF
test2.dat
test.dat
EOF
diff -s test1.txt test2.txt

Under Sonoma 14.2 the "diff" command returns:

1c1
< test1.dat
---
> test2.dat
Files test1.txt and test2.txt are identical

Notice that "diff" actually detects the difference in the two files (i.e., the first lines of each file differ), but then it reports that "Files test1.txt and test2.txt are identical" which is the reporting bug.

Hope this helps. I'm pretty sure that you won't find a problem under Monterey. If you do, please report back because that would mean that Apple's imporper changes to "diff" actually made their way back to Monterey through an update.

Solouki

 

[Brian33]

Thanks, @solouki !

'diff -s' works properly in Monterey 12.7.1 and 12.7.2, according to the test above and a few other simple tests.

I wonder what 'diff -qs' would return on Sonoma with differing files. With the bug, would it say both "Files... differ" and "Files... are identical"?? Ha!

Anyway, hopefully it gets fixed in Sonoma before I get there!

 

[solouki]

Thanks, @solouki !

'diff -s' works properly in Monterey 12.7.1 and 12.7.2, according to the test above and a few other simple tests.

I wonder what 'diff -qs' would return on Sonoma with differing files. With the bug, would it say both "Files... differ" and "Files... are identical"?? Ha!

Anyway, hopefully it gets fixed in Sonoma before I get there!

Hi Brian33,

In my post #3 you can see the output from "diff -s", it is contradictory: the top four lines of the output identifies (correctly) the difference between the test1.txt and test2.txt files, then the fifth line states that the two files are identical. Definitely a bug, you can't have it both ways, the files can't be both different and identical at the same time. Fortunately, in a number of my scripts, I compare sha256 codes of two files to see if they are different, and then only run diff on different files to identify the differences. Also, when using diff to make a patch file, this works correctly also, so the bug in diff is really just a reporting error, and thus not too serious unless you are relying on that reporting. That's why I decided to post this bug here, because I thought it would be bad if someone else was relying on diff reporting only. I've sent a bug report to Apple, but I don't expect to hear back from Apple for awhile -- I'm sure at some point some minor update to Sonoma 14.x.x will correct this reporting bug.

Since I've been using diff for decades, I can tell you that this is the first time I've encountered this bug, so diff worked correctly and reported correctly through Monterey and Ventura, but this bug appeared in Sonoma. If it appears in an earlier OS, then an update must have transferred the bug back to the older OS. You have already checked Monterey 12.7.2, so it hasn't been retrofitted into Monterey 12.7.2 -- thanks for checking and reporting this, by the way.

Solouki

 

[maccan]

Thanks for reporting!
It is not fixed in Sonoma 14.2.1!
Some scripts may be impacted by flaws like this.
How the hell can Apple mess up with very basic UNIX commands.

Best regards!

 

[maccan]

Just got the gnu diffutils from here:
http://ftp.gnu.org/gnu/diffutils/
and compiled it. Works as expected!
So you have a correct working "diff" until this has been fixed (if ever).

Best regards

 

[KALLT]

When you use the -s argument to compare identical files in different directories, it works as expected.

% mkdir test1 test2 test3
% echo "1" > test1/test
% echo "2" > test2/test
% cp test1/test test3
% diff -s test1 test2
1c1
< 1
---
> 2
% diff -s test1 test3
Files test1/test and test3/test are identical

The error only seems to appear when comparing text files directly:

% echo "1" > test4   
% echo "2" > test5
% diff -s test4 test5
1c1
< 1
---
> 2
Files test4 and test5 are identical

When you add the -q argument as well, it works as you would expect:

% diff -qs test4 test5
Files test4 and test5 differ

The error seems to be that it outputs "Files test4 and test5 are identical" at all. Passing only the -s argument should either report the differences or report that the files are identical, not both.

Side note: When I check the manual page, it mentions the -s / --report-identical-files argument only under "Directory comparison options". Similarly, the synopsis lists -s / --report-identical-files only when the arguments are directories. FreeBSD's diff (which Apple's diff is now based on) makes the same claims. I wonder why it works for direct file comparisons at all, if that is not the documented as such.

Apple seems to have switched from gnudiff to FreeBSD diff as of macOS 13, but that does not explain why this change appears only in macOS 14.

 

[solouki]

Just got the gnu diffutils from here:
http://ftp.gnu.org/gnu/diffutils/
and compiled it. Works as expected!
So you have a correct working "diff" until this has been fixed (if ever).

Best regards

Thanks maccan.

You can also get diffutils from Homebrew, if interested. Fortunately, my scripts rely on sha256 to detect differences and not on diff, so I'm fine with Apple's diff -- but the reporting error is serious. Apple's diff still does make correct patch files, the bug is in the reporting, not in the patching.

 

[solouki]

Side note: When I check the manual page, it mentions the -s / --report-identical-files argument only under "Directory comparison options". Similarly, the synopsis lists -s / --report-identical-files only when the arguments are directories. FreeBSD's diff (which Apple's diff is now based on) makes the same claims. I wonder why it works for direct file comparisons at all, if that is not the documented as such.

Thanks KALLT.

My reading of the man page indicated that "diff -s" is used to compare files and report when they are the same, even if they are in different directories. When they are in the same directory, then the "-s" means that they should still report files that are identical (essentially, the directories are both ./). I'm pretty sure that this is a reporting bug, "diff -s" should not report that two different files are identical. Do you have access to a FreeBSD system? It would be interesting if the FreeBSD diff makes the same reporting error.

Solouki

 

[maccan]

Thanks maccan.

You can also get diffutils from Homebrew, if interested. Fortunately, my scripts rely on sha256 to detect differences and not on diff, so I'm fine with Apple's diff -- but the reporting error is serious. Apple's diff still does make correct patch files, the bug is in the reporting, not in the patching.

You know, if some discrepancies occur, there is some lack of trust!
But yes, as pointed out by KALLT, the "-s option" in the man-pages is documented under "Directory comparison options:" indicating it is not really intended to be used with plain file comparisons. However, it is still a bug!

 

[KALLT]

I also think that it is a bug and I agree that it is probably a reporting bug only. Even if -s is not intended for direct file comparisons, it should not output a wrong statement all the same.

diff still seems to correctly identify the differences and the exit code is correct too. It simply returns exit code 0 when the files are the same, something else (plus output) when they are not. You can mitigate this by passing the -q argument as well or by checking the exit code instead.

Perhaps it is better to switch to diffutils instead, if and until Apple fixes it.

@solouki: When have you reported this to Apple?

 

[solouki]

You know, if some discrepancies occur, there is some lack of trust!
But yes, as pointed out by KALLT, the "-s option" in the man-pages is documented under "Directory comparison options:" indicating it is not really intended to be used with plain file comparisons. However, it is still a bug!

Hi maccan,

I have a slightly different understanding of the Apple man page. Yes, the "-s" is listed under "Directory comparison options", but it is referring to files being different in two directories as well as the two directories having different files. In particular, the man page states for -s:

"Causes diff to report files which are the same, which are otherwise not mentioned."

The man page goes on to state:

"If both arguments are directories, diff sorts the contents of the directories by name, and then runs the regular file diff algorithm, producing a change list, on text files which are different. Binary files which differ, common subdirectories, and files which appear in only one directory are described as such. In directory mode only regular files and directories are compared. If a non-regular file such as a device special file or FIFO is encountered, a diagnostic message is printed."

Please correct me if I misunderstand, but to me this man page means that if two directories (which, by the way, could be the same directory specification with two different filenames) have two identically named files that are identical, then the -s causes diff to report that they are identical. The two directories could be the same, in which case the -s option causes diff to report that two identical files are indeed identical -- this is how diff has worked for me for decades. But now diff -s reports that two different files are identical -- a reporting error. If only directory names are given, then diff -s makes its own listings of the files in those two directories and then performs the file diff on these directories's files, the -s specifying that identical files will also be reported.

Thanks,
Solouki

P.S. I'll get around to checking this next week.

 

[solouki]

@solouki: When have you reported this to Apple?

Hi KALLT:

Thanks for your reporting. I sent a bug report via Feedback Assistant to Apple a couple months ago, haven't heard anything back, and don't really expect to hear anything. I expect that this reporting bug will just be fixed in some future minor update.

Since diff still makes correct patch files, and my code actually calculates sha256 hash codes to check if two files are the same or different, this reporting bug doesn't really affect me -- I just posted it here in case someone else was only looking at the diff -s reporting that is in error when it states that two different files are identical.

Solouki