Performance issues running "sed" command on M1

[REBELinBLUE]

So I have been testing the M1 recently to make sure all our development tools work fine, most of them do apart from no MySQL 5.7 docker image, which is a known issue and easy to workaround.

The other issue I am having though is very strange and I am not able to figure out the cause, however, I don't know anyone else with an M1 to check if there is something wrong with my machine or it affects every M1.

Would someone be able to test something for me?

At work we have a horrible shell script that loops through a load of template files and runs sed on them, replacing placeholders with different values.

On my Intel MBP, this takes about 6 minutes to run, on the M1 MBP it takes nearly 25 minutes to run!

I have done a very stripped down version of this script https://github.com/REBELinBLUE/test

If I run sh -c "time ./test.sh"

on a 2.2 GHz 6-Core Intel Core i7 with 16 GB of RAM I get

❯ sh -c "time ./test.sh"
real    0m16.187s
user    0m2.187s
sys    0m6.264s

i.e. it takes about 16 seconds to run.

If I run it on the M1 MBP with 16GB of RAM I get

❯ sh -c "time ./test.sh"
real    0m52.319s
user    0m3.267s
sys    0m14.932s

Over 50 seconds, consistently, every single time. Several of my colleagues tried on their Intel MBPs and are all getting similar results to the 16s

We are hopefully replacing this script soon anyway as it is horrific but I am shocked about the massive performance difference and was wondering if it is this machine having problems or all M1 Macs

sed of course is compiled for arm

❯ file /usr/bin/sed
/usr/bin/sed: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/sed (for architecture x86_64):    Mach-O 64-bit executable x86_64
/usr/bin/sed (for architecture arm64e):    Mach-O 64-bit executable arm64e

 

[jdb8167]

Can you run it under Rosetta 2. It might just be poorly optimized for Arm64. It would be interesting to see if Apple’s optimizations in Rosetta do a better job.

 

[REBELinBLUE]

Had not thought to try that, I think this is how you do it, seems to be broadly the same

❯ env /usr/bin/arch -x86_64 sh -c "time ./test.sh"

real    0m58.784s
user    0m10.456s
sys    0m11.986s

Your comment about it maybe not being very well optimized for arm made me think to try the same thing on my raspberry pi 4

❯ sh -c "time -p ./test.sh"

real 19.43
user 7.51
sys 12.74

Closer to the Intel Mac than the ARM Mac

 

[Slartibart]

I have done a very stripped down version of this script https://github.com/REBELinBLUE/test

GitHub page is not up

 

[REBELinBLUE]

GitHub page is not up

Sorry, link was messed up as I pasted it from markdown, fixed now

 

[jdb8167]

Had not thought to try that, I think this is how you do it, seems to be broadly the same

❯ env /usr/bin/arch -x86_64 sh -c "time ./test.sh"

real    0m58.784s
user    0m10.456s
sys    0m11.986s

Your comment about it maybe not being very well optimized for arm made me think to try the same thing on my raspberry pi 4

❯ sh -c "time -p ./test.sh"

real 19.43
user 7.51
sys 12.74

Closer to the Intel Mac than the ARM Mac

The RPi version is interesting. I guess someone will have to take a look at the code to see what is slowing it down. There is no way that the RPi should be faster.

 

[REBELinBLUE]

The RPi version is interesting. I guess someone will have to take a look at the code to see what is slowing it down. There is no way that the RPi should be faster.

Indeed, which makes me wonder if this machine specifically is having some weird IO problem or if whatever is happening happens on all M1 macs

 

[wyrdness]

I've seen weird issues with using sed with the inline -i option on non-Intel architectures. Though the problem that I saw was where sed would just hang. That was caused by it creating random filenames for the inline editing, but using a broken random number generator that only generated a very few 'random' filenames. These files then weren't being deleted and subsequent sed commands would generate identical 'random' filenames and then hang for ages when they found an existing file with the same name.
Not sure if this is the same problem as you're seeing, but I though it worth mentioning. Would you be able to modify your script so that you're not using inline editing and see if that makes a difference?

When you've tried it on an Intel machine, does it have the same universal binary as the M1? If not, could you copy the universal binary to an Intel machine and see if the same slowdown occurs?

 

[Slartibart]

You can try GNU parallel. Something like

parallel  /usr/bin/sed "s/{env}/${env}/g" $file

Put this in a number of loops for the subsequent sed processing which equals the number of available CPU cores. I am not sure, but you might save additionally time when you pipe

output/$env/$env-$filename.$extension

directly in the following sed processing steps and not writing it to disk. But this has to be checked.

 

[leman]

I've seen weird issues with using sed with the inline -i option on non-Intel architectures. Though the problem that I saw was where sed would just hang. That was caused by it creating random filenames for the inline editing, but using a broken random number generator that only generated a very few 'random' filenames. These files then weren't being deleted and subsequent sed commands would generate identical 'random' filenames and then hang for ages when they found an existing file with the same name.
Not sure if this is the same problem as you're seeing, but I though it worth mentioning. Would you be able to modify your script so that you're not using inline editing and see if that makes a difference?

Ugh, these kind of bugs are the worst. I remember some years ago I was setting up an internal java-based web application for a research group. It had this super weird issue that sometimes it would start instantly and sometimes it would take a half an hour or more to launch. It was driving me crazy. I spent hours trying to figure out what was going on and then it turned out that Java random number generator was blocking the entire system as it was accumulating entropy for the initial seed. But this being a headless virtual machine and that particular Linux setup mainly gathering entropy from the peripheral events... the system was stalling indefinitely until I started poking around in the command line, and that mere fact would start generating entropy which would then magically start the service...

I mean, that's a super-hostile way to design a system. As a developer, I would expect the platform provided random number service to just work, not stall everything just because I happen to run a headless VM system. Talk about mental load. You can say anything you want about Apple, but at least they take this kind of stuff very seriously and collect the entropy during boot so that everything is ready to go when you need it.

 

[REBELinBLUE]

You can try GNU parallel. Something like

parallel  /usr/bin/sed "s/{env}/${env}/g" $file

Put this in a number of loops for the subsequent sed processing which equals the number of available CPU cores. I am not sure, but you might save additionally time when you pipe

output/$env/$env-$filename.$extension

directly in the following sed processing steps and not writing it to disk. But this has to be checked.

Interesting, thanks I'll try that. As I said this script is hopefully soon being dropped anyway, I was just surprised by the huge difference in runtime on the Intel and Apple Silicon Mac

 

[REBELinBLUE]

I've seen weird issues with using sed with the inline -i option on non-Intel architectures. Though the problem that I saw was where sed would just hang. That was caused by it creating random filenames for the inline editing, but using a broken random number generator that only generated a very few 'random' filenames. These files then weren't being deleted and subsequent sed commands would generate identical 'random' filenames and then hang for ages when they found an existing file with the same name.
Not sure if this is the same problem as you're seeing, but I though it worth mentioning. Would you be able to modify your script so that you're not using inline editing and see if that makes a difference?

When you've tried it on an Intel machine, does it have the same universal binary as the M1? If not, could you copy the universal binary to an Intel machine and see if the same slowdown occurs?

That is interesting, shall have a play around today

 

[REBELinBLUE]

Definitely appears to be an IO issue of some description... I rewrote the horrible script so instead of running sed for each pattern it has all the patterns in a single sed call so it only reads/writes once and it is massively improved, the full script down from about 25 minutes to 1 minute on the M1 mac

Honestly, a little embarrassed that no one thought about it before, shows the power of inertia... the script is working so why change it ? Still, slightly concerned, shall have to run a disk performance tool to check the drive

 

[REBELinBLUE]

Changing all the sed commands to a single call like

sed -i.bak -e "s/foo/bar/g; s/bar/baz/g" file.txt

certainly has massively improved the performance, although still weirdly noticeably slower on the M1

I tried without the inline option as suggested

sed -e "s/foo/bar/g; s/bar/baz/g" file.txt > file2.txt

and the runtime was the same (well within a margin of error, like 1s difference).

Odd

 

[bobcomer]

Honestly, a little embarrassed that no one thought about it before, shows the power of inertia... the script is working so why change it ? Still, slightly concerned, shall have to run a disk performance tool to check the drive

That's so common I could scream. At one job I had I was the guy that found the slowdowns and then fixed them. I've never worked anywhere else that had a guy that did that.

 

[REBELinBLUE]

That's so common I could scream. At one job I had I was the guy that found the slowdowns and then fixed them. I've never worked anywhere else that had a guy that did that.

Ha yeah, that is normally me too be honest, which is why I was asked to test the new Mac, just that this script has never been that much of an issue to bother with. Still really interested in finding out why the massive difference in performance between the Intel and Arm macs though. I will have to do more profiling of sed and some disk profiling but I suspect the issue is to do with sed or something related rather than the disk otherwise I would have experienced other issues.

 

[solouki]

I have done a very stripped down version of this script https://github.com/REBELinBLUE/test

Hi REBELinBLUE,

I wanted to run your sed test on a MBP M1 16GB, but for some reason I still get a 404 error even though you have "corrected" this.

Thanks,
Solouki

 

[REBELinBLUE]

Hi REBELinBLUE,

I wanted to run your sed test on a MBP M1 16GB, but for some reason I still get a 404 error even though you have "corrected" this.

Thanks,
Solouki

Sorry I deleted the repo since I'd worked around the issue, but I have pushed it again now. Thanks

 

[bobcomer]

Ha yeah, that is normally me too be honest, which is why I was asked to test the new Mac, just that this script has never been that much of an issue to bother with. Still really interested in finding out why the massive difference in performance between the Intel and Arm macs though. I will have to do more profiling of sed and some disk profiling but I suspect the issue is to do with sed or something related rather than the disk otherwise I would have experienced other issues.

I agree it's really odd! Let us know what you find, who knows, maybe something like that is slowing my stuff down even though there's nothing quite so obvious.

 

[REBELinBLUE]

I shall definitely keep poking it and report back if i find anything

 

[REBELinBLUE]

So I think it may actually be an issue with this Mac, someone else now has the exact same machine and it is much more like you would expect, like a 10th of the time it takes this machine.

Plus this machine has done some weird ****; twice on Friday, then yesterday when I was reading just a document and then again today, it completely locked up, then the screen flashed green and it rebooted. Also when we first turned on file vault it rebooted into recovery key mode but none had been reported to remote management (had forgotten about that).... so I am guessing there is some sort of disk issue.