I would continue to hit Eject before disconnecting any drive. Modern journaled filesystems such a JHFS+ and APFS are more resilient than previous PC and UNIX filesystems but they still aren't designed for random (to them) disconnects.
Recent example: intending to move one ejected drive to another system, accidentally unplugged an HDD with JHFS+. Plugged it back in and immediately ran Disk Aid. All good. Later doing some testing with it for unrelated reasons on another system, noticed that performance was down by 1/3rd (which is more painful with HDD than SSD). Ran First Aid, moved it back to the original system, repeated, etc. Still down by 1/3rd. More important than the actual performance was whether this was early signs of drive of failure? No indication elsewhere so tried one last thing: ran fsck_hfs at the command-line to rebuild the catalog btree. Performance instantly up 50% (i.e. returned to previous).
APFS is probably more robust but I am sure it's better to eject safely than not. Plus you never know when macOS, etc decides to write to a drive. If you haven't disabled Spotlight indexing, it could very well be scanning the drive and writing to its index when you yank it from the system. Which may only lead to a corrupted index that will be rebuilt automatically the next time it is plugged in but, again, best to always hit Eject like you were doing...