Something's probably stuck. I can't find an easy way to cause the circle button to appear, only the square. If force quitting Messages doesn't do it, then maybe power the phone on and off. If that fails, maybe you did something to turn on an accessibility mode?
Red circle usually means "press to record" and square usually means "recording". For me, I need to press and hold the waveform to start recording, and it goes directly to square. When I'm done, it shows a triangle play icon.