Could this have anything to do with where the text message is coming from? That is, if the person you last sent an SMS replies you don't get sound, but if the text is from someone else you do (regardless of whether the message app is open or closed when the SMS comes in)?
As someone said before, the phone might not make a sound if it thinks you're in a conversation.
Um, why would it not alert me if a message comes from someone I've texted before? Why on earth would an answer to a recently posed question not be important enough to alert me about?
Sure, if INSIDE the sms app, then the selected sms tone is replaced with a short shwoosh sound, but as soon as you're outside of that app — and especially when the phone is asleep and thus potentially in my pocket — an alert is more important than ever.
Luckily, though, your theory is easily dismissed. A lot of the time I text the same people, and simply put: the behavior isn't consistent. I can have the phone go to sleep after reading an sms, I can exit the sms app and have it go to sleep from the start screen — the bug doesn't really depend on that. Most of the time it will alert me as expected. But at times, the buggy behavior comes to life.
Also (I forgot to mention that) the buggy behavior may "stick" for a while. Or, again, it may immediately go back to behaving as expected. Again, I can find no pattern for when or why.
To further dismiss your theory, note that the problem described is that NO alert goes off whatsoever. If "in a conversation", as you put it, you should still get a shwoosh sound. This is just plain wrong.
Finally, there really is no situation where the phone could "think" I am in a conversation — either I am or I'm not.
Reading your guesses makes me quite sure you don't use an iPhone yourself, so let's just state that the distinction between the two situations (sms app open, or not) is really easy to make.
The fact that the behavior is unpredictable and intermittent is what makes it a bug.