Are you putting the phone close to your face? If you clasp your fingers into a fist and rest it on your nose (at least a foot away), the end of your elbow is a good distance your phone should be when unlocking it with Face ID. It could even be further too. If the first swipe up failed to authenticate with your face you can swipe up to reactivate Face ID again.
If holding further still doesn't work, you can reset Face ID. If you wear glasses or etc that can obstruct your face a lot, you can add it as an alternate face.