Are you trying to slide on it to act on it? or clear it?
You can 3D Touch on it to act on it, and swipe from the centre out to access the 'clear' button.
It doesn’t happen to me often, but boy is it irritating when it does. I think what happens is that the little vertical gap between notifications is counted as background (it is, so there is some logic in this) so when you swipe a little above/below the notification you intended it moves to the camera or widgets. IMO this little gap should either reject horizontal swipes (it should still react to vertical swipes accordingly and scroll) or just redirect the swipe to the nearest notification.
It’s a little thing, but I would agree is an irritant sometimes. I’ll see about sending some feedback, now I’ve been reminded.