Same here but other forum members write about their experience in the past. There are certainly a lot of individual factors at play but in essence we have plastic and metal surfaces in contact and when only held by magnets that can allow for the ever so slight wiggle of the pen along the iPad sides.
For those who use a case at that spot this should be a no-issue.