I have Background Refresh OFF in settings; always have.
When checking battery usage some background activity has always shown, and this is normal behaviour.
When an app is minimised/closed (not force closed) it is allowed to do some stuff for a short period (few seconds? half a minute?) regardless of the background refresh option. This is why most apps work fine with background refresh turned off globally. All the apps I've installed work fine.
Background refresh is more for stuff which will periodically update even if the app has been closed for ages, such as updating news or the weather etc every half an hour or so, so that when you next open the app it's updated and ready.
When iOS14 came out I did think my widgets wouldn't update due to my OFF setting, but it seems swiping to a page with widgets counts as opening the underlying app and they then update just fine.