When an application's option is set to "Keep in Dock", its Recent Items menu in the dock does not update. Unsetting that option partially fixes it; quitting and relaunching the app is still required to make the Recent Items menu update.
When "Keep in Dock" is set, the only solution I've found to updating the Recent Items menu in the dock is by executing "killall Dock" in Terminal.
A few google searches reveal that this has been a bug since the El Capitan betas.
I've submitted a bug report.
Yes, the Dock behaves weird since 10.11. I have shortcuts to encrypted .dmg volumes on the Dock and the icons and functionality is not working.