First, it's now called "Messages" - Apple changed the name of the app. Second, it's a Mac app issue and not an OS item issue - you'd likely get a quicker response by posting in the relevant Forum. Third, take the time to review some of your options in your User Account, using the Help and Apple Support portal to highlight the differences. I'm not being a jerk, there's some significant differences in the two OSes.
Given that there's no start-up item selected as Rodan52 offered, the OS 10.11.x default upon logging out or shutting down includes reopening app Windows upon restarting, illustrated in the screen shot of the dialog I've attached. If you want to not have Message start up with the windowing you're seeing, either uncheck the "Reopen windows when logging back in" option or quit the Messages app before logging in/shutting down/restarting.