Is OS X Lion Server an app that runs on lion or a standalone operating system?
Straight answer: No, it's not a standalone operating system. It's a package that you install on top of an existing Lion installation that adds various severs, tools and utilities. Many of these are just configuration utilities for server facilities already built into Lion (e.g. Apache and the email server) but there is some extra software as well. You are also given an option to "tune" various system settings to optimise your system for server, rather than desktop, use.
Whether that makes the result a "new" operating system or not is a matter of opinion, but practically speaking it feels more like installing an app than doing an OS upgrade, and shouldn't interfere with your existing applications. Lion Server is very different from older versions of OS X Sever in that respect.
NB: unless you'll be using something like the XGrid server or the mobile device profile manager I'd strongly advise first looking to see if there's an online HowTo showing how to add the facility you need to regular Lion, often just via config files or free software. My impression of Lion server was that unless you are setting up your system exactly as Apple envisioned, you're thrown back to the command line facilities included in Lion anyway.
There are also some glaring omissions for a home/small biz server: no friendly config for fetching POP/IMAP email or for name-based virtual hosting - again easily fixed with config-file edits but then firing up the server GUI tool will hose your config. YMMV depending on what features you use, but unless your ISP gives you multiple IP addresses and offers SMTP mail delivery, forget it as a mail/web server - it can be made to work, but you might as well set it up by hand on regular Lion.