Got it fixed!
1) I did not have a directory server installed - I only 2 macs, so why bother? Well, as it turns out, even though it is an option during setup to not use a directory server, it doesn't seem to particularly like doing things local. At least when it comes to mail server. The mail server doesn't pick up local users other than the initially created administrator. Properly setting up a local master ldap made life a lot easier. I just added the users using the workgroup manager, added the workgroup group and adding the email address to the short names made everything working honky dory. That last step is really important.
2) Connection issues were caused by a screwed up DNS config. After the reinstall, the DNS was properly setup again and after setting the local DNS as primary in the DHCP server on my router, everything works both locally and remote.
I still think OSX Server is very un-
like. Most tools needed and described in the OSX Server manual are not present until downloaded from some obscure kb article, it requires manual hacking config files through Terminal to get virtual hosts working and settings are all over the place. They make it look really simple, but unlike regular OSX which anybody can use, Server does require knowledge of servers, so at least you know where to look to fix things. They make things look easy with the Server.app, but I doubt Average Joe will be able to set this up for anything other than a 100% internal home server without real domain and incoming connections from the internet.