If you really want to write your own app for some reason (perhaps you want to add custom resources to the dmg or something) then you should probably look at wrapping the command line tools that create dmgs. I would start by looking at
hdiutil.
Note this assumes you already know how to write programs for the Mac in Cocoa or perhaps Carbon.