Is there any specific guide you followed?
I've also been meaning to try Ubuntu on my Nexus 4 but again, been too lazy
The CyanogenMod Wiki has build guides for all supported devices. Go to the device page and click "how to build for this phone" and it'll give you all the steps.