I have spent some time with the GTX 5xx series (that's what I had handy) and was able to make it become a Quadro or Tesla through changing hard-straps (resistdors). The Quadro drivers would install but SpecViewPerf scores were not there which made me suspect there's more to this than just changing PCI IDs.
At the same time I was playing with the BIOS itself, but the problem with all this was that I had a Mac Pro so it was twice as difficult (BIOS + EFI) and Mac hardware does not play nice, so flashing was pain in the butt (FreeDOS booting, etc)
With Titan it does sound like a challenge because things change from generation to generation, though it might be possible to do it since all Titan boards are reference designs and layouts appear to be shared across platforms (Titan, Quadro, Tesla) for the most part.
Of course actual GK ASICs might be (are?) different in which case it would all be futile.
Image
Here's a close-up of the ROM for the Titan. Generally one finds hard-straps around it (or on the flip side) for PCI IDs with most GTX cards. This requires tracing paths on the board while making measurements with a DMM, from specific pins of the ROM to the ground (pull-down) or positive rail (pull-up).
With Titan, it appears that it shares the board design with GTX 780, which is good because they might leave the hard-straps in place to allow for different board PCI ID configurations.
What you are looking for is even further away than just modifying the PCI IDs. You need two Titans, modded and in the same box to start with (unless you have some inter-connect like Infiniband, but then that makes things event more complicated).
Also, let's not forget the BIOS itself. You might need the Tesla BIOS which you could compare with the Titan one, and see whether things need to be changed and how. It's a whole other area of work...
Next would be to find out where does RDMA happen and where does it get blocked (I will presume it will, because this is still not a real Tesla card) and go from there.
It might be easier to attempt all this from Linux, if it has support for RDMA, since it is more open to peeking and poking than other OSes.
Of course this is all in theory - I don't have a Titan, or two, that I could just experiment with, and it feels like a long shot anyway because it might require lots and lots of time.
Is the effort worth it for this one feature alone, and how many people would really benefit from it?
The time and other resources involved might simply outweigh the purchase of Tesla cards that do this OOB.
. 
