Continuity\Handoff doesn't work that way! You do need BT enabled and to be on the same WiFi network, but you don't need to actually make a BT connection. There are a few threads in the Yosemite Forum explaining requirements and then there's Apple Support >Here< with the official route. Have a read.