Yep, wouldn't be a good idea. Simple example:
You have a server out there on the internet. So I don't have to type out a long IPv6 address or MAC address, lets say the network number is 1.2.3 and the MAC is 4.5.6. Using your example of using the MAC for the host part of the IP, we'd get an IP of (yes, I know this IP is completely invalid. Same with the MAC) 1.2.3.4.5.6, correct? You got your DNS for your domain pointing there, and who knows what else.
Now, that server's NIC card dies. You put in a new NIC, but it's MAC is 7.8.9. So now, that machine's IP becomes 1.2.3.7.8.9. Which means your DNS is invalid and needs updating, and anything else you have pointing to that IP breaks. Just adds a lot of extra work to what would've been a 5 minute NIC swap.
In general, MACs should only matter within the subnet. When 2 hosts are communicating over a network, if the 2 of them are in different subnets, they shouldn't be concerned with the other's MAC. That subnet's router takes care of the MAC stuff