I would think that unless you need a second video card to drive more displays, there's no real advantage to having it installed. Also, potentially a disadvantage as there's more memory, system management (PCIE lanes in use for TB controllers, DisplayPort upstream etc.) and increased thermal load inside the machine.
Unless you need the 580X, there will be 0 benefit to keeping it installed IMO. Though I guess it does provide additional Thunderbolt 3 ports, and if you do go dual or triple monitor it takes the load of one of the video processors?
I'd not want to waste a perfectly good card though either, so I feel your "hesitation".