Unfortunately I don't have any experience with a pfsense VM but that could be the reason. Maybe better to ask in the Proxmox forum. They are very helpful and experienced.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
MP 1,1-5,1 Do any of you run Proxmox on your Mac Pro 5,1?
- Thread starter Project Alice
- Start date
- Sort by reaction score
I've been running proxmox on a MacPro 4,1 (flashed to latest 5,1) for nearly 2 years 24x7. Running a pfsense VM using a dedicated Dell H092P Intel Pro/VT quad card (about $30 on ebay.) I didn't manually install any drivers, and passthrough setup was very straightforward (in the VM settings before initial boot, go to Hardware, Add Device, "PCI Device", Raw Device, and select the device in the list. Done.) For my specific card, I did this twice in order to access all four ports.
I recently upgraded the system from an SSD to an NVME drive, (cloned the system with about 5 terminal lines) and also upgraded to the latest proxmox version. This has been the most rock-solid OS I have ever used in 25+ years in IT, and automated snapshots with ZFS has been incredible. 32GB RAM chips are plug-and-play on this too, making it a real beast for cheap (I used HMT84GR7AMR4C-H9 HYNIX.)
If you have a specific question, I will try to help.
I recently upgraded the system from an SSD to an NVME drive, (cloned the system with about 5 terminal lines) and also upgraded to the latest proxmox version. This has been the most rock-solid OS I have ever used in 25+ years in IT, and automated snapshots with ZFS has been incredible. 32GB RAM chips are plug-and-play on this too, making it a real beast for cheap (I used HMT84GR7AMR4C-H9 HYNIX.)
If you have a specific question, I will try to help.
Ok, Thanks! I will try that out when I get a chance. Thanks for offering to help!
Great, good to know that it is working such well for you.
Do you also run other VM's beside pfsense on this Mac?
May I ask how much RAM do you need for your ZFS? How big is your ZFS storage and which RAID variant are you using? Did you ever try to passthrough the USB controllers?
Thank you.
Hey, I run a lot of various VMs including some instances of MacOS (just to experiment with), windows, and many linux systems. I started out mainly wanting it for pfsense, unify and pihole, which I run my whole home network thru. It's been extremely solid for this. I like cloning the VMs and making tweaks to them (adding complex routing and VPNs to pfsense, for example) and being able to choose which instance to enable at any given time.
I have not passed thru any USB controllers, but I do not anticipate it would pose any issue whatsoever to do so. (I would do this with a PCIe card as I would want better than the onboard USB2 speeds.) I don't currently pass a video card thru either (running headless trying to keep power consumption lower) but I may do this soon for Mac VMs as card prices seem more reasonable now.
ZFS will consume approximately half of your available RAM if you let it (I do) but there are ways to disable this. I just let it do it's thing! This consumption seems to start anew upon startup and will slowly grow to 50% of RAM--if RAM is not available to it, it won't use it. Hasn't been an issue in my use case. I say all this from my experience having run 96GB for about a year, and now have 192GB in the system--the system usage has remained similar, relative/percentage-wise.
Another note--I also have both a dual processor tray and a single as I have a few MacPros, and switched them out in the proxmox system (between reboots) and it handled this with zero complaints! I thought that was pretty cool!
I'm not currently using RAID. My current ZFS storage is 2TB currently all on the NVMe boot drive, 970EVO+ on a OWC Accelsior card. This is not a recommended setup having boot drive and rpool on the same drive, but it works fine for me. I ran for a long time on a single SSD also in an OWC pcie card, and was amazed at how simple the cloning process was once I figured it out.
I've experimented with adding 4 SSDs in the built-in bays and passing-thru the onboard controller to a TruNAS Scale VM. It worked in my testing but i haven't implemented yet since I currently have a large Synology box and am waiting to get 4x large capacity SSDs before fully migrating. (But I have already migrated everything else like docker containers to proxmox.) I'm just watching drive prices fall so I can maximize ROI!
Last edited:
Thank you so much for sharing this. It's very useful and interesting for me.
I mainly want to use my Proxmox system as a macOS and Windows workstation with additional Linux containers to run some background services (pfSense, Plex etc.) independently from the workstation systems.
I could passthrough may AMD RX 5700 graphics card but it took me a while to get the reset bug fixed. Yesterday I was able to fix the Intensity Pro 4K passthrough and now I am working to make my Quantum tape drive available in the VM's through iSCSI. But unfortunately I was not able to passthrough the onboard USB controllers. I think it's because of the dual USB 1 + USB 2 functionality they have. Proxmox's USB Device passthrough is unfortunately limited to 5.
I am still running it on a spare HDD as an experiment. My NVMe boot drive has macOS and Windows as bare-metal installations on it.
If you passthrough hard drives, like with your TrueNAS VM, you are loosing the ability to make snapshots. Did you find a workaround for this?
Are you using Proxmox Backup Server? How do you backup your Proxmox system? Any experience with disaster recovery?
Thank you.
Ah, this does make sense.
Great point. Thank you! I don't need snapshots of my large file storage array. (For me, it'd be too expensive for the excess capacity needed!) This storage will be in a RAID config and I could bare metal install a TrueNas instance and access the storage that way in an emergency if needed. The TrueNas VM system install would still be housed on my ZFS drive (with snapshot) and would be backed up externally.
I periodically backup my Synology box to a big USB drive which I keep in "fire-proof" safe. (Would be nice to have 2 of these large USB drives and rotate them out. Keeping another offsite would be ideal but I'm working on a budget here!) I'll do the same once I move to TrueNAS.
I'm not using Proxmox Backup Server. I'm currently backing up my VMs to my Synology box via automated script. I don't know of a good solution to backup the Proxmox host itself. I'd love to hear ideas on this. I've used dd command copying to an SSD via USB. No disasters yet, but I've upgraded my boot device at least 3 times. In case it helps anyone, here are my notes to myself I've used for this. (Research these commands and their flags and change attributes accordingly.)
sudo dd if=/dev/sda of=/dev/nvme0n1 bs=1M status=progress
parted /dev/nvme0n1
resizepart 3 -1
quit
zfs list
zpool online -e rpool /dev/nvme0n1p3
Last edited:
So I did properly setup pci pass through and followed your instructions on setting for the pfsense VM. I have the install setup screen showing. I have been learning about linux and and I usually set my router up with custom setting after initial setup. So bear with me. My question is. I want to trial this setup with pfsense leaving my current router in control. After I prove out it's stable enough. I want to move it as the master router right after my fiber modem. So I assume I need to know the MAC address of the port for the WAN also the MAC address for the LAN I want to use? I also assume with that information I would need to reserve the ip address for the Wan MAC address then in the pfsense install setup add the correct information for WAN, configure LAN setup ip addresss range for the LAN side of pfsense?
Last edited:
That's what I did too.
Yes. I believe all the interfaces should appear in a drop-down selection list to choose from. That's how mine is now but I don't recall if I had to do something to set this up initially or not (I dont think so.)
That sounds right!
Thank you! I appreciate you taking time to answer my questions! One other question for now. If it proves to work well and I'm ready to move it as the master router. I would run a jumper ether cable to my built-in Mac Pro ether port that I am currently running the Proxmox master PVE so I have access to the PVE in the browser or is there a better way to do that? Also the most secure option would be best.
No problem! Pretty sure you'd need either a special cable or simply a device such as a switch in between. I use a TP-Link TL-SG108E, one of cheapest good managed switches I could find that could do aggregate ports.
Last edited:
So it would connect to the LAN side of the pfsense just like any other device. I assume on pfsense I can configure the other 3 ports as different VLAN's. My Mac has 2 built in ether ports also, so I could PCI passthrough that one also if I wanted to also use it with pfsense. I know They also support VLAN. I also have a Ubiquiti UniFi Switch so I will support the VLAN. I want segment my network a few ways.
Yes, I recommend you connect the LAN side of pfsense to your switch, along with all your other devices.
(Wish I had a Ubiquiti switch--very nice!) Sounds like you are on the right track. Good luck!
BTW, you will see it recommended everywhere to not use a virtualized router. The main reason for this is that can be a real pain to troubleshoot when something goes wrong. But if you're using a MacPro 5,1 at this point, you are already accustomed to this level of pain. I say jump in, the water is fine! Having the pass-through ethernet card keeps things pretty clean once it's all setup.
(Wish I had a Ubiquiti switch--very nice!) Sounds like you are on the right track. Good luck!
BTW, you will see it recommended everywhere to not use a virtualized router. The main reason for this is that can be a real pain to troubleshoot when something goes wrong. But if you're using a MacPro 5,1 at this point, you are already accustomed to this level of pain. I say jump in, the water is fine! Having the pass-through ethernet card keeps things pretty clean once it's all setup.
Last edited:
Even if you don't have snapshots you still can make scheduled backups within Proxmox like with all other VM's. I mean not from the RAID, just from the TrueNAS VM if you want to make one.
There seems to be no real solution for Proxmox host backups. Some use Clonezilla, some use scripts or third party applications... there is also a Proxmox Backup Client. I guess the idea is to keep everything as much as possible in the VM's and have a quick deployment procedure for Proxmox VE and just bring back your VMs. There is also a so called Live Restore: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_live_restore
As mentioned above. Poxmox has a good backup feature for the VMs which you could use for your backups to your Synology box: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_integrated_backup_and_restore
May I ask what exactly your script is doing. I understand only the first part with the dd command, then I get lost
Thank you.
Do you have any experience with iSCSI? I am trying to access my tape drive through iSCSI for different scenarios but somehow I don't get it work properly. Thank you.
Great links. I've used most of those tools in the past. I have some clone copies of my Proxmox boot drive and have automated backups of my VMs, so I guess I am in a comfortable spot. The commands I pasted above were from the last time I cloned my system drive and then expanded the data partition on my boot/system drive. (I went from 1TB SSD to 2TB NVMe.)I guess the idea is to keep everything as much as possible in the VM's and have a quick deployment procedure for Proxmox VE and just bring back your VMs. There is also a so called Live Restore: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_live_restore
May I ask what exactly your script is doing. I understand only the first part with the dd command, then I get lost
Thank you.
Do you have any experience with iSCSI?
It has been many years since I worked with iSCSI (and SCSI!) and I would honestly be no help there. Reading your tape drive comments above gave me horrible flashbacks lol. I spent countless hours struggling with SCSI devices back in the day! Also makes my back ache thinking of moving heavy rack gear!
Did you ever try that solution? It has many good..?
Yes...I had done just that on a 2011 Mac Mini server with an Akitio thunderbolt 2 enclosure...placed an Intel i350-t4 in it and you can add intrusion protection...I had Suricata on WAN and Snort on LAN. You could do the same if you add a thunderbolt 2 interface.
If you had not figured it out yet, just Nano /etc/modules and add
[TUTORIAL] - PCI/GPU Passthrough on Proxmox VE 8 : Installation and configuration
In this article, I propose taking a closer look at the configuration process for setting up PCI Passthrough on Proxmox VE 8.0 (I had initially planned this article for Proxmox VE 7, but since the new version has just been released, it's an opportunity to test!). This article will be the...forum.proxmox.com
NIC PCI-e Passthrough Bottleneck on Guest pfSense
Hello, I have a 25-Gbe embedded Intel E823-L controller which I'm trying to get as close to wirespeed as possible. When running iperf3 directly on the proxmox host I can hit full wirespeed. As soon I I try on a pfSense guest is when I run into problems. The maximum I could hit using a VirtIO...
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
