I have this setup. Upfront, I would not recommend Proxmox, the update methods are annoying. The better way is straight Debian with Incus installed, then you get straightforward stable Debian updates automatically - they won’t break anything and you’re secure. Sometime I’ll redo it - I haven’t because, of course, it is my router and when its down I don’t have internet! So foreboding and on the back burner.

Also also Proxmox’s GUI leaves a lot to be desired (for me, it looks like ass and is confusing), Incus is nicer for VM control and Cockpit is nicer for host control. After typing all that I realize I’m a hater at this point

I haven’t really noticed downtime issues cause of Proxmox updates cause I just do it when nothing is happening. And Proxmox hasn’t bricked itself, though I am wary of it because that has happened to others due to their rolling release update style.

I’ve got a Dell Wyse 5070 Extended with a 2 port Intel NIC in it. I pass both ports through leaving the built-in port for managing Proxmox.

Here are my notes:

Set NIC PCIe Passthrough for Network Card

nano /etc/default/grub

• Edit this line by adding intel_iommu=on to get

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

update-grub

nano /etc/modules

• Add these lines

vfio  
vfio_iommu_type1  
vfio_pci  
vfio_virqfd  

update-initramfs -u -k all

reboot

Click on 2nd level thing named router on the left side vertical bar hierarchy thing and then click in the top right the blue Create VM button.

• General tab
  • Name: OPNsense
  • Start at boot: checked
  • Start/Shutdown order: 1
  • Startup delay: 15
• OS tab
  • Use media: DVD version (usb might work) of OPNsense.iso
• System tab
  • Machine: q35
  • Bios: OVMF (UEFI)
    • Storage: local-lvm
    • UNCHECK Pre-enroll Keys (HATE)
• Hard Disk tab
  • Disk size (GiB): 15
  • Discard: checked
  • SSD emulation: checked
• CPU tab
  • Cores: 4
  • Type: host {makes it not moveable between diff CPU types but will theoretically allow for more speed}
• Memory tab
  • Memory (MiB): 2048
  • Minimum memory (MiB): 512
• Network tab
  • No network device: checked
• Confirm tab
  • Do not start on creation
• After creation, go to Hardware tab in the 2nd left vertical list on the browser page and click add
• Click PCI Device
  • Device: ...01:00.0 I350 Gigabit... & ...01:00.1 I350 Gigabit... (1st & 2nd ones)
  • PCI-Express: checked

Go to the Console tab in the 2nd left vertical list on the browser page and hit enter to get to a command line in the OPNsense VM

!Add expand storage via command line!

And lastly, during setup I have these notes

It will choose wrong (WAN gets igb1 and LAN gets igb0 -> we want WAN gets igb0 and LAN gets igb1)  
Default User: root, PW: opnsense (they don't tell you anywhere, you don't have internet b/c this is your new router, fuck em)  
**Access at 192.168.1.1 via pluging an ethernet cable into the 1st port in a set of forwarded ports**  
*Note that we will move it so the 1st port is the WAN (can't access OPNsense from the WAN port for safety), so after following this you access via 2nd port*  

So watch out for those things. Not sure quite what I mean by the 1st and 2nd port things, may be related to on setup it had the order of the ports I wanted wrong so they’re switched till setup is complete and it reboots.

I don’t remember doing this at this point, but maybe this info dump will help!

Pangolin has a lot more going on than I expected, I thought it was just a mesh system. That might be a one-stop shop, thanks for sharing!!

I do see Authentik can apparently act as a reverse proxy, so it can sit at the very front. But I’d lose out on caddy + crowdsec then…

I’ll have to do some reading if caddy can actually just forward to caddy, and where the TLS is terminated and all that.

And I do use the internal setting now, but I need it off if I want to publish the port on the LAN so that the VM can see the ports on the LAN. But if I can have WAN caddy do the auth check and forward along good stuff to the LAN caddy, then I’ll only need to publish LAN caddy’s port and that’s not the worst at all.

Thanks for the ideas, I’ll try to cook “caddy (DMZ) -> auth OIDC (DMZ) -> caddy (LAN) -> services (LAN)”!

Rootful to get the cross-user networking
Edit: cross-user via running the containers with userNS & user or volume idmap from root Podman. Containers can be run by different users and share networks, but the containers don’t have root access