GPU(RTX 3090) passthrough is not working on Ubuntu 18.04 with VMware ESXi

RTX 2080 or RTX 3070 on VMware has no problem. Gpu passthrough is ok, but I'm stack since 1 week trying to have my graphic card (RTX 3090) working in a VMware host.

Specifications:

• Dell Precision 3650 Tower
• Core i9-10900K
• 48GB Memory
• NVIDIA GeForce RTX 3090 (24GB)
• VMware ESXi 7.0 U2(vSphere)
• Ubuntu 18.04 (Guest OS)

I noticed that RTX 3090 has 24GB memory, so EFI boot is required and additional parameters are required.

Additional Parameters (VMware)
hypervisor.cpuid.v0=FALSE
pciPassthru.use64bitMMIO=TRUE
pciPassthru.64bitMMIOSizeGB=64

Host BIOS
memory mapped I/O above 4GB=Enabled

But it had no effect. By the way, the graphic card works fine on Windows (not on VM).

# journalctl -ex
Oct 15 15:45:08 ubuntu kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 242
Oct 15 15:45:08 ubuntu kernel: nvidia 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
Oct 15 15:45:08 ubuntu kernel: NVRM: The NVIDIA GPU 0000:0b:00.0 (PCI ID: 10de:2204) NVRM: installed in this system is not supported by the NVRM: NVIDIA 470.63.01 driver release. NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products' NVRM: in this release's README, available on the operating system NVRM: specific graphics driver download page at
Oct 15 15:45:08 ubuntu kernel: nvidia: probe of 0000:0b:00.0 failed with error -1
Oct 15 15:45:08 ubuntu kernel: NVRM: The NVIDIA probe routine failed for 1 device(s).
Oct 15 15:45:08 ubuntu kernel: NVRM: None of the NVIDIA devices were initialized.
Oct 15 15:45:08 ubuntu kernel: nvidia-nvlink: Unregistered the Nvlink Core, major device number 242
Oct 15 15:45:08 ubuntu kernel: PKCS#7 signature not signed with a trusted key

# apt-get install nvidia-driver-470
Reading package lists... Done
Building dependency tree
Reading state information... Done
nvidia-driver-470 is already the newest version (470.63.01-0ubuntu0.18.04.2).
# uname -a
Linux ubuntu 4.15.0-159-generic #167-Ubuntu SMP Tue Sep 21 08:55:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# prime-select query
nvidia
# lspci | grep -i nvidia
0b:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)
# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I have no idea why I can't use RTX 3090.

Additional Info:

# grep -E "vmx|svm" /proc/cpuinfo
Nothing is daiplayed.
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.15.0-159-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
# dmesg | grep -e DMAR
Nothing is daiplayed.
# journalctl -b -k | grep DMAR
Nothing is daiplayed.
# lspci -vvv
0b:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1) (prog-if 00 [VGA controller]) Subsystem: Dell Device 3880 Physical Slot: 192 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at 3000 [size=128] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x32, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [250 v1] Latency Tolerance Reporting Max snoop latency: 71680ns Max no snoop latency: 71680ns Capabilities: [258 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=255us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [128 v1] Power Budgeting <?> Capabilities: [420 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900 v1] #19 Capabilities: [bb0 v1] #15 Capabilities: [c1c v1] #26 Capabilities: [d00 v1] #27 Capabilities: [e00 v1] #25 Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.