Support PF VFIO Passthrough for InfiniBand Devices

[winsopc]

Background

Currently, ib-sriov-cni only supports VF (Virtual Function) devices. However, there are use cases where passing through the entire PF (Physical Function) to a VM is necessary, particularly in virtualized GPU workloads with RDMA requirements.

Use Case: KubeVirt with High-Performance GPU and InfiniBand RDMA for east west traffic

Scenario

When running GPU-accelerated workloads in KubeVirt VMs, the GPU PCI device is passed through to the VM using VFIO. In multi-GPU setups requiring east-west RDMA communication (e.g., distributed training, HPC workloads), the InfiniBand network interface must also be passed through to the VM for optimal performance.

Architecture Components

The solution involves the following Kubernetes components:

1. KubeVirt: Provides VM runtime on Kubernetes
2. Multus CNI: Enables multiple network interfaces per pod
3. ib-sriov-cni: Manages InfiniBand VFIO network configuration
4. sriov-device-plugin: Discovers and advertises InfiniBand devices to Kubernetes
5. ib-kubernetes: Manages InfiniBand GUID allocation and Pkey binding for multi-tenancy

Why PF Passthrough is Needed

In certain scenarios, PF passthrough is required instead of VF:

1. GPU Affinity: When GPU and InfiniBand devices share the same PCIe root complex, passing through the entire IB PF provides better performance and reduces complexity
2. Full Hardware Feature Access: Some advanced InfiniBand features may only be available on the PF
3. VM-Level Control: Giving the VM full control over the InfiniBand adapter for specialized configurations

[winsopc]

@SchSeba @adrianchiris for viz

[winsopc]

[adrianchiris]

@winsopc we had a discussion in the community meeting. no objection in adding PF support for ib-sriov-cni

[winsopc]

@adrianchiris thanks for the update! I will file a PR asap.