Using non-discoverable devices connected to I/O virtualization block

Important

This feature might not be applicable to all Platforms. Please check individual Platform pages, section Supported Features to confirm if this feature is listed as supported.

Overview

The reference design platforms that support a IO Virtualization block as part of the compute subsystem allow connecting non-discoverable devices (non-PCIe) to it. The I/O virtualization block includes SMMUv3 to translate address and provide device isolation security, GIC-ITS to support MSI interrupts and NI-700 inter-connect to route transactions in and out of x16, x8, x4_1, and x4_0 ports.

The non-discoverable devices that are connected to the Io Virtualization block include - two PL011 UART, two PL330 DMA controllers and six regions of SRAM memory. The reference design software stack includes support to configure and use these devices and memory regions.

This document describes how to build the Neoverse reference design platform software stack and use it to test non-PCI devices that are connected on I/O virtualization blocks. Busybox Boot is used on the Neoverse RD FVP and tests are run from the command line to validate the devices.

NOTE: These tests are supported only on reference design platforms that designate one of the IO virtualization block for connecting non-discoverable devices.

Build the platform software

Note

This section assumes the user has completed the chapter Getting Started and has a functional working environment.

Refer to the Busybox Boot page to build the reference design platform software stack and boot into busybox on the Neoverse RD FVP.

Running tests for non-PCI devices on busybox

To begin the tests with the non-discoverable devices connected to the IO Virtualization block, boot to busybox using the command mentioned below (refer to Busybox Boot guide for details on the parameters).

./boot.sh -p <platform name> -a <additional_params> -n [true|false]

PL011 UART

There are two PL011 UART controllers connected to the non-discoverable IO Virtualization block. These UART controllers are initialized by edk2 firmware before booting to the Linux kernel.

These UART peripherals are enumerated by the Linux Kernel as Serial ports and can be tested by writing to the corresponding tty device.

  • After booting into busybox, verify that the two PL011 UART controllers are enumerated. The command provided below will list the serial ports detected by the Linux kernel.

    # dmesg | grep tty
[    0.034995] ARMH0011:00: ttyAMA0 at MMIO 0x1080000000 (irq = 14, base_baud = 0) is a SBSA
[    0.035195] ARMH0011:01: ttyAMA1 at MMIO 0x10a0000000 (irq = 15, base_baud = 0) is a SBSA
[    0.035595] ARMH0011:02: ttyAMA2 at MMIO 0xef70000 (irq = 36, base_baud = 0) is a SBSA
[    0.037095] printk: console [ttyAMA2] enabled

  • Here, ttyAMA0 and ttyAMA1 are the PL011 UART peripherals that are connected to the I/O virtualization block.

  • Now test the PL011 UART peripherals by writing to the corresponding tty device files using echo command:

    # echo "test message 0" > /dev/ttyAMA0
# echo "test message 1" > /dev/ttyAMA1

  • The above commands print the message on the FVP iomacro_terminal_0 and FVP iomacro_terminal_1 terminals respectively.

PL330 DMA

There are two PL330 DMA controllers connected to the non-discoverable IO Virtualization block. Each of these controllers support 8 data channels and one instruction channel.

To test these dma controllers, DMA test guide included in the Linux kernel documentation has to be followed. As mentioned in the guide, CONFIG_DMATEST has to be enabled in the Linux kernel.

  • After booting into busybox validate that the DMA PL330 controllers are probed fine and showing 8 channels on each dma controllers - dma0 and dma1.

    # ls /sys/class/dma
dma0chan0  dma0chan3  dma0chan6  dma1chan1  dma1chan4  dma1chan7
dma0chan1  dma0chan4  dma0chan7  dma1chan2  dma1chan5
dma0chan2  dma0chan5  dma1chan0  dma1chan3  dma1chan6

  • Also verify that the two dma controllers are attached to SMMUv3 of I/O Virtualization block. An example of this shown below.

    # ls /sys/class/iommu/smmu3.0x0000000048000000/devices
ARMH0330:00  ARMH0330:01

  • Following the DMA test guide, set the timeout and number of iterations. For example,

    # echo 2000 > /sys/module/dmatest/parameters/timeout
# echo 1 > /sys/module/dmatest/parameters/iterations

  • Start the test for different channels. For example, to run on dma0chan0 use the following command:

    # echo dma0chan0 > /sys/module/dmatest/parameters/channel
# echo 1 > /sys/module/dmatest/parameters/run

  • One can use loops to run the tests on all channels, for example for dma0:

    # for ch in 0 1 2 3 4 5 6 7; do echo dma0chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done

    • Similarly, for other controller - dma1:

    # for ch in 0 1 2 3 4 5 6 7; do echo dma1chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done

  • Test result: Test results are printed to the kernel log buffer with the format:

    "dmatest: result <channel>: <test id>: '<error msg>' with src_off=<val> dst_off=<val> len=<val> (<err code>)"

    • Below example if from running the test on all channels of dma0.

    # for ch in 0 1 2 3 4 5 6 7; do echo dma0chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done
[ 1375.118694] dmatest: Added 1 threads using dma0chan0
[ 1375.118694] dmatest: Started 1 threads using dma0chan0
[ 1375.123195] dmatest: dma0chan0-copy0: summary 1 tests, 0 failures 235.45 iops 3060 KB/s (0)
[ 1376.119025] dmatest: Added 1 threads using dma0chan1
[ 1376.119025] dmatest: Started 1 threads using dma0chan1
[ 1376.120894] dmatest: dma0chan1-copy0: summary 1 tests, 0 failures 615.76 iops 3078 KB/s (0)
[ 1377.119311] dmatest: Added 1 threads using dma0chan2
[ 1377.119311] dmatest: Started 1 threads using dma0chan2
[ 1377.123594] dmatest: dma0chan2-copy0: summary 1 tests, 0 failures 246.24 iops 2954 KB/s (0)

... and so on

SRAM Memory

There are six SRAM memory regions connected to the non-discoverable IO Virtualization block. Out of the six, two SRAM memory are connected to the the high bandwidth port of the I/O virtualization block and the remaining 4 are connected to the low bandwidth port. The size of each SRAM memory connected to the I/O virtualization block is 4MiB. The memory mapping for all the SRAM memory are listed in the below table:

SRAM that are connected to high bandwidth port:

Mem Name

Start Address

End Address

Size

MEM0

0x10_8001_0000

0x10_8001_FFFF

4MiB

MEM1

0x10_B002_0000

0x10_B002_FFFF

4MiB

SRAM that are connected to low bandwidth port:

iomacro_size = 0x2000000

iomacro_instance :
    RD-N2-Cfg1 = 1
    RD-N2      = 4

Mem Name

Start Address

End Address

Size

MEM2

0x4100_0000 + (iomacro_instance * iomacro_size)

0x413F_FFFF + (iomacro_instance * iomacro_size)

4MiB

MEM3

0x4140_0000 + (iomacro_instance * iomacro_size)

0x417F_FFFF + (iomacro_instance * iomacro_size)

4MiB

MEM4

0x4180_0000 + (iomacro_instance * iomacro_size)

0x41BF_FFFF + (iomacro_instance * iomacro_size)

4MiB

MEM5

0x41C0_0000 + (iomacro_instance * iomacro_size)

0x41FF_FFFF + (iomacro_instance * iomacro_size)

4MiB

  • The SRAM memory can be tested by using devmem busybox utility which can be used to read and write to physical memory using /dev/mem provided that the Linux Kernel is built with Kernel config option CONFIG_DEVMEM=y

  • After booting into busybox, use the following example to test the SRAM memory connected to the non-discoverable I/O virtualization block instance.

  • Type “devmem” to display the busybox devmem utility info

    # devmem
BusyBox v1.33.0 (2021-08-10 13:24:14 IST) multi-call binary.

Usage: devmem ADDRESS [WIDTH [VALUE]]

Read/write from physical address

        ADDRESS Address to act upon
        WIDTH   Width (8/16/...)
        VALUE   Data to be written

  • Test SRAM memory write by using the following example.

    # devmem 0x1080010000 32 0xabcd1234

  • Similarly, test SRAM memory read by using the following example.

    # devmem 0x1080010000 32
0xABCD1234

This completes the testing for non-PCI devices connected to the I/O virtualization block.