.. _NonPCIe_IO_virtualization_label: Using non-discoverable devices connected to I/O virtualization block ==================================================================== .. important:: This feature might not be applicable to all Platforms. Please check individual Platform pages, section **Supported Features** to confirm if this feature is listed as supported. Overview -------- The reference design platforms that support a IO Virtualization block as part of the compute subsystem allow connecting non-discoverable devices (non-PCIe) to it. The I/O virtualization block includes SMMUv3 to translate address and provide device isolation security, GIC-ITS to support MSI interrupts and NI-700 inter-connect to route transactions in and out of x16, x8, x4_1, and x4_0 ports. The non-discoverable devices that are connected to the Io Virtualization block include - two PL011 UART, two PL330 DMA controllers and six regions of SRAM memory. The reference design software stack includes support to configure and use these devices and memory regions. This document describes how to build the Neoverse reference design platform software stack and use it to test non-PCI devices that are connected on I/O virtualization blocks. :ref:`Busybox Boot ` is used on the Neoverse RD FVP and tests are run from the command line to validate the devices. NOTE: These tests are supported only on reference design platforms that designate one of the IO virtualization block for connecting non-discoverable devices. Build the platform software --------------------------- .. note:: This section assumes the user has completed the chapter :doc:`Getting Started ` and has a functional working environment. Refer to the :ref:`Busybox Boot ` page to build the reference design platform software stack and boot into busybox on the Neoverse RD FVP. Running tests for non-PCI devices on busybox -------------------------------------------- To begin the tests with the non-discoverable devices connected to the IO Virtualization block, boot to busybox using the command mentioned below (refer to :ref:`Busybox Boot ` guide for details on the parameters). :: ./boot.sh -p -a -n [true|false] PL011 UART ********** There are two `PL011 UART`_ controllers connected to the non-discoverable IO Virtualization block. These UART controllers are initialized by edk2 firmware before booting to the Linux kernel. These UART peripherals are enumerated by the Linux Kernel as Serial ports and can be tested by writing to the corresponding tty device. - After booting into busybox, verify that the two PL011 UART controllers are enumerated. The command provided below will list the serial ports detected by the Linux kernel. :: # dmesg | grep tty [ 0.034995] ARMH0011:00: ttyAMA0 at MMIO 0x1080000000 (irq = 14, base_baud = 0) is a SBSA [ 0.035195] ARMH0011:01: ttyAMA1 at MMIO 0x10a0000000 (irq = 15, base_baud = 0) is a SBSA [ 0.035595] ARMH0011:02: ttyAMA2 at MMIO 0xef70000 (irq = 36, base_baud = 0) is a SBSA [ 0.037095] printk: console [ttyAMA2] enabled - Here, ttyAMA0 and ttyAMA1 are the PL011 UART peripherals that are connected to the I/O virtualization block. - Now test the PL011 UART peripherals by writing to the corresponding tty device files using echo command: :: # echo "test message 0" > /dev/ttyAMA0 # echo "test message 1" > /dev/ttyAMA1 - The above commands print the message on the **FVP iomacro_terminal_0** and **FVP iomacro_terminal_1** terminals respectively. PL330 DMA ********** There are two `PL330 DMA`_ controllers connected to the non-discoverable IO Virtualization block. Each of these controllers support 8 data channels and one instruction channel. To test these dma controllers, `DMA test guide`_ included in the Linux kernel documentation has to be followed. As mentioned in the guide, CONFIG_DMATEST has to be enabled in the Linux kernel. - After booting into busybox validate that the DMA PL330 controllers are probed fine and showing 8 channels on each dma controllers - dma0 and dma1. :: # ls /sys/class/dma dma0chan0 dma0chan3 dma0chan6 dma1chan1 dma1chan4 dma1chan7 dma0chan1 dma0chan4 dma0chan7 dma1chan2 dma1chan5 dma0chan2 dma0chan5 dma1chan0 dma1chan3 dma1chan6 - Also verify that the two dma controllers are attached to SMMUv3 of I/O Virtualization block. An example of this shown below. :: # ls /sys/class/iommu/smmu3.0x0000000048000000/devices ARMH0330:00 ARMH0330:01 - Following the `DMA test guide`_, set the timeout and number of iterations. For example, :: # echo 2000 > /sys/module/dmatest/parameters/timeout # echo 1 > /sys/module/dmatest/parameters/iterations - Start the test for different channels. For example, to run on dma0chan0 use the following command: :: # echo dma0chan0 > /sys/module/dmatest/parameters/channel # echo 1 > /sys/module/dmatest/parameters/run - One can use loops to run the tests on all channels, for example for dma0: :: # for ch in 0 1 2 3 4 5 6 7; do echo dma0chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done - Similarly, for other controller - dma1: :: # for ch in 0 1 2 3 4 5 6 7; do echo dma1chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done - Test result: Test results are printed to the kernel log buffer with the format: :: "dmatest: result : : '' with src_off= dst_off= len= ()" - Below example if from running the test on all channels of dma0. :: # for ch in 0 1 2 3 4 5 6 7; do echo dma0chan$ch > /sys/module/dmatest/parameters/channel; echo 1 > /sys/module/dmatest/parameters/run; sleep 1; done [ 1375.118694] dmatest: Added 1 threads using dma0chan0 [ 1375.118694] dmatest: Started 1 threads using dma0chan0 [ 1375.123195] dmatest: dma0chan0-copy0: summary 1 tests, 0 failures 235.45 iops 3060 KB/s (0) [ 1376.119025] dmatest: Added 1 threads using dma0chan1 [ 1376.119025] dmatest: Started 1 threads using dma0chan1 [ 1376.120894] dmatest: dma0chan1-copy0: summary 1 tests, 0 failures 615.76 iops 3078 KB/s (0) [ 1377.119311] dmatest: Added 1 threads using dma0chan2 [ 1377.119311] dmatest: Started 1 threads using dma0chan2 [ 1377.123594] dmatest: dma0chan2-copy0: summary 1 tests, 0 failures 246.24 iops 2954 KB/s (0) ... and so on SRAM Memory *********** There are six SRAM memory regions connected to the non-discoverable IO Virtualization block. Out of the six, two SRAM memory are connected to the the high bandwidth port of the I/O virtualization block and the remaining 4 are connected to the low bandwidth port. The size of each SRAM memory connected to the I/O virtualization block is 4MiB. The memory mapping for all the SRAM memory are listed in the below table: SRAM that are connected to high bandwidth port: +----------+----------------+----------------+------+ | Mem Name | Start Address | End Address | Size | +==========+================+================+======+ | MEM0 | 0x10_8001_0000 | 0x10_8001_FFFF | 4MiB | +----------+----------------+----------------+------+ | MEM1 | 0x10_B002_0000 | 0x10_B002_FFFF | 4MiB | +----------+----------------+----------------+------+ SRAM that are connected to low bandwidth port: :: iomacro_size = 0x2000000 iomacro_instance : RD-N2-Cfg1 = 1 RD-N2 = 4 +----------+-----------------------------------------------------+-----------------------------------------------------+------+ | Mem Name | Start Address | End Address | Size | +==========+=====================================================+=====================================================+======+ | MEM2 | 0x4100_0000 + (`iomacro_instance` * `iomacro_size`) | 0x413F_FFFF + (`iomacro_instance` * `iomacro_size`) | 4MiB | +----------+-----------------------------------------------------+-----------------------------------------------------+------+ | MEM3 | 0x4140_0000 + (`iomacro_instance` * `iomacro_size`) | 0x417F_FFFF + (`iomacro_instance` * `iomacro_size`) | 4MiB | +----------+-----------------------------------------------------+-----------------------------------------------------+------+ | MEM4 | 0x4180_0000 + (`iomacro_instance` * `iomacro_size`) | 0x41BF_FFFF + (`iomacro_instance` * `iomacro_size`) | 4MiB | +----------+-----------------------------------------------------+-----------------------------------------------------+------+ | MEM5 | 0x41C0_0000 + (`iomacro_instance` * `iomacro_size`) | 0x41FF_FFFF + (`iomacro_instance` * `iomacro_size`) | 4MiB | +----------+-----------------------------------------------------+-----------------------------------------------------+------+ - The SRAM memory can be tested by using **devmem** busybox utility which can be used to read and write to physical memory using **/dev/mem** provided that the Linux Kernel is built with Kernel config option **CONFIG_DEVMEM=y** - After booting into busybox, use the following example to test the SRAM memory connected to the non-discoverable I/O virtualization block instance. - Type "devmem" to display the busybox devmem utility info :: # devmem BusyBox v1.33.0 (2021-08-10 13:24:14 IST) multi-call binary. Usage: devmem ADDRESS [WIDTH [VALUE]] Read/write from physical address ADDRESS Address to act upon WIDTH Width (8/16/...) VALUE Data to be written - Test SRAM memory write by using the following example. :: # devmem 0x1080010000 32 0xabcd1234 - Similarly, test SRAM memory read by using the following example. :: # devmem 0x1080010000 32 0xABCD1234 This completes the testing for non-PCI devices connected to the I/O virtualization block. .. _PL330 DMA: https://developer.arm.com/documentation/ddi0424/d/ .. _PL011 UART: https://developer.arm.com/documentation/ddi0183/g/ .. _DMA test guide: https://www.kernel.org/doc/html/latest/driver-api/dmaengine/dmatest.html#dma-test-guide