CMN-Cyprus Multichip Configuration

Introduction

This document gives an overview of the CMN-Cyprus multichip configuration in the RD-Fremont-Cfg2 platform.

Overview

The following diagram shows the cross-chip CCG connections in the the RD-Fremont-Cfg2 platform:

+----------------------------------+             +-------------------------------+
|                                  |             |                               |
|                              CCG6+-------------+CCG6                           |
|                                  |             |                           CCG4|
|CCG4                          CCG7+-------------+CCG7                           |
|             CHIP 0               |             |             CHIP 2            |
|                              CCG8+----+  +-----+CCG8                       CCG5|
|CCG5                              |    |  |     |                               |
|                              CCG9+-+  |  |  +--+CCG9                           |
|                                  | |  |  |  |  |                               |
|   CCG0    CCG1    CCG2    CCG3   | |  |  |  |  |   CCG0   CCG1   CCG2   CCG3   |
+----+-------+-------+-------+-----+ |  |  |  |  +-----+------+------+------+----+
     |       |       |       |       |  |  |  |        |      |      |      |
     |       |       |       |       |  |  |  |        |      |      |      |
+----+-------+-------+-------+-----+ |  |  |  |  +-----+------+------+------+----+
|   CCG3    CCG2    CCG1    CCG0   | |  |  |  |  |    CCG3  CCG2   CCG1    CCG0  |
|                                  | |  |  |  |  |                               |
|                              CCG4+-|--|--+  |  |                               |
|                                  | |  |     |  |                               |
|                              CCG5+-|--|-----+  |                               |
|                                  | |  |        |                               |
|CCG8                              | |  +--------+CCG4                       CCG8|
|             CHIP 1               | |           |             CHIP 3            |
|                                  | +-----------+CCG5                           |
|CCG9                              |             |                           CCG9|
|                              CCG6+-------------+CCG6                           |
|                                  |             |                               |
|                              CCG7+-------------+CCG7                           |
|                                  |             |                               |
+----------------------------------+             +-------------------------------+

CMN-Cyprus Driver config data for multichip

Please refer the following file in the workspace to get the complete info on the configuration data passed to the CMN-Cyprus driver module in SCP firmware:

<workspace>/scp/product/neoverse-rd/rdfremont/scp_ramfw/config_cmn_cyprus.c

The CML configuration structures can be found in the following file: <workspace>/scp/module/cmn_cyprus/include/mod_cmn_cyprus.h

The mod_cmn_cyprus_cml_config structure is used to describe the CCG blocks. Each chip in the RD-Fremont-Cfg2 platform is connected to the other chips via CCG blocks. This info is described using the following tables in the config data:

cml_config_table_chip_0: This table describes the CCG blocks that connect the Chip 0 to Chip 1, Chip 2 and Chip 3 respectively.
cml_config_table_chip_1: This table describes the CCG blocks that connect the Chip 1 to Chip 0, Chip 2 and Chip 3 respectively.
cml_config_table_chip_2: This table describes the CCG blocks that connect the Chip 2 to Chip 0, Chip 1 and Chip 3 respectively.
cml_config_table_chip_3: This table describes the CCG blocks that connect the Chip 3 to Chip 0, Chip 1 and Chip 2 respectively.

The following section explains how the Chip 0 CCG blocks are described in the cml_config_table_chip_0 table. Please note that LCN is enabled along with CPAG and two CCG nodes per CPAG group is considered in this config data.

Configure the logical IDs for the CCGs. The logical ID ranges from 0 to (total CCG block per chip - 1).
- Refer rdfremontcfg2_cmn_cyprus_ccg_port in config_cmn_cyprus.c file for the list of CCG logical IDs.
- For the CCG blocks that connect Chip 0 to Chip 1, CCG_PORT_0 and CCG_PORT_1 is used.
Configure HAIDs for the CCG blocks. This is a unique HAID across all the chips.
- The Chip ID is used to calculate a unique HAIDs for the CCG blocks. Hence, (CCG_PER_CHIP * PLATFORM_CHIP_0) + CCG_PORT_0 and (CCG_PER_CHIP * PLATFORM_CHIP_0) + CCG_PORT_1 is used.
Configure the table of remote memory regions. As LCN is enabled, the remote memory regions must be specified as System Cache Group(SCG) regions. The remote DRAM regions which are non-contiguous are specified as secondary regions for the same SCG. Apart from these, the start and the end node positions for the remote SCG is also configured.
- Since the CCG0, CCG1 blocks in Chip 0 is connected to Chip 1, the address range of the Chip 1 is specified in this table: i.e., [0x1000000000 - 0x1fffffffff] and [0x100000000000 - 0x1fffffffffff]
- Specify the unique HAIDs of the CCG3 and CCG2 blocks in Chip 1 as the target HAIDs.
Configure the Remote Chip ID. This is used to setup the AgentID (RAID and HAID) to LinkID LUT.
Set enable_smp_mode to true as the CCG block must be programmed for multichip SMP communications.
Set enable_direct_connect_mode to true in order to enable the upper link layer to upper link layer connection between the CCLAs of two chips. This enables the connection of CXS interface from the CCLA on one CMN-Cyprus to the CXS interface of the other.
Set enable_cpa_mode to true in order to enable CCIX Port Aggregation Mode.
Set cpag_id to 0, as the CPA group IDs are assigned sequentially and ccg_count to 2 as we consider two CCG ports per CPA group.

Steps 1 - 8 are repeated for configuring the CPAG that connect Chip 0 to Chip 2 and Chip 3 respectively. Similarly, the CPA groups in Chip 1, 2 and 3 are specified in the respective CCG config tables. Also, the enable_lcn flag is set in each chip’s CMN config data to enable LCN programming.

The CMN-Cyprus driver in each SCP, discovers and configures the local CMN mesh and then programs the CCGs to enable multichip SMP communication. For more info on the CMN-Cyprus driver programming, please refer CMN-Cyprus Driver.