CMN Cyprus Driver Module

Introduction

The CMN-Cyprus is a scalable configurable coherent interconnect that is designed to meet the Power, Performance, and Area (PPA) requirements for Coherent Mesh Network systems that are used in high-end networking and enterprise compute applications. This document gives an overview of the cmn cyprus driver module in the SCP firmware.

NOTE: The Super Home Node (HN-S) introduced in the CMN Cyprus is responsible for managing part of the local address space and managing local coherency for part of the remote address space. However, any descriptions related to the Fully coherent Home Node (HN-F) in this document also apply to the HN-S node.

Overview

The cmn cyprus driver module in the SCP firmware discovers the CMN interconnect present in the system and configures the CMN during the boot-time based on the address map info provided from the module configuration data.

The cmn cyprus module config data for RD-Fremont platform can be found here: <workspace>/scp/product/neoverse-rd/rdfremont/scp_ramfw/config_cmn_cyprus.c

The cmn cyprus driver module code can found in the following path: <workspace>/scp/module/cmn_cyprus/

NOTE: The CMN Cyprus configuration register space located in the AP memory map is accessible through Root access only. Hence, the Address Translation Unit in the RD-Fremont platform variants is configured in such a way that the SCP can access the CMN Configuration space with root permissions. Please refer to the SCP ATU Configuration document for more info on this.

HN-S Isolation

MXP Device Isolation feature in the CMN Cyprus allows HN-F or HN-S devices attached to a CAL2 device ports to be logically isolated from the corresponding MXP and mesh. If the isolation parameter is enabled in the mesh, then, by default, all the CAL2 connected HN-S devices are in isolated state.

Note: Isolated HN-S nodes must not accessed before disabling the isolation.

The MXP mxp_device_port_disable register of the corresponding MXP must be configured in order to bring the HN-S nodes out of isolation before accessing the HN-S nodes. The CMN Cyprus driver module config data has provision for passing a list of isolated HN-S node addresses and the coordinates. This info is used by the driver during the discovery to skip the isolated HN-S nodes(if any) and enable all the other MXP device ports before discovering the individual nodes connected to the cross-point.

Please refer mxp_enable_device() function in the <workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_mxp_reg.c file for more info.

Sequence

The CMN Cyprus configuration space is located in the SYSTOP power domain which requires the SYSTOP power domain to be turned on and clock to be configured before the CMN CFGM space can be accessed. Hence, the CMN Cyprus driver starts initialization after the clock and the power domain modules.

The cmn_cyprus_setup() function is invoked during the start phase. This function initiates the CMN boot-time programming sequence.

Note: In Multichip platforms, the CMN Cyprus driver in each chip’s SCP firmware programs the local CMN mesh, CCG blocks and enables the CML links as described in the following sections.

Discovery

During the module start phase, the cmn_cyprus_setup() function is invoked, which invokes cmn_cyprus_discovery() function to initiate the discovery.
The cmn_cyprus_discovery() function invokes the discover_mesh_topology() function which starts from the Rootnode base and traverses through the mesh to discover the interconnect topology and total number of each type of node present in the mesh.

During the discovery, disable_hns_isolation() function is used to disable mxp device isolation for non-isolated nodes attached in the MXP before traversing the MXP. In addition to this, before reading the nodes info register, each node address is compared with the list of isolated HN-S nodes using the is_node_base_isolated() function and if a match is found, the corresponding child pointer of the cross-point is skipped.

After the discovery, the cmn_cyprus_discovery() allocates the space for the internal descriptors and pointers and invokes cmn_cyprus_init_ctx() function.
In the cmn_cyprus_init_ctx() function, the driver traverses through the mesh starting from the Rootnode base and each RN-SAM, HN-F node, CCG node info is stored in the module context data. This completes the CMN discovery sequence.

Please refer the following for more details: <workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_discovery_setup.c

HN-F SAM Programming

After the discovery, cmn_cyprus_setup() invokes the function cmn_cyprus_setup_hnf_sam() to program the HN-F SAM.

This function checks the hnf_sam_mode in the hnf_sam_config and if Direct SN mapping mode is configured, then the function program_hnf_sam_direct_mapping() is invoked, which programs the SN-F node ID provided in the config data at the SN0 index.
If range based hashed SN mapping mode is enabled, then the function program_hnf_sam_range_based_hashing() is invoked where the target SN node IDs are configured along with the top address bit positions and finally the SN mode is enabled. At the moment, only 3-SN mode is supported.
Next, program_syscache_sub_regions() is invoked which iterates through each region in the memory map and for each system cache group sub-region i.e regions with the type configured as MOD_CMN_CYPRUS_MEM_REGION_TYPE_SYSCACHE_SUB (region serviced by dedicated SBSX node for non-hashed access) the address range is configured in the HN-F SAM in the configure_non_hashed_region() function.
Finally, the HN-F node’s PPU(Power Policy unit) register is configured.

Please refer <workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_hnsam_setup.c file for more details.

LCN SAM Programming

For multichip platforms that enable Local Coherency Node feature, the function cmn_cyprus_setup_lcn_sam() is invoked to program the remote hashed regions in the LCN SAM. The address range, target type, CAL mode and CPA are programmed.

RNSAM Programming

Following the HN-F SAM programming, cmn_cyprus_setup() function invokes cmn_cyprus_setup_rnsam() to program the RN SAM.

stall_rnsam_requests() function is invoked to stall the RNSAM requests initially.
For each region provided in the config mmap table, program_rnsam_region() is invoked, which checks the memory region type and invokes the function program_io_region() if the region is an I/O region or program_scg_region() function if the region is a System cache backed region.

Non-Hashed regions

Non-hashed memory regions are used when a memory partition has to be assigned to an individual target node. I/O space is the intended target of these regions.

The region type is MOD_CMN_CYPRUS_MEM_REGION_TYPE_IO and the function rnsam_configure_non_hashed_region() is invoked to configure the address range and the target node type.
The address range and the target node type is configured in the following RNSAM registers when range comparison mode is enabled.
- non_hash_mem_region_reg
- non_hash_mem_region_cfg2_reg
After that, the target node id is configured in the register non_hash_tgt_nodeid in rnsam_set_non_hashed_region_target() function and the region is marked as valid for comparision in the function rnsam_set_non_hash_region_valid().

Hashed regions

Hashed memory regions are used when a memory partition is distributed across many target nodes. These regions are typically used to support DRAM space to be hashed across multiple HN-F nodes.

The region type is MOD_CMN_CYPRUS_MEM_REGION_TYPE_SYSCACHE and the function rnsam_configure_hashed_region() is invoked to configure the address range and the target node type.
The address range and the target node type is configured in the following RNSAM registers when range comparision mode is enabled:
- sys_cache_grp_region
- hashed_tgt_grp_cfg2_region
Optionally, the secondary address range is also configured in the following RNSAM registers when range comparision mode is enabled:
- sys_cache_grp_secondary_region
- hashed_tgt_grp_secondary_cfg2_region
Next, the CAL2 (Component Aggregation Layer) mode is configured if specified in the config data.
Then, if hierarchical hashing mode is specified in the config data, the function configure_scg_hier_hashing() is invoked, which configures the number of clusters in the first level hierarchy, number of HN-S nodes per cluster, number of address bits to be removed at second level and enables the hierarchical hashing mode using RNSAM utility functions.
- The following registers are configured:
  - hashed_target_grp_hash_cntl_reg
  - sys_cache_grp_sn_attr
  - sys_cache_grp_sn_sam_cfg
After this, the hashed region is marked as valid for comparision.
Once the hashed memory region is programmed, the function configure_scg_target_nodes() programs the target HN-S node IDs that fall within the SCG region boundary specified in the config data.

Note: An SCG is a group of HN-Fs that share a contiguous address region. However, the addresses that are covered by each HN-F in an SCG are mutually exclusive. An HN-F belonging to an SCG is selected as the target based on a hash function.
- Typically, these HN-F nodes are bound by an arbitrary rectangle in the mesh and the module config data specifies the start and the end HN-F node positions of the SCG/HTG. If the node falls within the SCG, then the node id is programmed in the following RN-SAM registers:
  - sys_cache_grp_hn_nodeid_reg
Then, sys_cache_group_hn_count register is programmed with the number of HN-Ss in the hashed target group.
For multichip platforms, the remote chip memory regions and the CML port aggregation mode is also configured by the function program_rnsam_remote_regions().
Finally, the RNSAM is enabled by configuring the rnsam_status register, where the requests are unstalled and default target id selection is disabled in the rnsam_unstall() function.

Please refer <workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_rnsam_setup.c for more details.

CML Programming

For multichip platforms, the function cmn_cyprus_setup() invokes cmn_cyprus_setup_cml() to program the CML links and enable multichip communication.

The multichip connection is setup in the following sequence:

The remote memory regions are programmed in the RA SAM. The function program_ra_sam() is invoked to configure the remote address range, target HAID and set the RA SAM region as valid.
Assign LinkIDs to remote agents. program_agentid_to_linkid_lut() function programs the LinkID in the AgentID-to-LinkID register. The default value points to Link 0. Hence, the driver skips the configuration and sets the linkid programming as valid.
Program the Home Agent ID (HAID) in the CCG HA node using the function ccg_ha_configure_haid() function.
Assign expanded RAIDs to local request agents. The function program_ccg_ra_ldid_to_raid_lut() programs unique RAID values for the local RN-Fs, RN-Is, RN-Ds and optionally LCN in the respective LUT registers.
Assign LDIDs to remote caching agents. The function program_ccg_ha_raid_to_ldid_lut programs LDID values for each remote caching agent in the CCG HA RAID-to-LDID LUT registers.
Program CCG HA node IDs at the LDID index for remote caching agents in the HN-S nodes using the function program_hns_ldid_to_rn_nodeid(). The request nodes are set as remote and based on the config data, CPA is also configured.
Then, if the SMP mode is specified in the config data, the functions ccg_ra_set_cml_link_ctl() and ccg_ha_set_cml_link_ctl() are invoked to configure the SMP mode in CCG RA and CCG HA registers respectively.
If direct connect mode (ULL-to-ULL) mode is required, the function enable_ccla_direct_connect_mode() configures the CML direct connect mode.
The CML link is enabled, the link status is then verified and the link is brought up using the function setup_cml_link().
Then, cml_exchange_protocol_credit() function is used to exchange the protocol credits between the CML links.
Finally the cml_enter_coherency_domain() function is used to enable the CML links to enter snoop coherency domain and DVM domain.