:orphan:

.. _cmn_cyprus:

************************
CMN Cyprus Driver Module
************************

Introduction
============

The CMN-Cyprus is a scalable configurable coherent interconnect that is designed
to meet the Power, Performance, and Area (PPA) requirements for Coherent Mesh
Network systems that are used in high-end networking and enterprise compute
applications. This document gives an overview of the cmn cyprus driver module in
the SCP firmware.

.. note::

   The Super Home Node (HN-S) introduced in the CMN Cyprus is responsible for
   managing part of the local address space and managing local coherency for 
   part of the remote address space. However, any descriptions related to the
   Fully coherent Home Node (HN-F) in this document also apply to the HN-S
   node.

Overview
========

The cmn cyprus driver module in the SCP firmware discovers the CMN interconnect
present in the system and configures the CMN during the boot-time based on the
address map info provided from the module configuration data.

The cmn cyprus module config data for RD-V3 platform can be found here:
``<workspace>/scp/product/neoverse-rd/rdv3/scp_ramfw/config_cmn_cyprus.c``

The cmn cyprus driver module code can found in the following path:
``<workspace>/scp/module/cmn_cyprus/``

.. note::

   The CMN Cyprus configuration register space located in the AP memory map is
   accessible through Root access only. Hence, the Address Translation Unit in
   the RD-V3 platform variants is configured in such a way that the SCP
   can access the CMN Configuration space with root permissions. Please refer
   to the :ref:`SCP ATU Configuration <scp_atu_config>` document for more
   information.

HN-S Isolation
==============

MXP Device Isolation feature in the CMN Cyprus allows HN-F or HN-S devices
attached to a CAL2 device ports to be logically isolated from the corresponding
MXP and mesh. If the isolation parameter is enabled in the mesh, then, by
default, all the CAL2 connected HN-S devices are in isolated state.

.. important::
   
   Isolated HN-S nodes must not accessed before disabling the isolation.

The MXP ``mxp_device_port_disable`` register of the corresponding MXP must be
configured in order to bring the HN-S nodes out of isolation before accessing
the HN-S nodes. The CMN Cyprus driver module config data has provision for
passing a list of isolated HN-S node addresses and the coordinates. This
information is used by the driver during the discovery to skip the isolated
HN-S nodes (if any) and enable all the other MXP device ports before
discovering the individual nodes connected to the cross-point.

Please refer ``mxp_enable_device()`` function in the ``<workspace>/scp/module/
cmn_cyprus/src/cmn_cyprus_mxp_reg.c`` file for more information.

Sequence
========

The CMN Cyprus configuration space is located in the SYSTOP power domain which
requires the SYSTOP power domain to be turned on and clock to be configured
before the CMN CFGM space can be accessed. Hence, the CMN Cyprus driver starts
initialization after the clock and the power domain modules.

The ``cmn_cyprus_setup()`` function is invoked during the start phase. This
function initiates the CMN boot-time programming sequence.

Note: In Multichip platforms, the CMN Cyprus driver in each chip's SCP firmware
programs the local CMN mesh, CCG blocks and enables the CML links as described
in the following sections.

Discovery
=========

1. During the module start phase, the ``cmn_cyprus_setup()`` function is
   invoked, which invokes ``cmn_cyprus_discovery()`` function to initiate the
   discovery.

2. The ``cmn_cyprus_discovery()`` function invokes the ``discover_mesh_topology
   ()`` function which starts from the Rootnode base and traverses through the
   mesh to discover the interconnect topology and total number of each type of
   node present in the mesh.

   During the discovery, ``disable_hns_isolation()`` function is used to
   disable mxp device isolation for non-isolated nodes attached in the MXP
   before traversing the MXP. In addition to this, before reading the nodes
   info register, each node address is compared with the list of isolated HN-S
   nodes using the ``is_node_base_isolated()`` function and if a match is
   found, the corresponding child pointer of the cross-point is skipped.

3. After the discovery, the ``cmn_cyprus_discovery()`` allocates the space for
   the internal descriptors and pointers and invokes ``cmn_cyprus_init_ctx()``
   function.

4. In the ``cmn_cyprus_init_ctx()`` function, the driver traverses through the
   mesh starting from the Rootnode base and each RN-SAM, HN-F node, CCG node
   info is stored in the module context data. This completes the CMN discovery
   sequence.

Please refer the following for more details:
``<workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_discovery_setup.c``

HN-F SAM Programming
====================

After the discovery, ``cmn_cyprus_setup()`` invokes the function
``cmn_cyprus_setup_hnf_sam()`` to program the HN-F SAM.

1. This function checks the ``hnf_sam_mode`` in the ``hnf_sam_config`` and if
   Direct SN mapping mode is configured, then the function
   ``program_hnf_sam_direct_mapping()`` is invoked, which programs the SN-F
   node ID provided in the config data at the SN0 index.

2. If range based hashed SN mapping mode is enabled, then the function
   ``program_hnf_sam_range_based_hashing()`` is invoked where the target SN
   node IDs are configured along with the top address bit positions and finally
   the SN mode is enabled. At the moment, only 3-SN mode is supported.

3. Next, ``program_syscache_sub_regions()`` is invoked which iterates through
   each region in the memory map and for each system cache group sub-region i.e
   regions with the type configured as ``MOD_CMN_CYPRUS_MEM_REGION_TYPE_
   SYSCACHE_SUB`` (region serviced by dedicated SBSX node for non-hashed
   access) the address range is configured in the HN-F SAM in the ``configure_
   non_hashed_region()`` function.

4. Finally, the HN-F node's PPU (Power Policy unit) register is configured.

Please refer ``<workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_hnsam_setup.c``
file for more details.

LCN SAM Programming
===================

For multichip platforms that enable Local Coherency Node feature, the function
``cmn_cyprus_setup_lcn_sam()`` is invoked to program the remote hashed regions
in the LCN SAM. The address range, target type, CAL mode and CPA are
programmed.

RNSAM Programming
=================

Following the HN-F SAM programming, ``cmn_cyprus_setup()`` function invokes
``cmn_cyprus_setup_rnsam()`` to program the RN SAM.

1. ``stall_rnsam_requests()`` function is invoked to stall the RNSAM requests
   initially.

2. For each region provided in the config mmap table, ``program_rnsam_region()``
   is invoked, which checks the memory region type and invokes the function
   ``program_io_region()`` if the region is an I/O region or
   ``program_scg_region()`` function if the region is a System cache backed
   region.

Non-Hashed regions
------------------

Non-hashed memory regions are used when a memory partition has to be assigned to
an individual target node. I/O space is the intended target of these regions.

1. The region type is ``MOD_CMN_CYPRUS_MEM_REGION_TYPE_IO`` and the function
   ``rnsam_configure_non_hashed_region()`` is invoked to configure the address
   range and the target node type.

2. The address range and the target node type is configured in the following
   RNSAM registers when range comparison mode is enabled.

   - ``non_hash_mem_region_reg``
   - ``non_hash_mem_region_cfg2_reg``

3. After that, the target node id is configured in the register
   ``non_hash_tgt_nodeid`` in ``rnsam_set_non_hashed_region_target()`` function
   and the region is marked as valid for comparision in the function
   ``rnsam_set_non_hash_region_valid()``.

Hashed regions
--------------

Hashed memory regions are used when a memory partition is distributed across
many target nodes. These regions are typically used to support DRAM space to
be hashed across multiple HN-F nodes.

1. The region type is ``MOD_CMN_CYPRUS_MEM_REGION_TYPE_SYSCACHE`` and the
   function ``rnsam_configure_hashed_region()`` is invoked to configure the
   address range and the target node type.

2. The address range and the target node type is configured in the following
   RNSAM registers when range comparision mode is enabled:

   - ``sys_cache_grp_region``
   - ``hashed_tgt_grp_cfg2_region``

3. Optionally, the secondary address range is also configured in the following
   RNSAM registers when range comparision mode is enabled:

   - ``sys_cache_grp_secondary_region``
   - ``hashed_tgt_grp_secondary_cfg2_region``

4. Next, the CAL2 (Component Aggregation Layer) mode is configured if specified
   in the config data.

5. Then, if hierarchical hashing mode is specified in the config data, the
   function ``configure_scg_hier_hashing()`` is invoked, which configures the
   number of clusters in the first level hierarchy, number of HN-S nodes per
   cluster, number of address bits to be removed at second level and enables the
   hierarchical hashing mode using RNSAM utility functions. The following
   registers are configured:

   - ``hashed_target_grp_hash_cntl_reg``
   - ``sys_cache_grp_sn_attr``
   - ``sys_cache_grp_sn_sam_cfg``

6. After this, the hashed region is marked as valid for comparision.

7. Once the hashed memory region is programmed, the function
   ``configure_scg_target_nodes()`` programs the target HN-S node IDs that fall
   within the SCG region boundary specified in the config data.

   Note: An SCG is a group of HN-Fs that share a contiguous address region.
   However, the addresses that are covered by each HN-F in an SCG are mutually
   exclusive. An HN-F belonging to an SCG is selected as the target based on a
   hash function.

   Typically, these HN-F nodes are bound by an arbitrary rectangle in the mesh
   and the module config data specifies the start and the end HN-F node
   positions of the SCG/HTG. If the node falls within the SCG, then the node
   id is programmed in the following RN-SAM registers:

   - ``sys_cache_grp_hn_nodeid_reg``

8. Then, ``sys_cache_group_hn_count`` register is programmed with the number of
   HN-Ss in the hashed target group.

9. For multichip platforms, the remote chip memory regions and the CML port
   aggregation mode is also configured by the function
   ``program_rnsam_remote_regions()``.

10. Finally, the RNSAM is enabled by configuring the ``rnsam_status`` register,
    where the requests are unstalled and default target id selection is disabled
    in the ``rnsam_unstall()`` function.

Please refer ``<workspace>/scp/module/cmn_cyprus/src/cmn_cyprus_rnsam_setup.c``
for more details.

CML Programming
===============

For multichip platforms, the function ``cmn_cyprus_setup()`` invokes
``cmn_cyprus_setup_cml()`` to program the CML links and enable multichip
communication.

The multichip connection is setup in the following sequence:

1. The remote memory regions are programmed in the RA SAM. The function
   ``program_ra_sam()`` is invoked to configure the remote address range, target
   HAID and set the RA SAM region as valid. 

2. Assign LinkIDs to remote agents. ``program_agentid_to_linkid_lut()``
   function programs the LinkID in the AgentID-to-LinkID register. The default
   value points to Link 0. Hence, the driver skips the configuration and sets
   the linkid programming as valid.

3. Program the Home Agent ID (HAID) in the CCG HA node using the function
   ``ccg_ha_configure_haid()`` function.

4. Assign expanded RAIDs to local request agents. The function
   ``program_ccg_ra_ldid_to_raid_lut()`` programs unique RAID values for the
   local RN-Fs, RN-Is, RN-Ds and optionally LCN in the respective LUT registers.

5. Assign LDIDs to remote caching agents. The function
   ``program_ccg_ha_raid_to_ldid_lut`` programs LDID values for each remote
   caching agent in the CCG HA RAID-to-LDID LUT registers.

6. Program CCG HA node IDs at the LDID index for remote caching agents in the
   HN-S nodes using the function ``program_hns_ldid_to_rn_nodeid()``. The
   request nodes are set as remote and based on the config data, CPA is also
   configured.

7. Then, if the SMP mode is specified in the config data, the functions
   ``ccg_ra_set_cml_link_ctl()`` and ``ccg_ha_set_cml_link_ctl()`` are invoked
   to configure the SMP mode in CCG RA and CCG HA registers respectively.

8. If direct connect mode (ULL-to-ULL) mode is required, the function
   ``enable_ccla_direct_connect_mode()`` configures the CML direct connect mode.

9. The CML link is enabled, the link status is then verified and the link is
   brought up using the function ``setup_cml_link()``.

10. Then, ``cml_exchange_protocol_credit()`` function is used to exchange the
    protocol credits between the CML links.

11. Finally the ``cml_enter_coherency_domain()`` function is used to enable the
    CML links to enter snoop coherency domain and DVM domain.