From ba735d70e10ff12a4825a60335e44dde5f55aa78 Mon Sep 17 00:00:00 2001 From: Ved Shanbhogue Date: Sat, 13 Jul 2024 16:01:12 -0500 Subject: [PATCH] Include QoS ID standard extension --- src/iommu.bib | 40 +++++++++++++++ src/iommu_data_structures.adoc | 22 +++++++- src/iommu_extensions.adoc | 92 +++++++++++++++++++++++++++++++++- src/iommu_registers.adoc | 11 ++-- 4 files changed, 159 insertions(+), 6 deletions(-) diff --git a/src/iommu.bib b/src/iommu.bib index 49ef1d72..291ad131 100644 --- a/src/iommu.bib +++ b/src/iommu.bib @@ -25,3 +25,43 @@ @electronic{PR243 title = {Clarification updates to IOMMU v1.0.0}, url = {https://github.com/riscv-non-isa/riscv-iommu/pull/243/commits} } +@electronic{CBQRI, + title = {RISC-V Capacity and Bandwidth QoS Register Interface}, + url = {https://github.com/riscv-non-isa/riscv-cbqri} +} +@article{PTCAMP, + author = {Du Bois, Kristof and Eyerman, Stijn and Eeckhout, Lieven}, + title = {Per-Thread Cycle Accounting in Multicore Processors}, + year = {2013}, + issue_date = {January 2013}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + volume = {9}, + number = {4}, + issn = {1544-3566}, + url = {https://doi.org/10.1145/2400682.2400688}, + doi = {10.1145/2400682.2400688}, + journal = {ACM Trans. Archit. Code Optim.}, + month = {jan}, + articleno = {29}, + numpages = {22}, +} +@inproceedings{HERACLES, + author = {Lo, David and Cheng, Liqun and Govindaraju, Rama and Ranganathan, Parthasarathy and Kozyrakis, Christos}, + title = {Heracles: Improving Resource Efficiency at Scale}, + year = {2015}, + isbn = {9781450334020}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/2749469.2749475}, + doi = {10.1145/2749469.2749475}, + booktitle = {Proceedings of the 42nd Annual International Symposium on Computer Architecture}, + pages = {450–462}, + numpages = {13}, + location = {Portland, Oregon}, + series = {ISCA '15} +} +@electronic{SSQOSID, + title = {RISC-V Quality-of-Service (QoS) Identifiers}, + url = {https://github.com/riscv/riscv-ssqosid} +} diff --git a/src/iommu_data_structures.adoc b/src/iommu_data_structures.adoc index 2978f94c..9689003b 100644 --- a/src/iommu_data_structures.adoc +++ b/src/iommu_data_structures.adoc @@ -484,6 +484,7 @@ the PTEs from the first page table or the second page table. These are the only expected behaviors. ==== +[[DC_TA]] ===== Translation attributes (`ta`) .Translation attributes (`ta`) field @@ -492,7 +493,9 @@ expected behaviors. {reg: [ {bits: 12, name: 'reserved'}, {bits: 20, name: 'PSCID'}, - {bits: 32, name: 'reserved'}, + {bits: 8, name: 'reserved'}, + {bits: 12, name: 'RCID'}, + {bits: 12, name: 'MCID'}, ], config:{lanes: 2, hspace: 1024, fontsize: 16}} .... @@ -502,6 +505,21 @@ fences on a per-address-space basis. The `PSCID` field in `ta` is used as the address-space ID if `DC.tc.PDTV` is 0 and the `iosatp.MODE` field is not `Bare`. When `DC.tc.PDTV` is 1, the `PSCID` field in `ta` is ignored. +The `RCID` and `MCID` fields are added by the QoS ID extension. If +`capabilities.QOSID` is 0, these bits are reserved and must be set to 0. +IOMMU-initiated requests for accessing the following data structures use the +value configured in the `RCID` and `MCID` fields of `DC.ta`. + +* Process directory table (`PDT`) +* Second-stage page table +* First-stage page table +* MSI page table +* Memory-resident interrupt file (`MRIF`) + +The `RCID` and `MCID` configured in `DC.ta` are provided to the IO bridge on +successful address translations. The IO bridge should associate these QoS IDs +with device-initiated requests. + ===== First-Stage context (`fsc`) If `DC.tc.PDTV` is 0, the `DC.fsc` field holds the `iosatp` that provides the controls for first-stage address translation and protection. @@ -716,6 +734,8 @@ misconfigured" (cause = 259). . `DC.tc.SBE` value is not a legal value. If `fctl.BE` is writable then `DC.tc.SBE` may be 0 or 1. If `fctl.BE` is not writable then `DC.tc.SBE` must be the same as `fctl.BE`. +. `capabilities.QOSID` is 1 and `DC.ta.RCID` or `DC.ta.MCID` values + are wider than that supported by the IOMMU. [NOTE] ==== diff --git a/src/iommu_extensions.adoc b/src/iommu_extensions.adoc index 5332a466..301d3b84 100644 --- a/src/iommu_extensions.adoc +++ b/src/iommu_extensions.adoc @@ -2,4 +2,94 @@ == IOMMU Extensions -This chapter specifies the standard extensions to the IOMMU Base Architecture. +This chapter specifies the following standard extensions to the IOMMU Base +Architecture: + +[%autowidth,float="center",align="center",cols="^,^,^",options="header",] +|=== +| Specification |Version |Status +| <> + |*1.0* + |*Ratified* +|=== + +[[QOSID]] +=== Quality-of-Service (QoS) Identifiers Extension, Version 1.0 + +Quality of Service (QoS) is defined as the minimal end-to-end performance +guaranteed in advance by a service level agreement (SLA) to a workload. +Performance metrics might include measures such as instructions per cycle (IPC), +latency of service, etc. + +When multiple workloads execute concurrently on modern processors -- equipped +with large core counts, multiple cache hierarchies, and multiple memory +controllers -- the performance of any given workload becomes less +deterministic, or even non-deterministic, due to shared resource contention +cite:[PTCAMP]. + +To manage performance variability, system software needs resource allocation +and monitoring capabilities. These capabilities allow for the reservation of +resources like cache and bandwidth, thus meeting individual performance targets +while minimizing interference cite:[HERACLES]. For resource management, hardware +should provide monitoring features that allow system software to profile +workload resource consumption and allocate resources accordingly. + +To facilitate this, the QoS Identifiers ISA extension (Ssqosid) cite:[SSQOSID] +introduces the `srmcfg` register, which configures a hart with two identifiers: +a Resource Control ID (`RCID`) and a Monitoring Counter ID (`MCID`). These +identifiers accompany each request issued by the hart to shared resource +controllers. + +These identifiers are crucial for the RISC-V Capacity and Bandwidth Controller +QoS Register Interface cite:[CBQRI], which provides methods for setting resource +usage limits and monitoring resource consumption. The `RCID` controls resource +allocations, while the `MCID` is used for tracking resource usage. + +The IOMMU QoS ID extension provides a method to associate QoS IDs with requests +to access resources by the IOMMU, as well as with devices governed by it. This +complements the Ssqosid extension that provides a method to associate QoS IDs +with requests originated by the RISC-V harts. Assocating QoS IDs with device +and IOMMU originated requests is required for effective monitoring and +allocation of shared resources. + +The IOMMU `capabilities` register (<>) is extended with a `QOSID` field +which enumerates support for associating QoS IDs with requests made through the +IOMMU. When `capabilities.QOSID` is 1, the memory-mapped register layout is +extended to add a register named `iommu_qosid` (<>). This register is +used to configure the Quality of Service (QoS) IDs associated with +IOMMU-originated requests. The `ta` field of the device context (<>) is +extended with two fields, `RCID` and `MCID`, to configure the QoS IDs to +associate with requests originated by the devices. + +==== Reset Behavior + +If the reset value for `ddtp.iommu_mode` field is `Bare`, then the +`iommu_qosid.RCID` field must have a reset value of 0. + +[NOTE] +==== +At reset, it is required that the `RCID` field of `iommu_qosid` is set to 0 if +the IOMMU is in `Bare` mode, as typically the resource controllers in the +SoC default to a reset behavior of associating all capacity or bandwidth to the +`RCID` value of 0. When the reset value of the `ddtp.iommu_mode` is not `Bare`, +the `iommu_qosid` register should be initialized by software before changing +the mode to allow DMA. +==== + +==== Sizing QoS Identifiers + +The size (or width) of `RCID` and `MCID`, as fields in registers or in data +structures, supported by the IOMMU must be at least as large as that supported +by any RISC-V application processor hart in the system. + +==== IOMMU ATC Capacity Allocation and Monitoring + +Some IOMMUs might support capacity allocation and usage monitoring in the IOMMU +address translation cache (IOATC) by implementing the capacity controller +register interface. + +Additionally, some IOMMUs might support multiple IOATCs, each potentially having +different capacities. In scenarios where multiple IOATCs are implemented, such +as an IOATC for each supported page size, the IOMMU can implement a capacity +controller register interface for each IOATC to facilitate individual capacity +allocation. diff --git a/src/iommu_registers.adoc b/src/iommu_registers.adoc index 0b643ee6..986c6406 100644 --- a/src/iommu_registers.adoc +++ b/src/iommu_registers.adoc @@ -81,7 +81,8 @@ the register returns 0 and writes to that offset are ignored. control>> | if `capabilities.DBG==0` |616 |`tr_response` |8 |<> | if `capabilities.DBG==0` -|624 |Reserved |64 |Reserved for future use +|624 |`iommu_qosid` |4 |<> | if `capabilities.QOSID==0` +|628 |Reserved |60 |Reserved for future use (`WPRI`) | |688 |custom |72 |Designated for custom use (`WARL`) | @@ -153,7 +154,8 @@ the IOMMU. At reset, the register shall contain the IOMMU supported features. {bits: 1, name: 'PD8'}, {bits: 1, name: 'PD17'}, {bits: 1, name: 'PD20'}, - {bits: 15, name: 'reserved'}, + {bits: 1, name: 'QOSID'}, + {bits: 14, name: 'reserved'}, {bits: 8, name: 'custom'}, ], config:{lanes: 8, hspace:1024}} .... @@ -223,7 +225,8 @@ the IOMMU. At reset, the register shall contain the IOMMU supported features. |38 |`PD8` |RO | One level PDT with 8-bit process_id supported. |39 |`PD17` |RO | Two level PDT with 17-bit process_id supported. |40 |`PD20` |RO | Three level PDT with 20-bit process_id supported. -|55:41 | reserved |RO | Reserved for standard use. +|41 |`QOSID` |RO | Associating QoS IDs with requests is supported. +|55:42 | reserved |RO | Reserved for standard use. |63:56 |custom |RO | Designated for custom use. |=== @@ -1647,7 +1650,7 @@ The `iommu_qosid` register fields are defined as follows: [width=100%] [%header, cols="^1,2,^1,5"] |=== -|Bits ^|Field |Attribute ^| Description +|Bits |Field |Attribute | Description |11:0 |`RCID` |WARL | `RCID` for IOMMU-initiated requests. |15:12 |reserved |WPRI | Reserved for standard use. |27:16 |`MCID` |WARL | `MCID` for IOMMU-initiated requests.