gem5+SST13 Changes #269

kaustav-goswami · 2023-11-17T00:55:04Z

This PR compares the disaggregated memory changes.

This change updates the gem5 SST Bridge to use SST 13.0.0. Changes are made to replace SimpleMem class to StandardMem class as SimpleMem will be deprecated in SST 14 and above. In addition, the translator.hh is updated to translate more types of gem5 packets. A new parameter `ports` was added on SST's side when invoking the gem5 component which does not require recompiling the gem5 component whenever a new outgoing bridge is added in a gem5 config. Change-Id: I45f0013bc35d088df0aa5a71951422cabab4d7f7 Signed-off-by: Kaustav Goswami <[email protected]>

This change adds necessasry tools to simulate multiple gem5 nodes to simulate a disaggregated memory setup using gem5 and SST. Change-Id: I6e1e52d4ba8df7c161b3151c9b2c02b72fc7cc31 Signed-off-by: Kaustav Goswami <[email protected]>

This change adds functional accesses to the gem5/SST bridge to boot disk images using the bridge. The change also adds more information in INSTALL.md for MacOS users on how to get the bridge setup working. Signed-off-by: Kaustav Goswami <[email protected]>

This change updates the gem5 SST Bridge to use SST 13.0.0. Changes are made to replace SimpleMem class to StandardMem class as SimpleMem will be deprecated in SST 14 and above. In addition, the translator.hh is updated to translate more types of gem5 packets. A new parameter `ports` was added on SST's side when invoking the gem5 component which does not require recompiling the gem5 component whenever a new outgoing bridge is added in a gem5 config. Change-Id: I45f0013bc35d088df0aa5a71951422cabab4d7f7 Signed-off-by: Kaustav Goswami <[email protected]>

This change adds necessasry tools to simulate multiple gem5 nodes to simulate a disaggregated memory setup using gem5 and SST. Change-Id: I6e1e52d4ba8df7c161b3151c9b2c02b72fc7cc31 Signed-off-by: Kaustav Goswami <[email protected]>

This change adds functional accesses to the gem5/SST bridge to boot disk images using the bridge. The change also adds more information in INSTALL.md for MacOS users on how to get the bridge setup working. Signed-off-by: Kaustav Goswami <[email protected]>

…kg/dm-numa-sst

powerjg

This is a very quick review... can you please clean up the commits before sending this out for review? There are changes in commits that should be in other commits, and please don't use "merge" and instead use rebase. When there are merge commits in a PR, I cannot tell what are changes that I should be reviewing vs changes that are actually already in upstream.

ext/sst/translator.hh

src/sst/outgoing_request_bridge.cc

powerjg · 2023-11-17T16:06:10Z

ext/sst/sst_responder_subcomponent.cc

+            break;
+        }
+        default:
+            panic("handleRecvFunctional: Unable to convert gem5 packet: %s\n", pkt->cmd.toString());


Line is >80 chars

should be fixed in the latest patchset

src/sst/outgoing_request_bridge.cc

ext/sst/sst_responder_subcomponent.cc

powerjg · 2023-11-17T16:13:52Z

ext/sst/Makefile

Why is this file deleted in this commit?

powerjg · 2023-11-17T16:15:09Z

ext/sst/INSTALL.md

+```sh
+# go to the base gem5 directory
+cd ../..
+export DYLD_LIBRARY_PATH=:`pwd`/build/RISCV/


Why specifically build/RISCV/?

This would be better if it was generic. E.g., export DYLD_LIBRARY_PATH=:$GEM5_BASE/<build directory>

This being said, this change has nothing to do with "added functional accesses to gem5/SST bridge" it should be in a different change.

we are supporting the RISCV version as the default one. We can definitely do this but we'd also need to change the README and the INSTALL files to make it more generic. I agree on making it more generic as the default examples we have are example.py -> RISCV/default and arm_example.py -> ARM.

powerjg · 2023-11-17T16:16:17Z

ext/sst/gem5.hh

-        {"system_port", "Connection to gem5 system_port", "gem5.gem5Bridge"},
-        {"cache_port", "Connection to gem5 CPU", "gem5.gem5Bridge"}
-    )
+    // SST_ELI_DOCUMENT_SUBCOMPONENT_SLOTS(


WHy this change? and why in this commit?

This was updated in this patchset. It spilled over to this change as I felt it was no longer necessary to explicitly document system_port and memory_port as we are now using a vector of outgoing ports

This change adds a lot of new features to the gem5-SST bridge. The following list of changes describes each of them: 1. The bridge now supports limited functional accesses, which enables users to work with vda devices in ARM/RISCV systems. 2. There are new methods added to the bridge which allows future extension of the bridge to swap ports to enable delayed INIT phase for SST. This will allow the bridge to start the system in gem5 and then swap the memory ports to SST at RUN phase. 3. A lot of stray debug statements/cout statements were removed in this patch. 4. New documentation was added to the SST component for gem5. Signed-off-by: Kaustav Goswami <[email protected]>

This change fixes minor changes with the gem5's outgoing bridge, mostly in comments and gem5 syle. Signed-off-by: Kaustav Goswami <[email protected]>

This change drops all the boards and configs created previously to simulate disaggregated memory systems as those were not strictly adhering to the stdlib rules. This new change posts most of the previously written code to create a new abstract disaggregated memory board for both ARM and RISCV architectures, which minimizes code duplication and can be easily extended. Extension suggestions and tips are now commented thoughout the python files. This is a proper directory structure to navigate these changes. Signed-off-by: Kaustav Goswami <[email protected]>

This change create config script for both gem5 and SST and allows SST to run these scripts using one or more ISAs at the same time. Signed-off-by: Kaustav Goswami <[email protected]>

kaustav-goswami · 2023-11-22T04:20:24Z

@powerjg I am sorry that I had such a messy code. I restructured the whole disaggregated memory code. I had a conversation with @studyztp who was trying to extend this code for her work and it was not trivial for her. In this version, I have split the whole setup into boards, cachehierarchies, memories and configs, which should be easy to read and extend. The code extends the stdlib in a systematic manner. I have also removed a lot of unnecessary config files for both gem5 and SST and made the new configs flexible.

This change updates the numactl calls to correctly allocate memory in the local and the remote memory nodes. Signed-off-by: Kaustav Goswami <[email protected]>

This change adds stats counters to the gem5 SST external bridge to monitor the number and size of packets going out and coming into gem5. This numbers should also be reflected in SST's stats output file. Signed-off-by: Kaustav Goswami <[email protected]>

This change adds a private l1 private l2 and shared l3 cache hierarchy as a stdlib component. Signed-off-by: Kaustav Goswami <[email protected]>

This change updates the disaggregated memory scripts to use a 3-level cache and dumps starts after each STREAM kernel. Signed-off-by: Kaustav Goswami <[email protected]>

This commit fixes a compilation error on the gem5 SST bridge. Signed-off-by: Kaustav Goswami <[email protected]>

This patch fixes the previous git merge that duplicated a few lines in the ext library of sst. Signed-off-by: Kaustav Goswami <[email protected]>

…ma-sst

This commit fixes a compilation error on the gem5 SST bridge. Signed-off-by: Kaustav Goswami <[email protected]>

This commit further fixes repeted lines of code. Signed-off-by: Kaustav Goswami <[email protected]>

…ma-sst

powerjg · 2023-12-12T19:00:53Z

disaggregated_memory/boards/arm_sst_board.py

+
+    @overrides(ArmAbstractDMBoard)
+    def _set_remote_memory_ranges(self):
+        pass


Should you add an error here?

powerjg · 2023-12-12T19:01:12Z

disaggregated_memory/boards/arm_sst_board.py

+            clk_freq=clk_freq,
+            processor=processor,
+            local_memory=local_memory,
+            remote_memory_addr_range=self._remoteMemory.remote_memory.physical_address_ranges[


This should probably be a function of the remote memory.

powerjg · 2023-12-12T19:02:29Z

disaggregated_memory/boards/arm_sst_board.py

+        # we need to find whether there is any external latency. if yes, then
+        # add xbar to add this latency.
+
+        if self.get_remote_memory().is_xbar_required():


We should try to make this more generic

powerjg · 2023-12-12T19:03:58Z

disaggregated_memory/boards/riscv_dm_board.py

+        )
+
+    @overrides(RiscvBoard)
+    def generate_device_tree(self, outdir: str) -> None:


How is this different from the base?

powerjg · 2023-12-12T19:05:23Z

disaggregated_memory/boards/riscv_sst_board.py

+
+        :returns: The memory system.
+        """
+        raise NotImplementedError


?? This is odd. Can you explain why this is not implemented?

powerjg · 2023-12-12T19:07:19Z

disaggregated_memory/boards/riscv_sst_board.py

+        # we need to find whether there is any external latency. if yes, then
+        # add xbar to add this latency.
+
+        if self.get_remote_memory().is_xbar_required():


This xbar should bein the remote memory object and simply exposed as a single port.

Let's generalize this by assuming the remote memory will always be a single port (i.e., never channeled). Then, you can make this function be exactly the same for sst/gem5 and arm/riscv.

In fact, there should be a "mixin" class that allows disaggregated memory.

powerjg · 2023-12-12T19:08:22Z

disaggregated_memory/cachehierarchies/dm_caches_sst.py

@@ -0,0 +1,218 @@
+# Copyright (c) 2023 The Regents of the University of California


What are "dm" caches and why are they different from the normal cache hiearchies?

powerjg · 2023-12-12T19:08:47Z

disaggregated_memory/memories/remote_memory.py

+        ]
+
+    @overrides(ChanneledMemory)
+    def get_mem_ports(self) -> Sequence[Tuple[AddrRange, Port]]:


Let's make this a single port and put the xbar in this object.

powerjg · 2023-12-12T19:09:31Z

disaggregated_memory/memories/external_remote_memory.py

+from m5.objects import OutgoingRequestBridge, AddrRange, Tick
+
+
+class ExternalRemoteMemoryInterface:


I think this should inherit from AbstractMemorySystem

powerjg · 2023-12-12T19:11:28Z

disaggregated_memory/configs/riscv-sst-numa-nodes.py

+board._pre_instantiate()
+root = Root(full_system=True, board=board)
+board._post_instantiate()
+m5.instantiate()


These four lines are very odd. Can you explain what it's doing?

This change makes the remote memory a 4 channel DDR4-like memory installed on the remote node. Signed-off-by: Kaustav Goswami <[email protected]>

This change restricts any functional reads at runtime to go to SST. Signed-off-by: Kaustav Goswami <[email protected]>

This change adds a KVM capable runscript for testing the NUMA setup. Signed-off-by: Kaustav Goswami <[email protected]>

This change adds a WIP working x86 board that allows memory offlining with compatible kernel. Signed-off-by: Kaustav Goswami <[email protected]>

Signed-off-by: Kaustav Goswami <[email protected]>

kaustav-goswami added 7 commits October 4, 2023 17:11

ext,configs: adding a disaggregated memory setup

010650d

This change adds necessasry tools to simulate multiple gem5 nodes to simulate a disaggregated memory setup using gem5 and SST. Change-Id: I6e1e52d4ba8df7c161b3151c9b2c02b72fc7cc31 Signed-off-by: Kaustav Goswami <[email protected]>

ext,configs: adding a disaggregated memory setup

0cfb5f5

This change adds necessasry tools to simulate multiple gem5 nodes to simulate a disaggregated memory setup using gem5 and SST. Change-Id: I6e1e52d4ba8df7c161b3151c9b2c02b72fc7cc31 Signed-off-by: Kaustav Goswami <[email protected]>

Merge branch 'kg/dm-numa-sst' of https://github.com/darchr/gem5 into …

f239c92

…kg/dm-numa-sst

kaustav-goswami requested a review from powerjg November 17, 2023 00:55

kaustav-goswami self-assigned this Nov 17, 2023

powerjg requested changes Nov 17, 2023

View reviewed changes

kaustav-goswami added 4 commits November 21, 2023 18:46

ext: corrected minor changes in the outgoing bridge

b4d92cb

This change fixes minor changes with the gem5's outgoing bridge, mostly in comments and gem5 syle. Signed-off-by: Kaustav Goswami <[email protected]>

ext,configs: added stdlib examples to sst scripts

edbaf2a

This change create config script for both gem5 and SST and allows SST to run these scripts using one or more ISAs at the same time. Signed-off-by: Kaustav Goswami <[email protected]>

kaustav-goswami added 10 commits November 26, 2023 16:22

configs: updated numactl calls

1afc0a8

This change updates the numactl calls to correctly allocate memory in the local and the remote memory nodes. Signed-off-by: Kaustav Goswami <[email protected]>

configs,stdlib: added a private l1, l2 and shared l3 cache

57edb4b

This change adds a private l1 private l2 and shared l3 cache hierarchy as a stdlib component. Signed-off-by: Kaustav Goswami <[email protected]>

configs: updated disaggregated memory scripts

18fe2ee

This change updates the disaggregated memory scripts to use a 3-level cache and dumps starts after each STREAM kernel. Signed-off-by: Kaustav Goswami <[email protected]>

ext,base-stats: minor fix in the stat counter

8aabc15

This commit fixes a compilation error on the gem5 SST bridge. Signed-off-by: Kaustav Goswami <[email protected]>

ext: fixed repeated lines of code

24dfc0a

This patch fixes the previous git merge that duplicated a few lines in the ext library of sst. Signed-off-by: Kaustav Goswami <[email protected]>

Merge branch 'kg/dm-numa-sst' of github.com:darchr/gem5 into kg/dm-nu…

5a70b11

…ma-sst

ext,base-stats: minor fix in the stat counter

167bf8d

This commit fixes a compilation error on the gem5 SST bridge. Signed-off-by: Kaustav Goswami <[email protected]>

ext: fixed ext: fixed repeated lines of code

e2610b3

This commit further fixes repeted lines of code. Signed-off-by: Kaustav Goswami <[email protected]>

Merge branch 'kg/dm-numa-sst' of github.com:darchr/gem5 into kg/dm-nu…

eb80501

…ma-sst

kaustav-goswami requested a review from mbabaie December 5, 2023 00:01

powerjg reviewed Dec 12, 2023

View reviewed changes

kaustav-goswami added 3 commits December 12, 2023 18:56

ext: updated the example arm dm script

23e680d

This change makes the remote memory a 4 channel DDR4-like memory installed on the remote node. Signed-off-by: Kaustav Goswami <[email protected]>

ext: restrict any functional reads to go to sst

8c10039

This change restricts any functional reads at runtime to go to SST. Signed-off-by: Kaustav Goswami <[email protected]>

configs: added a KVM capable ARM NUMA runscript

cabe028

This change adds a KVM capable runscript for testing the NUMA setup. Signed-off-by: Kaustav Goswami <[email protected]>

kaustav-goswami added 2 commits January 19, 2024 13:44

arch-x86,configs: added a WIP x86 dm board

53f8b47

This change adds a WIP working x86 board that allows memory offlining with compatible kernel. Signed-off-by: Kaustav Goswami <[email protected]>

configs: added a arm kvm config for dm

8ec3229

Signed-off-by: Kaustav Goswami <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gem5+SST13 Changes #269

gem5+SST13 Changes #269

kaustav-goswami commented Nov 17, 2023

powerjg left a comment

powerjg Nov 17, 2023

kaustav-goswami Nov 22, 2023

powerjg Nov 17, 2023

powerjg Nov 17, 2023

kaustav-goswami Nov 22, 2023

powerjg Nov 17, 2023

kaustav-goswami Nov 22, 2023

kaustav-goswami commented Nov 22, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

powerjg Dec 12, 2023

		@@ -0,0 +1,218 @@
		# Copyright (c) 2023 The Regents of the University of California

		from m5.objects import OutgoingRequestBridge, AddrRange, Tick


		class ExternalRemoteMemoryInterface:

gem5+SST13 Changes #269

Are you sure you want to change the base?

gem5+SST13 Changes #269

Conversation

kaustav-goswami commented Nov 17, 2023

powerjg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaustav-goswami commented Nov 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment