[ReadingService] Add round robin sharding to support non-replicable DataPipe for Multiprocessing #919

ejguan · 2022-12-09T23:54:32Z

This PR is created on top of #555. And, this PR extends PrototypeMultiprocessingReadingService to accept non-replicable DataPipe.

And, this PR depends on pytorch/pytorch#90769

Main Changes

Add a way to launch a process to fetch data from non-replicable DataPipes and send data to worker processes in a round-robin manner
- Add ShardingRoundRobinDispatcher (functional name sharding_round_robin_dispatch) to indicate non-replicable DataPipe
- Add MultipleDataPipesToQueuesLoop to connect non-sharding process to request/response queues
- Add find_lca_non_replicable_dp as a graph function to determine the lowest common ancestor of all non-replicabble DataPipes. This would guarantee that all non-replicable DataPipes will be running in a single dispatching process
- In each multiprocessing worker process,
  - If All datapipes are replicable, apply multiprocessing sharding to the graph
  - If not, worker would use find_replicable_branches to apply mp sharding to those replicable branches, because all non-replicable branches have been properly sharded by routing data round-robinly to worker processes.
Properly get ResetEpochResponse from protocol via get_response_reset_epoch
Add tests for two graph functions
Add test to launch non-shardable DataPipe process
Add documents
- Replicable DataPipe/Non-replicable DataPipe in multiprocessing
- How PrototypeMPRS handles the above two types of DataPipe
  Please check the link for doc: https://ejguan.github.io/dataloader2.html#dynamic-sharding

nit Changes

Rename Spawn to Create as the process has not been started

test/dataloader2/test_dataloader2.py

torchdata/dataloader2/communication/iter.py

ejguan · 2022-12-09T23:56:35Z

torchdata/dataloader2/communication/protocol.py

@@ -225,6 +225,16 @@ def get_response_reset_iterator(self, block=False):
        if not isinstance(response, communication.messages.ResetIteratorResponse):
            raise Exception("Invalid response received")

+    def get_response_reset_epoch(self, block=False):


Not sure why we didn't have this response before

Non-blocking but - hmmmm.... did this cause any bug or unhandled messages? Do you happen to know why?
I know you were looking into unusually messages and responses

When new iteration starts, reset will be called for _IterateQueueDataPipes . And, an extra get_respondse_next is invoked to drop / response. So, all requests are served.

data/torchdata/dataloader2/reading_service.py

Lines 152 to 154 in 0a0ae5d

for dp in self.datapipes:

if dp.protocol.waiting_for_response():

dp.protocol.get_response_next(block=True)

I can give a try to remove this part.

It seems removing is fine

noob question: any docs / entry pointer to understand how these dataloader2/communication design / work? ~

Unfortunately no doc. I can talk about the components based on my understanding:

ProtocolClient remains in the main process to pass Request via the request_queue to the corresponding worker process

ProtocolServer is created in the worker process that takes request then send Response back to main process via reqponse_queue

DataPipeBehindQueues is the worker loop that holds a ProtocolServer to maniputlate DataPipe based on the Reqeust

QueueWrapper is the DataPipe that holds a ProtocolClient instance to issue Request and yield data from response_queue to the subsequent DataPipe graph.

We can talk about more detail offline if you want

Thanks! Something like this? ~

Let me try to draw it with mermaid

graph TD; Worker_1_1-->ProtocolServer_1_1-->ProtocolClient_1 Worker_1_2-->ProtocolServer_1_2-->ProtocolClient_1 Worker_1_3-->ProtocolServer_1_3-->ProtocolClient_1 ProtocolClient_1-->GPU1 ProtocolClient_2-->GPU2 ProtocolClient_3-->GPU3

Loading

facebook-github-bot · 2022-12-13T17:50:03Z

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-12-13T18:15:32Z

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-12-13T18:18:10Z

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

torchdata/dataloader2/communication/eventloop.py

NivekT

I am going to need more time to look through this. What is the use case of non-shardable DataPipe?

NivekT

I think we should provide a definition of non-shardable DataPipe and why that may occur (an abstract example would be helpful as well). In particularly, as you mentioned, note that we do not want to duplicate such DataPipe into multiple workers and sharding filter should not be applied to it. Instead it should be read round-robin (or something else) by downstream DataPipes that are sharded?

ejguan · 2022-12-14T18:03:07Z

I think we should provide a definition of non-shardable DataPipe and why that may occur (an abstract example would be helpful as well).

Make sense. I will add doc regarding non-shardbable DataPipe/shardable DataPipe to the documents for dataloader2.

In particularly, as you mentioned, note that we do not want to duplicate such DataPipe into multiple workers and sharding filter should not be applied to it. Instead it should be read round-robin (or something else) by downstream DataPipes that are sharded?

Actually, this is not True. sharding_filter is still needed if distributed sharding is needed. We only disabled multiprocessing sharding for the non-shardable branch. And, round-robin reading is also achieved automatically with ProtoTypeMPRS.

Edit: Updated the summary with a few topics need to be covered. Let me know if there is any other concern on the documentation.

NivekT · 2022-12-15T08:33:07Z

torchdata/dataloader2/communication/protocol.py

@@ -225,6 +225,16 @@ def get_response_reset_iterator(self, block=False):
        if not isinstance(response, communication.messages.ResetIteratorResponse):
            raise Exception("Invalid response received")

+    def get_response_reset_epoch(self, block=False):


Non-blocking but - hmmmm.... did this cause any bug or unhandled messages? Do you happen to know why?
I know you were looking into unusually messages and responses

NivekT · 2022-12-15T08:37:40Z

torchdata/dataloader2/utils/worker.py

@@ -84,9 +153,22 @@ def process_reset_fn(
    reset the random state of the ``DataPipe`` graph and the global random states for ``torch``,
    ``random`` and ``numpy``.
    """
+    # Reset non-sharding process first
+    graph = traverse_dps(datapipe)
+    non_sharding_process_dps = find_dps(graph, communication.iter._IterateQueueDataPipes)


Question: Is it always the case where _IterateQueueDataPipes is the only non-sharding process?

Yes because we will find the lowest common ancestor of all non-shardable DataPipes in the main process and replace it by this _IterateQueueDataPipes in the worker process.
So, it's guaranteed that there is only a single non-sharding process.

ejguan · 2022-12-15T23:40:45Z

@NivekT
Thanks to your question regarding example of non-shardable DataPipe. There are two kinds of non-shardable DataPipe. However, we have to treat them differently.

non-shardable data source that connects to a remote system that only accept one client per machine.
For non-shardable data source, we need to launch the non-sharding process to transfer data from non-shardable data source to worker processes
fullsync DataPipe that should not run in worker process.
This DataPipe is normally appended at the end of pipeine. If we use the lowest common ancestor of all non-shardable DataPipe as the non-shardable partial graph, the whole pipeline would be sent to non-sharding process. We have to make sure fullsync is attached to the output of worker processes in the main process

I propose to change to find lowest common ancestor of non-replicable Data Source and sent them to non-sharding process.
Simple example:

graph TD;
DP1(non-replicable DP1)-->DP2;
DP2-->DP5;
DP3(non-replicable DP3)-->DP4;
DP4-->DP5;
DP5-->DP6;
DP6-->fullsync;
fullsync-->output;

The lowest common ancestor of all non-shardable Data Source is DP5. If we include fullsync, the whole pipeline becomes non-shardable. Given the above example, the non-sharding process should contain two branches until DP5 and worker process should execute only DP6 and at the end of pipeline fullsync is attach in the main process

NivekT · 2022-12-16T09:32:26Z

That makes sense. How do you plan to implement that? Will we still have users calling .fullsync() and make modification to the graph in DL2? Or making full_sync an option within DL2? Probably the former?

ejguan · 2022-12-16T16:19:43Z

How do you plan to implement that? Will we still have users calling .fullsync() and make modification to the graph in DL2?

We should allow either users calling fullsync or let DistributedRS attach fullsync automatically. Then, ProtoMPRS should do graph modification. WDYT? I am not going to implement it in this PR for fullsync though.

wenleix · 2022-12-19T06:12:47Z

Replying to #919 :

Actually, this is not True. sharding_filter is still needed if distributed sharding is needed. We only disabled multiprocessing sharding for the non-shardable branch. And, round-robin reading is also achieved automatically with ProtoTypeMPRS.

An alternative approach for distributed sharding would be distribute the workload based on filename or some compression/encoding unit in file (in Parquet it's called "Page": https://parquet.apache.org/docs/concepts/) and in ORC I think it's called "Stripe". So it avoid reading the original data multiple times?

test/dataloader2/test_dataloader2.py

wenleix · 2022-12-19T07:12:31Z

torchdata/dataloader2/communication/protocol.py

@@ -225,6 +225,16 @@ def get_response_reset_iterator(self, block=False):
        if not isinstance(response, communication.messages.ResetIteratorResponse):
            raise Exception("Invalid response received")

+    def get_response_reset_epoch(self, block=False):


noob question: any docs / entry pointer to understand how these dataloader2/communication design / work? ~

ejguan · 2022-12-20T16:10:26Z

Non-shardable is an extremely bad name as it's actually being sharded by round-robin dispatching. The actual meaning here is to prevent copy of DataPipe to multiple processes. I might rename it to non-replicable DataPipe/dispatching process.

NivekT · 2022-12-20T17:45:52Z

Non-shardable is an extremely bad name as it's actually being sharded by round-robin dispatching. The actual meaning here is to prevent copy of DataPipe to multiple processes. I might rename it to non-replicable DataPipe/dispatching process.

I agree with renaming "non-shardable" to "non-replicated" DataPipe. I suppose sometimes it is replicable but the users won't want to?

ejguan · 2022-12-20T23:09:53Z

@wenleix @NivekT This PR has been updated. And updated document can be found in https://ejguan.github.io/dataloader2.html#dynamic-sharding

ejguan · 2022-12-21T15:24:02Z

torchdata/datapipes/iter/util/prefetcher.py

+        # Lazily import to prevent circular import
+        from torchdata.dataloader2 import communication
+


This is my temporary fix for the circular import problem.
cc: @NivekT

ejguan

The following review steps are added to make it easier to review the main logic in the PR. Let me know if there is anything fuzzy to you.

ejguan · 2022-12-21T15:25:43Z

torchdata/datapipes/iter/util/sharding.py

+@functional_datapipe("sharding_round_robin_dispatch")
+class ShardingRoundRobinDispatcherIterDataPipe(IterDataPipe):


Review Step 1: Add ShardingRoundRobinDispatcher is introduced to indicate where the pipeline should be non-replicable.

I am open to any suggestion on the name/functional name

ejguan · 2022-12-21T15:26:30Z

torchdata/datapipes/iter/util/sharding.py

+    def __iter__(self) -> Iterator[T_co]:
+        yield from self.source_datapipe


Review Step 1.1: Keep __iter__ as a noop here rather than raising Error to support single-process use case.

by "single-process use case". does it mean "eager mode"?

Yeah, pure eager currently. In the future, we might provide a by-default SingleProcessReadingService for users.

ejguan · 2022-12-21T15:29:01Z

torchdata/dataloader2/utils/dispatch.py

+    res_queue: Queue
+
+
+def find_lca_non_replicable_dp(graph: DataPipeGraph) -> Optional[DataPipe]:


Review Step 2: Add this graph function to find the lowest common ancestor of the non-replicable DataPipes (ShardingRoundRobinDispatcher)

ejguan · 2022-12-21T15:30:09Z

test/test_graph.py

+        graph = traverse_dps(end_dp)
+        return single_br_dp, multi_br_dp, ch1, ch2, fork_zip_dp, cir_br_dp, cir_map_dp, end_dp, graph
+
+    def test_single_non_replicable_dp(self):


Review Step 3.1: Tests for single non-replicable DataPipe

ejguan · 2022-12-21T15:30:20Z

test/test_graph.py

+        graph, cir_map_dp = make_dp_non_replicable(graph, cir_map_dp)
+        self.assertEqual(find_lca_non_replicable_dp(graph), cir_map_dp)
+
+    def test_multi_non_replicable_dps(self):


Review Step 3.2: Tests for multiple non-replicable DataPipes

ejguan · 2022-12-21T15:49:19Z

torchdata/dataloader2/communication/eventloop.py

@@ -91,3 +131,20 @@ def SpawnThreadForDataPipeline(datapipe):

    process = threading.Thread(target=DataPipeToQueuesLoop, args=(new_datapipe, req_queue, res_queue), daemon=True)
    return process, req_queue, res_queue, new_datapipe
+
+
+def CreateProcessForMultipleDataPipelines(multiprocessing_ctx, datapipes):


Review Step 7.1: Create num_workers pairs of req_queue and res_queue.
And launch MultipleDataPipesToQueuesLoop to iterate over the non-replicable DataPipe

ejguan · 2022-12-21T15:53:11Z

torchdata/dataloader2/communication/eventloop.py

 ]


+def MultipleDataPipesToQueuesLoop(source_datapipes, req_queues, res_queues, call_on_process_init=None):


Review 7.2: Launch a non-blocking DataPipeBehindQueues while-loop per child DataPipe from round_robin_demux.
Using zip_longest to mimic round robin calling next over each child DataPipe.

ejguan · 2022-12-21T15:56:02Z

torchdata/dataloader2/reading_service.py

+            # Dispatching process for non-replicable DataPipes exists
+            if self._dispatch_process is not None:
+                # Use the placehold to pass request/response queue to each worker process
+                dummy_dp.req_queue = self._dispatch_process[1][worker_id]
+                dummy_dp.res_queue = self._dispatch_process[2][worker_id]


Review 6.2: We only have one _DummyIterDataPipe in the main process but have num_workers pairs of req_queue and res_queue. To connect a pair to the corresponding worker process, inject the attributes from _DummyIterDataPipe before sending it to the worker process.

ejguan · 2022-12-21T15:58:40Z

torchdata/dataloader2/utils/worker.py

+    # Find if there is non-replicable DataPipe
+    graph = traverse_dps(datapipe)
+    non_replicable_dp = find_dps(graph, _DummyIterDataPipe)  # type: ignore
+
+    # There are two cases for DataPipe graph in terms of mp sharding:
+    # 1) All DataPipes are replicable, apply mp sharding to the whole graph
+    if len(non_replicable_dp) == 0:
+        torch.utils.data.graph_settings.apply_sharding(
+            datapipe, worker_info.num_workers, worker_info.worker_id, SHARDING_PRIORITIES.MULTIPROCESSING
+        )
+    # 2) There is non-replicable DataPipe. Since we have replaced the lowest common
+    #    ancestor by a `_DummyIterDataPipe`, we would only apply mp sharding
+    #    to replicable branches that don't have `_DummyIterDataPipe`.
+    else:
+        assert len(non_replicable_dp) == 1
+        replicable_branches = find_replicable_branches(graph)
+        for dp in replicable_branches:
+            torch.utils.data.graph_settings.apply_sharding(
+                dp, worker_info.num_workers, worker_info.worker_id, SHARDING_PRIORITIES.MULTIPROCESSING
+            )
+
+        req_queue = non_replicable_dp[0].req_queue
+        res_queue = non_replicable_dp[0].res_queue
+
+        queue_wrapper = communication.iter.QueueWrapper(
+            communication.protocol.IterDataPipeQueueProtocolClient(req_queue, res_queue)
+        )
+        dispatch_process_dp = communication.iter._IterateQueueDataPipes([queue_wrapper])
+        graph = replace_dp(graph, non_replicable_dp[0], dispatch_process_dp)
+        datapipe = list(graph.values())[0][0]


Review Step 8: In the worker process, find if there is _DummyIterDataPipe.
If not, it means the whole pipeline is replicable and do the sharding by filter
If there is, we would do sharding only over the replicable branches.

QueueWrapper and _IterateQueueDataPipes is used to wrap res_queue and req_queue as a DataPipe that can handle Request and Response based on the protocol.

ejguan · 2022-12-21T16:00:23Z

torchdata/dataloader2/utils/worker.py

+def dispatch_process_reset_fn(
+    datapipe: DataPipe,
+    worker_info: WorkerInfo,
+    dist_info: _DistInfo,
+) -> DataPipe:
+    r"""
+    Based on the distributed shared random seed, this function is used to set the random state
+    of the non-repliable ``DataPipe`` graph and the global random states for the dispatch process.
+    This function would guarantee that all distributed non-sharding process share the
+    same random states to ensure the same shuffle order.
+    """
+    worker_seed_generator = torch.Generator()
+    worker_seed_generator.manual_seed(dist_info.shared_seed)
+    torch.utils.data.graph_settings.apply_random_seed(
+        datapipe,
+        worker_seed_generator,
+    )
+
+    # Set global random states
+    _set_global_random_state(worker_seed_generator)
+
+    return datapipe


Review Step 9: When new epoch starts, we want to control the random seed based on distributed information.
We need to guarantee all distributed dispatching process share the same random seed.

wenleix

"Review Step 1: Add ShardingRoundRobinDispatcher is introduced to indicate where the pipeline should be non-replicable."

LGTM. Name (sharding_round_robin_dispatch) is a bit long but let's keep it for now...

…ed non-sharding process

wenleix

"Review Step 7: create req_queue and res_queueand execute with round_robin_demux"

High-level control flow looks good. But didn't have enough low-level context yet~

torchdata/dataloader2/communication/eventloop.py

wenleix · 2022-12-22T20:42:25Z

torchdata/dataloader2/communication/eventloop.py

+        res_queues
+    ), "``MultipleDataPipesToQueuesLoop`` requires the same number of datapipes, request queues and response queues"
+
+    torch.set_num_threads(1)


noob question: what's this for?

IIRC, this was introduced to disable OpenMP in dataloader workers. This is because OpenMP would create number of threads that equals to the number of CPU cores by default. And, with multiprocessing enabled, num_workers x num_threads_per_worker threads will be created. This won't provide any further benefit.

Besides, OpenMP features should not be enabled if any OpenMP features are utilized in the main process and before subprocesses are forked.

Any suggestion?

wenleix

"Review Step 8: Worker process handling (graph rewrite and receiving demux result from dispatch process)"

LGTM.

wenleix · 2022-12-22T20:58:15Z

torchdata/dataloader2/utils/worker.py

+        queue_wrapper = communication.iter.QueueWrapper(
+            communication.protocol.IterDataPipeQueueProtocolClient(req_queue, res_queue)
+        )
+        dispatch_process_dp = communication.iter._IterateQueueDataPipes([queue_wrapper])


IIUC:

IterDataPipeQueueProtocolClient will be wrapped into a QueueWrapper (but still not a IterDataPipe), and further wrapped into a _IterateQueueDataPipes which is a IterDataPIpe?

Nope. Both QueueWrapper and _IterateQueueDataPipes are IterDataPipe, this is one of the thing that we can optimize later.

wenleix

Review Step 9: LGTM % minor question...

torchdata/dataloader2/utils/worker.py

facebook-github-bot · 2022-12-27T19:21:06Z

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-12-28T15:20:40Z

@ejguan merged this pull request in e15e145.

NivekT · 2023-01-03T17:34:12Z

torchdata/dataloader2/communication/iter.py

+                    yield response.value
+
+    def reset(self):
+        # NonBlocking DataPipes do not reset automatically, have to do it manually


@ejguan Just noticed that the reset method has changed after the move. It used to have this:

# Collect all existing requests results to clear queues for dp in self.datapipes: if dp.protocol.waiting_for_response(): dp.protocol.get_response_next(block=True)

Is this no longer necessary?

I don't think it's necessary because we always want to do reset_epoch as well for NonBlocking, which will discard all existing requests. So, when at the point of reset, we should expect no request within the worker process queues.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 9, 2022

ejguan commented Dec 9, 2022

View reviewed changes

test/dataloader2/test_dataloader2.py Show resolved Hide resolved

ejguan commented Dec 9, 2022

View reviewed changes

torchdata/dataloader2/communication/iter.py Show resolved Hide resolved

ejguan commented Dec 9, 2022

View reviewed changes

ejguan added the topic: new feature topic category label Dec 9, 2022

ejguan force-pushed the proto2rs branch from de6e228 to 47201aa Compare December 12, 2022 22:48

ejguan requested review from NivekT and wenleix December 13, 2022 17:50

ejguan commented Dec 13, 2022

View reviewed changes

torchdata/dataloader2/communication/eventloop.py Outdated Show resolved Hide resolved

NivekT reviewed Dec 14, 2022

View reviewed changes

ejguan force-pushed the proto2rs branch from dc8a68c to 73f4ce6 Compare December 14, 2022 16:42

NivekT reviewed Dec 14, 2022

View reviewed changes

NivekT reviewed Dec 15, 2022

View reviewed changes

wenleix reviewed Dec 19, 2022

View reviewed changes

ejguan changed the title ~~[ReadingService] Add mp support for non-shardable DataPipe~~ [ReadingService] Add round robin sharding to support non-replicable DataPipe for Multiprocessing Dec 20, 2022

ejguan mentioned this pull request Dec 21, 2022

Potential circular import in prefetcher #931

Closed

ejguan commented Dec 21, 2022

View reviewed changes

wenleix reviewed Dec 21, 2022

View reviewed changes

ejguan added 16 commits December 22, 2022 17:03

Add LCA test

70ff50f

Add tests for shardable branches

eda375b

Fix LCA for branches and circular reference

3e9fb6b

Add tests for non-sharding process

d454ab2

Improve test suite

78b88f0

Fix mypy

4fe85ba

Miss one test

a5420dd

Remove wrong comment and unused import

b388ac7

Add non-sharding reset fn to keep the same random state for distribtu…

f74257e

…ed non-sharding process

Set sleep to 0

b3270c9

Remove request cleanup at reset for _IterateQueueDataPipes

a3805b4

Add documentation for non-shardable DataPipe

5866a8d

Add ShardingRoundRobinDispatcher to inidicate dispatching process

2586638

Add graphviz doc

0b4c7f6

Add assertion tests for the number of replicable branches

0262278

Change cache and helper function to more meaningful names

f93215b

ejguan force-pushed the proto2rs branch from 4274a8c to f93215b Compare December 22, 2022 17:19

wenleix reviewed Dec 22, 2022

View reviewed changes

torchdata/dataloader2/utils/worker.py Outdated Show resolved Hide resolved

torchdata/dataloader2/utils/worker.py Outdated Show resolved Hide resolved

wenleix approved these changes Dec 27, 2022

View reviewed changes

Resolve comment

0d3ceac

facebook-github-bot closed this in e15e145 Dec 28, 2022

facebook-github-bot added the Merged label Dec 28, 2022

ejguan mentioned this pull request Dec 28, 2022

_DataPipeSerializationWrapper doesn't work with multiprocessing Queue #934

Open

NivekT reviewed Jan 3, 2023

View reviewed changes

NivekT mentioned this pull request Jan 23, 2023

Tables in Documentation not rendering properly #959

Closed

	for dp in self.datapipes:
	if dp.protocol.waiting_for_response():
	dp.protocol.get_response_next(block=True)

		# Lazily import to prevent circular import
		from torchdata.dataloader2 import communication

		@functional_datapipe("sharding_round_robin_dispatch")
		class ShardingRoundRobinDispatcherIterDataPipe(IterDataPipe):

		def __iter__(self) -> Iterator[T_co]:
		yield from self.source_datapipe

		res_queue: Queue


		def find_lca_non_replicable_dp(graph: DataPipeGraph) -> Optional[DataPipe]:

		]


		def MultipleDataPipesToQueuesLoop(source_datapipes, req_queues, res_queues, call_on_process_init=None):

[ReadingService] Add round robin sharding to support non-replicable DataPipe for Multiprocessing #919

[ReadingService] Add round robin sharding to support non-replicable DataPipe for Multiprocessing #919

Conversation

ejguan commented Dec 9, 2022 • edited Loading

Main Changes

nit Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejguan Dec 19, 2022 • edited by wenleix Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenleix Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Dec 13, 2022

facebook-github-bot commented Dec 13, 2022

facebook-github-bot commented Dec 13, 2022

NivekT left a comment

Choose a reason for hiding this comment

NivekT left a comment

Choose a reason for hiding this comment

ejguan commented Dec 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejguan commented Dec 15, 2022 • edited by wenleix Loading

NivekT commented Dec 16, 2022

ejguan commented Dec 16, 2022

wenleix commented Dec 19, 2022

Choose a reason for hiding this comment

ejguan commented Dec 20, 2022 • edited Loading

NivekT commented Dec 20, 2022

ejguan commented Dec 20, 2022

Choose a reason for hiding this comment

ejguan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenleix left a comment

Choose a reason for hiding this comment

wenleix left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenleix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenleix left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Dec 27, 2022

facebook-github-bot commented Dec 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejguan commented Dec 9, 2022 •

edited

Loading

ejguan Dec 19, 2022 •

edited by wenleix

Loading

wenleix Dec 21, 2022 •

edited

Loading

ejguan commented Dec 14, 2022 •

edited

Loading

ejguan commented Dec 15, 2022 •

edited by wenleix

Loading

ejguan commented Dec 20, 2022 •

edited

Loading

wenleix left a comment •

edited

Loading