Cli #589

ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store hosHnGWDuRUhAKD6djMqmo.PxPE2ynCb6vFYX9iXZBGEH')>
Exception: Failed to get session store hosHnGWDuRUhAKD6djMqmo.PxPE2ynCb6vFYX9iXZBGEH

why is this and what does it mean?

oeway · 2023-06-11T21:32:16Z

It might because of the local proxy was removed when you call a certain function. I just tried your cli branch, the following worked (upgraded imjoy-rpc==0.5.21, and hypha==0.15.15):

    ....
    await poll_for_services(server)
    print('services found')
    
    services = await server.list_services()
    service = [s for s in services if s["name"] == "itkwidgets_client"][0]
    itk_viewer = await server.get_service(service)
    image = itk.imread(data)
    await itk_viewer.setImage(np.array(image))

I can see the image get loaded in the opened browser tab.

I saw you have some polling functions in the cli.py, maybe a better way to implement it is to register a service in Python, e.g. a image zarr store provider service. We register it before we trigger the browser, when browser is loaded, the viewer use the workspace/token to get the zarr store from the service. Something like this:

await server.register_service({
      "name": "ItkwidgetsFileLoader",
      "id": "itkwidgets-file-loader",
      "description": "Load a file from the server",
      "config": {
          "visibility": "protected"
      },
      "load_file": load_file
})

In the itk-vtk-viewer, we can get the service and load the remote zarr store, in a passive way. Alternatively, you can also pass the current itk-vtk-viewer api object to the load_file function, then the server will be able to control the viewer. Note that since hypha service function are suppose to be functional, so you won't be able to keep the viewer api object.

Another advantage is we can have multiple viewer window open at the same time.

We can support collaborative visualisation too, since the server service can push viewer parameters to the viewer clients.

itkwidgets/standalone/index.html

bnmajor · 2023-06-19T20:27:03Z

@thewtex @oeway Thank you both for all of your help understanding what we actually need here!

This branch has been updated to register services for communication between the client and server rather than registering a plugin and seems to work well for now! There is basic arg support for some of the args the createViewer expects and can be expanded to add support for all of the other kwargs that view currently accepts. It is still hard-coded to handle images using itk.imread and does not currently handle point_set files though. Before doing too much here though I thought it might be good to discuss best options for how to handle all the possible file inputs.

Additionally there is a convenience function now, standalone_viewer, which can be used to retrieve the itk-vtk-viewer object when a viewer has been created and served. Right now it just expects the full URL which is then parsed to use the workspace and token to connect to the server before using the service to get the viewer. The easiest way to use this is by starting IPython from the command line:

In [1]: from itkwidgets import standalone_viewer
In [2]: url = "http://127.0.0.1:37480/itkwidgets/index.html?workspace={...}&token={...}"
In [3]: viewer = await standalone_viewer(url)
In [4]: await viewer.setRotateEnabled(True)

This is achievable through the Python REPL as well but it is a bit more complicated:

>>> from itkwidgets import standalone_viewer
>>> import asyncio
>>> async def start_rotate(loop, url):
...     viewer = await standalone_viewer(url)
...     await viewer.setRotateEnabled(True)
>>> url = "http://127.0.0.1:37480/itkwidgets/index.html?workspace={...}&token={...}"
>>> loop = asyncio.new_event_loop()
>>> future = loop.create_task(start_rotate(loop, url))
>>> loop.run_until_complete(future)

oeway · 2023-06-26T12:27:30Z

FYI: i recently added “startup function” support which might be useful for this cli. Not sure how well it will fit the cli case though: https://ha.amun.ai/#/?id=custom-initialization-and-service-integration-with-hypha-server

oeway · 2023-06-28T22:37:04Z

Hi @bnmajor I added a synchronous wrapper for imjoy-rpc with hypha server: imjoy-team/imjoy-rpc#546 (It works by running the event loop in a separate thread)
This might improve how the user use the itkwidgets cli since they won't need to use async/await. Let me know if it works for you.

bnmajor · 2023-07-06T15:30:52Z

I added a synchronous wrapper for imjoy-rpc with hypha server: imjoy-team/imjoy-rpc#546 (It works by running the event loop in a separate thread)

Thanks @oeway, this is fantastic! With this change I was able to update the function to be synchronous, which means that starting up a basic python interpreter and updating the viewer programmatically is now significantly simpler:

>>> from itkwidgets import standalone_viewer
>>> url = "http://127.0.0.1:37480/itkwidgets/index.html?workspace={...}&token={...}"
>>> viewer = standalone_viewer(url)
>>> viewer.set_rotate_enabled(True)

This seems to work well for basic functionality, but if you try to set an image or label image you get the following:

ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.ZSuEC7FJWDivJZCDgdvLst')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.ZSuEC7FJWDivJZCDgdvLst
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.nvfDB8jVRH9oGHLv3jaFx6')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.nvfDB8jVRH9oGHLv3jaFx6
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.hRE4ARjrx6i4RFhnkDQxLM')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.hRE4ARjrx6i4RFhnkDQxLM
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.9hpyjmbg3gjWYwaDvqY6j4')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.9hpyjmbg3gjWYwaDvqY6j4
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.C5mTujPBemgJHiMhDf3AvR')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.C5mTujPBemgJHiMhDf3AvR
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.J8NVuLMh9dYdbUpyoCNeSj')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.J8NVuLMh9dYdbUpyoCNeSj
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.EEfogoASEdFZPjk3EkJfey')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.EEfogoASEdFZPjk3EkJfey
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.hGJQprwuK6k3g63VnM8ox7')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.hGJQprwuK6k3g63VnM8ox7
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.QGZbnUDgounxW965MoVHpj')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.QGZbnUDgounxW965MoVHpj
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.DapoRmvqGXH83zJpQUXrtx')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.DapoRmvqGXH83zJpQUXrtx
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.RRwihdq3r93Nx26G7zocEG')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.RRwihdq3r93Nx26G7zocEG
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.jpY6hpDbs2aFY9bBYQxrzP')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.jpY6hpDbs2aFY9bBYQxrzP
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.Ui5mEzzmcuNeNJ2HDTMcgu')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.Ui5mEzzmcuNeNJ2HDTMcgu
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.CNaMTuV4kgUxA76dvzDzGX')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.CNaMTuV4kgUxA76dvzDzGX
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.A9JK8qYBja5AT32tTypiyw')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.A9JK8qYBja5AT32tTypiyw
ERROR:asyncio:FuturePromise exception was never retrieved
future: <FuturePromise finished exception=Exception('Failed to get session store QKVLVpfxrfj3KzeyayFKLS.5ja3V8Xmpyditskz35JPSM')>
Exception: Failed to get session store QKVLVpfxrfj3KzeyayFKLS.5ja3V8Xmpyditskz35JPSM

@thewtex The function has also been updated to return a Viewer instance so that all functions are wrapped the same way that they are in the notebook.

bnmajor · 2023-07-06T16:49:39Z

FYI: i recently added “startup function” support which might be useful for this cli. Not sure how well it will fit the cli case though: https://ha.amun.ai/#/?id=custom-initialization-and-service-integration-with-hypha-server

@oeway I did look into using this approach, but it seems that the server object that is returned has no register_codec function - is this intentional?

itkwidgets/standalone/config.py

pyproject.toml

itkwidgets/__init__.py

thewtex · 2023-07-06T16:53:26Z

itkwidgets/standalone_server.py

+
+        loop.create_task(start_viewer(server_url))
+        loop.run_forever()
+        time.sleep(30)


itkwidgets/standalone_server.py

oeway · 2023-07-09T01:16:11Z

@oeway I did look into using this approach, but it seems that the server object that is returned has no register_codec function - is this intentional?

Have you tried the latest version, it should work:

>>> from imjoy_rpc.hypha import connect_to_server_sync
>>> server = connect_to_server_sync({"server_url": "https://ai.imjoy.io"})
>>> server.register_codec
<bound method RPC.register_codec of <imjoy_rpc.hypha.rpc.RPC object at 0x7fc248082520>>

oeway · 2023-07-10T00:43:34Z

BTW, I just added webrtc support for imjoy-rpc: imjoy-team/imjoy-rpc#547
This might be interesting for you too, it enables peer-to-peer connection between hypha clients, you can use our public server for signaling, the the rest will be p2p. We are setting up a STUN/TURN server for complicated firewall too. The plan is to support streaming microscope images and control it with hypha. It will be interesting for itkwidgets to enable remote rendering too!

bnmajor · 2023-07-10T15:40:02Z

@oeway I did look into using this approach, but it seems that the server object that is returned has no register_codec function - is this intentional?

Have you tried the latest version, it should work:
>>> from imjoy_rpc.hypha import connect_to_server_sync
>>> server = connect_to_server_sync({"server_url": "https://ai.imjoy.io"})
>>> server.register_codec
<bound method RPC.register_codec of <imjoy_rpc.hypha.rpc.RPC object at 0x7fc248082520>>

Thanks @oeway! That is great to know. What I am actually referring to is the server object that is passed into the startup function:

with subprocess.Popen([
    sys.executable, "-m", "hypha.server",
    f"--host={SERVER_HOST}", f"--port={PORT}",
    "--static-mounts", f"/itkwidgets:{viewer_mount_dir}",
    "--startup-functions=itkwidgets.standalone_server:start_viewer"
], env=hypha_server_env)

...

async def start_viewer(server):
    register_itkwasm_imjoy_codecs_cli(server)

...

File "/home/local/KHQ/brianna.major/itkwidgets/itkwidgets/imjoy.py", line 53, in register_itkwasm_imjoy_codecs_cli
    server.register_codec({'name': 'itkwasm-image', 'type': itkwasm.Image, 'encoder': encode_itkwasm_image})
TypeError: 'NoneType' object is not callable

bnmajor · 2023-07-11T21:38:23Z

Hi, sorry, missed another place which requires change, now added as a test case. It should work with hypha>=0.15.28 (available soon).

Thanks for all of the updates and fixes! Unfortunately in testing out the functionality a bit more I think that this may not be the move for our use case. We are using the services to trigger the next steps in the pipeline server-side, and this requires sharing some global state variables. Using the the startup function creates a new instance of the module so this global state is not available. If I am maybe missing something here please let me know though!

bnmajor · 2023-07-11T21:38:31Z

@thewtex This PR currently:

Is started with the cli entry point: itkwidgets --image {...} {...optional args}
Can accept a --verbose flag to output stdout to the terminal again, otherwise output is surpressed
Can accept a --repl flag which will automatically start an interactive REPL session
- This session makes the viewer object available and this is explained in the banner created with the session
The issue mentioned above regarding setting the label image seems to be the same issue described [before] (Cli #589 (comment)). This issue happens at the same place (in updateRenderedImage) and only for setImage and setLabelImage where we queue the functions. Creating and calling a simple asynchronous function succeeds.

oeway · 2023-07-11T22:52:27Z

Thanks for all of the updates and fixes! Unfortunately in testing out the functionality a bit more I think that this may not be the move for our use case. We are using the services to trigger the next steps in the pipeline server-side, and this requires sharing some global state variables. Using the the startup function creates a new instance of the module so this global state is not available. If I am maybe missing something here please let me know though!

I see, then I think you are right, startup functions won't work in this case.

thewtex · 2023-07-13T19:17:35Z

itkwidgets/standalone_server.py

+            exitmsg = "Exiting REPL. Press CTRL+C to teminate CLI tool."
+            code.interact(banner=banner, local={"viewer": viewer}, exitmsg=exitmsg)
+
+


Can we also add:

if __name__ == "__main__": main()

?

Sure! But should this be

if __name__ == "__main__": cli_entrypoint()

instead so that we still parse any flags?

bnmajor · 2023-07-16T21:44:30Z

@thewtex The bug that we discussed where we cannot load a label image still exists but it is not obvious why... Thanks to @PaulHax we know that the exact issue is that getItem is failing for some reason (here). What I know so far:

If we pass the image and label data to the client via the inputObject service function and then we pass in all of the data without the label image we can call await viewer.setLabelImage right after viewer resolves and it works.
If we try to call viewer.set_label_image (the wrapper for the client-side viewer.setLabelImage function) from the Python REPL session it fails at getItem.
If we create a client-side service function that calls viewer.setLabelImage and pass it the image after it is loaded from the Python REPL session it fails at getItem.
All approaches are taking the same path on the server-side to pass the same multiscale zarr store to the client.
The only other difference I can see between the successful and failed approaches is that it appears that any time the client is in charge of requesting the data we have no issues, but any time that the server sends the data without a client request it fails. This makes me think maybe there is a codec issue? But if so shouldn't we see the same issue in the notebooks where we register the same codecs and pass the data from the server in the same manner?

@oeway I am leaning towards this being something that I am doing wrong (or fundamentally misunderstanding) rather than an issue in the itk-vtk-viewer code, so if you have any thoughts or ideas I would welcome them!

thewtex · 2023-07-18T20:48:52Z

@bnmajor great work debugging.

Another debugging session with @PaulHax , a few more interesting points:

We confirmed that getItem is not returning on the JavaScript side, but only with the array chunks -- it succeeds for .zattrs and .zarray.
We tried a few more approaches, and we confirmed your observation that it succeeds only when pulling from the JavaScript side. The call flow: set_label_image passes a remote zarr store on which the JS side will make remote procedure calls before completing do not succeed.

It appears that there is some issue with our syncified Python RPC that requires JS RPC calls before returning?

In any case, is there a workaround that we can use to behave like the initial input data objects, pulling client-side?

oeway · 2023-07-18T22:46:37Z

@bnmajor great work debugging.

Another debugging session with @PaulHax , a few more interesting points:

We confirmed that getItem is not returning on the JavaScript size, but only with the array chunks -- it succeeds for .zattrs and .zarray.

We tried a few more approaches, and we confirmed your observation that it succeeds only when pulling from the JavaScript side. The call flow: set_label_image passes a remote zarr store on which the JS side will make remote procedure calls before completing do not succeed.

It appears that there is some issue with our syncified Python RPC that requires JS RPC calls before returning?

In any case, is there a workaround that we can use to behave like the initial input data objects, pulling client-side?

I think what you are observing is related to a dead lock in the syncified python RPC. It happens because we execute the syncified functions in the main thread. If we have a service function A registered, normally, it will respond to remote calls from the js side. However, if we initiate a call from the same main thread to call something on the js side, then the js side try to call A again, the thread gets locked because it will occupy the thread and function A will never get executed.

I need to check if I can solve it inside the rpc library. But for now, an easy way to solve this is to create two server connections and use one to register the internal services, and the other one for accessing from the Python REPL.

Edit: Fixed now, I increased the worker number in the thread executor of the syncifier and it seems fixes the issue of deadlock. Please upgrade imjoy-rpc to 0.5.42 and let me know if it works.

Thanks for reporting!

@thewtex @bnmajor

thewtex · 2023-07-20T00:47:35Z

itkwidgets/standalone/index.html

+        config: {
+          visibility: "protected",
+          require_context: false,
+          run_in_executor: true,


This won't have any effect in js.

thewtex · 2023-07-21T14:18:51Z

I added a commit that adds terminal output by default! 💻 🤓 We can still open in a browser with the -b or --browser flag, but this enables quick terminal-based inspection of 3D data. It requires a terminal that supports the iterm2 inline image protocol such as:

wezterm
VSCode's Terminal (with the latest version of VSCode)
iTerm2

In VSCode:

itkwidgets-vscode.webm

thewtex · 2023-07-21T14:23:52Z

VSCode terminal + tmux. Requires set-option -g allow-passthrough on in ~/.tmux.conf.

itkwidgets-vscode-tmux.webm

thewtex · 2023-07-21T14:27:00Z

wezterm on Linux:

itkwidgets-wezterm-linux.webm

thewtex · 2023-07-21T17:20:45Z

Verified on Windows 10, wezterm, ssh'ing into an ARM mac:

and Windows ssh'ing into a headless Linux:

wezterm with Windows 11 WSL with itkwidgets running on Windows also works 🎉 🌮

Install command will be:

pip install 'itkwidgets[cli]'
playwright install --with-deps chromium

Try to pull data from client-side rather than pushing from server-side

A few tweaks are also made to the cli flags for useability. Install: ```sh pip install 'itkwidgets[cli]' playwright install --with-deps chromium ``` Requires a terminal that supports the iterm2 inline image protocal, e.g. wezterm, iterm2, VSCode Terminal.

`import js` will succeed.

bnmajor · 2023-07-25T21:15:16Z

@thewtex We can now set images and label images! 🙂

Edit: Fixed now, I increased the worker number in the thread executor of the syncifier and it seems fixes the issue of deadlock. Please upgrade imjoy-rpc to 0.5.42 and let me know if it works.

@oeway Thank you for this! I apologize, I have been out for a few days. I did have a chance to try out your changes but unfortunately I was still seeing the same issue. I did end up with a workaround where I am saving the zarr store server-side and then using services to trigger the client-side to pull the store and set the image/label. I found that this exact same approach would fail if I used a service to save the store server-side though (and any attempts to push the store rather than pull it always faill) so I am still not sure that I understand why my current solution actually works. If you have any ideas I would love to understand this better but for now at least we are able to use the full API from the REPL session! Thank you!!

thewtex

💯 🍾 🎇 💖

WooT! wOOt!

As discussed with @PaulHax , for performance reasons, we are moving to having all the input datasets managed by a DataManager for the viewer that lives in a WebWorker. In that case, we will have a separate JS event loop in the web worker and separate connections to hypha -- so the workaround we have now will likely be a non-issue.

thewtex assigned bnmajor Dec 22, 2022

thewtex mentioned this pull request Jan 7, 2023

Widget not showing on VSCode with Remote SSH #411

Open

thewtex mentioned this pull request Feb 14, 2023

Add standalone Python / WebSocket Deployment document #467

Closed

bnmajor force-pushed the cli branch 2 times, most recently from 887492b to fdecb0c Compare February 21, 2023 15:29

bnmajor force-pushed the cli branch from fdecb0c to d088a7f Compare June 6, 2023 13:51

bnmajor force-pushed the cli branch from e6e2fcb to 1cc1be5 Compare June 19, 2023 19:33

oeway reviewed Jun 19, 2023

View reviewed changes

itkwidgets/standalone/index.html Outdated Show resolved Hide resolved

oeway reviewed Jun 19, 2023

View reviewed changes

itkwidgets/standalone/index.html Outdated Show resolved Hide resolved

bnmajor force-pushed the cli branch 2 times, most recently from 7365d15 to b667943 Compare June 26, 2023 11:37

bnmajor force-pushed the cli branch from ca9ea7c to 8605423 Compare July 6, 2023 15:30

bnmajor force-pushed the cli branch from 8605423 to 0e7ec08 Compare July 6, 2023 16:21

thewtex commented Jul 6, 2023

View reviewed changes

thewtex commented Jul 13, 2023

View reviewed changes

bnmajor added 3 commits July 16, 2023 16:47

BLD: Update CLI deps and env check

86ac98c

BUG: Fix call to Viewer - call view instead

b04537a

MAINT: Remove unsed Python 3.7 dependency

e9e63f6

oeway mentioned this pull request Jul 18, 2023

Fix thread locking by using multiple executor imjoy-team/imjoy-rpc#550

Merged

bnmajor added 2 commits July 19, 2023 18:34

BLD: Bumpy hypha to 0.5.42

e7c58c6

ENH: Add __main__ entrypoint

d6e19f2

thewtex commented Jul 20, 2023

View reviewed changes

thewtex force-pushed the cli branch from fee85ac to 79e301b Compare July 21, 2023 14:20

thewtex force-pushed the cli branch from 79e301b to 583ffb7 Compare July 21, 2023 17:02

bnmajor and others added 4 commits July 25, 2023 16:55

WIP: set_label_image

cb637c9

Try to pull data from client-side rather than pushing from server-side

ENH: Terminal CLI output

41a18d9

A few tweaks are also made to the cli flags for useability. Install: ```sh pip install 'itkwidgets[cli]' playwright install --with-deps chromium ``` Requires a terminal that supports the iterm2 inline image protocal, e.g. wezterm, iterm2, VSCode Terminal.

BUG: Fix hypha detection on Windows

5c5947c

`import js` will succeed.

MAINT: Some code cleanup and simplification

64ae3bf

bnmajor force-pushed the cli branch from 583ffb7 to 64ae3bf Compare July 25, 2023 21:08

bnmajor marked this pull request as ready for review July 26, 2023 12:28

thewtex commented Jul 26, 2023

View reviewed changes

thewtex merged commit 82b2181 into InsightSoftwareConsortium:main Jul 26, 2023

thewtex deleted the cli branch July 26, 2023 17:52

		exitmsg = "Exiting REPL. Press CTRL+C to teminate CLI tool."
		code.interact(banner=banner, local={"viewer": viewer}, exitmsg=exitmsg)

Cli #589

Cli #589

Conversation

thewtex commented Dec 22, 2022

thewtex commented Dec 22, 2022

thewtex commented Jun 6, 2023

bnmajor commented Jun 6, 2023

thewtex commented Jun 9, 2023

oeway commented Jun 10, 2023

thewtex commented Jun 11, 2023

thewtex commented Jun 11, 2023

oeway commented Jun 11, 2023 • edited Loading

bnmajor commented Jun 19, 2023

oeway commented Jun 26, 2023

oeway commented Jun 28, 2023

bnmajor commented Jul 6, 2023 • edited Loading

bnmajor commented Jul 6, 2023

thewtex Jul 6, 2023

Choose a reason for hiding this comment

bnmajor Jul 11, 2023

Choose a reason for hiding this comment

oeway commented Jul 9, 2023

oeway commented Jul 10, 2023

bnmajor commented Jul 10, 2023

bnmajor commented Jul 11, 2023

bnmajor commented Jul 11, 2023

oeway commented Jul 11, 2023

thewtex Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

bnmajor Jul 16, 2023

Choose a reason for hiding this comment

thewtex Jul 18, 2023

Choose a reason for hiding this comment

bnmajor commented Jul 16, 2023 • edited Loading

thewtex commented Jul 18, 2023 • edited Loading

oeway commented Jul 18, 2023 • edited Loading

thewtex Jul 20, 2023

Choose a reason for hiding this comment

oeway Jul 20, 2023

Choose a reason for hiding this comment

thewtex commented Jul 21, 2023

thewtex commented Jul 21, 2023

thewtex commented Jul 21, 2023

thewtex commented Jul 21, 2023

bnmajor commented Jul 25, 2023

thewtex left a comment

Choose a reason for hiding this comment

oeway commented Jun 11, 2023 •

edited

Loading

bnmajor commented Jul 6, 2023 •

edited

Loading

thewtex Jul 13, 2023 •

edited

Loading

bnmajor commented Jul 16, 2023 •

edited

Loading

thewtex commented Jul 18, 2023 •

edited

Loading

oeway commented Jul 18, 2023 •

edited

Loading