WorkerTaskManager #19650

kaisalmen · 2020-06-13T21:20:55Z

Last Update 2021-01-28

The TaskManager allows to register a task expressed by an initialization and an execution function with an optional comRounting function to be run a web worker. It creates one to a maximum number of workers that can be used for execution. Multiple execution requests can be handled in parallel of the main task, If all workers are currently occupied the requested are enqueued and the returned promise is fulfilled once a worker becomes available again.

What the implement status?

Functionality outlined in TaskManager: Proposed worker management class #18234 has been implemented with the exception of cost parameter. Some additional features are already available.
Standard workers are implemented including Main-Execution fallback
Dependency loading for standard workers is available
Module workers are implemented, dependencies are declared regularly and therefore no extra functionality is required
exec and init functions can be declared in module and then be packaged in standard worker if needed. I used this to define the worker code once and support both code paths
A new example is supplied
- It uses a dat.gui to configure the example. Let it run, stop and reset.
- It allows to select all outlined worker types (standard workers with dependencies, module workers that declare dependencies if required and standard workers that are executed on main).
- It also allows to set the maximum amount of workers, the overall executions count and the number of meshes to keep.
- It is a potentially infinitely running example executing over and over the same workers, This has already proven to be very helpful in identifying memory holes and in verifying that CPU is utilized as expected.
JSDoc has been completed and TypeScript definitions are generated for free from it, see WorkerTaskManager #19650 (comment)
it just utilizes the parser
Code has been moved to examples/jsm/taskmanager
Class notation eases creation of worker code in case of standard workers.
Execution queue has been added to WorkerTaskManager

OBJLoader2

OBJLoader2 and OBJLoader2Parallel have been removed from three.js with R125. I will ensure it will work with WorkerTaskManager and also use simplified worker transfer functions (for all interested see here)

TODO

Solve concurrent init issues when the same TaskManager is passed to multiple instances of the same loader/class requesting init
Optimize and review execution queue in enqueueForExecution and _kickExecutions
Ongoing Complete documentation
- Example
- JSDoc for typescript defintions
Ongoing Create a wrapper for OBJLoader (standard and module version).
Ongoing Provide easy to use utility functions to transfer BufferGeometry, Material (meta-information) and Mesh bi-driectionally between Main and workers. All "transferables" shall be treated as such. This replaces old functions available with OBJLoader2

Questions

Shall we add cost parameter later?
Is the example too complex?

Later

On init the maximum amount of possible workers for all tasks are created. This is time consuming. I could makes sense to make the init lazy (= create extra worker when needed until max. is reached), but this adds complexity to initialization especially when Transferables are involved as they need to be cached and cloned.

trusktr · 2020-06-14T04:36:21Z

(There's a "Draft" feature if the PR is in progress. Useful for signaling to others it is WIP. Under the "Reviewers" section of the right sidebar, where it says "Still in progress? Convert to draft")

donmccurdy

Thanks, this is looking good! A few comments and questions.

donmccurdy · 2020-06-14T23:35:00Z

examples/jsm/loaders/TaskManager.js

+ * @author Kai Salmen / https://kaisalmen.de
+ */
+
+import { FileLoaderBufferAsync } from "./obj2/utils/FileLoaderBufferAsync.js";


Is this import from the obj2 folder necessary? It looks like it just adds a custom onProgress callback to the FileLoader, which may not be applicable to all implementations? If it is necessary it should probably be moved into the TaskManager file.

The necessity for it is gone. I created it for other prototyping activity and used it because it was there. With the introduction of loadAsync in Loader it collapsed to what it is now and I agree whatever is needed (using FileLoader with setResponseType( 'arraybuffer') should be integrated directly in TaskManager.

donmccurdy · 2020-06-14T23:40:56Z

examples/jsm/loaders/TaskManager.js

+     * @param {String[]} [dependencyUrls]
+     * @return {TaskManager}
+     */
+    registerType ( type, maximumWorkerCount, initFunction, executeFunction, comRoutingFunction, dependencyUrls ) {


Are there benefits to having a separate maxWorkerCount for each type of task? The downside, I think, is that you can't effectively load balance across different types of task. If any worker can do any task, that balancing could be easier.

Maybe there is need for an overall count and a desired per taskType count. Was that your idea behind cost parameter?

If any worker can do any task, that balancing could be easier.

Is this a realistic scenario?

I consider the cost parameter unrelated to whether or not there are separate worker queues for different task types.

My original idea had been to have a configurable number of workers for the entire TaskManager, say N. Each worker can perform any type of task, and all tasks are balanced across those N workers. So there would be no per-task worker limit, just the TaskManager total limit.

Is this a realistic scenario?

Having multiple tasks running in parallel is realistic, at least. For example, when using both compressed geometry and compressed textures you'll need DRACOLoader and (one of) KTX2Loader or BasisTextureLoader.

Whether there are very many realistic situations where shared queues are likely to give better performance? I'm not sure. If you think this is more complicated, or worse for some other reason, I'm also fine with leaving this as you have it for now.

My original idea had been to have a configurable number of workers for the entire TaskManager, say N. Each worker can perform any type of task, and all tasks are balanced across those N workers. So there would be no per-task worker limit, just the TaskManager total limit.

Ok, now I see your point. Then we need to create N workers for every taskType either lazy or on init (cost are only extra memory and init time what becomes negligible if Workers really do intensive work). Then we can always use maximum of N for every taskType depending on current demand.
From my point of view the Worker should only include the code domain (init, exec) of one task as otherwise I fear mixing task and their dependencies into the worker will introduce unnecessary complexity and mixing module and non-module code will not be manageable. I hope this makes sense.

Then we need to create N workers for every taskType... Then we can always use maximum of N for every taskType depending on current demand.

This is a bit different than what I meant. Something like...

var manager = new TaskManager().setWorkerLimit(2); var objLoader = new OBJLoader(manager); var ktx2Loader = new KTX2Loader(manager);

... would create only two Web Workers total, no matter how many loaders are using that manager, and no matter how many resources are loaded. OBJ and KTX2 tasks could be balanced across those two workers, and a single worker's queue would have tasks of both types.

There are some very real advantages to that approach: you have firm control over the total number of Web Workers, and work is evenly distributed. But the concern of mixing module and non-module code makes sense. I'm not sure the module worker type is going to be worth a whole separate API in the long term, but I don't want to hold up this PR on the question – it's fine with me to continue as you have the PR for now.

When tasks are add/executed it only runs two in parallel ...

This will be very complicated to enforce. What happens if we get requests in this order:

10 OBJ parse tasks

2 KTX2 parse tasks

After step (1) I would expect that the manager assigns ~5 OBJ tasks to each OBJ worker. But then when it gets to step (2) there are already two workers running, do we have to wait until one of the workers clears its queue before assigning KTX tasks to a KTX worker? What if more OBJ tasks arrive in the meantime?

I would prefer to avoid all that... maybe we have a setWorkerLimit( n ) method on the TaskManager, and for now it will create N workers per task type and use them all. Then in the future, if we decide to, we can try to combine the worker pool so that tasks share workers for more equitable load balancing.

Add task puts the processing requests to a fifo queue (currently Promises are stored in array). Whenever a worker signals exec complete a task is taken from the queue and a fitting worker is kicked. What is done in the current TaskManager/example is not so far off expect for the overall count is per task type and not globally, but tasks are already intermixed in the queue. But getting there not be hard. I will prototype this.

I see! I wonder whether the time to signal the main thread that a task is done and then wait for a new task to come back has a measurable impact on completion times? Thinking of a model with 100+ Draco geometries for example... that doesn't seem like much of a problem though.

In lastest commit the maximum amount (N) of parallel worker execution is now handled on TaskManager level. N workers are created of each type (except for main fallback), but only the maximum defined in TaskManager are executed in parallel in the example.

The previous code did not work as described. I pushed another update that really works as described.
Init time of workers increases linearly with quantity even if Blobs/Strings are cached. For module workers there seems no obvious way to cache code except for pre-loading the module in the embedding html.

Execution on main is still strange. The FakeTaskWorker is now treated identically during execution, but when it solely used in the example than it completely blocks any rendering. When it is intermixed with real workers, then the rendering is not blocked. Manual triggering of requestAnimationFrame did not help. For now I leave it as it is

donmccurdy · 2020-06-14T23:42:36Z

examples/jsm/loaders/TaskManager.js

+     * @param {string} type The type as string
+     * @return boolean
+     */
+    supportsType ( type ) {


Minor: I think I find the term "task" or "taskType" clearer than just "type" here, and for the methods below.

Yes, this is inconsistent. I obviously forgot to rename it everywhere. Will change this.

Changed in latest commit.

donmccurdy · 2020-06-14T23:54:11Z

examples/jsm/loaders/TaskManager.js

+     * @param {string} workerJsmUrl The URL to be used for the Worker. Module must provide logic to handle "init" and "execute" messages.
+     * @return {TaskManager}
+     */
+    registerTypeJsm ( type, maximumWorkerCount, workerJsmUrl ) {


I've never seen the term "jsm" anywhere outside the name of this folder, so I'd prefer to use a clearer term here. It's also important that even though the "examples/js" folder is being removed later this year, that has very little to do with whether this "jsm" method should be used — DRACOLoader, BasisTextureLoader, and KTX2Loader will all use the non-JSM register method. What do you think of:

// Default. manager.registerTask( type, maxCount, initFn, execFn, ... ); // URL. (undecided which name..) manager.registerTaskModule( type, maxCount, moduleUrl ); manager.registerTaskUrl( ... ); manager.registerTaskSrc( ... );

I think the fact that the script happens to be an ES Module is less important than the fact that it's an external script. That means any user with a build system like webpack or browserify will need to specifically configure their application to output files in the right directories.

Agreed, let's remove jsm from method signatures. module is the best replacement term as it is also in line with the naming of the Worker constructor parameter, I think.

Changed in latest commit.

donmccurdy · 2020-06-14T23:55:59Z

Should we adjust loaders in scope of this PR or do it later?

I think one loader in this PR is probably enough to get the general idea.

…skManager. Maximum number of workers for every type are created, but only maximum number is executed. Removed the need for FileLoaderBufferAsync

TaskManager: - Adjusted naming (taskType and module) - Reorganize execution flow and renamed addTask to enqueueForExecution - Align FakeWorker main execution behaviuor with TaskWorkers: The maximum no of workers is created for all types webgl_loader_taskmanager: - Allow to define overall execution, loop count and meshes to keep visible - Adjusted naming (taskType and module)

kaisalmen · 2020-06-30T13:01:47Z

I think one loader in this PR is probably enough to get the general idea.

Do you have any preference? My comfort zone is OBJ, but I am also aware others loaders are more demanded these days. 😉 Should we collaborate? Is R119 a possible target?

kaisalmen · 2020-07-06T20:18:01Z

Cool, you can use the TypeScript compiler to generate ts.d files from the jsdoc declarations. Create a config file like this:

{
  "include": [
    "examples/jsm/loaders/TaskManager.js"
  ],
  "compilerOptions": {
    "allowJs": true,
    "declaration": true,
    "noEmit": false,
    "emitDeclarationOnly": true,
    "removeComments": true
  }
}

Write good jsdoc and get the definitions for free. 😄

kaisalmen · 2020-07-31T21:00:42Z

@donmccurdy and @mrdoob I have fully updated the description. R120 is a realistic target, I think. Sorry it took so long. If I could do this work in my regular job, we would have been here months ago. I should be able to solve open issues in the upcoming weeks. So, if you don't identify potential blockers, then we could merge this before end of August!

kaisalmen · 2020-08-26T07:29:07Z

@donmccurdy and @mrdoob I didn't make it for R120, but the TaskManager is now feature complete. OBJLoader2Parallel has been fully adapted to it. I stumbled upon some nasty async init problems while adapting it to the TaskManager especially when sharing the TaskManager in different loader instances. These are now under control.
I updated the PR description.

Shall we rename the TaskManager to something else? The name is wired to the Windows TaskMananger in my head. What's your opinion?

donmccurdy · 2020-08-28T18:14:05Z

Thanks @kaisalmen! I hope to review this in detail soon...

I also associate the name with windows task manager, but it seems like a pretty good name. I like the "Manager" suffix for its similarity to the existing LoadingManager. Maybe WorkerManager? Either that or TaskManager seem fine to me.

What do you mean by "legacy workers" in the PR description?

How confident are you in the updates to the various loaders that depend on this? I guess I'm wondering if we should try to get in the TaskManager itself first, as a clean PR with nothing depending on it, or merge this all at once.

kaisalmen · 2020-08-29T19:41:31Z

Maybe WorkerManager? Either that or TaskManager seem fine to me.

Then let's keep the name TaskManager. The name is good and fits. It is just the association, but that's alright. 😄

What do you mean by "legacy workers" in the PR description?

Classic, normal or standard workers, so one can distinguish between Module Workers and let's say Standard Workers instead of just Workers. What do you think is the best term? Standard Worker? It's really a pitty Module Workers are not supported by Firefox or Safari.

How confident are you in the updates to the various loaders that depend on this?

I have removed the need WorkerExecutionSupport and accompanying code from OBJLoader2Parallel and I have verified that all examples work (also the additional examples/tests I have in my dev repo https://github.com/kaisalmen/WWOBJLoader/tree/TaskManagerPrototype). Apart from that no other Loader has been touched, yet.
I also think that the new worker creation approach is better. Using the class notation simplifies the creation of strings (whole class or just single functions).
Separation would work, but I need to change the example and merge the OBJLoader2 related changes afterwards. Take your time to look at the code and then we can discuss?!

Resolved 2020-08-30: I need to look again at the execution loop of the TaskManager. It behaves like a thread that sleeps for a short while. I had created an endless loop and that is the current state of the solution, but it may not be good enough as it slows down processing, but seems robust at least.

donmccurdy · 2020-08-29T19:55:45Z

I'm worried about giving the impression that Module Workers are the better/recommended option, and using the term "legacy" to describe the alternative gives that impression. The key difference between the two is not the use of ES Modules, but the fact that Module Workers rely on external scripts with a pre-determined URL. That's going to be very hard to use with modern bundlers without requiring users to individually put their dependencies in the right place.

Note that we are in the process of deprecating examples/js in favor of ES Modules in examples/jsm, which has no particular relation to Module Workers. I expect that the loaders I'll update to use TaskManager (BasisTextureLoader, DRACOLoader, and KTX2Loader) will be ES Modules themselves, but will not use Module Workers. I'm worried that the Legacy/Module naming will be pretty confusing for users in that process.

kaisalmen · 2020-08-30T20:57:51Z

It was not my intention to make a recommendation here. I have already adjusted the wording to standard in the description. Honestly, Module Workers can't even be preferred as only Chromium-based browsers are able to use them. TaskManager is not biased here as you can load dependencies and add extra code pieces for standard workers. Standard and module workers have feature parity.

but the fact that Module Workers rely on external scripts with a pre-determined URL. That's going to be very hard to use with modern bundlers without requiring users to individually put their dependencies in the right place.

But, isn't that even more problematic when you use importScripts in standard workers? You can use the same module in main and module workers without problems (see OBJ2LoaderWorker usage in OBJLoader2Parallel for creation of both worker types https://github.com/kaisalmen/three.js/blob/TaskManagerProto/examples/jsm/loaders/OBJLoader2Parallel.js#L119-L130). Logically, I don't see why bundlers should not be able to handle this. But I need to acquire more knowledge regarding the js bundler black magic.

What I esthetically like about module workers is that they remove the inconsistency between main and worker. With standard workers you have to use a different syntax in the worker (importScripts vs. module syntax). Modules on both sides may also encourage one to write things only once. 😄

kaisalmen · 2020-09-02T21:05:06Z

Finally, the PR is ready for review. 🚀 I am still updating the code comments in the new example, but this does not block the review from my point of view. Wording regarding workers (standard and module) has been adapted in the new example,

mrdoob · 2020-09-03T08:13:46Z

How about naming this WebWorkerManager?

kaisalmen · 2020-09-03T20:01:07Z

... or WorkerTaskManager because tasks are executed by workers? I am fine with any of these names.

mrdoob · 2020-09-03T20:40:55Z

That works too 👌

Updated Typescript definitions Removed MaterialCloneInstruction

…k with the unaltered loader.

kaisalmen · 2021-09-28T18:44:37Z

@donmccurdy and @mrdoob What do we do? Are still interested with this? Is there anything I can do to make this any more desirable?

donmccurdy · 2021-10-01T05:25:30Z

Sorry for the long delay here —

I'm struggling with what role WorkerTaskManager should play, and where it should be (examples/jsm, separate repo, etc.) to fulfill that. My original motivation for the TaskManager proposal in #18234 was to consolidate worker management logic required by KTX2Loader and DRACOLoader, and WorkerPool.js has basically solved that need, with a very small amount of code.

Because this PR provides features like transferring Mesh, Material, and BufferGeometry instances over a wire, it's also about 20x larger. The serialization is valuable in its own right — like a better/faster toJSON/fromJSON — but wouldn't ideally be built into a Web Worker framework, since you might also want that feature for Web Sockets, Node.js, databases, etc. The combined package is a good fit for loading OBJ (and perhaps more complex formats like COLLADA, STEP, or USD?) but those same formats tend to be complex enough to merit their own native libraries and GitHub repositories...

Do we think it is likely that this framework is also a good fit for something like @gkjohnson's three-mesh-bvh, or doing physics in a worker with cannon-es? I'm not convinced that it is, unfortunately. In comparison, consider a combination of...

... the combination is more flexible for use cases like physics, and provides a threading implementation compatible with both Node.js and Web Workers. I'm wondering if that is the way we should go here instead.

gkjohnson · 2021-10-02T18:41:48Z

If code for transferring core types between web workers is going to be added I still strongly vote for the approach posed in #21035 or at least something that's usable separately from any worker infrastructure three.js is choosing to provide. The benefits of such an approach are not limited to use in transferring web workers, as well. Keeping geometry data as the original typed attributes (or even just cloning a typed array directly) is much faster and the typed array can be stored in browser caches like IndexedDB and sent over websockets if needed.

kaisalmen · 2021-10-02T20:37:02Z

and WorkerPool.js has basically solved that need, with a very small amount of code.

Yes, I guess your intention what should be achieved with WorkerTaskManager and what I interpreted were fairly different. Me involving OBJLoader2 from the beginning may have not been the best choice, because it brought many more requirements (support for different worker types, dependencies and no need for bundlers, etc.) to the table. Now, with it removed from three.js also the current need for the extra features in the scope of three.js is gone.

Because this PR provides features like transferring Mesh, Material, and BufferGeometry ... but wouldn't ideally be built into a Web Worker framework, since you might also want that feature for Web Sockets, Node.js, databases, etc.

Yes, I completely agree with you both. It is contained in this PR, but WorkerTaskManager is not dependent: It's just a handy tool that I needed again mostly to make OBJ2Loader work efficiently and make the example simpler. And, yes, if these transport or send-efficient-over-the-wire tools existed independently of any web worker stuff (e.g. #21035 is realized) that would be great.

... the combination is more flexible for use cases like physics, and provides a threading implementation compatible with both Node.js and Web Workers. I'm wondering if that is the way we should go here instead.

I actually thought about if using comLink makes sense. Currently, WorkerTaskManager is independent from other projects apart from three.js (and this dependency is not strong). This is good, but it may nowadays no longer be the best choice.

three-wtm, like OBJLoader2 exist as an independent projects now. three-wtm could evolve into something completely independent of three.js (with transfer tools moved to OBJ2 or no longer required). I have to think about where I want to go with it and what makes sense.

I have the feeling that this PR won't get merged because it does not bring the right value to three,js, but if you think my assumption is wrong or any of the code/work done here is useful in another scope (elsewhere in three.js or beyond), then let me know/let's talk. 🙂

kaisalmen · 2022-04-22T20:42:13Z

Dear @mrdoob @donmccurdy and @gkjohnson I will now close this PR. I have waited this long, because I wanted the external project three-wtm. to reach a good state, first. Now is the time.

The code has been transformed to TypeScript and I performed clean-up, fixed bugs and added new features (e.g. different worker count for each registered task, generic WorkerTask isolated from WorkerTaskDirector (new name)). I have isolated a core library without any dependencies, so the WorkerTaskDirector can be used in other contexts and I created an extension for three.js:
https://www.npmjs.com/package/wtd-core/v/2.0.0-beta.1
https://www.npmjs.com/package/wtd-three-ext/v/2.0.0-beta.1

The code and documentation needs some polishing, but V2.0.0 will be there early May (won't have much time to work on in the upcoming week). Maybe this work can be of to you use even if the connection to three.js is getting weaker. Feedback from you is very welcome. If you like drop by the repo. I will announce the release on Twitter.

Btw, OBJLoader2(Parallel) is still alive and is now using it as well:
https://www.npmjs.com/package/wwobjloader2/v/5.0.0-beta.1

kaisalmen · 2022-05-14T21:25:48Z

Release is almost ready. You can try the new examples here: http://wtd.kaisalmen.de/

kaisalmen marked this pull request as draft June 14, 2020 05:59

donmccurdy reviewed Jun 14, 2020

View reviewed changes

kaisalmen force-pushed the TaskManagerProto branch from 20d31aa to f6d5847 Compare June 26, 2020 21:43

kaisalmen force-pushed the TaskManagerProto branch from f6d5847 to 6593a08 Compare June 30, 2020 12:42

kaisalmen force-pushed the TaskManagerProto branch from f33c2ec to f49dda6 Compare July 23, 2020 12:00

kaisalmen force-pushed the TaskManagerProto branch from f49dda6 to 827c658 Compare July 30, 2020 13:18

kaisalmen force-pushed the TaskManagerProto branch from b1a9754 to 7643a0d Compare August 17, 2020 21:04

kaisalmen force-pushed the TaskManagerProto branch from 7643a0d to 6db95fe Compare August 25, 2020 21:47

donmccurdy mentioned this pull request Aug 28, 2020

ThreeJs ThreeMF loader is blocking the UI while upzipping the 3MF file. #20206

Closed

12 tasks

kaisalmen marked this pull request as ready for review September 2, 2020 20:57

kaisalmen changed the title ~~WIP: TaskManager~~ TaskManager Sep 2, 2020

kaisalmen force-pushed the TaskManagerProto branch from 1c871c0 to 424c4fc Compare September 2, 2020 20:58

mrdoob changed the title ~~TaskManager~~ WorkerTaskManager Sep 12, 2020

kaisalmen added 7 commits September 28, 2021 20:39

Updated documentation

607fae7

Updated Typescript definitions Removed MaterialCloneInstruction

Updated documentation of webgl_loader_workertaskmanager.html

2bd5698

Simplified example

687d35a

Removed unneeded workers

0fb53c3

Use fake function call in OBJLoader worker wrapper allowing it to wor…

67a8d4f

…k with the unaltered loader.

Fix missing EventDispatcher

07bb155

Align code with latest code from three-wtm (1.0.0-beta.6)

3405ccb

kaisalmen force-pushed the TaskManagerProto branch from 8cd5896 to 3405ccb Compare September 28, 2021 18:40

mrdoob modified the milestones: r133, r134 Sep 30, 2021

gkjohnson mentioned this pull request Oct 2, 2021

Object3D.toJSON : Add demo implementation to retain typed buffers in result #21035

Draft

mrdoob modified the milestones: r134, r135 Oct 28, 2021

mrdoob modified the milestones: r135, r136 Nov 26, 2021

mrdoob modified the milestones: r136, r137 Dec 24, 2021

mrdoob modified the milestones: r137, r138 Jan 26, 2022

mrdoob modified the milestones: r138, r139 Feb 23, 2022

mrdoob modified the milestones: r139, r140 Mar 24, 2022

kaisalmen closed this Apr 22, 2022

WorkerTaskManager #19650

WorkerTaskManager #19650

Conversation

kaisalmen commented Jun 13, 2020 • edited Loading

trusktr commented Jun 14, 2020

donmccurdy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jun 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jun 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jun 26, 2020 • edited Loading

Choose a reason for hiding this comment

kaisalmen Jun 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisalmen Jun 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

donmccurdy commented Jun 14, 2020

kaisalmen commented Jun 30, 2020 • edited Loading

kaisalmen commented Jul 6, 2020 • edited Loading

kaisalmen commented Jul 31, 2020 • edited Loading

kaisalmen commented Aug 26, 2020 • edited Loading

donmccurdy commented Aug 28, 2020

kaisalmen commented Aug 29, 2020 • edited Loading

donmccurdy commented Aug 29, 2020

kaisalmen commented Aug 30, 2020 • edited Loading

kaisalmen commented Sep 2, 2020 • edited Loading

mrdoob commented Sep 3, 2020

kaisalmen commented Sep 3, 2020 • edited Loading

mrdoob commented Sep 3, 2020

kaisalmen commented Sep 28, 2021 • edited Loading

donmccurdy commented Oct 1, 2021 • edited Loading

gkjohnson commented Oct 2, 2021

kaisalmen commented Oct 2, 2021 • edited Loading

kaisalmen commented Apr 22, 2022

kaisalmen commented May 14, 2022

kaisalmen commented Jun 13, 2020 •

edited

Loading

kaisalmen Jun 16, 2020 •

edited

Loading

kaisalmen Jun 17, 2020 •

edited

Loading

kaisalmen Jun 26, 2020 •

edited

Loading

kaisalmen Jun 30, 2020 •

edited

Loading

kaisalmen Jun 15, 2020 •

edited

Loading

kaisalmen commented Jun 30, 2020 •

edited

Loading

kaisalmen commented Jul 6, 2020 •

edited

Loading

kaisalmen commented Jul 31, 2020 •

edited

Loading

kaisalmen commented Aug 26, 2020 •

edited

Loading

kaisalmen commented Aug 29, 2020 •

edited

Loading

kaisalmen commented Aug 30, 2020 •

edited

Loading

kaisalmen commented Sep 2, 2020 •

edited

Loading

kaisalmen commented Sep 3, 2020 •

edited

Loading

kaisalmen commented Sep 28, 2021 •

edited

Loading

donmccurdy commented Oct 1, 2021 •

edited

Loading

kaisalmen commented Oct 2, 2021 •

edited

Loading