Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plugin: grab-canvas #366

Closed
TimPietrusky opened this issue Jun 12, 2020 · 12 comments · Fixed by #612
Closed

Plugin: grab-canvas #366

TimPietrusky opened this issue Jun 12, 2020 · 12 comments · Fixed by #612
Assignees
@TimPietrusky
Labels
enhancement Feature requests regression A feature that existed before that either no longer exists or works as intended released

Comments

@TimPietrusky
Copy link
Member

In modV2 we had the plugin grab-canvas that was used in combination with https://github.com/NERDDISCO/luminave to exchange color-information that is grabbed live from the output-canvas to control lights.

Idea: API via GraphQL

Maybe it would make sense to not directly connect to luminave over Websocket, but to provide an API via GraphQL that provides a subscription (which is also WebSocket), so that other applications could get that information out of modV, too.

Dependency

vcync/modv-3#133 needs to be done if we want to make this a plugin, but maybe if we decide that modV needs a general GraphQL API, it would go into core.
@2xAA
Copy link
Member

2xAA commented Jun 12, 2020

grab-canvas is a completely proprietary method to communicate with Luminave.

I think maybe we should look at ways of getting video data out of modV without processing it first, what do you think?

Really like the idea of a GraphQL API.
I'm not sure the GraphQL API would be super necessary for grab-canvas if we were able to use something like a frameserver to share video frames.

Luminave would have to grab the frames and process them separately, but I don't think this would be such a bad thing.

@TimPietrusky
Copy link
Member Author

TimPietrusky commented Jun 12, 2020 •
edited
Loading

What do you have in mind for a frameserver? Are you thinking about the Screen Capture API?

And don't you think we should process some stuff, so that other applications that don't know how to process the video, but just want to use some processed data, can also benefit from this?

@2xAA
Copy link
Member

2xAA commented Jun 12, 2020

I didn't have anything in mind for the frameserver exactly. Just putting it out there as a route to explore.

I'm concerned that we may end up inventing a new way of communicating between our audio/visual apps and isolate our apps from playing nicely with other applications.

We may want to process some stuff on modV's side. I'd be more comfortable to standardise the visual output of modV and let applications consume and manipulate however they like. Though I can see the benefit of keeping the existing grab-canvas API in modV.

Were the performance issues of grab-canvas fixed in the end?

Again I really like the idea of GraphQL, I think that could be very useful. Especially if we think about bringing things like a mobile controller to modV again (this was available in versions before 1.5).

@TimPietrusky
Copy link
Member Author

@2xAA can we please have a meeting about this and talk about the different scenarios that we want to achieve? And then create the correct tickets.

@2xAA
Copy link
Member

2xAA commented Jun 12, 2020

Yes, absolutely!

@TimPietrusky TimPietrusky self-assigned this Jun 15, 2020
@TimPietrusky
Copy link
Member Author

Scale the image down to the correct size and it should scale it down to the exact pixels that we need.

@2xAA
Copy link
Member

2xAA commented Jul 1, 2020 via email

@TimPietrusky
Copy link
Member Author

PoC: https://codesandbox.io/s/grab-canvas-image-scale-bcdfr?file=/src/index.js

@2xAA 2xAA transferred this issue from vcync/modv-3 Jul 2, 2020
@2xAA 2xAA added enhancement Feature requests regression A feature that existed before that either no longer exists or works as intended labels Jul 2, 2020
@2xAA
Copy link
Member

2xAA commented Jul 4, 2020

@TimPietrusky, do you think we should split out the GraphQL idea into a separate issue?

@TimPietrusky
Copy link
Member Author

@2xAA yes, that makes total sense!

2xAA added a commit that referenced this issue Jun 22, 2021
@2xAA
feat(grabcanvas): adds grabCanvas plugin
da30ec8 
also adds the plugin panel and adds some plugin lifecycle functions

#366
@2xAA
Copy link
Member

2xAA commented Jun 22, 2021

@2xAA yes, that makes total sense!

oh good :)

2xAA added a commit that referenced this issue Jun 23, 2021
@2xAA
feat(pluginprops): adds plugin props
8302b54 
splits the Control component out into ModuleControl and PluginControl to allow code re-use. Adds
store methods for plugin props.

re #366
@TimPietrusky TimPietrusky mentioned this issue Jul 1, 2021
Merged
@2xAA 2xAA closed this as completed in #612 Jul 1, 2021
2xAA added a commit that referenced this issue Jul 1, 2021
@TimPietrusky @2xAA
feat(grabCanvas): Adds grab-canvas as a plugin (#612)
d7ca887 
* feat(grabcanvas): adds grabCanvas plugin

also adds the plugin panel and adds some plugin lifecycle functions

#366

* feat(pluginprops): adds plugin props

splits the Control component out into ModuleControl and PluginControl to allow code re-use. Adds
store methods for plugin props.

re #366

* fix: removes unnecessary gl-stack for plugins

* feat: adds infoview directive for plugins panel

* feat: adds searchterms directive

* fix(grab-canvas): Clear mappingContext on every frame

* refactor(grab-canvas): Set default size to 7x7

* refactor(grab-canvas): Proper reconnect and restructure of code

Co-authored-by: Sam Wray <[email protected]>
2xAA pushed a commit that referenced this issue Jul 1, 2021
@semantic-release-bot
chore(release): 3.18.0
cd07cba 
# [3.18.0](3.17.1...3.18.0) (2021-07-01)

### Bug Fixes

* updates a group's outputContext name when group's name is updated ([#604](#604)) ([a0db3c9](a0db3c9)), closes [#599](#599)
* **deps:** Use correct format to load vcync/interactive-shader-format-js ([553f4f9](553f4f9))
* fixes loading presets with textures referencing groups ([#591](#591)) ([9dfeff5](9dfeff5)), closes [#590](#590) [#590](#590)
* **mathjs:** updates mathjs to v7 from v3 ([#595](#595)) ([32585cc](32585cc)), closes [#594](#594)

### Features

* **fft:** adds fft ([#611](#611)) ([f9f2b90](f9f2b90))
* **grabCanvas:** Adds grab-canvas as a plugin ([#612](#612)) ([d7ca887](d7ca887)), closes [#366](#366) [#366](#366)
* adds localStorage saving for audio and video inputs ([#608](#608)) ([33109cd](33109cd)), closes [#470](#470)
* **gallery:** updates ordering of modules to be alphabetical ([#606](#606)) ([48f0075](48f0075)), closes [#605](#605)
* **group-inheritance:** adds group inheritance option ([#607](#607)) ([1894eda](1894eda)), closes [#603](#603)
@2xAA
Copy link
Member

2xAA commented Jul 1, 2021

🎉 This issue has been resolved in version 3.18.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@2xAA 2xAA added the released label Jul 1, 2021
2xAA pushed a commit that referenced this issue Jan 7, 2022
@semantic-release-bot @2xAA
chore(release): 3.18.0
3663754 
# [3.18.0](3.17.1...3.18.0) (2021-07-01)

### Bug Fixes

* updates a group's outputContext name when group's name is updated ([#604](#604)) ([a0db3c9](a0db3c9)), closes [#599](#599)
* **deps:** Use correct format to load vcync/interactive-shader-format-js ([553f4f9](553f4f9))
* fixes loading presets with textures referencing groups ([#591](#591)) ([9dfeff5](9dfeff5)), closes [#590](#590) [#590](#590)
* **mathjs:** updates mathjs to v7 from v3 ([#595](#595)) ([32585cc](32585cc)), closes [#594](#594)

### Features

* **fft:** adds fft ([#611](#611)) ([f9f2b90](f9f2b90))
* **grabCanvas:** Adds grab-canvas as a plugin ([#612](#612)) ([d7ca887](d7ca887)), closes [#366](#366) [#366](#366)
* adds localStorage saving for audio and video inputs ([#608](#608)) ([33109cd](33109cd)), closes [#470](#470)
* **gallery:** updates ordering of modules to be alphabetical ([#606](#606)) ([48f0075](48f0075)), closes [#605](#605)
* **group-inheritance:** adds group inheritance option ([#607](#607)) ([1894eda](1894eda)), closes [#603](#603)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TimPietrusky TimPietrusky

Labels
enhancement Feature requests regression A feature that existed before that either no longer exists or works as intended released
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(grabCanvas): Adds grab-canvas as a plugin vcync/modV
2 participants
@TimPietrusky @2xAA