Live deployment on Cloudflare Pages available here.
Demo version of the gesture tracking app developed at RNCM PRiSM for the piece Forager, by composer and computer music pioneer George Lewis.
The app uses a k-Nearest-Neighbours (k-NN) model with Dynamic Time Warping (DTW) to track a specific set of musical gestures in real time.
For pitch estimation it makes use of CREPE. The demo is inspired by the (Chrome only) CREPE demo site.
This is a mono version of the app using the Web Audio API, and was not used in the first performance of Forager.
The full code for the production app will be released by RNCM PRiSM shortly. The production version uses SoX - the 'Swiss Army knife' of sound processing - for audio capture. It is multi-channel, and works with a mic or audio interface connected to a local machine.
Lead Developer: Dr Christopher Melen (PRiSM Research Software Engineer).
With invaluable contributions from Professor David De Roure, Dr Bofan Ma, George Lewis and Damon Holzborn.