A VTube Studio plugin that allows for connectivity between heart rate monitors (HRM) and VTube Studio!
Download the latest version here! Or, in the release section of this repo.
đź’“ Support for many heart rate monitors with pulsoid.net, hyperate.io, ANT+ and even Fitbit!
đź’“ Configurable model tinting that scales with pulse!
đź’“ Automatic expression and hotkey triggering at desired heartrate thresholds!
đź’“ Custom tracking parameters for pulse and breath!
đź’“ Support for Live2D Items!
đź’“ NEW! Dynamically control VTube Studio VFX with your heartbeat!
đź’“ NEW! Plugin API so that you can build your own apps that consume or write heartrate data!
This plugin is developed by Tom "Skeletom" Farro. If you need to contact him, the best way to do so is via Twitter.
If you're more of an email-oriented person, you can contact his support email: [email protected].
Getting up and running is relatively straightforward. The plugin will automatically connect to VTube Studio on launch. From there, do the following steps:
- Input an estimated minimum and maximum heartrate.
- Select a desired heartrate input method. You can connect over bluetooth using your phone and pulsoid.net or hyperate.io, connect directly to your PC with an ANT+ USB dongle, or even use your Fitbit with a special companion app.
- Add Art Mesh Tint modules and configure them to parts of your model!
- Add Expression and Hotkey Trigger modules and configure them to activate model functions automatically!
- Hook up Custom Tracking Parameters, to your model for things like breathing speed!
A comprehensive video tutorial has also been provided as reference.
In the interest of being widely accessible, this plugin features a wide set of possible input methods, which you may freely switch between at any time.
The slider is primarily useful for quick testing of different heartrate values. The slider range is from 0 to 255.
This input method allows you to read heartrate data from an external file. The file must simply contain the numeric heartrate value in plain text. File path must be absolute. Useful if you have another program that can output heartrate data.
This input method allows you to read heartrate data from the Input API. For more information about the API, consult the API Documentation.
Pulsoid is a free third-party app for Android/iOS which allows for easy, reliable connectivity to a wide set of heartrate monitors via the Bluetooth of your mobile device.
Once you have a Pulsoid account, you can use this input method to collect heartrate data from the service.
By clicking the 'Login' button in the plugin, you will be asked to grant this plugin permission to connect to your account. You will then be given an 'Authentication Token' which you must paste into the plugin.
ANT+ is a low-power protocol supported by many sport and fitness sensors.
This input method allows for direct connection to your ANT+ device, provided that you have a USB receiver plugged in.
By clicking the 'Refresh' button, this plugin will begin a continuous scan for devices that output heartrate data. Then, simply select one from the dropdown and click the 'Connect' button.
Please note that this plugin is not an officially licensed or certified affiliate of the ANT+ Brand.
HypeRate is a free third-party app for Android/iOS which allows for easy, reliable connectivity to a wide set of heartrate monitors via the Bluetooth of your mobile device.
Once you have a HypeRate account, you can use this input method to collect heartrate data from the service.
âš IMPORTANT! âš : As of October 2023, Pulsoid and HypeRate now natively support Fitbit devices through their app! As such, it is highly recommended to use one of these input methods for this plugin, and connect your Fitbit device through that instead.
Fitbit is wearable fitness device brand, with an app for Android/iOS.
In order to use your Fitbit with vts-heartrate, you will need to download a special companion app onto the Fitbit device itself.
To do this, select your watch model from the dropdown. Then scan the 'Download the Companion App' QR code with your phone, which will take you to the Fitbit App Gallery listing for the companion app version that is compatible with your model. Follow the prompts on your phone to install this companion app via the Fitbit phone app.
On the phone app, beneath the 'Install' button, there will be sections labelled as 'Permissions' and 'Settings'. Make sure to grant all requested permissions. Then navigate to 'Settings', where you will be prompted for Your Local IP, which is provided to you by the vts-heartrate PC app. Upon entering that information, your Fitbit watch should finally begin sending heartrate data to vts-heartrate after a short delay.
If you ever need to return to the 'Settings' menu again, in order to update Your Local IP, simply scan the 'Download the Companion App' QR code again.
This setup also requires that your phone is on the same network as the PC running vts-heartrate, because your phone is actually the device relaying information between the Fitbit and the PC.
A list of compatible Fitbit models can be found in the FAQ at the bottom of this README.
This output will gradually tint the matched Art Meshes the desired color, based on your current heartrate. For example, you can use this to make your model's face flushed for workouts, or run cold during horror games.
Art Meshes are matched as long as their names or tags contain any of the provided list of text to match. For example, providing the text 'head,mouth,neck' will match Art Meshes with names like 'forehead', 'outermouth2', and 'leftneckside1'. Text to match must be comma separated and should not contain spaces.
If you are unsure of what your Art Meshes are named, a great web-tool was developed by Hawkbar called VTubeStudioTagger, which offers an intuitive way to discover the names of your model's Art Meshes.
Additionally, as of version 1.3.0, you can select specific Art Meshes by clicking the 'Select Art Meshes in VTube Studio' button, which will open a prompt in VTube Studio.
This output will cause an Expression to activate or deactivate when the current heartrate is above or below a given threshold, based on the selected behavior settings.
For example, the configuration in the provided image will cause the angry Expression to automatically activate when the heartrate exceeds 120 BPM, and will deactivate when the heartrate falls back beneath 120 BPM.
This output will cause a Hotkey to be triggered when the current heartrate is above or below a given threshold, based on the selected behavior settings.
For example, the configuration in the provided image will cause the panic Item Scene to automatically toggle when the heartrate exceeds 120 BPM, and will toggle again when the heartrate falls back beneath 120 BPM.
This plugin outputs fifteen custom tracking parameters for use. They are as follows:
VTS_Heartrate_Linear: A value that scales from 0.0 to 1.0 as your heartrate moves across the expected range.VTS_Heartrate_Pulse: A value that bounces back and forth between 0.0 and 1.0 with a frequency exactly matching your heartrate.VTS_Heartrate_Breath: A value that bounces back and forth between 0.0 and 1.0 with a frequency slower thanVTS_Heartrate_Pulse, suitable for controlling your model'sParamBreathoutput parameter.
The following parameters are primarily suited for controlling Live2D props, but you may also find them useful of your model:
VTS_Heartrate_BPM: A value that represents the actual current BPM from 0 to 255, rather than a normalized value from 0.0 to 1.0.VTS_Heartrate_BPM_Ones: A value from 0 to 9 that represents the first digit of the current BPM. For example, with a heartrate of 95, this parameter would be 5.VTS_Heartrate_BPM_Tens: A value from 0 to 9 that represents the second digit of the current BPM. For example, with a heartrate of 95, this parameter would be 9.VTS_Heartrate_BPM_Hundreds: A value from 0 to 9 that represents the third digit of the current BPM. For example, with a heartrate of 95, this parameter would be 0.VTS_Heartrate_Repeat_1: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every heartbeat.VTS_Heartrate_Repeat_5: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 5 heartbeats.VTS_Heartrate_Repeat_10: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 10 heartbeats.VTS_Heartrate_Repeat_20: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 20 heartbeats.VTS_Heartrate_Repeat_30: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 30 heartbeats.VTS_Heartrate_Repeat_60: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 60 heartbeats.VTS_Heartrate_Repeat_120: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with every 120 heartbeats.VTS_Heartrate_Repeat_Breath: A value that scales linearly from 0.0 to 1.0 and resets back to 0.0 with the frequency of theVTS_Heartrate_Breathparameter.
For more information on how to integrate these tracking parameters into your model, please refer to the Official VTube Studio documentation.
There are several Live2D Item assets available which are already configured to take advantage of the above custom parameters. You can find them at the 7M|C Shop. These assets packs are:
These packs include things such as sweat drips, EKG monitors, steamy breath puffs and more.
Of course, if you're up the the task, you can also make your own Live2D Items using these custom parameters, as well. You can find the complete guide to making Live2D Items in the VTube Studio Manual.
VTube Studio offers a ton of cool post-processing rendering effects, called "visual effects" or "VFX" for short.
You can make the intensity of these effects scale with your heartrate using Custom Tracking Parameters! Simply select the desired tracking parameter that you want to use to control a given config value of the effect. You can then additionally modify the config value using the slider to increase or decrease its maximum intensity.
There is a lot of customizability with this feature, so it is recommended that you play around with different values and see how they look.
For example, the configuration in the provided image will cause the analog_glitch effect to have a pulsating intensity driven by the VTS_Heartrate_Pulse parameter, which is then multiplied by 0.50 to dampen the minimum and maximum strength of the overall effect. It will also cause the scanline_jitter aspect of the effect to have an intensity driven by VTS_Heartrate_Linear parameter, multiplied by 0.65 for the same reason.
In this way, you can enjoy having dynamic VFX that scale with your heartrate, while still being able to limit how intense they get, even at your highest heartrate values.
You can use the Stream Widget as a Browser Source in OBS, or as a Web Item in VTube Studio. It prints your heartrate in real time, and the backing image does a pulsing animation at the corresponding speed.
The widget will automatically attempt to connect to vts-heartrate when it loads, so if you open the widget before launching vts-heartrate, simply refresh the widget to connect.
You can get the URL for the widget from a button within vts-heartrate.
As of version 1.2.0, vts-heartrate supports profiles.
Profiles allow you to have multiple output configurations for the same model. For example, you may want one profile where you go blue in the face for Horror Games, and another profile where you flush red and activate a sweating expression for Workouts.
You can even copy profile configurations from one profile to another, even across models. This is useful if you have multiple variations of the same base model, such as different outfits.
When a model is loaded in VTube Studio, a DEFAULT profile is automatically loaded along with it. User-created profiles must be loaded manually.
As of version 1.2.0, vts-heartrate features its own Plugin API, so that you can build your own apps that consume or write heartrate data! That's right, this VTube Studio plugin now supports plugins of its own.
There are three underlying API endpoints, all accessible via WebSocket: the Data API, the Events API, and the Input API.
The Data API is a read-only endpoint accessible at ws://localhost:<your chosen port>/data. Upon connecting to this endpoint, your WebSocket will receive a message containing current heartrate and output parameter data once per frame (60 times per second).
The message structure is as follows:
{
apiVersion: "1.0",
messageType: "DataResponse",
timestamp: 1656382245785
data: {
heartrate: 103,
parameters: {
vts_heartrate_bpm: 103,
vts_heartrate_bpm_hundreds: 1,
vts_heartrate_bpm_ones: 3,
vts_heartrate_bpm_tens: 0,
vts_heartrate_breath: 0.29349666833877563,
vts_heartrate_linear: 1,
vts_heartrate_pulse: 0.994240403175354,
vts_heartrate_repeat_1: 0.27418050169944763,
vts_heartrate_repeat_5: 0.07212083041667938,
vts_heartrate_repeat_10: 0.3203545808792114,
vts_heartrate_repeat_20: 0.12136854976415634,
vts_heartrate_repeat_30: 0.4221900999546051,
vts_heartrate_repeat_60: 0.714303731918335,
vts_heartrate_repeat_120: 0.932239294052124,
vts_heartrate_repeat_breath: 0.29349666833877563
},
tints: [
{
baseColor: { r: 255, g: 128, b: 128, a: 255 },
currentColor: { r: 255, g: 200, b: 200, a: 255 },
matchers: ['head', 'face']
},
{
baseColor: { r: 255, g: 128, b: 128, a: 255 },
currentColor: { r: 255, g: 200, b: 200, a: 255 },
matchers: ['mouth', 'neck']
}
]
}
}
For more information about the output parameters, consult the Custom Tracking Parameter Documentation and Art Mesh Tinting Documentation.
The Event API is a read-only endpoint accessible at ws://localhost:<your chosen port>/events. Upon connecting to this endpoint, your WebSocket will receive a message containing event information every time an Expression Trigger or Hotkey Trigger is triggered.
For Expressions, the message structure is as follows:
{
apiVersion: "1.0",
messageType: "ExpressionEventResponse",
timestamp: 1656382245785
data: {
threshold: 120,
heartrate: 121,
expression: "angry.exp3.json",
behavior: 2,
activated: true
}
}
The complete list of possible behavior values is as follows:
UNKNOWN = -1,
ACTIVATE_ABOVE_DEACTIVATE_BELOW = 0,
DEACTIVATE_ABOVE_ACTIVATE_BELOW = 1,
ACTIVATE_ABOVE = 2,
DEACTIVATE_ABOVE = 3,
ACTIVATE_BELOW = 4,
DEACTIVATE_BELOW = 5,
For Hotkeys, the message structure is as follows:
{
apiVersion: "1.0",
messageType: "HotkeyEventResponse",
timestamp: 1656382245785
data: {
threshold: 120,
heartrate: 121,
hotkey: "Blush",
behavior: 1
}
}
The complete list of possible behavior values is as follows:
UNKNOWN = -1,
ACTIVATE_ABOVE_ACTIVATE_BELOW = 0,
ACTIVATE_ABOVE = 1,
ACTIVATE_BELOW = 2,
The Input API is a read and write endpoint accessible at ws://localhost:<your chosen port>/input. Upon connecting to this endpoint, your WebSocket will be able to write heartrate data for use with the API WebSocket input method.
First, you will need to authenticate your client before you are granted write permission. In order to authenticate, you must first request a token by sending a message with the following structure:
{
messageType: "AuthenticationRequest",
data: {
pluginName: "My Heartrate Plugin",
pluginAuthor: "Skeletom",
pluginAbout: "A plugin for connecting a third-party service."
}
}
The user will then be prompted to either Approve or Deny access to this plugin. If the request is approved, the API server will then respond with a message containing a token.
You only need to do this step if your plugin has not already been granted a token. If it has a token already, you can skip to the next step.
The message structure is as follows:
{
apiVersion: "1.0",
messageType: "AuthenticationResponse",
timestamp: 1656382245785
data: {
pluginName: "My Heartrate Plugin",
pluginAuthor: "Skeletom",
token: "e404d80b-c160-4af4-a0b9-9c0159f3010e",
authenticated: false
}
}
It is strongly recommended that you save this token so that your plugin may use it again in the future. Finally, you must now submit an authentication request using the token, by sending a message with the following structure:
{
messageType: "AuthenticationRequest",
data: {
pluginName: "My Heartrate Plugin",
pluginAuthor: "Skeletom",
token: "e404d80b-c160-4af4-a0b9-9c0159f3010e"
}
}
Assuming you have provided a token that the user has granted access for, the API Server will finally respond with a message indicating that your plugin is fully authenticated, and that you may begin writing heartwrate data.
The message structure is as follows:
{
apiVersion: "1.0",
messageType: "AuthenticationResponse",
timestamp: 1656382245785
data: {
pluginName: "My Heartrate Plugin",
pluginAuthor: "Skeletom",
token: "e404d80b-c160-4af4-a0b9-9c0159f3010e",
authenticated: true
}
}
Once your plugin is fully authenticated, you can write heartrate data sending a message with the following structure:
{
messageType: "InputRequest",
data: {
heartrate: 78
}
}
Upon successfully writing data, the API Server will echo your message back to you as an InputResponse, to confirm that it has been received.
In the event that something goes wrong, such as the API Server receiving a message it cannot parse, or a message from an unauthenticated client, your WebSocket will receive an error message.
The message structure is as follows:
{
apiVersion: "1.0",
messageType: "ErrorResponse",
timestamp: 1656382245785
data: {
errorCode: 403,
message: "This client is not authenticated!"
}
}
The complete list of possible errorCode values is as follows (they are HTTP status codes):
OK = 200,
BAD_REQUEST = 400,
FORBIDDEN = 403,
SERVER_ERROR = 500,
A: This plugin supports any device that is supported by
- Pulsoid (compatibility list)
- HypeRate (compatibility list)
- ANT+ with a USB Receiver
- The following Fitbit devices:
- Versa 3
- Sense
- Ionic
- Versa 2
- Versa Lite
- Versa
Unfortunately, Fitbit has not provided a way to develop or install third-party apps on their latest models (Versa 4 and later), so the latest devices cannot be supported at this time.
Planned features include the following:
- Localization into additional languages
- More device integrations (I take requests!)