feat(split): Transfer arbitrary tagged data from central to peripheral

[ReFil]

Currently the only mechanism for data transfer for secondary features (underglow and backlighting primarily) is through the behaviour invocation process, this can only hander user initiated changes, so automatic state changes are missed and the halves state can go out of sync if the central is changed with the peripheral disconnected. Some new features currently in PR initalise a new characteristic for central-> peripheral data transfer however for infrequently updated things abstracting it away means there's not a variety of implementations in the split source files.
This introduces a new mechanism by which a data tag, size value and up to 16 bytes of data can be transferred from central to peripheral(s). I tried 32 bytes but ran into some MTU limitation. The data tags are handled in an enum and, in a similar manner to the zephyr sensor channels, additional tags can be defined separately from the current ones and passed to this system without issue. There are currently 5 tags defined, one each for underglow and backlighting, one for the upcoming HID indicators work and one for keymap state (which could be used for passing layer information to a peripheral display widget). the fifth is a marker to show the start of the unallocated ranges for custom usages. That way any future custom uses automatically get "bumped down" as new upstream uses get added seamlessly without any conflicts.
This also implements this feature for both RGB underglow and single colour backlighting for a more seamless lighting experience.

[ReFil]

Rebased and changed the HID Indicators code to use this system too

[ReFil]

This should resolve #1494 for RGB and backlighting

[ReFil]

This has been rebased to solve the merge conflict as well as adding the ability to send keymap layer state to the peripherals for peripheral display widgets/rgb indications

[ReFil]

This could be expanded to include a peripheral side listener for the remaining global locality behaviours (reset, bootloader and ext power) with the intention of superceding the locality system, keen to hear other people's opinions on this idea

[ReFil]

Rebased after Zephyr 3.5 update, need to do some testing to confirm nothing broke

[ReFil]

Tested all 4 data channels and confirmed, RGB state, backlight state, HID indicators and keymap state all get successfully transmitted to the peripheral , a display widget or custom RGB underglow implementation can subscribe to the zmk_split_data_xfer_event event and process information from the central. automatic state changes are transferred as well. By disabling *_AUTO_OFF_IDLE on the peripheral for RGB and backlighting the lights stay in sync throughout all state changes on the central be it automated or manual.
The zmk_split_peripheral_status_event event is now triggered correctly however it usually triggers before the connection has had time to process. As a result the central often fails to update the peripheral on reconnection after a power cycle. Not sure how to solve this in a good way, would appreciate any suggestions on this

[skewty]

It has been over 3 months and it seems no review has taken place (don't see any review comments). Is something still pending?

[ReFil]

It has been over 3 months and it seems no review has taken place (don't see any review comments). Is something still pending?

From my perspective it's ready for a preliminary review

[skewty]

@petejohanson you were assigned. Will you be the person doing the review?

[infused-kim]

@skewty I am not one of the maintainers, but have been involved in the project for a while and have submitted a few PRs, so here are my 2 cents...

(Which are my opinions alone and do not represent the views of the project or maintainers. I could also be completely wrong about everything here adn they might disagree with it all.)

The zmk maintainer team is very small, the userbase is quite big and the amount of PRs submitted is a lot. There is no way they could review everything and schedule each PR based on the time the PR has been open.

I imagine they have an internal roadmap of the improvements they want to get in and not all the PRs match that roadmap.

On top of that, each feature has a cost, because most of the time the original person submitting it, won't be maintaining it. Updating zephyr, the underlying RTOS, frequently requires a lot of modifications to the code base. Once a feature is merged, all of that work is dumped on the maintainers.

Additionally, a lot of features are submitted by developers who are not familiar with zephyr and zmk. So many features work, but are not implemented in the correct and maintainable way. I'm not saying it's the case with this PR, but it's just one of the things that must be considered.

So, considering all of this, I would recommend you learn how to maintain your own fork with all of the PRs you want to use. It's really not very difficult. Urob's guide is a great intro:
https://gist.github.com/urob/68a1e206b2356a01b876ed02d3f542c7

That's the beauty of open source. You can be independent and include all teh features you want without relying on the maintainers prioritizing the features you want.

[petejohanson]

So... I think my main concern with this approach is that this introduces an extra level of indirection for consuming code on the peripheral sides. They no longer can subscribe to the specific event they want, they need to subscribe to this "grab bag" event and do two levels of checks/filtering.

I would much rather a system that uses some generic GATT bits as an internal implementation detail that allows domain specific events/data to be set and the rest of the code doesn't need to be aware of it.

Does that make sense?

[ReFil]

So the peripheral GATT handler raises the layer changed event or indicators changed event etc? I avoided that for a couple reasons:

1. If someone wants to send their own data across this transport in a module they can't without modifying upstream files, unless we have the data transfer event as a backstop if the known tags don't match maybe
2. We'd need to create a lot more events to handle RGB indicator state, backlight state etc
3. Any alternative transports (e.g. wired) would probably need to reimplement the same sorting and individual event raising functionality, rather than just raising the same event and letting the downstream files handle it

The alternative would be having a split communication layer in between the transport's and the rest of the system on each side e.g.

System - split_central.c - (wired, Bluetooth etc etc)_central.c <-> transport <-> (wired, Bluetooth etc etc)_peripherals.c - split_peripheral.c - System

Which could pave the way for wired splits, or even having multiple different split transports available for use simultaneously, plug the trrs cable in if you're in a coffee shop or unplug it for wireless operation but that's a far more major rework than what I originally had in mind here

[petejohanson]

So the peripheral GATT handler raises the layer changed event or indicators changed event etc? I avoided that for a couple reasons:

That was the thinking, but you raise some interesting points, so replying below.

1. If someone wants to send their own data across this transport in a module they can't without modifying upstream files, unless we have the data transfer event as a backstop if the known tags don't match maybe

Yeah, that brings up another area of this that I have some hesitation around, but am not necessarily opposed to; I see that you added the "custom start" tag for the enum to allow third-party code to add their own data type tags. On the one hand, having such a mechanism makes sense, and yet simultaneously if this "takes off" then we'll quickly hit conflicts with features re-useing the same tag ID and then wanting to be used together.

If we want to keep going on that, I think having a layered system where we still fire the "raw" xfer event for custom tags, and for known types, fire the "native" semantic event.

2. We'd need to create a lot more events to handle RGB indicator state, backlight state etc

Is this a bad thing?

3. Any alternative transports (e.g. wired) would probably need to reimplement the same sorting and individual event raising functionality, rather than just raising the same event and letting the downstream files handle it

The alternative would be having a split communication layer in between the transport's and the rest of the system on each side e.g.

System - split_central.c - (wired, Bluetooth etc etc)_central.c <-> transport <-> (wired, Bluetooth etc etc)_peripherals.c - split_peripheral.c - System

I think if the "transport" can handle the low level bits, we can have code layered on top that does the "take a complete xfer event, and do the right thing with it". I believe that matches what you've described above.

Which could pave the way for wired splits, or even having multiple different split transports available for use simultaneously, plug the trrs cable in if you're in a coffee shop or unplug it for wireless operation but that's a far more major rework than what I originally had in mind here

I think that would be a start.

[ReFil]

Would a good path forward to be add an extra source file for a peripheral handler that captures all data transfer events and raises the appropriate events for recognised tags? Do we also want a central handler that captures events and turns that into data transfer requests.

1: I'm very keen on maintaining support for custom tags. I'll have a think on how best to avoid ID collisions, ideally some sort of compile time check but no obvious solution for this comes immediately to mind. The results of an ID collision would also be unpredictable which would lead to a challenging problem to diagnose. Good logging might help on this one.

2: no, I suppose it's not. Off the top of my head we'll need new events for backlight and underglow. For the keymap it gets a little trickier, because there's the current event for layer but not the display name, which a peripheral widget may well want to display. Should we treat the keymap information as a special case and add some extra code on the peripheral side to handle storing the keymap name or add a new event on both sides for highest layer keymap's name?

[Nick-Munnich]

[...] having multiple different split transports available for use simultaneously, plug the trrs cable in if you're in a coffee shop or unplug it for wireless operation [...]

Just wanted to add that this is probably the feature I'd want the most from wired split in ZMK (aside from it functioning).