ChibiOS upgrade to v19.1.3

[tzarc]

Description

As per the title, this is a PR in order to upgrade ChibiOS to the latest release version.

The newer ChibiOS itself provides some scripts that are able to upgrade the conf files in use by each board. For chconf.h and halconf.h this applies to all boards, however, for mcuconf.h this only applies to F303, as that's the only "upgrade script" supplied with the distribution. Other MCUs have been manually updated using diff's and the ChibiOS testhal definitions.

To combat making zillions of other changes, I've also added an interim script -- util/chibios-upgrader.sh -- which currently builds the 3rdparty dependencies for the ChibiOS upgrade scripts, and performs the update. In order to identify duplicate conf file and board definitions, each file gets pumped through clang-format before generating a checksum, in order to ensure only one distinct copy of each of the updated files is required for deployment into the rest of the repository. Unfortunately, this script is known to be non-functional in its current form on macOS. I'm unsure if this script (along with the "deployed files") should be included in the final merge.

The last commit on this PR (" Add updated board/conf files ") is the result of the execution of the upgrade script.

At this stage, the only testing that has been done was with a Proton C and a Blue Pill, both the handwired/onekey variants.

I've produced binaries from make all:default and uploaded them to the following location:
https://qmk.tzarc.io/chibios-upgrade/

There are a few outstanding pieces of development required, namely:

Dealing with the submodules:

• ChibiOS is a direct copy of the main ChibiOS repository, and the referenced revision is specifically the ver19.1.3 tag. This points to my own copy of the repo, and will need to be updated on QMK's side to ensure consistency.
• ChibiOS contrib is also a direct copy of the contrib repo, pointing at a recent revision of master. I haven't investigated if there have been subsequent updates -- I'd imagine that'll get done when the conversion to QMK-official repositories get done
• uGFX is a direct copy of their master branch, with one additional change in order to bump compatibility with the newer ChibiOS. QMK uses a version of uGFX that is in maintenance mode, and they haven't seen fit to update it to be compatible with latest ChibiOS.

The other aspect is there have been a few definitions that have gone missing due to people modifying the *conf.h files (such as CORTEX_ENABLE_WFI_IDLE), which the ChibiOS upgrade scripts nuke during update.

I've kicked off this PR for now to get some official discussion going, as well as provide a forum to decide on a formal plan if/when these changes are integrated.

Types of Changes

• Core
• Bugfix
• New feature
• Enhancement/optimization
• Keyboard (addition or update)
• Keymap/layout/userspace (addition or update)
• Documentation

Issues Fixed or Closed by This PR

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

TODO

• ChibiOS-Contrib modifications made by QMK already upstreamed, apparently
• Fix timer subsystem
• Work out missing #defines automatically removed when doing chibios update scripts
• Removed: CORTEX_ENABLE_WFI_IDLE -- see Move CORTEX_ENABLE_WFI_IDLE=TRUE to rules.mk files #7766
• Removed: usb_lld_disconnect_bus/usb_lld_disconnect_bus --in board.h, not modified by ChibiOS
• Look into LTO build problems
• Move personal repo's for ChibiOS/Contrib/uGFX to QMK org
• Work out what to do with the update script and associated files added to util/

[tzarc]

Regarding converter/siemens_tastatur, discussion on Discord:

[2:06 AM] tz: ahhh, just the person I need to talk to 😛
[2:08 AM] tz: converter/siemens_tastatur was the last one I had to update
[2:08 AM] tz: don't suppose a test would be achievable?
[2:26 AM] yiancar: nah
[2:26 AM] yiancar: u cnt test it lol
[2:26 AM] yiancar: :p
[2:26 AM] yiancar: there is only 1/1
[2:26 AM] yiancar: i think the guy dissasembled it anyway
[2:26 AM] yiancar: so assume its working lol
[2:33 AM] tz: perfect!

[yiancar]

I dont want to be the annoying dude but I have just 1 question:)
Why did we take out the space alignment in mcuconf.h? Was it to make easier to script?

[tzarc]

I dont want to be the annoying dude but I have just 1 question:)
Why did we take out the space alignment in mcuconf.h? Was it to make easier to script?

Honestly, a lot of it was automated -- I'd imagine a clang-format was involved at some point.
The script is capable of backing out all the board+conf changes, will see if it's just a question of using the non-space-removed copy. Will check it out.

[yulei]

It's hard to test all the keyboards using chibios, keeping the current and introducing the v19 chibios as another protocol would be easier.

[tzarc]

It's hard to test all the keyboards using chibios, keeping the current and introducing the v19 chibios as another protocol would be easier.

That may be the case, but it's unlikely that we'll want to support two ChibiOS versions for eternity, given the frequency of people copypasting of other keyboards. At some point "no" will need to be said when targeting the older ChibiOS with newer boards, and realistically the earlier the better.

Given that QMK doesn't seem to remove existing boards, there's only one course of action if the eventual removal of the v17.x ChibiOS is deemed necessary.

[yanfali]

It's hard to test all the keyboards using chibios, keeping the current and introducing the v19 chibios as another protocol would be easier.

That's just not sustainable long term. We don't have the resources. If you want to maintain your own fork that might be a possibility, but I would not want to do that. One thing we're currently discussing is having stable and development branches, this would probably be a good approach rather than maintaining an old version of chibios and a current one. We'll update everyone once we've discussed things among collaborators and have something like a plan.

[yiancar]

Also in theory we dont need to test "each keyboard" rather each platform

[tzarc]

I dont want to be the annoying dude but I have just 1 question:)
Why did we take out the space alignment in mcuconf.h? Was it to make easier to script?

Confirmed that it was clang-format -- the ChibiOS upgrade scripts depend on the existence of #define in the code (without spaces after the #), and whilst that might be fine for some keyboards, because we have IndentPPDirectives: AfterHash set in the config, any existing values aren't carried across.

Have modified the scripts to rollback any formatting changes if the ChibiOS upgrade scripts didn't touch them.

[yiancar]

<3

[yanfali]

tested clueboard 66 rev 4, seemed to work F303
tested clueboard 60 - unresponsive after flash F303
@tzarc any way you can get the CLI to print the version of chibios being built?

[yanfali]

hadron v3 with default keymap seems to work.
OK verified that this is head of chibios dir
commit 313416b8fda90d9973a749a0a35970956852c286 (HEAD, tag: ver19.1.3)

[yanfali]

hadron ver3 weirdly if I flash my personal keymap it refuses to boot at all. flashing master qmk back on to board brings it back to life. branch is here: https://github.com/yanfali/qmk_firmware/tree/yanfali-chibios-upgrade

[yanfali]

tested my little foot port using CTPC and it also doesn't work on a proton-c. Doesn't boot

[yanfali]

@tzarc I pushed the littlefoot stuff to my fork so you can try it out for yourself since it's only proton c based. You can build it using make kingly_keys/little_foot:yanfali:flash CTPC=yes. plays some nice music on startup, so you know if it's working if a speaker is installed

[drashna]

@yanfali just to make sure, you've ran make clean, right? (at least once).

Also, this breaks the bitbang WS2812 implementing, and it breaks solenoid support. And LT, MT, and one shot timeout. And probably more :(
Anything that relies on a (software) timer seems to be broken.

Edit: Also, LTO breaks the new chibiOS. If it's enabled, it will basically compile a nice brick.

[yanfali]

yeah used a new clone so it was brand new, had to do that to get sane behavior on the submodules. I use LT and MT so this is definitely broken for me then.

…

On Sun, Dec 15, 2019 at 8:39 PM Drashna Jaelre ***@***.***> wrote: @yanfali <https://github.com/yanfali> just to make sure, you've ran make clean, right? (at least once). Also, this breaks the bitbang WS2812 implementing, and it breaks solenoid support. And LT, MT, and one shot timeout. And probably more :( Anything that relies on a (software) timer seems to be broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7597?email_source=notifications&email_token=AARLSUYRM54MBQYHJNK7TKDQY4BABA5CNFSM4JZAM2QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG5PT7I#issuecomment-565901821>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARLSU3P7R3DVN5ZPY4X4JLQY4BABANCNFSM4JZAM2QA> .

-- "We do not learn so much by our successes as we learn by failures -- our own and others'. Especially if we see the failures properly corrected." -- Frank Lloyd Wright

[tzarc]

tested my little foot port using CTPC and it also doesn't work on a proton-c. Doesn't boot

......built directly from your branch, started playing music fine. Weird?

[yanfali]

tested my little foot port using CTPC and it also doesn't work on a proton-c. Doesn't boot

......built directly from your branch, started playing music fine. Weird?

In discord tz figured out it was compiler versions. 8.3.1 arm doesn't build quite right for latest chibios branch.

[tzarc]

Timers seem like they're cooperating now, rgblight animations are cycling at a (seemingly) normal speed.
Later today I'll work out a test to toggle a GPIO using timer_elapsed, confirm with an oscilloscope.