Streaming Upgrades, and mark as Alpha (#1129)

This commit is a large commit which provides us the tooling so we can more efficiently make and modify streams for appropriate situation, and does a bunch of work to optimize our stream. (A mid sized patch shrunk by half with these changed). It's a large change which closes #1069 and should allow us to be streaming stable from hereon out. At least that's the plan! - Make -1/-1/-1 Zone Address stream as {} - Processor Storage none is streamed as {} now - Zone::AssociatedSampleData inactive -> {} - Move current streaming version to configuration.,h - Remove the extra streaming version variable on patch - Have a human readable streaming version fn and include it in engine json and log messages - Add capability to support streaming an enum with default, but be careful - Use that on bus aux - Fix a channel strip display bug that it didn't show the routing on refresh properly - Use default routing row to shrink stream size / parse time - Add a "StreamReason" like "Unstream" data which lets you know if you are streaming for process, daw or multi - Don't send route extraPalyload for daw or multi - Move the streamguards into engine next to the thread local states - Add template functions to add and addUnlessDefault for an object - Bus mute solo pan and level default stream - Stream sample udpate and up streaming version for big streamathon - Move to SC_STREAMDEF everywhere (except ID for reasons) - Compact a whole bunch of keys and do some optionals - Some more key compaction - Finally, Pre-Alpha to Alpha Closes #1069
surge-synthesizer · Aug 16, 2024 · 365b2a5 · 365b2a5
1 parent 7850be6
commit 365b2a5
Show file tree

Hide file tree

Showing 27 changed files with 706 additions and 645 deletions.
diff --git a/.github/workflows/build-release.yml b/.github/workflows/build-release.yml
@@ -124,8 +124,8 @@ jobs:
         with:
           webhook: ${{ secrets.DISCORD_SHORTCIRCUIT_WEBHOOK }}
           tag: Nightly
-          title: "A New ShortCircuit Nightly is Available"
-          subtitle: "Still pre-alpha! Use care!"
+          title: "A New Shortcircuit XT Nightly is Available"
+          subtitle: "Shortcircuit XT is alpha software. Please use care."
 
 
   publish-scxt-release:

diff --git a/clients/clap-first/scxt-juce-standalone/scxt-juce-standalone.cpp b/clients/clap-first/scxt-juce-standalone/scxt-juce-standalone.cpp
@@ -117,6 +117,7 @@ struct SCXTApplicationWindow : juce::DocumentWindow, juce::Button::Listener
         engine->getSampleManager()->purgeUnreferencedSamples();
         try
         {
+            auto sg = scxt::engine::Engine::StreamGuard(scxt::engine::Engine::FOR_DAW);
             auto engineXml = scxt::json::streamEngineState(*engine);
             SCLOG("Streaming State Information: " << engineXml.size() << " bytes");
             properties->setValue("engineState", juce::String(engineXml));

diff --git a/clients/clap-first/scxt-plugin/scxt-plugin.cpp b/clients/clap-first/scxt-plugin/scxt-plugin.cpp
@@ -111,6 +111,7 @@ bool SCXTPlugin::stateSave(const clap_ostream *ostream) noexcept
     engine->getSampleManager()->purgeUnreferencedSamples();
     try
     {
+        auto sg = scxt::engine::Engine::StreamGuard(engine::Engine::FOR_DAW);
         auto xml = scxt::json::streamEngineState(*engine);
 
         auto c = xml.c_str();

diff --git a/clients/juce-plugin/SCXTProcessor.cpp b/clients/juce-plugin/SCXTProcessor.cpp
@@ -318,6 +318,7 @@ void SCXTProcessor::getStateInformation(juce::MemoryBlock &destData)
     engine->getSampleManager()->purgeUnreferencedSamples();
     try
     {
+        auto sg = scxt::engine::Engine::StreamGuard(scxt::engine::Engine::FOR_DAW);
         auto xml = scxt::json::streamEngineState(*engine);
         SCLOG("Streaming State Information: " << xml.size() << " bytes");
         destData.replaceAll(xml.c_str(), xml.size() + 1);

diff --git a/libs/sst/sst-basic-blocks b/libs/sst/sst-basic-blocks
diff --git a/resources/NightlyBlurb.md b/resources/NightlyBlurb.md
@@ -1,11 +1,12 @@
-ShortCircuit XT is a sample-initiated instrument being actively developed by the Surge Synth Team. The build below is our latest pre-alpha github version, but it is very incomplete.
+ShortCircuit XT is a sample-initiated instrument being actively developed by the Surge Synth Team. The build below is our latest alpha github version, but it is very incomplete.
 
 The instrument has many features incomplete, still has unresolved crashing bugs, and may have DSP errors which cause unbounded sound. Still, it is becoming more stable and usable and we do have testers and developers running it.
 
 If you use ShortCircuit a few things of note:
 
 - You may want to use a limiter on the SC bus, in case there is a DSP error, and definitely do not use in-ear headphones
 - Most of the discussion about what works and doesn't happens on our discord. Please join!
-- [This figma document](https://www.figma.com/proto/LWyY0E29tISj1djAp40EDL/ED-SST-Wireframes?node-id=3228-2774&starting-point-node-id=3228%3A2774) serves as a design guide for where we are going
+- [This figma document](https://www.figma.com/proto/LWyY0E29tISj1djAp40EDL/ED-SST-Wireframes?node-id=3228-2774&starting-point-node-id=3228%3A2774)
+  serves as a design guide for where we are going
 - The more the merrier! If you are a dev jump in.
 
diff --git a/src-ui/components/SCXTEditorMenus.cpp b/src-ui/components/SCXTEditorMenus.cpp
@@ -95,7 +95,17 @@ void SCXTEditor::showMainMenu()
     auto dp = juce::PopupMenu();
     dp.addSectionHeader("Developer");
     dp.addSeparator();
-    dp.addItem("Pretty JSON", [w = juce::Component::SafePointer(this)]() {
+    dp.addItem("Pretty JSON (DAW)", [w = juce::Component::SafePointer(this)]() {
+        if (!w)
+            return;
+        w->sendToSerialization(cmsg::RequestDebugAction{cmsg::DebugActions::pretty_json_daw});
+    });
+    dp.addItem("Pretty JSON (MULTI)", [w = juce::Component::SafePointer(this)]() {
+        if (!w)
+            return;
+        w->sendToSerialization(cmsg::RequestDebugAction{cmsg::DebugActions::pretty_json_multi});
+    });
+    dp.addItem("Pretty JSON (PROCESS)", [w = juce::Component::SafePointer(this)]() {
         if (!w)
             return;
         w->sendToSerialization(cmsg::RequestDebugAction{cmsg::DebugActions::pretty_json});

diff --git a/src-ui/components/WelcomeScreen.cpp b/src-ui/components/WelcomeScreen.cpp
@@ -38,7 +38,7 @@ namespace scxt::ui
 WelcomeScreen::WelcomeScreen(SCXTEditor *e) : HasEditor(e)
 {
     setAccessible(true);
-    setTitle("Welcome to ShortCircuit XT Pre-Alpha. Please use care. Press escape to use sampler.");
+    setTitle("Welcome to ShortCircuit XT Alpha. Please use care. Press escape to use sampler.");
     setWantsKeyboardFocus(true);
 }
 void WelcomeScreen::visibilityChanged()
@@ -64,14 +64,15 @@ void WelcomeScreen::okGotItDontShowAgain()
 }
 
 auto txt =
-    "Shortcircuit XT is in a pre-alpha release. This version has incomplete and "
-    "missing features, may have crashing bugs, may generate improper sounds, and is not streaming "
-    "stable (so saved sessions may not work in the future).\n"
+    "Shortcircuit XT is in a alpha release. This version has incomplete and "
+    "missing features, may have crashing bugs, and may generate improper sounds\n"
     "\n"
     "We welcome testers in this pre-alpha period but recommend a few precautions:\n\n"
     "- Consider using limiter and don't use in-ear headphones when experimenting.\n"
-    "- The platform is not streaming stable; if you make music you like, bounce stems.\n"
     "- There is no missing sample resolution; don't move underlying files.\n"
+    "- We beleive we have both the streaming and Plugin ID/Params stable as of mid Aug 24\n"
+    "   but we could be wrong. If you make music you love, please do bounce stems so you can\n"
+    "   recreate it.\n"
 
     "\n"
     "We love early testers, documenters, and designers on all our projects. The best way to "
@@ -89,7 +90,7 @@ void WelcomeScreen::paint(juce::Graphics &g)
 
     g.fillAll(editor->themeColor(theme::ColorMap::bg_1).withAlpha(0.6f));
 
-    auto bd = r.reduced(140, 130);
+    auto bd = r.reduced(140, 120);
 
     g.setColour(editor->themeColor(theme::ColorMap::bg_1));
     g.fillRect(bd);
@@ -99,8 +100,7 @@ void WelcomeScreen::paint(juce::Graphics &g)
     g.setColour(editor->themeColor(theme::ColorMap::generic_content_highest));
     g.drawText("Welcome to ShortcircuitXT", bd.reduced(10), juce::Justification::centredTop);
     g.setColour(editor->themeColor(theme::ColorMap::warning_1a));
-    g.drawText("Pre-Alpha Release", bd.reduced(10).translated(0, 50),
-               juce::Justification::centredTop);
+    g.drawText("Alpha Release", bd.reduced(10).translated(0, 50), juce::Justification::centredTop);
 
     g.setFont(editor->themeApplier.interLightFor(22));
     g.setColour(editor->themeColor(theme::ColorMap::generic_content_high));

diff --git a/src-ui/components/WelcomeScreen.h b/src-ui/components/WelcomeScreen.h
@@ -40,7 +40,7 @@ struct WelcomeScreen : juce::Component, HasEditor
      * If you update this version the user will see the welcome screen
      * on their next startup even if they dismissed a prior version
      */
-    static constexpr int welcomeVersion{1};
+    static constexpr int welcomeVersion{2};
     WelcomeScreen(SCXTEditor *e);
 
     void visibilityChanged() override;

diff --git a/src-ui/components/mixer/ChannelStrip.h b/src-ui/components/mixer/ChannelStrip.h
@@ -98,6 +98,11 @@ struct ChannelStrip : public HasEditor, sst::jucegui::components::NamedPanel
     void onDataChanged()
     {
         labelPluginOutput();
+        for (int i = 0; i < numAux; ++i)
+        {
+            if (auxPrePost[i])
+                resetAuxRoutingGlyph(i);
+        }
         repaint();
     }
 

diff --git a/src/configuration.h b/src/configuration.h
@@ -33,6 +33,8 @@
 
 namespace scxt
 {
+static constexpr uint64_t currentStreamingVersion{0x2024'08'16};
+
 static constexpr uint16_t blockSize{16};
 static constexpr uint16_t blockSizeQuad{16 >> 2};
 static constexpr double blockSizeInv{1.0 / blockSize};

diff --git a/src/engine/engine.cpp b/src/engine/engine.cpp
@@ -148,6 +148,7 @@ Engine::~Engine()
 }
 
 thread_local bool Engine::isFullEngineUnstream{false};
+thread_local Engine::StreamReason Engine::streamReason{StreamReason::IN_PROCESS};
 thread_local uint64_t Engine::fullEngineUnstreamStreamingVersion{0};
 
 voice::Voice *Engine::initiateVoice(const pathToZone_t &path)

diff --git a/src/engine/engine.h b/src/engine/engine.h
@@ -333,7 +333,56 @@ struct Engine : MoveableOnly<Engine>, SampleRateSupport
      * to do this, but the easiest is to have a thread local static set up in the unstream
      */
     static thread_local bool isFullEngineUnstream;
+    enum StreamReason
+    {
+        IN_PROCESS,
+        FOR_MULTI,
+        FOR_DAW
+    };
+    static thread_local StreamReason streamReason;
     static thread_local uint64_t fullEngineUnstreamStreamingVersion;
+    struct UnstreamGuard
+    {
+        bool pIs{false};
+        uint64_t pVer{0};
+        explicit UnstreamGuard(uint64_t sv)
+        {
+            SCLOG("Unstreaming engine with streaming version = " << scxt::humanReadableVersion(sv));
+            pIs = engine::Engine::isFullEngineUnstream;
+            pVer = engine::Engine::fullEngineUnstreamStreamingVersion;
+            engine::Engine::isFullEngineUnstream = true;
+            engine::Engine::fullEngineUnstreamStreamingVersion = sv;
+        }
+        ~UnstreamGuard()
+        {
+            engine::Engine::isFullEngineUnstream = pIs;
+            engine::Engine::fullEngineUnstreamStreamingVersion = pVer;
+        }
+    };
+    struct StreamGuard
+    {
+        StreamReason pIs{IN_PROCESS};
+        explicit StreamGuard(StreamReason sr)
+        {
+            switch (sr)
+            {
+            case IN_PROCESS:
+                SCLOG("Engine Stream Guard - In Process");
+                break;
+
+            case FOR_DAW:
+                SCLOG("Engine Stream Guard - For DAW");
+                break;
+
+            case FOR_MULTI:
+                SCLOG("Engine Stream Guard - For Multi");
+                break;
+            }
+            pIs = engine::Engine::streamReason;
+            engine::Engine::streamReason = sr;
+        }
+        ~StreamGuard() { engine::Engine::streamReason = pIs; }
+    };
 
     /**
      * The VoiceDisplayState structure is updated at some relatively low (like 30hz)

diff --git a/src/engine/patch.h b/src/engine/patch.h
@@ -163,10 +163,6 @@ struct Patch : MoveableOnly<Patch>, SampleRateSupport
 
     void resetToBlankPatch()
     {
-        // If it is the year 2112 and you just had a regtest fail because
-        // this is earlier than the streaming version you just changed, then
-        // this software lived too long
-        streamingVersion = 0x2112'01'01;
         for (int i = 0; i < numParts; ++i)
         {
             parts[i] = std::make_unique<Part>(i);
@@ -178,7 +174,6 @@ struct Patch : MoveableOnly<Patch>, SampleRateSupport
     bool usesOutputBus(int bus) { return busses.usesOutput[bus]; }
     void setupBussesOnUnstream(Engine &e);
     PatchID id;
-    uint64_t streamingVersion{0}; // we use hex dates for these
 
     Engine *parentEngine{nullptr};
 

diff --git a/src/engine/zone.h b/src/engine/zone.h
@@ -202,7 +202,7 @@ struct Zone : MoveableOnly<Zone>, HasGroupZoneProcessors<Zone>, SampleRateSuppor
 
         int16_t pbDown{2}, pbUp{2};
 
-        int16_t exclusiveGroup;
+        int16_t exclusiveGroup{0};
 
         float velocitySens{1.0};
         float amplitude{0.0};   // decibels

diff --git a/src/json/datamodel_traits.h b/src/json/datamodel_traits.h
@@ -40,59 +40,60 @@ namespace scxt::json
 {
 
 SC_STREAMDEF(datamodel::pmd, SC_FROM({
+                 assert(SC_STREAMING_FOR_IN_PROCESS);
                  std::vector<std::pair<int, std::string>> dvStream;
                  for (const auto &[k, mv] : t.discreteValues)
                      dvStream.emplace_back(k, mv);
 
-                 v = {{"type", (int)t.type},
-                      {"name", t.name},
-                      {"minVal", t.minVal},
-                      {"maxVal", t.maxVal},
-                      {"defaultVal", t.defaultVal},
-                      {"canTemposync", t.canTemposync},
-                      {"canExtend", t.canExtend},
-                      {"canDeactivate", t.canDeactivate},
-                      {"canAbsolute", t.canAbsolute},
-                      {"temposyncMultiplier", t.temposyncMultiplier},
-                      {"supportsStringConversion", t.supportsStringConversion},
-                      {"displayScale", (int)t.displayScale},
+                 v = {{"t", (int)t.type},
+                      {"n", t.name},
+                      {"mn", t.minVal},
+                      {"mx", t.maxVal},
+                      {"df", t.defaultVal},
+                      {"ts", t.canTemposync},
+                      {"ex", t.canExtend},
+                      {"de", t.canDeactivate},
+                      {"ab", t.canAbsolute},
+                      {"tsm", t.temposyncMultiplier},
+                      {"ssc", t.supportsStringConversion},
+                      {"dsc", (int)t.displayScale},
                       {"unit", t.unit},
-                      {"customMinDisplay", t.customMinDisplay},
-                      {"customMaxDisplay", t.customMaxDisplay},
-                      {"discreteValues", dvStream},
-                      {"decimalPlaces", t.decimalPlaces},
-                      {"quantization", (int)t.quantization},
-                      {"quantizationParam", t.quantizationParam},
+                      {"qt", (int)t.quantization},
+                      {"qtp", t.quantizationParam},
+                      {"cmid", t.customMinDisplay},
+                      {"cmxd", t.customMaxDisplay},
+                      {"discv", dvStream},
+                      {"dep", t.decimalPlaces},
                       {"svA", t.svA},
                       {"svB", t.svB},
                       {"svC", t.svC},
                       {"svD", t.svD}};
              }),
              SC_TO({
-                 findEnumIf(v, "type", to.type);
-                 findIf(v, "name", to.name);
-                 findIf(v, "minVal", to.minVal);
-                 findIf(v, "maxVal", to.maxVal);
-                 findIf(v, "defaultVal", to.defaultVal);
-                 findIf(v, "canTemposync", to.canTemposync);
-                 findIf(v, "canDeactivate", to.canDeactivate);
-                 findIf(v, "canExtend", to.canExtend);
-                 findIf(v, "canAbsolute", to.canAbsolute);
-                 findIf(v, "temposyncMultiplier", to.temposyncMultiplier);
-                 findIf(v, "supportsStringConversion", to.supportsStringConversion);
-                 findEnumIf(v, "displayScale", to.displayScale);
-                 findEnumIf(v, "quantization", to.quantization);
-                 findIf(v, "quantizationParam", to.quantizationParam);
+                 findEnumIf(v, "t", to.type);
+                 findIf(v, "n", to.name);
+                 findIf(v, "mn", to.minVal);
+                 findIf(v, "mx", to.maxVal);
+                 findIf(v, "df", to.defaultVal);
+                 findIf(v, "ts", to.canTemposync);
+                 findIf(v, "de", to.canDeactivate);
+                 findIf(v, "ex", to.canExtend);
+                 findIf(v, "ab", to.canAbsolute);
+                 findIf(v, "tsm", to.temposyncMultiplier);
+                 findIf(v, "ssc", to.supportsStringConversion);
+                 findEnumIf(v, "dsc", to.displayScale);
+                 findEnumIf(v, "qt", to.quantization);
+                 findIf(v, "qtp", to.quantizationParam);
                  findIf(v, "unit", to.unit);
-                 findIf(v, "customMinDisplay", to.customMinDisplay);
-                 findIf(v, "customMaxDisplay", to.customMaxDisplay);
+                 findIf(v, "cmid", to.customMinDisplay);
+                 findIf(v, "cmxd", to.customMaxDisplay);
 
                  std::vector<std::pair<int, std::string>> dvStream;
-                 findIf(v, "discreteValues", dvStream);
+                 findIf(v, "discv", dvStream);
                  for (const auto &[k, mv] : dvStream)
                      to.discreteValues[k] = mv;
 
-                 findIf(v, "decimalPlaces", to.decimalPlaces);
+                 findIf(v, "dep", to.decimalPlaces);
                  findIf(v, "svA", to.svA);
                  findIf(v, "svB", to.svB);
                  findIf(v, "svC", to.svC);

diff --git a/src/json/dsp_traits.h b/src/json/dsp_traits.h
@@ -43,7 +43,7 @@ SC_STREAMDEF(scxt::dsp::processor::ProcessorStorage, SC_FROM({
                  auto &t = from;
                  if (t.type == dsp::processor::proct_none)
                  {
-                     v = {{"type", scxt::dsp::processor::getProcessorStreamingName(t.type)}};
+                     v = tao::json::empty_object;
                  }
                  else
                  {
@@ -62,8 +62,13 @@ SC_STREAMDEF(scxt::dsp::processor::ProcessorStorage, SC_FROM({
                  auto &result = to;
                  const auto &object = v.get_object();
 
-                 auto optType = scxt::dsp::processor::fromProcessorStreamingName(
-                     v.at("type").template as<std::string>());
+                 std::string type;
+                 findOrSet(
+                     v, {"t", "type"},
+                     scxt::dsp::processor::getProcessorStreamingName(dsp::processor::proct_none),
+                     type);
+
+                 auto optType = scxt::dsp::processor::fromProcessorStreamingName(type);
 
                  if (optType.has_value())
                      result.type = *optType;
+14 −2		include/sst/basic-blocks/mod-matrix/ModMatrix.h
+41 −0		tests/mod_matrix_tests.cpp