Support usage in Babashka environments

[RickMoynihan]

Babashka is a dialect of Clojure that supports the Small Clojure Interperter sci as a graalvm compiled native image. It's main usecase is quick startup times, so it's suitable as a bash replacement, but also in other environments such as in AWS lambda deployments.

As an experiment I tried porting integrant to work with babashka, only minimal changes are required. You can see this work in my integrant fork, and in my fork of dependency.

In the process of this work we've identified a few areas of improvement within sci and babashka that will require even less changes to integrant and dependency. Specifically these issues:

• Add java.io.FileNotFoundException to babashka.impl.classes babashka/babashka#581 (fixed on babashka master)
• defrecord: constructor call not recognized in protocol impls babashka/sci#419 (fixed in sci and now on babashka master -- should mean we no longer need to maintain a fork of the dependency lib)
• [#420] Add support for clojure.core/find-var babashka/sci#424 (will make running the tests in :bb env easier, and potentially also help making spec optional).

@weavejester I'm wondering if once these issues are resolved, whether you'd consider officially incorporating babashka support into integrant? And whether I should work my forks changes up into a revised PR, that reverts the commits related to those issues and includes running the tests under babashka in the travis CI environment?

[RickMoynihan]

I should state another option is obviously just to not incorporate the changes upstream, and to simply use my fork in place of integrant. If I leave the namespace names the same, users will only need to include my dependency in place of integrant, and any components on the classpath will then work in the bb environment.

Indeed this may be a better option in the short/medium term.

I mainly just wanted to make you aware of the work, and how small the required changes are.

[borkdude]

For those not familiar with babashka: it supports reader conditionals (:bb) so there are no limitations for the JVM or CLJS when deciding on support for babashka.

[RickMoynihan]

Also many, many thanks to @borkdude for filing and fixing those issues so quickly!

[weavejester]

That seems fine in principle. I don't like the extra dependency needed for babashka's spec, but we could make spec support optional anyway by dynamically loading it in assert-pre-init-spec.

[RickMoynihan]

Ok, can you ellaborate some more on how you think that might work?

Whilst we can load spec dynamically as it stands integrant still needs to call s/valid? etc so does it not need spec to be available at read time? I guess we could possibly use something like dynaload but that's also adding an extra dependency, and under babashka s/valid? etc will want to resolve to spartan.spec.

[weavejester]

At least in Clojure (I'm not sure about babashka but I assume it works the same way), you can lookup vars dynamically via find-var:

(defn- assert-pre-init-spec [system key value]
  (when-let [spec (pre-init-spec key)]
    (require 'clojure.spec.alpha)
    (let [valid?       (find-var 'clojure.spec.alpha/valid?)
          explain-data (find-var 'clojure.spec.alpha/explain-data)]
      (when-not (valid? spec value)
        (throw (spec-exception system key value spec (explain-data spec value)))))))

[RickMoynihan]

@weavejester well it's funny you should mention find-var as look at what I'm in the process of adding to sci babashka/sci#424 😄 . It should make it into a babashka release shortly.

Regarding assert-pre-init-spec, firstly if we're using find-var is it actually even necessary to do that require, as can't we rely on the namespace that defines the pre-init-spec key to have already required spec or spartan.spec for us? If so we could just test if find-var returns a var and only check the specs if it does.

Secondly for the babashka support I'll still need to put a :bb reader conditional with a reference to spartan.spec over those find-var calls, I'm guessing are you ok with that?

Finally I should add that it's unclear at this stage how babashka will support clojure.spec in the long term. If they add spec 1 this issue will disappear, however this also raises the question about how integrant might support spec2?

Obviously spec2 is a bit of an unknown at this point, however it may be useful to think about how integrant might support it. I guess the easy thing would be to either add a pre-init-spec2 function or to do some dispatch dependent on the kind of spec returned by pre-init-spec.

If the later happened it would potentially allow users to pick their own framework for validating component config, e.g. mali, spec1, spec2 or spartan. However I worry that having a less opiniated stance of specing config would potentially lead to a hotchpotch of validation frameworks being used and required in apps, so would probably err against it, with perhaps the notable exception of spec.

[RickMoynihan]

Just a note I've been updating the status of the linked issues, as fixes to sci/babashka get pushed upstream.

I'll try and update my fork at some point soon to account for the recent upstream changes (which will mean less changes in integrant).

[weavejester]

Regarding assert-pre-init-spec, firstly if we're using find-var is it actually even necessary to do that require, as can't we rely on the namespace that defines the pre-init-spec key to have already required spec or spartan.spec for us?

Likely most of the time we don't, but there's no harm in adding it as if the namespace is already loaded it does nothing, and if there's some edge condition we haven't thought of, it guarantees we do have the necessary namespace available.

Secondly for the babashka support I'll still need to put a :bb reader conditional with a reference to spartan.spec over those find-var calls, I'm guessing are you ok with that?

Yep.

Finally I should add that it's unclear at this stage how babashka will support clojure.spec in the long term. If they add spec 1 this issue will disappear, however this also raises the question about how integrant might support spec2?

Good question; I'm not sure yet.

[ieugen]

Hi, any plans to make progress on this?

[borkdude]

I think integrant should already work in babashka, or at least, some version in the past did since I'm running all integrants unit tests on every new commit in babashka:

https://github.com/babashka/babashka/tree/master/test-resources/lib_tests/integrant

[borkdude]

I'm testing with integrant/integrant {:mvn/version "0.8.0"} on babashka CI.

[borkdude]

Tested locally with 0.8.1, this also still worked, but 0.9.0-alpha1 gave:

Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: ig/pre-init-spec
Data:     {:type :sci/error, :line 27, :column 1, :file "/Users/borkdude/dev/babashka/test-resources/lib_tests/integrant/core_test.cljc", :phase "analysis"}
Location: /Users/borkdude/dev/babashka/test-resources/lib_tests/integrant/core_test.cljc:27:1
Phase:    analysis

----- Context ------------------------------------------------------------------
23:
24: (defmethod ig/init-key ::k [_ v] v)
25:
26: (defmethod ig/init-key ::n [_ v] (inc v))
27: (defmethod ig/pre-init-spec ::n [_] nat-int?)
    ^--- Could not resolve symbol: ig/pre-init-spec
28:
29: (defmethod ig/init-key ::r [_ v] {:v v})
30: (defmethod ig/resolve-key ::r [_ {:keys [v]}] v)
31: (defmethod ig/resume-key ::r [k v _ _] (ig/init-key k v))
32:

[borkdude]

Ah wait, this is likely because I also should update the test code. Hang on.

[borkdude]

I get all tests passing with 0.9.0-alpha1 with this change in try-require:

#?(:clj
   (defn- try-require [sym]
     (try (do (require sym) sym)
          (catch Exception _))))

rather than catching a java.io.FileNotFoundException which is more specific than the exception type that babashka throws.

$ script/run_lib_tests integrant.core-test

Testing integrant.core-test

Ran 21 tests containing 120 assertions.
0 failures, 0 errors.
{:test 21, :pass 120, :fail 0, :error 0}

Would this be an acceptable change @weavejester ?
If you're open to this, I could also submit a test runner for bb to detect these issues early on.

[borkdude]

I guess I could also change babashka to throw a FileNotFoundException to align more with Clojure:

user=> (require 'dude)
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate dude__init.class, dude.clj or dude.cljc on classpath.
user=> (type *e)
java.io.FileNotFoundException

I'll do that instead.

[borkdude]

Merged that change an all existing integrant tests now run with bb master. @ieugen you could try to run the master version with:

bash <(curl https://raw.githubusercontent.com/babashka/babashka/master/install) --dev-build --dir /tmp

when the current master build finishes.

If there's interest in a bb test runner in this project, let me know.

[weavejester]

A bb test runner might help prevent accidental future regressions.

[borkdude]

OK, will send one tomorrow.

[borkdude]

As discussed: #103