Unit testing framework

[gitonthescene]

Let’s use this issue to discuss plans for a unit testing framework. There are probably enough decisions to be made to split this off the head issue.

[gitonthescene]

• I did some preliminary reconnaissance work here

[gitonthescene]

Just to reiterate, we’re targeting this API.

There doesn’t seem to be any “official” BQN unit testing framework though there are obviously tests in the BQN source. The API is mostly about how test results are reported more than how they are run.

Poking around some other test runners people seemed to be able to adapt pre-built frameworks to the API, so following some best practices will probably get us there.

@razetime points out that the J test runner uses a well established framework written in an array language. That may be the easiest to adapt but no decisions have been made yet.

[gitonthescene]

FWIW, creating the actual runner is mostly Docker work, which is probably a separate task from making a framework for creating tests and producing summaries of the results. It’s probably best to leave that for another issue.

[razetime]

One reference we can use is the Fortran test runner

[razetime]

One idea from @RocketRace was to have a way to group tests, similar to most languages. pass/fail data is kept in each namespace.

"foo should error on invalid numbers" TestCase {
    # where x is some namespace exposing the assert functions, now with extra context
    Foo x._must_error 42
}

I have a different idea, storing everything within the library namespace.

[aecepoglu]

I think you are imagining building a testing framework and adding an exercism track at the same time.

1. I would go for the least-effort path for now; which is to have :

tests ⇐ ⟨
"multiplication" Describes 15 ≡ 3 Foo 5
,"mul by zero" Describes 0 ≡ 0 Foo 5
⟩

I don't think assertions for errors and whatnot are priorities.

If anyone (including one of us) has written a testing framework soon enough, we can simply use it. Unless that's the case though, I don't think we'll need more than ≡ or ⎊for exercism tests.

2. A CLI program test-runner.bqn <slug> <input-dir> <output-dir> will ⟨tests⟩←•Import slug∾".bqn" and then provide an output that exercism expects. We have a JSON lib now so this should be easier than before.

So basically, I think any idea will work. Lets just pick one and go with it :) But w/ regards to how comprehensive a testing framework we have, I think we need very little.

[razetime]

Yeah basic assertions should do well enough for tests, but there seems enough incentive to add some simple grouping for assertions.

[aecepoglu]

I come from a school of writing tests where each test has a single assertion, so the need to group didn't initially occur to me. But I understand the need.

Could we open a {block} and {!assert1⋄!assert2}

And I also feel like some of these may be easier to build than to discuss :) So I will simply say I will follow the decisions taken and figure out what I can contribute to them.

[razetime]

i think an immediate block is not as useful, we need something like

"title" _group_ {𝕊:
  Assert thing
  Assert thing2
}

and so on, so we can store test data. It can only be advantageous anyway since we will need to pretty print it and also use it for the test runner

[razetime]

beginnings of this framework are at https://github.com/razetime/bqn-unit. react to this post for an invitation.

[aecepoglu]

i think an immediate block is not as useful, we need something like

"title" _group_ {𝕊:
  Assert thing
  Assert thing2
}

I think BQN has all the syntax necessary to write tests and that we genuinely don't need much more utils besides a pretty diff printer.

And I understand the desire to group multiple assertions under one test but it's neither a need nor is it commonly recommended:

So we really don't need to do stuff like this:

"foo returns prime divisors" TestCase {
 a ← Foo 42
 Assert 3 expect.LengthEquals a
 Assert ⟨2,3,7⟩ expect.Equals a
}

Because one assertion is(should be) enough for each test:

"foo returns prime divisors" TestCase_v1 {
 a ← Foo 42
 ⟨2,3,7⟩ expect.Equals a
}

Where expect.Equals←⋈ is sufficient, by the way. We only need to check if ≡ is truthy and print the left and right values in the final test report.

[razetime]

The thing is, eventually we may have to link the test data into the runner.

So now I am thinking it is better that we have a list of namespaces, like so. For what it's worth, not too complicated and based on ideas from minitest:

unit ← •Import "unit-test.bqn"

tests ← ⟨
  {
    describe ← "description"
    criteria ← ⟨
      it         ← "does a certain thing"
      assertions ← ⟨
        ⟨{𝕊: Fun⟜∾"thing"}, "a thing"⟩
      ⟩
    ⟩
  },
 (More describe clauses here...)
⟩

•args unit.RunWithArgs tests

This way we have test data usable elsewhere, and a script that runs the tests with args. If this sounds good, I will continue with it in razetime/bqn-unit and complete it very soon.

[aecepoglu]

I still feel like this API is too verbose. Do you have an example for how you'd write tests for a real-life modulle?

[razetime]

hmm, yeah, it is verbose. Namespaces are a little nicer than having global state. however.

the point is tests is a list of namespaces,

• having a describe field implies that it is a group
• having an it field implies that it will have a list of assertions in assertions

unit.RunWithArgs will check for command line args so that importing a test file does not do anything by itself. We can also look at environment variables for this.

If you have a better idea for doing the same, we can try that.

unit ← •Import "unit-test.bqn"

tests ← ⟨
  {
    describe ← "Basic Tests"
    criteria ← ⟨
      it         ← "thinks 1 + 2 = 3"
      assertions ← ⟨
        ⟨{𝕊: 1 Fun 2}, 3⟩
      ⟩
    ⟩
  }
⟩

•args unit.RunWithArgs tests

[aecepoglu]

OK, I'll very quickly and briefly share my idea for the framework with little regard for what we have:

# arithmetic.test.bqn
unit <- •Import "unit-tests.bqn"

"summation" Test {𝕊x:
  2 AssertEquals 1 + 1
}

# Stack Machine Tests
{
  # Do once before all assertions
  machine ← 3 StackMachinePush MakeStackMachine 0

  "stack machine summation" Test {𝕊x:
    4 AssertEquals StackMachinePop "+" StackMachinePush 1 StackMachinePush machine
  }
  "stack machine subtraction" Test {𝕊x:
    2 AssertEquals StackMachinePop "-" StackMachinePush 1 StackMachinePush machine
  }
{

# Impure Stack Machine Tests
{
  # Do once before all assertions
  machine_id ← RegisterStackMachine MakeStackMac3 StackMachinePush hine 0

  ResettingTest ← Test∘{𝕊x: ResetStackMachine machine_id ⋄ x}

  "stack machine summation" ResettingTest {𝕊x:
    4 AssertEquals ImpureStackMachinePop "+" ImpureStackMachinePush 1 ImpureStackMachinePush machine_id
  }
  "stack machine subtraction" ResettingTest {𝕊x:
    2 AssertEquals ImpureStackMachinePop "-" ImpureStackMachinePush 1 ImpureStackMachinePush machine_id
  }
}

# setup.bqn
unit ←•Import "unit-test-library.bqn"
PrettyPrintResults RunTests¨ ⟨"./arithmetic.test.bqn"⟩

# In a shell, do this to run the tests
bqn setup.bqn

[razetime]

ok, this seems good for now.
there's some ideas on the exercism forums but we can extend to accomodate those later.