Minitest::Smartdiff

Have you ever found yourself staring cross-eyed and bewildered at walls of diffs that are absolutely the same damn it.

Well, it's the future, and you don't have to do that anymore. Let the endlessly patient Robot go looking for the needle in the needle stack.

Minitest::Smartdiff is a minitest plugin that replaces Minitest's (notoriously lacking) diff with an LLM call. The LLM will read the two values in your assertion, and provide you with a plain english description of what the head-smackingly obvious difference is.

It works with Strings, JSON strings and Ruby primitives that can be serialized to JSON.

Installation

Add this line to your application's Gemfile:

gem 'minitest-smartdiff'

And then execute:

$ bundle

Or install it yourself as:

$ gem install minitest-smartdiff

Usage

In your test_helper.rb file:

require 'minitest-smartdiff'

And in your test file:

class MyTestThatProducesAnAnnoyingDiff < Minitest::Test
  include Minitest::Smartdiff

  def test_that_produces_an_annoying_obscure_diff
    some_result = my_method()

    smart_diff do
      assert_equal "Mayonnaise is delicious", "Mayonnaise is delicioous"
    end
  end
end

Alternatively - you can call smart_diff_on and smart_diff_off to turn the functionality on/off without using a block.

Smartdiff is smart enough not to ask the LLM for the difference if the assertion passes, so you don't accrue OpenAI costs for passing tests. However - you should consider this primarily a debugging tool - use at high volume at your own risk.

You can configure the OpenAI client like so:

class MyTestThatProducesAnAnnoyingDiff < Minitest::Test
  include Minitest::Smartdiff

  openai({
    access_token: ENV['OPENAI_KEY']
  })

  model('gpt-3.5-turbo')

  prompt(<<~ERB
    You are a differ - you diff things.  Diff this:
    <%= expected %>
    <%= actual %>
    <%= mode %> is either `:json`, `:object` or `:text`
    ERB
  )


  def test_that_produces_an_annoying_obscure_diff
    some_result = my_method()

    smart_diff do
      assert_equal "Mayonnaise is delicious", "Mayonnaise is delicioous"
    end
  end
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/stephenprater/minitest-smartdiff.