Examples | Contributing | Code of Conduct | License
Experiment is a Go package to test and evaluate new code paths without interfering with the users end result.
This is inspired by the GitHub Scientist gem.
Imagine a web application where you're generating images. You decide to investigate a new imaging package which seems to fit your needs more than the current package you're using. Tests help you transition from one package to the other, but you want to see how this behaves under load.
package main
import (
"github.com/jelmersnoeck/experiment/v3"
)
func main() {
exp := experiment.New[string](
experiment.WithPercentage(50),
experiment.WithConcurrency(),
)
// fetch arbitrary data
userData := getUserData()
exp.Control(func(context.Context) (string, error) {
return dataToPng.Render(userData)
})
exp.Candidate("", func(context.Context) (string, error) {
return imageX.Render(userData)
})
result, err := exp.Run(context.Background())
}This allows you to serve the original content, dataToPng.Render() to the user
whilst also testing the new package, imageX, in the background. This means
that your end-user doesn't see any impact, but you get valuable information
about your new implementation.
This package uses go modules. To import it, use github.com/jelmersnoeck/experiment/v3 as import path.
Control(func(context.Context) (any, error)) should be used to implement your
current code. The result of this will be used to compare to other candidates.
This will run as it would run normally.
A control is always expected. If no control is provided, the experiment will panic.
func main() {
exp := experiment.New[string](
experiment.WithPercentage(50),
)
exp.Control(func(context.Context) (string, error) {
return fmt.Sprintf("Hello world!"), nil
})
result, err := exp.Run(context.Background())
if err != nil {
panic(err)
} else {
fmt.Println(result)
}
}The example above will always print Hello world!.
Candidate(string, func(context.Context) (any, error)) is a potential
refactored candidate. This will run sandboxed, meaning that when this panics,
the panic is captured and the experiment continues.
A candidate will not always run, this depends on the WithPercentage(int)
configuration option and further overrides.
func main() {
exp := experiment.New[string](
experiment.WithPercentage(50),
)
exp.Control(func(context.Context) (string, error) {
return fmt.Sprintf("Hello world!"), nil
})
exp.Candidate("candidate1", func(context.Context) (string, error) {
return "Hello candidate", nil
})
result, err := exp.Run(context.Background())
if err != nil {
panic(err)
} else {
fmt.Println(result)
}
}The example above will still only print Hello world!. The candidate1
function will however run in the background 50% of the time.
Run(context.Context) will run the experiment and return the value and error of the control
function. The control function is always executed. The result value of the
Run(context.Context) function is an interface. The user should cast this to the expected
type.
Force(bool) allows you to force run an experiment and overrules all other
options. This can be used in combination with feature flags or to always run
the experiment for admins for example.
Ignore(bool) will disable the experiment, meaning that it will only run the
control function, nothing else.
Compare(any, any) bool is used to compare the control value
against a candidate value.
If the candidate returned an error, this will not be executed.
Clean(any) any is used to clean the output values. This is
implemented so that the publisher could use this cleaned data to store for later
usage.
If the candidate returned an error, this will not be executed and the
CleanValue field will be populated by the original Value.
Due to the fact that it is not guaranteed that a test will run every time or in what order a test will run, it is suggested that experiments only do stateless changes.
When enabling the WithConcurrency() option, keep in mind that your tests will
run concurrently in a random fashion. Make sure accessing your data concurrently
is allowed.
By default, the candidates run sequentially. This means that there could be a significant performance degradation due to slow new functionality.
When running with the WithConcurrency() option, the tests will run
concurrently and the control result will be returned as soon as possible. This
does however mean that the other candidates are still running in the background.
Be aware that this could lead to potential memory leaks and should thus be
monitored closely.
An Observation contains several attributes. The first one is the Value. This
is the value which is returned by the control function that is specified. There
is also an Error attribute available, which contains the error returned.
When the control errors, this will be returned in the Run(context.Context)
method. When a candidate errors, this will be attached to the Error field in
its observation.
An error marks the experiment as a failure.
When the control panics, this panic will be respected and actually be triggered.
When a candidate function panics, the experiment will swallow this and assign
this to the Panic field of the observation, which you can use in the
Publisher. An ErrCandidatePanic will also be returned.
If the WithConcurrency() configuration option is passed to the constructor,
the experiment will run its candidates in parallel. The result of the control
will be returned as soon as it's finished. Other work will continue in the
background.
This is disabled by default.
WithPercentage(int) allows you to set the amount of time you want to run the
experiment as a percentage. Force and Ignore do not have an impact on this.
This is set to 0 by default to encourage setting a sensible percentage.
Publishers are used to send observation data to different locations to be able to get insights into said observations. There is a simple Publisher, the LogPublisher, which writes all observations to the logger you provide in it.
WithPublisher(Publisher) marks the experiment as Publishable. This means that
all the results will be pushed to the Publisher once the experiment has run.
This is nil by default.
By default, there is the LogPublisher. This Publisher will log out the
Observation values through a provided logger or the standard library logger.
func main() {
exp := experiment.New[string](
experiment.WithPercentage(50),
).WithPublisher(experiment.NewLogPublisher[string]("publisher", nil))
exp.Control(func(context.Context) (string, error) {
return fmt.Sprintf("Hello world!"), nil
})
exp.Candidate("candidate1", func(context.Context) (string, error) {
return "Hello candidate", nil
})
exp.Force(true)
result, err := exp.Run(context.Context)
if err != nil {
panic(err)
} else {
fmt.Println(result)
}
}When the experiment gets triggered, this will log out
[Experiment Observation: publisher] name=control duration=10.979µs success=false value=Hello world! error=<nil>
[Experiment Observation: publisher] name=candidate1 duration=650ns success=false value=Hello candidate error=<nil>