Releasing a new version

https://dev-accel.shopify.io/guides/package_cloud_gems/

enterprise-script-service

The enterprise script service (aka ESS) is a thin Ruby API layer that spawns a process, the enterprise_script_engine, to execute an untrusted Ruby script.

The enterprise_script_engine executable ingests the input from stdin as a msgpack encoded payload; then spawns an mruby-engine; uses seccomp to sandbox itself; feeds library, input and finally the Ruby scripts into the engine; returns the output as a msgpack encoded payload to stdout and finally exits.

Data format

Input

The input is expected to be a msgpack MAP with three keys (Symbol): library, sources, input:

• library: a msgpack BIN set of MRuby instructions that will be fed directly to the mruby-engine

• input: a msgpack formated payload for the sources to digest

• sources: a msgpack ARRAY of ARRAY with two elements each (tuples): path, source; the actual code to be executed by the mruby-engine

Output

The output is msgpack encoded as well; it is streamed to the consuming end though. Streamed items can be of different types. Each element streamed is in the format of an ARRAY of two elements, where the first is a Symbol describing the element type:

• measurement: a msgpack ARRAY of two elements: a Symbol describing the measurement, and an INT64 with the value in µs.

• output: a msgpack MAP with two entries (keys are symbols):

  • extracted with whatever the script put in @output, msgpack encoded; and

  • stdout with a STRING containing whatever the script printed to "stdout".

• stat: a MAP keyed with symbols mapping to their INT64 values

Errors

When the ESS fails to serve a request, it communicates the error back to the caller by returning a non-zero status code. It can also report data about the error, in certain cases, over the pipe. In does so in returning a tuple, as an ARRAY with the type being the symbol error and the payload being a MAP. The content of the map will vary, but it always will have a __type symbol key that defines the other keys.

Build

Run ./bin/rake to build the project. This effectively runs the spec target, which builds all libraries, the ESS and native tests; then runs all tests (native and Ruby).

To rebuild the entire project (which is useful when switching from one OS to another), use ./bin/rake mrproper.

Using it

The sample script bin/sandbox reads Ruby input from a file or stdin, executes it, and displays the results.

You can invoke ESS from your own Ruby code as follows:

result = EnterpriseScriptService.run(
  input: {result: [26803196617, 0.475]}, # (1)
  sources: [
    ["stdout", "@stdout_buffer = 'hello'"],
    ["foo", "@output = @input[:result]"], # (2)
  ],
  instructions: nil, # (3)
  timeout: 10.0, # (4)
  instruction_quota: 100000, # (5)
  instruction_quota_start: 1, # (6)
  memory_quota: 8 << 20  # (7)
)
expect(result.success?).to be(true)
expect(result.output).to eq([26803196617, 0.475])
expect(result.stdout).to eq("hello")

1. invokes the ESS, with a map as the input (available as @input in the sources)

2. two "scripts" to be executed, one sets the @stdout_buffer to a value, the second returns the value associated with the key :result of the map passed in in <1>

3. some raw instructions that will be fed directly into MRuby; defaults to nil

4. a 10 second time quota to spawn, init, inject, eval and finally output the result back; defaults to 1 second

5. a 100k instruction limit that that the engine will execute; defaults to 100k

6. starts counting the instructions at index 1 of the sources array

7. creates an 8 megabyte memory pool in which the script will run

Where are things?

C++ sources

Consists of our code base, plus seccomp and msgpack libraries, as well as the mruby stuff. All in ext/enterprise_script_service

Note: lib seccomp is omitted on Darwin.

Ruby layer

Ruby code is in lib/

Tests

• googletest tests are in tests/, which also includes the Google Test library.

• RSpec tests are in spec/

Other useful things

• There is a CMakeLists.txt that’s mainly there for CLion support; we don’t use cmake to build any of this.

• You can use UTM to bootstrap an x86_64 VM to test with Linux while on MacOS; this is useful when testing seccomp. Ubuntu 24.04 Server for AMD64 is the recommended Linux image to use. Note that the script_runner_test.defaults_to_counting_all_instructions and script_runner_test.can_bypass_deserialization_instructions tests will fail with execution_time_us being greater than 0 due to execution time taking longer because the tests are being run in an emulated environment. As long as all tests pass in CI, it’s fine if they fail in UTM when running on emulated hardware.

Ubuntu VM

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-add-repository -y ppa:rael-gc/rvm
$ sudo apt-get update
$ sudo apt-get install rvm build-essential git libncurses5-dev libgmp-dev libssl-dev openssh-server net-tools gdb
$ sudo systemctl start ssh
$ ifconfig # record the IP address for the non-loopback interface
$ sudo usermod -a -G rvm $USER
$ sudo reboot
$ rvm install 3.3.0 # (this may take a while)
$ git clone https://github.com/Shopify/ess.git
$ cd ess
$ git submodule update --init --recursive
$ bundle install
$ bin/rake compile

To SSH in:
$ ssh <vm_username>@<vm_ipaddress>

If you use VS Code, you can also use the _Remote-SSH: Connect to Host..._ functionality in VS Code to connect to the VM.