ARROW-2104: [C++] take kernel functions for nested types #4531

bkietz · 2019-06-12T16:32:42Z

Take now supports gathering from List, FixedSizeList, Map, and Struct arrays. Union is not yet supported

pitrou

The implementation is neat! Some comments below.

python/pyarrow/tests/test_compute.py

pitrou · 2019-06-25T12:28:37Z

cpp/src/arrow/compute/kernels/take-internal.h

+}
+
+static Status UnsafeAppend(BinaryBuilder* builder, util::string_view value) {
+  RETURN_NOT_OK(builder->ReserveData(static_cast<int64_t>(value.size())));


Hmm... it would probably be more efficient to do it in two passes:

first pass to compute the overall data length

second pass to do the actual builder appends after having reserved the right data size at once

What do you think? Is it easily doable to switch to such a scheme?

That's not a problem. I'll write a benchmark so we can watch the difference.

@pitrou as it turns out, it's actually a little faster to ReserveData for each string:

# ReserveData on append --------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... --------------------------------------------------------------------------------------------- FilterString/1048576/1/min_time:1.000 1586715 ns 1586705 ns 884 null_percent=1 size=1048.58k 630.237MB/s FilterString/8388608/1/min_time:1.000 23555110 ns 23555006 ns 72 null_percent=1 size=8.38861M 339.631MB/s # separate ReserveData pass --------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... --------------------------------------------------------------------------------------------- FilterString/1048576/1/min_time:1.000 1656093 ns 1654443 ns 841 null_percent=1 size=1048.58k 604.433MB/s FilterString/8388608/1/min_time:1.000 19849301 ns 19809235 ns 71 null_percent=1 size=8.38861M 403.852MB/s

Would you like me to push this change so you can inspect it?

It's faster on the 1MB case, but slower on the 8MB case. I expect it to become even slower on larger data. Can you try with e.g. 32MB?

(what is the size of your L3 cache?)

Update: we discussed and investigated it a bit with @bkietz . It turns out that the dominating contributor to Filter performance is probably reading the selection bitmap, so iterating twice on the input instead of once has a negative impact.

cpp/src/arrow/compute/kernels/take-internal.h

cpp/src/arrow/compute/kernels/take-test.cc

bkietz · 2019-06-25T13:16:08Z

The benchmarks for Filter with this refactor show a slight improvement over previous:

----------------------------------------------------------------------------------------------------------
Benchmark                                                   Time           CPU Iterations UserCounters...
----------------------------------------------------------------------------------------------------------
FilterInt64/32768/0/min_time:1.000                      16321 ns      16321 ns      85904 null_percent=0 size=32.768k   1.86981GB/s
FilterInt64/32768/1/min_time:1.000                      16431 ns      16431 ns      84487 null_percent=1 size=32.768k   1.85729GB/s
FilterInt64/32768/10/min_time:1.000                     19935 ns      19935 ns      70396 null_percent=10 size=32.768k   1.53088GB/s
FilterInt64/32768/50/min_time:1.000                     39153 ns      39153 ns      35976 null_percent=50 size=32.768k   798.148MB/s
FilterInt64/1048576/1/min_time:1.000                   882180 ns     882176 ns       1590 null_percent=1 size=1048.58k   1.10699GB/s
FilterInt64/8388608/1/min_time:1.000                  7038037 ns    7038002 ns        200 null_percent=1 size=8.38861M   1.11005GB/s
FilterFixedSizeList1Int64/32768/0/min_time:1.000        44414 ns      44414 ns      31993 null_percent=0 size=32.768k   703.614MB/s
FilterFixedSizeList1Int64/32768/1/min_time:1.000        45416 ns      45416 ns      30649 null_percent=1 size=32.768k    688.08MB/s
FilterFixedSizeList1Int64/32768/10/min_time:1.000       57006 ns      57006 ns      24561 null_percent=10 size=32.768k   548.193MB/s
FilterFixedSizeList1Int64/32768/50/min_time:1.000      104088 ns     104088 ns      13318 null_percent=50 size=32.768k   300.227MB/s
FilterFixedSizeList1Int64/1048576/1/min_time:1.000    1590805 ns    1590798 ns        887 null_percent=1 size=1048.58k   628.615MB/s
FilterFixedSizeList1Int64/8388608/1/min_time:1.000   12718793 ns   12718723 ns        110 null_percent=1 size=8.38861M   628.994MB/s

pitrou · 2019-06-25T16:21:00Z

cpp/src/arrow/compute/kernels/filter-benchmark.cc

+static void FilterString(benchmark::State& state) {
+  RegressionArgs args(state);
+
+  const int64_t array_size = args.size / sizeof(int64_t);


It doesn't seem right to divide by int64_t here.

@fsaintjacques could you comment?

My intent was to make the benchmark share a memory budget (args.size) and let them size the input. The goal is to be able to compare operations. So if you don't input any physical sizes and just input length, no need to divide.

Right. So the problem here is to generate a random string array roughly consistent with the given memory budget.

bkietz · 2019-06-26T00:49:05Z

CI failures seem spurious:

pitrou · 2019-06-26T11:56:29Z

Rebased to fix unrelated CI failures.

bkietz · 2019-06-26T14:21:32Z

@pitrou when you rebased you missed the last two commits somehow ( a4bfaac 9cee25e ).

I'll update the filter benchmark to produce a StringArray with ~(memory budget) total characters and rebase again.

pitrou · 2019-06-26T14:23:37Z

@bkietz Wow, not sure how that happened :-/ Sorry for that.

bkietz · 2019-06-26T14:29:22Z

No worries, easily fixed

bkietz · 2019-06-26T19:34:48Z

Spurious Appveyor failure while downloading R installer

wesm · 2019-06-27T13:48:06Z

There was an idiosyncratic Maven failure. I restarted the one failing Travis job

wesm · 2019-06-27T13:48:48Z

Appveyor failed due to the ongoing cloud.r-project.org DNS issue. We should probably add this test to "allow_failures" until that is resolved

bkietz · 2019-06-27T15:08:12Z

@wesm anything I need to do to accomplish that?

ursabot · 2019-06-27T17:37:15Z

AMD64 Ubuntu 18.04 C++ Benchmark (#27346) builder has been succeeded.

Revision: 73262bd

  ===================================  ===========  ===========  ==========
  benchmark                               baseline    contender      change
  ===================================  ===========  ===========  ==========
  FilterString/32768/1/min_time:1.000  4.33183e+09  4.25759e+09  -0.0171373
  ===================================  ===========  ===========  ==========

fsaintjacques · 2019-06-27T17:39:36Z

@ursabot benchmark --benchmark-filter=TakeString HEAD^1

ursabot · 2019-06-27T17:39:50Z

AMD64 Ubuntu 18.04 C++ Benchmark (#27347) builder failed.

Revision: 73262bd

Archery: 'archery benchmark ...' step's stderr:

Selected compiler gcc 7.4.0
Using ld linker
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
CMake Warning at cmake_modules/ThirdpartyToolchain.cmake:176 (find_package):
  No "Findbenchmark.cmake" found in CMAKE_MODULE_PATH.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:1610 (resolve_dependency)
  CMakeLists.txt:389 (include)


CMake Warning (dev) at cmake_modules/ThirdpartyToolchain.cmake:176 (find_package):
  Findbenchmark.cmake must either be part of this project itself, in this
  case adjust CMAKE_MODULE_PATH so that it points to the correct location
  inside its source tree.

  Or it must be installed by a package which has already been found via
  find_package().  In this case make sure that package has indeed been found
  and adjust CMAKE_MODULE_PATH to contain the location where that package has
  installed Findbenchmark.cmake.  This must be a location provided by that
  package.  This error in general means that the buildsystem of this project
  is relying on a Find-module without ensuring that it is actually available.

Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:1610 (resolve_dependency)
  CMakeLists.txt:389 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

Cloning into '/tmp/arrow-bench-t9n1i_r7/'HEAD^1'/arrow'...
done.
error: pathspec ''HEAD^1'' did not match any file(s) known to git.
Traceback (most recent call last):
  File "/usr/local/bin/archery", line 11, in <module>
    load_entry_point('archery', 'console_scripts', 'archery')()
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/cli.py", line 384, in benchmark_diff
    suite_filter=suite_filter, benchmark_filter=benchmark_filter)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/benchmark/runner.py", line 75, in from_rev_or_path
    src_rev, _ = src.at_revision(rev_or_path, clone_dir)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/utils/source.py", line 90, in at_revision
    git.checkout(revision, git_dir=clone_dir)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/utils/git.py", line 30, in wrapper
    return fn(self, sub_cmd, *argv, **kwargs)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/utils/git.py", line 52, in checkout
    return self.run_cmd(*argv, **kwargs)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/utils/git.py", line 44, in run_cmd
    return self.run(*opts, cmd, *argv, **kwargs)
  File "/buildbot/AMD64_Ubuntu_18_04_C___Benchmark/dev/archery/archery/utils/command.py", line 68, in run
    return subprocess.run(invocation, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/git', '-C', "/tmp/arrow-bench-t9n1i_r7/'HEAD^1'/arrow", 'checkout', "'HEAD^1'"]' returned non-zero exit status 1.

fsaintjacques · 2019-06-27T17:39:56Z

@ursabot benchmark --help

ursabot · 2019-06-27T17:39:57Z

Usage: @ursabot benchmark [OPTIONS] [BASELINE]

  Trigger all benchmarks registered for this pull request.

Options:
  --suite-filter <regex>      Regex filtering benchmark suites.
  --benchmark-filter <regex>  Regex filtering benchmarks.
  --help                      Show this message and exit.

fsaintjacques · 2019-06-27T17:45:17Z

@ursabot benchmark --benchmark-filter=TakeString HEAD~2

ursabot · 2019-06-27T17:55:01Z

AMD64 Ubuntu 18.04 C++ Benchmark (#27353) builder failed with an exception.

Revision: 73262bd

Archery: 'archery benchmark ...' step's traceback:

Traceback (most recent call last):
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/process/buildstep.py", line 566, in startStep
    self.results = yield self.run()
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/ursabot/ursabot/ursabot/steps.py", line 43, in run
    await log.addContent(content)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/process/log.py", line 130, in addContent
    return self.lbf.append(text)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/util/lineboundaries.py", line 62, in append
    text = self.newline_re.sub('\n', text)
builtins.TypeError: expected string or bytes-like object

fsaintjacques · 2019-06-27T17:57:45Z

@ursabot benchmark --benchmark-filter=TakeString HEAD~1

ursabot · 2019-06-27T18:07:29Z

AMD64 Ubuntu 18.04 C++ Benchmark (#27354) builder failed with an exception.

Revision: 73262bd

Archery: 'archery benchmark ...' step's traceback:

Traceback (most recent call last):
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/process/buildstep.py", line 566, in startStep
    self.results = yield self.run()
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/ursabot/ursabot/ursabot/steps.py", line 43, in run
    await log.addContent(content)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/process/log.py", line 130, in addContent
    return self.lbf.append(text)
  File "/home/ursabot/.conda/envs/ursabot/lib/python3.7/site-packages/buildbot/util/lineboundaries.py", line 62, in append
    text = self.newline_re.sub('\n', text)
builtins.TypeError: expected string or bytes-like object

fsaintjacques · 2019-06-27T18:29:31Z

@ursabot benchmark --benchmark-filter=TakeString eaf8302

fsaintjacques · 2019-06-27T18:32:24Z

@ursabot benchmark --help

ursabot · 2019-06-27T18:32:25Z

Usage: @ursabot benchmark [OPTIONS] [BASELINE]

  Trigger all benchmarks registered for this pull request.

Options:
  --suite-filter <regex>      Regex filtering benchmark suites.
  --benchmark-filter <regex>  Regex filtering benchmarks.
  --help                      Show this message and exit.

fsaintjacques · 2019-06-27T18:34:10Z

@ursabot benchmark --help

ursabot · 2019-06-27T18:34:11Z

Usage: @ursabot benchmark [OPTIONS] [<baseline>]

  Run the benchmark suite in comparison mode.

  This command will run the benchmark suite for tip of the branch commit
  against `<baseline>` (or master if not provided).

  Examples:

  # Run the all the benchmarks
  �
  @ursabot benchmark

  # Compare only benchmarks where the name matches the `^Sum` regex
  �
  @ursabot benchmark --benchmark-filter=^Sum

  # Compare only benchmarks where the suite matches the `compute-` regex.
  # A suite is the C++ binary.
  �
  @ursabot benchmark --suite-filter=compute-

  # Sometimes a new optimization requires the addition of new benchmarks to
  # quantify the performance increase. When doing this be sure to add the
  # benchmark in a separate commit before introducing the optimization.
  #
  # Note that specifying the baseline is the only way to compare a new
  # benchmark, otherwise the intersection of benchmarks with master will be
  # empty (no comparison possible).
  #
  # Suppose that the "MyBenchmark" benchmark is introduced in HEAD~2 and the
  # optimization is found in HEAD. The following command will show the
  # difference with and without the optimization on said benchmark.
  �
  @ursabot benchmark --benchmark-filter=MyBenchmark HEAD~2

Options:
  --suite-filter <regex>      Regex filtering benchmark suites.
  --benchmark-filter <regex>  Regex filtering benchmarks.
  --help                      Show this message and exit.

fsaintjacques · 2019-06-27T18:47:41Z

@ursabot benchmark --help

ursabot · 2019-06-27T18:47:42Z

Usage: @ursabot benchmark [OPTIONS] [<baseline>]

  Run the benchmark suite in comparison mode.

  This command will run the benchmark suite for tip of the branch commit
  against `<baseline>` (or master if not provided).

  Examples:

  # Run the all the benchmarks
  @ursabot benchmark

  # Compare only benchmarks where the name matches the /^Sum/ regex
  @ursabot benchmark --benchmark-filter=^Sum

  # Compare only benchmarks where the suite matches the /compute-/ regex.
  # A suite is the C++ binary.
  @ursabot benchmark --suite-filter=compute-

  # Sometimes a new optimization requires the addition of new benchmarks to
  # quantify the performance increase. When doing this be sure to add the
  # benchmark in a separate commit before introducing the optimization.
  #
  # Note that specifying the baseline is the only way to compare a new
  # benchmark, otherwise the intersection of benchmarks with master will be
  # empty (no comparison possible).
  #
  # The following command compares the results of matching benchmarks,
  # compiling against HEAD and the provided baseline commit (HEAD~2). You can
  # use this to quantify the performance improvement of new optimizations or
  # to check for regressions.
  @ursabot benchmark --benchmark-filter=MyBenchmark HEAD~2

Options:
  --suite-filter <regex>      Regex filtering benchmark suites.
  --benchmark-filter <regex>  Regex filtering benchmarks.
  --help                      Show this message and exit.

fsaintjacques · 2019-06-27T18:48:37Z

@ursabot benchmark --benchmark-filter=TakeString eaf8302

ursabot · 2019-06-27T19:00:12Z

AMD64 Ubuntu 18.04 C++ Benchmark (#27385) builder has been succeeded.

Revision: 73262bd

  ===================================  ===========  ===========  ===========
  benchmark                               baseline    contender       change
  ===================================  ===========  ===========  ===========
  TakeString/32768/50/min_time:1.000   1.33361e+09  1.27811e+09  -0.0416122
  TakeString/32768/1/min_time:1.000    1.54006e+09  1.54192e+09   0.00120504
  TakeString/8388608/1/min_time:1.000  7.39268e+08  8.28827e+08   0.121145
  TakeString/32768/0/min_time:1.000    1.74939e+09  1.73941e+09  -0.00570733
  TakeString/32768/10/min_time:1.000   1.52927e+09  1.51239e+09  -0.0110394
  TakeString/1048576/1/min_time:1.000  1.37075e+09  1.36925e+09  -0.00109342
  ===================================  ===========  ===========  ===========

bkietz · 2019-06-27T19:02:19Z

@wesm @pitrou the only remaining CI failure is an abort in test_flight.py

bkietz · 2019-06-27T19:07:13Z

Follow up JIRA for optimization of these kernels:https://issues.apache.org/jira/browse/ARROW-5760

wesm · 2019-06-27T19:32:18Z

can we fix the hideous benchmark formatting to show human readable timings?

bkietz · 2019-06-27T19:33:20Z

Not with a change to Take(), I think

wesm · 2019-06-27T19:34:50Z

Sorry that was a general comment to @fsaintjacques =)

I restarted the failed Travis build to see if it's transient

https://travis-ci.org/apache/arrow/jobs/551366263

definitely concerning that Flight is able to crash like that

fsaintjacques · 2019-06-27T19:35:57Z

@wesm actively working to this next. See voltrondata-labs/ursabot#98

codecov-io · 2019-06-27T20:10:16Z

Codecov Report

❗ No coverage uploaded for pull request base (master@7ae6a58). Click here to learn what that means.
The diff coverage is 96.49%.

@@            Coverage Diff            @@
##             master    #4531   +/-   ##
=========================================
  Coverage          ?   88.87%           
=========================================
  Files             ?      717           
  Lines             ?    98750           
  Branches          ?        0           
=========================================
  Hits              ?    87768           
  Misses            ?    10982           
  Partials          ?        0

Impacted Files	Coverage Δ
cpp/src/arrow/compute/kernels/filter.h	`66.66% <ø> (ø)`
cpp/src/arrow/array.h	`98.29% <ø> (ø)`
cpp/src/arrow/compute/kernels/util-internal.h	`100% <ø> (ø)`
cpp/src/arrow/compute/kernels/take.cc	`97.91% <100%> (ø)`
cpp/src/arrow/buffer-builder.h	`100% <100%> (ø)`
cpp/src/arrow/compute/kernels/take-test.cc	`100% <100%> (ø)`
cpp/src/arrow/compute/kernels/filter.cc	`98.03% <100%> (ø)`
cpp/src/arrow/array/builder_primitive.cc	`92.53% <100%> (ø)`
cpp/src/arrow/compute/kernels/take.h	`75% <100%> (ø)`
cpp/src/arrow/array.cc	`80.33% <100%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ae6a58...73262bd. Read the comment docs.

wesm

+1

bkietz marked this pull request as ready for review June 13, 2019 13:27

fsaintjacques added the Component: C++ label Jun 14, 2019

bkietz force-pushed the 2104-Implement-take-kernel-functions-nested-a branch 2 times, most recently from 50c77f8 to f7bd12b Compare June 24, 2019 15:21

pitrou requested changes Jun 25, 2019

View reviewed changes

pitrou reviewed Jun 25, 2019

View reviewed changes

pitrou approved these changes Jun 25, 2019

View reviewed changes

pitrou force-pushed the 2104-Implement-take-kernel-functions-nested-a branch from a4bfaac to 73ab241 Compare June 26, 2019 11:56

bkietz force-pushed the 2104-Implement-take-kernel-functions-nested-a branch from 73ab241 to f16a047 Compare June 26, 2019 16:41

wesm force-pushed the 2104-Implement-take-kernel-functions-nested-a branch from 5e2c9eb to 06a6889 Compare June 27, 2019 04:34

bkietz added 10 commits June 27, 2019 11:13

refactor Take and Filter to share code through Taker<>

65dcd90

add tests for Take(nested types)

227ea55

lint fixes

6c453f3

clang-format, explicit cast

6a14c93

repair bounds checking

c7f2e40

rewrite python Take() test

c3e8129

simplify looping through IndexSequences

abc1733

add Take() permutation inversion test

d9c4a1a

remove redundant bounds checking in Struct case

e6081b0

add doccomments

5585483

wesm approved these changes Jun 27, 2019

View reviewed changes

wesm closed this in da752fd Jun 27, 2019

bkietz deleted the 2104-Implement-take-kernel-functions-nested-a branch February 25, 2021 16:39

This was referenced Jun 27, 2019

[C++] Implement take kernel functions - nested array value type #18077

Closed

[C++] Optimize Take implementation #22185

Closed

ARROW-2104: [C++] take kernel functions for nested types #4531

ARROW-2104: [C++] take kernel functions for nested types #4531

Conversation

bkietz commented Jun 12, 2019

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkietz commented Jun 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fsaintjacques Jun 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkietz commented Jun 26, 2019

pitrou commented Jun 26, 2019

bkietz commented Jun 26, 2019

pitrou commented Jun 26, 2019

bkietz commented Jun 26, 2019

bkietz commented Jun 26, 2019

wesm commented Jun 27, 2019

wesm commented Jun 27, 2019

bkietz commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

ursabot commented Jun 27, 2019

bkietz commented Jun 27, 2019 • edited Loading

bkietz commented Jun 27, 2019

wesm commented Jun 27, 2019

bkietz commented Jun 27, 2019

wesm commented Jun 27, 2019

fsaintjacques commented Jun 27, 2019

codecov-io commented Jun 27, 2019

Codecov Report

wesm left a comment

Choose a reason for hiding this comment

fsaintjacques Jun 26, 2019 •

edited

Loading

bkietz commented Jun 27, 2019 •

edited

Loading