improve random string performance

[lstrzebinczyk]

I'm am running a worker which is trying to move over a fair bit of data from one location to a second one. This consists of a large number of small files. Tesla is somewhere in the dependencies, and I discovered that more than 50% of my cpu time is right now being spent generating random numbers for multipart. I created faster version to do more-or-less the same, and as for this script:

defmodule PerformanceTest do
  @boundary_chars "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
                  |> String.split("")

  def unique_string(length) do
    Enum.reduce(1..length, [], fn _i, acc ->
      [Enum.random(@boundary_chars) | acc]
    end)
    |> Enum.join("")
  end

  def fast_unique_string do
    16
    |> :crypto.strong_rand_bytes()
    |> Base.encode16(case: :lower)
  end
end

defmodule ProfilingExamples do
  import ExProf.Macro

  def run_slow do
    profile do
      for i <- 0..200 do
        PerformanceTest.unique_string(32)
      end
    end
  end

  def run_fast do
    profile do
      for i <- 0..200 do
        PerformanceTest.fast_unique_string()
      end
    end
  end
end

ProfilingExamples.run_slow()
ProfilingExamples.run_fast()

The current version takes 872331ms to finish, while new version I'm submiting takes 1819ms, so is ~479 times faster.

If it is an issue that new version generates random strings from a smaller number of characters, I can adjust, but it looks good enough, if I might say so.

Please note that this is not an academic exercise, I am running into a serious performance bottleneck with this specific method.

EDIT:

I don't quite understand why this test is failing, I'm happy to collaborate on it. All tests pass locally for me.

[lstrzebinczyk]

Take a look at this:

FUNCTION                                              CALLS        %  TIME  [uS / CALLS]
--------                                              -----  -------  ----  [----------]
lists:reverse/1                                           1     0.00     0  [      0.00]
'Elixir.Enum':join/2                                      2     0.00     0  [      0.00]
rand:seed/1                                               1     0.00     0  [      0.00]
rand:seed_s/2                                             1     0.00     0  [      0.00]
rand:'-mk_alg/1-fun-10-'/1                                1     0.00     0  [      0.00]
'Elixir.ProfilingExamples':'-run_slow/0-fun-0-'/2         2     0.00     0  [      0.00]
erlang:send/2                                             1     0.00     0  [      0.00]
erlang:unique_integer/0                                   1     0.00     0  [      0.00]
'Elixir.PerformanceTest':unique_string/1                  2     0.01     1  [      0.50]
'Elixir.Range':new/2                                      2     0.01     1  [      0.50]
rand:seed_s/1                                             1     0.01     1  [      1.00]
rand:seed58/1                                             3     0.01     1  [      0.33]
erlang:system_time/0                                      1     0.01     1  [      1.00]
'Elixir.Enum':reduce/3                                    3     0.02     2  [      0.67]
rand:mk_alg/1                                             1     0.02     2  [      2.00]
erlang:phash2/1                                           1     0.02     2  [      2.00]
rand:exsss_seed/1                                         1     0.03     3  [      3.00]
erlang:iolist_to_binary/1                                 2     0.04     4  [      2.00]
'Elixir.ProfilingExamples':'-run_slow/0-fun-1-'/0         1     0.05     5  [      5.00]
rand:splitmix64_next/1                                    3     0.07     7  [      2.33]
'Elixir.Enum':take_list/2                                64     0.09     9  [      0.14]
'Elixir.Enum':'-join/2-lists^foldl/2-0-'/3               66     0.09     9  [      0.14]
'Elixir.Enum':take/2                                     64     0.10    10  [      0.16]
'Elixir.Enum':reduce_range_inc/4                         66     0.12    12  [      0.18]
erlang:tuple_to_list/1                                   64     0.12    12  [      0.19]
erlang:make_tuple/2                                      64     0.13    13  [      0.20]
'Elixir.Enum':'-join/2-fun-0-'/3                         64     0.17    17  [      0.27]
erlang:min/2                                             64     0.17    17  [      0.27]
'Elixir.PerformanceTest':'-unique_string/1-fun-0-'/2     64     0.18    18  [      0.28]
'Elixir.Enum':random/1                                   64     0.18    18  [      0.28]
'Elixir.Enum':take_random/2                              64     0.19    19  [      0.30]
erlang:setelement/3                                     370     0.64    62  [      0.17]
rand:'-mk_alg/1-fun-13-'/2                             4032     5.39   526  [      0.13]
erlang:put/2                                           4033     5.86   572  [      0.14]
'Elixir.Enum':'-take_random/2-lists^foldl/2-1-'/3      4160     6.54   638  [      0.15]
rand:seed_get/0                                        4032    10.59  1033  [      0.26]
rand:uniform/1                                         4032    10.98  1071  [      0.27]
rand:exsss_uniform/2                                   4032    11.11  1084  [      0.27]
rand:seed_put/1                                        4033    11.17  1090  [      0.27]
'Elixir.Enum':'-take_random/2-fun-1-'/3                4096    11.83  1154  [      0.28]
'Elixir.Enum':random_integer/2                         4096    11.94  1165  [      0.28]
rand:uniform_s/2                                       4032    12.05  1175  [      0.29]
----------------------------------------------------  -----  -------  ----  [----------]
Total:                                                41687  100.00%  9754  [      0.23]

Just calling the unique_string once calls random_integer 4000 times!

[teamon]

Whoaaa, nice find!

I've reviewed the PR code and it looks good.

The only tests failing are the 1.0.0 compatibility ones which seems to be caused by some changes to github actions - definitely not a blocker.

I'll merge it now.

Thanks a lot 🙇

[lstrzebinczyk]

Thank you very much! Can you publish a new version with this?

[teamon]

Released https://github.com/teamon/tesla/releases/tag/v1.3.1