benchee v0.6.0 Release Notes

Release Date: 2016-11-30 // over 7 years ago
  • ๐Ÿš€ One of the biggest releases yet. Great stuff in here - more elixir like API for Benchee.run/2 with the jobs as the primary argument and the optional options as the second argument and now also as the more idiomatic keyword list!

    The biggest feature apart from that is the possibility to use multiple inputs - which you all should do now as quite many functions behave differently with bigger, smaller or differently shaped inputs. Apart from that a bulk of work has gone into making and supporting benchee_html!

    ๐Ÿ”‹ Features (User Facing)

    • New :inputs configuration key that allows you to specify a map from input name to input value so that each defined benchmarking job is then executed with this input. For this to work the benchmarking function is called with the appropriate input as an argument. See samples/multiple_inputs.exs for an example. #21
    • ๐Ÿ—„ The highlevel Benchee.run/2 is now more idiomatic elixir and takes the map of jobs as the first argument and a keyword list of options as the second (and last) argument. The old way of passing config as a map as the first argument and the jobs as the second argument still works, but might be deprecated later on #47
    • Along with that Benchee.init/1 now also accepts keyword lists of course

    ๐Ÿ’ฅ Breaking Changes (User Facing)

    • ๐Ÿšš The old way of providing the jobs as a list of tuples now removed, please switch to using a map from string to functions

    ๐Ÿ”‹ Features (Plugins)

    • Benchee.Utility.FileCreation module to help with creating files from a map of multiple inputs (or other descriptors) mapping to input and an interleave function that spits out the correct file names especially if the :__no_input marker is used
    • Benchee.System is available to retrieve Elixir and Erlang versions but it's also already added to the suite during Benchee.run/2

    ๐Ÿ’ฅ Breaking Changes (Plugins)

    • ๐Ÿ‘ท The structure of the output from Benchee.Benchmark.measure/1 to Benchee.Statistics.statistics/1 has changed to accommodate the new inputs feature there is now an additional level where in a map the input name then points to the appropriate results of the jobs. When there were no inputs the key is the value returned by Benchee.Benchmark.no_input/0.

    ๐Ÿ›  Bugfixes

    • ๐Ÿšš prewarming (discarding the first result due to some timer issues) during run time was removed, as it should already happen during the warmup period and would discard actual useful results especially for longer running macro benchmarks.
    • ๐Ÿ‘ท when the execution time of the benchmarking job exceeds the given :time it will now execute exactly once (used to be 2) #49
    • ๐Ÿ‘€ run_times are now in the order as recorded (used to be reverse) - important when wants to graph them/look at them to see if there are any anomalies during benchmarking
    • โœ‚ Remove elixir 1.4.0-rc.0 warnings