Changelog History
Page 2
-
v0.10.0
October 24, 2017This release focuses on 2 main things: the internal restructuring to use scenarios and the new hooks system. Other than that we also have some nice convenience features and formatters can be generated in parallel now.
๐ Features (User Facing)
- Hooks system - basically you can now do something before/after a benchmarking scenario or the benchmarking function, too much to explain it in a Changelog, check the README
- Don't show more precision than we have - i.e. 234.00 microseconds (measurements are in microseconds and .00 doesn't gain you anything)
- Limit precision of available memory displayed, you don't need to know 7.45678932 GB. Thanks to
@elpikel
. - Display the 99th percentile runtime. Thanks to
@wasnotrice
. - ๐ง
:unit_scaling
is now a top level configuration option that can now also be used and picked up by formatters, like the HTML formatter - formatters can now be specified as a module (which should implement the
Benchee.Formatter
behaviour) - this makes specifying them nice and now at least theirformat/1
functions can be executed in parallel
๐ Bugfixes (User Facing)
- Determining CPUs was too strict/too assuming of a specific pattern breaking in certain environments (like Semaphore CI). That is more relaxed now thanks to
@predrag-rakic
! - Memory is now correctly converted using the binary (1024) interpreation, instead of the decimal one (1000)
๐ Features (Plugins)
- the statistics now also provide the mode of the samples as well as the 99th percentile
- There is a new
Benchee.Formatter
behaviour to adopt and enforce a uniform format for formatters, best to douse Benchee.Formatter
๐ Breakin Changes (Plugins)
- ๐
:run_times
,:statistics
and:jobs
have been removed and folded together into:scenarios
- a scenario holds the benchmarking function, porentially the input, the raw run times measures and the computed statistics. With this data structure, all the relevant data for one scenario is one place although it takes a lot to change, this seems to be the best way going forward. Huge thanks to@devonestes
!
-
v0.9.0
June 08, 2017This release focuses on adding more system specific information like CPU etc. and for better erlang compatibilityif you wanna use benchee from erlang. There is an example project but calling Elixir from Erlang hasn't been as easy as I hoped :)
๐ Features (User Facing)
- Gather more system data like number of cores, Operating System, memory, cpu speed - thanks @devonestes and @OvermindDL1
- ๐ท the names for jobs in the map of
Benchee.run/2
or inBenchee.benchmark/3
may now be given as strings or atoms - atoms will be converted to strings internally though for consistency and avoiding name duplicates - ๐ง the names of inputs in the benchee configuration may now be given as strings or atoms - atoms will be converted to strings internally though for consistency and avoiding name duplicates
- ๐
Benchee is now also available "Erlang Style" to be called on an atom like
:benchee.run(_, _)
for better Erlang compatibility
-
v0.8.0
May 07, 2017๐ Another smaller release that focuses on adding type specs and structs in appropriate places along with fixing a couple of small bugs.
๐ Features (User Facing)
- ๐ง Providing an unrecognized configuration option (say
runNtime
instead ofruntime
) will now raise an exception - ๐ง Durations in the configuration will now be scaled appropriately (minutes, microseconds etc)
- ๐ Major functions are type specced for your viewing pleasure in the docs and your dialyzer pleasure at type check time.
๐ Bugfixes (User Facing)
- ๐ In 0.7.0 statistics generation might time out if Millions of run times were captured so that it takes longer than 5 seconds, this is fixed by waiting infinitely - thanks @devonestes for the report.
- ๐ Unintended line break in the fast function warning removed
- All necessary dependencies added to
:applications
(deep_merge was missing)
๐ฅ Breaking Changes (User Facing)
- โฌ๏ธ Dropped support for elixir 1.2, new support is elixir 1.3+
- ๐ง
Benchee.Config
was renamed toBenchee.Configuration
(important when you use the more verbose API or used it in a Plugin)
๐ Features (Plugins)
- Major public interfacing functions are now typespecced!
- ๐ง A couple of major data structures are now proper structs e.g.
Benchee.Suite
,Benchee.Configuration
,Benchee.Statistics
๐ฅ Breaking Changes (Plugins)
- ๐ง The
config
key is nowconfiguration
to go along with the Configuration name change - ๐ As
Benchee.Configuration
is a proper struct now, arbitrary keys don't end up in it anymore. Custom data for plugins should be passed in throughformatter_options
orassigns
. Existing plugin keys (csv
,json
,html
andconsole
) are automatically put into theformatter_options
key space for now.
- ๐ง Providing an unrecognized configuration option (say
-
v0.7.0
April 23, 2017Smaller convenience features in here - the biggest part of work went into breaking reports in benchee_html apart :)
๐ Features (User Facing)
- ๐จ the print out of the Erlang version now is less verbose (just major/minor)
- โ the fast_warning will now also tell you how to disable it
- ๐จ When
print: [benchmarking: false]
is set, information about which input is being benchmarked at the moment also won't be printed - generation of statistics parallelized (thanks hh.ex - @nesQuick and @dszam)
๐ฅ Breaking Changes (User Facing)
- If you use the more verbose interface (
Benchee.init
and friends, e.g. notBenchee.run
) then you have to insert aBenchee.system
call beforeBenchee.measure
(preferably right afterBenchee.init
)
๐ Features (Plugins)
- ๐
Benchee.Utility.FileCreation.interleave/2
now also accepts a list of inputs which are then all interleaved in the file name appropriately. See the doctests for more details.
๐ฅ Breaking Changes (Plugins)
- ๐ง
Benchee.measure/1
now also needs to have the system information generated byBenchee.system/1
present if configuration information should be printed.
-
v0.6.0
November 30, 2016๐ One of the biggest releases yet. Great stuff in here - more elixir like API for
Benchee.run/2
with the jobs as the primary argument and the optional options as the second argument and now also as the more idiomatic keyword list!The biggest feature apart from that is the possibility to use multiple inputs - which you all should do now as quite many functions behave differently with bigger, smaller or differently shaped inputs. Apart from that a bulk of work has gone into making and supporting benchee_html!
๐ Features (User Facing)
- New
:inputs
configuration key that allows you to specify a map from input name to input value so that each defined benchmarking job is then executed with this input. For this to work the benchmarking function is called with the appropriateinput
as an argument. Seesamples/multiple_inputs.exs
for an example. #21 - ๐ The highlevel
Benchee.run/2
is now more idiomatic elixir and takes the map of jobs as the first argument and a keywordlist of options as the second (and last) argument. The old way of passing config as a map as the first argument and the jobs as the second argument still works, but might be deprecated later on #47 - Along with that
Benchee.init/1
now also accepts keyword lists of course
๐ฅ Breaking Changes (User Facing)
- ๐ The old way of providing the jobs as a list of tuples now removed, please switch to using a map from string to functions
๐ Features (Plugins)
Benchee.Utility.FileCreation
module to help with creating files from a map of multiple inputs (or other descriptors) mapping to input and aninterleave
function that spits out the correct file names especially if the:__no_input
marker is usedBenchee.System
is available to retrieve elixir and erlang versions but it's also already added to the suite duringBenchee.run/2
๐ฅ Breaking Changes (Plugins)
- ๐ท The structure of the output from
Benchee.Benchmark.measure/1
toBenchee.Statistics.statistics/1
has changed to accommodate the new inputs feature there is now an additional level where in a map the input name then points to the appropriate results of the jobs. When there were no inputs the key is the value returned byBenchee.Benchmark.no_input/0
.
๐ Bugfixes
- ๐ prewarming (discarding the first result due to some timer issues) during run time was removed, as it should already happen during the warmup period and would discard actual useful results especially for longer running macro benchmarks.
- ๐ท when the execution time of the benchmarking job exceeds the given
:time
it will now execute exactly once (used to be 2) #49 - ๐
run_times
are now in the order as recorded (used to be reverse) - important when wants to graph them/look at them to see if there are any anomalities during benchmarking - โ Remove elixir 1.4.0-rc.0 warnings
- New
-
v0.5.0
October 13, 2016๐ This release focuses on scaling units to more appropriate sizes. Instead of always working with base one for counts and microseconds those values are scaled accordingly to thousands, milliseconds for better readability. This work was mostly done by new contributor @wasnotrice.
๐ Features (User Facing)
- Console output now scales units to be more friendly. Examples:
- instead of "44556677" ips, you would see "44.56 M"
- instead of "44556.77 ฮผs" run time, you would see "44.56 ms"
- Console output for standard deviation omits the parentheses
- ๐ Scaling of console output can be configured with the 4 different strategies
:best
,:largest
,:smallest
and:none
. Refer to the documentation for their different properties. - โ Shortened the fast function warning and instead linked to the wiki
๐ Features (Plugins)
- The statistics module now computes the
minimum
,maximum
andsample_size
(not yet shown in the console formatter) - you can rely on
Benchee.Conversion.Duration
,Benchee.Conversion.Count
andBenchee.Conversion.DeviationPercent
for help with formatting and scaling units
๐ฅ Breaking Changes (Plugins)
- The
Benchee.Time
module is gone, if you relied on it for one reason or another it's succeeded by the more powerfulBenchee.Conversion.Duration
- Console output now scales units to be more friendly. Examples:
-
v0.4.0
September 11, 2016๐ง Focuses on making what benchee print out configurable to make it fit to your preferences :)
๐ Features (User Facing)
- ๐ง The configuration now has a
:print
key where it is possible to configure in a map what benchee prints out during benchmarking. All options are enabled by default (true). Options are::benchmarking
- print when Benchee starts benchmarking a new job (Benchmarking name ..):configuration
- a summary of configured benchmarking options including estimated total run time is printed before benchmarking starts:fast_warning
- warnings are displayed if functions are executed too fast leading to inaccurate measures
- ๐ง There is also a new configuration option for the built-in console formatter. Which is also enabled by default:
:comparison
- if the comparison of the different benchmarking jobs (x times slower than) is shown
- ๐ง The pre-benchmarking output of the configuration now also prints the currently used Erlang and Elixir versions (similar to
elixir -v
) - โ Add a space between the benchmarked time and the unit (microseconds)
- ๐ง The configuration now has a
-
v0.3.0
July 11, 2016๐ This release switches internal data structures from lists of tuples to maps, allows the configuration of formatters, aggregates all values and hands them down so formatters can access the whole configuration, prints general configuration information and much more great stuff :)
๐ฅ Breaking Changes (User Facing)
- The recommended data structure handed to
Benchee.run
was changed from a list of 2-element tuples to a map ("Name" => benchmark_function
). However, the old list of tuples still works but may be removed in future releases (so it's not "breaking" strictly speaking). - ๐ท You can not have benchmark jobs with the same names anymore, the last one wins here. This was the reason why previously the data structure was a list of tuples. However, having benchmarks with the same name is nonsensical as you can't discern their results in the output any way.
๐ฅ Breaking Changes (Plugins)
- ๐ท main data structure to hold benchmarks and results was changed from a list of 2-element tuples to a map (
"Name" => values
). That is for the jobs, the run times as well as the statistics. However, if you used something likeEnum.each(data, fn({name, value}) -> .. end)
you are still fine though, cause Elixir is awesome :)
๐ Features (User Facing)
- ๐ง now takes a
parallel: number
configuration option and will then execute each job in parallel in as many parallel processes as specified innumber
. This way you can gather more samples in the same time and also simulate a system more under load. This is tricky, however. One of the use cases is also stress testing a system. Thanks @ldr - the name column width is now determined based on the longest name. Thanks @alvinlindstam
- ๐ง Print general configuration information at the start of the benchmark, including warmup, time, parallel and an estimated total run time
- ๐ New method
Benchee.Formatters.Console.output/1
that immediately prints to the console - ๐ง now takes a
formatters: [&My.Format.function/1, &Benchee.Formatters.console.output/1]
configuration option with which multiple formatters for the same benchmarking run can be configured when usingBenchee.run/2
. E.g. you can print results to the console and create a csv from that same run. Defaults to the builtin Console formatter.
๐ Features (Plugins)
- All previous configuration options are preserved after
Benchee.Statistics.statistics/1
, meaning there is access to raw run times as well as custom options etc. E.g. you could grab custom options like%{csv: %{file: "my_file_name.csv"}}
to use.
๐ Bugfixes
- name columns are no longer truncated after 30 characters. Thanks @alvinlindstam
- The recommended data structure handed to
-
v0.2.0
June 11, 2016๐ This release introduces warmup for benchmarks, nicer console output and the new
Benchee.measure
that runs the benchmarks previously defined instead of running them instantly.๐ฅ Breaking Changes (User Facing)
- ๐ท
Benchee.benchmark/3
now doesn't run the benchmark anymore but simply adds it to:jobs
in the config. The whole benchmark suite is then run viaBenchee.measure/1
. This only affects you if you used the more verbose way of defining benchmarks,Benchee.run/2
still work as before.
๐ฅ Breaking Changes (Plugins)
- ๐ท the defined benchmarking are now preserved after running the benchmark under the
:jobs
key of the suite. Run times are added to the:run_times
key of the suite (important for alternative statistics implementations)
๐ Features (User Facing)
๐ง configuring a warmup time to run functions before measurements are taken can be configured via the
warmup
key in the config defaulting to 2 (seconds)๐ statistics in console output are aligned right now for better comparisons
๐ last blank line of console output removed
๐ Features (Plugins)
- additionally supply the total standard deviation of iterations per second as
std_dev_ips
afterBenchee.Statistics.statistics
๐ Bugfixes
- if no time/warmup is specified the function won't be called at all
- ๐ท
-
v0.1.0
June 05, 2016๐ Initial release