Changelog History
Page 2
-
v2.7.2 Changes
June 10, 2021๐ Fixed
[Oban.Plugins.Pruner] Consider
cancelled_at
ordiscarded_at
timestamps when querying prunable jobs. The previous query required anattempted_at
value, even forcancelled
ordiscarded
jobs. If a job was cancelled before it was attempted then it wouldn't ever be pruned.๐ [Oban.Plugins.Gossip] Correct exit handling during safe checks. Occasionally, producer checks time out and the previous
catch
block didn't handle exits properly.
-
v2.7.1 Changes
May 26, 2021๐ Fixed
- โช [Oban.Telemetry] Restore the
:job
key to[:oban, :job, :start]
events. It was mistakenly removed during a refactoring.
- โช [Oban.Telemetry] Restore the
-
v2.7.0 Changes
May 25, 2021๐ Pluggable Notifier
The PubSub functionality Oban relies on for starting, stopping, scaling, and pausing queues is now pluggable. The previous notifier was hard-coded to use Postgres for PubSub via
LISTEN/NOTIFY
. Unfortunately,LISTEN/NOTIFY
doesn't work for PG Bouncer in "Transaction Mode." While relatively few users run in "Transaction Mode," it is increasingly common and deserved a work-around.0๏ธโฃ
Oban.Notifier
defines a minimal behaviour and Oban ships with a default implementation,Oban.PostgresNotifier
, based on the oldLISTEN/NOTIFY
๐ง module. You can specify a different notifier in your Oban configuration:config :my_app, Oban, notifier: MyApp.CustomNotifier, repo: MyApp.Repo, ...
An alternate [
pg/pg2
based notifier][pgn] is available in Oban Pro.Replacing Opts on Unique Conflict
โก๏ธ The
replace_args
option for updating args on unique conflict is replaced with โก๏ธ a highly flexiblereplace
option. Usingreplace
, you can update any job field when there is a conflict. Replace works for any of the following fields: โฑargs
,max_attempts
,meta
,priority
,queue
,scheduled_at
,tags
, ๐ทworker
.For example, to change the
priority
to 0 and increasemax_attempts
to 5 when there is a conflict:BusinessWorker.new( args, max_attempts: 5, priority: 0, replace: [:max_attempts, :priority] )
โฑ Another example is bumping the scheduled time to 1 second in the future on โฑ conflict. Either
scheduled_at
orschedule_in
values will work, but the โฑ replace option is alwaysscheduled_at
.UrgentWorker.new(args, schedule_in: 1, replace: [:scheduled_at])
โ Added
๐ท [Oban.Job] A new
conflict?
field indicates whether a unique job conflicted with an existing job on insert.[Oban.Telemetry] Always populate the
unsaved_error
field forerror
ordiscard
events. Previously, the field was empty fordiscard
events.๐ท [Oban.Queue.Engine] Define a
cancel_job/2
callback in the engine and move cancellation from the query module to theBasicEngine
.
๐ Changed
๐จ [Oban.Testing] Thanks to a slight refactoring, the
perform_job/3
helper now emits the same telemetry events that you'd have in production. The refactoring also exposed and fixed an inconsistency around invalid snooze handlers.โ [Oban.Testing] Expose
Testing.all_enqueued/0
, an alternate clause that doesn't require any options. This new clause makes it possible to check for any enqueued jobs without filters.๐ [Oban.Plugins.Stager] Limit the number of jobs staged at one time to prevent timeout errors while staging a massive number of jobs.
๐ Fixed
- [Oban.Repo] Respect ongoing transactions when using
get_dynamic_repo
. Prior to this fix, calls that useOban.Repo
, such asOban.insert/2
, could use a different repo than the one used in the current transaction.
-
v2.6.1 Changes
April 02, 2021๐ Fixes
- [Oban.Drainer] Always use the
BasicEngine
for draining, regardless of the currently configured engine.
- [Oban.Drainer] Always use the
-
v2.6.0 Changes
April 02, 2021โฌ๏ธ ๐ Web and Pro users should check the [v2.6 upgrade guide][v26ug] for a complete walkthrough.
๐ Pluggable Queue Engines
Queue producers now use an "engine" callback module to manage demand and work โจ with the database. The engine provides a clean way to expand and enhance the functionality of queue producers without modifying the solid foundation of queue supervision trees. Engines make enhanced functionality such as global concurrency, local rate limiting and global rate limiting possible!
The
BasicEngine
is the default, and it retains the exact functionality of previous Oban versions. To specify the engine explicitly, or swap in an ๐ง alternate engine, set it in your configuration:config :my_app, Oban, engine: Oban.Queue.BasicEngine, ...
โฌ๏ธ See the [v2.6 upgrade guide][v26ug] for instructions on swapping in an alternate engine.
Recursive Queue Draining
The ever-handy
Oban.drain_queue/1,2
function gained a newwith_recursion
๐ท option, which makes it possible to test jobs that insert more jobs when they execute. Whenwith_recursion
is enabled the queue will keep executing until no โฑ jobs are available. It even composes withwith_scheduled
!โ In practice, this is especially useful for testing dependent workflows.
๐ Gossip Plugin for Queue Monitoring
๐ The new gossip plugin uses PubSub to efficiently exchange queue state information between all interested nodes. This allows Oban instances to broadcast state information regardless of which engine they are using, and without storing anything in the database.
โฌ๏ธ See the [v2.6 upgrade guide][v26ug] for details on switching to the gossip ๐ plugin.
โ Added
๐ท [Oban.Job] Inject the current conf into the jobs struct as virtual field, making the complete conf available within
perform/1
.[Oban.Notifier] Add
unlisten/2
, used to stop a process from receiving notifications for one or more channels.
๐ Changed
[Oban.Telemetry] Stop emitting circuit events for queue producers. As producers no longer poll, the circuit breaker masked real errors more than it guarded against sporatic issues.
[Oban.Telemetry] Delay
[:oban, :supervisor, :init]
event until the complete supervision tree finishes initialization.[Oban.Migration] Stop creating an
oban_beats
table entirely.
๐ Fixed
- โฑ [Oban.Plugins.Cron] Prevent schedule overflow right before midnight
-
v2.5.0 Changes
February 26, 2021Top of the Minute Cron
โฑ Rather than scheduling cron evaluation 60 seconds after the server starts, โฑ evaluation is now scheduled at the top of the next minute. This yields several ๐ improvements:
- ๐ท More predictable timing for cron jobs now that they are inserted at the top of the minute. Note that jobs may execute later depending on how busy queues are.
- ๐ท Minimize contention between servers inserting jobs, thanks to the transaction lock acquired by each plugin.
- ๐ท Prevent duplicate inserts for jobs that omit the
completed
state (when server start time is staggered the transaction lock has no effect)
๐ Repeater Plugin for Transaction Pooled Databases
Environments that can't make use of PG notifications, i.e. because they use ๐ท PgBouncer with transaction pooling, won't process available jobs reliably. The ๐ new
Repeater
plugin provides a workaround that simulates polling functionality for producers.๐ Simply include the
Repeater
in your plugin list:config :my_app, Oban, plugins: [ Oban.Plugins.Pruner, Oban.Plugins.Stager, Oban.Plugins.Repeater ], ...
๐ Include Perform Result in Job Telemetry Metadata
๐ The metadata for
[:oban, :job, :stop]
telemetry events now include the job's ๐ทperform/1
return value. That makes it possible to extract job output from other processes:def handle_event([:oban, :job, :stop], _timing, meta, _opts) do IO.inspect(meta.result, label: "return from #{meta.job.worker}") :ok end
โ Added
๐ [Oban] Support a changeset function in addition to a changeset in
insert_all/4
. Inserting jobs within a multi is even more powerful.๐ท [Oban.Queue.Executor] Stash a timed job's pid to enable inner process messaging, notably via telemetry events.
๐ Changed
โฌ๏ธ [Oban] Upgrade minimum Elixir version from 1.8 to 1.9.
๐ [Oban.Plugins.Cron] Individually validate crontab workers and options, providing more descriptive errors at the collection and entry level.
๐ท [Oban] Simplify the job cancelling flow for consistency. Due to race conditions it was always possible for a job to complete before it was cancelled, now that flow is much simpler.
[Oban.Queue.Producer] Replace dispatch cooldown with a simpler debounce mechanism.
๐ [Oban.Plugins.Stager] Limit the number of notify events to one per queue, rather than one per available job.
๐ Fixed
[Oban.Queue.Producer] Handle unclean exits without an error and stack, which prevents "zombie" jobs. This would occur after multiple hot upgrades when the worker module changed, as the VM would forcibly terminate any processes running the old modules.
[Oban] Specify that the
changeset_wrapper
type allows keys other than:changesets
, fixing a dialyzer type mismatch.
โ Removed
- ๐ [Oban] Remove the undocumented
version/0
function in favor of using the standardApplication.spec/2
-
v2.4.3 Changes
February 07, 2021๐ Fixed
- ๐ท [Oban.Telemetry] Use
conf
rather thanconfig
in meta for:job
and:circuit
telemetry events.
- ๐ท [Oban.Telemetry] Use
-
v2.4.2 Changes
January 28, 2021๐ Fixed
- ๐ [Oban.Plugins.Stager] Notify queues of all available jobs, not only jobs that were most recently staged. In some circumstances, such as reboot or retries, jobs are available without previously being scheduled.
-
v2.4.1 Changes
January 27, 2021๐ Fixed
[Oban.Migrations] Correctly migrate up between sequential versions. The V10 migration adds check constraints, which can't be re-run safely. An attempted work-around prevented re-runs, but broke sequential upgrades like 9->10.
[Oban.Queue.Producer] Kick off an initial dispatch to start processing available jobs on startup, to compensate for a lack of incoming jobs or scheduling.
โฑ [Oban.Plugins.Stager] Keep rescheduling staging messages after the initial run.
-
v2.4.0 Changes
January 26, 2021๐ Centralized Stager Plugin
โฑ Queue producers no longer poll every second to stage scheduled jobs. Instead, ๐ the new
Oban.Plugins.Stager
plugin efficiently handles staging from a single location. This reduces overall load on the BEAM and PostgreSQL, allowing apps to easily run hundreds of queues simultaneously with little additional overhead.In a test with a single Ecto connection and 512 queues the CPU load went from 60.0% BEAM / 21.5% PostgreSQL in v2.3.4 to 0.5% BEAM / 0.0% PostgreSQL.
๐ The stager plugin is automatically injected into Oban instances and there isn't ๐ง any additional configuration necessary. However, if you've set a
poll_interval
๐ for Oban or an individual queue you can safely remove it.Overhauled Cron
The CRON parser is entirely rewritten to be simpler and dramatically smaller. ๐ The previous parser was built on
nimble_parsec
and while it was fast, the ๐ compiled parser added ~5,500LOC to the code base. Thanks to a suite of property โ tests we can be confident that the new parser behaves identically to the ๐ previous one, and has much clearer error messages when parsing fails.๐ Along with a new parser the
crontab
functionality is extracted into the ๐Oban.Plugins.Cron
plugin. For backward compatibility, top-levelcrontab
and ๐timezone
options are transformed into a plugin declaration. If you'd like to ๐ง start configuring the plugin directly change your config from:config :my_app, Oban, crontab: [{"* * * * *", MyApp.Worker}], timezone: "America/Chicago" ...
To this:
config :my_app, Oban, plugins: [ {Oban.Plugins.Cron, crontab: [{"* * * * *", MyApp.Worker}], timezone: "America/Chicago"} ]
๐ The plugin brings a cleaner implementation, simplified supervision tree, and ๐ง eliminates the need to set
crontab: false
in test configuration.๐ Improved Indexes for Unique Jobs
๐ท Applications may have thousands to millions of completed jobs in storage. As the ๐ table grows the performance of inserting unique jobs can slow drastically. A new V10 migration adds necessary indexes, and paired with improved query logic it alleviates insert slowdown entirely.
๐ท For comparison, a local benchmark showed that in v2.3.4 inserting a unique job ๐ท into a table of 1 million jobs runs around 4.81 IPS. In v2.4.0 it runs at 925.34 IPS, nearly 200x faster.
The V10 migration is optional. If you decide to migrate, first create a new migration:
mix ecto.gen.migration upgrade_oban_jobs_to_v10
Next, call
Oban.Migrations
in the generated migration:defmodule MyApp.Repo.Migrations.UpdateObanJobsToV10 do use Ecto.Migration def up do Oban.Migrations.up(version: 10) end def down do Oban.Migrations.down(version: 10) end end
โ Added
[Oban] Add
[:oban, :supervisor, :init]
event emitted when an Oban supervisor starts.๐ [Oban.Telemetry] Wrap built-in plugins with telemetry spans and consistently include
conf
in all telemetry events.[Oban.Config] Improve the error messages raised during initial validation. Also, the
Config
module is now public with light documentation.โ [Oban.Testing] Support specifying a default prefix for all test assertions. This comes with improved assertion messages that now include the prefix.
[Oban.Repo] Add
delete/3
as a convenient wrapper aroundc:Ecto.Repo.delete/2
.๐ท [Oban.Job] New check constraints prevent inserting jobs with an invalid
priority
, negativemax_attempts
or anattempt
number beyond the maximum.
๐ Changed
- ๐ [Oban.Telemetry] Deprecate and replace
span/3
usage in favor of the official:telemetry.span/3
, which wasn't available whenspan/3
was implemented.
๐ Fixed
- [Oban] Inspect the error reason for a failed
insert!/2
call before it is raised as an error. Wheninsert!/2
was called in a transaction the error could be:rollback
, which wasn't a valid error.