Popularity
5.1
Growing
Activity
6.8
Growing
31
4
13

Monthly Downloads: 865
Programming language: Elixir
License: MIT License
Latest version: v0.18.0

pillar alternatives and similar packages

Based on the "ORM and Datamapping" category.
Alternatively, view pillar alternatives based on common mentions on social networks and blogs.

Do you think we are missing an alternative of pillar or a related project?

Add another 'ORM and Datamapping' Package

README

Pillar

github.com hex.pm hex.pm hex.pm hex.pm github.com

Elixir client for ClickHouse, a fast open-source Online Analytical Processing (OLAP) database management system.

Features

Usage

Direct Usage with connection structure

conn = Pillar.Connection.new("http://user:[email protected]:8123/database")

# Params are passed in brackets {} in SQL query, and map strtucture does fill
# query by values.
sql = "SELECT count(*) FROM users WHERE lastname = {lastname}"

params = %{lastname: "Smith"}

{:ok, result} = Pillar.query(conn, sql, params)

result
#=> [%{"count(*)" => 347}]

Pool of workers

Recommended usage, because of limited connections and supervised workers.

defmodule ClickhouseMaster do
  use Pillar,
    connection_strings: [
      "http://user:[email protected]:8123/database",
      "http://user:[email protected]:8123/database"
    ],
    name: __MODULE__,
    pool_size: 15
end

ClickhouseMaster.start_link()

{:ok, result} = ClickhouseMaster.select(sql, %{param: value})

Async insert

connection = Pillar.Connection.new("http://user:[email protected]:8123/database")

Pillar.async_insert(connection, "INSERT INTO events (user_id, event) SELECT {user_id}, {event}", %{
  user_id: user.id,
  event: "password_changed"
}) # => :ok

Buffer for periodical bulk inserts

For this feature required Pool of workers.

defmodule BulkToLogs do
  use Pillar.BulkInsertBuffer,
    pool: ClickhouseMaster,
    table_name: "logs",
    # interval_between_inserts_in_seconds, by default -> 5
    interval_between_inserts_in_seconds: 5,
    # on_errors is optional
    on_errors: &__MODULE__.dump_to_file/2

  @doc """
  dump to file function store failed inserts into file 
  """
  def dump_to_file(_result, records) do
    File.write("bad_inserts/#{DateTime.utc_now()}", inspect(records))
  end

  @doc """
  retry insert is dangerous (but it is possible and listed as proof of concept)

  this function may be used in `on_errors` option
  """
  def retry_insert(_result, records) do
    __MODULE__.insert(records)
  end
end
:ok = BulkToLogs.insert(%{value: "online", count: 133, datetime: DateTime.utc_now()})
:ok = BulkToLogs.insert(%{value: "online", count: 134, datetime: DateTime.utc_now()})
:ok = BulkToLogs.insert(%{value: "online", count: 132, datetime: DateTime.utc_now()})
....

# All this records will be inserted with 5 second interval.

on_errors parameter allows you to catch any error of bulk insert (for example: one of batch is bad or clickhouse was not available )

Migrations

Migrations can be generated with mix task mix pillar.gen.migration migration_name.

mix pillar.gen.migration events_table

But for launching them we have to write own task, like this:

defmodule Mix.Tasks.MigrateClickhouse do
  use Mix.Task
  def run(_args) do
    connection_string = Application.get_env(:my_project, :clickhouse_url)
    conn = Pillar.Connection.new(connection_string)
    Pillar.Migrations.migrate(conn)
  end
end

And launch this via command.

mix migrate_clickhouse

Contribution

Feel free to make a pull request. All contributions are appreciated!