Etcher

1.3.0

Etcher allows you to take raw settings and/or user input and etch them into a concrete and valid configuration for use within your application. As quoted from Wikipedia, to etch is to:

[Use] strong acid or mordant to cut into the unprotected parts of a metal surface to create a design in intaglio (incised) in the metal.

By using Etcher, you have a reliable way to load default configurations (i.e. Environment, JSON, YAML) which can be validated and etched into frozen records (i.e. Hash, Data, Struct) for consumption within your application which doesn’t violate the Law of Demeter. This comes complete with transformations and validations all via a simple Object API. Pairs well with the XDG, Runcom, and Sod gems.

Table of Contents

Features

Supports contracts which respond to #call to validate a Hash before building the final record. Pairs well with the Dry Schema and Dry Validation gems.
Supports models which respond to .[] for consuming a splatted Hash to instantiate new records. Pairs well with primitives such as: Hash, Data, and Struct.
Supports loading of default configurations from the Environment, a JSON configuration, a YAML configuration, or anything that can answer a hash.
Supports multiple transformations which can process loaded configuration hashes and answer a transformed hash.
Supports Hash overrides as a final customization which is handy for Command Line Interfaces (CLIs), as aided by Sod, or anything that might require user input at runtime.

Requirements

Ruby.

Setup

To install with security, run:

# 💡 Skip this line if you already have the public certificate installed.
gem cert --add <(curl --compressed --location https://alchemists.io/gems.pem)
gem install etcher --trust-policy HighSecurity

To install without security, run:

gem install etcher

You can also add the gem directly to your project:

bundle add etcher

Once the gem is installed, you only need to require it:

require "etcher"

Usage

Basic usage is to new up an instance:

etcher = Etcher.new
etcher.call({one: 1, two: 2})

# Success({:one=>1, :two=>2})

Notice you get a monad — either a Success or Failure — as provided by the Dry Monads gem. This allows you to create more sophisticated pipelines as found with the Pipeable gem or any kind of failsafe workflow you might need.

By default, any attributes you message the instance with will only pass through what you gave it and always answer a Success. This is nice for initial experimentation but true power comes with full customization of the instance. Here’s an advanced configuration showing all features:

require "dry/monads"
require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:user).filled :string
  required(:home).filled :string
end

model = Data.define :user, :home

transformer = lambda do |content, key = :user|
  Dry::Monads::Success content.merge! key => content[key].upcase
end

Etcher::Registry.new(contract:, model:, transformers: [transformer])
                .add_loader(Etcher::Loaders::Environment.new(%w[USER HOME]))
                .then { |registry| Etcher.new(registry).call }

# Success(#<data user="DEMO", home="/Users/demo">)

The above can be broken down into a series of steps:

A Dry Schema contract — loaded with Dry Monads extensions — is created to verify untrusted attributes.
A model is created with attributes: user and home.
A registry is constructed with a custom contract, model, loader, and transformer.
Finally, we see a successfully built configuration for further use.

While this is a more advanced use case, you’ll usually only need to register a contract and model. The loaders and transformers provide additional firepower in situations where you need to do more with your data. We’ll look at each of these components in greater detail next.

ℹ️ All keys are converted to symbols before being processed. This is done to ensure consistency and improve debugablity when dealing with raw input that might be a mix of strings and/or symbols.

Steps

As hinted at above, the complete sequence of steps are performed in the order listed:

Load: Each loader, if any, is called and merged with the previous loader to build initial content.
Override: Any overrides are merged with the result of the last loader to produce updated content. In Version 2.0.0, this step happen after the Transform step.
Transform: Each transformer, if any, is called to transform and manipulate the content.
Validate: The contract is called to validate the content as previously loaded, overwritten, and transformed.
Model: The model consumes the content of the validated contract and creates a new record for you to use as needed.

You can use the above steps as a reference when using this gem. Each step is explained in greater detail below.

Registry

The registry is provided as a way to register any/all complexity for before creating a new Etcher instance. Here’s what you get by default:

Etcher::Registry.new
# #<data Etcher::Registry contract=#<Proc:0x000000010e393550 contract.rb:7 (lambda)>, model=Hash, loaders=[], transformers=[]>

Since the registry is a Data, you can initialize with everything you need:

Etcher::Registry[
  contract: MyContract,
  model: MyModel,
  loaders: [MyLoader.new],
  transformers: [MyTransformer]
]

You can also add additional loaders and/or transformers after the fact:

registry = Etcher::Registry.new
                           .add_loader(MyLoader.new)
                           .add_transformer(MyTransformer)

💡 Order matters so ensure you list your loaders and transformers in the order you want them processed.

Contracts

Contracts are critical piece of this workflow as they provide a way to validate incoming data, remove unwanted data, and create a sanitized record for use in your application. Any contract that has the following behavior will work:

#call: Must be able to consume a Hash and answer an object which can respond to #to_monad.

The best gems which adhere to this interface are: Dry Schema and Dry Validation. You’ll also want to make sure the Dry Monads extensions are loaded as briefly shown earlier so the result will respond to #to_monad. Here’s how to enable monad support if using both gems:

Dry::Schema.load_extensions :monads
Dry::Validation.load_extensions :monads

Using Dry Schema syntax, we could create a contract for verifying email addresses and use it to build a new Etcher instance. Example:

require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:from).filled :string
  required(:to).filled :string
end

etcher = Etcher::Registry[contract:].then { |registry| Etcher.new registry }
etcher.call

# Failure({:step=>:validate, :payload=>{:from=>["is missing"], :to=>["is missing"]}})

etcher.call from: "Mork", to: "Mindy"
# Success({:from=>"Mork", :to=>"Mindy"})

Here you can see the power of using a contract to validate your data both as a failure and a success. Unfortunately, with the success, we only get a Hash as a record and it would be nice to structured model which which we’ll look at next.

Types

To support contracts further, there are a couple custom types which might be of interest. Each custom type, as described below, is made possible via Dry Types.

Pathnames

Etcher::Types::Pathname

The above allows you to use pathname types in your contracts to validate and cast as pathnames:

contract = Dry::Schema.Params do
  required(:path).filled Etcher::Types::Pathname
end

contract.call(path: "a/path").to_monad
# Success(#<Dry::Schema::Result{:path=>#<Pathname:a/path>} errors={} path=[]>)

Versions

Etcher::Types::Version

The above allows you to validate and cast versions within your contracts — via the Versionaire gem — as follows:

contract = Dry::Schema.Params do
  required(:version).filled Etcher::Types::Version
end

contract.call(version: "1.2.3").to_monad
# Success(#<Dry::Schema::Result{:version=>"1.2.3"} errors={} path=[]>)

Models

A model is any object which responds to .[] and can accept a splatted hash. Example: Model[**attributes]. These primitives are excellent choices: Hash, Data, and Struct.

ℹ️ Keep in mind that using a Hash is the default model and will only result in a pass through situation. You’ll want to reach for the more robust Data or Struct objects instead.

The model is used in the last step of the etching process to create a frozen record for further use by your application. Here’s an example where a Data model is used:

model = Data.define :from, :to
etcher = Etcher::Registry[model:].then { |registry| Etcher.new registry }

etcher.call
# Failure({:step=>:record, :payload=>"Missing keywords: :from, :to."})

etcher.call from: "Mork", to: "Mindy"
# Success(#<data Model from="Mork", to="Mindy">)

Notice we get an failure if all attributes are not provided but if we supply the required attributes we get a success.

ℹ️ Keep in mind the default contract is always a pass through so no validation is being done when only using a Hash. Generally you want to supply both a custom contract and model at a minimum.

Loaders

Loaders are a great way to load a default configuration for your application which can be in multiple formats. Loaders can either be defined when creating a new registry instance or added after the fact. Here are a few examples:

# Initializer
registry = Etcher::Registry[loaders: [MyLoader.new]]

# Method
registry = Etcher::Registry.new.add_loader MyLoader.new

There are a few guidelines to using them:

They must respond to #call with no arguments.
All keys are symbolized which helps streamline merging and overriding values from the same keys across multiple configurations.
All nested keys will be flattened after being loaded. This means a key structure of {demo: {one: "test"}} will be flattened to demo_one: "test" which adheres to the Law of Demeter when a new recored is etched for you.
The order in which you define your loaders matters. This means the first loader defined will be processed first, then the second, and so forth. Loaders defined last take precedence over previously defined loaders when overriding the same keys.

The next couple of sections will help you learn about the supported loaders and how to build your own custom loader.

Environment

Use Etcher::Loaders::Environment to load configuration information from your Environment. By default, this object wraps ENV, uses an empty array for keys to include, and answers a filtered hash where all keys are downcased. If you don’t specify keys to include, then an empty hash is answered back. Here’s a few examples:

# Default behavior.
loader = Etcher::Loaders::Environment.new
loader.call
# Success({})

# With specific includes.
loader = Etcher::Loaders::Environment.new %w[RACK_ENV DATABASE_URL]
loader.call
# Success({"rack_env" => "test", "database_url" => "postgres://localhost/demo_test"})

# With a custom environment and specific include.
loader = Etcher::Loaders::Environment.new "USER", source: {"USER" => "Jack"}
loader.call
# Success({"user"=>"Jack"})

This loader is great for pulling from environment variables as a fallback configuration for your application.

JSON

Use Etcher::Loaders::JSON to load configuration information from a JSON file. Here’s how to use this loader (using a file that doesn’t exist):

# Default behavior (a custom path is required).
loader = Etcher::Loaders::JSON.new "your/path/to/configuration.json"
loader.call  # Success({})

You can also customize the fallback and logger used. Here are the defaults:

loader = Etcher::Loaders::JSON.new "your/path/to/configuration.json",
                                   fallback: {},
                                   logger: Logger.new(STDOUT)
loader.call  # Success({})

If the file did exist and had content, you’d get a Success with a Hash of the contents.

ℹ️ The logger is only used to log debug information when issues are encountered when reading from the file. This is done to reduce noise in your console when a configuration might have issues and can safely revert to the fallback in order to load the rest of the configuration.

YAML

Use Etcher::Loaders::YAML to load configuration information from a YAML file. Here’s how to use this loader (using a file that doesn’t exist):

# Default behavior (a custom path is required).
loader = Etcher::Loaders::YAML.new "your/path/to/configuration.yml"
loader.call  # Success({})

You can also customize the fallback and logger used. Here are the defaults:

loader = Etcher::Loaders::YAML.new "your/path/to/configuration.yml",
                                   fallback: {},
                                   logger: Logger.new(STDOUT)
loader.call  # Success({})

If the file did exist and had content, you’d get a Success with a Hash of the contents.

Custom

You can always create your own loader if you don’t need or want any of the default loaders provided for you. The only requirement is your loader must respond to #call and answer a monad with a Hash for content which means you can use a class, method, lambda, or proc. Here’s an example of creating a custom loader, registering, and using it:

require "dry/monads"

class Demo
  include Dry::Monads[:result]

  def initialize fallback: {}
    @fallback = fallback
  end

  def call = Success fallback

  private

  attr_reader :fallback
end

etcher = Etcher::Registry[loaders: [Demo.new]].then { |registry| Etcher.new registry }
etcher.call  # Success({})

While the above isn’t super useful since it only answers whatever you provide as fallback information, you can see there is little effort required to implement and customize as desired.

Transformers

Transformers are great for modifying specific keys and values. They give you finer grained control over your configuration and are the last step before validating and creating an associated record of your configuration. Transformers can either be defined when creating a new registry instance or added after the fact. Here are a few examples:

# Initializer
registry = Etcher::Registry[transformers: [MyTransformer]]

# Method
registry = Etcher::Registry.new.add_transformer MyTransformer

Here are a few guidelines to using them:

They can be initialized with whatever requirements you might need.
They must respond to #call which takes a required content positional argument and answers a modified representation of this content as a monad with a Hash for content.
A second optional positional key parameter should follow your content parameter when implementing your transformer. This allows you to quickly refactor the key later while also reducing key duplication throughout your implementation.
The content passed to your transformer will have symbolized keys so you don’t need to do this yourself.

Here are a few examples of where you could go with this:

The following capitalizes all values (which may or may not be good depending on your data structure).

require "dry/monads"

Capitalize = -> content { Dry::Monads::Success content.transform_values!(&:capitalize) }
Capitalize.call(name: "test")

# Success({:name=>"Test"})

The following updates current time relative to when configuration was transformed.

require "dry/monads"

CurrentTime = lambda do |content, key = :at, at: Time.now|
  content.fetch(key) { at }
         .then { |value| Dry::Monads::Success content.merge!(key => value) }
end

CurrentTime.call({})
# Success({:at=>2023-04-23 15:22:23.746408 -0600})

CurrentTime.call({at: Time.utc(2023, 10, 15)})
# Success({:at=>2023-10-15 00:00:00 UTC})

CurrentTime.call({}, at: Time.utc(2023, 1, 10))
# Success({:at=>2023-01-10 00:00:00 UTC})

The following obtains the current Git user’s email address from the global Git configuration using the Gitt gem.

require "dry/monads"
require "gitt"

class GitEmail
  def initialize key = :author_email, git: Gitt::Repository.new
    @key = key
    @git = git
  end

  def call(content) = git.get("user.email").fmap { |value| content[key] = value }

  private

  attr_reader :key, :git
end

GitEmail.new.call({})

# Success("demo@alchemists.io")

To use all of the above, you’d only need to register and use them:

registry = Etcher::Registry[transformers: [Capitalize, CurrentTime, GitEmail.new]]
etcher = Etcher.new(registry)
etcher.call

Overrides

Overrides are what you pass to the Etcher instance when called. Example:

etcher = Etcher.new
etcher.call name: "test", label: "Test"

# Success({:name=>"test", :label=>"Test"})

Overrides are applied after any loaders are processed and before any transformations. They are a nice way to deal with user input during runtime or provide any additional configuration not supplied by the loading of your default configuration while still allowing you to transform them if desired.

⚠️ In Version 2.0.0, this step will be changed to occur after the Transform step for maximum flexibility.

Resolver

In situations where you’d like Etcher to handle the complete load, transform, validate, and build steps for you, then you can use the resolver. This is provided for use cases where you’d like Etcher to handle everything for you and abort if otherwise. Example:

Etcher.call name: "demo"
# {:name=>"demo"}

When called and there are no issues, you’ll get the fully formed record as a result (in this case a Hash which is the default model). You’ll never a get a monad when using Etcher.call because this is meant to resolve the monadic pipeline for you. If any failure is encountered, then Etcher will abort with a fatal log message. Here’s a variation of earlier examples which demonstrates fatal errors:

require "dry/monads"
require "dry/schema"

Dry::Schema.load_extensions :monads

contract = Dry::Schema.Params do
  required(:to).filled :string
  required(:from).filled :string
end

model = Data.define :to, :from
registry = Etcher::Registry.new(contract:, model:)

Etcher.call registry

# 🔥 Unable to load configuration due to the following issues:
#   - to is missing
#   - from is missing

Etcher.call registry, to: "Mindy"

# 🔥 Unable to load configuration due to the following issues:
#   - from is missing


registry = Etcher::Registry.new(model: Data.define(:name, :label))
Etcher.call registry, to: "Mindy"

# 🔥 Build failure: :record. Missing keywords: :name, :label.

💡 When using a custom registry, make sure it’s the first argument. All arguments afterwards can be any number of key/values overrides which is similar to how Etcher.new works.

Development

To contribute, run:

git clone https://github.com/bkuhlmann/etcher
cd etcher
bin/setup

You can also use the IRB console for direct access to all objects:

bin/console

Architecture

The following illustrates the full sequences of events when etching new records:

Tests

To test, run:

bin/rake

Credits

Built with Gemsmith.
Engineered by Brooke Kuhlmann.

Etcher

1.3.0

Features

Requirements

Setup

Usage

Steps

Registry

Contracts

Types

Pathnames

Versions

Models

Loaders

Environment

JSON

YAML

Custom

Transformers

Overrides

Resolver

Development

Architecture

Tests

License

Security

Code of Conduct

Contributions

Versions

Community

Credits