Railway Pattern | Alchemists

Railway Pattern

This pattern is coined from the Railway Oriented Programming presentation — and subsequent book: Domain Modeling Made Functional — by Scott Wlaschin which explains how to pipe functions together in a fault tolerant manner. The good news is Ruby is great at blending objects with functions for maximum effect. In order to learn how to apply this pattern within your own code, we’ll leverage the following foundational gems that make this pattern shine:

Dry Monads: Provides low level monad functionality. There are several types of monads included in this gem but we’ll focus on the Result monad.
Pipeable: Implements this pattern by building upon Ruby’s native function composition capabilities via the forward (#>>) and backward (#<<) composition methods. When coupled with Dry Monads, we get a clean and succinct Domain Specific Language (DSL) for piping (composing) multiple functions together as a single fault tolerant pipeline.

In order to illustrate how powerful this pattern is, we’ll focus on building an implementation that can obtain information from an external third-party API service. The problem with external APIs is that they can be faulty, suffer from outages, network delays, have ugly data structures, and a slew of other problems which makes this pattern perfect for implementing an elegant solution to this common problem.

Table of Contents

Fundamentals
Setup
Approaches
- Naive
- Traditional
- Functional
- Railway
Advantages
Disadvantages
Guidelines
Conclusion

Fundamentals

At a high level, one way to conceptualize this pattern is to think of train tracks which can either fork to the left (success) and right (failure).

Alternatively, you can think of this pattern as a series of functional steps in a pipeline. The result of each step is either a success or failure. Even better, each success or failure can be transformed further before producing the final result. This is what makes a fault tolerant pipeline powerful because you have the opportunity to transform (heal) a failure by turning it into a success. To illustrate, here’s a visualization of a fault tolerant pipeline using Dry Monads Result syntax:

You don’t need to know what each method of a Result does to understand the above. You only need to see that, for each step within the pipeline, you’ll get a success or failure which can be transformed or left as is. This means there are multiple ways to yield a success or failure along with the opportunity to turn a failure into a success (i.e. fault tolerance).

If it helps, you can also think of the Result monad as a specialized boolean: true (success) and false (failure). Unlike a boolean, a Result carries information which can be unwrapped for further processing and then rewrapped to produce an updated result. Definitely more powerful than a simple boolean yet not losing basic boolean functionality!

Setup

Let’s say we want to obtain Studio Ghibli films by implementing an API client. To pull this off, we need to:

Use the Studio Ghibli API, specifically the films endpoint.
Use an HTTP client to request and parse API requests.
Validate the JSON response.
Model the response as a new records for immediate interaction.

Visually, this is:

As you can see, several steps are required for pulling this off. Additionally, we can have bad data, network issues, and potential exceptions along the way. In order to tackle these steps, we can start by building up a Bundler Inline script with the dependencies and objects we need for quick experimentation. Here are the dependencies:

#! /usr/bin/env ruby
# frozen_string_literal: true

# Save as `demo`, then `chmod 755 demo`, and run as `./demo`.

require "bundler/inline"

gemfile true do
  source "https://rubygems.org"

  gem "amazing_print"
  gem "debug"
  gem "http"
  gem "dry-monads"
  gem "dry-schema"
  gem "pipeable"
end

Now we can add a few objects to the above script to implement our solution:

Requester

The following provides a low-level HTTP requester which is built upon the HTTP gem for making basic HTTP requests specifically tailored for the Ghibli API. We only need to make HTTP GET requests but, in practice, you’d have methods for POST, PUT, DELETE, and so forth.

module Ghibli
  module API
    class Requester
      def initialize uri: "https://ghibliapi.vercel.app", http: HTTP
        @uri = uri
        @http = http
      end

      def get(path) = http.get("#{uri}/#{path}")

      private

      attr_reader :uri, :http
    end
  end
end

Contract

The following provides a Dry Schema contract for validating the HTTP response while also casting attributes to the appropriate type. The contract ensures all records be an array (value) of the data key. The data key is necessary because Dry Schema can’t validate raw arrays, they must always be wrapped in a hash.

module Ghibli
  module API
    module Contracts
      Film = Dry::Schema.JSON do
        required(:data).array(:hash) do
          required(:id).filled :string
          required(:title).filled :string
          required(:original_title).filled :string
          required(:original_title_romanised).filled :string
          required(:description).filled :string
          required(:director).filled :string
          required(:producer).filled :string
          required(:release_date).filled :string
          required(:running_time).filled :string
          required(:rt_score).filled :string
          required(:people).filled :array
          required(:species).filled :array
          required(:locations).filled :array
          required(:vehicles).filled :array
          required(:url).filled :string
        end
      end
    end
  end
end

Model

The following models validated data from our contract so we can immediately interact with record objects instead of a primitive hash. We use the .for method as a convenient solution map over the array of data entries and create new records.

module Ghibli
  module API
    module Models
      Film = Data.define(
        :id,
        :title,
        :original_title,
        :original_title_romanised,
        :description,
        :director,
        :producer,
        :release_date,
        :running_time,
        :rt_score,
        :people,
        :species,
        :locations,
        :vehicles,
        :url
      ) do
        def self.for(**attributes)
          attributes[:data].map { |entry| new(**entry) }
        end
      end
    end
  end
end

Client

The following ties all of the above into our primary API client so we can make the HTTP GET request, parse the response, validate the response, and model the response as records we can immediately interact with.

module Ghibli
  module API
    class Client
      def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film
        @requester = requester
        @contract = contract
        @model = model
      end

      def films
        requester.get("films")
                 .parse
                 .then { |payload| contract.call(data: payload).to_h }
                 .then { |attributes| model.for(**attributes) }
      end

      private

      attr_reader :requester, :contract, :model
    end
  end
end

Execution

The last thing we need to is initialized the client and message the #films method:

client = Ghibli::API::Client.new
films = client.films

You should find that films is an array of film objects. Example (truncated for brevity):

[
  #<data Ghibli::API::Models::Film>,
  #<data Ghibli::API::Models::Film>,
  #<data Ghibli::API::Models::Film>
]

These objects are the core building blocks when designing an API client because you need to:

Make HTTP requests to the API server.
Validate the HTTP responses since you sometimes can’t rely on the data being correct or always in the same structure.
Model the data so you have a quick way to interact with structured objects instead of raw hashes which avoids the Primitive Obsession code smell.

As you follow along, take the time to copy and paste the above code into a Bundler Inline script so you can quickly run the code. This’ll come in handy as we tweak and evolve each of these objects into our final solution.

Approaches

Now that you have all of the objects necessary for implementing our API client we can discuss the various approaches to solving the problem.

Naive

The naive approach is exactly what you see in the Ghibli::API::Client#films implementation which has several problems. Here’s the code, again, for convenience:

requester.get("films")
         .parse
         .then { |payload| contract.call(data: payload).to_h }
         .then { |attributes| model.for(**attributes) }

If any of the above steps fail, we’ll get an exception with a stack dump. Worse, each step assumes the previous step was successful with no way to deal with errors, invalid data, etc.

This approach only solves the happy path at the cost of being brittle due to not being fault tolerant.

Traditional

The traditional approach is to use conditional and exception logic. Here’s what this looks like after updating our Ghibli::API::Client#films implementation:

def films
  result = validate parse

  case result
    in data: Array then model.for(**result)
    else result
  end
end

private

attr_reader :requester, :contract, :model

def parse
  response = requester.get "films"
  body = response.parse

  response.status.success? ? body : {error: body}
rescue HTTP::Error, JSON::ParserError => error
  {error: error.message}
end

def validate payload
  return payload if payload.is_a? Hash

  result = contract.call data: payload

  result.success? ? result.to_h : {error: result.errors.to_h}
end

If you apply the above changes and run the code again, you’ll get the same result as before. Conditionals and rescues are introduced to make the code more robust so we get a films array or an error hash. Definitely an improvement at the cost of becoming more complex due to extra conditional logic and use of array (films) or hash (error) type checking. Definitely not ideal.

There is one major improvement with this implementation which is we are starting to see the beginning of a fault tolerant pipeline. This means each step (i.e. request, parse, validate, and model) is robust enough to either produce the desired result of film records or an error. If an error is encountered in any of the steps, then the error bubbles up so we can do further processing of the result as desired.

We can do better by switching to monads which brings us to a more functional approach.

Functional

For this approach, we need to make a few modifications to our original objects. Here’s the changes:

Requester

The following modifies the original implementation so that we include the Dry Monads Result type and modify #get so it answers either a Success or Failure. This includes capturing any HTTP::Error as a failure as well.

class Requester
  include Dry::Monads[:result]

  def initialize uri: "https://ghibliapi.vercel.app", http: HTTP
    @uri = uri
    @http = http
  end

  def get path
    http.get("#{uri}/#{path}").then do |response|
      response.status.success? ? Success(response) : Failure(response)
    end
  rescue HTTP::Error => error
    Failure error.message
  end

  private

  attr_reader :uri, :http
end

Contract

For our contract, we only need to teach it how to enable monad support by loading the monads extension as follows:

Dry::Schema.load_extensions :monads

Client

The following updates the #films method to use the Fluent Interface of Dry Monads while deleting all of the previously added private methods.

class Client
  def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film
    @requester = requester
    @contract = contract
    @model = model
  end

  def films
    requester.get("films")
             .fmap(&:parse)
             .bind { |payload| contract.call(data: payload).to_monad }
             .fmap { |result| model.for(**result.to_h) }
  rescue JSON::ParserError => error
    Failure error: error.message
  end

  private

  attr_reader :requester, :contract, :model
end

In case you are not familiar with how the Result monad works, here’s a breakdown of each step:

requester.get("films"): Answers either a Success or Failure because we taught the Requester to always answer a monad.
fmap(&:parse): The #fmap method is short for function map and means the block will only be called if the result is a Success, otherwise any Failure passes on to the next step. So when we have a Success, that success will be automatically unwrapped to pass the HTTP response to the block which we can then parse. Once parsed, the result (i.e. JSON body) will automatically be wrapped back up as a Success to pass on to the next step.
bind { |payload| contract.call(data: payload).to_monad }: The #bind method only executes the block if the result is a Success. On Success, the contract will be called to answer a monad via the #to_monad message and — this is important — bind doesn’t automatically wrap the result of the block as a Success which means that when #to_monad is called we’ll answer back either a Success or Failure.

We still have to account for JSON parsing errors which is why the rescue is still in the #films method. In this case, we wrap the hash as a Failure with an error key and string message. You could also wrap the entire error as a failure (i.e. Failure error:) in order for you to use more advanced Pattern Matching due to knowing the type of error that caused this failure instead of a simple string as the error value. For the purposes of this implementation, we’ll stick with a string.

If you apply the above changes and run the code, you’ll see films is wrapped as a Success. Example (truncated for brevity):

Success(
  [
    #<data Ghibli::API::Models::Film>,
    #<data Ghibli::API::Models::Film>,
    #<data Ghibli::API::Models::Film>
  ]
)

This means we can use Pattern Matching to process the result:

include Dry::Monads[:result]
client = Ghibli::API::Client.new

case client.films
  in Success(*films) then puts films
  in Failure(error: message) then puts "ERROR: #{message}"
  else puts "Unknown type."
end

We now have the beginnings of a functional flow where each step expects either a Success or Failure. If any of the steps fail, then the failure gracefully passes through all of the subsequent steps for further processing or skipped altogether.

At this point, we could stop here because we have an implementation that adheres to the Railway Pattern. Each step will either continues down the Success track of the railway or forks to end up as a failure.

We can do better, which leads us to our final approach that fully implements the Railway Pattern.

Railway

We only need to update our Client to use the Pipeable gem which allows us to use Function Composition. Here’s the updated implementation:

class Client
  include Pipeable

  def initialize requester: Requester.new, contract: Contracts::Film, model: Models::Film
    @requester = requester
    @contract = contract
    @model = model
  end

  def films
    pipe requester.get("films"),
         try(:parse, catch: JSON::ParserError),
         merge(as: :data),
         validate(contract, as: :to_h),
         to(model, :for)
  end

  private

  attr_reader :requester, :contract, :model
end

Two important changes to call out are:

We include Pipeable functionality at the top of the class.
The #films method has been updated to pipe all steps as one continuous pipeline. As with the monad approach, if any of the steps fails, the Failure will pass through all subsequent steps without further operation or exception.

If you’ve not used the Pipeable gem before, the gem allows you to pipe (compose) a series of functional steps together as a single operation. In this case, we are using the default steps that come with the gem but you can build your own custom steps too. Here’s what each step is doing (which is similar to what we did in the monads approach only with a concise syntax):

We start by using pipe to supply all steps. The first step expects a Result. If not a Result it’ll be automatically wrapped as a Success in order to kick off processing (handy when you have a raw value as input without having to wrap it first).
The next step tries to parse the HTTP result. Any errors will be caught and wrapped as Failures. Otherwise, the result is a wrapped as a Success.
The next step merges the array of films into a hash with data as the key because the contract requires a hash to validate.
The next step validates the payload using the injected contract. Upon success, it’ll be cast as a Hash.
Finally, the last step will delegate the result of the previous step (i.e. the validated payload) to the injected model using the .for method to convert the payload of data into actual Film records.

For Elm fans, the pipe method becomes the pipe operator (i.e. |>) which would yield this blended Elm/Ruby pseudo code:

requester.get("films")
  |> try(:parse, catch: JSON::ParserError)
  |> merge(as: :data)
  |> validate(contract, as: :to_h)
  |> to(model, :for)

Make no mistake, the above is syntactically incorrect Elm code but the shape is similar to how’d you implement a pipeline in Elm. With Ruby, we don’t have a pipe operator so, by using Pipeable, we can achieve the desired effect using the pipe method. Each step ends up being composable using Function Composition courtesy of the forward composition method (#>>) which is available to procs, lambdas, methods, and — with Pipeable — classes. 🎉

Advantages

There are several advantages to this pattern:

You can pipe multiple steps together as a single, fault tolerant, operation.
Each step can produce a success or failure for which you can transform and/or repair further.
You can build super pipes which allows you to chain multiple pipes together with their own series of steps.
You can interchangeably compose procs, lambdas, methods, and classes together which makes blending functions with objects seamless.
You can apply Pattern Matching to process the final result to powerful effect.

Disadvantages

There can be a few disadvantages to this pattern depending on your situation:

Requires gem dependencies since Ruby doesn’t have native support.
Each pipeline answers a monad which you may or may not always want. That said, Dry Monads does provide convenience methods, such as #value!, to get around this.
Can degrade performance especially in hot code paths so monitor, measure, and adjust accordingly.

Guidelines

When using this pattern, there are a few guidelines worth noting:

Use to gracefully bubble up errors, problems, and/or data you can’t repair or massage into a success. There are times, like capturing a JSON parse error, and bubbling it up as a failure where the failure is not catastrophic but worth informing the user so they can take corrective action. Other times, you might want to capture an HTTP timeout exception as a failure but then retry the HTTP request with an exponential backoff after three attempts to make your pipeline more fault tolerant.
Use to build more advanced architectures where you can mix and chain multiple pipes together as super pipes.
Avoid when needing to fail fast. Letting an exception through or even exiting the program can be useful for catastrophic errors that need immediate attention.
Avoid when not needing to process any failures, don’t need to chain pipelines together (i.e. super pipes), or don’t need to unwrap the result each time.
Avoid this pattern if you need to be highly performant. Both the Pipeable and Dry Monads gems introduce some overhead which may not be desired in all code paths.

Conclusion

If you take a moment to look at where we started with the naive approach and compare that to the final railway solution, you see the steps are, roughly, the same:

The key difference is the Railway Pattern allows us to functionally compose our steps in a fault tolerant manner while remaining reusable and composable via more complex architectures. That’s a far cry from what we started with while still remaining syntactically concise. You won’t always need this pattern but definitely is handy when you need to build fault tolerant code that can self heal while keeping your implementation clean and concise.

You’ve now learned what the Railway Pattern is, why it is useful, and how to apply it to your own code. As a bonus, you have working code you can play and experiment with further. Enjoy!