Dependency Management in Clojure

Holger Schauer / holger.schauer@gmx.de

:clojureD 2016 / Berlin / 2016-02-20

Director's cut: The original detailed version

whoami

  • Working as <inject role here> in a variety of projects, companies, languages and frameworks since 2002
    • web projects, CMS, eCommerce
    • natural language processing, data analysis
  • Just someone using Lisp for many years with an interest in software architecture

What this is not about

What this is about

They call it a classico

Classical designs for procedural languages show a very characteristic structure of dependencies. As (the following figure) illustrates, these dependencies typically start from a central point, for instance the main program or the central module of an application. At this high level, these modules typically implement abstract processes. However, they are directly depending on the implementation of the concrete units, i.e. the very functions. These dependencies are causing big problems in practice. They inhibit changing the implementation of concrete functions without causing impacts on the overall system.

Gernot Starke, "Effektive Softwarearchitekturen"

A mace of dependencies

  • The more entangled the code, the more difficult it becomes to change
  • Abstract processes depend on concrete implementations
  • Dependencies often cross architectural boundaries

Explicit vs. implicit

  • Explicit: everything that is obvious from the outside
    • function signature, argument lists to functions (defn)
    • :refered names in namespace declaration
    • compile time errors
  • Implicit: everything inside only (implementation)
    • using names not given as arguments
    • requiring namespaces only or use :refer :all
    • correctness at run time

Explicit vs. implicit

  • Known unknowns are better than unknown unknowns
  • Or are they?
    • Encapsulation: hiding details that should not matter
    • Relevant information readily available
  • Architectural boundaries and decisions should be obvious

So many questions

  • Developer
    • If and how should a user be able to influence behavior?
    • What do I want to hide so I can change the implementation without breaking assumptions?
  • User
    • What arguments need to be provided?
    • Does this function use something internally I know nothing about that might come back to haunt me?
    • Can I influence the behavior of this function?

Onion architecture

  • Dependencies always point inwards
  • Inner levels might define and use abstractions of outer layers
  • Enables changing outer layers without requiring changes to inner layers

Type of dependencies

  • Usage dependency
    • Function calls
    • Namespace dependency
    • Function call arguments
  • Creation dependency
    • Data objects
    • Closures
  • Run time dependency
  • Compile time dependency

Simple code


(ns playitloud.ui-player
  (:require [playitloud.simple-player :refer [play]]))

(defn play-pressed [randomize]
  (println "User pressed play")
  (play randomize))
	    

(ns playitloud.simple-player
  (:require [playitloud.speaker :refer [blare]]
            [playitloud.musiccoll :refer [get-songs]]))

(defn play [randomize]
  (let [songs (get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blare song))
         songs)))
            

(ns playitloud.speaker)

(defn blare [song]
  (let [result (str "Speaker plays " song)]
    (println result)
    result))
	    
## Simple code - Consists of straight-forward layers + UI + some business logic + I/O (`speaker`) + database (`musiccoll`) - Functions depend on functions - Imply namespace dependencies

Not so simple

Higher order functions


(ns playitloud.ho.player)

(defn play [blarefn songgetterfn randomize]
  (let [songs (songgetterfn)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blarefn song))
         songs)))
	    

	    ; ---------------------------------------------------
(ns playitloud.ho.ui-player
  (:require [playitloud.ho.player :as ho :refer [play]]
            [playitloud.speaker :as speaker]
            [playitloud.headphone :as headphone]
            [playitloud.blue-streamer :as streamer]
            [playitloud.musiccoll :refer [get-songs]]))

(defn play-pressed [speaker randomize]
  (println "User pressed play")
  (condp = speaker
    :speaker   (play speaker/blare get-songs randomize)
    :headphone (play headphone/blare get-songs randomize)
    :stream    (play streamer/blare get-songs randomize)))
	    

Higher order functions

  • Functions as arguments
  • Module doesn't depend on other namespaces
  • Dependency in parameter explicit, but signature implicit
  • Shifts dependency problem to caller

Close over dependencies


(defn make-playfn
  "Returns a function that will play all songs"
  [blarefn songgetterfn]
  
  (fn ; Plays all songs, potentially randomized
    [randomize]
    (let [songs (songgetterfn)
          songs (if randomize (shuffle songs) songs)]
      (map (fn [song]
             (blarefn song))
           songs))))
	      

Close over dependencies

  • Factory function returns a function that closes over dependencies
  • Returned closure takes only domain objects as arguments

Close over functions


(ns playitloud.ho.play-config [...]
(def speaker-play   (make-playfn speaker/blare get-songs))
(def headphone-play (make-playfn headphone/blare get-songs))
(def stream-play    (make-playfn streamer/blare get-songs))

(defn select-playfn [speaker]
  (condp = speaker
    :speaker   speaker-play
    :headphone headphone-play
    :stream    stream-play))
	    

(ns playitloud.ho.clos-ui-player
  (:require [playitloud.ho.play-config :refer [select-playfn]]))

(defn play-pressed [output randomize]
  (let [playfn (select-playfn output)]
    (playfn randomize)))
	      

TL;DR Higher order functions

  • Higher order functions don't depend on concrete implementation at compile time
  • Signature makes explicit that some function is needed
  • Implicit dependency on a callable with matching signature that's only checked at run time

Service locator pattern


(ns playitloud.services.config)

(def ^:dynamic *services*
  {:blare playitloud.speaker/blare
   :get-songs playitloud.musiccoll/get-songs})
	    

(ns playitloud.services.player
  (:require [playitloud.services.config :refer [*services*]]))

(defn- blare [sound]
  ((:blare *services*) sound))

(defn- get-songs []
  ((:get-songs *services*)))
	    

(defn play [randomize]
  (let [songs (get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blare song))
         songs)))
	    
## Service locator pattern - Indirection uses functions as first-class objects - Dependency on lookup function at compile time - Explicit signature definition, but only checked at run time - Dynamic binding of `*services*` allows changes at run time

Funsig library


(ns playitloud.sig.output-device
  (:require [de.find-method.funsig :refer [defsig]]))

(defsig blare "Play sound loudly!" [sound])
	    

(ns playitloud.sig.speaker
  (:require [de.find-method.funsig :refer [defimpl]]
            [playitloud.sig.output-device :refer [blare]]))

(defimpl blare [sound]
  (let [result (str "Speaker plays " sound)]
    (println result)
    result))
	    

(ns playitloud.sig.player
  (:require [playitloud.sig.output-device :refer [blare]]
            [playitloud.musiccoll :as mc :refer [get-songs]]))

(defn play [randomize]
  (let [songs (get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blare song))
         songs)))
	  
## Funsig library - Hides the service locator pattern behind macros - Signature validation & multiple implementations - Caller has no say in choice of implementation - Implementation and caller depend on abstraction at compile time

clj-di library


(ns playitloud.di.config
  (:require [clj-di.core :refer [register!]]
            [playitloud.speaker :as speaker :refer [blare]]))

(register! :blare blare)
; ---------------------------------------------------
(ns playitloud.di.player
  (:require [clj-di.core :refer [get-dep]]
  	    [playitloud.musiccoll :refer [get-songs]]))

(defn play [randomize]
  (let [blarefn (get-dep :blare) ; <-- get dependency
        songs (get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blarefn song))
         songs)))
	    
## clj-di - clj-di also uses service locator - clj-di makes 'dependency injection' explicit - Signatures of registered functions are implicit only, check at run time - Provides a `with-registered` for substitutions in tests

Protocols


(ns playitloud.proto.output-device)

(defprotocol OutputDevice
  (blare [device sound])
  (inc-volume [device])
  (dec-volume [device]))
	    

(ns playitloud.proto.speaker
  (:require [playitloud.proto.output-device
             :as output :refer :all]))

(defrecord Speaker [volume]
    OutputDevice
  (blare [_ sound]
    (let [result (str "Speaker plays " sound)]
      (println result)
      result))
  ;[...]
	    

Protocols


(ns playitloud.proto.player
  (:require [playitloud.proto.output-device :as output]
            [playitloud.musiccoll :as mc]))

(defn play [output-device randomize]
  (let [songs (mc/get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (output/blare output-device song))
         songs)))
	    

Protocols imply types

  • Protocols group functions with high cohesion
  • Implementation in terms of types
    • defrecord, extend-type, reify
    • Single type can implement multiple protocols
  • Feels natural when state is required
    ... feels like object-oriented programming again

TL;DR Protocols

  • Inversion of control
    • Caller depends on abstraction at compile-time
    • Implementation depends on protocol at compile-time
  • Caller has to provide an instance of a type implementing the protocol at run-time
  • Signatures of functions are explicitly declared in protocols
  • Protocols are the standard way to decouple dependencies

Multi-methods


(ns playitloud.multi.output-device)

(defmulti blare
  "Play sounds on an output device"
  (fn [device sound]
    device))
	    

(ns playitloud.multi.speaker
  (:require [playitloud.multi.output-device :refer [blare]]))

(defmethod blare :speaker [device song]
  (let [result (str "Speaker plays " song)]
    (println result)
    result))
	    

(ns playitloud.multi.headphone
  (:require [playitloud.multi.output-device :refer [blare]]))

(defmethod blare :headphone [device song]
  (let [result (str "Headphone plays " song)]
    (println result)
    result))
	    

Multi-methods usage


(ns playitloud.multi.player
  (:require [playitloud.multi.output-device :refer [blare]]
            [playitloud.musiccoll :as mc]))

(defn play [output-device randomize]
  (let [songs (mc/get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blare output-device song))
         songs)))
	    

(ns playitloud.multi.ui
  (:require [playitloud.multi.player :as player :refer [play]]))

(defn play-pressed [output-selection randomize]
  (println "User pressed play")
  (play output-selection randomize))
	    
## Multi-methods - `defmulti` provides abstract "signature" specification plus dispatch function - Implementation is chosen via dispatch function - Dispatch does not have to happen in terms of types - Can't use multi-methods to group different methods like protocols, simply use namespaces
## TL;DR Multi-methods - Inversion of control + Caller depends on abstraction at compile time + Implementation depends on abstraction (multimethod) - Implicit _detailed_ knowledge of dispatch function required - If you don't need flexibility of dispatch function, using multi-methods for resolving dependencies is over-engineered

Stateful dependencies

- DDD distinguishes between domain entities and services - In OOP, services implemented by objects + objects manage state locally - In FP, functions implement services + strive for "pure" functions without side-effects + state management is still important

Resource management

- Services often dependend on states that are _resources_ + not domain entities + technical "details" like connections, channels, ... - Lifecycle management of resources in applications is often critical

Closures, revisited


(ns playitloud.closure-streamer)

(defn blare [connection-config]
  (let [connection (connect connection-config " closure streamer")]
    (fn [noise]
      (let [result (str "Streaming " noise " to " connection)]
        (println result)
        (transmit connection noise)
        result))))
	    
## Closures, revisited - We can configure state before function call - This separates dependency setup from call time - But we can't change the state easily

Dynamic variables


(ns playitloud.dynvar.config)

(def ^:dynamic *streamer*
  {:connection ; [...]})
	    

(ns playitloud.dynvar.streamer
  (:require [playitloud.dynvar.config :as config]))

(defn blare [noise]
  (let [result (str "Streaming " noise " to " config/*connection*)]
    (transmit (:connection config/*streamer* noise)
    result)))
	    

Dynamic variables

  • Refer to a (dynamic) var binding the state
  • Common pattern in with-... style macros
  • Dependency is often quite implicit
    • Difficult to change
      • dynamic scope is thread-local binding
      • Who is responsible for creating, changing or stopping resource in a multi-threaded program?
    • binding doesn't mix with lazyness (map,...)
  • Check out Stuart Sierra's On the perils of dynamic scope
## Component - Explicit management of complex dependencies - Components need and obey a _life-cycle_ protocol - All components are created and all dependencies are “injected” when _system_ starts up - Components make up an application, so don't use component in your library - Supports Stuart Sierra's reloaded workflow

Component lifecycle implementation


(ns playitloud.comp.streamer
  (:require [com.stuartsierra.component :as component]
            [playitloud.comp.output-device :as output :refer :all]
            [playitloud.comp.remote-connection :as remote]))

(defrecord BlueStreamer [connection volume]
    OutputDevice
  (blare [streamer sound]
     ;... normal protocol/record implementation ...)
     
    component/Lifecycle
  (start [streamer]
    (println "BlueStreamer starting")
    (connect (:connection streamer) streamer)
    (assoc streamer :status :connected))
  (stop [streamer]
    (println "BlueStreamer stopping")
    (disconnect (:connection streamer) streamer)
    (->> (assoc streamer :status :disconnected)
         (dissoc :connection))))

(defn new-blue-streamer [config]
  (->BlueStreamer nil (:default-volume config)))
             

Component usage


(ns playitloud.comp.player
  (:require [com.stuartsierra.component :as component]
            [playitloud.comp.output-device :as output]
            [playitloud.musiccoll :as mc]))

(defrecord Player [output-device]
  component/Lifecycle
  (start [player]
    (println "Player started with "
             "output-device " (:output-device player))
    player)
  (stop [player]
    (println "Player stopped")
    player))

(defn play [player randomize]
  (let [songs (mc/get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (output/blare (:output-device player) song))
         songs)))

(defn new-player []
  (->Player nil))
	      

Component systems


(ns playitloud.comp.config
  (:require [com.stuartsierra.component :as component]
            [playitloud.comp.blue-connection :as blueconn]
            [playitloud.comp.streamer :as streamer]
            [playitloud.comp.player :as player]))

(def default-config
  {:bt-conn {:name "Fabuluous-Connectivity" :port "1234"}
   :default-volume 5})

(defn make-player-system
  "Setup the player system"
  ([]
   (make-player-system default-config))
  ([config]
   (component/system-map
    :connection (blueconn/new-connection config)
    :output-device (component/using
                    (streamer/new-blue-streamer config)
                    [:connection])
    :player (component/using
             (player/new-player)
             [:output-device]))))
	      

More components

  • When some piece of code fun-a depends on some managed state, it needs to become a component A
  • In turn, some fun-b that wants to call fun-a now needs to depend on component A, too and hence also needs to become a component (B)
  • B will hold a reference to a component A at run time
## TL;DR Component - Dependencies of components can be inspected and changed (entire sub-graph of dependencies) - “Full buy-in” can lead to code in which most functionality is managed with protocols and records - Only “inject” needed dependencies + System map is a global state (map) of dependencies + Don't pass around system map or access the system map as a var (global environment) in code
## Mount - _Making application state enjoyably reloadable_ - Same purpose as component, but explicitly takes a different approach + Claims to be “more clojure-ish” + provides lifecycle functions (no protocol) - _trust the Clojure compiler for dependency resolution_ + aka `:require`/`:refer`

Mount states


(ns playitloud.mount.streamer
  (:require [mount.core :refer [defstate]]
            [playitloud.mount.connection
                 :refer [conn transmit send-command]]))

(defn create-streamer [connection]
  (println "Connecting to connection ..." connection)
  {:connection connection})

(defstate streamer :start (create-streamer conn))

(defn blare [streamer sound]
  (let [result (str "Will stream " sound
                     " to  " (:connection streamer))]
    (println result)
    (transmit conn sound)
    result))
	      

Mount usage


(ns playitloud.mount.player
  (:require [mount.core :refer [defstate]]
            [playitloud.mount.streamer :refer [streamer blare]]
            [playitloud.musiccoll :as mc]))

(defn create-player [streamer]
  (println "Will use streamer " streamer)
  {:streamer streamer})

(defstate player :start (create-player streamer))

(defn play [randomize]
  (let [songs (mc/get-songs)
        songs (if randomize (shuffle songs) songs)]
    (map (fn [song]
           (blare (:streamer player) song))
         songs)))
	    

(facts "Handling state with mount"
       (with-state-changes [(before :facts (mount/start))
                            (after :facts (mount/stop))]
         (fact "We can play over the configured device"
               (play false)
               => (just ["Will stream David Bowie -- Blackstar " ; ...
	  
## Mount characteristics - States have explicit start and stop functions (or values) - Dependencies involve only vars and namespaces - Doesn't mandate use of records or protocols + behavior implemented with normal functions - Vars hold `DeferableState` objects which implement `IDeref` interface and which behave like values when `:started`
## TL;DR Mount - mount focusses on state, but doesn't offer an obvious way to change behavior + _orthogonal_ to what mount strives to do well + you can use protocols for abstracting behavior - Code using states rely on global state -- the global environment which holds vars - Implicit state dependencies vs. explicit argument passing in component library
## TL;DR Dependencies in Clojure - Clojure provides many ways to manage dependencies - A lot of choice wrt. being explicit or leaving things implicit - Variety regarding time and place of dependency resolution + either once and global per application ("singleton" like) + or everywhere locally within the application + there is no controlled middle ground (e.g. inject new state on request / transaction boundary)
## Keep it simple - Don't overengineer, focus on important things - Always be composing – cf. Zach Tellman + Prefer small pure functions + Reduce dependency problem to “integration functions” + Testing motivation for DI mostly goes away - Split out dependencies of code on components / services + that you think could change + or of which you already have multiple implementations
# THANK YOU
- Source code [https://github.com/schaueho](https://github.com/schaueho) - Blog [http://www.find-method.de/](http://www.find-method.de/) - Google+ [https://plus.google.com/Holger.Schauer](https://plus.google.com/Holger.Schauer)
## Image credits
- [Needle](http://www.torange.us/Backgrounds-textures/wallpapers/Medical-background-19870.html) from www.tOrange.us under CC-SA 4.0 - [Balls](https://commons.wikimedia.org/wiki/File:Colorful_Super_ball.jpg) from 高橋 宗史 under CC-by-SA 2.5 - [Wasserhahn](https://commons.wikimedia.org/wiki/File:Wasserhahn_aus_Holzwand.JPG) from High Contrast under CC-by-3.0 DE