Functional Service in Scala

Often, a backend application is a black box, even if we look at its code, we still can’t simply reason about it locally without some assosiated context. It is hard to figure out what is going on at particular layer of the application, even though it can be a mainly stateless application. One of the main problem here is that we can’t reason about an application code, because of too weak method or function type signatures.

Functional programming is great paradigm to bring clarity into any codebase. What would it mean for a backend developer to implement a REST service in functional style? The answer to this is referential transparency[1], usage of functions as first-class citizens, and explicit side-effect control.

The Scala ecosystem provides lots of libraries to build backend applications. One popular choice to build REST APIs is Play and Akka-HTTP. However, if we want to build a pure-functional application with full control of side-effects and work with programs like with values, then HTTP4s library is a perfect choice for that. Often, we need to manage some state in some SQL database. Slick library is a good and mature option to work with an SQL database via the JDBC. However, to have a pure-functional persistence layer, one could use the Doobie library instead of Slick.

Goals

What we want to achieve:

Get a pure-functional program/service in Scala.
Look at transition to Typelevel libraries.
Bonus goal: have an opportunity to choose an effect type by specifying it at a single place.

Before we start achieving these goals, let’s define what we mean at the goal #1 based on this talk. It is important to get an idea what the pure functional program is.

Referential transparency

Everything starts with the referential transparency:

An expression e is referentially transparent if for all programs p every occurrence of e in p can be replaced with the result of evaluating e without changing the meaning of p.

A pure function

A function f is pure if f(x) is RT when x is RT.
A pure function does not depend on anything other than its argument.

A pure program

A pure functional program consists of the referentially transparent (RT) expressions.

Solution

We start an implementation of a service with Akka + Slick and migrate to HTTP4s + Doobie to get pure-functional program using monads.

REST API

We want to register user trips by taxi, bike, car within some distance and price. This is to be implemented by a REST API.

Add: POST /api/v1/trips, body = JSON
Update: PUT /api/v1/trips/<id>, body = JSON
Delete: DELETE /api/v1/trips/<id>
Select all: GET /api/v1/trips?sort=id&page=1&pageSize=100
Select one: GET /api/v1/trips/<id>

Model

final case class Trip(
  id: Int,
  city: String,
  vehicle: Vehicle,
  price: Int,
  completed: Boolean,
  distance: Option[Int],
  endDate: Option[LocalDate]
)

final case class Trips(trips: Seq[Trip])

object Vehicle extends Enumeration {
  type Vehicle = Value
  val Bike, Taxi, Car = Value
}

final case class CommandResult(count: Int)

Service

Our service logic and data layer representation must be 3rd-party libraries-free. That means, our core abstractions should not depend or import some modules from the infrastructure libraries, which we are going to use in our application. Let’s take care of that, before we implement and combine all the layers.

// service layer
trait TripAlg[F[_]] {
  def selectAll(page: Option[Int], pageSize: Option[Int], sort: Option[String])
  : F[Trips]
  def select(id: Int): F[Option[Trip]]
  def insert(trip: Trip): F[Int]
  def update(id: Int, trip: Trip): F[Int]
  def delete(id: Int): F[Int]
}

// data layer mimics almost everything the service does
trait Repository[F[_]] {
  def delete(id: Int): F[Int]
  def update(id: Int, row: Trip): F[Int]
  def createSchema(): F[Unit]
  def insert(row: Trip): F[Int]
  def selectAll(page: Int, pageSize: Int, sort: String): F[Seq[Trip]]
  def select(id: Int): F[Option[Trip]]
  def sortingFields: Set[String]
}

F[_] stands for some generic high-order type, which represents an effect type. This is used for almost every return type of the service and repository layers. For now, this effect type remains abstract. It will be defined when we are ready to launch the application or a unit test for this application.

Build with Akka-Http and Slick

Akka and Slick are both asynchronous libraries and they work with the standard Scala Future. That means, we have to go with Scala Future when we fix F[_] at the very beginning of the program execution path.

We could use another effect type, like Cats IO, but this would require transformation of IO to Scala Future behind the scene, which would change the IO behaviour from lazy to eager. Basically, this would break the original idea to get pure-functional service. Why? Because Scala Future is running immediately upon its construction and it is caching its computed value. This leads to violation of referential transparency. Here is very good and short explanation how Future brakes that.

// we use Macwire to automatically wire dependencies
import com.softwaremill.macwire.wire

class AkkaModule(cfg: Config) {
  val db = Database.forConfig("storage", cfg)
  val repo = wire[SlickTripRepository]
  val service = wire[TripService[Future]]
  val routes = concat(wire[QueryRoutes].routes, wire[CommandRoutes].routes)
}

Macwire is a lightweight Scala dependency injection library. Method wire is a Scala macro, which generates object creation code at compile time. See Macwire documentation for more details.

Above AkkaModule combines all the parts together. In the result, we can use routes to start the Akka-HTTP server.

Akka-HTTP routes

This is what Routing DSL of Akka-HTTP looks like:

class CommandRoutes(service: TripService[Future])(implicit ...) {

  def routes: Route = pathPrefix("api" / "v1" / "trips") {
      concat(
        pathEndOrSingleSlash {
          post {
            entity(as[Trip]) { trip =>              
              val inserted = service.insert(trip)
              complete {
                // converting to HTTP response with JSON body
                toCommandResponse(inserted, CommandResult)
              }
            }
          }
        },
        ...
      )
  }

In a similar way, we define the rest endpoints (see source code for the full code example).

Slick mapping

Slick is a functional relational mapping library, which works with the SQL Databases over the JDBC. We map each case class to its own Slick Table entity:

class Trips(tag: Tag) extends Table[Trip](tag, "trips") {

    def id = column[Int]("id", O.AutoInc, O.PrimaryKey)
    def city = column[String]("city")
    ... // rest of the columns here

    def * =
      (id, city, vehicle, price, completed, distance, endDate) <>
        (Trip.tupled, Trip.unapply)
  }

Next, we write our Repository implementation. It uses a Slick table instance:

class SlickTripRepository(db: Database) extends Repository[Future] {
  val trips = TableQuery[Trips] 

  override def select(id: Int): Future[Option[Trip]] =
    db.run(trips.filter(_.id === id).take(1).result.headOption)

  override def update(id: Int, row: Trip): Future[Int] = 
    db.run(trips.filter(_.id === id).update(row))
  ...  
}

As we can see, Future is used in implementation of Akka-HTTP routes (implicitly) and in Slick database actions. That is an actual problem to achieve our goal, since Scala Future is not referentially transparent as we concluded above.

Build with HTTP4s and Doobie

HTTP4s and Doobie are both Typelevel projects and very well integrated with the Cats library. Therefore, our effectful[2] code can be conveniently wrapped with the Cats-Effect IO monad.

class Http4sModule[F[_]: Async: ContextShift](cfg: JdbcConfig) {

  val xa = Transactor.fromDriverManager[F](
    cfg.driver.value, cfg.url.value, 
    cfg.user.value, cfg.password.value
  )
  val repo = wire[DoobieTripRepository[F]]
  val service = wire[TripService[F]]
  val apiPrefix = "/api/v1/trips"
  val routes: HttpRoutes[F] = Router(apiPrefix -> (
      wire[QueryRoutes[F]].routes <+> wire[CommandRoutes[F]].routes
  ))

This module structure is very similar to previous Akka module. However, using HTTP4s and Doobie we are able to abstract effect type. Implicit Async[F] and ContextShift[F] are required by the Doobie transactor.

What is an IO Monad?

An IO stands for input and output. An IO Monad is a special monad used for effectful programs. These are programs that eventually produce some side-effects in the external world. Before we run the IO programs, they stay referentially transparent. When we use IO Monad, we basically lift side-effects to the type signature. The type signature shows that the program does interact with the external world from its inside, and thus must be used carefully.

For quick comprehension, a short implementation version of an IO Monad can be:

@ class IO[A](body: => A) {
    def flatMap[B](f: A => IO[B]): IO[B] =
      new IO[B](f(body).run)

      def run: A = body
  }
defined class IO

@ val printHi = new IO(println("hi there"))
printHi: IO[Unit] = ammonite.$sess.cmd20$IO@2e40ea48

@ printHi.run
hi there

A chunk of code inside the body is lazy. Method run is here to execute the I/O action, which is captured by the IO constructor.

When to run an IO program?

In functional programming, we tend to avoid side-effects by delaying them. Usually, we launch such a side-effectful program inside the entry point of an application, for example from the main method of a Scala program. IO monad is a wrapper around some code and this captured code is talking to the external world. Some examples of I/O actions can be read/write to console, database, file or network.

HTTP4s routes

class CommandRoutes[F[_]: Sync](service: TripService[F]) 
  extends Http4sDsl[F] with CirceJsonCodecs {

  val routes = HttpRoutes.of[F] {
    case req @ POST -> Root =>
      for {
        trip <- req.as[Trip]
        i <- service.insert(trip)
        resp <- Ok(CommandResult(i))
      } yield resp
      ...
  }

Other HTTP endpoints are to be defined in a similar way.

To stress one important point: HTTP4s and Doobie take a type of the effect on the API level. That allows us to keep abstraction even further by saying that our effect is some F[_]. So it is very easy to parametrise the HTTP and data layers of our application when using HTTP4s and Doobie with some concrete effect type like Cats IO, Monix Task, Scalaz Task, Identity or even Scala Future (like IO -> Future via IO.unsafeToFuture).

Doobie implementation

Doobie is a pure functional layer on top the JDBC API. This means, we have to write our SQL queries manually, like we would do with the JDBC API using e.g. PreparedStatement, etc. There are no ORM[3] or FRM[4] capabilities, but this is perfectly fine for most projects. SQL queries can be written using SQL fragments, which helps to reduce code duplication.

class DoobieTripRepository[F[_]: Sync](xa: Transactor[F]) 
  extends Repository[F] {

  override def select(id: Int): F[Option[Trip]] =
    sql"SELECT * FROM trips WHERE id = $id"
      .query[Trip]
      .to[List]
      .map(_.headOption)
      .transact(xa)

  override def update(id: Int, row: Trip): F[Int] = {
    val valuesFrag =
      fr"(${row.id}, ${row.city}, ${row.vehicle}, ${row.price}, " +
      fr"${row.completed}, ${row.distance}, ${row.endDate})"

    (updateFrag ++ valuesFrag).update.run.transact(xa)
  }

    ...    
}

object DoobieTripRepository {
  val columnsWithComma = List(
        "id", "city", "vehicle", "price", "completed", "distance", "end_date")
      .mkString(",")
  val updateFrag = fr"UPDATE trips SET (" 
    ++ Fragment.const(columnsWithComma) ++ fr") = "
}

There above two examples use the Doobie API. One is done via string interpolation, the other creates SQL query via SQL fragments.

Similar to HTTP routes, we use the implicit cats.effect.Sync monad to let the Doobie to wrap the result into F. The Doobie transactor instance is also parameterised with the target effect type F. Again, we will define the concrete type to be used for F in the main method of our application. Stay tuned.

Composition with IO monad

We are going to use the Cats IO monad when it comes to define our abstract F[_]. Our Scala program can be represented using below pseudo-code:

type F[A] = IO[A] 

main[F] ->  fs2.Stream[F, ExitCode] -> HTTP4s[F] -> DoobieRepository[F]

Assignment of _Cats_ `IO` to `F` is done in the main method.

Note that HTTP4s is based on the fs2 library, that is why, at some point, we construct a fs2.Stream instance to serve the HTTP requests. However, not much knowledge regarding the fs2 API is needed to implement HTTP servers that are similar to the example given here.

Main Program

The Cats-Effect library provides an IOApp trait, which leaves run method to be implemented. Its signature very similar to the standard Scala program main method, however we need to return a cats.effect.IO instance instead of a Unit type.

object Http4sMain extends IOApp {
  val (server, jdbc, _) = ... // loading configuration via case classes

  // main program, which is still of type F[_]
  def stream[F[_]: ConcurrentEffect: Applicative: ContextShift]
    : Stream[F, ExitCode] =

    for {
      mod <- Stream.eval(new Http4sModule(jdbc).pure[F])
      _ <- Stream.eval(mod.init())

      apiV1App = mod.routes.orNotFound
      finalHttpApp = Logger(logHeaders = true, logBody = true)(apiV1App)

      exitCode <- BlazeServerBuilder[F]
        .bindHttp(server.port.value, server.host.value)
        .withHttpApp(finalHttpApp)
        .serve
    } yield exitCode

  // main method of the IOApp, 
  // here we set the target effect type for the first time to IO[_] !
  override def run(args: List[String]): IO[ExitCode] = 
    stream[IO].compile.drain.as(ExitCode.Success)  
}

The main composition is done inside the stream method and based on for-comprehension of fs2.Stream type. There are three IO instances and one block of non-effectful code, which creates an instance of HTTP4s app. The first IO instance creates all the objects needed. At the moment, it does not do any IO action, but this may change when it starts to depend on the external world checks, for example checking database connection. The second IO calls the init method on the module to initialize the database schema. The third IO creates an instance of HTTP server and binds it to a particular user port and hostname.

The AkkaMain program consist of similar blocks, but relies on Scala Future at some point. See Akka-HTTP version here.

Summary

Pure-functional libraries as building blocks help to achieve the functional approach within the entire application. This brings the following benefits:

As a developer, one can see that a particular part of the program is doing IO action based on the type signature.
IO actions become composable when the effect type is a monad.
Referential transparency in the whole application helps us to reason locally and test program pieces in isolation.
Separation of IO code from the business logic code.
To represent the program effect type, an abstract generic type is used. This can greatly help when it comes to unit testing or while experimenting with different effect types. We might want to switch from Cats IO to something else. A change of a couple of lines makes this happen. However this benefit comes from generic programming, rather than functional paradigm itself.

Web-Links

Cats-Effect IO Data Type
HTTP4s site and documentation
Doobie microsite
Deep talk on Purely Functional I/O from Runar Bjarnson
Source code of the sample project

Footnotes

for an expression to be referentially transparent—in any program, the expression can be replaced by its result without changing the meaning of the program. And we say that a function is pure if calling it with RT arguments is also RT. (Functional Programming in Scala. Paul Chiusano and Runar Bjarnason) ↩
Effectful code is a code which is doing some side–effect to the external world. ↩
Object–Relational Mapping ↩
Functional–Relational Mapping ↩

Blog Post