Generic programming in Scala with Shapeless
In my last post about evolutionary computing, I mentioned I started building the project partly just so I could have a play with Shapeless. Shapeless is a generic programming library for Scala, which is growing in popularity - but can be fairly complicated to get started with.I had used Shapeless from a distance up until this point - using libraries like Circe or Spray-Json-Shapeless that use Shapeless under the hood to do stuff like JSON de/serialisation without boilerplate overhead of having to define all the different case classes/sealed traits that we want to serialise.
You can read the previous post for more details (and the code) if you want to understand exactly what the goal was, but to simplify, the first thing I needed to achieve was for a given case class, generate a brand new instance.
Now I wanted the client of the library to be able to pass in any case class they so wished, so I needed to be able to generically inspect all attributes of any given case class* and then decide how to generate a new instance for it.
Thinking about this problem in traditional JVM terms, you may be thinking you could use reflection - which is certainly one option, although some people still have a strong dislike for reflection in the JVM, and also has the downside that its not type safe - that is, if someone passes in some attribute that you don't want to support (a DateTime, UUID, etc) then it would have to be handled at runtime (which also means more exception handling code, as well as loosing the compile time safety). Likewise, you could just ask clients of the code to pass in a Map of the attributes, which dodges the need for reflection but still has no type safety either.
Enter Shapeless
Shapeless allows us to represent case classes and sealed traits in a generic structure, that can be easily inspected, transformed and put into new classes.A case class, product and HList walk into a bar
A case class can be thought of as a product (in the functional programming, algebraic data type sense), that is, case class Person(name: String, age: Int, living: Boolean) is the product of name AND age AND living - that is, every time you have an instance of that case class you will always have an instance of name AND age AND living. Great. So, as well as being a product, the case class in this example also carries semantic meaning in the type itself - that is, for this instance as well as having these three attributes we also know an additional piece of information that this is a Person. This is of course super helpful, and kinda central to the idea of a type system! But maybe sometimes, we want to be able to just generically (but still type safe!) say I have some code that doesn't care about the semantics of what something is, but if it has three attributes of String, Int, Boolean, then I can handle it - and that's what Shapeless provides - It provides a structure called a HList (a Heterogeneous List) that allows you to define a type as a list of different types, but in a compile time checked way.Rather that a standard List in Scala (or Java) where by we define the type that the list will contain, with a HList, we can define the list as specific types, for example:
val person: String :: Int :: Boolean :: HNil |
Case class to HList and back again
Ok, cool - we have a way to explicitly define very specific heterogeneous lists of types - how can this help us? We might not want to be explicitly defining these lists everywhere. Once again Shapeless steps in and provides a type class called Generic.The Generic type class allows conversion of a case class to a HList, and back again, for example:
val genericPerson = Generic[Person] | |
// will be the same type as previously defined | |
//String :: Int :: Boolean :: HNil |
val pCaseClass = Person("Jon", 17, true) | |
val pHlist = genericPerson.to(pCaseClass) | |
assert pCaseClass == genericPerson.from(pHlist) |
Back to the problem at hand
So, we now have the means to convert any given case class to a HList (that can be inspected, traversed etc) - which is great, as this means we can allow our client to pass in any case class and we just need to be able to handle the generation of/from a HList.To start with, we will define our type class for generation:
trait Generator[A] { | |
def generate: A | |
} |
Now, in the companion object we will define a helper method, that can be called simply without needing anything passed into scope, and the necessary Generator instance will be provided implicitly (and if there aren't any appropriate implicits, then we get our compile time error, which is our desired behaviour)
object Generator { | |
def generate[A](implicit gen: Generator[A]) = gen.generate | |
} |
Next, we need to define some specific generators, for now, I will just add the implicits to support some basic types:
implicit def intGenerator = new Generator[Int] { | |
override def generate = Random.nextInt | |
} | |
implicit def doubleGenerator = new Generator[Double] { | |
override def generate = Random.nextDouble | |
} | |
implicit def StringGenerator = new Generator[String] { | |
val loremWords = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.".split(" ") | |
override def generate = Random.shuffle(loremWords.toList).take(5).mkString(" ") | |
} | |
implicit def booleanGenerator = new Generator[Boolean] { | |
override def generate = Random.nextBoolean | |
} |
Now comes the part where our generate method can handle a case class being passed in, and subsequently the Shapeless HList representations of a case class (we will recursively traverse a HList structure using appropriate implict generators). First up, lets look at how we can implicitly handle a case class being passed in - at first this may start to look daunting, but its really not that bad.. honestly:
implicit def genericToGenerator[T, L <: HList](implicit generic: Generic.Aux[T, L], lGen: Generator[L]): Generator[T] = | |
new Generator[T] { | |
override def generate = generic.from(lGen.generate) | |
} |
Let's start with the base-case, as we saw earlier, HNil is the type for an empty HList so lets start with that one:
implicit def hnilGenerator = new Generator[HNil] { | |
override def generate = HNil | |
} |
Next we need to handle a generator for a non-empty HList:
implicit def hconsGenerator[H, T <: HList](implicit headGen: Generator[H], tailGen: Generator[T]) = | |
new Generator[H :: T] { | |
override def generate = headGen.generate :: tailGen.generate | |
} |
Now we can simply define a case class, pass it to our helper method and it will generate a brand new instance for us:
import Generator._ | |
case class Sample(a: Int, b: Boolean, c: Double, d: String) | |
generate[Sample] |
I was using this approach to generate candidates for my evolutionary algorithm, however the exact same approach could be used easily to generate test data (or much more). I used the same approach documented here to later transform instances of case classes as well.
Here is the completed code put together:
import org.scalatest.FunSpec | |
import shapeless.{HList, HNil, _} | |
import scala.util.Random | |
trait Generator[A] { | |
def generate: A | |
} | |
object Generator { | |
def generate[A](implicit gen: Generator[A]) = gen.generate | |
implicit def intGenerator = new Generator[Int] { | |
override def generate = Random.nextInt | |
} | |
implicit def doubleGenerator = new Generator[Double] { | |
override def generate = Random.nextDouble | |
} | |
implicit def StringGenerator = new Generator[String] { | |
val loremWords = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.".split(" ") | |
override def generate = Random.shuffle(loremWords.toList).take(5).mkString(" ") | |
} | |
implicit def booleanGenerator = new Generator[Boolean] { | |
override def generate = Random.nextBoolean | |
} | |
implicit def hnilGenerator = new Generator[HNil] { | |
override def generate = HNil | |
} | |
implicit def hconsGenerator[H, T <: HList](implicit headGen: Generator[H], tailGen: Generator[T]) = | |
new Generator[H :: T] { | |
override def generate = headGen.generate :: tailGen.generate | |
} | |
implicit def genericToGenerator[T, L <: HList](implicit generic: Generic.Aux[T, L], lGen: Generator[L]): Generator[T] = | |
new Generator[T] { | |
override def generate = generic.from(lGen.generate) | |
} | |
} | |
class TestSpec extends FunSpec { | |
describe("example") { | |
it("s") { | |
import Generator._ | |
case class Sample(a: Int, b: Boolean, c: Double, d: String) | |
println(generate[Sample]) | |
} | |
} | |
} |
0 comments: