As our running examples of polymorphic functions, let’s take the singleton function from last time (which given an argument of type T should return a single element Set[T] containing it), the headOption function (which given an argument of type List[T] gives us back its head as an Option[T]), the identity function (which returns its argument unchanged), and a generic size function which will compute an integer size appropriate to the type of its argument (eg. the size of a List or a String will be its length).

This is how we expect them to behave in the REPL,

scala> singleton("foo")
res0: Set[String] = Set(foo)

scala> identity(1.0)
res1: Double = 1.0

scala> headOption(List(1, 2, 3))
res2: Option[Int] = Some(1)

scala> size("foo")
res3: Int = 3

scala> size(List(1, 2, 3, 4))
res4: Int = 4

The function-like signatures for each of these are as follows,

  singleton    (∀T) T => Set[T]

  identity     (∀T) T => T

  headOption   (∀T) List[T] => Option[T]

  size         (∀T) T => Int

I say “function-like” here because, of course, Scala can’t directly express generic function value types of this form — that’s the problem we’re trying to solve.

Polymorphism lost, polymorphism regained

Recall from the preceeding article that the explanation for Scala’s function values being monomorphic is that the polymorphism of the FunctionN traits is fixed at the point at which they’re instantiated rather than the point at which they’re applied. This follows immediately from the position that their type parameters occur in their definition. For example, for Function1,

trait Function1[-T, +R] {
  def apply(t : T) : R
}

As you can see, the argument and result type parameters are declared at the trait level and hence are fixed for each invocation of the apply method.

The natural move at this point is try to shift the type parameters off Function1 and onto the apply method making it polymorphic independently of its enclosing trait — as we saw last time, the combination of polymorphic methods and call site eta-expansion gets us something that looks very much like a polymorphic function value.

We still want to be left with a first class type, values of which can be passed as arguments to higher-order functions, so we have to keep an enclosing type of some sort. A first naive pass at this might look something like,

trait PolyFunction1 {
  def apply[T, R](t : T) : R
}

But this won’t do at all, as we discover as soon as we try to implement it.

The problem we immediately run up against is that the result type of the apply method of our PolyFunction1 trait is completely unconstrained. But the signatures we’re trying to implement require that the result type be determined by the argument type (singleton, identity, headOption) or constant (size). There’s no way that we can map those signatures into the form required for the common trait.

Unconstrained result type parameters are in any case problematic when it comes to type inference, as I discussed in an earlier article, so let’s start by focussing on the singleton case where it’s easy to view the result type as a simple function of the argument type. This leads us to a second pass at a polymorphic function trait which captures that idea directly in terms of a higher-kinded trait-level type parameter — the higher-kinded type parameter is going act as a type-level function,

trait PolyFunction1[F[_]] {
  def apply[T](t : T) : F[T]
}

We can now define singleton as follows,

object singleton extends PolyFunction1[Set] {
  def apply[T](t : T) : Set[T] = Set(t)
}

and this behaves more or less as you’d expect — in particular, note the inferred result types,

scala> singleton(23)
res0: Set[Int] = Set(23)

scala> singleton("foo")
res1: Set[String] = Set(foo)

So far so good. Now, can we squeeze identity into the same mould? To do that we need to find a higher-kinded type for the F type-argument to PolyFunction1 such that F[T] = T — a type-level identity function in fact! Scala’s type aliases make such a type extremely straightforward to define — it’s just,

type Id[T] = T

Now we can define and apply the identity function like so,

object identity extends PolyFunction1[Id] {
  def apply[T](t : T) : T = t
}

scala> identity(23)
res0: Int = 23

scala> identity("foo")
res1: java.lang.String = foo

Next up is headOption. In this case we have a signature that has constraints on its argument type as well, not just on its result type as was the case for singleton and identity. Hopefully, though, it should be clear that we can repeat the same trick, and view both the argument type and the result type as functions of a common underlying type. This leads us to a third pass at the polymorphic function trait which this time has two higher-kinded trait-level type parameters — one to constrain the argument type and one to constrain the result type,

trait PolyFunction1[F[_], G[_]] {
  def apply[T](f : F[T]) : G[T]
}

And now we can define our first three functions as follows,

object singleton extends PolyFunction1[Id, Set] {
  def apply[T](t : T) : Set[T] = Set(t)
}

object identity extends PolyFunction1[Id, Id] {
  def apply[T](t : T) : T = t
}

object headOption extends PolyFunction1[List, Option] {
  def apply[T](l : List[T]) : Option[T] = l.headOption
}

scala> singleton("foo")
res0: Set[java.lang.String] = Set(foo)

scala> identity(1.0)
res1: Double = 1.0

scala> headOption(List(1, 2, 3))
res2: Option[Int] = Some(1)

That just leaves the size function. Handling that entails making the constant result type Int take the form of a higher-kinded type as well. We can do that with the help of a type lambda representing a type-level function from an arbitrary type T to some constant type C,

type Const[C] = {
  type λ[T] = C
}

For the particular case of type Const[Int], it is a structural type with a higher-kinded type member λ[_] which is equal to Int no matter what type argument it is applied to. So the type Const[Int]#λ[T] will be equal to type Int whatever type we substitute for T. Here’s short REPL session demonstrating that,

scala> implicitly[Const[Int]#λ[String] =:= Int]
res0: =:=[Int,Int] = 

scala> implicitly[Const[Int]#λ[Boolean] =:= Int]
res1: =:=[Int,Int] = 

This is a type-level rendering of the value-level constant function that you might also know as the K combinator from the SKI calculus (or as a “Kestrel” if you’re a Ray Smullyan fan),

def const[T](t : T)(x : T) = t

scala> val const3 = const(3) _
const3: Int => Int = 

scala> const3(23)
res6: Int = 3

With this in hand we can begin to define a size function that implements the same PolyFunction1 trait as singleton, identity and headOption,

object size extends PolyFunction1[Id, Const[Int]#λ] {
  def apply[T](t : T) : Int = 0
}

scala> size(List(1, 2, 3, 4))
res0: Int = 0

scala> size("foo")
res1: Int = 0

We have the signature right, at least, but what about the implementation of the apply method? Just returning a constant 0 isn’t particularly interesting. Unfortunately we don’t have much to work with — the use of Id on the argument side is what allows this function to be applicable to both Lists and Strings, but the direct consequence of that generality is that within the body of the method we have no knowledge about the type of the argument, so we have no immediate way of computing an appropriate result.

We can pattern match here of course, but as we’ll see that’s not a particularly desirable solution. For now let’s just go with that, and note a distinct lingering code smell,

object size extends PolyFunction1[Id, Const[Int]#λ] {
  def apply[T](t : T) : Int = t match {
    case l : List[_] => l.length
    case s : String  => s.length
    case _ => 0
  }
}

scala> size(List(1, 2, 3, 4))
res0: Int = 4

scala> size("foo")
res1: Int = 3

scala> size(23)
res2: Int = 0

A spoon full of sugar

Parenthetically, I’d like to flag up a small syntactic tweak that we can make to the PolyFunction1 trait which takes advantage of symbolic names in Scala and the ability to write types with two type arguments using an infix notation. The latter allows us to write types of the form T[X, Y] as X T Y. And if we choose the name T carefully this can give us a very syntactically elegant way of expressing the concept we’re trying to render.

Here we’re talking about the types of function-like things, so something which puns on Scala’s function arrow symbol => is a good choice — let’s use ~>. Our trait now looks like this,

trait ~>[F[_], G[_]] {
  def apply[T](f : F[T]) : G[T]
}

and our definitions look a lot more visibly function-like,

object singleton extends (Id ~> Set) {
  def apply[T](t : T) : Set[T] = Set(t)
}

object identity extends (Id ~> Id) {
  def apply[T](t : T) : T = t
}

object headOption extends (List ~> Option) {
  def apply[T](l : List[T]) : Option[T] = l.headOption
}

object size extends (Id ~> Const[Int]#λ) {
  def apply[T](t : T) : Int = t match {
    case l : List[_] => l.length
    case s : String  => s.length
    case _ => 0
  }
}

Function-like?

I’ve been careful to describe these things as “function-like” values rather than as functions to emphasize that they don’t and can’t conform to Scala’s standard FunctionN types. The immediate upshot of this is that they can’t be directly passed as arguments to any higher-order function which expects to receive an ordinary Scala function argument. For example,

scala> List(1, 2, 3) map singleton
:11: error: type mismatch;
 found   : singleton.type (with underlying type object singleton)
 required: Int => ?
              List(1, 2, 3) map singleton
                                ^

We can fix this however — whilst ~> can’t extend Function1, we can use an implicit conversion to do a job similar to the one that eta-expansion does for polymorphic methods,

implicit def polyToMono[F[_], G[_], T](f : F ~> G) : F[T] => G[T] = f(_)

This is along the right lines, but unfortunately due to a current limitation in Scala’s type inference this won’t work for functions like singleton that are parametrized with Id or Const because those types will never be inferred for F[_] or G[_]. We can help out the Scala compiler with a few additional implicit conversions to cover all the relevant permutations of those cases,

implicit def polyToMono2[G[_], T](f : Id ~> G) : T => G[T] = f(_)
implicit def polyToMono3[F[_], T](f : F ~> Id) : F[T] => T = f(_)
implicit def polyToMono4[T](f : Id ~> Id) : T => T = f[T](_)
implicit def polyToMono5[F[_], G, T](f : F ~> Const[G]#λ) : F[T] => G = f(_)
implicit def polyToMono6[G, T](f : Id ~> Const[G]#λ) : T => G = f(_)

With these in place we can map singleton over an ordinary Scala List,

scala> List(1, 2, 3) map singleton
res0: List[Set[Int]] = List(Set(1), Set(2), Set(3))

Natural transformations and their discontents

This encoding of polymorphic function values in Scala has been around for quite some time — in fact more or less from the point at which higher-kinded types arrived in the language. And, as a representation of a natural transformation, it’s been put to good use in scalaz.

So we’re done, right? Well, no not really. Whilst function-like values of this form are undoubtedly useful, they have a number of shortcomings which make them less than ideal in general. Let’s have a look at some of them now.

The first problem we saw earlier in the implementation of the size function,

object size extends (Id ~> Const[Int]#λ) {
  def apply[T](t : T) : Int = t match {
    case l : List[_] => l.length
    case s : String  => s.length
    case _ => 0
  }
}

Because the apply method’s type parameter T is completely unconstrained, the type of the argument t within the method body is effectively Any. In other words, the compiler knows nothing at all about its shape, specifically it can’t know that it has a length method yielding an Int.

We can pattern match to recover some type information as we’ve done above, but this is unsatisfactory for several reasons. First, we have to be careful to handle all cases or risk being hit by a MatchError — that forces us to include a possibly artifical default case, or take our chances at runtime. It’s also hopelessly non-modular — if we want to add cases for additional types then we have to modify this definition rather than adding orthogonal code to handle the new cases. Not good.

Second we have to be aware of the limitations of pattern matching in the face of type erasure. For example, suppose we had wanted to define the size of a List[String] as the sum of the lengths of its String elements. We might try something like,

object size extends (Id ~> Const[Int]#λ) {
  def apply[T](t : T) : Int = t match {
    case l : List[String] => l.map(_.length).sum
    case l : List[_] => l.length
    case s : String  => s.length
    case _ => 0
  }
}

We get a warning from this definition,

warning: non variable type-argument String in type pattern List[String] is
  unchecked since it is eliminated by erasure
           case l : List[String] => l.map(_.length).sum
                    ^

but let’s carry on regardless and try it out on the REPL,

scala> size(List("foo", "bar", "baz"))
res0: Int = 9

So far so good. But suppose we try with a list of non-Strings,

scala> size2(List(1, 2, 3))
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String

Oops! — that unchecked warning was telling us that Scala’s pattern matching runtime infrastructure (or rather, the parts of the JVM’s runtime infrastructure which support Scala’s pattern matching) is unable to verify the types of a List‘s elements because the element type is erased at runtime. Consequently the first case, List[String], is always selected, and this fails at runtime if handed a list of anything other than Strings. Also not good.

The conclusion to draw from this is that implementing a polymorphic function of this form via pattern matching is unworkable if we want type safety, modularity or type-specific cases which are distinguished by particular type arguments of a common type constructor.

If pattern matching is ruled out, then what can we do? Well, we can do anything which doesn’t depend on the shape of T. That’s trivially the case for the identity function — it simply returns it’s argument unexamined. And it’s also the case for methods which are themselves polymorphic in the the same way that our apply method is. That’s what’s happening in the definition of singleton. Here it is again with a little less syntactic sugar,

object singleton extends (Id ~> Set) {
  def apply[T](t : T) : Set[T] = Set.apply[T](t)
}

The apply method (parametric in T) is implemented in terms of the Set companion object’s apply method (also parametric in T). No information is needed about the shape of T here because the Set factory method doesn’t need to examine its arguments, it just needs to wrap them in a fresh container.

Things like headOption are also fine, this time because we do have a constraint on the type of the argument to the apply method — here we know that the outer type constructor is List[_]. That means that we can implement the method in terms of any methods defined on List which don’t themselves need to know anything about the shape of T (this excludes methods like sum and product for example).

Clearly this gives us quite a bit to work with, and if you can manage with the constraints that parametricity imposes, then it’s definitely the way to go. But it’s a real constraint with sharp teeth — we won’t be able to implement polymorphic functions like size in this way.

The fact that our problems are caused by lacking information about the shape of T might encourage us to explore the option of modifying the ~> trait to add bounds on T (ordinary type bounds or view or context bounds). This is an interesting exercise to attempt, but ultimately it adds a lot of complexity without solving the problem fully generally. I encourage you to give it a try — if you do, you’ll very quickly find yourself wanting to be able to abstract over function signatures more generally, and that’s going to take us in a different direction as we’ll see in the next part of this series.

scala> import shapeless._ ; import HList._

scala> List(1, 2, 3) map singleton                    // List[Int]
res0: List[Set[Int]] = List(Set(1), Set(2), Set(3))

scala> List("foo", "bar", "baz") map singleton        // List[String]
res1: List[Set[String]] = List(Set(foo), Set(bar), Set(baz))

scala> (23 :: "foo" :: false :: HNil) map singleton   // HList
res2: Set[Int] :: Set[String] :: Set[Boolean] :: HNil =
      Set(23)  :: Set(foo)    :: Set(false)   :: HNil

(here singleton() is a function which makes a single element Set from its argument).

Arranging for things to work out this smoothly involves quite a bit of hidden sophistication, and in this short series of articles I’ll explain what that sophistication is and why it’s necessary.

The first observation to make about the REPL transcript above is that the singleton() function is an argument to the map() higher-order functions defined for List and HList. This is part and parcel of Scala making good on its claim to blend object-oriented and functional programming styles by representing functions as values.

The second observation to make is that this function is being applied to arguments of several different types — Ints, Strings, and Booleans. In other words, the function is polymorphic in its argument type.

Given that there’s nothing at all unusual looking about the first two maps above (over the vanilla Lists) you might put these two observations together and quite reasonably conclude that Scala directly supports polymorphic function values.

That’s not the case, however. Appearances to the contrary, all Scala values (and hence Scala function values in particular) are monomorphic (in the sense relevant here, see qualifications below), and representing polymorphic functions requires some work.

Before we get to that, we need to understand method-level parametric polymorphism in Scala. And we need to understand the differences between between Scala’s methods and Scala’s function values. With that in hand we’ll be able to see why Scala’s standard function values can only be monomorphic. That will set the stage for our exploration of ways to encode polymorphic function values in parts 2 and 3 of this series.

Method-level parametric polymorphism

The best way to get to grips with method-level parametric polymorphism is to compare the definitions and uses of a monomorphic and a polymorphic method. At the point of definition the constrast is simply between those methods with arguments of fixed types and those with arguments which have a type that is bound by a method-level type parameter. So, for example,

// Monomorphic methods have type parameter-free signatures
def monomorphic(s : String) : Int = s.length

monomorphic("foo") 

// Polymorphic methods have type parameters in their signatures
def polymorphic[T](l : List[T]) : Int = l.length

polymorphic(List(1, 2, 3))
polymorphic(List("foo", "bar", "baz"))

Monomorphic methods can only be applied to arguments of the fixed types specified in their signatures (and their subtypes, I’ll come back to this in a moment), whereas polymorphic methods can be applied to arguments of any types which correspond to acceptable choices for their type parameters — in the example just given we can apply monomorphic() to Strings only, but we can apply polymorphic() to values of type List[Int] or List[String] or … List[T] for any type T.

Of course, Scala is both an object-oriented and a functional programming language, so as well as parametric polymorphism (ie. polymorphism captured by type parameters) it also exhibits subtype polymorphism. That means that the methods that I’ve been calling monomorphic are only monomorphic in the sense of parametric polymorphism and they can in fact be polymorphic in the traditional object-oriented way. For instance,

trait Base { def foo : Int }
class Derived1 extends Base { def foo = 1 }
class Derived2 extends Base { def foo = 2 }

def subtypePolymorphic(b : Base) = b.foo

subtypePolymorphic(new Derived1) // OK: Derived1 <: Base
subtypePolymorphic(new Derived2) // OK: Derived2 <: Base

Here the method subtypePolymorphic() has no type parameters, so it's parametrically monomorphic. Nevertheless, it can be applied to values of more than one type as long as those types stand in a subtype relationship to the fixed Base type which is specified in the method signature — in other words, this method is both parametrically monomorphic and subtype polymorphic.

It's parametric polymorphism that I'll be talking about in the remainder of this article and the sequel — I won't be mentioning subtype polymorphism again, and from now on I'll just talk about methods and functions being monomorphic or polymorphic without a "parametric" qualifier.

Methods vs. function values

So far I've been talking about Scala methods ... now we need to understand how they differ from Scala function values. Scala methods are exactly like Java methods: they're components of classes or traits and aren't first-class values in their own right. Scala function values, on the other hand, are first-class values and are represented by JVM-level classes rather than being components of some other class. It's the first-class value nature of Scala function values which allows them to be passed as arguments to the higher-order functions and methods which give Scala it's functional flavour.

Nevertheless, Scala methods can have a very function-like feel, particularly when they're defined on Scala objects used as modules, or nested inside other method or function definitions. In these cases they appear to be free floating, lacking the implicit left-hand-side "receiver" argument characteristic of object-oriented methods. And in many cases they can be used in applications of higher-order functions without any visible additional ceremony. For example,

scala> object Module {
     |   def stringSingleton(s : String) = Set(s)
     | }
defined module Module

scala> import Module._
import Module._

scala> stringSingleton("foo")
res0: Set[String] = Set(foo)

scala> List("foo", "bar", "baz") map stringSingleton
res1: List[Set[String]] = List(Set(foo), Set(bar), Set(baz))

The stringSingleton() method of the Module object appears to be indistinguishable from a first-class function value. But the appearances are deceptive. The method isn't free-standing: we could have used this in its body and it would have referred to the Module singleton object, even after the import. And it's not the method which is passed to map — instead a transient function value is implicitly created to invoke the stringSingleton() method (this is a process known as eta-expansion) and it's that function-value which is passed to map.

Fortunately these mechanics are completely invisible most of the time, but they're relevant to us now, so let's make them a bit more visible. We can ask explicitly for the Scala compiler to give us the eta-expanded function value, allowing us to give it a name and discover its type. We do this using Scala's multipurpose '_' — in this manifestation it's acting as a function-value-forming operator,

scala> val stringSingletonFn = stringSingleton _
stringSingletonFn: (String) => Set[String] = 

scala> stringSingletonFn("foo")
res2: Set[String] = Set(foo)

scala> List("foo", "bar", "baz") map stringSingletonFn
res3: List[Set[String]] = List(Set(foo), Set(bar), Set(baz))

This sequence is exactly equivalent to what we had before, but now we can see that a new function value of type (String) => Set[String] has been created — it's this which is passed to map.

In this instance both the method and the eta-expanded function value are monomorphic. Let's see what happens if we try to do the same thing with a polymorphic method,

scala> def singleton[T](t : T) = Set(t)
singleton: [T](t: T)Set[T]

scala> singleton("foo")
res4: Set[java.lang.String] = Set(foo)

scala> singleton(23)
res5: Set[Int] = Set(23)

So far so good — our method can be applied at arbitrary types as expected. Now let's try explicitly eta-expanding this, as we did in the monomorphic case,

scala> val singletonFn = singleton _
singletonFn: (Nothing) => Set[Nothing] = 

scala> singletonFn("foo")
:14: error: type mismatch;
 found   : java.lang.String("foo")
 required: Nothing
       singletonFn("foo")
                   ^

scala> singletonFn(23)
:14: error: type mismatch;
 found   : Int(23)
 required: Nothing
       singletonFn(23)
                   ^

Ouch! Something's gone badly wrong here. Look at the type inferred for singletonFn! In the earlier monomorphic case, the type of our eta expanded function was quite straightforward and unsurprising: (String) => Set[String]. But here we have (Nothing) => Set[Nothing] — where has this Nothing come from? And what's happened to the polymorphism of the underlying method? To see what's going on we'll need to dig a little deeper into Scala's function types and how function values are represented.

Scala function types

One of the joys of Scala's mixed object-functional design is that there's nothing particularly special about function types: (String) => Set[String] is simply syntactic sugar for one of a family of FunctionN traits, one for each function arity between 0 and (somewhat arbitrarily) 22. In this particular case we have a function with a single argument, so the function value is an instance of the Function1 trait. Simplifying a little, that trait and the implementation of the stringSingletonFn instance of it looks like this,

trait Function1[-T, +R] {
  def apply(v : T) : R
}

val stringSingletonFn = new Function1[String, Set[String]] {
  def apply(v : String) : Int = Module.stringSingleton(v)
}

The crucial thing to notice is that the function's argument and result type parameters are all declared at the trait level. This has two consequences. The first is that when we create an instance of Function1 we (or the compiler) must choose the argument and result types at that point. In the eta-expansion expression above we haven't explicitly specified the argument type, so the compiler is left to infer it, and since there is no useful information for it to work with, it fills the argument type in as Nothing.

The second is an immediate consequence of the first — because we (or the compiler) must choose the argument and result types at the point at which the function value is created they are fixed, once and for all, from that point on. This means that even if we were to eliminate the Nothings by specifying an argument type we would still have a problem,

scala> val singletonFn : String => Set[String] = singleton _
singletonFn: (String) => Set[String] = 

scala> singletonFn("foo")
res6: Set[String] = Set(foo)

scala> singletonFn(23)
:14: error: type mismatch;
 found   : Int(23)
 required: String
       singletonFn(23)
                   ^

Having specified that we want our function value instantiated to accept String arguments we are stuck with that choice forever after. Or, in other words, we have completely lost the polymorphism of the underlying method.

Returning to our example from the introduction — where we have ordinary Scala Lists, if we take the above polymorphic method definition of singleton() we get an appearance of polymorphic function values,

scala> def singleton[T](t : T) = Set(t)
singleton: [T](t: T)Set[T]

scala> List(1, 2, 3) map singleton             // eta-expanded to Int => Set[Int]
res0: List[Set[Int]] = List(Set(1), Set(2), Set(3))

scala> List("foo", "bar", "baz") map singleton // eta-exp'd to String => Set[String] 
res1: List[Set[String]] = List(Set(foo), Set(bar), Set(baz))

This is because the polymorphic method is eta-expanded each time it's used as an argument to List's map(), first with T instantiated as Int and then with it instantiated as String. But in both cases the resulting function value has a fixed argument type, ie. is monomorphic. And that's just fine, because each particular List only has elements of one fixed type as well.

But now consider the HList case — here we have only one application of map(), hence we only have one opportunity for eta-expansion. This can only deliver one standard Scala function value, and that function value can be applied to arguments of just one fixed type. But we need more than one ... Oops!

So here we have the crux of the problem — we can have first-class monomorphic function values and we can have second-class polymorphic methods, but we can't have first-class polymorphic function values ... at least we can't with the standard Scala definitions. And yet it's first-class polymorphic function values that we need to map over an HList ... what to do?

In the next article in this series I'll start explaining how we can get the best of both worlds ...

Selection sort is just about the simplest possible sorting algorithm — select the least element from your unsorted list as the first element of the sorted list, then recursively sort the remainder — so I’ll take it as a given, and focus on the interesting bit … how can this possibly be done at the type-level?

The first thing we need is a type-level representation of lists of sortable things. For that, we’re going to use HLists of type-level natural numbers.

import shapeless._
import Nat._
import HList._

def typed[T](t : => T) {}

val unsorted = _3 :: _1 :: _4 :: _0 :: _2 :: HNil
typed[_3 :: _1 :: _4 :: _0 :: _2 :: HNil](unsorted)

Here unsorted is a value-level list of Nat‘s, and we can see that its structure is exactly mirrored in its type — in shapeless each Nat value (_0, _1, …) has a corresponding Nat type with the same name (_0, _1, …). We use the typed function here to verify the type of unsorted rather than use a type annotation on its val declaration to ensure that the act of verifying the inferred type doesn’t itself contribute to type inference.

Now we need a way of capturing order over these type-level Nat‘s. We do that using a type class that witnesses the relationship at the type level,

trait LTEq[A <: Nat, B <: Nat]

object LTEq {
  import Nat._0

  type <=[A <: Nat, B <: Nat] = LTEq[A, B]

  implicit def ltEq1 = new <=[_0, _0] {}
  implicit def ltEq2[B <: Nat] = new <=[_0, Succ[B]] {}
  implicit def ltEq3[A <: Nat, B <: Nat](implicit lt : A <= B) =
    new <=[Succ[A], Succ[B]] {}
}

If you're familiar with Peano arithmetic it should be pretty clear how this works — we have two base cases: _0 is less than or equal to _0, and _0 is less than or equal to the successor of any number; and we have one induction case: if A <= B then A+1 <= B+1. With these definitions in place, the magic of Scala's implicit resolution allows us to ask the compiler about relationships between different Nat types,

scala> import shapeless._ ; import Nat._ ; import LTEq._
import shapeless._
import Nat._
import LTEq._

scala> implicitly[_2 <= _5] // OK
res0: shapeless.LTEq.<=[shapeless.Nat._2,shapeless.Nat._5] =
  shapeless.LTEq$$anon$5@4fc8927c

scala> implicitly[_4 <= _2] // Does not compile
:17: error: could not find implicit value for parameter 
  e: shapeless.LTEq.<=[shapeless.Nat._4,shapeless.Nat._2]
    implicitly[_4 <= _2]
              ^

Before we go any further, let's define an operation which witnesses whether or not an HList of Nats is in sorted order — we'll want this to verify that our type-level sort is correct,

trait NonDecreasing[L <: HList]
implicit def hnilNonDecreasing = new NonDecreasing[HNil] {}
implicit def hlistNonDecreasing1[H] = new NonDecreasing[H :: HNil] {}
implicit def hlistNonDecreasing2[H1 <: Nat, H2 <: Nat, T <: HList]
  (implicit ltEq : H1 <= H2, ndt : NonDecreasing[H2 :: T]) =
    new NonDecreasing[H1 :: H2 :: T] {}
      
def acceptNonDecreasing[L <: HList](l : L)(implicit ni : NonDecreasing[L]) = l
  
// Verify type-level relations
implicitly[NonDecreasing[_1 :: _2 :: _3 :: HNil]]   // OK
implicitly[NonDecreasing[_1 :: _3 :: _2 :: HNil]]   // Does not compile
      
// Apply at the value-level
acceptNonDecreasing(_1 :: _2 :: _3 :: HNil)         // OK
acceptNonDecreasing(_1 :: _3 :: _2 :: HNil)         // Does not compile  

This is a fairly straighforward induction — we have two base cases: an empty list is in sorted order, as is a list of exactly one element; and we have one induction case: a list is in sorted order if its tail is, and if its first element is less than or equal to its second element.

Now we can define an operation to select the least element from an HList of Nats, returning both it and the remainder of the list,

trait SelectLeast[L <: HList, M <: Nat, Rem <: HList] {
  def apply(l : L) : (M, Rem)
}
  
trait LowPrioritySelectLeast {
  implicit def hlistSelectLeast1[H <: Nat, T <: HList] =
    new SelectLeast[H :: T, H, T] {
      def apply(l : H :: T) : (H, T) = (l.head, l.tail)
    }
}
  
object SelectLeast extends LowPrioritySelectLeast {
  implicit def hlistSelectLeast3[H <: Nat, T <: HList, TM <: Nat, TRem <: HList]
    (implicit tsl : SelectLeast[T, TM, TRem], ev : TM < H) =
      new SelectLeast[H :: T, TM, H :: TRem] {
        def apply(l : H :: T) : (TM, H :: TRem) = {
          val (tm, rem) = tsl(l.tail) 
          (tm, l.head :: rem)
        }
      }
}

val (l1, r1) = selectLeast(_1 :: _2 :: _3 :: HNil)
typed[_1](l1)
typed[_2 :: _3 :: HNil](r1)
  
val (l2, r2) = selectLeast(_3 :: _1 :: _4 :: _0 :: _2 :: HNil)
typed[_0](l2)
typed[_3 :: _1 :: _4 :: _2 :: HNil](r2)

Here're we're working at both the type-level and at the value-level — we're selecting the least value-level Nat from each of the examples HLists, and these are being assigned the corresponding Nat types. Hopefully the recursive pattern is beginning to look fairly familiar, even in this slightly more complicated case — we have a base case: the least element of a singleton list is its only element; and the least element of a non-singleton list is the least element of its tail unless its head is already the least element of the entire list.

And now we can sort!

trait SelectionSort[L <: HList, S <: HList] {
  def apply(l : L) : S
}
  
trait LowPrioritySelectionSort {
  implicit def hlistSelectionSort1[S <: HList] = new SelectionSort[S, S] {
    def apply(l : S) : S = l
  }
}
  
object SelectionSort extends LowPrioritySelectionSort {
  implicit def hlistSelectionSort2[L <: HList, M <: Nat, Rem <: HList, ST <: HList]
    (implicit sl : SelectLeast[L, M, Rem], sr : SelectionSort[Rem, ST]) =
      new SelectionSort[L, M :: ST] {
        def apply(l : L) = {
          val (m, rem) = sl(l)
          m :: sr(rem)
        }
      }
}
  
def selectionSort[L <: HList, S <: HList](l : L)
  (implicit sort : SelectionSort[L, S]) = sort(l)

val unsorted = _3 :: _1 :: _4 :: _0 :: _2 :: HNil
typed[_3 :: _1 :: _4 :: _0 :: _2 :: HNil](unsorted)
acceptNonDecreasing(unsorted)  // Does not compile!
  
val sorted = selectionSort(unsorted)
typed[_0 :: _1 :: _2 :: _3 :: _4 :: HNil](sorted)
acceptNonDecreasing(sorted)    // Compiles!     

As you can see, the static type of the sorted value reflects the fact that it is in sorted order, and that consequently, unlike the initial unsorted value, it has the correct type to be passed to the acceptNonDecreasing function that we wrote earlier.

Update — Joni Freeman has done a nice job of porting the algorithm used to Prolog here.

If you've made it this far, and want to play around with it yourself, you can find shapeless on github along with the example source this article is based on. You'll also find a mailing list for discussion around shapeless here. Oh, and if you're going to be at Nescala then I'll see you there ...

In advance of that, I’d also be keen to talk to any Scala-using organizations who would be interested in a two/three day Advanced Scala training course on the East coast in the run up to, or just after the event — in depth coverage of the techniques involved in the creation of shapeless, and how they can be applied elsewhere, will be included. Please get in touch if you’re interested.

In the new year I’ll be posting a (long overdue) series of articles here on the implementation techniques used — heavily type class based, with essential use of dependent types at various junctures, which together enable the relatively smooth encoding of type level functions. In the meantime you’ll find Olivera, Moors and Odersky Type Classes as Objects and Implicits useful background material.

Selected highlights of shapeless include …

• A new encoding of polymorphic function values which optionally supports type specific cases, and which is interoperable with Scala’s ordinary monomorphic function values.

  // choose is a function from Sets to Options with no type specific cases
  object choose extends (Set ~> Option) {
    def default[T](s : Set[T]) = s.headOption 
  }
  
  // choose is convertible to an ordinary monomorphic function value
  val lo = List(Set(1, 3, 5), Set(2, 4, 6)) map choose
  lo == List(Option(1), Option(2))
  
  // size is a function from values of arbitrary type to a 'size' which is
  // defined via type specific cases
  object size extends (Id ~> Const[Int]#λ) {
    def default[T](t : T) = 1
  }
  implicit def sizeInt = size.λ[Int](x => 1)
  implicit def sizeString = size.λ[String](s => s.length)
  implicit def sizeList[T] = size.λ[List[T]](l => l.length)
  implicit def sizeOption[T](implicit cases : size.λ[T]) =
    size.λ[Option[T]](t => (t map size).getOrElse(0))
  implicit def sizeTuple[T, U](implicit st : size.λ[T], su : size.λ[U]) =
    size.λ[(T, U)](t => size(t._1)+size(t._2))
  
  size(23) == 1
  size("foo") == 3
  size((23, "foo")) == 4
  

• An implementation of Scrap your Boilerplate with Class which provides generic map and fold operations over arbitrarily nested data structures,

  val nested = List(Option(List(Option(List(Option(23))))))
      
  val succ = everywhere(inc)(nested)
  succ == List(Option(List(Option(List(Option(24))))))
  

• A Typeable type class which provides a type safe cast operation,

  val a : Any = List(Vector("foo", "bar", "baz"), Vector("wibble"))
  
  val lvs : Option[List[Vector[String]]] = a.cast[List[Vector[String]]]
  lvs.isDefined == true
  
  val lvi : Option[List[Vector[Int]]] = a.cast[List[Vector[Int]]]
  lvi.isEmpty == true
  

• The mother of all Scala HList‘s which, amongst other things, is covariant,

  trait Fruit
  case class Apple() extends Fruit
  case class Pear() extends Fruit
      
  type FFFF = Fruit :: Fruit :: Fruit :: Fruit :: HNil
  type APAP = Apple :: Pear :: Apple :: Pear :: HNil
      
  val a : Apple = Apple()
  val p : Pear = Pear()
      
  val apap : APAP = a :: p :: a :: p :: HNil
  val ffff : FFFF = apap  // APAP <: FFFF 
  

  It has a map operation, applying a polymorphic function value (possibly with type specific cases) across its elements. This means that it subsumes both typical HList's and also KList's (HList's whose elements share a common outer type constructor),

  type SISS = Set[Int] :: Set[String] :: HNil
  type OIOS = Option[Int] :: Option[String] :: HNil
      
  val sets : SISS = Set(1) :: Set("foo") :: HNil
  val opts : OIOS = sets map choose
  opts == Option(1) :: Option("foo") :: HNil 
  

  It has a zipper for traversal and persistent update,

  val l = 1 :: "foo" :: 3.0 :: HNil
  
  val l2 = l.toZipper.right.put("wibble", 45).toHList
  l2 == 1 :: ("wibble", 45) :: 3.0 :: HNil
    
  val l3 = l.toZipper.right.delete.toHList
  l3 == 1 :: 3.0 :: HNil
  
  val l4 = l.toZipper.last.left.insert("bar").toHList
  l4 == 1 :: "foo" :: "bar" :: 3.0 :: HNil
  

  It has a unify operation which converts it to an HList of elements of the least upper bound of the original types,

  val ffff = apap.unify // type inferred as FFFF
  

  It supports conversion to an ordinary Scala List of elements of the least upper bound of the original types,

  val lf = apap.toList  // type inferred as List[Fruit]
  lf == List(a, p, a, p)
  

  And it has a Typeable type class instance, allowing, eg. vanilla List[Any]'s or HList's with elements of type Any to be safely cast to precisely typed HList's,

  val ffff : FFFF = apap.unify               // discard precise typing 
  val apap2 : Option[APAP] = ffff.cast[APAP] // reestablish precise typing
  apap2.get == apap  
  

  These last three features make this HList dramatically more practically useful than HList's are typically thought to be — normally the full type information required to work with them is too fragile to cross subtyping or I/O boundaries. This implementation supports the discarding of precise information where necessary (eg. to serialize a precisely typed record after construction), and its later reconstruction (eg. a weakly typed deserialized record with a known schema can have it's precise typing reestabilished).

• Conversions between tuples and HList's, and between ordinary Scala functions of arbitrary arity and functions which take a single corresponding HList argument,

  // Round trip from tuple to HList and back
  val t1 = (23, "foo", 2.0, true)
      
  val l1 = t1.hlisted
  h1 == 23 :: "foo" :: 2.0 :: true :: HNil
      
  val t2 = l1.tupled
  t1 == t2
  

  One application of this is the liftO function which lifts an ordinary function of arbitrary arity into Option.

  // Lift these ordinary Scala function values into Option 
  val sum : (Int, Int) => Int = _ + _
  val prd : (Int, Int, Int) => Int = _ * _ * _
      
  // Nb. liftO abstracts over the arity of its function arguments 
  val sumO = liftO(sum) // (Option[Int], Option[Int]) => Option[Int]
  val prdO = liftO(prd) // (Option[Int], Option[Int], Option[Int]) => Option[Int]
  
  val s1 = sumO(Some(1), Some(2))
  s1 == Option(3)
  
  val s2 = sumO(Some(1), None)
  s2 == None
  
  val p1 = prdO(Some(2), Some(3), Some(4))
  p1 == Option(24)
  
  val p2 = prdO(Some(2), None, Some(4))
  p2 == None
  

The library is targetted at Scala 2.10-SNAPSHOT by default, but currently should build against Scala 2.9.1.final with the -Ydependent-method-types switch enabled. I make no promises that it'll continue to build with 2.9.x, or even vanilla Scala 2.10-SNAPSHOT — in 2012 I plan to investigate, amongst other things, what can be done with singleton types for literal values — at a minimum they would make the clunky encoding of type level natural numbers redundant, and might enable a whole lot more.

Fork it, kick the tires over the holiday and let me know how you get on!
]]> http://www.chuusai.com/2011/12/19/shapeless-preview/feed/ 14 http://www.chuusai.com/2011/07/16/fundeps-in-scala/ http://www.chuusai.com/2011/07/16/fundeps-in-scala/#comments Sat, 16 Jul 2011 17:02:24 +0000 Miles Sabin http://www.chuusai.com/?p=103 Continue Reading]]> Functional dependencies are a near-standard extension to Haskell (present in GHC and elsewhere) which allow constraints on the type parameters of type classes to be expressed and then enforced by the compiler. They allow the programmer to state that some of the type parameters of a multi-parameter type class are completely determined by the others. One particularly common application of this is allowing the result type of a function to be a parameter but nevertheless determined by the type(s) of it’s argument(s) — the examples on the Haskell wiki page that I linked to above illustrate that extremely well.

Does this have an equivalent in Scala? We certainly have the equivalent problem — whenever we have multiple type parameters (of a type or method) we might want to be able to express functional dependency-like constraints between those type parameters and have those constraints checked by the compiler. And ideally we want that to get along well with type-inference — we would like the compiler to be able to infer any types which are determined by types which it is already able to infer.

It turns out that there’s a very simple translation of Haskell-style functional dependencies to Scala, but for some reason this doesn’t seem to have been commented on. Quite the opposite in fact — I’ve seen it reported that encoding fundeps in Scala is either impossible, or possible but not useful because of issues with type inference. What makes this even more surprising is that a crucial feature of Scala’s (now not so) new collections framework (CanBuildFrom) appears to be a perfect example of this encoding in practice … strangely this isn’t advertised in Type Classes as Objects and Implicits (Olivera, Odersky and Moors, 2010) either.

Maybe this has all been noticed before and assumed to be too obvious to mention, nevertheless, I think it’s useful to be able to make the connection. I’ll illustrate the translation using the examples from the Haskell wiki. By the time I’m done translating them into Scala the fact that CanBuildFrom is an instance of a fundep should be quite clear.

Let’s start with the matrix/vector/scalar multiplication example. Here we want to express the fact that the result type of the multiplication is fully determined by it’s arguments,

Here we have a triple of types where the first two (the argument types) determine the third (the result type). Lets express this relationship directly via a trait with three type parameters accompanied by some implicit definitions which capture the relationship we want to enforce,

trait Matrix // Dummy definitions for expository purposes
trait Vector

trait MultDep[A, B, C]
// trait MultDep[-A, -B, +C]    // for Scala <= 2.9.0-1 these variance
//                              // annotations are required
  
implicit object mmm extends MultDep[Matrix, Matrix, Matrix]
implicit object mvv extends MultDep[Matrix, Vector, Vector]
implicit object mim extends MultDep[Matrix, Int, Matrix]
implicit object imm extends MultDep[Int, Matrix, Matrix]

Given these definitions we can ask the compiler what is and what isn't allowed,

scala> implicitly[MultDep[Matrix, Matrix, Matrix]] // OK: Matrix * Matrix -> Matrix
res0: MultDep[Matrix,Matrix,Matrix] = mmm$@1ddeda2

scala> implicitly[MultDep[Matrix, Vector, Vector]] // OK: Matrix * Vector -> Vector
res1: MultDep[Matrix,Vector,Vector] = mvv$@1a8fb1b

scala> implicitly[MultDep[Matrix, Vector, Matrix]] // Error
:15: error: could not find implicit value for 
  parameter e: MultDep[Matrix,Vector,Matrix]
       implicitly[MultDep[Matrix, Vector, Matrix]]

The first two cases are fine, because they both correspond to one of the implicit instances we provided earlier. The third fails, quite rightly, because there is no instance corresponding to Matrix * Vector -> Matrix.

So, we've been able to capture the desired relationship between the type variables. Here's how we put that to use,

def mult[A, B, C](a : A, b : B)(implicit instance : MultDep[A, B, C]) : C =
  error("TODO")

// Type annotations soley to verify that the correct result type has been inferred
val r1 : Matrix = mult(new Matrix {}, new Matrix{}) // Compiles
val r2 : Vector = mult(new Matrix {}, new Vector{}) // Compiles
val r3 : Matrix = mult(new Matrix {}, 2)            // Compiles
val r4 : Matrix = mult(2, new Matrix {})            // Compiles

// This next one doesn't compile ...
val r5 : Matrix = mult(new Matrix {}, new Vector{}) 

Notice how the third type parameter, C, isn't used in the first (explicit) parameter list. If it weren't for its appearance in the second (implicit) parameter list the compiler would normally infer it as Nothing — not what we want at all. But the combination of type inference and implicit search saves the day. The first two type parameters A and B are inferred from the explicit arguments to mult(), then implicit search kicks in, attempting to locate an implicit definition of MultDep[_, _, _] consistent with those two types. By construction, it will only ever find one, and that is sufficient to uniquely determine C for use as the result type of the function.

We have the signature of our mult() function sorted out, now for the implementation. The (hopefully fairly obvious) trick here is to make our MultDep instances provide the implementations of the different cases as well as the type constraints,

implicit object mmm extends MultDep[Matrix, Matrix, Matrix] {
  def apply(m1 : Matrix, m2 : Matrix) : Matrix = error("TODO")
}

implicit object mvv extends MultDep[Matrix, Vector, Vector] {
  def apply(m1 : Matrix, v2 : Vector) : Vector = error("TODO")
}

implicit object mim extends MultDep[Matrix, Int, Matrix] {
  def apply(m1 : Matrix, i2 : Int) : Matrix = error("TODO")
}

implicit object imm extends MultDep[Int, Matrix, Matrix] {
  def apply(i1 : Int, m2 : Matrix) : Matrix = error("TODO")
}

(filling out the bodies of the dummy apply() methods is an exercise left for the reader). We can now complete the definition of mult(),

def mult[A, B, C](a : A, b : B)(implicit instance : MultDep[A, B, C]) : C = 
  instance(a, b)

And that's all there is to it. The implicit objects correspond exactly to the type class instances in the Haskell original, and the combination of type inference and implicit search captures the functional dependency directly.

Here's the second example from Haskell wiki page, translated in the same way,

trait ExtractDep[A, B] {
  def apply(a : A) : B
}
  
implicit def ep[A, B] = new ExtractDep[(A, B), A] {
  def apply(p : (A, B)) : A = p._1
}
  
def extract[A, B](a : A)(implicit instance : ExtractDep[A, B]) : B = instance(a)

// Type annotation soley to verify that the correct result type has been inferred
val c : Char = extract(('x', 3))

Of course, neither of these two examples are particularly exciting in Scala as they stand — Scala has Java-style overloading, which makes the matrix/vector/scalar example mostly redundant; and the second example is a little too artificial to be compelling. Nevertheless, there are many situations where being able to express similar constraints amongst type parameters can be extremely useful — Scala embedded DSLs often have to deal with situations which are equivalent to wanting a function result type to be determined by its argument types, and I know of a number of DSL designers who've struggled with the problem of unhelpful Nothings being inferred for unconstrained result types.

But the best widely known example of a fundep is right under our noses in the Scala standard library itself: CanBuildFrom. Let's take a look at (a slight simplification of) of the part of its definition which expresses type constraints,

trait CanBuildFrom[From, Elem, To] {
  def apply ...
}

Instances of this trait capture collection-type-specific constraints on whether or not a collection of type To with elements of type Elem can be created from a collection of type From. For "regular" collection types (collections which can hold elements of any type) the corresponding instance is equivalent to,

implicit def regularCanBuildFrom[CC[_], A] = new CanBuildFrom[CC[_], A, CC[A]] {
  def apply ...
}

In effect this asserts that for a regular collection type it's possible to create a new collection of the same shape, but with an arbitrarily substituted element type. We can ask the compiler about these relationships like so,

scala> import scala.collection.generic._
import scala.collection.generic._

scala> implicitly[CanBuildFrom[List[Int], String, List[String]]]
res0: scala.collection.generic.CanBuildFrom[List[Int],String,List[String]] = ...

scala> implicitly[CanBuildFrom[Set[String], Double, Set[Double]]]
res1: scala.collection.generic.CanBuildFrom[Set[String],Double,Set[Double]] = ...

scala> implicitly[CanBuildFrom[List[String], Int, List[Double]]]
:11: error: Cannot construct a collection of type List[Double] with 
  elements of type Int based on a collection of type List[String].
       implicitly[CanBuildFrom[List[String], Int, List[Double]]]
                 ^

scala> implicitly[CanBuildFrom[List[String], Int, Set[Int]]]
:11: error: Cannot construct a collection of type Set[Int] with 
  elements of type Int based on a collection of type List[String].
       implicitly[CanBuildFrom[List[String], Int, Set[Int]]]
                 ^

As you can see from these examples, the third type argument in CanBuildFrom is determined by its first two — it takes its type constructor from the first type argument and its element type from the second type argument. Any attempt to deviate from that pattern results in a static compile time error. This should be sounding familiar ...

Now lets look at where this abstraction is used: the definition of TraversibleLike.map(),

def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That = {
  val b = bf(repr)
  b.sizeHint(this) 
  for (x <- this) b += f(x)
  b.result
}

It should be apparent that here, like the matrix/vector/scalar example, we have a method with a parametrized result type which is not used in any of its explicit argument types, and hence which can't be inferred from its explicit arguments. Just as in the earlier case, it's the implicit argument which is determining the otherwise unconstrained result type via implicit search for a CanBuildFrom instance with type parameters B (determined by type inference from the explicit parameter list) and Repr (effectively a constant). It should also be apparent that the implicit argument is providing a significant component of the implementation of this method via the call bf(repr) — this is exactly analogous to the calls of instance(a, b) in the earlier example.

Hopefully the discussion above will have persuaded you that these are the hallmarks of a Haskell-like functional dependency, right at the heart of the Scala collections framework. And in my book, that makes fundeps in Scala neither impossible nor useless.

Update — I’ll be giving a talk on this material on the 16th of August in London. Please come along.

Scala has a highly expressive type system, but it doesn’t include everything you might find yourself hankering after — at least, not as primitives. There are a few genuinely useful things which fall under this heading — higher-rank polymorphic function types and recursive structural types are two I’ll talk about more in later posts. Today I’m going to show how we can encode union types in Scala, in the course of which I’ll have an opportunity to shed a little light on the Curry-Howard isomorphism and show how it can be put to work for us.

So, first up, what is a union type? A union type is pretty much what you’d expect: it’s the union of two (or more, but I’ll limit this discussion to just two) types. The values of that type are all of the values of each of the types that it’s the union of. An example will help to make this clear, but first a little notation — for reasons which will become apparent later I’ll write the union of types T and U as T ∨ U (ie. the two types flanking the logical ‘or’ operator), and so we write the union of the types Int and String as Int ∨ String. The values of this union type are all the Ints and all the Strings.

What does this mean more concretely? It means that if we could express this type directly in Scala we would be able to write,

def size(x : Int ∨ String) = x match {
  case i : Int => i
  case s : String => s.length
}

size(23) == 23   // OK
size("foo") == 3 // OK
size(1.0)        // Not OK, compile time error

In other words, the size method would accept arguments of either type Int or type String (and their subtypes, Null and Nothing) and nothing else.

It’s important to recognize the difference between this use of a union type and the similar use of Scala’s standard Either. Either is what’s known as a sum type, the analog of union types in languages which don’t support subtyping. Recasting our example in terms of Either we get,

def size(x : Either[Int, String]) = x match {
  case Left(i) => i
  case Right(s) => s.length
}

size(Left(23)) == 23    // OK
size(Right("foo")) == 3 // OK

Either[Int, String] can model the union type Int ∨ String because there is an isomorphism between the two types and their values. But equally clearly the Either type manages this by way of a layer of boxed representation, rather then by being an unboxed primitive feature of the type system. Can we do better than Either? Can we find a way of representing union types in Scala which doesn’t require boxing, and which provides all of the static guarantees we would expect?

It turns out that we can, but to get there we have take a detour through first-order logic via the Curry-Howard isomorphism. Curry-Howard tells us that the relationships between types in a type system can be viewed as an image of the relationships between propositions in a logical system (and vice versa). There are various ways that we can fill that claim out, depending on the type system we’re talking about and the logical system we’re working with, but for the purposes of this discussion I’m going to ignore most of the details and focus on simple examples.

To illustrate Curry-Howard (in the context of a type system with subtyping like Scala’s), we can see that there is a correspondence between intersection types (A with B in Scala) and logical conjunction (A ∧ B); between my hypothetical union types (A ∨ B) and logical disjunction (also A ∨ B, as hinted earlier); and between subtyping (A <: B in Scala) and logical implication (A ⇒ B). On the left hand side of each row in the table below we have a subtype relationship which is valid in Scala (although, in the case of the union types at the bottom, not directly expressible), and on the right hand side we have a logical formula which is obtained from the type relationship on the left by simply rewriting ("with" to "∧" and "<:" to "⇒") — in each case the result of the rewriting is a logically valid.

The essence of Curry-Howard is that this mechanical rewriting process (whichever direction you go in) will always preserve validity — valid type formulae will always rewrite to valid logical formulae, and vice versa. This isn’t only true for conjunction, disjunction and implication. We can also generalize the correspondence to logical formulae which include negation (the key one for us here) and universal and existential quantification.

So what would it mean to add negation to the mix? The conjunction of two types (ie. A with B) has values which are instances of both A and B, so similarly we should expect the negation of a type A (I’ll write it as ¬[A]) to have as it’s values everything which isn’t an instance of A. This is also something which can’t be directly expressed in Scala, but suppose it was?

If it was, then we would be able to crank on the Curry-Howard isomorphism and De Morgan’s laws to give us a definition of union types in terms of intersection types (A with B) and type negation. Here’s how that might go …

First recall the De Morgan equivalence,

(A ∨ B) ⇔ ¬(¬A ∧ ¬B)

Now apply Curry-Howard (using Scala’s =:= type equality operator),

(A ∨ B) =:= ¬[¬[A] with ¬[B]]

If we could work out a way of expressing this in Scala, we’d be home and dry and have our union types. So can we express type negation?

Unfortunately we can’t. But what we can do is transform all of our types in a way which allows negation to be expressed in the transformed context. We’ll then need to work out how make that work for us back in the original untransformed context.

Some readers might have been a little surprised earlier when I illustrated Curry-Howard using intersection types as the correlate of conjunction, union types as the correlate of disjunction and the subtype relation as the correlate of implication. That’s not how it’s normally done — usually product types (ie. (A, B)) model conjunction, sum types (ie. Either[A, B]) model disjunction and function types model implication. If we recast our earlier table in terms of products, sums and functions we end up with this,

On the left hand side we’re no longer looking for validity with respect to the subtype relation, instead we’re looking for evidence of the principle of parametricity, which allows us to determine if a function type is implementable just by reading it’s signature. It’s clear that all the function signatures on the left in the table above can be implemented — for the first two we have an (A, B) pair as our function argument, so we can easily evalutate to either an A or a B, using _1 or _2,

val conj1 : ((A, B)) => A = p => p._1
val conj2 : ((A, B)) => B = p => p._2

and for the last two we have either an A or a B as our function argument, so we can evalute to Either[A, B] (as Left[A] or Right[B] respectively).

val disj1 : A => Either[A, B] = a => Left(a)
val disj2 : B => Either[A, B] = b => Right(b)

This is the form in which the Curry-Howard isomorphism is typically expressed for languages without subtyping. Because this mapping doesn’t reflect subtype relations it isn’t going to be much direct use to us for expressing union types which, like intersection types, are inherently characterized in terms of subtyping. But it can help us out with negation, which is the missing piece that we need.

Either with or without subtyping, the bottom type (Scala’s Nothing type) maps to logical falsehood, so for example, the following equivalences all hold,

because the function signatures on the left are once again all implementable, and the logical formulae on the right are again all valid (see this post from James Iry for an explanation of why I haven’t shown the corresponding cases for products/conjunctions). Now we need to think about what a function signature like,

A => Nothing

corresponds to. On the logical side of Curry-Howard this maps to A ⇒ false, which is equivalent to ¬A. This seems fairly intuitively reasonable — there are no values of type Nothing, so the signature A => Nothing can’t be implemented (other than by throwing an exception, which isn’t allowed).

Let’s see what happens if we take this as our representation of the negation of a type,

type ¬[A] = A => Nothing

and apply it back in the subtyping context that we started with to see if we can now use De Morgan’s laws to get the union types we’re after,

type ∨[T, U] = ¬[¬[T] with ¬[U]]

We can test this using the Scala REPL, which will very quickly show us that we’re not quite there yet,

scala> type ¬[A] = A => Nothing
defined type alias $u00AC

scala> type ∨[T, U] = ¬[¬[T] with ¬[U]]
defined type alias $u2228

scala> implicitly[Int <:< (Int ∨ String)]
<console>:11: error: Cannot prove that Int <:< 
  ((Int) => Nothing with (String) => Nothing) => Nothing.
       implicitly[Int <:< (Int ∨ String)]

The expression “implicitly[Int <:< (Int ∨ String)]” is asking the compiler if it can prove that Int is a subtype of Int ∨ String, which it would be if we had succeeded in coming up with an encoding of union types.

So what’s gone wrong? The problem is that we have transformed the types on the right hand side of the <:< operator into function types so that we can make use of the encoding of type negation as A => Nothing. This means that the union type is itself a function type. That’s clearly not consistent with Int being a subtype of it — as the error message from the REPL shows. To make this work, then, we also need to transform the left hand side of the <:< operator into a type which could possibly be a subtype of the type on the right. What could that transformation be? How about double negation?

type ¬¬[A] = ¬[¬[A]]

Lets see what the compiler says now,

scala> type ¬¬[A] = ¬[¬[A]]
defined type alias $u00AC$u00AC

scala> implicitly[¬¬[Int] <:< (Int ∨ String)]
res5: <:<[((Int) => Nothing) => Nothing,
  ((Int) => Nothing with (String) => Nothing) => Nothing] = <function1>

scala> implicitly[¬¬[String] <:< (Int ∨ String)]
res6: <:<[((String) => Nothing) => Nothing,
  ((Int) => Nothing with (String) => Nothing) => Nothing] = <function1>

Bingo! ¬¬[Int] and ¬¬[String] are both now subtypes of Int ∨ String!

Let's just check that this isn't succeeding vacuously,

scala> implicitly[¬¬[Double] <:< (Int ∨ String)]
<console>:12: error: Cannot prove that ((Double) => Nothing) => Nothing <:<
  ((Int) => Nothing with (String) => Nothing) => Nothing.
       implicitly[¬¬[Double] <:< (Int ∨ String)]

We're almost there, but there's one remaining loose end to tie up — we have subtype relationships which are isomorphic to the ones we want (because ¬¬[T] is isomorphic to T), but we don't yet have a way to express those relationships with respect to the the untransformed types that we really want to work with.

We can do that by treating our ¬[T], ¬¬[T] and (T ∨ U) as phantom types, using them only to represent the subtype relationships on the underlying type rather that working directly with their values. Here's how that goes for our motivating example,

def size[T](t : T)(implicit ev : (¬¬[T] <:< (Int ∨ String))) = t match {
  case i : Int => i
  case s : String => s.length
}

This is using a generalized type constraint to require the compiler to be able to prove that any T inferred as the argument type of the size method must be such that it's double negation is a subtype of (Int ∨ String). That's only ever true when T is Int or T is String, as this REPL session shows,

scala> def size[T](t : T)(implicit ev : (¬¬[T] <:< (Int ∨ String))) = t match {
     |     case i : Int => i
     |     case s : String => s.length
     |   }
size: [T](t: T)(implicit ev: <:<[((T) => Nothing) => Nothing,
  ((Int) => Nothing with (String) => Nothing) => Nothing])Int

scala> size(23)
res8: Int = 23

scala> size("foo")
res9: Int = 3

scala> size(1.0)
:13: error: Cannot prove that ((Double) => Nothing) => Nothing <:<
  ((Int) => Nothing with (String) => Nothing) => Nothing.
       size(1.0)

One last little trick to finesse this slightly. The implicit evidence parameter is syntactically a bit ugly and heavyweight, and we can improve things a little by converting it to a context bound on the type parameter T like so,

type |∨|[T, U] = { type λ[X] = ¬¬[X] <:< (T ∨ U) }

def size[T : (Int |∨| String)#λ](t : T) = t match {
    case i : Int => i
    case s : String => s.length
}

And there you have it — an unboxed, statically type safe encoding of union types in unmodified Scala!

Obviously it would be nicer if Scala supported union types as primitives, but at least this construction shows that the Scala compiler has all the information it needs to be able to do that. Now we just need to pester Martin and Adriaan to make it directly accessible.

Nigel Warren, designer of Fly, said “I’m delighted that Miles and his team at Chuusai will be bringing all of their deep understanding of Scala in the High Performance, Financial and Scientific Computing domains to bear on projects using Fly to cluster their distributed Scala apps. The Fly team has worked very hard to make the Scala interface natural for Scala developers by, for example, including direct support for Actors, whilst maximising performance by writing native Scala objects directly to the Fly server.”

Miles Sabin of Chuusai said “We’re delighted to be working with the Fly team to bring the combination of tuple spaces and Scala to a wider audience. Tuple spaces are coming into their own in our new environment of ubiquitous parallelism and distribution, and Fly, being the foremost implementation available on the Java Virtual Machine, and the only one with a Scala interface, deserves serious attention from everyone working in demanding computational domains.”

BCS SPA 2010 – An Introduction to Scala for Java Developers

]]> http://www.chuusai.com/2010/05/20/spa-workshop-slides/feed/ 0 http://www.chuusai.com/2010/04/17/scala-days-slides/ http://www.chuusai.com/2010/04/17/scala-days-slides/#comments Sat, 17 Apr 2010 23:16:32 +0000 Miles Sabin http://www.chuusai.com/?p=27

Scaladays 2010 – The Scala IDE for Eclipse – Retrospect and Prospect for 2.8 and Beyond

]]> http://www.chuusai.com/2010/04/17/scala-days-slides/feed/ 0 http://www.chuusai.com/2010/03/26/eclipsecon-slides/ http://www.chuusai.com/2010/03/26/eclipsecon-slides/#comments Fri, 26 Mar 2010 23:03:44 +0000 Miles Sabin http://www.chuusai.com/?p=26

Eclipsecon 2010 – Scala Support in Eclipse

]]> http://www.chuusai.com/2010/03/26/eclipsecon-slides/feed/ 0 http://www.chuusai.com/2010/02/26/bnp-paribas-slides/ http://www.chuusai.com/2010/02/26/bnp-paribas-slides/#comments Fri, 26 Feb 2010 19:23:13 +0000 Miles Sabin http://www.chuusai.com/?p=25

An Introduction to Scala – Blending OO and Functional Paradigms

]]> http://www.chuusai.com/2010/02/26/bnp-paribas-slides/feed/ 0 http://www.chuusai.com/2010/01/29/intro-to-scala-slides/ http://www.chuusai.com/2010/01/29/intro-to-scala-slides/#comments Fri, 29 Jan 2010 19:11:41 +0000 Miles Sabin http://www.chuusai.com/?p=24

An Introduction to Scala for Java Developers

]]> http://www.chuusai.com/2010/01/29/intro-to-scala-slides/feed/ 0 http://www.chuusai.com/2010/01/27/scala-ide-2-8-beta-1-released/ http://www.chuusai.com/2010/01/27/scala-ide-2-8-beta-1-released/#comments Wed, 27 Jan 2010 19:08:28 +0000 Miles Sabin http://www.chuusai.com/?p=23 Read the full announcement here.

Scala Support in Eclipse – Monkey-patching the JDT for fun and profit?

]]> http://www.chuusai.com/2009/11/25/eclipse-democamp-london-slides/feed/ 0 http://www.chuusai.com/2009/11/20/51cto-interview/ http://www.chuusai.com/2009/11/20/51cto-interview/#comments Fri, 20 Nov 2009 18:56:23 +0000 Miles Sabin http://www.chuusai.com/?p=20 Read the full article here.

JVM Languages Support in Eclipse – Monkey-patching the JDT for fun and profit?

]]> http://www.chuusai.com/2009/09/27/jvm-lang-summit-slides/feed/ 0 http://www.chuusai.com/2009/09/27/base-slides/ http://www.chuusai.com/2009/09/27/base-slides/#comments Sun, 27 Sep 2009 13:02:44 +0000 Miles Sabin http://www.chuusai.com/?p=15

The Scala IDE for Eclipse – Retrospect and Prospect for 2.8.0

]]> http://www.chuusai.com/2009/09/27/base-slides/feed/ 0 http://www.chuusai.com/2009/09/10/scala-ide-2-7-6-released/ http://www.chuusai.com/2009/09/10/scala-ide-2-7-6-released/#comments Thu, 10 Sep 2009 12:59:02 +0000 Miles Sabin http://www.chuusai.com/?p=14 Read the full announcement here.