Compile-time Extension Interfaces

[raulraja]

Proposal

The goal of this proposal is to enable compile-time dependency resolution through extension syntax. Overall, we want to enable "contract" interfaces to be declared as program constraints in function or class constructor arguments and enable the compiler to automatically resolve and inject those instances that must be provided with evidence in one of a given set of scopes. The resolution scopes to look for those evidences are predefined and ordered, more details inside the proposal.

In the case of not having evidence of a required interface (program constraints), the compiler would fail and provide the proper error messages both by IntellIJ IDEA inspections and compile-time errors.

This would bring first-class named extension families to Kotlin. Extension families allow us to guarantee that a given data type (class, interface, etc.) satisfies behaviors (a group of functions or properties) that are decoupled from the type's inheritance hierarchy.

Unlike the traditional subtype style composition where users are forced to extend and implement classes and interfaces, extension families favor horizontal composition based on compile-time resolution between types and their extensions.

Current code status

Thanks to @truizlop and @JorgeCastilloPrz,we already have a working POC where we've coded a big part of the KEEP-87. There are still some additional behaviors we'd need to code that have been clarified on the KEEP though, and we'd love to get JetBrains help on iterating those. Instructions about how to test it have been provided within the "How to try?" section of the keep, where we have also provided a code sample.

Rewording

We have revamped the initial proposal given Type Classes are not really our aim here, but compile-time dependency resolution for Kotlin. The nature of the proposal has not changed a lot, just suffered a major refactor so it's more in the line of what a generic language feature like this one would be, and also to fit the compiler implementation feedback we've got from some JetBrains devs these later months.

We believe the compiler could support compile-time dependency resolution out of the box, as other modern laguages already do. We believe some infrastructure patterns would fit better as built-in language features, so users can get their canonical solution for that already provided and focus their work on the actual domain requirements and business logics of their programs. We've seen similar things in other languages that provide built in support for concurrency and other stuff.

How to contribute

Want to help to bring compile time extension resolution to Kotlin?. A fork is being provisioned where a reference implementation based on this proposal will take place at https://github.com/arrow-kt/kotlin

[elizarov]

The proposed syntax for companion constrains looks a bit confusing. Take a look at this example:

typeclass Monoid {
    fun combine(b: Self): Self 
    fun Self.Companion.empty(): Self
}

It is not perfectly clear from this declaration what is the receiver type of combine and empty functions. It is, in fact, Self for combine and Self.Companion for empty, but the syntax is not regular enough to make it clear.

I was thinking a bit on how to fit companion-extension syntax in a nicer way and came up with the following alternative syntactic approach.

Let's eschew the idea that extension declaration is a way to group multiple extension functions together, but instead leave the regular extension function syntax intact inside both typeclass and extension scopes like this:

typeclass Monoid {
    fun Self.combine(b: Self): Self
    fun Companion.empty(): Self
}

Here, in the context of typeclass declaration (and only there) we treat Self and Companion as context-sensitive keywords that refer, respectively, to the actual class that this typeclass applies to and its companion (regardless of how its companion is in fact named). Now, when declaring the actual extension we'll have:

extension Int : Monoid {
    fun Int.combine(b: Int): Int = this + b
    fun Int.Companion.empty(): Int = 0
}

In this approach extension block serves just a wrapper and an additional safety net. It does not bring any scope with it. All the functions declared inside extension are going to be treated normally (as if they were on the top level), but compiler will complain if they do not match the signatures of the corresponding type-class.

Unfortunately, this proposal also opens the whole can of worms on companion-vs-static debate. One of the issues being the fact, that not all classes in Kotlin (and even in Kotlin stdlib) have companions and there is currently no way to declare a companion outside of the class. Moreover, it is hardly possible to fix or to work around due to dynamic nature of JVM.

One solution to this conundrum is to stop piggybacking onto companion objects at all. Let us, instead, declare functions on the typeclass itself, like this:

typeclass Monoid {
    fun Self.combine(b: Self): Self
    fun empty(): Self
}

In this approach typeclass does declare a scope in a similar way to interface declaration. We interpret combine as a function with two receivers (an instance of the typeclass and Self), while empty has only one receiver (an instance of typeclass). In fact, it meshes perfectly with how the typeclasses are going to be represented on JVM anyway. Now the extension declaration is going to look like this:

extension Int : Monoid {
    fun Int.combine(b: Int): Int = this + b
    fun empty(): Int = 0
}

The interesting thought direction here is that this kind of extension mechanism can work as a replacement for companion objects altogether. We might just deprecate companions, because any declaration of the form:

class Foo {
    companion object {
       /** body **/
    }
}

can be replaced, for the same effect, with

class Foo {
    extension Foo {
       /** body **/
    }
}

or, which is not possible with companion objects, if the access to private members of Foo is not needed, it can be moved outside:

class Foo {
}

extension Foo {
    /** body **/
}

It is cleaner and more modular, since you are not limited to one companion object anymore, but can define as many as you want.

However, with this alternative-to-companion approach to typeclasses we have to work out on how the typeclass instances are going to be named and what is the syntactic approach to explicitly specify this name (this is, at least, needed for JVM interop). We can also try to somehow reuse companion modifier for extension blocks by imbuing it with an additional syntax. Lots of things to think about.

[raulraja]

@elizarov makes perfect sense. I'll update the proposal to the new proposed syntax

[mkobit]

Small note seeing this link, you may want to copy/paste the snippets if you believe it provides important context because the Kotlin slack is free and only has the most recent 10k messages.

[raulraja]

@mkobit thanks, I copied most of the snippets from @elizarov into the proposal so I think we are good if that goes away.

[mikehearn]

I realise all this stuff comes from Haskell and people who love category theory, but I strongly suggest that to gain better acceptance you drop the conventional FP names for things and use more appropriate names that are less jargony and more Java-like:

Monoid -> Combinable
Functor -> Mappable

etc

[raulraja]

@mikehearn those are just examples and not part of the proposal. Anyone is free to name their own typeclasses whatever they want and this proposal does not address what names users will choose nor it proposes new types for the stdlib.
Also the examples don't come from Haskell. Those names and type classes I used as examples are part of many libraries and langs by now including libs in Kotlin.

[dnpetrov]

How type class implementations (extensions) are resolved?
When two implementations of the type classes are in conflict?

Elaborating on the example with ProdConfig and TestConfig, suppose I have

package monoid

typeclass Monoid {
    fun Self.combine(b: Self): Self
    fun empty(): Self
}

and, for example,

package org.acme

extension Int : Monoid {
    fun Int.combine(b: Int): Int = this + b
    fun empty(): Int = 0
}

and

package org.fizzbuzz

extension Int : Monoid {
    fun Int.combine(b: Int): Int = this * b
    fun empty(): Int = 1
}

Now, in package myapp, I want to use one particular implementation of Monoid.
What should I import?

[raulraja]

@dnpetrov an import matching the contained instances would bring them into scope:

import org.fizzbuzz.*

In the case where you have two instances that are conflicting the compiler should bail because they are ambiguous and the user needs to disambiguate them.

import org.fizzbuzz.*
import org.acme.*

val x: Int = 1.combine(2) // // Ambiguous instances for Monoid: Int. Please redefine`extension Monoid: Int` in the current scope or use tagged instances if they provide different behavior.

A natural way for disambiguation may be redefining the instance in the current scope which would take precedence over the imported one.
If the compiler generated synthetic names for the instances or supported users naming them you could just import the FQN.

import org.fizzbuzz.monoidIntAdd 

Another approach would be to place further constrains in the instances like is done today with the <A: Any> trick where it is used to ensure non nullable values are always passed as args to functions.

typeclass Add : Int
typeclass Multiply : Int

extension Int : Monoid : Add {
    fun Int.combine(b: Int): Int = this + b
    fun empty(): Int = 0
}

extension Int : Monoid : Multiply {
    fun Int.combine(b: Int): Int = this * b
    fun empty(): Int = 1
}

fun <Tag, A: Monoid: Tag> combine(a: A, b: A): A = a.combine(b)

combine<Add, Int>(1, 1) // 2
combine<Multiply, Int>(1, 1) // 1
combine(1, 1) // Ambiguous instances for `Monoid: Int`. Please redefine`extension Monoid: Int` in the current scope or use tagged instances if they provide different behavior.

[elizarov]

Let's assume the example by @dnpetrov

package org.acme

extension Int : Monoid {
    fun Int.combine(b: Int): Int = this + b
    fun empty(): Int = 0
}

It is pretty straightforward with star imports:

import org.acme.*

To disambiguate extension in the case of start import from the other package, I suggest to use the simple name of the class that extension is being defined for, like this:

import org.acme.Int

This will bring in scope all Int extensions defined in this package.

This approach has a nice and quite well-intended side-effect that if a class and its extension are both defined in the same package, then a qualified import of a class automatically brings into scope all the extensions defined for this class in the same package. We can even go as far as to forbid redefining class extensions in other packages if they are defined in the original package. This will improve predictability and is quite consistent with a class member vs extension resolution in Kotlin (class member, being defined by the class author, always takes precedence).

However, if there is no extension in the class's original package, then different clients can define and/or import different extension instances for this class with the same typeclass.

Also note, that a generic method with typeclass upper-bounds should have an implict instance of typeclass in scope behind the scenes (actually passed to it by invoker) and this instance should always take precedence.

[elizarov]

The interesting side-effect of this proposal is that we can almost deprecate reified modifier and reimplement it with type-classes instead (not really):

typeclass Reified {
    val selfClass: KClass<Self>
}

Now a function that was doing something like:

fun <reified T> foo() { .... T::class ... }

can be replaced with:

fun <T : Reified> foo() { .... T.selfClass ... }

This analogy might help to understand how typeclasses shall behave in various scenarios. In particular, generic classes with typeclass upper bounds shall be supported if and only if we roll out support of generic classes with reified type parameters.

[raulraja]

@elizarov that will simplify most of our code a ton since we are currently relying on inline <reified A>... all over the place and we frequently find places inside generic classes where we can't use those methods with the classes type args.

[raulraja]

Updated the proposal with a section on overcoming some of the limitations of inline reified with Typeclass constrain evidences as demonstrated in #87 (comment)

[dnpetrov]

What if I have multiple type class implementations for the same type in the same package?
Like

package org.acme.bigint

class BigInt { ... }

extension BigInt : Monoid { ... }
extension BigInt : Show { ... }

I'd expect extensions to have names or something. Implicit imports look somewhat fragile. You don't know what else you bring into the scope.

Maybe even just follow familiar class inheritance syntax:

typeclass Monoid<T> {
    fun T.combine(b: T): T
    fun empty(): T
}

extension AdditiveInt : Monoid<Int> {
    override fun Int.combine(b: Int): Int = this + b
    override fun empty(): Int = 0
}

Then typeclass becomes just a special kind of interface, and extension - a special kind of object, with its members implicitly imported in file scope, and all regular resolution rules just work as they should. Note that you also get multi-parameter type classes for free.

[dnpetrov]

With parametric type classes, type class dependent functions might need some different syntax, like:

fun <T> combineWith(others: Iterable<T>): T where Monoid<T> { ... }

[raulraja]

@dnpetrov As for the syntax makes sense. That syntax is similar to what I originally proposed as well since it is also necessary to implement the fact that certain type classes can't target a single type but target the combinations of many. For example a natural transformation to go from one type to another:

typeclass FunctionK<F<_>, G<_>> {
  fun <A> invoke(fa: F<A>): G<A>
}

extension Option2List : FunctionK<Option, List> {
  fun <A> invoke(fa: Option<A>): List<A> =
    fa.fold({ emptyList() }, { listOf(it) })
}

In the case above the instance does not target neither Option nor List but those are part of the type args and folks should be able to summon the instance somehow if they can prove that such instance exists for both types. I'm not sure how that case could be expressed with extension syntax.

@dnpetrov In your example above an alternative way to express combineWith could be:

fun <T> combineWith(others: Iterable<T>, instance MT : Monoid<T>): T = ...

That would allow call sites to replace the instance by explicitly providing one if needed so the method could be invoked in a similar way as:

combineWith(others) //compiles if Monoid<T> instance is found and fails if not.
combineWith(others, customInstance) // always compiles

That also is an alternative way to handle ambiguity when collisions exists beside using concrete imports.

@elizarov @dnpetrov thoughts? Thanks!

[dnpetrov]

@raulraja Looks good. Need some time to reshuffle my thoughts and probably show this to the owner of Resolution & Inference. @erokhins pls take a look...

[elizarov]

The latest syntactic proposal by @dnpetrov for typeclass declaration is indeed more general and is easier to adapt to to the can of typeclasses binding multiple types.

However, I don't like the need to name typeclasses and more so the need to declare and name typeclass instances in function signature. It is just a boilerplate. IMHO, it defeats the whole purpose of having type-classes as a language feature, since you can already do all that.

At this point I would recommend to (re)read Cedric's blog post on ad-hoc polymorphism. In our example, we can already write just:

interface Monoid<T> {
    fun T.combine(b: T): T
    fun empty(): T
}

then write

object AdditiveInt : Monoid<Int> {
    override fun Int.combine(b: Int): Int = this + b
    override fun empty(): Int = 0
}

then declare functions with an explicit type class instance:

fun <T> combineWith(others: Iterable<T>, MT : Monoid<T>): T = ...

It is even shorter than the latest syntactic proposal for typeclasses, since I don't have to add instance modifier before MT. So, what is the value of the type classes proposal then?

IMHO, the only valuable benefit of having type classes directly supported in the language is making them easier to grasp for beginners by capitalizing on the mental model of interfaces on steroids (aka Swift protocols), and eschewing as much boilerplate as we can in the process. Having to name your type class instance is just a boilerplate, having to pass your type-class instances explicitly is just a boilerplate. Neither should be required.

[cbeust]

A nitpick but in the examples, I think you should be more specific about the name of the Monoid types, e.g. MonoidAddition, MonoidMultiplication, etc...

[raulraja]

@elizarov Kategory uses interfaces for typeclasses now and does it like that. The issue is that:

fun <T> combineWith(others: Iterable<T>, MT : Monoid<T>): T = ...

Requires explicitly passing the instance manually which defeats the purpose of type classes. Having a keyword to require the compiler to verify that call sites such as combineWith(listOf(1)) compiles because AdditiveInt is in scope is the real feature because we get compile time verification of constrains.

I agree though that the variant based on extensions is much easier to grasp for beginners. We can perhaps encode all complex cases where typeclasses require multiple types if typeclasses allowed for type args:

typeclass FunctionK<G<_>> {
  fun <A> Self<A>.convert(): G<A>
}

extension Option : FunctionK<List> {
  fun <A> Option<A>.convert(): List<A> =
    this.fold({ emptyList() }, { listOf(it) })
}

val x: List<Int> = Option(1).convert() // [1] 
val y: List<Int> = None.convert() // [ ] 

And in a polymorphic context:

fun <G<_>, F<_>: FunctionK<G>, A> convert(fa: F<A>): G<A> = fa.convert()
val l: List<Int> = convert(Option(1))
val s: Set<Int> = convert(Option(1)) // No `Option: FunctionK<Set>` extension found in scope.

[elizarov]

Having played more with @dnpetrov syntactic proposal I start to like certain parts of it. First of all, the generic-like declaration for interface indeed looks more concise than my original proposal with the Self keyword. Being able to eschew the "magic" nature of Self is also good.

Moreover, instead of a having to reserve a new typeclass keyword we can just be explicit about the fact that it just an interface, but tag this interface with an extension modifier like it is customary in Kotlin's syntactic tradition:

extension interface Monoid<T> {
    fun T.combine(b: T): T
    fun empty(): T
}

This syntax also scales to multiple types (and, maybe HKTs, but I will not cover them here). For example, you can have:

extension interface Bijection<A, B> {
    fun A.toB(): B
    fun B.toA(): A
}

However, as soon as you allow applying this syntax to multiple types, the story with "standalone" (static? but not really) functions like empty becomes complicated. We do want those functions to extend "the type" and we want them being applicable on the type itself (like companion object functions for the type). You should be able to use Int.empty() expression in the context of Monoid<Int>, but with multiple types it becomes unclear which type is being extended this way.

In order to overcome this difficulty, we can use some fresh syntactic construct to bind the type to its typeclass. Look at how the dot (.) can work for this purpose and how it is consistent with Kotlin's approach to extension functions:

extension interface T.Monoid {
    fun T.combine(b: T): T
    fun empty(): T
}

In this context, T will be called "an extension receiver type" (that is, the type that receives the extension).

Note, that we can allow to add additional types in <...> after the extension interface name and allow "receiver-less" extensions like Bijection, too.

Similarly, declaration of typeclass instances can ready follow the model of Kotlin's companion object. Name can be optional, too, so that an instance without a name can be written like this:

extension object : Int.Monoid {
    override fun Int.combine(b: Int): Int = this + b
    override fun empty(): Int = 0
}

Since we have indicated that Int is an "extension receiver type", it becomes clear and syntactically transparent on why one can write Int.empty() in the context of this extension object. It is also clear on how to provide an explicit name, if needed.

Alas, this notation makes it less obvious on how to specify typeclass constraints for functions. The interface-inspired syntax of T : Monoid does not look consistent with the declaration of Monoid anymore and breaks altogether with receiver-less type classes like Bijection<A,B>. I don't think that reusing a where clause works either, but we can introduce a new clause for this purpose, for example:

fun <T> combineWith(others: Iterable<T>): T given T.Monoid = ...

[dnpetrov]

@elizarov

However, as soon as you allow applying this syntax to multiple types, the story with "standalone" (static? but not really) functions like empty becomes complicated. We do want those functions to extend "the type" and we want them being applicable on the type itself (like companion object functions for the type). You should be able to use Int.empty() expression in the context of Monoid, but with multiple types it becomes unclear which type is being extended this way.

This approach has some issues with overloading. E.g., what if I have several different type classes (extension interfaces, whatever) for T with method empty()?
I'd suppose it should rather be more explicit, like in #87 (comment), where type class implementation is passes as a named parameter, which has special default value resolution strategy.

I'd expect something like:

extension interface Monoid<T> {
    fun T.combine(b: T): T
    fun empty(): T
}

extension object AdditiveInt : Monoid<Int> {
    override fun Int.combine(b: Int): Int = this + b
    override fun empty(): Int = 0
}

extension object StringConcatenation : Monoid<String> {
    override fun String.combine(b: String): String = this + b
    override fun empty(): String = ""
}

fun <T> T.timesN(n: Int, extension monoid: Monoid<T>): T {
    var result = empty() // 'monoid' is an implicit dispatch receiver
        // or, if you need to disambiguate, 'monoid.empty()'
    for (i in 0 until n) {
        result = result.combine(this) // 'monoid' is an implicit dispatch receiver, 'combine' is a member extension
          // or 'monoid.run { result.combine(this) }'
    }
    return result
}

fun test() {
    assertEquals("abcabcabcabc", "abc".timesN(4))
}

[dnpetrov]

Now, when I look at my example above, it starts to remind me "extension providers" from Xtend (https://www.eclipse.org/xtend/documentation/202_xtend_classes_members.html#extension-methods).
Probably we'll also need 'extension val's.

@raulraja @elizarov
I think we need some more practical motivating example. Yeah, all those Monoids and BiFunctions and HKTs thrown in here and there are nice. Can we think of a concise, meaningful piece of code to demonstrate why type classes (extension interfaces) make life better? Like HTML builders for extension lambdas, for example.

[d3xter]

One possible motivation could be ad-hoc polymorphism as described by Cedric.
An Example is Limiting the possible values put into a JSON.

For example in klaxon to create a new Json-Object you have the following method:
fun obj(vararg args: Pair<String, *>): JsonObject Link

This basically allows everything to be put inside.
Other JSON-Libraries like json-simple have many methods called putString/putInt and so on.

With typeclasses we can define a JSONValue:

typeclass T.JSONValue {
    fun T.toJSON() = String
    fun fromJSON(String) = T
}

extension object : Int.JSONValue {
    fun Int.toJSON() = this.toString()
    fun fromJSON(a: String) = a.toInt()
}

Then all you need inside your JSON-Library is a method like that:

fun <T> JsonObject.put(key: String, value: T) given T.JSONValue { ... }

[raulraja]

@dnpetrov here are a few:

Type classes cover many use cases, here are a few interesting ones syntax aside:

Compile-Time Dependency Injection

package common

extension interface Dependencies<T> {
    fun config(): Config
}

package prod

extension object ProdDeps: Dependencies<Service> {
    fun config(): Config = ProdConfig
}

package test

extension object TestDeps: Dependencies<Service> {
    fun config(): Config = TestConfig
}

import test.*

Service.config() // TestConfig

import prod.*

Service.config() // ProdConfig

In the example above we used config but you can apply a similar case to the Android Activity which managing it one of the biggest issues for developers.

Type restrictions for safer operations

extension interface Numeric<T> {
    fun T.plus(y: T): T
    fun T.minus(y: T): T
}

extension Fractional<T> : Numeric<T> {
    def T.div(y: T): T
}

extension object : Numeric<Int> {
    override fun Int.plus(y: Int): Int = this + y
    override fun Int.minus(y: Int): Int = this + y
}

extension object : Fractional<Double> {
    override fun Double.plus(y: Double): Double = this + y
    override fun Double.minus(y: Double): Double = this + y
    override fun Double.div(y: Double): Double = this / y
}

fun <T> add(x: T, y: T, extension N: Numeric<T>): T = x.plus(y)
fun <T> div(x: T, y: T, extension F: Fractional<T>): T = x.div(y)

add(1, 1) // 2
add(1.0, 1.0) // 2.0
div(3.0, 5.0) // 0.6
div(3, 5) // No `Fractional<Int>` instance found in scope.
//In Kotlin now 3/5 == 0 loosing precision.

Type safe composable encoder / decoders and reflectionless instrospection

Expanding on what @d3xter mentions in #87 (comment)

sealed class JValue {
    object JNull: JValue()
    class JString(s: String): JValue()
    class JDouble(num: Double): JValue()
    class JDecimal(num: BigDecimal): JValue()
    class JInt(num: Int): JValue()
    class JBool(value: Boolean): JValue()
    class JObject(obj: List<JField>): JValue()
    class JArray(arr: List<JValue>): JValue()
}

typealias JField = Pair<JString, JValue>

extension Writes<T> {
  fun writes(o: T): JValue
}

extension Reads<T> {
  fun reads(json: JValue): T
}

extension Format<T> : Reads<T>, Writes<T>

//Primitive to JValue instances omitted for simplicity

fun <T> T.toJson<T>(instance tjs: Writes<T>): JValue = tjs.writes(this)

fun <T> JValue.to(intance fjs: Reads<T>): T = fjs.reads(this)

data class Person(val name: String, val age: Int)

object extension Writes<Person> {
    fun writes(o: Person): JValue = JObject(listOf(
        "name".toJson() to o.name.toJson(), // fails to compile if no `Reads<String>` is defined
        "age".toJson() to o.age.toJson(), // fails to compile if no `Reads<Int>` is defined
    ))
}

Person("William Alvin Howard", 91).toJson() // if both `Reads<String>` `Reads<Int>` is defined : { "name": "William Alvin Howard", age: "91" }

The most important use case of them all is that users and library authors can write code with typeclasses and default instances providing complete solutions to their users. At the same time users have the power to override library exposed typeclasses behavior at any time by providing other instances in scope.
In the example above if you need keys serialized with Pascal Case style the user can just override the instance and leave the rest of the library intact:

Defined in user code (not in the json library), this implementation capitalizes the keys.

object extension Writes<Person> {
    fun writes(o: Person): JValue = JObject(listOf(
        "Name".toJson() to o.name.toJson(), // fails to compile if no `Reads<String>` is defined
        "Age".toJson() to o.age.toJson(), // fails to compile if no `Reads<Int>` is defined
    ))
}

Person("William Alvin Howard", 91).toJson() // { "Name": "William Alvin Howard", "Age": "91" }

The library author did not have to add specific support for Pascal Case because overriding a tiny bit of functionaly is easy enough. Entire system and libraries can be composed with typeclass evidences deferring dependencies to users but providing reasonable defaults that are verified at compile time therefore making programs more secure and making easier not to recurre to runtime reflection, costly initialization graph of dependencies etc.

I'll be happy to provide more use cases if needed but I believe the controlled and safe power type classes would bring to Kotlin is a great step forward in terms of compile time verification, type safety and performance for many common patterns like the ones demonstrated above.

There is also the use case of supporting typed FP with all the abstractions we used in examples previously but I will not bring those examples here because Type Classes is much more than FP, it's all about polymorphism and evidence based compile time verification opening the door to many other use cases most of them in trivial tasks users face in daily programming.

[elizarov]

@dnpetrov I would suggest that "type extension" like empty in Monoid are added to the scope of the corresponding type, so the examples below should just work:

fun <T> T.timesN(n: Int): T given T.Monoid {
    var result = T.empty() // because T is a Monoid.
    for (i in 0 until n) {
        result = result.combine(this) // again, because we are given that T is Monoid
    }
    return result
}

[raulraja]

@elizarov is :T given T.Monoid<T> needed or would it be :T given Monoid<T> ?

[elizarov]

@raulraja That was a type. Fixed, than you. You can use either :T given Monoid<T> or :T given T.Monoid depending on the definition of Monoid -- either as extension interface Monoid<T> or extension interface T.Monoid.

The problem with the former, as was discussed before, is that it is not clear how to disambiguate the functions like empty without a Scala-like implicit parameters, which I particularly despise, because they wreck the symmetry between declaration and use of a function.

With the later declaration, given T.Monoid we can use T.empty() to disambiguate. The open questions, which I cannot answer without a larger design discussion and analysis of use-case if whether we should support unqualified empty() at all and whether a declaration of Monoid<T> (without a receiver type) with receiver-less empty() function should be even allowed.

[fatjoem]

I agree with @fvasco. I still think that @edrd-f example should be added to the keep. It shows in a simple way what the downsides of existing language features are.

[ulrikrasmussen]

@edrd-f This doesn't work in general. Example:

interface Monoid<A> {
  fun z(): A
  fun A.add(other: A): A
}

class PairIsMonoid<A,B>(
  val left: Monoid<A>, val right: Monoid<B>
): Monoid<Pair<A, B>>, Monoid<A> by left, Monoid<B> by right { 
  /*implementation omitted */
}

The compiler fails because I cannot implement Monoid<*> twice due to type erasure. I tried to implement this exact pattern last week for JSON serialization and some other application-specific interfaces I had, and could not do it due to this problem.

I therefore think that the assertion that the only viable alternative being explicit with(...) is valid and shouldn't be weakened.

[fatjoem]

@ulrikrasmussen I am sure your real-world json serialization problem would be a much appreciated example for many who are not used to monoid terminology.

[edrd-f]

@fvasco indeed, but it's possible to have interfaces that solely define extension functions:

interface ValidatorExtensions<T> {
    fun T.isValid(): Boolean
}

These extensions are only visible within the class implementing it, so although the class can be considered a ValidatorExtensions, the functions are not accessible, thus not changing the public API.

But in case one doesn't want to expose this interface publicly, the approach would be to simply call the validator methods:

interface Validator<A> {
    fun isValid(record: A): Boolean
}

class ValidatedRepository<A>(private val validator: Validator<A>) : Repository<A> {

    val storedUsers: MutableMap<Int, A> = mutableMapOf() // e.g. users stored in a DB

    override fun loadAll(): Map<Int, A> {
        return storedUsers.filter { validator.isValid(it.value) }
    }

    override fun loadById(id: Int): A? {
        return storedUsers[id]?.let { if (validator.isValid(it)) it else null }
    }

    override fun A.save() {
        storedUsers[generateKey(this)] = this
    }
}

What's the difference in this case?

[BenWoodworth]

@edrf-f

These extensions are only visible within the class implementing it

The extension functions will be visible when used with scoping functions, among other things:

val validatedRepository: ValidatedRepository<Foo>
val Foo: Foo
validatedRepository.run {
    foo.isValid() // visible here
}

[ulrikrasmussen]

@fatjoem Sure. Simplified, I just want to have a .toJson() method I can call to encode objects as JSON:

interface ToJson<A> {
  fun A.toJson(): JsonElement
}

I want to be able to serialize lists if the elements are serializable. I can write the instance as a class which takes the element ToJson<A> instance as a constructor argument and produces a class implementing ToJson<List<A>>. Using the proposed approach to avoid explicit use of with(...), I would have to write the class definition as follows:

class ListToJson<A>(
  val elementToJson: ToJson<A>
): ToJson<List<A>>, ToJson<A> by elementToJson {
  override fun List<A>.toJson() = /* implement encoder using the A.toJson() instance that is implicitly in scope */
}

This fails to compile, however, because I am not allowed to define a class that implements the same generic interface with different type arguments (it tries to implement ToJson<List<A>> and ToJson<A> at the same time). The reason is, I assume, due to type erasure. The JVM signatures for the A.toJson() and List<A>.toJson() extension functions are identical, so it is impossible to generate code for the class. This problem does not arise with nested uses of with(..) because the implementations live in different objects.

[ulrikrasmussen]

@edrd-f

What's the difference in this case?

Apart from the verbosity (it is always possible to translate type class code to explicit record passing), you also lose the ability to enforce coherency in the compiler, which can make it harder to reason about your code. That is, the implementations that are passed to you are chosen by the caller, and you cannot rely on the fact that there is at most one instance for a given type.

[edrd-f]

@ulrikrasmussen – quoting @elizarov:

On a more philosophical note. In Kotlin corehent behaviors are supported via regular interfaces. A class implements an interface as an evidence of providing certain behavior in a coherent way. We don't need yet another mechanism in the language for that. Typeclasses, as proposed, are designed to solve the problem of adhoc polymorphism, which is useful exactly where there is no need for coherence.

[edrd-f]

@fatjoem Sure. Simplified, I just want to have a .toJson() method I can call to encode objects as JSON:

interface ToJson<A> {
  fun A.toJson(): JsonElement
}

I want to be able to serialize lists if the elements are serializable. I can write the instance as a class which takes the element ToJson<A> instance as a constructor argument and produces a class implementing ToJson<List<A>>. Using the proposed approach to avoid explicit use of with(...), I would have to write the class definition as follows:

class ListToJson<A>(
  val elementToJson: ToJson<A>
): ToJson<List<A>>, ToJson<A> by elementToJson {
  override fun List<A>.toJson() = /* implement encoder using the A.toJson() instance that is implicitly in scope */
}

This fails to compile, however, because I am not allowed to define a class that implements the same generic interface with different type arguments (it tries to implement ToJson<List<A>> and ToJson<A> at the same time). The reason is, I assume, due to type erasure. The JVM signatures for the A.toJson() and List<A>.toJson() extension functions are identical, so it is impossible to generate code for the class. This problem does not arise with nested uses of with(..) because the implementations live in different objects.

I think you can workaround this compilation issue using @JvmName. Take a look here: https://twitter.com/kotlin/status/1199251632904495104

[ulrikrasmussen]

@ulrikrasmussen – quoting @elizarov:

On a more philosophical note. In Kotlin corehent behaviors are supported via regular interfaces. A class implements an interface as an evidence of providing certain behavior in a coherent way. We don't need yet another mechanism in the language for that. Typeclasses, as proposed, are designed to solve the problem of adhoc polymorphism, which is useful exactly where there is no need for coherence.

I don't understand this line of reasoning. Because object interfaces are coherent, then typeclasses shouldn't have that property? I also don't understand how ad-hoc polymorphism is supposed to be synonymous with incoherence.

Typeclasses by the way are well suited for implementing compositional behaviors, something that is not well supported by regular interfaces. Returning to the ToJson interface implemented on List, if that was to be a regular interface, then the type parameter has to be bounded in the class definition, like so:

class List<A : ToJson>(): ToJson { ... }

But that prevents me from ever using a list with something that isn't serializable as JSON, which is clearly not what we want. Typeclasses allow me to express this more naturally.

[ulrikrasmussen]

I think you can workaround this compilation issue using @JvmName. Take a look here: https://twitter.com/kotlin/status/1199251632904495104

No, that won't work, for two reasons.

1. It doesn't seem to work for interface declared methods. That is, the compiler rejects my attempt to do the following:

interface ToJson<A> {
    @JvmName("foo")
    fun A.toJson(): String
}

saying that the annotation is not applicable to this declaration.
2. Even if it was, it wouldn't make a difference, unless I could choose a JVM name that actually depended on the type argument (which I can't). The problem is that at compile time, ToJson<A> and ToJson<List<A>> are different interfaces, but due to type erasure, they must both be compiled to the run-time interface ToJson<*>. A class simply cannot implement the same generic interface with different arguments and still be compatible with Java.

[SkittishSloth]

I agree with @ulrikrasmussen - the title of this KEEP is even "Extension Interfaces". My understanding was the point was to introduce a new "type" of interface that can be implemented for pre-existing classes. I can add code and say "an integer is a monoid" and then use it everywhere I'm using monoids. The only difference is that it's added after the integer type is defined, by someone who has no access to the original code.

[ulrikrasmussen]

I actually found an alternative to the "plain Kotlin" pattern proposed by @edrd-f which does not have the limitations I pointed out above. You need to define your extensions in an object inside the body of a function which invokes with on all the extensions that your definition depends on. But then you don't have to invoke with again at the call sites. For example, for the .toJson() example we have:

interface ToJson<A> {
    fun A.toJson(): String
}

fun <A> ListToJson(
    innerToJson: ToJson<A>
): ToJson<List<A>> =
    with(innerToJson) {
        object: ToJson<List<A>> {
            override fun List<A>.toJson(): String =
                "[${joinToString(",") { elem -> elem.toJson() }}]"
        }
    }

Here is another example with two extension constraints:

interface Monoid<A> {
    fun z(): A
    fun A.add(other: A): A
}

fun <A, B> PairIsMonoid(
    left: Monoid<A>,
    right: Monoid<B>
): Monoid<Pair<A, B>> =
    with(left) { with(right) {
        object: Monoid<Pair<A, B>> {
            // I have to call these explicitly on `left` and `right` because Kotlin cannot dispatch on return type.
            override fun z(): Pair<A, B> = Pair(left.z(), right.z())

            override fun Pair<A, B>.add(other: Pair<A, B>): Pair<A, B> {
                val (x1, y1) = this
                val (x2, y2) = other
                // The `.add` extensions on `A` and `B` can be used here.
                return Pair(x1.add(x2), y1.add(y2))
            }
        }
    } }

There may be problems I have overlooked.

[hannomalie]

I agree with @ulrikrasmussen - the title of this KEEP is even "Extension Interfaces". My understanding was the point was to introduce a new "type" of interface that can be implemented for pre-existing classes. I can add code and say "an integer is a monoid" and then use it everywhere I'm using monoids. The only difference is that it's added after the integer type is defined, by someone who has no access to the original code.

I don't think that's the only difference. Regular interfaces have constraints that every instance implementing it must satisfy. With extension interfaces, you can have constraints for a type, which means you can have constraints on "static" things. This is the most important part from my pov.

[SkittishSloth]

I'm not sure I follow, but I want to - can you elaborate a bit more? I think I'm kinda stuck somewhere in type vs class land or something along those lines and that's where my confusion comes from.

[hannomalie]

No problem, the terminology is indeed confusing because many paradigms get mixed here.

When you have an interface, you implement it in your class and than you have to satisfy the constraints of that interface. That means when using the interface as a reference type for any implementing instance, you can use those interface's features. Easy.

interface Struct
class MyStruct: Struct {
  var myField = 0
  companion object {
    val instanceSizeInBytes = 4
  }
}
class MyOtherStruct: Struct {
  var myField = 0
  var myOtherField = 0
  companion object {
    val instanceSizeInBytes = 8
  }
}
...
println(MyStruct().myField) <-- Using an instance here

Look at how each class implements a companion object, because the instanceSizeInBytes is static for a class and only changes when the class's layout changes. Imagine you want to have a function like the following

fun allocateBuffer<T: Struct>(instanceCount: Int) = StructBuffer(ByteBuffer.allocateDirect(instanceCount * T.instanceSizeInBytes)) <-- Impossibru

T.instanceSizeInBytes won't work here, because the companion stuff is not part of the interface. You now have to somehow workaround that... for example you can take a second parameter in the allocateBuffer function that is something like interface StructType { val instanceSizeInBytes } and have you companions implement that. But the user of such an API would still have to pass in the companion instances, which is strange, because he already uses the type parameter, so everything should be clear. And it's also not coherent, which means one can pass in 2-3 different instances and get wrong results because of changing byte sizes...

With the proposed extension interfaces, you can enforce constraints for a type, that means you don't need an instance of that type to do something. The function would look like

fun allocateBuffer<T: Struct>(instanceCount: Int, with StructType<T> structType) = StructBuffer(ByteBuffer.allocateDirect(instanceCount * T.instanceSizeInBytes)) <-- Possible!
...

allocateBuffer<MyStruct>(15)

Depending on how extension interfaces are implemented, T.instanceSizeInBytes has to be written as T.Companion.instanceSizeInBytes or only instanceSizeInBytes. But the point is: The user of the API doesn't need to pass in a strange StructType instance and we can (optionally?) enforce coherence.

Some might find my example odd, but there are really a lot of other use cases, for example factory/constructor constraints, serializer stuff etc.

[fatjoem]

I rewrote the example provided in this keep's proposal by using an ordinary extension function together with the "companion values" proposal (#106).

It would look like this:

interface Repository<A> {
    fun loadById(id: Int): A?
    // ...trimmed for simplicity
} 

class UserRepository: Repository<User> {
    val storedUsers: MutableMap<Int, User> = mutableMapOf()
  
    override fun loadById(id: Int): User? {
        return storedUsers[id]
    }
} 

fun <A> Repository<A>.fetchById(id: Int): A? {
    return loadById(id)
}

object UsageExample {
    private companion val userRepository = UserRepository()

    fun test() {
        val user: User = fetchById(5)
    }
}

I have now done the same thing with a classical monoid example.

In my opinion, using the companion values proposal (from keep 106) leads to more idiomatic kotlin code, than what is proposed in this keep. The classical monoid example would look like this when using companion values:

interface IMonoid<A> {
  fun zero(): A
  fun A.append(other: A): A
}

class StringMonoid : IMonoid<String> {
  override fun zero() = ""
  override fun String.append(other: String) = this + other
}

fun <A> IMonoid<A>.doMonoidStuff(a: A) {
    ...
}

fun main() {
  // the companion val can go into different scopes.
  // in this example we are scoping it locally in this function:
  companion val stringMonoid = StringMonoid()

  val someString = "hi world"
  doMonoidStuff(someString)
}

In this example, everything is ordinary Kotlin, except for the companion val, which is the proposed feature of keep 106. It basically is just syntactic sugar for:

fun main() = with(StringMonoid()) {
  val someString = "hi world"
  doMonoidStuff(someString)
}

[AhmedMourad0]

Most examples here require the interface to have a type parameter (the class we are extending), which is fine if we have control of the interface, but sometimes we actually don't, eg. Android's Parcelable.

I believe the extension class should define both what interface it is implementing and what class it is implementing it for. So building on fatjoem's suggestion:

class StringMonoid : IMonoid for String {
  override fun zero() = ""
  override fun append(other: String) = this + other
}

fun doMonoidStuff(m: IMonoid) {
    ...
}

fun main() {

  private companion val stringMonoid = StringMonoid()

  val someString = "hi world"
  doMonoidStuff(someString)
}

• It has similar syntax structure to delegation.
• It doesn’t force a type parameter on the interface.
• No new keywords are added.
• It can work with both third-party classes and interfaces.
• It is explicit both at declaration and usage.

[ZkHaider]

Why make things complicated with typeclass or extension interface? :-)

Just use the existing syntax provided by Kotlin, and if need be introduce the extension keyword.

Correct me if I am wrong here but Kotlin seems to be mostly used by mobile engineers on Android mostly (due to Googles support), and if this is use-case, again I would call out that keeping parity between Swift would make this PR great!

Consider the following example in Swift.

Swift::

protocol Event {} 

protocol Eventable: class {
  func send(_ event: Event)
}

// I am able to extend Apple’s UIKit class here 
// this gives me amazing flexibility 
extension UIViewController: Eventable {

  func send(_ event: Event) {
    // Implementation...
  }
}

struct AnEvent: Event {}

// Can reference only Eventable now
let eventables: [Eventable] = //...
eventables.forEach { 
  $0.send(AnEvent())
} 

Now it would be amazing if I do the following in Kotlin:

Kotlin:

interface Event {}

interface Eventable {
  fun send(event: Event)
}

// Extend Androids APIs giving me 
// amazing flexibility 
extension Fragment: Eventable {

  fun send(event: Event) {
    // Implementation..
  }
}

object AnEvent: Event {}

val eventables: List<Eventable> = //...
eventables.forEach {
  it.send(event = AnEvent())
}

Now Fragment an Android API which was not possible to allow more conformance via interface can be extended with another interface via extension.

Furthermore I can now hold an array of List<Eventable> and Fragment would be included in that. This keeps my code concise and elegant.

What is preventing the above implementation? :-)

• Has parity with Swift
• Easy to read

[LouisCAD]

@ZkHaider The JVM and ART VM prevent this from working with interfaces. It would need to be a special kind of interfaces that would behave like protocols in Swift/Obj-C rather than like interfaces in Java.

I've been thinking about inline interface but I wasn't completely agreeing with that naming regarding what it actually is.

[uberto]

Having re-read all this thread after some years, I still think the proposal from @elizarov on Oct 2, 2017 is the best one.

Looking at the Swift world, there are lots of use for Self (which doesn't seem more magic than this to me)and ad-hoc extension which are not heavily related to category theory or Haskell type system.

[elizarov]

This issue has had some awesome discussions, rewrite, and a lot of other evolution. Thanks a lot to @raulraja personally and to all the other contributors who are too numerous to name them here. One thing became clear: there is an enormous community interest in adding some kind of extension interfaces to Kotlin. 🚀

The main use-case that gets people excited about this feature is almost available today in Kotlin via extension functions, so we have decided that we will double-down, first, on this particular Kotlin language design aspect and will extend it to support multiple receivers. Not only this will address the long-standing irregularities in the original Kotlin design but will enable many of the use-cases that were mentioned in this discussion without having to introduce entirely new concepts into the language.

This work on multiple receivers is tracked at issue 👉 https://youtrack.jetbrains.com/issue/KT-10468

The abovementioned issue does not have a concrete design proposal yet, but lists some key use-cases and basic requirements that were already identified. Please, vote for it, comment on your specific use-cases for multiple receivers and continue discussions there 👍

I understand that multiple receivers will not cover the full scope of features that were envisioned by the original "extension interface proposal". At the current stage of the design process, there are no provisions for any kind of automated creation/import or another kind of acquisition of "extension interface" instances beyond them being explicitly brought into the scope. However, just by itself, it is a big change to the language, so it is imperative to take small and careful steps. The very support for multiple receivers is the necessary first step that will enable other improvements in the future. It is not enough to introduce a new feature into the langauge. We also need to learn how to use it effectively to improve the clarity of code that we write in Kotlin. There are many open questions down this road, and that is why we are starting just with a prototype.

Having said that, I'm closing this issue to shift our focus onto the shorter-term goal of supporting multiple receivers in Kotlin first.

[raulraja]

Thanks so much, @elizarov. We look forward to multiple receivers and appreciate the work the Kotlin team is putting in the compiler to accommodate these community use cases 🙏