kotlin support request

[MartinDong]

please support convert to kotlin

[liujisi]

Right now it is not planned. We recommend using the Java bindings instead given the interoperability, mostly due to the code size bloat when both kotlin and Java are generated in the same app. Will keep this posted when anything changes.

[saturnism]

Kotlin support would be so much better than the current Java builders. :)

[banshee]

Don't forget that Kotlin can compile to non-jvm environments, so just saying 'use java' actually doesn't solve the kotlin problem.

[xenomachina]

If a Kotlin code generator is ever created, it would be great if "oneofs" were turned into sealed classes, rather than having the "*Case" enum.

Also, given how nice null handling is in Kotlin, I wish optional fields could simply be implemented as nullable properties, rather than having the separate "has*" methods.

[christophsturm]

Btw kotlinx.serialization supports protobuf already without any code generation.

_{Sent with GitHawk}

[banshee]

@christophsturm that's interesting, but it doesn't seem like it really addresses the issue. The problem is that you usually start with a .proto file that you really do want to use for code generation, and it looks like there's only partial support for protobuf anyway - Kotlin/kotlinx.serialization#67.

[christophsturm]

yes it has a different focus and i would love to see kotlin code generation in protobuf.

[cretz]

I am toying around with implementing this.

The ideal situation would be to generate all the required data classes (yes, that means immutable and sealed hierarchies for oneofs) and abstract the sizing, marshalling, and unmarshalling via multiplatform expect/actual classes and defer to existing protobuf serialization code for the three primary targets (JVM, JS, native). Maybe could have hand written serialization, but the existing language impls are highly optimized, especially in the JVM case. Also, for protoc-gen-kotlin, I am bootstrapping with the JVM impl for now but once complete, I am planning on it actually using Kotlin native so it can ship as a single binary (so I'm not using KotlinPoet). I am leaving enumerates open ended as data classes w/ a single int value and the known values as vals on the companion object. I may only support proto3 at first, we'll see.

I will update if/when complete.

[cretz]

Initial version complete: https://github.com/cretz/pb-and-k. I'll broadcast to Kotlin forum and/or Reddit tomorrow so no need to clutter up this issue discussing. It only has basic features now, but I'll implement more if the project becomes popular.

[daudrain]

An another projet looks promising kroto-plus

[lowasser]

This is a project we are (and specifically I am) working on for Google, with an unstable prototype landing in open source sometime in the late Q2-early Q3 range.

Some specifics, just to be clear:

• We're initially focusing on building extensions atop the generated Java APIs. A major goal of Kotlin is to support incremental migration of existing Java projects without compatibility hassles; we would intend the same approach here. Down the line, we might investigate generating pure Kotlin, but it would be API compatible with the generated Java APIs.
• Oneofs can't be sealed without breaking on future additions to the oneof, which are supposed to be compatible changes. I think we'll be generating something that "looks like" the sealed class would, except for the sealed keyword.
• We're still discussing nullable representations of optional fields.

[jeremiemartinez]

@lowasser interesting! Can we expect data class generation rather Java builders?

[lowasser]

Neither. Data classes as in data class have difficult conflicts with ways protos are supposed to be able to evolve forward-compatibly. (For example, new fields can be inserted "between" old fields.) We're thinking in terms of DSLs:

myMessage {
   field1 = 5
   field2 = "foobar"
}

...which is not actually that different, all things considered, from

MyMessage(
  field1 = 5, 
  field2 = "foobar"
)

[lowasser]

I've posted https://github.com/lowasser/protobuf/blob/master/kotlin-design.md , which is a complete description of the design our internal process came up with.

[marcoferrer]

@lowasser it’s really great to see this starting to gain some attention. I author a library that provides dsl generation for proto messages https://github.com/marcoferrer/kroto-plus

I still think it falls short of true first party kotlin support though.

A couple of notes I wanted to add. One thing we experienced early on was the need to have message types annotated with kotlins DslMarker annotation. This provides compile time safety so that one nested lambda doesn’t mutate the scope of a parent lambda.

Also I saw that you mentioned that there is a separate discussion for improvements to grpc apis. Is this public yet? The kroto project has been working on refining grpc integration with coroutines and I wouldn’t mind helping contribute

[dsilvasc]

@lowasser I understand the design constraints might be different for google's kotlin generator (for instance with regards to positional args, data classes, or a copy method), but if your team hasn't yet, it might be helpful to take a look at the approach @thesamet took for ScalaPB:

• support for optional scalars from proto2 sources, repeated fields as immutable vectors, and map fields as immutable maps:
  https://scalapb.github.io/generated-code.html#messages

• withX methods for updating field X in a message, an update method that takes a list of mutations, and mutations on deeply nested fields:
  https://scalapb.github.io/generated-code.html#updating-messages

• all proto objects immutable

• oneofs as sealed types:
  https://scalapb.github.io/generated-code.html#oneof-fields

• alternative representation for oneofs:
  https://scalapb.github.io/sealed-oneofs.html

• google's well-known wrapper types for proto3 scalars as optionals:
  https://scalapb.github.io/customizations.html#primitive-wrappers

• support for wrapping primitives in inline classes:
  https://scalapb.github.io/customizations.html#custom-types

• It's a replacement for the protoc java output so your application doesn't have to pay the cost of generated code twice, but there is an option to generate interop code:
  https://scalapb.github.io/generated-code.html#java-conversions

Although the target language for ScalaPB is different, some of the ideas would translate fairly well to a Kotlin code generator.

[lowasser]

@marcoferrer
The DslMarker annotation is already included in our prototype implementation. I'd be interested to hear what you think the gap is for first-party Kotlin design -- just direct Kotlin source generation at all levels, or something else?

No, the gRPC design isn't public yet. I am aware of kroto+, and have reviewed its design, and I think we'll be doing something along similar lines, but with a number of small differences -- including keeping the proto enhancements separate from the gRPC enhancements. Please feel free to contact me personally, though, and we can talk about that. I'd prefer to keep the proto issue to the proto discussion :)

[lowasser]

@dsilvasc Point by point:

• support for optional scalars from proto2 sources, repeated fields as immutable vectors, and map fields as immutable maps:
  https://scalapb.github.io/generated-code.html#messages

We already do surface repeated fields as immutable lists and map fields as immutable maps. (I don't believe there's currently a way to expose the unmodifiable interface while keeping the base proto in Java, unless there's an annotation available for that...?) Optional scalars is something we may well do for proto2, though we're mostly focusing on proto3 right now. (If I didn't list that in the "Open Questions" section, that was an oversight on my part.)

• withX methods for updating field X in a message, an update method that takes a list of mutations, and mutations on deeply nested fields:
  https://scalapb.github.io/generated-code.html#updating-messages
  I think we're seeing that as all-encompassed by the copy method: myMessage.copy { field = foo } is about even with myMessage.withField(foo). Deeply-nested changes aren't too bad either: myMessage.copy { field = field.copy { subField = foo } }.

• all proto objects immutable
  This is already part of the Java API.

• oneofs as sealed types: https://scalapb.github.io/generated-code.html#oneof-fields
  FWIW, the doc discusses this as an idea we specifically considered and decided was actively a bad idea, because protos are supposed to support evolution. It doesn't actually make that much difference, FWIW.

• google's well-known wrapper types for proto3 scalars as optionals:
  https://scalapb.github.io/customizations.html#primitive-wrappers
  Definitely falls under the category of the "Open Questions" discussion.

• support for wrapping primitives in inline classes:
  https://scalapb.github.io/customizations.html#custom-types
  Huh. Interesting. I feel kind of weird about this, and I'm not sure the implicit type mapping is in scope.

[thesamet]

@lowasser - can you clarify why sealing oneofs inhibits proto evolution? In ScalaPB, we generate an Empty case, so users can pattern match on Empty to handle unexpected future developments. The advantage of sealing is, of course, that you can get a compiler warning if there are unhandled cases. Agree this is not much of a difference, but as author/maintainer of ScalaPB I'd be happy to fully understand the perspective that considers this an actively bad idea.

[lowasser]

@thesamet In Kotlin, my understanding is that an unhandled case in a when-expression is an error, not a warning. When expressions are required to be exhaustive. That makes adding new cases a source-incompatible change. (Demo: https://pl.kotl.in/yWJloqTKg, for a scenario where code that worked before the addition of DoubleOneof stops compiling when it's added.)

[lowasser]

(I'm actually a little surprised to hear it's not an error in Scala -- what actually happens when there's an unhandled case in a match expression?)

[thesamet]

@lowasser Thanks for clarifying. In Scala you get a warning at compile time and a MatchError exception at runtime: https://scastie.scala-lang.org/RWeVMPvoSJypVulnb8DjLw. Some people use a compiler option to promote warnings into errors so they don't miss on cases.

Taking an optional field removal as a potential proto evolution step: this breaks source compatibility for all compiled languages since getters/setters will not be generated. I personally find it desirable to get a compiler warning/error at the use-site when a case or a field is added or removed so I know everything is being handled. Also is scala, it is easy to fix for future unknown cases by pattern matching on _ (which is a catch all).

[lowasser]

We've never supported optional field removal as a source-compatible change -- do any language proto APIs? -- but supporting adding new fields without breaking existing code's compilation has always been a business-critical use case for us. As far as I know, all the other officially supported proto APIs have similar support.

It's easy to handle unknown future cases in Kotlin, too -- just have an else branch in your when, which behaves analogously to how you describe _. Omitting the sealed keyword on the class really only means users should match else instead of NotSet, which seemed like an acceptable balance to guarantee protos can evolve the ways we expect without breaking builds or tests.

[dsilvasc]

Giving users an option to represent oneofs as sealed would be helpful for teams who want their source to break when they update their schemas -- teams who want to be forced to handle new possible cases.

The java code generated for enums already does this and it's helpful that teams can choose on a case-by-case basis whether they want source-breaking exhaustive matches or source-forward-compatible else/_ branches (in addition to the generated Unknown case, which is different from default/NotSet).

[lowasser]

An option to make oneofs effectively sealed seems like a reasonable addition, though in some ways I wonder if that should really be a Kotlin-exclusive feature. Do other languages for protobufs already have such an annotation? Or was it suggested, and rejected?

[dsilvasc]

I don't think sealed hierarchies are easily expressible in the languages targeted by this repo (Java, C++, Python, ObjC, C#, JS, Ruby).

In languages with sealed types (Scala, Rust, Swift), their protobuf code generators opted for representing oneofs as sealed:

https://github.com/apple/swift-protobuf/blob/master/Documentation/API.md#oneof-fields
https://github.com/danburkert/prost#oneof-fields
https://github.com/stepancheg/rust-protobuf/blob/ad2518b81351399ab8f0e6e3a88367e0b066399c/protobuf-codegen/src/oneof.rs#L196

[TeBoring]

Sorry, we don't have time to support this in the near future. Close this for now.

[aberasarte]

There is an official initial release, you can check it here https://github.com/grpc/grpc-kotlin.

[jan-xyz]

There is an official initial release, you can check it here https://github.com/grpc/grpc-kotlin.

This is the part to generate Kotlin gRPC services, right? The generated messages are still Java, or am I wrong?

[lowasser]

There have been some significant hangups; the protobuf team has expressed interest in owning this in the long term and making Kotlin a fully supported language for protos, but requires the code generation to be rewritten in C++ for a number of reasons. That work is ongoing. Additionally, we're continuing to work on both optimization and on future-proof-ness; for example, we're leaning towards dropping the oneof class feature at least for the initial release, because we think it's probable that Java will want to add the same feature down the line and we'd very much want to avoid that breaking compatibility.

The related but separate project of Kotlin coroutine APIs for gRPC did indeed release yesterday, but that's not the same thing.

[garyp]

@daudrain This looks interesting! Thank you for the pointer! One thing I can't figure out: does it have support for kotlin.js?

It looks like Wire recently switched to Kotlin Multiplatform. They don't say anything about JS or Native support on their website but they are publishing a Kotlin/JS artifact: https://search.maven.org/artifact/com.squareup.wire/wire-runtime-js/3.1.0/jar

Another option is https://github.com/streem/pbandk (I'm one of the maintainers). pbandk is a Kotlin Multiplatform protobuf library with support for JVM and JS (and Native support is in review).

[oleksiyp]

Here is my attempt to enhance protobufs with builders: #5341

I just want to remind you about this possibility in case this technique is forgotten.
Not sure if it is the best one, especially in relationship to multiplatform.

All it's doing it is generating additional extension functions in protoc.

import com.example.tutorial.*
import com.example.tutorial.AddressBookProtosDSL.PhoneNumber.buildPhoneNumber
import com.example.tutorial.AddressBookProtosDSL.buildAddressBook
import com.example.tutorial.AddressBookProtosDSL.buildPerson

fun main(args: Array<String>) {
    println(buildPerson {
        id = 2
        name = "fff"
    })

    println(buildPhoneNumber {
        number = "123"
    })

    println(buildAddressBook {
        people {
            addPerson {
                id = 1234
                name = "John Doe"
                email = "[email protected]"
                dedicatedPhone {
                    type = AddressBookProtos.Person.PhoneType.WORK
                    name = "abc"
                    number = "2222-3333"
                }
                phones {
                    addPhoneNumber {
                        type = AddressBookProtos.Person.PhoneType.HOME
                        name = "def"
                        number = "555-4321"
                    }
                    addPhoneNumber {
                        type = AddressBookProtos.Person.PhoneType.WORK
                        name = "ghi"
                        number = "555-3421"
                    }
                }
            }
        }
    })
}

// Generated by the protocol buffer compiler.  DO NOT EDIT!
// source: tutorial.proto

package com.example.tutorial;

object AddressBookProtosDSL {
  fun buildPerson(
    block: com.example.tutorial.AddressBookProtos.Person.Builder.() -> Unit
  ) = com.example.tutorial.AddressBookProtos.Person.newBuilder()
    .apply(block)
    .build()

  object PhoneNumber {
    fun buildPhoneNumber(
      block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
    ) = com.example.tutorial.AddressBookProtos.Person.PhoneNumber.newBuilder()
      .apply(block)
      .build()

  }

  fun buildAddressBook(
    block: com.example.tutorial.AddressBookProtos.AddressBook.Builder.() -> Unit
  ) = com.example.tutorial.AddressBookProtos.AddressBook.newBuilder()
    .apply(block)
    .build()


  // @@protoc_insertion_point(outer_dsl_object_scope)
}

fun com.example.tutorial.AddressBookProtos.Person.Builder.phones(
  block: MutableList<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>.() -> Unit
) {
  addAllPhones(
    mutableListOf<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>()
      .apply(block)
  )
}

fun com.example.tutorial.AddressBookProtos.Person.Builder.dedicatedPhone(
    block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
) {
  dedicatedPhoneBuilder.apply(block)
}

fun com.example.tutorial.AddressBookProtos.AddressBook.Builder.people(
  block: MutableList<com.example.tutorial.AddressBookProtos.Person>.() -> Unit
) {
  addAllPeople(
    mutableListOf<com.example.tutorial.AddressBookProtos.Person>()
      .apply(block)
  )
}

fun MutableList<com.example.tutorial.AddressBookProtos.Person>.addPerson(
    block: com.example.tutorial.AddressBookProtos.Person.Builder.() -> Unit
) {
    add(
      com.example.tutorial.AddressBookProtos.Person.newBuilder()
       .apply(block)
       .build()
    )
}

fun MutableList<com.example.tutorial.AddressBookProtos.Person.PhoneNumber>.addPhoneNumber(
    block: com.example.tutorial.AddressBookProtos.Person.PhoneNumber.Builder.() -> Unit
) {
    add(
      com.example.tutorial.AddressBookProtos.Person.PhoneNumber.newBuilder()
       .apply(block)
       .build()
    )
}

fun MutableList<com.example.tutorial.AddressBookProtos.AddressBook>.addAddressBook(
    block: com.example.tutorial.AddressBookProtos.AddressBook.Builder.() -> Unit
) {
    add(
      com.example.tutorial.AddressBookProtos.AddressBook.newBuilder()
       .apply(block)
       .build()
    )
}

[edvorg]

For those who is still in need for Kotlin protobuf support, we successfully applied pbandk after latest release 0.8.1 which updates the runtime to latest Kotlin and Kotlin serialization.

We're able to use same schema library on frontend and backend making it easier to write crossplatform Kotlin code.

[tandel-pratik]

For those who is still in need for Kotlin protobuf support, we successfully applied pbandk after latest release 0.8.1 which updates the runtime to latest Kotlin and Kotlin serialization.

We're able to use same schema library on frontend and backend making it easier to write crossplatform Kotlin code.

Are there examples of people using pbandk with gRPC servers? There don't seem to be a ton of examples of gRPC servers using the pbandk bindings.

[garyp]

Are there examples of people using pbandk with gRPC servers? There don't seem to be a ton of examples of gRPC servers using the pbandk bindings.

@pratik-brex There isn't explicit support for gRPC in pbandk. But there is a generic service code generation mechanism which is intended for use cases like gRPC: https://github.com/streem/pbandk#service-code-generation. I'm not aware of anyone actively using it yet, so do let us know your feedback/suggestions if you try it.

[YuriDenison]

@lowasser It's been over 7 month since your last update on Protobuf team plans for first-class Kotlin support.
Could you please share current status with community?

[RaphaelTarita]

I would also want to see an update here. I'd like to use gRPC in my Kotlin application, yet the not at all idiomatic code generated by protoc-java is really deterring me from using it. Google has made great efforts in embracing Kotlin, especially in the Android world, but just saying that Kotlin is the preferred language for this and that is not enough. Kotlin stands out because of its unique and ergonomic language features. Simply translating Java code into Kotlin completely defies the point of using Kotlin instead of Java.

In my opinion, writing and generating Kotlin-idiomatic code instead of Java-like code (as explained in Koans) should be an integral part of promoting Kotlin as, for example, the preferred Android language. This also includes proper Kotlin support in both ProtoBuf and gRPC.

Probably one of the most important features that a hypothetical protoc-kotlin compiler should have is to implement the officially promoted type-safe builder pattern instead of a Java-style builder pattern. This has already been addressed in this discussion.

Concerning the discussion about sealed classes, a few points seem to be relevant to add:

1. Since the discussion is quite old by now, things have changed: sealed classes are now not only a thing in Kotlin, but also an experimental feature in Java 15. I think that by now the discussion is almost language-agnostic, since closed polymorphism becomes more and more popular and gets added to more and more languages.
2. Adding an else branch in an (exhaustive) when-expression is as well possible when working with sealed classes. It is however true that this might issue a warning because it is unnecessary in an exhaustive case.
3. when-statements do not have to be exhaustive in any case. As well as if-statements, they only have to cover all outcomes if and only if the statement is used as an expression (which makes kinda sense). Also, the presence of an else branch automatically makes any when-expression exhaustive (but then again, it might issue warnings..)

Anyway, I hope that this feature is still being worked on, because I really think it is a necessity as of today. Since the discussion is already 5 months old, I am not sure about the state of third-party Proto/gRPC for Kotlin projects. If anyone can point me towards the currently active projects, that would be very nice.

I think we deserve some kind of update on this feature by now.

[lowasser]

The state is that yes, it is still being worked on extremely actively; no, there is not much to show for it publicly yet.

Some things that have happened in the past few months:

• the code generator has been rewritten entirely in C++ as part of the main protobuf compiler, in particular to make it easier for the main protobuf team to own the project long term and make it easier to open source
• the design has been somewhat simplified and thoroughly reviewed, because pretty literally every bit of Google's Kotlin code will use this and backwards-incompatible changes would be extraordinarily painful; the sealed class oneofs are being left off for the moment; the typesafe builder pattern is clearly the best way to do this in Kotlin and that's what we're doing
• the protobuf team has hired a new employee who is working on this as one of their main projects

[andrewparmet]

If you're looking for third-party support, as mentioned above (disclaimer: I'm a developer), https://github.com/open-toast/protokt generates idiomatic Kotlin code and supports ServiceDescriptor and MethodDescriptor generation that is compatible with both grpc-java and grpc-kotlin. It does not interoperate at the Java/Kotlin level with code generated with protobuf-java, which is something the protobuf team is doing. It has the potential for cross-platform functionality but that's not currently in the roadmap. It is used in production at Toast.

If you're looking for a solution that does have interop with protobuf-java's generated code, you might look at https://github.com/marcoferrer/kroto-plus. My guess is that it will be mostly straightforward to migrate from that to an official library when that time comes.

Note also that you can use grpc-kotlin with the Java bindings that exist today and get coroutine support. Later, when an official Kotlin generator is released, you can migrate message building separately.

[garyp]

To add another option, https://github.com/streem/pbandk generates idiomatic Kotlin code and supports Kotlin Mulitplatform (JVM, JS, and Native). Disclaimer: I'm the primary maintainer of the project. We use it in production at Streem in our backend and Android code.

pbandk is pretty similar in spirit to protokt. I'd say the main differences with protokt are that protokt includes built-in support for gRPC (which pbandk lacks), but pbandk includes support for JSON and Kotlin Multiplatform (which protokt lacks). pbandk also focuses on implementing serialization/deserialization in the runtime library, whereas protokt implements it in the generated code. There are of course lots of smaller differences too.

pbandk currently generates Kotlin data classes for each proto message, which are idiomatic Kotlin but run into some backwards-compatibility concerns when the proto message definition evolves (as discussed above). We plan to replace those with Kotlin-style type-safe builders in an upcoming version.

[RdeWilde]

@lowasser Is there any date that is being aimed for? I think many people would like to support the project, but as long as it's not out in the open, no one can.

[glerchundi]

It seems like something is happening in the repo! 17 days ago an interesting PR landed into master!

Does this mean that we're that close? 😍

@lowasser @perezd

[LouisCAD]

Does it support unsigned integers that are now stable in Kotlin 1.5?

[Fleshgrinder]

Looking at what landed in master it seems that no Kotlin code is generated, it's all Java and the existing generated code is extended via extension functions and additional wrapper types. I actually feared that we are not going to see idiomatic Kotlin code and only get overhead. If this sounds negative then it does because it is. I would wish so much to have some clean Kotlin code with all its possible beauty over Java, e.g. no more has* functions but actual nullability. Protobuf is often a foot gun because people don't understand that it always gives you something and that you have to call other functions to find out what is going on. Similar situation with the open Enums which are not supported by Java but could easily be created with sealed classes in Kotlin. Sure, Java 17 will have sealed support but 20 years will go by until we can move to Java 17 only...

[bubenheimer]

See also my recent request to actively support kotlinx.serialization in addition to protobuf in the grpc-kotlin project. Grpc is not strictly tied to protobuf. grpc/grpc-kotlin#255

[lowasser]

We were holding off on posting specifically about this because we were planning to mention it at I/O, but as that talk has gone up, yes, we’ve launched Kotlin support in protobuf.

There were definitely some design decisions that disappointed people – including me – but we think we made them for good reasons. Most of them boil down to the fact that we see Java compatibility as a business requirement: almost all of our Kotlin codebases at Google, and most Kotlin codebases in the world, are mixed with Java. For example, this is why we haven’t adopted unsigned types, and are continuing to use Java hazzers and the like.

For many features where Kotlin adds potential value, we wanted to be sure that we have a plan for ensuring compatibility with Java even if Java protos add the same features down the road. For example, Java is finalizing sealed types in September with Java 17, and optionality was added “back” into proto3 as recently as February. We thought it was best to hold off on adding special oneof support via sealed types, or orNull extension properties for optional fields, until these issues were settled. We don't need to wait until those features are actually added to Java protos, but we needed enough information to make a compatibility plan.

We've carefully considered a lot of design space, and we will continue to evolve protobufs for Kotlin, likely including the features discussed above. It’s even conceivable that someday we might compatibly convert protobufs for Java to depend on Kotlin, instead of the other way around, to try to achieve an ideal experience for developers of both languages. However, such an effort would have to capture the years of engineering effort that have gone into optimizing protos for Java, especially the “lite runtime” for Android. Those optimizations are mission critical for our applications, so we can’t “just” do a complete rewrite.

In any event, we hope you find value in the Kotlin support we’ve released, and we look forward to improving it in future. If you have new feature requests for Kotlin protos, please file those as separate issues.

[RdeWilde]

Awesome, thank you so much