Carbon <-> C/C++ interoperability #80

jonmeow · 2020-06-16T23:40:06Z

Co-authored by: chandlerc

This includes the core structure of an interoperability design. It is not complete, and I've tried to indicate key missing pieces with TODOs. These portions would ideally be addressed in the future, as part of other proposals.

Move firebase support into a src directory. (#15)

Update fork

Merge

geoffromer

For the record, I don't consider my top-level concerns with this PR resolved, and I'm not confident I'll be able to affirm this proposal until they are, but I think for now they're best discussed in the context of PR #83 (see forum post).

geoffromer · 2020-07-23T17:18:30Z

docs/design/interoperability/README.md

+  - Carbon must be able to compile C++ headers in order to translate names and
+    types.


This is certainly true, but the toolchain used to compile C++ object files doesn't need to be the same as the toolchain that compiles Carbon object files. Allowing them to differ could permit a much more incremental migration: rather than a massive up-front project to switch C++ toolchains (and standard libraries) across the codebase before you even start using Carbon, you could incrementally migrate C++ headers if and when they are included from Carbon.

Noted, added "even if it's not used to compile the C++ object files"

I think your statement about migrating headers is off though, as if you don't have the underlying .cc files compiling with the Carbon toolchain, then Carbon tools shouldn't be expected to be able to migrate the code. Similarly, that cc code may not be able to call into Carbon, for the reasons noted in the next bullet.

geoffromer · 2020-07-23T17:20:14Z

docs/design/interoperability/README.md

+  - While arbitrary C++ code may be able to call into Carbon code that has been
+    pre-compiled into a library, a more complex interaction like C++ code
+    calling Carbon templates requires compilation of _both_ languages together.


Could this take the form of e.g. compiling the Carbon template to portable C++ (rather than Clang AST), for use by the user's C++ toolchain?

This seems like an infeasible constraint on Carbon to me, considering it's mainly for templates. Noted as an alternative approach.

geoffromer · 2020-07-23T17:35:27Z

docs/design/interoperability/primitive_types.md

+$if platform == LP64
+fn ToCLong(var Int64: val) -> Int64 { return val; }
+$else
+fn ToCLong(var Int64: val) -> Int32 { return (Int32)val; }


My comment isn't about the particular usage depicted in the example, it's about any possible usage. As far as I can tell my comment is still accurate, but evidently it's still not clear. Maybe we should set up a VC to try to get on the same page about this?

geoffromer · 2020-07-23T18:28:32Z

docs/design/interoperability/primitive_types.md

+  - For example, users may still write platform-specific code like
+    `var Cpp.long: x = ...; var Int32: y = (Int32)x;`.


How is this any different from the user writing code like var Int32: x = CppCompat.FromCLong(...), or var auto: x = (Int32)...;?

Pragmatically, it's not that different. That's why "may still" instead of "uniquely" -- the point is to emphasize that this solution has problems, too.

geoffromer · 2020-07-23T18:40:06Z

docs/design/interoperability/primitive_types.md

+  - If platform-specific types are added, it may be worth considering whether we
+    should promote these to Carbon primitive types.


We might as well start considering it now, because I guarantee you that people will introduce types like Cpp.long if Carbon doesn't provide them. I can't be sure how prevalent they will be, but I'm not going to be the only one who will be uncomfortable using an inherently type-unsafe API like the one proposed here.

Thank you for being explicit, and I think I'd correctly guessed your leaning from your other comments. I think my response is captured on the line above.

geoffromer · 2020-07-23T18:46:19Z

docs/design/interoperability/primitive_types.md

+  - These types are likely to leak beyond C++ interoperability code, creating
+    friction in using APIs that are designed either only for variable-size types
+    or for fixed-size types, hindering API reuse. Overlapping implementations
+    and increased maintenance costs are a likely result.


These arguments still seem too vague and speculative to be persuasive to me. Won't the very friction you're concerned about tend to discourage the API leakage that would cause it? A concrete example might help me envision why this leakage is likely to happen, and why it's likely to be harmful.

Sorry, I'm not sure I can provide concrete examples, other than to say that it's my understanding Swift has seen similar problems with their Obj-C layer that they didn't anticipate.

I view this as something that could flip in the decision, but my sense is variable size types probably won't be accepted -- thus the choice of defaults. Unless there's more discussion in the review that clearly indicates a leaning in the other direction, I'll leave it as is.

docs/design/interoperability/vocabulary_types.md

geoffromer · 2020-07-23T18:59:29Z

docs/design/interoperability/vocabulary_types.md

+### Mapping similar built-in types
+
+When it is not possible to convert a non-owning reference or pointer to a C++
+data structure or vocabulary type into a suitable Carbon type, the actual C++


gribozavr

Sending the comments that I wrote so far, only reviewed two files.

gribozavr · 2020-07-15T17:27:45Z

proposals/p0080.md

+
+## Acknowledgements
+
+This borrows significantly from the structure of Swift's interoperability plan


Suggested change

This borrows significantly from the structure of Swift's interoperability plan

This proposal borrows significantly from the structure of Swift's interoperability plan

docs/design/interoperability/README.md

gribozavr · 2020-07-23T08:36:49Z

docs/design/interoperability/README.md

+    vocabulary types.
+- Mappings should be easy to maintain.
+  - We should provide a syntax for transparently, automatically exposing a
+    subset of Carbon types and interfaces to C++ code without custom bridge


Do you really mean Carbon interfaces or just Carbon APIs in general, including free functions and constants that are not nested within a type?

gribozavr · 2020-07-23T08:38:55Z

docs/design/interoperability/README.md

+
+Non-goals:
+
+- We will not make Carbon -> C++ migrations as easy as C++ -> Carbon migrations.


I feel like you meant "interop" instead of "migration". I don't think we are designing any support for Carbon -> C++ migration (so I find it surprising to even mention it here), but calling Carbon from C++ is necessary for C++ -> Carbon migration.

So to avoid confusion I'd say "We prioritize calling C++ from Carbon. Calling Carbon from C++ will not be necessarily as easy."

Done, roughly.

gribozavr · 2020-07-23T15:50:03Z

docs/design/interoperability/README.md

+
+The design for interoperation between Carbon and C++ hinges on:
+
+1. A focus on types, and simple overload sets built from those types.


I'm not sure what "a focus on types" means. Does it mean that interop for free functions would be not as good as interop for structs?

I'm not sure either. :)

How about:

The ability to interoperate with a wide variety of code, such as
classes/structs, not just free functions.

docs/design/interoperability/README.md

gribozavr · 2020-07-24T14:08:40Z

docs/design/interoperability/README.md

+
+> References: [Templates and generics](templates_and_generics.md).
+
+Carbon generics will require bridge code that hides the generic. This bridge


I'm not sure why -- is there some implementation difficulty that you foresee? Exposing a Carbon generic as a C++ template should be rather doable even if Carbon generics are compiled separately.

I'm not clear that's that straightforward, given how generics are done. But maybe a template could be auto-generated? I haven't really thought that through. Added an open question to templates_and_generics.md

Indeed, it seems substantially harder to expose Carbon templates than to expose Carbon generics. This makes me wonder whether "template" and "generic" somehow got reversed in these two sections, and what we mean is that Carbon templates must be wrapped in a Carbon generic before they can be exposed?

gribozavr · 2020-07-24T16:03:50Z

docs/design/interoperability/enums.md

+  DIRECTION_WEST,
+  DIRECTION_NORTH,
+  DIRECTION_SOUTH,
+} __attribute__((carbon_enum("East:West:North:South"));


In Swift we instead strip a common prefix from enumerators.

Good point, added a note.

gribozavr · 2020-07-24T16:04:36Z

docs/design/interoperability/enums.md

+
+## C/C++ enums in Carbon
+
+C++ enums will generally translate naturally to Carbon, whether using `enum` or


It would be good to mention the cases that don't fall into the general pattern.

C and C++ APIs sometimes (how often? IDK) rely on enums being implicitly convertible to integers. Ignoring this issue will lead to some APIs being non-ergonomic so it is OK to punt on it being ergonomic, but we should provide a technical ability for C++ enums specifically. It might be the case that by default Carbon enums will not be convertible to integers at all to avoid even a remote possibility of anyone relying on numeric values (I'd certainly argue for it).

C and C++ APIs also sometimes cast arbitrary bit patterns into enum values, which Carbon enums might decide to prohibit. Ignoring this issue will lead to miscompiles, so I think we have to think about it. Maybe the Carbon enums imported from C++ should not be assumed to have free bit patterns.

Adding open questions to cover this. Is that sufficient for now?

gribozavr · 2020-07-24T16:06:47Z

docs/design/interoperability/enums.md

+}
+```
+
+We would expect to generate equivalent C++ code:


The C++ code will likely need an attribute to specify the size of the enum (sizeof(Direction)), because the size should match the Carbon ABI exactly to enable direct bridging, and Carbon should feel free to choose and change its rules about determining the enum size.

BTW, I'd really like this proposal to introduce some terms around bridging, specifically:

(1) a term that denotes zero-cost bridging of types that have identical memory layout in C++ and Carbon,
(2) a term that denotes bridging that involves running code that converts the memory layout (the code can be either compiler-written or user-defined, does not matter),
(3) (maybe) a more specific version of (2) where we create an independent copy of the value,
(4) (maybe) a more specific version of (2) where we destroy the source of the value being bridged.

Is the attribute something that exists? For now, I've added an open question. Although, would setting appropriate types on enum classes actually solve it?

For terms, any suggestions for a glossary? Does Swift have such terms?

Co-authored-by: Dmitri Gribenko <[email protected]> Co-authored-by: Geoff Romer <[email protected]>

Co-authored-by: Geoff Romer <[email protected]>

jonmeow

Partly through comments, aware I still have some left.

jonmeow · 2020-07-24T22:10:51Z

docs/design/interoperability/README.md

+    vocabulary types.
+- Mappings should be easy to maintain.
+  - We should provide a syntax for transparently, automatically exposing a
+    subset of Carbon types and interfaces to C++ code without custom bridge


Sure, I think it's fine to change this to APIs

jonmeow · 2020-07-24T22:14:36Z

docs/design/interoperability/README.md

@@ -0,0 +1,346 @@
+# Carbon &lt;-> C/C++ interoperability


This is trying to echo the BLUF style, so I think it's better to keep, even though it can get out of date. The intent is to summarize, and help people get a picture in their head before they delve into details (if they even choose to).

I can move the goals and philosophy out, though.

jonmeow · 2020-07-24T22:22:12Z

docs/design/interoperability/README.md

+
+Non-goals:
+
+- We will not make Carbon -> C++ migrations as easy as C++ -> Carbon migrations.


Done, roughly.

jonmeow · 2020-07-24T22:25:56Z

docs/design/interoperability/README.md

+  - Carbon must be able to compile C++ headers in order to translate names and
+    types.


Noted, added "even if it's not used to compile the C++ object files"

I think your statement about migrating headers is off though, as if you don't have the underlying .cc files compiling with the Carbon toolchain, then Carbon tools shouldn't be expected to be able to migrate the code. Similarly, that cc code may not be able to call into Carbon, for the reasons noted in the next bullet.

jonmeow · 2020-07-24T22:26:32Z

docs/design/interoperability/README.md

+  - While arbitrary C++ code may be able to call into Carbon code that has been
+    pre-compiled into a library, a more complex interaction like C++ code
+    calling Carbon templates requires compilation of _both_ languages together.


This seems like an infeasible constraint on Carbon to me, considering it's mainly for templates. Noted as an alternative approach.

docs/design/interoperability/README.md

jonmeow · 2020-07-24T22:52:12Z

docs/design/interoperability/README.md

+C++ enums will generally translate nautrally to Carbon, whether using `enum` or
+`enum class`. In the other direction, we expect Carbon enums to always use
+`enum class`.


@gribozavr, for clarity, would you voice in support of leaving this for now pending further enum design, or would you prefer it be removed?

docs/design/interoperability/README.md

jonmeow · 2020-07-24T23:10:43Z

docs/design/interoperability/README.md

+
+> References: [Templates and generics](templates_and_generics.md).
+
+Carbon generics will require bridge code that hides the generic. This bridge


I'm not clear that's that straightforward, given how generics are done. But maybe a template could be auto-generated? I haven't really thought that through. Added an open question to templates_and_generics.md

jonmeow · 2020-07-24T23:16:08Z

docs/design/interoperability/functions_and_overload_sets.md

+Carbon will provide specialized operator template functions for C++ types which
+are implemented as-if calling a C++ function template in bridge code which in
+turn did the exact operator call, including ADL-based name lookup and overload
+resolution. Carbon code can then override this behavior by providing specialized
+patterns for operators when interacting with Carbon types.


Honestly, I'm not familiar enough with ADL to give a good example. @chandlerc can you help here?

jonmeow

A few more addressed...

jonmeow · 2020-07-25T00:05:13Z

docs/design/interoperability/primitive_types.md

+$if platform == LP64
+fn ToCLong(var Int64: val) -> Int64 { return val; }
+$else
+fn ToCLong(var Int64: val) -> Int32 { return (Int32)val; }


Sure, feel free to grab a time.

jonmeow · 2020-07-25T00:14:06Z

docs/design/interoperability/primitive_types.md

+  - These types are likely to leak beyond C++ interoperability code, creating
+    friction in using APIs that are designed either only for variable-size types
+    or for fixed-size types, hindering API reuse. Overlapping implementations
+    and increased maintenance costs are a likely result.


Sorry, I'm not sure I can provide concrete examples, other than to say that it's my understanding Swift has seen similar problems with their Obj-C layer that they didn't anticipate.

I view this as something that could flip in the decision, but my sense is variable size types probably won't be accepted -- thus the choice of defaults. Unless there's more discussion in the review that clearly indicates a leaning in the other direction, I'll leave it as is.

jonmeow · 2020-07-25T00:14:13Z

docs/design/interoperability/primitive_types.md

+  - If platform-specific types are added, it may be worth considering whether we
+    should promote these to Carbon primitive types.


Thank you for being explicit, and I think I'd correctly guessed your leaning from your other comments. I think my response is captured on the line above.

jonmeow · 2020-07-25T00:16:00Z

docs/design/interoperability/primitive_types.md

+  - For example, users may still write platform-specific code like
+    `var Cpp.long: x = ...; var Int32: y = (Int32)x;`.


Pragmatically, it's not that different. That's why "may still" instead of "uniquely" -- the point is to emphasize that this solution has problems, too.

…lang into interop-proposal

jonmeow

And now I think I've addressed comments...

jonmeow · 2020-07-25T01:01:04Z

docs/design/interoperability/enums.md

+
+## C/C++ enums in Carbon
+
+C++ enums will generally translate naturally to Carbon, whether using `enum` or


Adding open questions to cover this. Is that sufficient for now?

jonmeow · 2020-07-25T01:04:47Z

docs/design/interoperability/enums.md

+  DIRECTION_WEST,
+  DIRECTION_NORTH,
+  DIRECTION_SOUTH,
+} __attribute__((carbon_enum("East:West:North:South"));


Good point, added a note.

jonmeow · 2020-07-25T01:07:10Z

docs/design/interoperability/enums.md

+}
+```
+
+We would expect to generate equivalent C++ code:


Is the attribute something that exists? For now, I've added an open question. Although, would setting appropriate types on enum classes actually solve it?

For terms, any suggestions for a glossary? Does Swift have such terms?

zygoloid

I think we should be seriously considering removing the $extern mechanism, and instead making the interoperability much more symmetric: permit Carbon libraries to be imported into C++ just as we permit C++ libraries to be imported into Carbon. I don't think it's feasible to ask for any Carbon type that's used with a C++ template to be $extern'd; that would create problems for the use of C++ templates from Carbon generics and from Carbon templates, where the requirement to $extern would be imposed on the facet type or on the user of the Carbon template, respectively.

I think a lot of the detail here is in areas where we can't really agree on that level of detail yet, because we don't know how that language feature of Carbon should work. So I think what we really need to decide first is the form we want interoperability to take: do we have the full bidirectional interoperability I described above, or do we treat use of Carbon from C++ as somewhat second-class, per this document, or something else? Do we expect to have a single toolchain that can understand and generate code for both Carbon and C++ sources, or do we expect the Carbon compiler to spit out a header file that a C++ compiler that doesn't understand Carbon can consume? I would want the outcome of this PR to be that we have clarity and agreement on those kinds of questions.

The exploration of details here is useful, but once we have the high-level decisions about the form of interoperability, it seem to me that we should be feeding that into the design review of all the other aspects of Carbon rather than trying to handle it centrally. To that end, I would suggest keeping all of the specific areas of design details, to flesh out the direction you're describing, but marking them as "to be finalized later", much like was done for the overall design document.

zygoloid · 2020-07-24T23:01:45Z

docs/design/interoperability/README.md

+- `$extern("Cpp")`: Indicates that Carbon code should be exposed for C++.
+  Similarly, `$extern("Swift")` might be used to indicate exposure for Swift at
+  some point in the future.


This description wasn't enough for me to infer what this syntax means or where it would appear. From reading ahead, it appears the intent is that this is an annotation attached to individual Carbon declarations, and that it causes the C++ wrapper header to provide a corresponding declaration to C++ code that matches the Carbon declaration.

Does this in any way change the meaning of the Carbon declaration, or only expose it? For example, extern "C" in C++ can change the calling convention and name mangling. Is the idea that $extern("Cpp") changes the calling convention and provides a C++-compatible symbol name? Or does it just indicate that the entity is exposed, and leave the compatibility / interoperability to the generated .6c.h file?

My hope would be that the Carbon declaration doesn't change meaning from within Carbon code.

Instead, any semantic differences should be only in the exposed form of the API as it is accessed from that language.

+1 to @chandlerc. I also think it is better to make any necessary calling convention changes only in symbols usable from C++, while leaving symbols used by Carbon unchanged. That can mean that a function might have two entry points, one with the Carbon calling convention, one with the C++ calling convention.

zygoloid · 2020-07-24T23:06:35Z

docs/design/interoperability/README.md

+
+Notable elements are:
+
+- `$extern("Cpp")`: Indicates that Carbon code should be exposed for C++.


Are you proposing concrete syntax here, or is this just a placeholder? If it's concrete syntax, I find it a bit strange: what is a prefix $ operator doing here? (Also I think we can find a better word for this than extern).

zygoloid · 2020-07-24T23:18:50Z

docs/design/interoperability/README.md

+`Foo` would become `::Carbon::Widget::Foo` in C++. This may be renamed for
+backwards compatibility for C++ callers when migrating code, for example
+`$extern("Cpp", namespace="widget")` for `::widget`.


Do you imagine this as declaring the entity in namespace widget (only), or as still declaring Carbon::Widget and then pulling it into namespace widget by using-declaration or similar? (The former would be more consistent and would let people incrementally stop using the backwards-compatibility names, whereas the latter would provide more consistent ADL behavior, in particular if there are C++ functions in namespace widget.)

zygoloid · 2020-07-24T23:20:32Z

docs/design/interoperability/README.md

+The behavior of mapped types will not always be identical; they need only be
+similar. For example, we expect Carbon's `UInt32` to map to C++'s `uint32_t`.
+While behavior is mostly equivalent, where C++ would use modulo wrapping, Carbon
+will instead have trapping behavior.


This ("will") seems definitive but I don't think we've agreed that. Maybe "may"?

zygoloid · 2020-07-24T23:22:38Z

docs/design/interoperability/README.md

+  - We will try to transfer ownership to a Carbon type where possible, but may
+    need to copy to the Carbon type in complex cases.


What would this "copy" option entail?

zygoloid · 2020-07-25T00:48:44Z

docs/design/interoperability/functions_and_overload_sets.md

+Carbon will provide specialized operator template functions for C++ types which
+are implemented as-if calling a C++ function template in bridge code which in
+turn did the exact operator call, including ADL-based name lookup and overload
+resolution. Carbon code can then override this behavior by providing specialized
+patterns for operators when interacting with Carbon types.


If I'm understanding right, the idea is that C++ types exposed to Carbon implement the corresponding Carbon operator interfaces by calling bridge code in C++ that uses the C++ operator. (I think this was written before we decided we probably want to use interfaces for operator overloading, which is why it talks about "specialized operator template functions" instead.)

Presumably when exposing a Carbon type to C++, if the Carbon type implements operator interfaces, we'll inject a suitable declaration of a C++ overloaded operator to match too?

zygoloid · 2020-07-25T00:56:36Z

docs/design/interoperability/goals_and_philosophy.md

+    vocabulary types.
+- Mappings should be easy to maintain.
+  - We should provide a syntax for transparently, automatically exposing a
+    subset of Carbon types and APIs to C++ code without custom bridge code,


By "types and APIs" do you mean "types and functions"? "APIs" isn't really a well-defined term in this context, but if you want to use it to preserve some amount of imprecision, it presumably includes types.

zygoloid · 2020-07-25T00:59:01Z

docs/design/interoperability/goals_and_philosophy.md

+- We prioritize making it easy to call C++ APIs from Carbon. Calling Carbon APIs
+  from C++ must be possible, but need not be as easy.


Do you mean "function" instead of "API" here? Or "use" instead of "call"?

zygoloid · 2020-07-25T01:08:19Z

docs/design/interoperability/goals_and_philosophy.md

+- We may choose not to provide full support for unwinding exceptions across
+  Carbon and C/C++ boundaries.
+- Interoperability features should not be expected to work for arbitrary C++
+  toolchains. While pre-compiled C++ libraries may be callable, the Carbon


I think there's a key decision to be made here. If we require all Carbon and C++ code in project to be built with the same toolchain, then we can essentially require all the C++ code to be written in a Carbon-flavored C++, which is backwards-compatible with normal C++ but has extensions to work better with Carbon, uses Carbon-specific ABI rules, and can directly talk to Carbon types. We don't need to generate C++ headers from Carbon code, or anything like that; we just have one toolchain that speaks both languages. We may not even need explicit $extern syntax in that case, and could instead let the C++ code import Carbon like we let Carbon code import C++.

But if we want to support use of Carbon from C++ code that's built with an unmodified C++ toolchain using a regular C++ ABI, then our situation is very different.

This really is a huge point, and thanks for surfacing it.

I have a bunch of thoughts here, but I think this discussion is important enough to optimize it a bit so I started a Discourse thread here:
https://forums.carbon-lang.dev/t/interop-implementation-strategies/108

Notably, I think several other in-file comments end up tying back to the same core point.

That said, I think half of the $extern syntax is actually not tied up in this. I'll talk a bit about the $extern syntax more broadly in response to your top-level comment.

zygoloid · 2020-07-25T01:15:27Z

docs/design/interoperability/primitive_types.md

+Similarly, `float` and `double` may end up being different sizes on particular
+platforms.


I think we said in another doc that we're not interested in supporting such platforms.

chandlerc

I think we should be seriously considering removing the $extern mechanism, and instead making the interoperability much more symmetric: permit Carbon libraries to be imported into C++ just as we permit C++ libraries to be imported into Carbon. I don't think it's feasible to ask for any Carbon type that's used with a C++ template to be $extern'd; that would create problems for the use of C++ templates from Carbon generics and from Carbon templates, where the requirement to $extern would be imposed on the facet type or on the user of the Carbon template, respectively.

FWIW, I agree that we should make the interop much more symmetric and not require the $extern stuff when, for example, nistantiating C++ templates with Carbon types.

However, I want to point out that there are two different uses of $extern. One of them is to address the consumption direction where it involves templates and thus Carbon types need to be visible to C++. I agree that we should make that 100% transparent (see my comment on your implementation question below).

But the other use is to designated what parts of a Carbon API become available for export to C++ consumers. Not as part of templates (which would still be within the purview of a Carbon compilation), but generically to a layer of completely C++ code. There, I think the utility remains. That is where we are likely to want it to be explicit that the Carbon interface may have constraints on what it can do (for example, Carbon templates might not work). Those constraints shouldn't apply to the prior case. Concretely: Carbon code consuming a C++ template and then instantiating it on a Carbon template type should be fine. We started in Carbon code, and so we can instantiate Carbon templates. It is when the root consumer is C++ code that restrictions enter the picture.

But I also don't think this will need much special syntax. We will want explicit syntax to control which Carbon APIs are exported from a library for any consumption. We should take that syntax and build on it to designate when that export includes C++ export. Not sure we have clearly thought through what that syntax is, but if the $export stuff in this proposal were reduced to a placeholder for "building on whatever normal export syntax we end up with..." for this purpose, I'd be fine with it.

I think a lot of the detail here is in areas where we can't really agree on that level of detail yet, because we don't know how that language feature of Carbon should work. So I think what we really need to decide first is the form we want interoperability to take: do we have the full bidirectional interoperability I described above, or do we treat use of Carbon from C++ as somewhat second-class, per this document, or something else? Do we expect to have a single toolchain that can understand and generate code for both Carbon and C++ sources, or do we expect the Carbon compiler to spit out a header file that a C++ compiler that doesn't understand Carbon can consume? I would want the outcome of this PR to be that we have clarity and agreement on those kinds of questions.

I've started a forum thread to dive into the requirements we want here:
https://forums.carbon-lang.dev/t/interop-implementation-strategies/108

The exploration of details here is useful, but once we have the high-level decisions about the form of interoperability, it seem to me that we should be feeding that into the design review of all the other aspects of Carbon rather than trying to handle it centrally. To that end, I would suggest keeping all of the specific areas of design details, to flesh out the direction you're describing, but marking them as "to be finalized later", much like was done for the overall design document.

+1 (But I also think it would be useful to try to get some clear directionality on the high level decision.)

chandlerc · 2020-07-25T08:48:22Z

docs/design/interoperability/README.md

+- `$extern("Cpp")`: Indicates that Carbon code should be exposed for C++.
+  Similarly, `$extern("Swift")` might be used to indicate exposure for Swift at
+  some point in the future.


My hope would be that the Carbon declaration doesn't change meaning from within Carbon code.

Instead, any semantic differences should be only in the exposed form of the API as it is accessed from that language.

chandlerc · 2020-07-25T08:51:55Z

docs/design/interoperability/README.md

+C++ enums will generally translate nautrally to Carbon, whether using `enum` or
+`enum class`. In the other direction, we expect Carbon enums to always use
+`enum class`.


FWIW, I do have lots fo qusetions around what enums will actually end up looking like in Carbon -- I'd not want to really speculate too far about that.

But I think @zygoloid hits on a nice point that we can telegraph meaningfully -- we don't intend to have the name leakage by default, whatever it is that ends up forming the basis of mapped-to-enums.

chandlerc · 2020-07-25T08:54:53Z

docs/design/interoperability/README.md

+Simple C++ class templates are directly made available as Carbon templates. For
+example, ignoring allocators and their associated complexity, `std::vector<T>`
+in C++ would be available as `Cpp.std.vector(T)` in Carbon. More complex C++
+templates may need explicit bridge code.


Huge plus one, but I wonder if there is an effective way to basically telegraph this, and come in with a revision that specifically tries to address this? I think this is one of the most complex aspects of interop and it might be helpful to have a focused discussion just around that. Thoughts?

chandlerc · 2020-07-25T08:56:33Z

docs/design/interoperability/README.md

+
+> References: [Templates and generics](templates_and_generics.md).
+
+Carbon templates should be usable from C++.


We should really dig into this strategy question you raised... Going to do that in a separate thread -- in particular, I think it might make sense to pop that discussion out to its own Discourse thread:
https://forums.carbon-lang.dev/t/interop-implementation-strategies/108

chandlerc · 2020-07-25T09:41:20Z

docs/design/interoperability/README.md

+However, function overloading is supported in both languages, and presents a
+much more complex surface to translate. Carbon's overloading is designed to be
+largely compatible with C++ so that this can be done reasonably well, but it
+isn't expected to be precisely identical. Carbon formalizes the idea of overload
+resolution into pattern matching. C++ already works in an extremely similar way,
+although without the formalization. We expect to be able to mirror most function
+overloads between the two approaches.


FWIW, I wrote this before we had really explored interfaces as the primary extension point mechanism.

I think it might be worthwhile to revisit much of this and think about whether there is a better way to bridge C++ overloads (at least those intending to be extension points) and Carbon interfaces.

chandlerc · 2020-07-25T09:43:12Z

docs/design/interoperability/README.md

+
+- C typedefs are generally mapped to Carbon aliases.
+- C/C++ macros that are defined as constants will be imported as constants.
+  Otherwise, macros will be unavailable in Carbon.


Just my two cents, but I wouldn't try to overly infer semantics or structure here.

They aren't namespaced in C++ and so I'd expect them to be named in a way that copes with that.

chandlerc · 2020-07-25T09:45:41Z

docs/design/interoperability/functions_and_overload_sets.md

+For a collection of Carbon function patterns to be exposed to C++ code, all the
+types involved must also be exposed to C++ code. These patterns will be
+expressed by synthesizing an overload set in C++ code that as accurately as
+possible reflects the expected pattern match that would occur with native Carbon
+code.
+
+There is no need to rely on the complexity of C++ conversion sequences to
+precisely match any conversions triggered by the Carbon pattern match. Instead,
+this logic can be produced by explicitly generating all the necessary C++
+overloads and managing conversion within them.


To be fair, technically that forms an overload set. ;]

I think the text here is somewhat written from the perspective of the user of the API, not the implementation strategy? And I don't think we'd want users to interact with the function as a function template, but as an overload that does deduction, which is the usual way to call function templates like this...

chandlerc · 2020-07-25T09:48:02Z

docs/design/interoperability/functions_and_overload_sets.md

+supported, Carbon needs strong support for most common and idiomatic overload
+sets it encounters.
+
+Overload sets will be translated in a series of steps:


This is an interesting idea...

On first glance, I don't really see strong motivation for either direction over the other here... That's ok, we can pick one and see how it goes, and I'm reasonably happy with either. But maybe you have some motivating factors in mind that would be worth capturing in the document?

chandlerc · 2020-07-25T09:49:58Z

docs/design/interoperability/functions_and_overload_sets.md

+Carbon will provide specialized operator template functions for C++ types which
+are implemented as-if calling a C++ function template in bridge code which in
+turn did the exact operator call, including ADL-based name lookup and overload
+resolution. Carbon code can then override this behavior by providing specialized
+patterns for operators when interacting with Carbon types.


@zygoloid's memory is correct -- this long predates the interface based stuff. And I agree with his suggestion of how this is likely to work: by implementing the Carbon operator interface with calls to the C++ operator overloads.

I also agree about how the reverse will work: injecting overloads based on the interfaces implemented and dispatching from the overload through the interface.

chandlerc · 2020-07-25T09:56:08Z

docs/design/interoperability/goals_and_philosophy.md

+- We may choose not to provide full support for unwinding exceptions across
+  Carbon and C/C++ boundaries.
+- Interoperability features should not be expected to work for arbitrary C++
+  toolchains. While pre-compiled C++ libraries may be callable, the Carbon


This really is a huge point, and thanks for surfacing it.

I have a bunch of thoughts here, but I think this discussion is important enough to optimize it a bit so I started a Discourse thread here:
https://forums.carbon-lang.dev/t/interop-implementation-strategies/108

Notably, I think several other in-file comments end up tying back to the same core point.

That said, I think half of the $extern syntax is actually not tied up in this. I'll talk a bit about the $extern syntax more broadly in response to your top-level comment.

gribozavr · 2020-07-27T09:43:43Z

docs/design/interoperability/README.md

+- `$extern("Cpp")`: Indicates that Carbon code should be exposed for C++.
+  Similarly, `$extern("Swift")` might be used to indicate exposure for Swift at
+  some point in the future.


+1 to @chandlerc. I also think it is better to make any necessary calling convention changes only in symbols usable from C++, while leaving symbols used by Carbon unchanged. That can mean that a function might have two entry points, one with the Carbon calling convention, one with the C++ calling convention.

gribozavr · 2020-07-27T09:45:41Z

docs/design/interoperability/README.md

+
+## Type mapping
+
+Carbon and C/C++ will have a number of types with direct mappings between the


What does the word "direct" mean here?

jonmeow · 2020-07-28T17:49:21Z

Per discussion regarding implementation strategies and what to do with the RFC, I'm going to pause here. This proposal is back in WIP.

gribozavr · 2020-07-29T06:06:58Z

This proposal is back in WIP.

Should we continue reviewing?

#83) Co-authored by: chandlerc - Based on [PR 22](#83) - [Idea topic](https://forums.carbon-lang.dev/t/proposal-for-an-incomplete-rough-high-level-overview-ready-for-early-feedback/52) - [RFC](https://forums.carbon-lang.dev/t/rfc-an-incomplete-early-and-in-progress-overview-of-the-language-design/73) - [Decision announcement](https://forums.carbon-lang.dev/t/accepted-an-incomplete-early-and-in-progress-overview-of-the-language-design/110) This proposal should be considered a starting point of the language design. It's not intended to be final; language details may change. This is intended to offer a reasonable starting point for: - Example code. - Conceptualizing Carbon at a high level. - Reasonable, but not necessarily final, approaches to features in README.md. - If any idea is obviously bad, we can clean it up here. This proposal is not intended to achieve: - A whole language design. - This is way too much work for a single proposal; this is a skeletal framework only. - As we work on feature-specific designs, we may decide to use other approaches. That's fine: we only need somewhere to start. - The summaries in README.md may be expected to change over time. - Feature-specific files aren't intended to be well-written or comprehensive. They are a quick jot of prior thoughts. - We want to avoid getting stuck on language details that we should consider more carefully regardless. If you're passionate about a feature, please feel free to start a new proposal for it. - Each and every aspect of the suggested overview should be subject to careful examination and justification before it becomes a settled plan of record. Chandler started this with #22. I've taken it over with the following changes: - More of a directory hierarchy. - Trying to thin out the main file (now README.md) to lighter summaries of features. - Details/rationale/alternatives should be in feature-specific files. - Draft files are linked as references where added. For an example of how we may proceed with feature-specific designs, see #80. In this structure: - docs/design/README.md mentions interoperability, with a light overview. - The light overview is not yet in #80. - docs/design/interoperability/README.md goes into more depth on interoperability, covering key points of the approach. - Individual files in docs/design/interoperability/* go into more depth on interoperability. Simple designs may not have a subdirectory. All current feature-specific designs do not -- they may be moved later.

Related threads: - From austern, [Initial draft of C++ Interoperability principles doc #62](#62) - From me, [Carbon <-> C/C++ interoperability #80](#80) - [Doc](https://docs.google.com/document/d/1va8VgvDdA966WG3znJyUrlComYqNfNBV7__hUd9XxxU/edit) - [Ideas topic](https://forums.carbon-lang.dev/t/draft-carbon-c-c-interoperability/77) - [RFC topic](https://forums.carbon-lang.dev/t/rfc-carbon-c-c-interoperability/89) - From chandlerc, [Interop implementation strategies](https://forums.carbon-lang.dev/t/interop-implementation-strategies/108) For this PR: - [RFC topic](https://forums.carbon-lang.dev/t/rfc-c-interoperability-goals-175/156) - [Decision topic](https://forums.carbon-lang.dev/t/request-for-decision-c-interoperability-goals/171) - [Decision announcement](https://forums.carbon-lang.dev/t/accepted-c-interoperability-goals/175) - [Decision PR](#200)

jonmeow added 11 commits May 20, 2020 08:36

Merge pull request #1 from carbon-language/master

2ae6b6f

Move firebase support into a src directory. (#15)

Merge remote-tracking branch 'upstream/master'

5bba3a8

Merge remote-tracking branch 'upstream/master'

98562a6

Merge pull request #2 from carbon-language/master

a05e9c3

Update fork

Merge pull request #3 from carbon-language/master

dfe42cb

Merge

Merge pull request #4 from carbon-language/master

3931951

Merge

Merge pull request #5 from carbon-language/master

43cfde9

Merge

Merge pull request #6 from carbon-language/master

b489c23

Merge

Merge pull request #7 from carbon-language/master

5b10b4d

Merge

Merge remote-tracking branch 'upstream/master'

6ce0f97

Initializing proposal

16e6242

jonmeow added proposal A proposal WIP labels Jun 16, 2020

jonmeow changed the title ~~Carbon: Carbon ↔ C/C++ interoperability~~ Carbon ↔ C/C++ interoperability Jun 16, 2020

jonmeow added 4 commits June 16, 2020 16:58

Import interoperability design

5a9c853

Rename proposal

38e73de

tocs

4310965

Summaries

3b49e31

jonmeow mentioned this pull request Jun 19, 2020

An incomplete, early, and in-progress overview of the language design. #83

Merged

googlebot added the cla: yes PR meets CLA requirements according to bot. label Jun 19, 2020

jonmeow added 3 commits June 22, 2020 10:01

Merge remote-tracking branch 'upstream/master'

218df9f

Merge remote-tracking branch 'upstream/master'

5c33fbf

codespell

e4223f4

jonmeow changed the title ~~Carbon ↔ C/C++ interoperability~~ Carbon <-> C/C++ interoperability Jun 22, 2020

jonmeow added 6 commits June 22, 2020 16:03

Merge branch 'master' into interop-proposal

471df44

↔ -> <-> due to github emoji weirdness

21097a8

De-emphasize alternatives

116edd6

Lang markers

7be07be

Proposal

72a3382

Cleanup

69ba0aa

Drop vice versa

80c8698

jonmeow added the final comment period label Jul 22, 2020

geoffromer reviewed Jul 23, 2020

View reviewed changes

gribozavr reviewed Jul 24, 2020

View reviewed changes

jonmeow and others added 4 commits July 24, 2020 15:10

Apply suggestions from code review

38ce17f

Co-authored-by: Dmitri Gribenko <[email protected]> Co-authored-by: Geoff Romer <[email protected]>

Extract out goals

394f723

Comments

46aaff3

Update docs/design/interoperability/vocabulary_types.md

b7bdcb0

Co-authored-by: Geoff Romer <[email protected]>

jonmeow commented Jul 25, 2020

View reviewed changes

jonmeow added 2 commits July 24, 2020 17:51

Merge branch 'interop-proposal' of https://github.com/jonmeow/carbon-…

bedd467

…lang into interop-proposal

Comments

891fb23

jonmeow commented Jul 25, 2020

View reviewed changes

zygoloid reviewed Jul 25, 2020

View reviewed changes

chandlerc reviewed Jul 25, 2020

View reviewed changes

gribozavr reviewed Jul 28, 2020

View reviewed changes

jonmeow added WIP and removed proposal rfc Proposal with request-for-comment sent out comment deadline labels Jul 28, 2020

jonmeow changed the title ~~Carbon <-> C/C++ interoperability~~ (WIP) Carbon <-> C/C++ interoperability Jul 28, 2020

jonmeow changed the title ~~(WIP) Carbon <-> C/C++ interoperability~~ Carbon <-> C/C++ interoperability Jul 28, 2020

Merge branch 'trunk' into interop-proposal

525f503

Merge

7101c2e

jonmeow closed this Sep 23, 2020

jonmeow deleted the interop-proposal branch September 23, 2020 17:55

jonmeow mentioned this pull request Oct 20, 2020

C++ interoperability goals #175

Merged

		- Carbon must be able to compile C++ headers in order to translate names and
		types.

		- For example, users may still write platform-specific code like
		`var Cpp.long: x = ...; var Int32: y = (Int32)x;`.

		- If platform-specific types are added, it may be worth considering whether we
		should promote these to Carbon primitive types.


		## Acknowledgements

		This borrows significantly from the structure of Swift's interoperability plan

	This borrows significantly from the structure of Swift's interoperability plan
	This proposal borrows significantly from the structure of Swift's interoperability plan


		Non-goals:

		- We will not make Carbon -> C++ migrations as easy as C++ -> Carbon migrations.


		The design for interoperation between Carbon and C++ hinges on:

		1. A focus on types, and simple overload sets built from those types.


		> References: [Templates and generics](templates_and_generics.md).

		Carbon generics will require bridge code that hides the generic. This bridge


		## C/C++ enums in Carbon

		C++ enums will generally translate naturally to Carbon, whether using `enum` or


		Notable elements are:

		- `$extern("Cpp")`: Indicates that Carbon code should be exposed for C++.

		- We will try to transfer ownership to a Carbon type where possible, but may
		need to copy to the Carbon type in complex cases.

		- We prioritize making it easy to call C++ APIs from Carbon. Calling Carbon APIs
		from C++ must be possible, but need not be as easy.

		Similarly, `float` and `double` may end up being different sizes on particular
		platforms.


		> References: [Templates and generics](templates_and_generics.md).

		Carbon templates should be usable from C++.


		## Type mapping

		Carbon and C/C++ will have a number of types with direct mappings between the

Carbon <-> C/C++ interoperability #80

Carbon <-> C/C++ interoperability #80

Conversation

jonmeow commented Jun 16, 2020 • edited Loading

geoffromer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gribozavr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonmeow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonmeow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonmeow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zygoloid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chandlerc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonmeow commented Jun 16, 2020 •

edited

Loading