Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First cut at typescript-lite generator implementation. #20

Merged
merged 11 commits into from
Apr 15, 2016
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
language: scala
scala: 2.10.5
jdk: oraclejdk8
node_js:
- "4"

script:
- sbt ++$TRAVIS_SCALA_VERSION fulltest

Expand Down
13 changes: 12 additions & 1 deletion project/Build.scala
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,14 @@ object Courier extends Build with OverridablePublishSettings {
lazy val swiftGenerator = Project(id = "swift-generator", base = swiftDir / "generator")
.dependsOn(generatorApi)

private[this] val typescriptLiteDir = file("typescript-lite")
lazy val typescriptLiteGenerator = Project(id = "typescript-lite-generator", base = typescriptLiteDir / "generator")
.dependsOn(generatorApi)

lazy val typescriptLiteGeneratorTest = Project(
id = "typescript-lite-generator-test", base = typescriptLiteDir / "generator-test")
.dependsOn(typescriptLiteGenerator)

lazy val courierSbtPlugin = Project(id = "sbt-plugin", base = file("sbt-plugin"))
.dependsOn(scalaGenerator)

Expand Down Expand Up @@ -166,6 +174,7 @@ object Courier extends Build with OverridablePublishSettings {
s";project android-generator;$publishCommand" +
s";project android-runtime;$publishCommand" +
s";project swift-generator;$publishCommand" +
s";project typescript-lite-generator;$publishCommand" +
s";++$sbtScalaVersion;project scala-generator;$publishCommand" +
s";++$currentScalaVersion;project scala-generator;$publishCommand" +
s";++$sbtScalaVersion;project sbt-plugin;$publishCommand" +
Expand All @@ -182,7 +191,9 @@ object Courier extends Build with OverridablePublishSettings {
androidGenerator,
androidGeneratorTest,
androidRuntime,
swiftGenerator)
swiftGenerator,
typescriptLiteGenerator,
typescriptLiteGeneratorTest)
.settings(runtimeVersionSettings)
.settings(packagedArtifacts := Map.empty) // disable publish for root aggregate module
.settings(
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
namespace org.coursera.arrays

record WithAnonymousUnionArray {
`unionsArray`: array[union[int, string]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these surrounded by ticks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completely unnecessary...just didn't think to change it from the file I copied to make this test-case.

I'll update that in a followup pr...

`unionsMap`: map[string, union[string, int]]
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
namespace org.coursera.enums

/**
* An enum dedicated to the finest of the food groups.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:-)

*/
enum Fruits {
/**
* An Apple.
Expand Down
1 change: 0 additions & 1 deletion swift/generator/build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,3 @@ mainClass in assembly := Some("org.coursera.courier.SwiftGenerator")
assemblyOption in assembly := (assemblyOption in assembly).value.copy(prependShellScript = Some(defaultShellScript))

assemblyJarName in assembly := s"${name.value}-${version.value}.jar"

11 changes: 11 additions & 0 deletions typescript-lite/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
.gradle
build/

generator/build
testsuite/testsuiteTests/generated

# Ignore Gradle GUI config
gradle-app.setting

# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)
!gradle-wrapper.jar
259 changes: 259 additions & 0 deletions typescript-lite/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,259 @@
Typescript-Lite Courier Data Binding Generator
==============================================

Experimental!

Lightweight Courier bindings for Typescript.

The guiding philosophy here was to achieve type safety with as small a runtime footprint as possible -- use the fact
that JSON is native to javascript to lightly define types and interfaces that describe your courier specs.

These bindings will achieve their goal if:

* You can just cast the result of a JSON HTTP response (or other JSON source) into the base expected
type and stay typesafe without having to cast anything more afterwards.
* You can hand-write JSON in your test-cases and API requests, and those test-cases
fail to compile when the record contract breaks.
* You can enjoy the sweet auto-complete and compile-time warnings you've come to enjoy from typescript.

These bindings require Typescript 1.8+

Features missing in action
--------------------------

* **Coercer support**. The strong runtime component of Coercers don't gel easily with the philosophy of casting records
lightly into target typescript interfaces, so there wasn't an easy fit.
* Maybe these get included when Typescript-lite becomes just plain ol typescript.
* **Keyed maps**. Javascript objects naturally only support string-keyed maps. Since we are avoiding runtime
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the behavior when a keyed map is encountered? Will it just fail to compile? Do you get a warning? What is the generated type? (Apologies in advance if you already have a test case enforcing a particular behavior that I haven't gotten to yet.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that the generated type will be a Map<String, ValueT>, which should work at runtime too as long as they are being sent down the wire as valid string-keyed json maps. I'll double-check the test cases.

How are typed maps sent down the wire? Are they just sent down as json objects or has the wire protocol been updated to include type information?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK yeah typed maps are converted into string -> value maps. And given what I understand of the wire protocol that should be fine.

overhead as much as possible we did not want to introduce a new type, nor the means to serialize said type.
* Integrating `Immutable.js` would make a lot of sense when we want to expand into this direction.
* **Non-JSON serialization**. Although courier supports more compact binary formats (PSON, Avro, BSON), Typescript-lite
bindings currently only supports valid JSON objects.
* **Default values**. As with the other points, default values require a runtime.
* **Flat-typed definitions**. This was just an oversight. Gotta add these.

Running the generator from the command line
-------------------------------------------

Build a fat jar from source and use that. See the below section *Building a fat jar*.

How code is generated
---------------------

**Records:**

* Records are generated as an interface within the typescript `namespace` specified by your record.
* If you don't use a namespace, the interface will be injected at the top-level

e.g. the result of [Fortune.courier](https://github.com/coursera/courier/blob/master/reference-suite/src/main/courier/org/example/Fortune.courier):

```typescript
// ./my-tslite-bindings/org.example.Fortune.ts
import { FortuneTelling } from "./org.example.FortuneTelling";
import { DateTime } from "./org.example.common.DateTime";

/**
* A fortune.
*/
export interface Fortune {

/**
* The fortune telling.
*/
telling : FortuneTelling;

createdAt : DateTime;
}
```

**Enums:**

* Enums are represented as [string literal types](https://basarat.gitbooks.io/typescript/content/docs/types/stringLiteralType.html).
* Convenience constants matching the string literals are provided despite having a runtime cost
* TODO(erem) We should consider changing this. Not sure if the benefit outweighs the wire cost.
* Unlike other bindings, Typescript-lite does not include an `UNKNOWN$` option. In case of wire inconsistency
you will have to just fall through to `undefined`.

e.g. the result of [MagicEightBallAnswer.courier](https://github.com/coursera/courier/blob/master/reference-suite/src/main/courier/org/example/MagicEightBallAnswer.courier):

```typescript
// ./my-tslite-bindings/org.example.MagicEightBallAnswer.ts
/**
* Magic eight ball answers.
*/
export type MagicEightBallAnswer = "IT_IS_CERTAIN" | "ASK_AGAIN_LATER" | "OUTLOOK_NOT_SO_GOOD" ;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eboto Sorry, I'm not familiar with the idioms of typescript. Can you provide a rationale for not including UNKNOWN$ in this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! We use UNKNOWN$ at runtime in existing client libraries to surface the case when an unknown enum value, probably from a later version of the software, appears down the wire. However, the enums as written here (string-literal types) do not have a run-time component so it would be impossible to coerce that unknown string into UNKNOWN$ as part of the type. So I chose not to include it in the type.

That pretty much leaves it on the client to compare to all the KNOWN enums, and let the unknown one fall off the end. e.g. if you have enum Fruits { APPLE, ORANGE } then the client would have to do

const fruit: Fruit = /* get the fruit */
if (fruit == Fruit.APPLE) { 
  // something
} else if (fruit == Fruit.ORANGE) {
  // something
} else {
  // handle unknown
}

Now thinking about it though, I could provide a utility function like Fruit.isUnknown(enumValue: Fruit) that returns boolean if we got an unknown value. But I'm not sure how much that would help the experience.

Did that make sense? What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty reasonable. Given that they're represented as strings, it should be reasonable to use them in a switch statement as well. Having the enum variants generated I think is the right call. I'd expect they'll gzip really well, such that the overhead of including them won't be much higher than having a bunch of string constants in the source code. (And this way, you get autocomplete, and the potential for tooling to better understand what's going on in the future.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Don't know why I reached for this gross if-elses first.

I'm going to give the switch usage a workup in the testsuite then update the docs.

According to this typescript issue, we will get compiler warnings future versions for strings that are not in the string literal type...so that will make it even nicer (as long as it doesn't stop you from falling through to default.

EDIT: Also I think I agree with you about the constants down the wire. I'm going to keep them. =)

export module MagicEightBallAnswer {

export const IT_IS_CERTAIN: MagicEightBallAnswer = "IT_IS_CERTAIN";

export const ASK_AGAIN_LATER: MagicEightBallAnswer = "ASK_AGAIN_LATER";

export const OUTLOOK_NOT_SO_GOOD: MagicEightBallAnswer = "OUTLOOK_NOT_SO_GOOD";
}
```

**Arrays:**

* Arrays are represented as typescript arrays.

**Maps:**

* Maps are represented as javascript objects, as interfaced by `courier.Map<ValueT>`
* Only string-keyed maps are currently supported.

**Unions:**

* Unions are represented as an intersection type between all the members of the union.
* A Run-time `unpack` accessor is provided to test each aspect of the union
* Unlike other serializers, no `UNKNOWN$` member is generated for the union. If all provided accessors end up yielding
undefined, then conclude that it was an unknown union member (see second example)

e.g. The result of [FortuneTelling.pdsc](https://github.com/coursera/courier/blob/master/reference-suite/src/main/pegasus/org/example/FortuneTelling.pdsc):
```typescript
// file: my-tslite-bindings/org.example.FortuneTelling.ts
import { MagicEightBall } from "./org.example.MagicEightBall";
import { FortuneCookie } from "./org.example.FortuneCookie";

export type FortuneTelling = FortuneTelling.FortuneCookieMember | FortuneTelling.MagicEightBallMember | FortuneTelling.StringMember;
export module FortuneTelling {
export interface FortuneTellingMember {
[key: string]: FortuneCookie | MagicEightBall | string;
}

export interface FortuneCookieMember extends FortuneTellingMember {
"org.example.FortuneCookie": FortuneCookie;
}

export interface MagicEightBallMember extends FortuneTellingMember {
"org.example.MagicEightBall": MagicEightBall;
}

export interface StringMember extends FortuneTellingMember {
"string": string;
}

export function unpack(union: FortuneTelling) {
return {
fortuneCookie: union["org.example.FortuneCookie"] as FortuneCookie,
magicEightBall: union["org.example.MagicEightBall"] as MagicEightBall,
string_: union["string"] as string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you're doing some escaping of some of the types. We've used $ for escaping elsewhere in Courier. Does $ not work in typescript?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that $ will work fine! I was unaware that it was a convention. I will patch the code to use $.

};
}
}
```

Here's how you would use one:
```typescript
import { FortuneTelling } from "./my-tslite-bindings/org.example.FortuneTelling";
import { MacigEightBall } from "./my-tslite-bindings/org.example.MagicEightBall";
import { FortuneCookie } from "./my-tslite-bindings/org.example.FortuneCookie";

const telling: FortuneTelling = /* Get the union from somewhere...probably the wire */;

const { fortuneCookie, magicEightBall, string_ } = FortuneTelling.unpack(telling);

if (fortuneCookie) {
// do something with fortuneCookie
} else if (magicEightBall) {
// do something with magicEightBall
} else if (string_) {
// do something with str
} else {
throw 'a fit because no one will tell your fortune';
}
```

Projections and Optionality
---------------------------

It is common to send and receive partial data through REST. This
is very commonly used when a subset of fields of a resources are "projected".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've been tussling internally with how to maintain type safety while supporting projections. While for our JS clients it's not important, for our mobile clients it is. Can you elaborate on why you went down this route, and what the advantages and disadvantages have been in your experience?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll freely admit that I haven't thought much about projections because they aren't part of my company's use case right now. That was just a part of the documentation that I overlooked. I should revisit it.

I think Intersection types may be a good approach in typescript

Imagine the following courier type:

record Message {
  id: string;
  subject: string;
  body: string;
}

If we wanted to support projections, we could generate the following types

module Message {
  interface Id {
    id: string;
  }
  interface Subject {
    subject: string;
  }

  interface Body {
     body: string;
  }
}
type Message = Message.Id & Message.Subject & Message.Body;

In your application code when you request some projection, instead of using the Message type you could just safely cast the message down to its component projections! For example:

function getMessageIdAndBody(id: string): Promise<Message.Id & Message.Body> {
  return http.get('/messages/' + id + '?projection=(id,body)').then((resp) => {
    return resp.data as (Message.Id & Message.Body);
  })
}

Any attempt to access message.subject from the results of that function would of course fail at compile time.

Anyways. I haven't actually tried any of that out, but I'm pretty certain it would work. So projections work pretty smoothly in typescript. =)

If it sounds good, I can put this intent into the docs and replace the current description of projections.


Since even fields that are marked as required in a Pegasus schema may be absent when data is
projected, Courier's [Optionality](https://github.com/coursera/courier/blob/master/typescript/generator/src/main/java/org.coursera.courier.tslite.TSProperties.java#L31)
settings defaults to REQUIRED_FIELDS_MAY_BE_ABSENT. This allows a single
generated typescript struct to be used for bindings to unprojected and projected data.

If this behaviour is not desired, one may set `Optionality` to `STRICT`.

See the [Optionality property docs](https://github.com/coursera/courier/blob/master/typescript/generator/src/main/java/org.coursera.courier.tslite.TSProperties.java#L8)
for details on how to set the `Optionality` property.

Custom Types
------------

Custom Types allow any Typescript type to be bound to any pegasus primitive type.

For example, say a schema has been defined to represent a "date time" as a unix timestamp long:

```
namespace org.example

typeref DateTime = long
```

This results in a typescript file:
```typescript
// ./my-tslite-bindings/org.coursera.customtypes.DateTime.ts
export type DateTime = number;
```

JSON Serialization / Deserialization
------------------------------------

JSON serialization is trivial in typescript. Just take use `JSON.stringify` on any courier type that compiles.

```typescript
import { Message } from "./my-tslite-bindings/org.coursera.records.Message";

const message: Message = {"body": "Hello Pegasus!"};
const messageJson = JSON.stringify(message);
```

And of course you can read results as well.

```typescript
import { Message } from "./my-tslite-bindings/org.coursera.records.Message";

const messageStr: string = /* Get the message string somehow */
const message = JSON.parse(messageStr) as Message;
```

Runtime library
---------------

All generated Typescript-lite bindings depend on a `CourierRuntime.ts` class. This class provides the very minimal
functionality and type definitions necessary for generated bindings to work.

Building from source
--------------------

See the main CONTRIBUTING document for details.

Building a Fat Jar
------------------

```sh
$ sbt
> project typescript-lite-generator
> assembly
```

This will build a standalone "fat jar". This is particularly convenient for use as a standalone
commandline application.

Testing
-------

1. No testing has been done yet except verifying that the generated files from reference-suite pass `tsc`.
That's next on the list =)

TODO
----

* [ ] Add support for flat type definitions
* [ ] Figure out the best way to distribute the 'fat jar'.
* [ ] Consider getting rid of the string literal constants in generated enums. They may not give much for the wire cost.
* [ ] Automate distribution of the Fat Jar
* [ ] Publish Fat Jar to remote repos? Typically fat jars should not be published to maven/ivy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely happy to work with you on this. I've been looking at Bintray, but haven't investigated further. Do you have any experience here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm unfortunately I don't have much experience with it. My first thoughts are just to be messy and toss the damned thing up onto S3, or even onto IPFS which would be amazing.

repos, but maybe it should be hosted for easy download elsewhere?
45 changes: 45 additions & 0 deletions typescript-lite/generator-test/build.sbt
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import sbt.inc.Analysis

name := "courier-typescript-lite-generator-test"

packagedArtifacts := Map.empty // do not publish

libraryDependencies ++= Seq(
ExternalDependencies.JodaTime.jodaTime)

autoScalaLibrary := false

crossPaths := false

// Test Generator
forkedVmCourierGeneratorSettings

forkedVmCourierMainClass := "org.coursera.courier.TypeScriptLiteGenerator"

forkedVmCourierClasspath := (dependencyClasspath in Runtime in typescriptLiteGenerator).value.files

forkedVmSourceDirectory := (sourceDirectory in referenceSuite).value / "main" / "courier"

forkedVmCourierDest := file("typescript-lite") / "testsuite" / "src" / "tslite-bindings"

forkedVmAdditionalArgs := Seq("STRICT")

(compile in Compile) := {
(forkedVmCourierGenerator in Compile).value

Analysis.Empty
}

lazy val npmTest = taskKey[Unit]("Executes NPM test")

npmTest in Test := {
(compile in Compile).value

val result = """./typescript-lite/testsuite/full-build.sh"""!

if (result != 0) {
throw new RuntimeException("NPM Build Failed")
}
}

test in Test := (npmTest in Test).value
2 changes: 2 additions & 0 deletions typescript-lite/generator/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
src/test/mainGeneratedPegasus
src/main/mainGeneratedPegasus
Loading