PostGraphQL V4 - announcing graphile-build

[benjie]

TL;DR:

• This is v4 of PostGraphQL
• PostGraphQL is renaming to "PostGraphile"
• Massively improved performance and added extensibility via plugins
• Breaking changes have been kept to a minimum; see bottom of this post
• Remaining tasks: https://trello.com/b/x4xS0bVG/postgraphile-v4-remaining-tasks
• Support my work https://www.patreon.com/benjie 🙏

---

In spirit, this is a follow on to Caleb's "My vision for the future of PostGraphQL" #87, to learn about our history I suggest you read that.

Hello everyone, I'm @benjie - the new primary maintainer of PostGraphQL!

I've been a PostGraphQL user for almost a year and have used it on various projects; and I really love how easy it makes starting a new project - all I have to do is build my data schema in my favourite database (PostgreSQL), making sure that the schema itself is secure (using such features as Row Level Security), run postgraphql and voila - I've got a fully working GraphQL API that does what I'd expect. It's an extremely impressive piece of software and I think we're all extremely grateful to @calebmer for authoring it and bringing it to fruition!

One of the issues that pops up time and again on the GitHub issues is extensibility - you can't add your own types/resolvers to the schema - everything has to be done in PostgreSQL. There's some work in #448 to partially resolve this, but I think I've found a better way...

Another issue has been the performance of the system - it's an excellent reference implementation in terms of accuracy and returning what you'd expect, but under the covers it sometimes generates and runs a very large number of SQL queries and over-fetches data. I attempted to solve this in #342, but since then I think I've found a better way...

Starting with GraphQL as a foundation I tried to imagine a system that could replicate PostGraphQL's benefits whilst remaining customisable, extensible and performant. I imagined PostGraphQL implemented as a number of plugins into a GraphQL schema builder - if you don't want certain functionality, simply don't use that plugin; if you want to add a bit of custom code, write a plugin; if you want to wrap one of PostGraphQL's resolvers with your own, write a plugin; if you... well: you get the point!

Introducing graphile-build!

To enable this goal, over the past few weeks I've been busy building the following packages in a new monorepo at https://github.com/graphile/graphile-build

• graphile-build: the core plugin system that enables you to build a GraphQL schema out of plugins

• graphile-build-pg: A selection of plugins related to PostgreSQL: schema introspection, generation of fields and types for all tables, computed columns, query procedures, etc

• postgraphile-core: a tiny compatibility layer between PostGraphQL and graphile-build/graphile-build-pg

• graphql-parse-resolve-info: Parses a GraphQLResolveInfo object into a tree of the fields that are being requested to enable optimisations to the GraphQL schema (e.g. we use it to determine which fields are required from the SQL database)

I believe this plugin interface can really enable PostGraphQL to grow and flourish. Changes can be implemented as replacements for existing plugins which can be tested outside of core until stable, at which point they can be merged. New functionality can be added in a similar way (subscriptions anyone?) without requiring to fork core. Experiments can be ran; not to mention that individual users can fix their issues with the system without having to have long running forked branches running!

But, alas, it's not quite ready to go just yet. So far I've managed to get the core working sufficiently that I can run my existing projects against it, but there's more to be done before it's release ready. If you're interested in contributing (code, testing, or other support) please get in touch!

Thanks for reading this far, to quote Blaise Pascal: "I would have written a shorter letter, but I did not have the time."

Features required before release:

• Pass the PostGraphQL integration tests for queries
• Pass the PostGraphQL integration tests for mutations
• Only fetch the columns that are needed
• Perform as few SQL queries as possible, using the database instead of DataLoader to batch requests
• Fix issues with case insensitive unique keys
• Surface all the inflections to enable them to be overridden
• Optimise generated SQL queries
• Domain (alias) support
• Ensure all PG types are supported at least as well as PostGraphQL v3
• Pass the PostGraphQL schema tests (small changes are acceptable since we're doing a major version bump so are allowed breaking changes)
• Re-implement schema watching
• Ensure the JWT creation functionality still works
• Ensure GraphiQL still works (including with schema watching)
• Set up CI for graphile-build
• Don't crash on empty mutations
• Look into nodeId/id issue commented on in PR review below
• Resolve issues with mutations and computed columns
• Solve Ensure optional arguments are handled correctly graphile-engine#3
• Solve @danscan's issue
• Restore PgCatalog functionality Feat: Build schema from prebuild PgCatalog #306
• Have some beta-testers approve
• Update documentation where necessary
• Write a migration guide (if necessary)

Features that would be nice to have, but can be implemented after release:

• Enable users to specify their own inflection
• More Documentation
• Add plugin customisation functionality to postgraphile-core
• Tidy up the remaining PostGraphQL codebase
• Enable users to specify additional plugins to load via the CLI
• Enable users to specify plugins to not load via the CLI
• Enable users to specify settings to be passed through to the plugin system via the CLI

---

Breaking changes (should be a drop-in replacement for v3 for most people)

Breaking changes I'm aware of (that aren't listed in the checkboxes to be solved above):

• Minimum PostgreSQL version is now 9.4
• Minimum Node.js version is now 4.0.0 (but you get significant performance boost by using v8+)
• Json and Uuid have been renamed to JSON and UUID
• UUID is enabled by default
• Watch mode works differently, so if you had to set up that schema yourself (rather than having PostGraphQL do it for you) you'll need to set it up again
• The introspection query has changed slightly, so if you use this directly for anything you may need to tweak your code.
• A small number of things that were nullable before are not nullable now (deliberately - e.g. an empty list of results is now an empty array rather than null)
• A small number of things that were not nullable before are nullable now (deliberately - e.g. setof procedures can return nulls (though I'd argue that doing so is bad practice!))
• Many of the descriptions of fields have changed slightly
• I use a different library for changing case, so it's possible that parts of the schema differ subtly, nothing in the acceptance tests has been affected by this
• Procedure connections that return setof table have gone back to the V2 behaviour of returning the generic table connection rather than a connection specific to the procedure
• Procedures that previously only supported orderBy: NATURAL no longer have orderBy
• Ranges are handled differently, so there will likely be some subtly different behaviours there (the interface is basically the same however)
• Scalar procedures returning setof no longer have pageInfo nor totalCount (this might be temporary depending on demand - let me know if this is a problem for you)
• Make nodes array items in Pg Connection non null. graphile-engine#89 (comment)
• security definer mutations that return a type from a private schema that the requesting PostgreSQL user is not allowed to view may now result in permission denied for schema xxxx see gitter

Despite the long list, PostGraphQL v4 should be a drop-in replacement for most people - if you see any major issues please let me know.

Have a look at postgraphqlIntegrationSchema-test.js.snap to get an idea what sort of changes you're dealing with - they're mostly minor.

[bradleyayers]

Really exciting to hear that there is a full time maintainer (and that it's you 😄) for this project. It was definitely one of my concerns with adopting it myself. I'm pumped that you're tackling the big gnarly challenges, and really interested in digging in and understanding the design.

I must admit it was disappointing to see it's no longer in TypeScript — I had a quick look and code new code and found it really challenging to make sense of it without having all the type information to tie everything together. If you could share some background or rationale for that I'm sure there'd be an audience for that.

[benjie]

UPDATE: this comment is mostly out of date now - we have Flow, linting, CI and a fairly decent amount of documentation on graphile.org

---

@bradleyayers Sadly I'm not full time; my work on PostGraphQL is completely unpaid; which also explains the lack of TypeScript: I was working to get a new system up and running as a proof of concept ASAP without knowing exactly what form it would ultimately take - strong typing would have slowed me down considerably at the rate I was iterating. I decided it would be fun to write the code in native JS because that way I wouldn't need transpilers, source maps, or any of that other mess that comes along with compile-to-JS languages - and the debugging tools for Node.js now built into Chrome are outstanding!

Now that the shape of the system is better known I'll probably start typing the system a little at a time with Flow which is pretty useful out of the box and doesn't require as much markup as TypeScript. I also find it to be more flexible and with clearer errors but that may be a familiarity thing.

My hope is that taking the monorepo approach will mean the various parts of the system can be kept fairly separate with clear boundaries and APIs; I've started documenting some of these but haven't got very far yet. I've not even enabled strong linting yet, or CI - all things on my todo list! At the moment the plugins are not as loosly coupled as I'd ultimately like, but I think that's something that will improve over time. My next task is to get schema watching working - once that's up and running I think the interfaces will be fairly stable. Though I should probably think about a "requires/optional/provides" specification for the plugins.

[benjie]

Any help on this would be extremely welcome and I'm more than happy to offer guidance, answer questions or even jump on a call. It doesn't even have to be code - help documenting the new system would be hugely valuable!

[chadfurman]

@benjie I like what's going on here: https://postgraphql.github.io/

For documentation, we have two main sources currently: https://github.com/postgraphql/postgraphql/tree/master/docs
https://github.com/postgraphql/postgraphql/tree/master/examples (specifically the forum example / tutorial)

We can have a "tutorial" page that uses docco to generate a nice UI from comments in a file, then walk the user through it that way.

The docs section consists of two parts, currently mixed together:
Information on Postgres stuff like procedures
Information on PostgraphQL stuff like using it as a library

I can get a PR together for the existing content in a format that will work on https://github.com/postgraphql/postgraphql.github.io

What else will we need in the v4 docs?

[benjie]

@chadfurman Thanks for pitching in!

For documentation, we have two main sources currently: /docs@master
/examples@master (specifically the forum example / tutorial)

We've also got a tiny amount on the wiki and absolutely loads spread throughout the GitHub issues. Would be good to centralise this (and remove the current places so that they don't get built up/added to again!)

We can have a "tutorial" page that uses docco to generate a nice UI from comments in a file, then walk the user through it that way.

Nice.

The docs section consists of two parts, currently mixed together:
Information on Postgres stuff like procedures
Information on PostgraphQL stuff like using it as a library

Yes - basically usage instructions (PostGraphQL) and advice/best practices (PostgreSQL)

I can get a PR together for the existing content in a format that will work on postgraphql/postgraphql.github.io

That would be excellent.

What else will we need in the v4 docs?

The various sub-components of graphql-build will require further documentation also, e.g. how to write your own plugins or what the current plugins are responsible for doing (and how to turn them off/configure them). I don't think we're quite ready for this to be written yet until the API solidifies fully, but we're almost there!

The v4 usage instructions should be basically the same as V3 (even the library instructions), we may tweak the createPostGraphQLSchema API slightly to enable configuration of plugins/inflection and add a few more CLI options, and the DEBUG envvar will require different configuration to e.g. view the raw SQL, but other than that most of what's true for V3 should be true for V4.

[chadfurman]

@benjie postgraphql/postgraphql.github.io#5

[benjie]

Schema watching now works (and fixes #502)

[bradleyayers]

Shouldn't POSTGRES_SCHEMA be POSTGRES_DATABASE?

[benjie]

Yup

[benjie]

Just pushed 4.0.0-alpha2.28 which includes the last major feature required to replace v3 - caching the introspection results to disk (see #632). This is particularly important to anyone using PostGraphile on AWS Lambda or similar environments where startup time is at a premium. (Sidenote: if you're using PostGraphile on AWS Lambda, please allocate at least 512MB of memory or you may risk significantly larger response times according to @jmparsons on gitter - we have not optimised PostGraphile for memory usage yet).

• ✨ --write-cache <path> command line option enables you to write introspection results (and anything else that the various plugins, including your own, decide to cache) to a JSON file
• ✨ --read-cache <path> command line option reads the cache back from disk allowing plugins to skip certain time-consuming tasks like introspection

We're so close to v4 beta now I can taste it! There's just one more breaking change I want to introduce - and that's #432, an improvement to create table bar (foo_id int primary key references foo, col text)-style one-to-one relations, so that you can query fooById(id: 7) { id barByFooId { col } } rather than what is currently fooById(id: 7) { id barsByFooId { edges { nodes { col } } } } - i.e. it removes a level of indirection. I hope to add an option to allow you to disable this if you need to restore backwards compatibility, the breaking change will be the change in default behaviour.

---

I really appreciate the support of the 7 patrons on my Patreon ❤️ Please consider joining them: https://www.patreon.com/benjie

[benjie]

Yeah... That's right... I just merged this to master! 🎉

Also note this repo has moved to https://github.com/graphile/postgraphile