Improved Docs & Removed monorepo-scripts from packages #924

fabioberger · 2018-07-29T20:09:26Z

Description

Currently monorepo-scripts is a dependency of every public package. The reason? We are using Lerna's postpublish.js script hook to generate/upload docs and publish release notes to Github. This PR changes that.

As the number of packages published in the repo has grown, it has become untenable to have an individual release note for each package. This PR therefore aggregates them into a single release note that looks like:

### 0x.js

- Fixed a tiny, inconsequential bug (#789)
- Add a `findALover` method to `ZeroEx` (#793)

### web3-wrapper

- Removed unused param from `getBlockAsync` (#794)

...and so on for each package

Additionally the doc generation/upload step has been moved to the publish.ts script and happens after the packages are published. Importantly, the way we generate the doc JSON representations has changed. It now does the following:

Find all exports in packages index.ts
Find the files for each export (if it's internal or within another monorepo package)
Call Typedoc with list of files containing the public interface of the package
Go through the resulting Typedoc JSON, and remove all exported elements not exported in the main package's index.ts.
Add entries for any external packages re-exported by our package (previously the docs simply omitted these).

These steps have resulted in our doc JSON files being 3X smaller then they used to be! 🍾 🎆

Additionally, we've gotten rid of a lot of package specific configurations using the above procedure. This PR also adds checks to make sure that any types that are present in the package's public interface are exported by it's index.ts file. It will even complain if the package exports superfluous type declarations. These checks now happen in every CircleCI run.

In order to make all package's pass these new constraints, I had to modify their index.ts. Some larger changes were also needed, like moving types from sol-compiler to types that are used by many packages now.

Miscelaneous fixes:

Run yarn install twice in CircleCI since I was encountering the same issue as locally.
Modified the public interface of 0x.js so that it simply re-exports it's sub-packages.
Added OrderWatcher back into 0x.js
Fixed a bunch of TSLint errors from across many unchanged packages that suddenly popped up.
Fixed scroll offset issue

Testing instructions

Run the tests. Render the doc pages.

Types of changes

Checklist:

Prefixed the title of this PR with [WIP] if it is a work in progress.
Prefixed the title of this PR with bracketed package name(s) corresponding to the changed package(s). For example: [sol-cov] Fixed bug.
Added tests to cover my changes, or decided that tests would be too impractical.
Updated documentation, or decided that no doc change is needed.
Added new entries to the relevant CHANGELOG.jsons.

…ggregate release note publishing to publish script

…ends on it

…hash

…se TypeDoc JSON

…y positioned div to work correctly. See: fisshy/react-scroll#330

…be defined

…ut .json imports

… the longest match

albrow · 2018-08-22T22:45:56Z

@fabioberger 👍 sounds good to me. I'll take any chance we can get to reduce maintenance burden right now.

fragosti

This is amazing but also a lot. Really hard to get some confidence that there aren't any bugs. Overall looks good -- I'm down to ship and then discover what bugs there are since it doesn't touch critical pieces toooo much. Must've been a lot of work so 🙏🏼

fragosti · 2018-08-22T22:46:56Z

packages/connect/package.json

@@ -82,7 +74,7 @@
        "nyc": "^11.0.1",
        "shx": "^0.2.2",
        "tslint": "5.11.0",
-        "typedoc": "~0.8.0",
+        "typedoc": "0xProject/typedoc",


What does this do?

I changed this just so that we are using the same version everywhere. There used to be a bug in Typedoc that was causing us grief. Refactors since then have made it a non-issue so I changed all versions to the latest one of 0.12.0.

fragosti · 2018-08-22T23:47:08Z

packages/contract_templates/contract.handlebars

@@ -1,9 +1,9 @@
 // tslint:disable:no-consecutive-blank-lines ordered-imports align trailing-comma whitespace class-name
 // tslint:disable:no-unused-variable
+// tslint:disable:no-unbound-method


Why do you suddenly need to add these?

I upgraded our tslint-eslint-rules dependency since it's dep tsutils@^1.4.0 wasn't playing nicely with TS 3.0. I believe this rule was added to it.

fragosti · 2018-08-23T00:12:44Z

packages/monorepo-scripts/src/utils/doc_generate_and_upload_utils.ts

+                    );
+                });
+            } else if (_.isObject(nodeValue)) {
+                updatedReferenceNames = updatedReferenceNames = DocGenerateAndUploadUtils._getAllReferenceNames(


Woah this compiles?

Yikes, good catch.

fragosti · 2018-08-23T00:15:14Z

packages/monorepo-scripts/src/utils/doc_generate_and_upload_utils.ts

+        });
+        if (!_.isEmpty(externalExportsWithoutLinks)) {
+            throw new Error(
+                `Found the following external exports in ${


fragosti · 2018-08-23T00:23:27Z

packages/monorepo-scripts/src/utils/github_release_utils.ts

+    const changelogs = JSON.parse(changelogJSON);
+    const latestLog = changelogs[0];
+    // If only has a `Dependencies updated` changelog, we don't include it in release notes
+    if (latestLog.changes.length === 1 && latestLog.changes[0].note === constants.dependenciesUpdatedMessage) {


Should we make this more explicit and put a shouldOmitFromReleaseNotes boolean in CHANGELOG.json?

Hm, the thing is that dev's don't add the Dependencies updated entry in CHANGELOG.json, it's automatically added if no entry was manually added to the CHANGELOG.

fragosti · 2018-08-23T00:24:22Z

packages/monorepo-scripts/src/utils/utils.ts

@@ -48,7 +48,7 @@ export const utils = {
                    };
                    packages.push(pkg);
                } catch (err) {
-                    utils.log(`Couldn't find a 'package.json' for ${subpackageName}. Skipping.`);
+                    // Couldn't find a 'package.json' for package. Skipping.


Yeah, too noisy and very unlikely to be an issue.

fragosti · 2018-08-23T00:30:22Z

packages/react-docs-example/public/css/basscss_responsive_type_scale.css

@@ -1,84 +0,0 @@
-/* Basscss Responsive Type Scale */
-/* Modified by Fabio Berger to include xs prefix */


Why delete all these files?

when I created the react-docs-example package, we had just separated out the docs rendering code from the website and were under the impression that others would want to use it. This didn't pan out. Additionally, I just don't feel confident given this most recent refactor that this package is ready for prime-time. We can always add it back once it is, but right now, I'd rather not give the impression that it's ready.

fragosti · 2018-08-23T00:32:32Z

packages/website/translations/chinese.json

@@ -13,7 +13,7 @@
    "TRADITIONAL_ASSETS": "传统资产",
    "DIGITAL_GOODS": "数字商品",
    "OFFCHAIN_ORDER_RELAY": "链下订单中继",
-    "OONCHAIN_SETTLEMENT": "链上最终结算",


Was wondering about this lol

Lol, silly typo.

fabioberger added 30 commits July 29, 2018 21:58

Remove all in-package monorepo-scripts by adding doc gen/upload and a…

f4a2e22

…ggregate release note publishing to publish script

Add missing exports discovered by generating compact typedoc JSON

4579e16

Improve doc gen script

9f74797

Fix package.json

454af2b

Remove no longer necessary include

11778db

Add back the 'clean' command

fe2787f

Make monorepo-scripts a private package now that no other package dep…

bb9237b

…ends on it

Move publish_release_notes to own script and other publish cleanup

1b24064

Move type

b7ff1fe

Move logic to publishUtils so can use as command-line and script import

b56fc69

Fix bugs in doc gen

79b1b6c

Update yarn.lock

79faf7d

Standardize child naming

b4f916d

Remove console

ba00cd9

Remove unused dep so that it's not re-compiled onWatcher

92fda0a

Change order of index.ts since it not dictates to ordering of the docs

0413e3e

Also add artifacts to src

c5fbc5c

Add /src/artifacts in order-utils to gitignore

9b24459

Update React-scroll, getting rid of the need to manually set the url …

1186912

…hash

Start refactoring docs to remove unnecessary configs given more conci…

3bdf600

…se TypeDoc JSON

Use our fork of react-scroll that fixes scrolling within an absolutel…

5e3bbbb

…y positioned div to work correctly. See: fisshy/react-scroll#330

Add padding above anchor titles so they show up nice when scrolled to

32e1c2a

Add callpath to properties

e5b93d1

Prefix menu and sections with markdown as that is all the should now …

2494af9

…be defined

Add contract-wrappers page and make topBar component more concise

0132286

Don't ignore compiler errors

b92b745

Fix connect docs

238eef6

We need to always include the globals.d.ts otherwise TS complains abo…

71a2f2d

…ut .json imports

Fix bug where if there were multiple matches, it wouldn't always take…

987971b

… the longest match

Remove unused var

b3473ff

Replace complicated camelCase to spaces logic with change-case package

27d329f

merge development

6125530

fabioberger removed request for BMillman19 and LogvinovLeon August 22, 2018 22:53

fabioberger added 4 commits August 23, 2018 00:08

Prettier fix

04e4fe2

Break with -1 so that the exit code of the loop if hit is non-zero

1417ee2

Missing/superfluous type exports from connect

2cef8d3

update yarn.lock

2a63592

fragosti approved these changes Aug 23, 2018

View reviewed changes

fabioberger added 18 commits August 23, 2018 13:56

Enable dry run of release publishing and handle git tags existing

2b38163

Fix prettier issues

15a34dc

Upgrade Typedoc to 0.12.0, which works with TS 3.x

0577ab9

Fix double assignment

f0f4f87

Fix CHANGELOG entry

aa9efd7

Remove superfluous dep

6c7dca7

Add catch and exit with non-zero

ae937cf

Look for all TS mapped types

29a9e1f

Fix comments

610caef

Remove docs catch-all endpoint

ae154f4

Improve doc commebnt

52e2c60

Remove excessive timestamp

1ca176e

Add missing keyu

3563fab

Add support for rending the Tuple type

28f077b

Remove duplicate type and remove nested interface

f9b222c

Stop nesting interfaces and add necessary type exports

c4c47d9

Fix linter

0ca64e3

Remove trailing slash

8eb4b6f

fabioberger merged commit 6e27324 into development Aug 23, 2018

fabioberger deleted the wrap-typedoc branch August 23, 2018 17:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved Docs & Removed monorepo-scripts from packages #924

Improved Docs & Removed monorepo-scripts from packages #924

fabioberger commented Jul 29, 2018 •

edited

Loading

albrow commented Aug 22, 2018

fragosti left a comment

fragosti Aug 22, 2018

fabioberger Aug 23, 2018

fragosti Aug 22, 2018

fabioberger Aug 23, 2018

fragosti Aug 23, 2018

fabioberger Aug 23, 2018

fragosti Aug 23, 2018

fragosti Aug 23, 2018

fabioberger Aug 23, 2018

fragosti Aug 23, 2018

fabioberger Aug 23, 2018

fragosti Aug 23, 2018

fabioberger Aug 23, 2018

fragosti Aug 23, 2018

fabioberger Aug 23, 2018

		@@ -1,84 +0,0 @@
		/* Basscss Responsive Type Scale */
		/* Modified by Fabio Berger to include xs prefix */

Improved Docs & Removed monorepo-scripts from packages #924

Improved Docs & Removed monorepo-scripts from packages #924

Conversation

fabioberger commented Jul 29, 2018 • edited Loading

Description

Testing instructions

Types of changes

Checklist:

albrow commented Aug 22, 2018

fragosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabioberger commented Jul 29, 2018 •

edited

Loading