Publishing the PK CLI for General Use

[CMCDragonkai]

Requirements of this design

The new structure of our projects is:

Polykey

• js-polykey renames to Polykey
• keeps all issues in js-polykey
• license: GPL3 (possible move to APL3)
• npm: polykey
• not hosted on nix (it's not an application)
• supports network API
• embeddable as a library
• bin/ code is extracted to Polykey-CLI
• maintains wiki: theory, reference, guides, tutorials
• maintains community discussions
• main README.md (open source entry point)
• requires cross-platform gitlab runner for building and QA of possibly-native addons

Polykey-CLI

• license: GPL3
• npm: polykey-cli
• nix: polykey-cli
• extracts bin/ code out of js-polykey
• requires cross-platform gitlab runner for building and QA

Polykey-Desktop - electron/tauri wrapper around library code

• license: GPL3
• not hosted on npm (it's not a JS library or application)
• nix: polykey-desktop
• requires cross-platform gitlab runner for building & QA

Polykey-Mobile - nativescript wrapper around library code

• license: unknown
• not hosted on npm (it's not a JS library or application)
• not hosted on nix (it's not a desktop application)
• requires cross-platform gitlab runner for building & QA

We may not need to split out to Polykey-CLI just yet, but it can be done later.

Additional context

• Verify that pre-release and release jobs work #390 - this should all be done automatically in the CI/CD
• Polykey naming and repository structure #426 - which setup the repository structure
• https://docs.npmjs.com/deprecating-and-undeprecating-packages-or-package-versions

Specification

1. - Undeprecate the NPM polykey, as Polykey Core library now uses polykey
2. - Publish to polykey
3. - Deprecate @matrixai/polykey and make it point to polykey and make the deprecated js-polykey and make it point to polykey
   • https://www.npmjs.com/package/@matrixai/polykey
   • https://www.npmjs.com/package/polykey
   • https://www.npmjs.com/package/js-polykey

[CMCDragonkai]

Review CLI exit codes in relation to sysexit standards

Exceptions are currently using sysexit codes from this stanard: https://www.freebsd.org/cgi/man.cgi?query=sysexits&apropos=0&sektion=0&manpath=FreeBSD+4.3-RELEASE&format=html

It's a good idea to review these exit codes and see how to ensure we all of our CLI commands are returning the proper exit codes.

[CMCDragonkai]

Bundling and Minification Step in our Release Executables

Recommend attempting the usage of a minifier before we package it up using pkg. Right now tsc doesn't minify the JS code nor does it do any optimisations.

Here's some resources:

• http://typescript-react-primer.loyc.net/minification.html
• https://github.com/privatenumber/minification-benchmarks
• https://github.com/vercel/ncc - this is by the same people using vercel/pkg, the ncc would be used first to take our dist into a single file, so I'm not sure what happens to our json data, and then pkg can run against the single file

Basically, right now tsc produces the dist. We would want to pass the dist into a minifier/bundler like esbuild or terser or ncc, and then pass the final single file result to pkg.

Ideally the "bundling" step should also perform optimisations like DCE.

This should reduce the size of the final executable by a significant amount, possibly 70%, and it should also help with the performance of the code as in #277 since it won't need to load different files.

However for npm publishing, it will still use dist as normal unminified and all. This makes it useful for libraries. This bundling step would only be done temporarily within our release builds to executables. It does not apply to application nor docker target.

[CMCDragonkai]

Should review the CLI error handling overall, see this thread: #278 (comment) and https://github.com/MatrixAI/js-polykey/wiki/API-Design

[CMCDragonkai]

Noticed that sometimes we need our --verbose messaging because by default only WARN messages are shown. Now some message are useful for the user. Some messages are useful for the debugger/developer. So we may need to raise our message levels internally to DEBUG. That way info messages useful to the user can still be shown. On the other hand, I'm not sure if I like this as well.

Perhaps instead we need another level, or level of messages that is given interactively. That is one could say that messages intended for interactive usage can be this.logger.warn.

Alternatively, interactive messages may need to go directly to process.stderr.write rather than using the logger even though the logger does use STDERR.

Not entirely sure atm. Because this has to interact with formatting and error handling too.

[CMCDragonkai]

Pkg seems quite limited:

• https://github.com/vercel/pkg/discussions/1388
• https://github.com/vercel/ncc

We need to see if alternatives are better.

Related issues:

• Optimise pkg bundling so that it is not bundling useless markdown files (combine with esbuild?) Polykey-CLI#23

[CMCDragonkai]

@tegefaulkes this is our epic for PK CLI. We can attach issues related to it here.

Also all of the issues relating CLI stuff will get moved to Polykey-CLI.

[CMCDragonkai]

First we factor out, then we move issues.

[tegefaulkes]

K. I can split out the CLI soon. There are a lot of pending changes in the agent migration PR, so doing them side by side is not ideal. I can get the agent migration to a working state and merge that, some of the remaining tasks for that can be a 2nd PR.

So I can merge #525, then split out the CLI, and complete the remainder of the migration. The remaining part should be consolidating the quic server and reverse connection handling into the node connection manager.

[CMCDragonkai]

Moving to Polykey-CLI.

[CMCDragonkai]

I'd say this is done... sort of. There's nothing more to do for this issue, but just massaging Polykey-CLI to be better.

More specific issues can be created for integrating a bundler, compression, and maybe even replacing pkg with our own creation.

[CMCDragonkai]

Reopening this right now, cause there are few subissues to solve here as part of 6th testnet deployment.

[CMCDragonkai]

Now that things are built and PK CLI master and staging is synced, this is done.