Skip to content

Commit

Permalink
README updates and cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
cwensel committed Sep 29, 2023
1 parent 750bb08 commit c9c5c19
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 68 deletions.
91 changes: 26 additions & 65 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ See the [ROADMAP](ROADMAP.md) for planned capabilities.

For example scenarios, see

- [How To](https://docs.clusterless.io/guide/1.0-wip/howtos/index.html) - step-by-step guides
- [clusterless-aws-example](https://github.com/ClusterlessHQ/clusterless-aws-examples) - simple examples to start with
- [aws-s3-log-pipeline](https://github.com/ClusterlessHQ/aws-s3-log-pipeline) - end-to-end sample pipeline for
processing AWS S3 access logs
Expand Down Expand Up @@ -76,7 +77,7 @@ And can be implemented as:

- Docker images (Python, Java, Node, etc)
- Serverless functions (Python, Java, Node, etc)
- Native services (AWS SageMaker, AWS Athena, AWS Glue, etc)
- Native services (AWS SageMaker, AWS Athena, AWS Glue, etc.)

Where the intent isn't to have functional parity across substrates, but to ease secure and reliable interoperability
between them.
Expand All @@ -85,82 +86,42 @@ Currently supported cloud substrates:

- AWS

## Prerequisites
## Installing

_If you already have Node and npm, skip to the AWS CDK install._

Install [nvm](https://github.com/nvm-sh/nvm):

> curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
Install node/npm:

> nvm install --lts
Install the AWS CDK:

> npm install -g [email protected]
after install, make sure to bootstrap your account and region:

https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html#bootstrapping-howto

> cdk bootstrap aws://ACCOUNT-NUMBER-1/REGION-1 aws://ACCOUNT-NUMBER-2/REGION-2 ...
Install Java 17 via [sdkman](https://sdkman.io):

> sdk install java 17.0.8-graalce
## Installing Clusterless

The Clusterless CLI app `cls` is downloadable from GitHub releases:

- https://github.com/ClusterlessHQ/clusterless/releases

The `bin` folder will have:

- `cls` - the root CLI interface
- `cls-aws` - the AWS interface only used as an escape hatch and testing
See: https://docs.clusterless.io/guide/1.0-wip/install-quickstart.html

## Running

> cls --help
```text
Usage: cls [-hVv] [-D=<String=String>]... [-P=<providerNames>]... [COMMAND]
Usage: cls [-hVv] [--output=<format>] [-D=<String=String>]...
[-P=<providerNames>]... [COMMAND]
-D, --property=<String=String>
key=value properties, will be passed down
-h, --help Show this help message and exit.
Optional key=value properties, will be passed down.
-h, --help Show this help message and exit.
--output=<format> Print results using given format.
Values: table, json, csv, tsv (default: table)
-P, --providers=<providerNames>
provider substrates to target
-v, --verbose Specify multiple -v options to increase verbosity.
For example, `-v -v -v` or `-vvv`
-V, --version Print version information and exit.
Provider substrates to target.
-v, --verbose Specify multiple -v options to increase verbosity.
For example, `-v -v -v` or `-vvv`
-V, --version Print version information and exit.
Commands:
help Display help information about the specified command.
config manage local and global configuration settings
show display details about providers, components, and project models
bootstrap initialize a cloud provider placement
deploy deploy a project into a declared placement
destroy destroy a project deployed a declared placement
diff compare local project changes with a deployed a declared placement
local support for executing workloads locally
verify verify project changes with a provider
help Display help information about the specified command.
config Manage local and global configuration settings.
show Display details about providers, components, and project models.
bootstrap Initialize a cloud provider placement.
deploy Deploy a project into a declared placement.
destroy Destroy a project deployed a declared placement.
diff Compare local project changes with a deployed a declared
placement.
local Support for executing workloads locally in a terminal.
verify Verify project changes with a provider.
placements List all visible placements with deployments.
projects List all deployed projects.
```

Every region must be bootstrapped by the `cls` app before it can be used:

> cls bootstrap help
To bootstrap a region:

> cls bootstrap --region us-east-1 --account 123456789012 --profile my-profile --stage DEV
If any arguments are missing, `cls` will prompt for them.

Note the `--stage` option creates a namespace for all resources, so you can have multiple environments in the same
account.

## Building (optional)

To build the CLI app locally from source, clone the repo and run:
Expand Down
12 changes: 10 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,19 @@ Most things in the cli have yet to be implemented.
3. Custom Lambda based arc workloads
4. High frequency S3 listener boundary
1. For aggregating objects that arrive within a lot interval
2. https://docs.clusterless.io/reference/1.0-wip/components/aws-core-s3-put-listener-boundary.html
5. Native resources and workloads
1. AWS Glue database and catalog updates
1. https://docs.clusterless.io/reference/1.0-wip/components/aws-core-glue-database.html
2. https://docs.clusterless.io/reference/1.0-wip/components/aws-core-glue-table.html
3. https://docs.clusterless.io/reference/1.0-wip/components/aws-core-glue-add-partitions-arc.html
2. AWS Athena CTAS/INSERT INTO queries (for chaining SQL)
3. AWS Sagemaker training/validation
6. Common data processing workloads
1. Data reformatting (from text/json to binary/parquet)
2. https://github.com/ClusterlessHQ/tessellate
1https://github.com/ClusterlessHQ/tessellate
2. Dynamic data repartitioning (partitions based on data like timestamps)
3. https://github.com/ClusterlessHQ/tessellate
1. https://github.com/ClusterlessHQ/tessellate
3. Predicate/duplicate index creation and data filtering
7. Join Barrier implementations
8. Scheduled arc executions
Expand All @@ -39,3 +43,7 @@ Most things in the cli have yet to be implemented.
10. Pluggable modules for providing third-party services
11. Localstack support for faster testing AWS scenarios
12. Alternate substrates/providers
1. Azure
2. GCP
3. Digital Ocean
4. OCI
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

@CommandLine.Command(
name = "projects",
description = "list all deployed projects"
description = "List all deployed projects."
)
public class ProjectsCommand extends CommandWrapper<ProjectsCommandOptions> {
public ProjectsCommand() {
Expand Down

0 comments on commit c9c5c19

Please sign in to comment.