Project skeleton for a Benthos plugin.
Bootstrap a new Benthos plugin in five minutes or less.
- Deploy Benthos with custom plugins to AWS Lambda with Serverless.
- Multi-arch and multi-platform builds published to GitHub with GoReleaser.
- Multi-stage Docker build for optimized production images.
- Images tagged using package version and commit checksum.
- Images pushed to GitHub Container Registry.
- Linting with golangci-lint.
- Unlimited, modular, and testable configs packaged with Blobpack.
- Node.js's npm package structure.
- Prettier code.
- Continuous testing and deployment with GitHub Actions.
- Keep a CHANGELOG.
- Consistent coding with EditorConfig.
- Create an empty (non-initialized) repository on GitHub.
- Clone the master branch of this repository with
Optionally, reset to the latest version with
$ git clone --single-branch [email protected]:makenew/benthos-plugin.git <new-blob-plugin> $ cd <new-blob-plugin>$ git reset --hard <version-tag> - Run
This will replace the boilerplate, delete itself, remove the git remote, remove upstream tags, and stage changes for commit.
$ ./makenew.sh - Create the required GitHub repository secrets.
- Review, commit, and push the changes to GitHub with
$ git diff --cached $ git commit -m "Replace makenew boilerplate" $ git remote add origin [email protected]:<user>/<new-blob-plugin>.git $ git push -u origin master - Ensure the GitHub action passes,
then release the initial version with
$ nvm install $ npm install $ npm version patch
If you want to pull in future updates from this skeleton, you can fetch and merge in changes from this repository.
Add this as a new remote with
$ git remote add upstream [email protected]:makenew/benthos-plugin.git
You can then fetch and merge changes with
$ git fetch --no-tags upstream
$ git merge upstream/master
Note that CHANGELOG.md is just a template for this skeleton.
The actual changes for this project are documented in the commit history
and summarized under Releases.
The service is distributed as a Docker container on GitHub Container Registry. The image is built identically to the standard Benthos container using the custom built Benthos binary.
$ docker run --read-only --init --publish 4195:4195 \
--volume "$(pwd)/benthos.yaml:/benthos.yaml" \
ghcr.io/makenew/benthos-plugin
$ git clone https://github.com/makenew/benthos-plugin.git
$ cd benthos-plugin
$ nvm install
$ npm install
$ make
$ npm test
Primary development tasks are defined under scripts in package.json
and available via npm run.
View them with
$ npm run
The source code is hosted on GitHub. Clone the project with
$ git clone [email protected]:makenew/benthos-plugin.git
A Go version compatible with the one specified in go.mod, GoReleaser, and golangci-lint.
To build the Serverless artifacts you will need Node.js with npm.
Be sure that all commands run under the correct Node version, e.g., if using nvm, install the correct version with
$ nvm install
Set the active version for each shell session with
$ nvm use
Install the development dependencies with
$ npm install
- Benthos configuration and unit tests live side-by-side in the
configdirectory. - Configuration is bundled with the deployed artifact to avoid the AWS environment variable 4K size limit.
- For each item in the
benthos.artifactsarray inpackage.json, a Serverless zip artifact will be built todistcontaining the Benthos binary and the corresponding config.
To add a new Serverless function nameed, e.g., foo:
- Create
config/foo.yamlandconfig/foo_benthos_test.yaml. - Update the
benthos.artifactsarray inpackage.jsonto includefoo. - Set the new Serverless function's
package.artifacttodist/foo.zip.
Serverless deployment is triggered by a release repository_dispatch on GitHub Actions.
Deployment may be triggered using on the web using a release workflow_dispatch on GitHub Actions.
GitHub Actions should already be configured: this section is for reference only.
The following repository secrets must be set on GitHub Actions:
AWS_DEFAULT_REGION: The AWS region Serverless will deploy to.AWS_ACCESS_KEY_ID: AWS access key ID.AWS_SECRET_ACCESS_KEY: AWS secret access key.GH_USER: The GitHub user's username to pull and push containers.GH_TOKEN: A personal access token that can trigger workflows.GPG_PRIVATE_KEY: The GitHub user's GPG private key.GPG_PASSPHRASE: The GitHub user's GPG passphrase.
These must be set manually.
The version and format GitHub actions require a user with write access to the repository, including access to trigger workflows. Set these additional secrets to enable the action:
GH_USER: The GitHub user's username.GH_TOKEN: A personal access token for the user.GIT_USER_NAME: The GitHub user's real name.GIT_USER_EMAIL: The GitHub user's email.
Please submit and comment on bug reports and feature requests.
To submit a patch:
- Fork it (https://github.com/makenew/benthos-plugin/fork).
- Create your feature branch (
git checkout -b my-new-feature). - Make changes.
- Commit your changes (
git commit -am 'Add some feature'). - Push to the branch (
git push origin my-new-feature). - Create a new Pull Request.
This Benthos plugin is licensed under the MIT license.
This software is provided by the copyright holders and contributors "as is" and any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the copyright holder or contributors be liable for any direct, indirect, incidental, special, exemplary, or consequential damages (including, but not limited to, procurement of substitute goods or services; loss of use, data, or profits; or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this software, even if advised of the possibility of such damage.