Long time no sync! #9
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
May 7, 2018 11:37
Expecting documents will want to populate a field called `type` to disambiguate from other "types" in the index.
The parser is customizable. Not yet sure how easy it easy to break the bits that help parse the metadata & attributes that users may want to aggregate on (eg partition).
- Handle "partitions" & "partitions:". - Handle partition in extra metadata specially.
- Named args for the regex & field instead of [0] & [1..] - Better handling of shard failures which would otherwise result in lots of log spam.
- Plugins no longer defined as prototypes to avoid `this` headaches. State stored in the plugin's closure. - No longer using elasticsearch.js since no support for the kind of auth I need. - A plugin's start() should return a Promise. Still have some work to clean up my async messiness. - No longer supporting pure node kafka client library for now. - New `parse` & `serialize` class of plugins to help disambiguate.
Useful for building the url based on environment `this.env` is defined for the function.
Useful for doing things like batch processing prior day's index.
- Need to use nock().delay() instead of delayBody() or will return something unexpected to superagent. - Support multiple inflight bulk requests, particularly when shutting down.
- Minor version bump since this includes major async/await change. - Using yarn instead of npm cuz I wanna be cool. - Not using my version of superagent.
Makes for simpler tests in that don't have to dig out of Logbus.events.
This allows to factor out boiler-plate pipeline bits into separate files. Then pipelines can define a `templates` section to inform logbus where to load them from. Then pipeline stages can provide `template` to merge in that template's definition in with the stage. The implementation is janky, mostly a copy-paste of how plugins are loaded. The index.js is a poor example of how to write code, but it works.
- using native EventEmitter until something else actually needed - addressed a lot of the awkward code in main index.js script
Users can more easily build their own images via `npm install logbus` and then add whatever dependencies they need.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Not sure if still want to be bothered with this... I just wrapped up some work. Didn't realize that I never submitted a PR that included 832b28c which is me trying to nodejs better. I did give some thought to compatibility, but wouldn't surprise me if has breaking changes. No one is using this so I'm not sweating it too much. Oh, I added a simple ETL example. I'm using it for one-off re-indexing efforts. Might be useful to you.