Pipeline

[localvar]

If you'd like to try the new feature in this PR, please follow this link.

Background

Initially, Easegress only supported the HTTP protocol, so many technical decisions only considered HTTP, and many implementation details were tightly bound to the HTTP protocol. In this way, when users request to support more protocols such as MQTT, TCP, etc., we have to re-implement a new PIPELINE for the protocol, which is not conducive to code maintenance and user use. Therefore, we hope to implement a general PIPELINE that supports all protocols, and we also hope instances of this PIPELINE can orchestrate filters of different protocols at the same time.

On the other hand, as the handler of request and response, filters should focus on the implementation of business logic, but the resilience filters, such as retryer and circuitbreaker, are for control logic. The mix of business logic and control logic results in that easegress must leverage the responsibility chain pattern to orchestrate filters, making the code difficult to understand and an ambiguous processing flow.

Finally, although one of the design goals of Easegress is traffic orchestration, and it achieves this goal well, it lacks the orchestration capabilities of complex APIs. For example, it can read a list of articles from an RSS feed, or send a message to Slack. But can't read a list of articles from an RSS feed and then post it to Slack.

Design

To solve the above problems, we redesigned the PIPELINE for the following goals：

• Protocol Independent: One pipeline implementation for all protocols and all protocols in one pipeline instance.
  • The context object should be a data container, data can be protocol dependent, but context should not.
  • Filters focus on business logic, they can be protocol dependent.
  • The pipeline is for processing flow orchestration, it focuses on control logic and must be protocol independent.
• Isolation of business logic and control logic.
  • Remove the resilence filters.
  • Define resilience policies in the pipeline.
  • The pipeline injects resilience policies into filters that need to support resilience when creating the filter.
• Multiple APIs Orchestration.
  • The context can hold multiple requests and responses, the requests and responses are grouped by namespace, and each namespace can have at most one request and one response.
  • Enhance the pipeline flow definition to bound a filter to a namespace, most filters can only access the request/response of that namespace. The new pipeline DSL is like below:
  • Add two new filters, RequestBuilder and ResponseBuilder, they can create a new request or response based on the existing requests and responses in the context.

The above goals are all for traffic orchestration. But with the enhanced orchestration capability, the ability of business logic extension has also been greatly improved. For example, Easegress can support the scenario we just mentioned via a pipeline, that's, get an RSS feed URL from the client request, read the RSS feed, then send the result to Kafka and post the article list to Slack.

                                                                    ┌────────────────┐
                                                         ┌─────────►│    RSS Feed    │
                                                         │          └────────────────┘
                                                         │
                                                         │
                                                         │
┌──────────────┐             ┌─────────────────┐         │          ┌────────────────┐
│    Client    ├────────────►│    Easegress    ├─────────┼─────────►│     Kafka      │
└──────────────┘             └─────────────────┘         │          └────────────────┘
                                                         │
                                                         │
                                                         │
                                                         │          ┌────────────────┐
                                                         └─────────►│     Slack      │
                                                                    └────────────────┘

And the pipeline definition would be (only the flow section is listed):

flow:
- filter: validator
- filter: rssRequestBuilder
  namespace: rss
- filter: rssProxy
  namespace: rss
- filter: responseAdaptor
  namespace: rss
- filter: slackRequestBuilder
  namespace: slack
- filter: kafka
  namespace: slack
- filter: slackProxy
  namespace: slack
- filter: responseBuilder

Detailed Changes

• filter

  • rename folder filter to filters
  • add filters.Kind for the filter meta information
  • filter interface is refactored to focus on runtime information
    • DefaultSpec, Description, and Results are moved to filters.Kind
    • Spec and Name are added
    • change the return value type of Kind from string to *filters.Kind
    • update the comment of Inherit to state it should not handle the lifecycle of the previous generation
  • add interface filters.Spec to simplify the spec related logic
  • add RequestBuilder and ResponseBuilder
  • remove Retryer, CircuitBreaker and TimeLimiter
  • filters are not called via the chain of responsibility pattern any more
  • the Proxy filter is refactored, especially the file pool.go

• pipeline

  • rename folder httppipeline to pipeline
  • add built-in filter END
  • add the support of namespace
  • add the support of filter alias
  • add the support of resilience
  • refactored to make pipeline protocol independent
  • refactored according to changes in filters

• context

  • now context is a struct instead an interface, and only focus on data (request/response) management

• protocol independent

  • add protocols package and interface Request, Response
  • add sub-packages inside protocols for protocol specific implementation
  • easemonitormetrics, trafficcontroller are refactored to be protocol independent

• resilience

  • add resilience package

• tracing

  • tracing are now using zipkin B3 format

• cluster

  • drop the support of dynamic cluster management
  • depreciated configuration method is removed

• other changes

  • HTTPServer is refactored, especially the file mux.go
  • legacy files are removed
  • some new utility packages are added

[codering]

one typo：pkg/object/httpserver/mux.go#95

mathAllHeader should be matchAllHeader.

[localvar]

If you'd like to try the new feature in this PR, please follow this script:

1. Clone Easgress repository with the command git clone https://github.com/megaease/easegress.git, if you have already cloned the repository, please execute git pull --all to pull the latest code.
2. Enter the local repository and switch to the pipeline branch with the command git checkout pipeline.
3. Build Easegress (please make sure you have Go v1.17+ installed) with the command make.
4. Download the attached easegress.zip and extract the 4 YAML files in it into the bin folder.
5. Edit rss-pipeline.yaml to replace the Slack webhook URL, to create a new slack webhook URL, please follow this document.
6. Edit stock-pipeline.yaml, same as rss-pipeline.yaml, the Slack webhook URL in this file needs to be changed. And beside that, the app_id & app_secret of the buildXchRateRequest filter need to be updated every day, please open https://www.mxnzp.com/doc/detail?id=27, click "获取指定货币编号的汇率信息" and copy the app_id & app_secret in the example.
7. In the bin folder, execute ./easegress-server -f config.yaml to start Easegress.
8. Open a new shell and execute the below commands to create the HTTP server and pipelines:

./egctl object create -f http-server.yaml
./egctl object create -f rss-pipeline.yaml
./egctl object create -f stock-pipeline.yaml

9. Execute curl -H X-Rss-Url:https://www.coolshell.cn/rss http://127.0.0.1:8080/rss, easegress will read the RSS feed, build the article list into a Slack message, and then send it to Slack. You may use an RSS feed other than coolshell, but please note the maximum message size Slack allowed is about 3K, so you will need to limit the number of articles returned by the RSS feed of some sites(e.g. Hack News).
10. Execute curl http://127.0.0.1:8080/stock?code=000001, easegress will fetch the stock price in RMB and foreign exchange rate, calculate out the stock price in USD, build the prices into a Slack message, and then send it to Slack. Please note the backend service only support the Chinese stock market currently, and please make sure the stock code is a valid one(start with 0 or 6 and contain six digits, but not all codes that meet this requirement are valid)
11. If you'd like to try more, you may update the YAML files and apply the changes to Easegress. For config.yaml, you have to restart Easegress to take the changes into effect, for the other 3 files, you may execute ./egctl object update -f <yaml file> to apply the changes dynamically. And if something went wrong and you don't know what to do, you can execute rm -rf member data log to reset the environment and then restart from step 7.

[haoel]

LGTM