Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The custom plug-in returns the result garbled #1559

Closed
griffenliu opened this issue May 9, 2020 · 10 comments · Fixed by #1598
Closed

The custom plug-in returns the result garbled #1559

griffenliu opened this issue May 9, 2020 · 10 comments · Fixed by #1598
Assignees
Labels
good first issue Good for newcomers

Comments

@griffenliu
Copy link
Contributor

code:

function _M.access(conf, ctx)
    ...
    log.warn(">>>>>>>>>>>>>>>>>>", data)
    local accept_type = ngx.req.get_headers()["Accept"]
    if str_contains(accept_type, "text/html") then
        return 200, data
    else
        return 200, decode(data)
    end
end

The data comes from the cache

log:

2020/05/09 10:55:59 [warn] 23640#0: *812 [lua] dfy-cache-plugin.lua:380: phase_fun(): >>>>>>>>>>>>>>>>>>{"data":"测试服务器8001"}
, client: 172.30.212.72, server: , request: "GET /read?_YCD_={%22read%22:%20[%2211111111111%22]} HTTP/1.1", host: "172.30.212.72:9080"

result:

{"data":"测试�务器8001"}

I think the reason is that the encoding is not configured in nginx.conf. Can we add utf-8 encoding into it or specify the encoding through configuration?

@moonming
Copy link
Member

I think the reason is that the encoding is not configured in nginx.conf.

can you show a example? I don't konw how to add utf-8 encoding in nginx.conf.

@griffenliu
Copy link
Contributor Author

http {
include mime.types;
default_type text/html;
charset utf8;
}

@moonming
Copy link
Member

moonming commented May 11, 2020 via email

@membphis membphis added beginner good first issue Good for newcomers labels May 12, 2020
@xxm404
Copy link
Contributor

xxm404 commented May 12, 2020

Can I modify APISIX Nginx template to solve this problem

@moonming
Copy link
Member

moonming commented May 12, 2020 via email

@xxm404
Copy link
Contributor

xxm404 commented May 12, 2020

ok please assign this issue to me.

@xxm404
Copy link
Contributor

xxm404 commented May 13, 2020

Add charset utf-8; to APISIX Nginx template
image
Modify example-plugin to test
image
restart apisix and execute curl -i 127.0.0.1:9080/api/bench/testSimple It still garbled
image

@xxm404
Copy link
Contributor

xxm404 commented May 14, 2020

Hi @griffenliu can you give me a minimal reproducible example, I can not reproduce this problem
In my test, the response normally

my test code

function _M.access(conf, ctx)
    -- core.log.warn("plugin access phase, conf : ", core.json.encode(conf))
    -- return 200, core.json.decode(data)
    local redis = redis_new()
    local ok, err = redis:connect('127.0.0.1', 6379)
    if not ok then
        return false, err
    end
    local data = redis:get('data')
    core.log.warn(">>>>>>>>>>>>>>>", data)
    return 200, core.json.decode(data)
end

log

2020/05/14 13:02:55 [warn] 17681#17681: *332 [lua] example-plugin.lua:70: phase_fun(): >>>>>>>>>>>>>>>{"data":"测试服务器8001"}, client: 127.0.0.1, server: , request: "GET /api/bench/testSimple HTTP/1.1", host: "127.0.0.1:9080"

response

HTTP/1.1 200 OK
Date: Thu, 14 May 2020 13:04:23 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX web server

{"data":"测试服务器8001"}

@griffenliu
Copy link
Contributor Author

sorry, I'm late. There is no problem with your example, which is the same as mine. The encoding was normal when I used curl to test, but it was garbled when I used the browser.

Using the browser response results below, you can see the difference between a correct response and an error response:

# this is garbled response
Connection: keep-alive
Content-Type: text/plain
Date: Mon, 18 May 2020 09:53:34 GMT
Server: APISIX web server
Transfer-Encoding: chunked
# this is normal response,  Content-Type with encoding: charset=utf8.
Connection: keep-alive
Content-Type: text/plain; charset=utf8
Date: Mon, 18 May 2020 09:55:06 GMT
Server: APISIX web server
Transfer-Encoding: chunked

@griffenliu
Copy link
Contributor Author

Hi @griffenliu can you give me a minimal reproducible example, I can not reproduce this problem
In my test, the response normally

my test code

function _M.access(conf, ctx)
    -- core.log.warn("plugin access phase, conf : ", core.json.encode(conf))
    -- return 200, core.json.decode(data)
    local redis = redis_new()
    local ok, err = redis:connect('127.0.0.1', 6379)
    if not ok then
        return false, err
    end
    local data = redis:get('data')
    core.log.warn(">>>>>>>>>>>>>>>", data)
    return 200, core.json.decode(data)
end

log

2020/05/14 13:02:55 [warn] 17681#17681: *332 [lua] example-plugin.lua:70: phase_fun(): >>>>>>>>>>>>>>>{"data":"测试服务器8001"}, client: 127.0.0.1, server: , request: "GET /api/bench/testSimple HTTP/1.1", host: "127.0.0.1:9080"

response

HTTP/1.1 200 OK
Date: Thu, 14 May 2020 13:04:23 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX web server

{"data":"测试服务器8001"}

Thanks for helping me solve my problem :)

SaberMaster pushed a commit to SaberMaster/incubator-apisix that referenced this issue Jun 30, 2020
nic-chen added a commit to nic-chen/apisix that referenced this issue Jul 6, 2020
* feature: implemented plugin `sys logger`. (apache#1414)

* bugfix(CORS): using rewrite phase and add lru cache for multiple origin  (apache#1531)

* change: updated the dashboard submodule to latest version. (apache#1540)

* doc: alter logger plugins documentations. (apache#1541)

* bugfix: Adding function to remove stale objects from kafka logger (apache#1526)

* bugfix: removed stale objects from tcp logger (apache#1543)

* bugfix: removing stale objects from udp logger (apache#1544)

* optimize: use buffer for plugin `syslog`. (apache#1551)

* plugin: add HTTP logger for APISIX  (apache#1396)

* bugfix: got 500 error when using post method in grpc-transcode plugin(apache#1566)

* bugfix: removed stale object in sys log. (apache#1557)

* feature(prometheus): support to collect metric `overhead` (apache#1576)

Fix apache#1534 .

* feature: support new field `exptime` for SSL object. (apache#1575)

fix apache#1571.

* doc: Added FAQ about how to reload your own plugin (apache#1568)

* doc: repair the white paper's url of README (apache#1582)

* chore: fix function name typo in ip-restriction (apache#1586)

* doc: added http logger Chinese docs (apache#1581)

* feature: support discovery center (apache#1440)

* doc:add chinese version for install doc (apache#1590)

* bugfix: incorrect variable name `hostCount` (apache#1585)

* doc: update kakfa logger plugin's cn version (apache#1594)

* doc: fix the doc style for *_logger.md (apache#1605)

* bugfix: raise error when none of the configured etcd can be connected (apache#1608)

Close apache#1561.

* test: updated style. (apache#1606)

* release: released 1.3 version. (apache#1558)

* bugfix(CLI): fixed garbled Chinese response in browser.  (apache#1598)

fix apache#1559

* change: updated prometheus to version 1.1 . (apache#1607)

* doc: add asf.yaml. (apache#1612)

* fix some doc style for response-rewrite* and health-check.md (apache#1611)

* makefile: add default check for install command (apache#1591)

* test cases: add doc and test cases for how to redirect http to https. (apache#1595)

* add FAQ about redirect http To https

* add test cases for serverless plugin and redirect plugin

Co-authored-by: rhubard <[email protected]>

* feature: add skywalking plugin. (apache#1241)

* doc: removed external links and docs. (apache#1619)

* doc: add coc file (apache#1589)

* bugfix: change the version of skywalking to 1.0-0 (apache#1624)

* bugfix(prometheus): the `overhead` should use milliseconds. apache#1615 (apache#1616)

Fix apache#1615

* feature: add option to include request body in log util (apache#1545)

* bugfix: fix typo of `instance_id` in skywalking plugin. (apache#1629)

* doc: added the link to discovery.md (apache#1631)

* change(ASF): add notifications to mailing list. (apache#1635)

* change(doc): style for HttpResponse section (apache#1634)

* doc(limit-count): fixed document description does not match source code.  (apache#1628)

close apache#1627

* bugfix(batch-requests): support cookie (apache#1599)

* feat(admin api): enhance `PATCH` method, allow to update partial data. (apache#1609)

* test: added test tests for skywalking. (apache#1621)

* doc: add skywalking plugin instructions (apache#1636)

* feature: support http_to_https in redirect plugin. (apache#1642)

* test: add test case for apache#1625 to test the filed of overhead (apache#1645)

* CLI: compatibility of benchmark script and apisix reload command on OSX (apache#1650)

* feature:  support to enable HTTPS for admin API (apache#1648)

* [log] Optimize the buffer size and flush time (apache#1570)

* yousali:<log>Optimize the buffer size and flush time

1. buffer=4096 is better for Writes of more than PIPE_BUF bytes may be nonatomic
2. flush=1. Since the log buffer is lowered, the flush time should also be lowered.

* yousali:<fix>

hi, I also made a test.

```
4096  Requests/sec:  16079.75
8192 Requests/sec:  16389.52
16384 Requests/sec:  16395.30
32768 Requests/sec:  16459.71
```
I think a log buffer size of 8192 or 16384 would be appropriate.

On the other hand, the refresh time of 3 seconds is still relatively long, and 1 or 3 seconds doesn't particularly affect QPS.

So I also agree with `buffer=16384 flush=1; `

* doc: add 'X-API-KEY' parameter for each interface of Admin API. (apache#1661)

* bugfix: wildcard certificates cannot match multi-level subdomains in … (apache#810)

* plugin: add consumer-restriction (apache#1437)

* feat: support resource name for route, service and upstream object. (apache#1655)

* [bugfix(CLI)]: check whether  the user has enabled etcd v2 protocol. (apache#1665)

* bugfix(CLI): generate the 'worker_cpu_affinity' config for Linux OS (apache#1658)

Fix apache#1657

* test case: formatted by `reindex`. (apache#1651)

* change: disable reuseport in development mode, it more easy to manage worker process. (apache#1175)

* test: add test case for route with `filter_func`. (apache#1683)

* doc: rename grpc-transcoding-cn.md to grpc-transcode-cn.md (apache#1694)

* fix bug: Execute command 'make run' multiple times, will start multiple processes (apache#1692)

Fix apache#1690

* doc(FAQ): added example for gray release. (apache#1687)

* change: set default reject code for some plugins (apache#1696)

plugin list:

limit-count
limit-conn
limit-req

* feature: ssl enhance (apache#1678)

support enable or disable ssl by patch method
support encrypted storage of the SSL private key in etcd
support multi snis

Fix apache#1668

* feature:  support body filter plugin `echo`. (apache#1632)

* doc: Update README_CN.md (apache#1705)

* change: use `iterate` to scan items in etcd.  (apache#1717)

related issue: apache#1685

* doc: added doc of key for limit-* plugins. (apache#1714)

* feature: support authorization Plugin for Keycloak Identity Server (apache#1701)

* feat[batch-request]: cp all header to every request (apache#1697)

* doc: updated main picture. (apache#1719)

* doc: update echo-cn.md (apache#1726)

* update `resty-etcd` to version 1.0 . (apache#1725)

* doc: health-check-cn.md (apache#1723)

* doc: add Chinese translation of authz-keycloak plugin (apache#1729)

* doc: Refactoring docs to support docsify (apache#1724)

* change: update `resty-radixtree` to version 1.9 . (apache#1730)

* feature: support the use of independent files to implement the load a… (apache#1732)

* feature: support the use of independent files to implement the load algorithm,
which is convenient for expanding different algorithms in the future.

* feature(echo): support header filter and access phases. (apache#1708)

* bugfix: id can be string object, which contains `^[a-zA-Z0-9-_]+$`. (apache#1739)

Fix apache#1654

* test: add test cases about the string id in `service` apache#1659 (apache#1750)

* update `lua-resty-raditree` to ver 2.0 . (apache#1748)

* refactory: collect `upstream` logic and put them in a single file. (apache#1734)

feature: support dynamic upstream in plugin.

here is a mini  example in `access` phase of plugin:

```lua
    local up_conf = {
        type = "roundrobin",
        nodes = {
            {host = conf.upstream.ip, port = conf.upstream.port, weight = 1},
        }
    }

    local ok, err = upstream.check_schema(up_conf)
    if not ok then
        return 500, err
    end

    local matched_route = ctx.matched_route
    upstream.set(ctx, up_conf.type .. "#route_" .. matched_route.value.id,
                 ctx.conf_version, up_conf, matched_route)
    return
```

* feature: implemented plugin `uri-blocklist` . (apache#1727)

first step: apache#1617

* doc: update `http-logger` plugins Chinese docs. (apache#1755)

* doc: update admin-api docs (apache#1753)

* doc: add oauth plugins Chinese docs. (apache#1754)

* bugfix: fixed configures of nginx.conf for security reasons (apache#1759)

removed working_directory and removed TLSv1 TLSv1.1 from ssl_protocols

* doc: update Chinese README.md (apache#1758)

* test: use longer ttl, avoid the cached item expired. (apache#1760)

* doc: updated k8s doc (apache#1757)

* bugfix: Fix for remote open ID connect introspection (apache#1743)

fix apache#1741

* test: added test cases. (apache#1752)

* bugfix: added `content-type` for admin API responses (apache#1746)

* feature: support etcd auth (apache#1769)

Fix apache#1713 , apache#1770

* plugin(heartbeat): use `info` log level when failed to report heartbeat. (apache#1771)

* optimize: Use lru to avoid resolving IP addresses repeatedly . (apache#1772)

* optimize: Use lru to avoid resolving IP addresses repeatedly .
Cached the global rules to `ctx` .

* optimzie: used a longer time interval for etcd and flush access log.

* optimize: return upstream node directly if the count is 1 .

* optimize: avoid to cache useless variable.

* doc: update Chinese README.md (apache#1763)

* doc: remove router `r3` . (apache#1764)

* release: released 1.4-0 version (apache#1742)

* bugfix(config etcd): when we reset the fetched data, `sync_times` also needs to be reset. (apache#1785)

* change: remove authentication type for cors plugin (apache#1788)

fix apache#1787

* rocks: fixed wrong source of 1.4. (apache#1783)

* change: 'get_plugin_list' API sorts the return list base on priority (apache#1779)

* test: format by tool `reindex`. (apache#1775)

* bugfix: missing argument `premature` because it was called by ngx.timer . (apache#1796)

* bugfix: return `404 Not Found` when the dashboard folder is empty.  (apache#1799)

close apache#1794

* doc: add guides for installing dependencies on fedora (apache#1800)

* doc: fixed some punctuation error in the document sample shell (apache#1803)

Co-authored-by: Ayeshmantha Perera <[email protected]>
Co-authored-by: Vinci Xu <[email protected]>
Co-authored-by: Nirojan Selvanathan <[email protected]>
Co-authored-by: YuanSheng Wang <[email protected]>
Co-authored-by: Yousa <[email protected]>
Co-authored-by: hiproz <[email protected]>
Co-authored-by: 罗泽轩 <[email protected]>
Co-authored-by: Scaat Feng <[email protected]>
Co-authored-by: qiujiayu <[email protected]>
Co-authored-by: dengliming <[email protected]>
Co-authored-by: dabue <[email protected]>
Co-authored-by: Wen Ming <[email protected]>
Co-authored-by: xxm404 <[email protected]>
Co-authored-by: rhubard <[email protected]>
Co-authored-by: Gerrard-YNWA <[email protected]>
Co-authored-by: 月夜枫 <[email protected]>
Co-authored-by: 仇柯人 <[email protected]>
Co-authored-by: stone4774 <[email protected]>
Co-authored-by: 琚致远 <[email protected]>
Co-authored-by: Kev.Hu <[email protected]>
Co-authored-by: QuakeWang <[email protected]>
Co-authored-by: agile6v <[email protected]>
Co-authored-by: Corey.Wang <[email protected]>
Co-authored-by: hellmage <[email protected]>
Co-authored-by: Eric Shi <[email protected]>
Co-authored-by: Shenal Silva <[email protected]>
Co-authored-by: jackstraw <[email protected]>
Co-authored-by: morrme <[email protected]>
Co-authored-by: ko han <[email protected]>
Co-authored-by: Joey <[email protected]>
Co-authored-by: YuanYingdong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants