Lua transform not able to find lua-http package

[atibdialpad]

Use-Case : Make an http call from the transform.
Since there is no such vector transform which does that, I resorted to using the LUA transform and use lua-http to make the http requests.

Steps:

1. Install and run vector
2. Install lua rocks via : apt-get install luarocks
3. Install lua headers: apt install liblua5.3-dev
4. Install lua http (which install all the dependencies compat53, bit32, cqueues, luaossl, basexx, lpeg, lpeg_patterns, binaryheap >= 0.3, fifo : luarocks --tree=/usr/share/lua/5.3/ install http
5. Now, when I use lua http from my transform, I get the below stacktrace :
   May 03 09:09:14.232 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("[string "?"]:4: module 'http.request' not found: no field package.preload['http.request'] no file '/usr/share/lua/5.3/http/request.lua' no file '/usr/local/share/lua/5.3/http/request.lua' no file '/usr/local/share/lua/5.3/http/request/init.lua' no file '/usr/local/lib/lua/5.3/http/request.lua' no file '/usr/local/lib/lua/5.3/http/request/init.lua' no file './http/request.lua' no file './http/request/init.lua' no file '/usr/local/lib/lua/5.3/http/request.so' no file '/usr/local/lib/lua/5.3/loadall.so' no file './http/request.so' no file '/usr/local/lib/lua/5.3/http.so' no file '/usr/local/lib/lua/5.3/loadall.so' no file './http.so' stack traceback: [C]: in ? [C]: in function 'require' [string "?"]:4: in function <[string "?"]:1>") } internal_log_rate_secs=30
6. I have a feeling it is something to do with where Vector is looking for the relevant packages and where it is installed but am not able to fix it.
   @jszwedko any ideas on why this can be happening ?

My transform config has search_dirs = ["/usr/share/lua/5.3"]

[atibdialpad]

I changed the search_dirs to "/usr/share/lua/5.3/share/lua/5.1" becasue :
>>VECTOR_DEV# ls /usr/share/lua/5.3/share/lua/5.1/ basexx.lua binaryheap.lua compat53 cqueues cqueues.lua fifo.lua http lpeg_patterns openssl openssl.lua re.lua
After which vector was able to find the http package, now it fails in finding the lpeg package

May 03 09:43:37.578 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: module lpeg not found:
no field package.preload['lpeg']
no file '/usr/share/lua/5.3/share/lua/5.1/lpeg.lua'
no file '/usr/share/lua/5.3/lib/lua/5.1/lpeg.lua'
no file '/usr/local/share/lua/5.3/lpeg.lua'
no file '/usr/local/share/lua/5.3/lpeg/init.lua'
no file '/usr/local/lib/lua/5.3/lpeg.lua'
no file '/usr/local/lib/lua/5.3/lpeg/init.lua'
no file './lpeg.lua'
no file './lpeg/init.lua'
no file '/usr/local/lib/lua/5.3/lpeg.so'
no file '/usr/local/lib/lua/5.3/loadall.so'
no file './lpeg.so'
stack traceback:
[C]: in ?
[C]: in function 'require'
/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: in main chunk
[C]: in function 'require'
[string "?"]:4: in function <[string "?"]:1>") } internal_log_rate_secs=30

Wish there was a simple installation as part of vector rather than having to manually install packages and take care of correct versions and package paths.

[atibdialpad]

Managed to link the lpeg.so (it was a compiled library) by adding package.cpath to the lua code.
Current status :
May 03 10:27:23.921 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("error loading module 'lpeg' from file '/usr/share/lua/5.3/lib/lua/5.1/lpeg.so':
/usr/share/lua/5.3/lib/lua/5.1/lpeg.so: undefined symbol: lua_tointeger
stack traceback:
[C]: in ?
[C]: in ?
[C]: in function 'require'
/usr/share/lua/5.3/share/lua/5.1/http/request.lua:1: in main chunk
[C]: in function 'require'
[string "?"]:5: in function <[string "?"]:1>") } internal_log_rate_secs=30

[daurnimator]

4\. **luarocks --tree=/usr/share/lua/5.3/ install http**

you probably wanted: luarocks --tree /usr/ install http
inside of the tree luarocks adds share/lua/5.3 for you.

I changed the search_dirs to "/usr/share/lua/5.3/share/lua/5.1" becasue :
>>VECTOR_DEV# ls /usr/share/lua/5.3/share/lua/5.1/ basexx.lua binaryheap.lua compat53 cqueues cqueues.lua fifo.lua http lpeg_patterns openssl openssl.lua re.lua

looks like you may need to add --lua-version 5.3. your luarocks thinks it's compiling for lua 5.1

[atibdialpad]

Thanks @daurnimator will add the lua-version and check.
Also @jszwedko I do not want luarocks to install Lua for me. I want it to use the Lua already installed as part of Vector. Is it possible to do that ?

[jszwedko]

@atibdialpad Vector will use the version of Lua bundled with it. I see that the luarocks package has a dependency on the lua package so it will install it when you install luarocks, but Vector won't use that one.

[atibdialpad]

@daurnimator does "--lua-version 5.3" work in the luarocks cmd line ?
I still see luarocks installing packages for lua 5.1 even when I specify the lua-version. I see this is still open luarocks/luarocks#719

[atibdialpad]

[Update]
So I could get lua http work with lua5.3 but not with vector

1. install lua 5.3

2. install liblua5.3-dev

3. install luarocks

4. install luahttp (--tree=/usr/ --lua-version 5.3)
   ~# luarocks config --lua-ver
   5.3

5. http code works
   package.path = package.path .. ';/usr/share/lua/5.3/?.lua'
   package.cpath = package.cpath .. ';/usr/lib/lua/5.3/?.so'
   local lpeg = require "lpeg"
   local http_request = require "http.request"
   local my_req = http_request.new_from_uri("http://httpbin.org/get")
   local headers, stream = assert(my_req:go())
   for field, value in headers:each() do
   print(field, value)
   end
   local body = assert(stream:get_body_as_string())
   print(body)

   {
   "args": {},
   "headers": {
   "Host": "httpbin.org",
   "User-Agent": "lua-http/0.4",
   "X-Amzn-Trace-Id": "Root=1-6090fbf7-36933a012d2959b04e41d834"
   },
   "origin": "103.35.127.33",
   "url": "http://httpbin.org/get"
   }

6. However when I run Vector Lua transform with http req, I get this error
   May 04 07:49:32.101 ERROR transform{component_kind="transform" component_name=my_transform_id component_type=lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("error loading module '_cqueues' from file '/usr/lib/lua/5.3/_cqueues.so':
   /usr/lib/lua/5.3/_cqueues.so: undefined symbol: luaL_openlibs
   stack traceback:
   [C]: in ?
   [C]: in ?
   [C]: in function 'require'
   /usr/share/lua/5.3/cqueues/auxlib.lua:2: in function </usr/share/lua/5.3/cqueues/auxlib.lua:1>
   (...tail calls...)
   [C]: in function 'require'
   /usr/share/lua/5.3/http/client.lua:1: in main chunk
   [C]: in function 'require'
   /usr/share/lua/5.3/http/request.lua:5: in main chunk
   [C]: in function 'require'
   [string "?"]:6: in function <[string "?"]:1>") } internal_log_rate_secs=30

Looking into this however @jszwedko @daurnimator any leads ?

[atibdialpad]

Hey @daurnimator is there any dependency of lua cqueues that I am missing here ?

[atibdialpad]

[Update]
I was not able to make the vector pipeline work with lua http :-( I switched to socket and vector seems happy with it . Something like this
local io = require("io")
local http = require("socket.http")
local ltn12 = require("ltn12")
b,c,h = http.request{
url = "https://xyz",
sink = ltn12.sink.file(io.stdout)
}
event.log.http_b = b
event.log.http_c = c

[khvzak]

@atibdialpad
You probably need to re-build vector with .cargo/config:

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-args=-rdynamic"]

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-args=-rdynamic"]

to export Lua symbols.

[atibdialpad]

Hey @khvzak , thanks for the info. Can you please explain a bit what that will do. Sorry I new to both Lua and Vector.

[atibdialpad]

Hey @khvzak , I already see these in https://github.com/timberio/vector/blob/master/.cargo/config . I am using a x86_64 linux gnu machine.

[khvzak]

Sorry @atibdialpad I can confirm that symbol luaL_openlibs does not exported (you can check dynamic symbols using nm -D).
Unfortunately with bundled lua there is not simple way to include this symbol :(

[atibdialpad]

Thanks @khvzak
I will go ahead with the socket way.

[khvzak]

I believe this issue should be solved in the new vector 0.15.1 after switching to mlua.

[jszwedko]

Just noting that the mlua change won't go out until 0.16.0, but you can try it now with the latest nightly build to see if it resolves your issue.

0.15.1 just includes a regression bug fix

[jszwedko]

I believe this issue is resolved, but please leave a comment if you still see it.