Builtin modules overloading #7774

igormunkin · 2022-10-04T14:38:08Z

@Totktonada: The idea is to offer something similar to Perl's dual-file modules. We should elaborate package loader rules to allow convenient writing of built-in libraries replacements.

After embedding checks to the core binary, plain require('checks') will return embedded module. So it won't be possible to run repository module tests (in reusable_testing or with repository CI) with repository code until it overloads build-in code. Until [1] is resolved there are no any other way to deal with it. (And even after [1] would be resolved, it may worth to leave it like this to support older versions with built-in checks.) 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7726

After embedding checks to the core binary, plain require('checks') will return embedded module. So it won't be possible to run repository module tests (in reusable_testing or with repository CI) with repository code until it overloads built-in code. Until [1] is resolved there are no any other way to deal with it. (And even after [1] would be resolved, it may worth to leave it like this to support older versions with built-in checks.) 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7726

After embedding checks to the core binary, plain require('checks') will return embedded module. So it won't be possible to run repository module tests (in reusable_testing or with repository CI) with repository code until it overloads build-in code. Until [1] is resolved there are no any other way to deal with it. (And even after [1] would be resolved, it may worth to leave it like this to support older versions with built-in checks.) 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7726

This commit starts a series, which refactors built-in module registration mechanisms. The series aims several goals: * Offer a standard tarantool specific way to register a built-in module. It allows to change the registration process for all built-in modules at once. * Make the API of such functions simple and clean. * Eliminate known problems of the existing approach to register built-in modules. This particular commit offers two new functions: * luaT_setfuncs() * luaT_newmodule() The former replaces `luaL_register(L, NULL, funcs)`. The latter replaces `luaL_register()` and `luaL_register_module()` to some extent. Let's look on examples below. ## How it works now First of all, consider the [documentation][1] about the `luaL_register()` function. Then look on the examples. ```c luaL_register(L, "foo.bar", funcs); ``` Creates a table `package.loaded['foo.bar']` and `_G.foo.bar` (if neither of them exists), fill it with the functions and pushes it to the stack. What is not ok: * Pollutes `_G`. * `package.loaded` is written flat, but `_G` is written deeply. * No control over (un)intended module rewritting. It also worth to note that we can't change this function, because it is part of the Lua API. So we need another function for our registration process. Another usage of the function makes it even more confusing: ```c luaL_register(L, NULL, funcs); ``` Does it creates a table, fills it with the functions and pushes it to the stack? No. Unlike other usages, it fills a table on the top of the stack with the functions and leaves it on the stack. And it actually has no relation to module registration. There is tarantool specific function `luaL_register_module()`, which looks as an attempt to solve some of the problems. ```c luaL_register_module(L, "foo.bar", funcs); ``` It doesn't touch `_G` and, surprisingly, changes how a module is written to `package.loaded`. The call creates a table (or finds it) in `package.loaded.foo.bar`, fills it with the functions and pushes to the stack. *This* function writes `package.loaded` deeply. It leaves us with `luaL_register()` for assigning `_G.foo.bar` (with pollution of `package.loaded`). It leaves us with `lua_register()` for setting functions into a table on the stack (with confusing name in this context and confusing contract). And we has no a function to just assign `package.loaded[modname]` without the deep tables creation and with appropriate checks. ## How it will work ```c luaT_newmodule(L, "foo.bar", funcs); ``` Create a table, set it to `package.loaded['foo.bar']` and push into the stack. Assert that the table doesn't exist. ```c luaT_setfuncs(L, funcs); ``` Add functions into a table on top of the stack. ```c luaL_findtable(L, LUA_GLOBALSINDEX, "foo.bar", 0); luaT_setfuncs(L, funcs); ``` Find/create a table `_G.foo.bar` and add functions to it. [1]: https://www.lua.org/manual/5.1/manual.html#luaL_register Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

See the previous commit regarding the idea behind this refactoring. The src/box/lua part is updated in a next commit, only `src/lua` is updated here. It is split to ease reading. The patch is mostly straightforward, I'll highlight several interesting places. * `compat` `_G.internal.compat` was created by `luaL_register()`. Removing of this call drops `_G.internal`: it is not created anymore. * `fiber.channel` and `fiber.cond` The initialization code add fields into fiber module. So we call `require('fiber')` here. The alternative is to access `package.loaded.fiber` directly, however I plan to eliminate such accesses in further patches. (Built-in modules registration process will use its own table instead of `package.loaded`.) * `http.client` Here we also eliminate a direct access to `package.loaded`, which is important for implementing tarantool#7774. * `utf8` This module is also set to `_G.utf8` for compatibility with Lua 5.3. It remains unchanged. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

The idea of the refactoring is described in a previous commit. The change is straighforward: * Use `luaL_findtable()` to find/create a nested table in `_G`. * Use `luaT_setfuncs()` to assign functions to a table. * Use `luaT_newmodule()` to register a module for `require()`. `require('box.internal.<...>')` calls are replaced with `box.internal.<...>`, because `box` submodules are not registered as modules anymore (IOW, are not placed into `package.loaded`). However `_G.box == require('box')`, it remains unchanged. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

There are modules that are implemented as two parts: a Lua/C module for internal use and a public module written on Lua. There is a practice to name both parts the same: just capture the internal part within the public part and rewrite `package.loaded` then. This name overlap is confusing at reading the sources and complicates debugging. And it conflicts with a built-in module loading logic that will be implemented for tarantool#7774. Let's use `foo.lib` for the internal part and `foo` for the public one. This approach is already used in some built-in modules. src/box/lua/upgrade.lua requires src/box/lua/xlog.lua, so changed the loading order. Eliminated extra `internal` field in `uri.lib`, becase the whole module is internal. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

The module can't be pulled as `require('compat')`, because it is tarantool specific tuning mechanism, not a general compatibility module. The API is exposed as `require('tarantool').compat`. Before this patch the module was loaded as `package.loaded.compat`, captured and then removed from `package.loaded`. However it would be easier to follow if we'll just assign an internal name for the module. The module is renamed to `internal.compat`. The Lua/C part of the module is renamed to `internal.compat.lib`. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

It is planned to allow to overload a built-in module by an external one. The implementation assumes storing of the built-in modules in a table that is different from `package.loaded`. It means that accessing a module as `package.loaded[<module_name>]` may fail. However we still have a guarantee that a registered module is accessible using `require()`. Use it for accessing built-in modules from our code. Notes on different modules: * `console` Reorder the print/log callbacks to place the print callback first. If something is going very wrong (say, someone removes the internal.print module), things will be consistent: both callback will not be set. * `session` `box` is accessible as `require('box')` and as `_G.box`. Use the latter. src/lua/serializer.c and src/lua/digest.c also access `package.loaded` directly. I left them out of scope here, it requires a bit more in-depth look. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

This commit starts a series, which refactors built-in module registration mechanisms. The series aims several goals: * Offer a standard tarantool specific way to register a built-in module. It allows to change the registration process for all built-in modules at once. * Make the API of such functions simple and clean. * Eliminate known problems of the existing approach to register built-in modules. This particular commit offers two new functions: * luaT_setfuncs() * luaT_newmodule() The former replaces `luaL_register(L, NULL, funcs)`. The latter replaces `luaL_register()` and `luaL_register_module()` to some extent. Let's look on examples below. ## How it works now First of all, consider the [documentation][1] about the `luaL_register()` function. Then look on the examples. ```c luaL_register(L, "foo.bar", funcs); ``` Creates a table `package.loaded['foo.bar']` and `_G.foo.bar` (if neither of them exists), fill it with the functions and pushes it to the stack. What is not ok: * Pollutes `_G`. * `package.loaded` is written flat, but `_G` is written deeply. * No control over (un)intended module rewritting. It also worth to note that we can't change this function, because it is part of the Lua API. So we need another function for our registration process. Another usage of the function makes it even more confusing: ```c luaL_register(L, NULL, funcs); ``` Does it creates a table, fills it with the functions and pushes it to the stack? No. Unlike other usages, it fills a table on the top of the stack with the functions and leaves it on the stack. And it actually has no relation to module registration. There is tarantool specific function `luaL_register_module()`, which looks as an attempt to solve some of the problems. ```c luaL_register_module(L, "foo.bar", funcs); ``` It doesn't touch `_G` and, surprisingly, changes how a module is written to `package.loaded`. The call creates a table (or finds it) in `package.loaded.foo.bar`, fills it with the functions and pushes to the stack. *This* function writes `package.loaded` deeply. It leaves us with `luaL_register()` for assigning `_G.foo.bar` (with pollution of `package.loaded`). It leaves us with `lua_register()` for setting functions into a table on the stack (with confusing name in this context and confusing contract). And we has no a function to just assign `package.loaded[modname]` without the deep tables creation and with appropriate checks. ## How it will work ```c luaT_newmodule(L, "foo.bar", funcs); ``` Create a table, set it to `package.loaded['foo.bar']` and push into the stack. Assert that the table doesn't exist. ```c luaT_setfuncs(L, funcs); ``` Add functions into a table on top of the stack. ```c luaL_findtable(L, LUA_GLOBALSINDEX, "foo.bar", 0); luaT_setfuncs(L, funcs); ``` Find/create a table `_G.foo.bar` and add functions to it. [1]: https://www.lua.org/manual/5.1/manual.html#luaL_register Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

See the previous commit regarding the idea behind this refactoring. The src/box/lua part is updated in a next commit, only `src/lua` is updated here. It is split to ease reading. The patch is mostly straightforward, I'll highlight several interesting places. * `compat` `_G.internal.compat` was created by `luaL_register()`. Removing of this call drops `_G.internal`: it is not created anymore. * `fiber.channel` and `fiber.cond` The initialization code add fields into fiber module. So we call `require('fiber')` here. The alternative is to access `package.loaded.fiber` directly, however I plan to eliminate such accesses in further patches. (Built-in modules registration process will use its own table instead of `package.loaded`.) * `http.client` Here we also eliminate a direct access to `package.loaded`, which is important for implementing tarantool#7774. * `utf8` This module is also set to `_G.utf8` for compatibility with Lua 5.3. It remains unchanged. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

The idea of the refactoring is described in a previous commit. The change is straightforward: * Use `luaL_findtable()` to find/create a nested table in `_G`. * Use `luaT_setfuncs()` to assign functions to a table. * Use `luaT_newmodule()` to register a module for `require()`. `require('box.internal.<...>')` calls are replaced with `box.internal.<...>`, because `box` submodules are not registered as modules anymore (IOW, are not placed into `package.loaded`). However `_G.box == require('box')`, it remains unchanged. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

There are modules that are implemented as two parts: a Lua/C module for internal use and a public module written on Lua. There is a practice to name both parts the same: just capture the internal part within the public part and rewrite `package.loaded` then. This name overlap is confusing at reading the sources and complicates debugging. And it conflicts with a built-in module loading logic that will be implemented for tarantool#7774. Let's use `foo.lib` for the internal part and `foo` for the public one. This approach is already used in some built-in modules. src/box/lua/upgrade.lua requires src/box/lua/xlog.lua, so changed the loading order. Eliminated extra `internal` field in `uri.lib`, because the whole module is internal. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

The module can't be pulled as `require('compat')`, because it is tarantool specific tuning mechanism, not a general compatibility module. The API is exposed as `require('tarantool').compat`. Before this patch the module was loaded as `package.loaded.compat`, captured and then removed from `package.loaded`. However it would be easier to follow if we'll just assign an internal name for the module. The module is renamed to `internal.compat`. The Lua/C part of the module is renamed to `internal.compat.lib`. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

It is planned to allow to overload a built-in module by an external one. The implementation assumes storing of the built-in modules in a table that is different from `package.loaded`. It means that accessing a module as `package.loaded[<module_name>]` may fail. However we still have a guarantee that a registered module is accessible using `require()`. Use it for accessing built-in modules from our code. Notes on different modules: * `console` Reorder the print/log callbacks to place the print callback first. If something is going very wrong (say, someone removes the internal.print module), things will be consistent: both callback will not be set. * `session` `box` is accessible as `require('box')` and as `_G.box`. Use the latter. src/lua/serializer.c and src/lua/digest.c also access `package.loaded` directly. I left them out of scope here, it requires a bit more in-depth look. Part of tarantool#7774 NO_DOC=user visible behavior is unchanged, pure refactoring change NO_TEST=see NO_DOC NO_CHANGELOG=see NO_DOC

After embedding checks to the core binary, plain require('checks') will return embedded module. So it won't be possible to run repository module tests (in reusable_testing or with repository CI) with repository code until it overloads build-in code. Until [1] is resolved there are no any other way to deal with it. (And even after [1] would be resolved, it may worth to leave it like this to support older versions with built-in checks.) 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7726

The ability to override built-in modules was introduced with [1]. After [2], metrics will be embedded to core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new metrics. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7725 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. After [1], checks is embedded to the core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new checks. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7726 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. After [1], checks is embedded to the core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new checks. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7726

The ability to override built-in modules was introduced with [1]. After [2], metrics will be embedded to core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new metrics. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7725 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. To use it, you must install the package files using the override.package path for each file included. Using soft links inside the rock seems like a bad idea -- luarocks try to resolve them on files deploy and fails to do so. The only stable approach is seems to be to duplicate the files to the override folder. builtin rockspec doesn't allow to specify a single file twice, so we need to store the soft link in the repo to workaround it. To build override package with builtin rockspec, we'll also need to manually maintain two lists of package files. cmake approach allows to automatize it. On the other hand, to work with cmake-built rock, you'll need to `make` the rock while working locally. Since `make .rocks` is a dependency of `make test`, it shouldn't be a big issue. 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. After [2], metrics will be embedded to core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new metrics. cartridge role, which was embedded to tarantool/cartridge in [3], is also would be override. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7725 3. tarantool/cartridge#2047 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. To use it, you must install the package files using the override.package path for each file included. Using soft links inside the rock seems like a bad idea -- luarocks try to resolve them on files deploy and fails to do so. The only stable approach is seems to be to duplicate the files to the override folder. builtin rockspec doesn't allow to specify a single file twice, so we need to store the soft link in the repo to workaround it. To build override package with builtin rockspec, we'll also need to manually maintain two lists of package files. On the other hand, soft link in repo allows to work with files locally ever without package rebuild. After [2], metrics will be embedded to core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new metrics. cartridge role, which was embedded to tarantool/cartridge in [3], is also would be override. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7725 3. tarantool/cartridge#2047 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. To use it, you must install the package files using the override.package path for each file included. Using soft links inside the rock seems like a bad idea -- luarocks try to resolve them on files deploy and fails to do so. The only stable approach is seems to be to duplicate the files to the override folder. builtin rockspec doesn't allow to specify a single file twice, so we need to store the soft link in the repo to workaround it. To build override package with builtin rockspec, we'll also need to manually maintain two lists of package files. cmake approach allows to automatize it. On the other hand, to work with cmake-built rock, you'll need to `make` the rock while working locally. Since `make .rocks` is a dependency of `make test`, it shouldn't be a big issue. 1. tarantool/tarantool#7774 Part of tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. After [2], metrics will be embedded to core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new metrics. cartridge role, which was embedded to tarantool/cartridge in [3], is also would be override. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7725 3. tarantool/cartridge#2047 Closes tarantool/tarantool#7727

The ability to override built-in modules was introduced with [1]. After [1], checks is embedded to the core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new checks. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7726

The ability to override built-in modules was introduced with [1]. After [1], checks is embedded to the core Tarantool. The ability to override them with installed rock will make it possible to use old Tarantool with new checks. Core Tarantool tests for master should be fine after this patch, see [3]. 1. tarantool/tarantool#7774 2. tarantool/tarantool#7726 3. tarantool/tarantool#8475

We already reset readline configuration in interactive_tarantool.lua. In particular this fixes gh_8613_new_cli_behaviour_test run with my custom .inputrc. Follows up ground works done for tarantool#7774 NO_DOC=test harness NO_TEST=test harness NO_CHANGELOG=test harness

This fixes gh_8613_new_cli_behaviour_test run with my custom .inputrc (I use vi-cmd-mode-string/vi-ins-mode-string). We already reset readline configuration in interactive_tarantool.lua. Follows up ground works done for tarantool#7774 NO_DOC=test harness NO_TEST=test harness NO_CHANGELOG=test harness

This fixes gh_8613_new_cli_behaviour_test run with my custom .inputrc (I use vi-cmd-mode-string/vi-ins-mode-string). We already reset readline configuration in interactive_tarantool.lua. Follows up ground works done for #7774 NO_DOC=test harness NO_TEST=test harness NO_CHANGELOG=test harness

This fixes gh_8613_new_cli_behaviour_test run with my custom .inputrc (I use vi-cmd-mode-string/vi-ins-mode-string). We already reset readline configuration in interactive_tarantool.lua. Follows up ground works done for tarantool#7774 NO_DOC=test harness NO_TEST=test harness NO_CHANGELOG=test harness (cherry picked from commit 028c65e)

This fixes gh_8613_new_cli_behaviour_test run with my custom .inputrc (I use vi-cmd-mode-string/vi-ins-mode-string). We already reset readline configuration in interactive_tarantool.lua. Follows up ground works done for #7774 NO_DOC=test harness NO_TEST=test harness NO_CHANGELOG=test harness (cherry picked from commit 028c65e)

igormunkin added feature A new functionality lua in design Requires design document app 2.11 Target is 2.11 and all newer release/master branches labels Oct 4, 2022

igormunkin assigned Totktonada Oct 5, 2022

Totktonada mentioned this issue Jan 12, 2023

Embed tarantool/checks in core tarantool #7726

Closed

DifferentialOrange mentioned this issue Jan 16, 2023

Embedding preparation tarantool/checks#42

Merged

NeraverinTarantool removed the 2.11 Target is 2.11 and all newer release/master branches label Jan 16, 2023

Totktonada removed the in design Requires design document label Jan 18, 2023

Totktonada mentioned this issue Jan 18, 2023

lua: preliminary refactoring for built-in modules overloading [1/2] #8173

Merged

NeraverinTarantool added the tmp label Jan 18, 2023

DifferentialOrange mentioned this issue Mar 17, 2023

rock: override built-in checks tarantool/checks#54

Merged

DifferentialOrange mentioned this issue Mar 23, 2023

Override builtin alternative tarantool/metrics#444

Closed

nshy mentioned this issue Jul 17, 2023

test: reset readline configuration for justrun too #8891

Merged

Totktonada mentioned this issue Oct 19, 2023

Remove select with custom merger tarantool/crud#365

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Builtin modules overloading #7774

Builtin modules overloading #7774

igormunkin commented Oct 4, 2022 •

edited by Totktonada

Loading

Builtin modules overloading #7774

Builtin modules overloading #7774

Comments

igormunkin commented Oct 4, 2022 • edited by Totktonada Loading

igormunkin commented Oct 4, 2022 •

edited by Totktonada

Loading