Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler error starting in v3.2.2 with non-copyable user type #1070

Closed
no-more-secrets opened this issue Nov 20, 2020 · 16 comments
Closed

Compiler error starting in v3.2.2 with non-copyable user type #1070

no-more-secrets opened this issue Nov 20, 2020 · 16 comments
Assignees
Milestone

Comments

@no-more-secrets
Copy link
Contributor

no-more-secrets commented Nov 20, 2020

Hi,

After upgrading my sol2 version from v3.2.1 to v3.2.2 I started seeing a compile error which I traced it to commit 0e1aafe. It is preventing me from registering a callable whose return type is a const & to a non-copyable type (although the problem may well be more general than that). Below I provided all of the info that you might need, but let me know if you need anything else.

Environment:

  • Language: C++, -std=c++20
  • OS: Linux/Pop!_OS 20.10
  • Compiler: Clang trunk as of a couple weeks ago:
    $ clang++ --version
    clang version 12.0.0 (https://github.com/llvm/llvm-project.git 6c2ad4cf8758335e0896ba0540b21bbf6d239fbd)
    Target: x86_64-unknown-linux-gnu
    Thread model: posix
  • Stdlib: libstdc++ from gcc 10.2.0
  • sol2 compiler defines:
    • SOL_USING_CXX_LUA=1
    • SOL_CXX17_FEATURES=1
    • SOL_ALL_SAFETIES_ON=1
    • SOL_EXCEPTIONS_ALWAYS_UNSAFE=1
    • SOL_PRINT_ERRORS=0

Standalone Reproducer

struct no_copy_type {
  no_copy_type()                      = default;
  no_copy_type( no_copy_type const& ) = delete;
  no_copy_type( no_copy_type&& )      = default;
};

struct my_callable {
  no_copy_type const& operator()() const {
    static no_copy_type o;
    return o;
  }
};

void test() {
  auto st = sol::state{};
  st["AAA"] = my_callable{};  // compile error here!!
}

This is impacting me because some of my types are movable but not copyable, and I used to be able to register callables that return const & of one of those non-copyable types but can't seem to anymore as of v3.2.2. Am I doing something wrong?

Compiler Error

/usr/lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/alloc_traits.h:514:4: error: no matching function for call to 'construct_at'
          std::construct_at(__p, std::forward<_Args>(__args)...);
          ^~~~~~~~~~~~~~~~~
../../extern/sol2/include/sol/stack_push.hpp:120:46: note: in instantiation of function template specialization 'std::allocator_traits<std::allocator<rn::lua::(anonymous namespace)::no_copy_type>>::construct<rn::lua::(anonymous namespace)::no_copy_type, const rn::lua::(anonymous namespace)::no_copy_type &>' requested here
                        std::allocator_traits<std::allocator<T>>::construct(alloc, obj, std::forward<Args>(args)...);
                                                                  ^
../../extern/sol2/include/sol/stack_push.hpp:127:11: note: in instantiation of function template specialization 'sol::stack::unqualified_pusher<sol::detail::as_value_tag<rn::lua::(anonymous namespace)::no_copy_type>, void>::push_fx<sol::stack::stack_detail::undefined_metatable &, const rn::lua::(anonymous namespace)::no_copy_type &>' requested here
                        return push_fx(L, fx, std::forward<Args>(args)...);
                               ^
../../extern/sol2/include/sol/stack_push.hpp:137:12: note: in instantiation of function template specialization 'sol::stack::unqualified_pusher<sol::detail::as_value_tag<rn::lua::(anonymous namespace)::no_copy_type>, void>::push_keyed<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, const rn::lua::(anonymous namespace)::no_copy_type &>' requested here
                                return push_keyed(L, usertype_traits<T>::metatable(), std::forward<Arg>(arg), std::forward<Args>(args)...);
                                       ^
../../extern/sol2/include/sol/stack_core.hpp:919:14: note: in instantiation of function template specialization 'sol::stack::unqualified_pusher<sol::detail::as_value_tag<rn::lua::(anonymous namespace)::no_copy_type>, void>::push<const rn::lua::(anonymous namespace)::no_copy_type &>' requested here
                                return p.push(L, std::forward<Arg>(arg), std::forward<Args>(args)...);
                                         ^
../../extern/sol2/include/sol/stack_push.hpp:324:19: note: in instantiation of function template specialization 'sol::stack::push<sol::detail::as_value_tag<rn::lua::(anonymous namespace)::no_copy_type>, const rn::lua::(anonymous namespace)::no_copy_type &, void>' requested here
                                return stack::push<detail::as_value_tag<T>>(L, std::forward<Args>(args)...);
                                              ^
../../extern/sol2/include/sol/stack_core.hpp:919:14: note: (skipping 29 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
                                return p.push(L, std::forward<Arg>(arg), std::forward<Args>(args)...);
                                         ^
../../extern/sol2/include/sol/table_core.hpp:630:4: note: in instantiation of function template specialization 'sol::basic_table_core<true, sol::basic_reference<false>>::set<std::tuple<char const (&)[4]>, sol::function_arguments<sol::function_sig<>, rn::lua::(anonymous namespace)::my_callable>>' requested here
                        set(std::forward<Key>(key), as_function_reference(std::forward<Fx>(fx), std::forward<Args>(args)...));
                        ^
../../extern/sol2/include/sol/table_core.hpp:603:4: note: in instantiation of function template specialization 'sol::basic_table_core<true, sol::basic_reference<false>>::set_fx<rn::lua::(anonymous namespace)::my_callable, std::tuple<char const (&)[4]>, sol::meta::enable_t::_>' requested here
                        set_fx(types<>(), std::forward<Key>(key), std::forward<Args>(args)...);
                        ^
../../extern/sol2/include/sol/table_proxy.hpp:100:8: note: in instantiation of function template specialization 'sol::basic_table_core<true, sol::basic_reference<false>>::set_function<std::tuple<char const (&)[4]>, rn::lua::(anonymous namespace)::my_callable>' requested here
                        tbl.set_function(std::move(key), std::forward<Args>(args)...);
                            ^
../../extern/sol2/include/sol/table_proxy.hpp:119:29: note: in instantiation of function template specialization 'sol::table_proxy<sol::basic_table_core<true, sol::basic_reference<false>> &, std::tuple<char const (&)[4]>>::set_function<rn::lua::(anonymous namespace)::my_callable>' requested here
                                return std::move(*this).set_function(std::forward<T>(other));
                                                        ^
../../src/lua.cpp:89:16: note: in instantiation of function template specialization 'sol::table_proxy<sol::basic_table_core<true, sol::basic_reference<false>> &, std::tuple<char const (&)[4]>>::operator=<rn::lua::(anonymous namespace)::my_callable>' requested here
  st["AAA"] = my_callable{};
            ^
/usr/lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_construct.h:94:5: note: candidate template ignored: substitution failure [with _Tp = rn::lua::(anonymous namespace)::no_copy_type, _Args = <const rn::lua::(anonymous namespace)::no_copy_type &>]: call to deleted constructor of 'rn::lua::(anonymous namespace)::no_copy_type'
    construct_at(_Tp* __location, _Args&&... __args)
    ^
1 error generated.
@no-more-secrets
Copy link
Contributor Author

I would have given a godbolt link, but v3.2.2 does not seem to appear there yet.

@ThePhD
Copy link
Owner

ThePhD commented Dec 17, 2020

This was causing problems in another issue, where const& return types were not having the const properly stripped out before being considered a copy. Constant references are meant to used as a proxy for copying things in Lua, but we behave differently for non-const...

This is likely drilling down to the fact that we don't have a "const reference" table: everything in Lua is either a plain, non-const reference or a value. We should probably fix this.

@no-more-secrets
Copy link
Contributor Author

Thanks for the response, and yes fixing would be great (how hard is it?), since it is currently blocking me from updating my sol version.

@no-more-secrets
Copy link
Contributor Author

no-more-secrets commented Dec 17, 2020

By the way -- the fact that this broke (or at least changed behavior) would seem to imply that there are no unit tests covering this case... perhaps sol2 needs some better unit test coverage.

@ThePhD
Copy link
Owner

ThePhD commented Dec 17, 2020

Pull requests welcome.

@no-more-secrets
Copy link
Contributor Author

I am happy to look into making a unit test for this -- but I think fixing it would be a bit tough since I don't have any familiarity with the code base. Unless you think it is simple and could give me a sketch of how to do it...

@ThePhD
Copy link
Owner

ThePhD commented Dec 18, 2020

This can't be fixed without breaking a fairly fundamental tenent that:

Arguments:

  • T&, std::reference_wrapper, T* is not copied
  • const T& is copied
  • T&& is moved

Returns:

  • T&, std::reference_wrapper, T* is not copied
  • const T& is copied
  • T&& is moved

The old behavior where const T& wasn't copied but stored in a (non-const) reference was reminiscent of a bug. The workaround is to return a pointer or a std::reference_wrapper<const T>. That means there's room here to either tweak the way return values and arguments are pushed into Lua on a more fundamental level, and let the user override the behavior appropriately. Thinking it over, it's probably best to allow:

  • sol_lua_push_argument, identical to the sol_lua_push extension point
  • sol_lua_push_return, identical to the sol_lua_push extension point

Folks could override these 2 to change the behavior. A greater change of "add a const reference table" is overkill and adds to the "too much runtime data" problem, albeit that's solved with an orthogonal feature to "pin the set of allowed operations at compile time to completely avoid issues of needing to grow or add things at runtime". So, we might just add a const & table to the already 3 internal tables we have, bringing it to 3 (value, reference, unique usertype, and the new const reference table).

The extension points are probably a better fundamental fix for this issue, that goes beyond the already-in-develop solutions of marking things as value-semantic.

@ThePhD ThePhD self-assigned this Dec 18, 2020
@ThePhD ThePhD added this to the Feature milestone Dec 18, 2020
@no-more-secrets
Copy link
Contributor Author

Here is probably a dumb question: why does sol associate const ref return types with copying? Shouldn't it only copy (or move) if a function returns the type by value? What was the original reasoning for wanted to copy const ref return types?

In any case... I guess for now I will just use non-const refs for return types and deal with it.
Thanks

@ThePhD
Copy link
Owner

ThePhD commented Dec 18, 2020

¯\_(ツ)_/¯

#include <string>

const std::string& jekw (const std::string& xD) {
     return xD;
}

int main () {
     const std::string* ptr = &jekw(":)");
     return ptr->size();
}

@ThePhD
Copy link
Owner

ThePhD commented Dec 18, 2020

(In other words, const blah& binds to temporaries which artificially extend lifetime in ways I cannot predict. blah& does not extend the lifetime of temporaries, unless you're on MSVC with their really bad extension turned on, but they've started shutting that one off by default because it's error prone and buggy in newer project builds under /Zc:conforming or whatever that compiler flag was supposed to be.)

@no-more-secrets
Copy link
Contributor Author

Right, but do you think we could say something like, generally in C++, a function should not return a reference unless it is sure that it is not referring to a temporary. In other words, the above example would seem to be more the fault of the person who wrote jekw and not sol. I think that the API of sol should not be designed to try to prevent people from making dangling-ref bugs in their code, otherwise it will limit support for the proper use cases of returning const&, such as my own use case which is to return const& to entities stored in global state which are not temporaries.

Is this something that we can/should change with sol? I know that it might break backward compatibility, but it seems like the right thing to do if I understand correctly what's going on here.

@ThePhD
Copy link
Owner

ThePhD commented Dec 18, 2020

This is, unfortunately, not how people write code in the wild and not how legacy code behaves. People have been returning const whatever& values and taking const whatever& values to "save space" and "only materialize temporaries when necessary". sol2 can't assume the benefit of the doubt that someone will do it properly because it ultimately puts it into a system which does not -- and cannot -- play by C++'s rules for statically evaluated programs. It goes into a runtime system, there is no lifetime extension detection I can do to figure out when the compiler's made the call to extend lifetime, so I have 2 choices:

  • let everyone's code crash when they (inevitably) make this mistake (a higher chance when dealing with C++03 and before-times code)
  • let everyone's code compile and run without issue, at the cost of others with pure, correct code requiring markup

Compilers warn on more direct examples but protracted ones still break, without warning. These bugs would be hard to find and hard to track down because the nature of their brokenness means they go into the Lua system and would demand quite the suite of runtime tools to catch. (Or just a lucky break with AddressSanitizer and UBSanitizer.)

@no-more-secrets
Copy link
Contributor Author

I hear you. I guess a counter could be that, if sol is "silently" copying types that are returned by const&, that could introduce bugs as well, where a user thinks they're operating on the "original" object in Lua when they are in fact just using a copy.

It sounds like for the time being I will just have to return non-const refs in my code. But thanks for your prompt responses, I appreciate it.

P.S. when are we getting your Unicode stuff in C++?

@ThePhD
Copy link
Owner

ThePhD commented Dec 27, 2020

Still working on this.

P.S.: Unicode stuff is coming. We had some wins in the C Committee - https://soasis.org/posts/planted-seeds-unicode-c-c++-2021/

@ThePhD
Copy link
Owner

ThePhD commented Jan 22, 2021

So I've spent some time on this trying to think about making a const table that propagate const-ness so you can have it work in Lua and get errors when you call functions on logically and morally const data. And, unfortunately,

it just doesn't work.

It's easy to detect for const member functions or similar: we just strip them from the const metatables in Lua and have a const-only version of the metatables (sol2 already has 3 versions of the metatable: self-destructing values, non-owning references, and unique usertype-style).

The problem gets more interesting (read: more annoying) when you start doing things like this:

sol::state lua;
lua["f"] = [] (int count, std::string& value) {
     // ...
};

static const std::string my_string = "hi";
lua["f"](1, std::cref(my_string)); // Uh???

What am I supposed to do here? Using std::cref produces a std::reference_wrapper<const std::string>. Am I supposed to blow up the code for lua["f"] and error? That implies that not only am I type checking, but I'm not qualifier checking. This gets more interesting when you throw in other qualifiers:

  • & and && reference qualifiers for member functions
  • const
  • volatile

The approach of "checking if the qualifiers match and performing C++-style resolution for each argument" is bad enough; doing it to also make things like sol::overload(...) choose the best overload is just asking for not only a HUGE performance drag but also a sincere problem in the whole system. The reality of the matter is that C++ only enforces const through its optimizer and through its compile-time, because that's the only sane time to enforce such constraints (alongside volatile and other qualifiers). Doing it at runtime is a bonkers task, and by saying "well if you bind const stuff it should track that and treat it like it, including const references" in the system, you're opening that same pandora's box.

So, unfortunately, I can't really implement this one. I think we'll just have to stay with the copies, and in the new docs make sure it's noted pretty clearly what happens with arguments vs. returns vs. function calls.

@ThePhD ThePhD closed this as completed Jan 22, 2021
@no-more-secrets
Copy link
Contributor Author

@ThePhD Thanks for taking a detailed look at this. Given that it looks like things need to stay the way they are, I am wondering if you could just confirm my understanding so that I can better use the sol2 API:

  • When a C++ function returns a const & to Lua, the value will be copied.
  • When a C++ function accepts a const & parameter from Lua, it will be copied and the const & as seen from the C++ function will refer to a temporary.

Are those correct? (second one seems unlikely, but just making sure)

And one other thing as a last attempt to make something work: I noticed above in your analysis that you are consider function parameters. For my use case I happen to only be concerned with const & return types from C++ functions into Lua; the semantics of function parameters are working fine for me as they are. So therefore, would it make it any easier to address this issue only for const & return types, leaving the behavior for function parameters as-is? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants