Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Format string checking incorrectly assumes repeated evaluation of string literal will produce the same pointer #4177

Closed
zygoloid opened this issue Sep 27, 2024 · 11 comments

Comments

@zygoloid
Copy link

zygoloid commented Sep 27, 2024

In the fstring constructor from a type that is convertible to string view, the s argument is passed to two different functions, which both internally convert it to a string_view. This s argument is frequently created by the FMT_STRING macro which has a conversion to string_view that evaluates a string literal expression.

Each evaluation of a string literal expression can produce a different pointer. Therefore it is not correct to assume that the two conversions of s to a string_view produce related string_views, but the code does assume this. Historically, compilers have let you get away with this in compile-time evaluation, but clang recently gained a more strict implementation of this rule and now rejects.

I think the fix is probably to do the same thing in this constructor that is done a few lines below in the next constructor:

   template <typename S,
             FMT_ENABLE_IF(std::is_convertible<const S&, string_view>::value)>
   FMT_CONSTEVAL FMT_ALWAYS_INLINE fstring(const S& s) : str(s) {
     if (FMT_USE_CONSTEVAL)
+      FMT_CONSTEXPR auto sv = string_view(S());
-      detail::parse_format_string<char>(s, checker(s, arg_pack()));
+      detail::parse_format_string<char>(sv, checker(sv, arg_pack()));
 #ifdef FMT_ENFORCE_COMPILE_STRING
     static_assert(
         FMT_USE_CONSTEVAL && sizeof(s) != 0,
         "FMT_ENFORCE_COMPILE_STRING requires format strings to use FMT_STRING");
 #endif
   }
   template <typename S,
             FMT_ENABLE_IF(std::is_base_of<detail::compile_string, S>::value&&
                               std::is_same<typename S::char_type, char>::value)>
   FMT_ALWAYS_INLINE fstring(const S&) : str(S()) {
     FMT_CONSTEXPR auto sv = string_view(S());
     FMT_CONSTEXPR int ignore =
         (parse_format_string(sv, checker(sv, arg_pack())), 0);
     detail::ignore_unused(ignore);
   }
@vitaut
Copy link
Contributor

vitaut commented Sep 28, 2024

Applied the patch (with a small tweak) in cacc310. Thanks!

@vitaut vitaut closed this as completed Sep 28, 2024
@phprus
Copy link
Contributor

phprus commented Sep 28, 2024

@vitaut
Why

FMT_CONSTEXPR auto sv = string_view(S());

?

Instead of

FMT_CONSTEXPR auto sv = string_view(s);

?

S() vs s variable.

@vitaut
Copy link
Contributor

vitaut commented Sep 28, 2024

I don't think it matters much but since the object is already available, I don't see why we need to construct another one.

@vitaut
Copy link
Contributor

vitaut commented Sep 28, 2024

Adding a regression test: 1c5883b.

@vitaut
Copy link
Contributor

vitaut commented Sep 28, 2024

I don't think it matters much but since the object is already available, I don't see why we need to construct another one.

Nevermind, I misinterpreted which is the current version of the code. You are right, we could use s there.

@zeroomega
Copy link
Contributor

@zygoloid

I did some local test by rolling our fmtlib to ToT and building our project with ToT clang but still getting errors:

FAILED: host_x64/obj/third_party/fmtlib/src/src/fmtlib.format.cc.o                                                                                                                                                
../../../clang/clang_fmt/bin/clang++ -MD -MF host_x64/obj/third_party/fmtlib/src/src/fmtlib.format.cc.o.d -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES -I../.. -Ihost_x64/gen -I.
./../third_party/fmtlib/src/include -fcolor-diagnostics -fcrash-diagnostics-dir=clang-crashreports -fcrash-diagnostics=all -gen-reproducer=error -ffp-contract=off --sysroot=../../prebuilt/third_party/sysroot/li
nux --target=x86_64-unknown-linux-gnu -ffile-compilation-dir=. -no-canonical-prefixes -fomit-frame-pointer -fdata-sections -ffunction-sections -O0 -Xclang -debug-info-kind=constructor -g3 -grecord-gcc-switches 
-gdwarf-5 -gz=zstd -Wall -Wextra -Wconversion -Wextra-semi -Wimplicit-fallthrough -Wnewline-eof -Wstrict-prototypes -Wwrite-strings -Wno-sign-conversion -Wno-unused-parameter -Wnonportable-system-include-path -
Wno-missing-field-initializers -Wno-extra-qualification -Wno-cast-function-type-strict -Wno-cast-function-type-mismatch -Wno-unknown-warning-option -Wno-missing-template-arg-list-after-template-kw -Wno-deprecat
ed-pragma -fvisibility=hidden -Werror -Wa,--fatal-warnings -Wno-error=deprecated-declarations --sysroot=../../prebuilt/third_party/sysroot/linux --target=x86_64-unknown-linux-gnu -fPIE -fvisibility-inlines-hidd
en -stdlib=libc++ -stdlib=libc++ -std=c++20 -Wno-deprecated-this-capture -fno-exceptions -fno-rtti -ftemplate-backtrace-limit=0 -stdlib=libc++ -c ../../third_party/fmtlib/src/src/format.cc -o host_x64/obj/third
_party/fmtlib/src/src/fmtlib.format.cc.o                                                                                                                                                                          
In file included from ../../third_party/fmtlib/src/src/format.cc:8:                                                                                                                                               
In file included from ../../third_party/fmtlib/src/include/fmt/format-inl.h:23:                                                                                                                                   
../../third_party/fmtlib/src/include/fmt/format.h:2677:4: error: extra ';' after member function definition [-Werror,-Wextra-semi]                                                                                
 2677 |   };                                                                                                                                                                                                      
      |    ^                                                                                                                                                                                                      
1 error generated.                                                                                                                                                                                                  

and

[49936/153879](31) CXX host_x64/obj/third_party/android/platform/system/tools/aidl/aidl_gen.aidl_to_cpp_common.cpp.o
FAILED: host_x64/obj/third_party/android/platform/system/tools/aidl/aidl_gen.aidl_to_cpp_common.cpp.o 
../../../clang/clang_fmt/bin/clang++ -MD -MF host_x64/obj/third_party/android/platform/system/tools/aidl/aidl_gen.aidl_to_cpp_common.cpp.o.d -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_REMOVE_TRANSITIVE_INCLUDES -I../../third_party/android/platform/system/tools/aidl -I../../src/lib/android/aidl/generated-files -I../../third_party/googletest/src/googletest/include -I../../third_party/googletest/src/googletest -I../.. -Ihost_x64/gen -I../../third_party/android/platform/system/libbase/include -I../../third_party/fmtlib/src/include -I../../third_party/android/platform/system/logging/liblog/include -I../../zircon/system/ulib/zx-panic-libc/include -I../../zircon/system/public -fcolor-diagnostics -fcrash-diagnostics-dir=clang-crashreports -fcrash-diagnostics=all -gen-reproducer=error -ffp-contract=off --sysroot=../../prebuilt/third_party/sysroot/linux --target=x86_64-unknown-linux-gnu -ffile-compilation-dir=. -no-canonical-prefixes -fomit-frame-pointer -fdata-sections -ffunction-sections -O0 -Xclang -debug-info-kind=constructor -g3 -grecord-gcc-switches -gdwarf-5 -gz=zstd -Wall -Wextra -Wconversion -Wextra-semi -Wimplicit-fallthrough -Wnewline-eof -Wstrict-prototypes -Wwrite-strings -Wno-sign-conversion -Wno-unused-parameter -Wnonportable-system-include-path -Wno-missing-field-initializers -Wno-extra-qualification -Wno-cast-function-type-strict -Wno-cast-function-type-mismatch -Wno-unknown-warning-option -Wno-missing-template-arg-list-after-template-kw -Wno-deprecated-pragma -fvisibility=hidden -Werror -Wa,--fatal-warnings -Wno-error=deprecated-declarations --sysroot=../../prebuilt/third_party/sysroot/linux --target=x86_64-unknown-linux-gnu -fPIE -fvisibility-inlines-hidden -stdlib=libc++ -stdlib=libc++ -std=c++20 -Wno-deprecated-this-capture -fno-exceptions -fno-rtti -ftemplate-backtrace-limit=0 -stdlib=libc++ -Wno-c++98-compat-extra-semi -Wno-c99-designator -Wno-deprecated-declarations -Wno-extra-semi -Wno-implicit-int-conversion -Wno-inconsistent-missing-override -Wno-newline-eof -Wno-range-loop-construct -Wno-reorder-init-list -Wno-shorten-64-to-32 -Wno-sign-compare -Wno-string-conversion -Wno-unused-but-set-variable -Wno-unused-result -Wno-unknown-warning-option -Wno-vla-cxx-extension -Wno-c++98-compat-extra-semi -Wno-c99-designator -Wno-deprecated-declarations -Wno-extra-semi -Wno-implicit-int-conversion -Wno-inconsistent-missing-override -Wno-newline-eof -Wno-range-loop-construct -Wno-reorder-init-list -Wno-shorten-64-to-32 -Wno-sign-compare -Wno-string-conversion -Wno-unused-but-set-variable -Wno-unused-result -c ../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp -o host_x64/obj/third_party/android/platform/system/tools/aidl/aidl_gen.aidl_to_cpp_common.cpp.o
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:16:
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.h:23:
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_language.h:26:
In file included from ../../third_party/android/platform/system/libbase/include/android-base/result.h:104:
In file included from ../../third_party/android/platform/system/libbase/include/android-base/format.h:23:
In file included from ../../third_party/fmtlib/src/include/fmt/chrono.h:23:
In file included from ../../third_party/fmtlib/src/include/fmt/format.h:41:
../../third_party/fmtlib/src/include/fmt/base.h:2670:24: error: constexpr variable 'sv' must be initialized by a constant expression
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                        ^    ~~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:374:24: note: in instantiation of function template specialization 'fmt::fstring<fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>>::fstring<const char *, 0>' requested here
  374 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("min_tag", min_tag),
      |                        ^
../../third_party/fmtlib/src/include/fmt/base.h:524:32: note: read of dereferenced null pointer is not allowed in a constant expression
  524 |       size_ = __builtin_strlen(detail::narrow(s));
      |                                ^
../../third_party/fmtlib/src/include/fmt/base.h:2670:29: note: in call to 'basic_string_view(nullptr)'
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                             ^~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:374:24: error: call to consteval function 'fmt::fstring<fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>>::fstring<const char *, 0>' is not a constant expression
  374 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("min_tag", min_tag),
      |                        ^
../../third_party/fmtlib/src/include/fmt/base.h:524:32: note: read of dereferenced null pointer is not allowed in a constant expression
  524 |       size_ = __builtin_strlen(detail::narrow(s));
      |                                ^
../../third_party/fmtlib/src/include/fmt/base.h:2670:29: note: in call to 'basic_string_view(nullptr)'
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                             ^~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:374:24: note: in call to 'fstring<const char *, 0>(tmpl)'
  374 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("min_tag", min_tag),
      |                        ^~~~
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:16:
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.h:23:
In file included from ../../third_party/android/platform/system/tools/aidl/aidl_language.h:26:
In file included from ../../third_party/android/platform/system/libbase/include/android-base/result.h:104:
In file included from ../../third_party/android/platform/system/libbase/include/android-base/format.h:23:
In file included from ../../third_party/fmtlib/src/include/fmt/chrono.h:23:
In file included from ../../third_party/fmtlib/src/include/fmt/format.h:41:
../../third_party/fmtlib/src/include/fmt/base.h:2670:24: error: constexpr variable 'sv' must be initialized by a constant expression
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                        ^    ~~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:735:24: note: in instantiation of function template specialization 'fmt::fstring<fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>>::fstring<const char *, 0>' requested here
  735 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("typelist", typelist));
      |                        ^
../../third_party/fmtlib/src/include/fmt/base.h:524:32: note: read of dereferenced null pointer is not allowed in a constant expression
  524 |       size_ = __builtin_strlen(detail::narrow(s));
      |                                ^
../../third_party/fmtlib/src/include/fmt/base.h:2670:29: note: in call to 'basic_string_view(nullptr)'
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                             ^~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:735:24: error: call to consteval function 'fmt::fstring<fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>>::fstring<const char *, 0>' is not a constant expression
  735 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("typelist", typelist));
      |                        ^
../../third_party/fmtlib/src/include/fmt/base.h:524:32: note: read of dereferenced null pointer is not allowed in a constant expression
  524 |       size_ = __builtin_strlen(detail::narrow(s));
      |                                ^
../../third_party/fmtlib/src/include/fmt/base.h:2670:29: note: in call to 'basic_string_view(nullptr)'
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                             ^~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:735:24: note: in call to 'fstring<const char *, 0>(tmpl)'
  735 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("typelist", typelist));
      |                        ^~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:790:24: error: call to consteval function 'fmt::fstring<fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>, fmt::detail::named_arg<char, std::string>>::fstring<const char *, 0>' is not a constant expression
  790 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("default_name", default_name),
      |                        ^
../../third_party/fmtlib/src/include/fmt/base.h:524:32: note: read of dereferenced null pointer is not allowed in a constant expression
  524 |       size_ = __builtin_strlen(detail::narrow(s));
      |                                ^
../../third_party/fmtlib/src/include/fmt/base.h:2670:29: note: in call to 'basic_string_view(nullptr)'
 2670 |     FMT_CONSTEXPR auto sv = string_view(S());
      |                             ^~~~~~~~~~~~~~~~
../../third_party/android/platform/system/tools/aidl/aidl_to_cpp_common.cpp:790:24: note: in call to 'fstring<const char *, 0>(tmpl)'
  790 |     out << fmt::format(tmpl, fmt::arg("name", name), fmt::arg("default_name", default_name),
      |                        ^~~~
5 errors generated.

@zygoloid
Copy link
Author

zygoloid commented Oct 2, 2024

I guess we should be using sv rather than S() after all.

@zeroomega
Copy link
Contributor

I guess we should be using sv rather than S() after all.

You mean FMT_CONSTEXPR auto sv = string_view(s);? It fails as well.

@zygoloid
Copy link
Author

zygoloid commented Oct 2, 2024

You would also need to remove the FMT_CONSTEXPR. If that doesn't work, you'll need to share the error messages you're seeing :)

@zeroomega
Copy link
Contributor

With diff like this:

diff --git a/include/fmt/base.h b/include/fmt/base.h
index f91f8e8..ebb8fdf 100644
--- a/include/fmt/base.h
+++ b/include/fmt/base.h
@@ -2667,7 +2667,7 @@ template <typename... T> struct fstring {
   template <typename S,
             FMT_ENABLE_IF(std::is_convertible<const S&, string_view>::value)>
   FMT_CONSTEVAL FMT_ALWAYS_INLINE fstring(const S& s) : str(s) {
-    FMT_CONSTEXPR auto sv = string_view(S());
+    auto sv = string_view(s);
     if (FMT_USE_CONSTEVAL)
       detail::parse_format_string<char>(sv, checker(sv, arg_pack()));
 #ifdef FMT_ENFORCE_COMPILE_STRING
diff --git a/include/fmt/format.h b/include/fmt/format.h
index 4ad1eff..52af87b 100644
--- a/include/fmt/format.h
+++ b/include/fmt/format.h
@@ -2674,7 +2674,7 @@ class bigint {
 
   FMT_CONSTEXPR auto get_bigit(int i) const -> bigit {
     return i >= exp_ && i < num_bigits() ? bigits_[i - exp_] : 0;
-  };
+  }
 
   FMT_CONSTEXPR void subtract_bigits(int index, bigit other, bigit& borrow) {
     auto result = double_bigit(bigits_[index]) - other - borrow;

The build passed.

@zeroomega
Copy link
Contributor

Uploaded PR #4187

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants