Core Dump while running sample program

[asimarora]

I was testing this robust framework and got core dump while running sample program mentioned here,

http://docs.seastar-project.org/master/md_doc_tutorial.html

[avikivity]

It would be helpful if you said exactly which program you ran, what the output was, and a stack trace if possible.

[asimarora]

Thanks for the fast reply.

Code Snippet:

#include "core/app-template.hh"
#include "core/reactor.hh"
#include
int main(int argc, char** argv) {
seastar::app_template app;
app.run(argc, argv, [] {
std::cout << "Hello world\n";
return seastar::make_ready_future<>();
});
}

Compile: g++ pkg-config --cflags --libs $SEASTAR/build/release/seastar.pc one.cpp -g -std=c++17

Additional Info:

gcc version 5.4.1 20160904 (Ubuntu 5.4.1-2ubuntu1~16.04)

(gdb) n
45 _opts.add(reactor::get_options_description(cfg.default_task_quota));
(gdb) n
n
Program received signal SIGSEGV, Segmentation fault.
seastar::memory::cpu_pages::free_span (this=0x7ffff7fba900, span_start=0, nr_pages=0) at core/memory.cc:513
513 if (before->free) {
(gdb) n

Program terminated with signal SIGSEGV, Segmentation fault.

Please let me know if u need more information.

[vkrivopalov]

I have built this sample with latest Seastar on Ubuntu 16.04 and it runs just fine.

Can you give the following information:

1. What Seastar shapshot (git commit) are you on?
2. What are the configuration options you're using when building Seastar?
3. How much RAM does your box have?

[asimarora]

1 Master Directly Cloned.
2 Give on https://github.com/scylladb/seastar/blob/master/doc/building-ubuntu.md
3 8 GB

I don't know whether it will be helpful to you or not, But i got this error while running

./configure.py --compiler=g++-5

Notice: -fsanitize=vptr is broken, disabling; some debug mode tests are bypassed.

[vkrivopalov]

Tried this on another Ubuntu 16.04 machine but still works well.
Can you try to build the same app in debug mode and tell is ASAN finds any memory misuse?

[asimarora]

Thanks , Can you please give Any reference doc to build in debug Mode.

[vkrivopalov]

You can configure Seastar using
./configure.py --mode=debug --compiler=g++-5

then build using ninja (same as for release)

And then specify "debug" instead of "release" for your command line:
g++ pkg-config --cflags --libs $SEASTAR/build/debug/seastar.pc one.cpp

[vkrivopalov]

It seems this issue might be the one addressed with this recent patch:
https://groups.google.com/forum/#!topic/seastar-dev/rCLrvfVrEV4

[asimarora]

I did a git pull and tested it again, I am sorry to say still same problem. I don't know much about this code but it seems from core that at line : 513 , It is dumping

page* before = &pages[span_start - 1];

Since span_start is 0 (As shown in core)

[vkrivopalov]

@asimarora,

Can you try to pull the up-to-date code from 'master' branch and give it a try?
So far it looks like the patch mentioned in my previous comment may be of help here.

What commit are you currently on?

[asimarora]

I tried pulling from master and building in debug mode. I got a bad free error

/usr/local/include/boost/filesystem/operations.hpp:294:64: runtime error: load of value 4226454032, which is not a valid value for type 'file_type'

==15874==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x603000000054 in thread T0
#0 0x7fb8b4c270a0 in operator delete(void*) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe10a0)
#1 0x860197 in __gnu_cxx::new_allocator::deallocate(char*, unsigned long) /usr/include/c++/7/ext/new_allocator.h:125
#2 0x86677f in std::allocator_traits<std::allocator >::deallocate(std::allocator&, char*, unsigned long) /usr/include/c++/7/bits/alloc_traits.h:462
#3 0x1628166 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_destroy(unsigned long) /usr/include/c++/7/bits/basic_string.h:210
#4 0x1627cf1 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_dispose() /usr/include/c++/7/bits/basic_string.h:205
#5 0x1627587 in std::__cxx11::basic_string<char, std::char_traits, std::allocator >::~basic_string() /usr/include/c++/7/bits/basic_string.h:620
#6 0x775afe in boost::filesystem::path::~path() (/media/asim/Vol2/asim/codes/cplusplus/seastar/a.out+0x775afe)
#7 0x49cd4f in virtualized core/reactor.cc:4312
#8 0x49ce7c in seastar::reactor::calculate_poll_time() core/reactor.cc:4323
#9 0x48b1f1 in seastar::reactor::get_options_description(std::chrono::duration<double, std::ratio<1l, 1l> >) core/reactor.cc:3547
#10 0xc3f602 in seastar::app_template::app_template(seastar::app_template::config) core/app-template.cc:45
#11 0x1626e39 in main /media/asim/Vol2/asim/codes/cplusplus/seastar/one.cpp:5
#12 0x7fb8b021d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#13 0x416a38 in _start (/media/asim/Vol2/asim/codes/cplusplus/seastar/a.out+0x416a38)

0x603000000054 is located 20 bytes inside of 24-byte region [0x603000000040,0x603000000058)
allocated by thread T0 here:
#0 0x7fb8b4c26228 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0228)
#1 0x7fb8b1d9ca1d in google::protobuf::internal::InitializeDefaultRepeatedFields() (/usr/lib/x86_64-linux-gnu/libprotobuf.so.9+0x55a1d)

SUMMARY: AddressSanitizer: bad-free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe10a0) in operator delete(void*)
==15874==ABORTING

Version : gcc version 7.2.0 (Ubuntu 7.2.0-1ubuntu1~16.04)

[asimarora]

Hello, Any update

[vkrivopalov]

I tried reproducing your issue using Ubuntu 16.04 and GCC 7.2 but to no avail. Both debug and release builds of a simple demo app with Seastar work fine for me.

Can you share the binaries along with your program source somewhere to try reproducing it?

Can you also describe your system (RAM, disk type/space, etc)? Looking at the path /media/asim/Vol2/asim/codes/cplusplus/seastar/one.cpp, do you use an external drive?

One more question, which exact commit are you on in Seastar? Please provide SHA of the commit in master.

[tchaikov]

@asimarora i ran into the same issue as you put in #345 (comment), after debugging it with the debug build of seastar, i realized that the root cause was in libboost: file_status status(const path&p, system::error_code* ec=0)) returned a file_status with an very large m_value, 4294955312, which should have been a enum file_type. but its value does not match with any of this enum. that's why ASAN=undefined complains. and i believe that's why you got a bad free error.

after installing an new libboost, the problem went away. in my case, the seastar crashes with libboost 1.62 shipped by Debian sid, but they work fine with libboost 1.63.

[asimarora]

@tchaikov , I commented some part of code for my testing purpose , But surely will try that .. Thanks for your input.

[asimarora]

Great , It worked with gcc-7 and Boost : 1.66