Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running memcached on intel with SMP=24 distributes incoming traffic only to CPUs 0..15 #26

Open
slivne opened this issue Feb 12, 2015 · 1 comment
Open

Running memcached on intel with SMP=24 distributes incoming traffic only to CPUs 0..15 #26

slivne opened this issue Feb 12, 2015 · 1 comment

Comments

@slivne
Copy link
Contributor

slivne commented Feb 12, 2015

I ran memcached from seastar-dev/linearize on Intel servers

on intel1

sudo build/release/apps/memcached/memcached --network-stack native --dhcp 0 --host-ipv4-addr 192.168.20.101 --netmask-ipv4-addr 255.255.255.0 --gw-ipv4-addr 192.168.20.185 --smp 24 --dpdk-pmd --collectd 1 --collectd-address 192.168.20.185:25826 --collectd-hostname=intel1-1

on intel2

for i in {1..48} ; do memaslap -s 192.168.20.101:11211 -t 12000s -X 64 -T 1 -c 64 & done

When checking incoming traffic via http://system.cloudius:18080/ machine intel1-1:

  • network-18->total-operations-rx-packets it can be seen that no traffic is received on this cpu its 0
  • netwrok-12-total-operations-rx-packets is ~200K

Please note that not all collectd information is avilable for all CPUs it seems that some packets are dropped along the way

When booting the memcached process the following is printed:

.
.
PMD: eth_i40e_dev_init(): FW 4.1 API 1.1 NVM 04.01.00 eetrack 80001121
PMD: eth_i40e_dev_init(): Failed to stop lldp
PMD: i40e_pf_parameter_init(): Max supported VSIs:130
PMD: i40e_pf_parameter_init(): PF queue pairs:64
PMD: i40e_pf_parameter_init(): Max VMDQ VSI num:63
PMD: i40e_pf_parameter_init(): VMDQ queue pairs:4
ports number: 1
Port 0: max_rx_queues 316 max_tx_queues 316
Port 0: using 24 queues
Port 0: RSS table size is 512
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
Port 0 init ... done: 
Creating mbuf pool 'dpdk_net_pktmbuf_pool0' [3072 mbufs] ...
.
.
@gleb-cloudius
Copy link
Contributor

i40e dpdk rss driver rounds down queue number to power of two when it configures rss redir table. Need to ask intel why and configure redir table by ourselves.

@nlfox nlfox mentioned this issue Jan 13, 2018
Open
@frank8989 frank8989 mentioned this issue Feb 26, 2018
Open
@closertotheheart closertotheheart mentioned this issue Jan 30, 2019
Closed
@Howe2015 Howe2015 mentioned this issue Aug 31, 2020
Open
avikivity pushed a commit that referenced this issue Apr 6, 2021
@bhalevy @avikivity
test: closeable: allocate variables accessed by continuations using d…
4d69526 
…o_with

Fixes failures in debug mode:
```
$ build/debug/tests/unit/closeable_test -l all -t deferred_close_test
WARNING: debug mode. Not for benchmarking or production
random-seed=3064133628
Running 1 test case...
Entering test module "../../tests/unit/closeable_test.cc"
../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test"
../../src/testing/seastar_test.cc(43): info: check true has passed
==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
terminate called after throwing an instance of 'seastar::broken_promise'
  what():  broken promise
==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592)
False positive error reports may follow
For details see google/sanitizers#189
=================================================================
==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
    #0 0x7fbf45f39d0b  (/lib64/libasan.so.6+0xb3d0b)
    #1 0x7fbf45f57d4e  (/lib64/libasan.so.6+0xd1d4e)
    #2 0x7fbf45f3e724  (/lib64/libasan.so.6+0xb8724)
    #3 0x7fbf45eb3e5b  (/lib64/libasan.so.6+0x2de5b)
    #4 0x7fbf45eb51e8  (/lib64/libasan.so.6+0x2f1e8)
    #5 0x7fbf45eb7694  (/lib64/libasan.so.6+0x31694)
    #6 0x7fbf45f39398  (/lib64/libasan.so.6+0xb3398)
    #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b)
    #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116
    #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121
    #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810)
    #11 0x7fbf44bcd897  (/lib64/libgcc_s.so.1+0xf897)
    #12 0x7fbf44bcea5f  (/lib64/libgcc_s.so.1+0x10a5f)
    #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8)
    #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148
    #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb)
    #16 0x7fbf45e10a21  (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21)
    #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0)
    #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094)
    #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921)
    #20 0x7fbf45e5eae1  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1)
    #21 0x7fbf45e5ed31  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31)
    #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547)
    #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618)
    #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77
    #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65
    #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd)
```

Signed-off-by: Benny Halevy <[email protected]>
Message-Id: <[email protected]>
dotnwat pushed a commit to dotnwat/seastar that referenced this issue Jul 15, 2022
@BenPope
Merge pull request scylladb#26 from VladLazar/aggregation-support
da64789 
Add metrics aggregation capabilities
@nlescoua nlescoua mentioned this issue Mar 28, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@slivne @gleb-cloudius