Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Pluggable thread pool #22631

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
3bd9fc0
PTP: Use uv_executor_queue_work and add options for zlib, crypto
davisjam Aug 31, 2018
bc0f42e
PTP: First pass at node::threadpool
davisjam Sep 3, 2018
7200111
pTP: rewire NodePlatform to use Threadpool
davisjam Sep 6, 2018
662825f
PTP: Refactor out a LibuvExecutor
davisjam Sep 6, 2018
a894655
PTP: linting
davisjam Sep 6, 2018
8771ad8
PTP: Add state tracking for Tasks
davisjam Sep 6, 2018
3a2bbb6
PTP: Bugfix: start and stop DelayedTaskScheduler
davisjam Sep 6, 2018
8f6df02
PTP: BlockingDrain for Threadpool and TaskQueue
davisjam Sep 6, 2018
5892ef7
PTP: CreatePlatform creates its own TP
davisjam Sep 7, 2018
95bdaad
PTP: linting, comments, etc.
davisjam Sep 7, 2018
76ca768
PTP: More unit tests
davisjam Sep 7, 2018
83e91b4
PTP: Threadpool RAII; UV_THREADPOOL_SIZE
davisjam Sep 7, 2018
a654044
PTP: Cancel Tasks Post'ed to the Threadpool
davisjam Sep 7, 2018
706f2c6
PTP: Add WorkerGroup class to manage Workers
davisjam Sep 9, 2018
d79920f
PTP: Expose NodeThreadpool as public TP face
davisjam Sep 9, 2018
223cfc4
PTP: Subclass NodeThreadpool for CPU/IO split
davisjam Sep 9, 2018
f462078
PTP: Generalize subclass type
davisjam Sep 9, 2018
153dcdb
PTP: ByTaskOriginPartitonedNodeThreadpool
davisjam Sep 9, 2018
8cf91cd
PTP: ByTaskOriginAndTypePartitonedNodeThreadpool
davisjam Sep 10, 2018
21a931a
PTP: Track Task queue and run times
davisjam Sep 10, 2018
076ccca
PTP: Refactor to DRY in PartitionedNodeThreadpool
davisjam Sep 11, 2018
ec5e624
PTP: Profiling: Sample TaskQueue lengths
davisjam Sep 11, 2018
2547f52
PTP: Add UnpartitionedPartitionedNodeThreadpool
davisjam Sep 11, 2018
61b4cd3
PTP: Visualize threadpool behavior
davisjam Sep 11, 2018
0c40738
PTP: change env vars for consistency
davisjam Sep 16, 2018
003c147
PTP: dump stats in either [Signal]Exit or Start
davisjam Sep 16, 2018
c289dcd
PTP: PrintStats tweaks for easy parsing
davisjam Sep 17, 2018
c98c8d6
PTP: address changes in the libuv PR
davisjam Sep 30, 2018
64ce6e1
REMOVE ME: PTP: include libuv changes so people can try prototype
davisjam Sep 30, 2018
883d7fc
PTP: make linter happy
davisjam Sep 30, 2018
83e4214
REMOVE ME: PTP: include libuv changes so people can try prototype
davisjam Sep 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions deps/uv/AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -346,3 +346,8 @@ Paolo Greppi <[email protected]>
Shelley Vohr <[email protected]>
Ujjwal Sharma <[email protected]>
Michał Kozakiewicz <[email protected]>
Emil Bay <[email protected]>
Jeremiah Senkpiel <[email protected]>
Andy Zhang <[email protected]>
dmabupt <[email protected]>
Ryan Liptak <[email protected]>
5 changes: 3 additions & 2 deletions deps/uv/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ elseif(CMAKE_C_COMPILER_ID MATCHES "AppleClang|Clang|GNU")
endif()

set(uv_sources
src/executor.c
src/fs-poll.c
src/inet.c
src/threadpool.c
Expand Down Expand Up @@ -139,8 +140,8 @@ set(uv_test_sources
test/test-tcp-writealot.c
test/test-thread-equal.c
test/test-thread.c
test/test-threadpool-cancel.c
test/test-threadpool.c
test/test-executor-cancel.c
test/test-executor.c
test/test-timer-again.c
test/test-timer-from-check.c
test/test-timer.c
Expand Down
2 changes: 1 addition & 1 deletion deps/uv/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ Bug fixes and features should come with tests. Add your tests in the
If you add a new test file, it needs to be registered in three places:
- `CMakeLists.txt`: add the file's name to the `uv_test_sources` list.
- `Makefile.am`: add the file's name to the `test_run_tests_SOURCES` list.
- `uv.gyp`: add the file's name to the `sources` list in the `run-tests` target.
- `test/test.gyp`: add the file's name to the `sources` list in the `run-tests` target.

Look at other tests to see how they should be structured (license boilerplate,
the way entry points are declared, etc.).
Expand Down
46 changes: 46 additions & 0 deletions deps/uv/ChangeLog
Original file line number Diff line number Diff line change
@@ -1,3 +1,49 @@
2018.09.22, Version 1.23.1 (Stable), d2282b3d67821dc53c907c2155fa8c5c6ce25180

Changes since version 1.23.0:

* unix,win: limit concurrent DNS calls to nthreads/2 (Anna Henningsen)

* doc: add addaleax to maintainers (Anna Henningsen)

* doc: add missing slash in stream.rst (Emil Bay)

* unix,fs: use utimes & friends for uv_fs_utime (Jeremiah Senkpiel)

* unix,fs: remove linux fallback from utimesat() (Jeremiah Senkpiel)

* unix,fs: remove uv__utimesat() syscall fallback (Jeremiah Senkpiel)

* doc: fix argument name in tcp.rts (Emil Bay)

* doc: notes on running tests, benchmarks, tools (Jamie Davis)

* linux: remove epoll syscall wrappers (Ben Noordhuis)

* linux: drop code path for epoll_pwait-less kernels (Ben Noordhuis)

* Partially revert "win,code: remove GetQueuedCompletionStatus-based poller"
(Jameson Nash)

* build: add compile for android arm64/x86/x86-64 (Andy Zhang)

* doc: clarify that some remarks apply to windows (Bert Belder)

* test: fix compiler warnings (Jamie Davis)

* ibmi: return 0 from uv_resident_set_memory() (dmabupt)

* win: fix uv_udp_recv_start() error translation (Ryan Liptak)

* win,doc: improve uv_os_setpriority() documentation (Bartosz Sosnowski)

* test: increase upper bound in condvar_5 (Jamie Davis)

* win,tty: remove deadcode (Jameson Nash)

* stream: autodetect direction (Jameson Nash)


2018.08.18, Version 1.23.0 (Stable), 7ebb26225f2eaae6db22f4ef34ce76fa16ff89ec

Changes since version 1.22.0:
Expand Down
1 change: 1 addition & 0 deletions deps/uv/MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

libuv is currently managed by the following individuals:

* **Anna Henningsen** ([@addaleax](https://github.com/addaleax))
* **Bartosz Sosnowski** ([@bzoz](https://github.com/bzoz))
* **Ben Noordhuis** ([@bnoordhuis](https://github.com/bnoordhuis))
- GPG key: D77B 1E34 243F BAF0 5F8E 9CC3 4F55 C8C8 46AB 89B9 (pubkey-bnoordhuis)
Expand Down
5 changes: 3 additions & 2 deletions deps/uv/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ libuv_la_SOURCES = src/fs-poll.c \
src/heap-inl.h \
src/inet.c \
src/queue.h \
src/executor.c \
src/threadpool.c \
src/timer.c \
src/uv-data-getter-setters.c \
Expand Down Expand Up @@ -265,8 +266,8 @@ test_run_tests_SOURCES = test/blackhole-server.c \
test/test-tcp-write-queue-order.c \
test/test-thread-equal.c \
test/test-thread.c \
test/test-threadpool-cancel.c \
test/test-threadpool.c \
test/test-executor-cancel.c \
test/test-executor.c \
test/test-timer-again.c \
test/test-timer-from-check.c \
test/test-timer.c \
Expand Down
79 changes: 77 additions & 2 deletions deps/uv/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,8 +282,31 @@ Make sure that you specify the architecture you wish to build for in the

Run:

For arm

```bash
$ source ./android-configure-arm NDK_PATH gyp [API_LEVEL]
$ make -C out
```

or for arm64

```bash
$ source ./android-configure-arm64 NDK_PATH gyp [API_LEVEL]
$ make -C out
```

or for x86

```bash
$ source ./android-configure-x86 NDK_PATH gyp [API_LEVEL]
$ make -C out
```

or for x86_64

```bash
$ source ./android-configure NDK_PATH gyp [API_LEVEL]
$ source ./android-configure-x86_64 NDK_PATH gyp [API_LEVEL]
$ make -C out
```

Expand All @@ -310,14 +333,66 @@ $ ninja -C out/Release

### Running tests

Run:
#### Build

Build (includes tests):

```bash
$ ./gyp_uv.py -f make
$ make -C out
```

#### Run all tests

```bash
$ ./out/Debug/run-tests
```

#### Run one test

The list of all tests is in `test/test-list.h`.

This invocation will cause the `run-tests` driver to fork and execute `TEST_NAME` in a child process:

```bash
$ ./out/Debug/run-tests TEST_NAME
```

This invocation will cause the `run-tests` driver to execute the test within the `run-tests` process:

```bash
$ ./out/Debug/run-tests TEST_NAME TEST_NAME
```

#### Debugging tools

When running the test from within the `run-tests` process (`run-tests TEST_NAME TEST_NAME`), tools like gdb and valgrind work normally.
When running the test from a child of the `run-tests` process (`run-tests TEST_NAME`), use these tools in a fork-aware manner.

##### Fork-aware gdb

Use the [follow-fork-mode](https://sourceware.org/gdb/onlinedocs/gdb/Forks.html) setting:

```
$ gdb --args out/Debug/run-tests TEST_NAME

(gdb) set follow-fork-mode child
...
```

##### Fork-aware valgrind

Use the `--trace-children=yes` parameter:

```bash
$ valgrind --trace-children=yes -v --tool=memcheck --leak-check=full --track-origins=yes --leak-resolution=high --show-reachable=yes --log-file=memcheck.log out/Debug/run-tests TEST_NAME
```

### Running benchmarks

See the section on running tests.
The benchmark driver is `out/Debug/run-benchmarks` and the benchmarks are listed in `test/benchmark-list.h`.

## Supported Platforms

Check the [SUPPORTED_PLATFORMS file](SUPPORTED_PLATFORMS.md).
Expand Down
23 changes: 0 additions & 23 deletions deps/uv/android-configure

This file was deleted.

2 changes: 1 addition & 1 deletion deps/uv/configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

AC_PREREQ(2.57)
AC_INIT([libuv], [1.23.0], [https://github.com/libuv/libuv/issues])
AC_INIT([libuv], [1.23.1], [https://github.com/libuv/libuv/issues])
AC_CONFIG_MACRO_DIR([m4])
m4_include([m4/libuv-extra-automake-flags.m4])
m4_include([m4/as_case.m4])
Expand Down
4 changes: 4 additions & 0 deletions deps/uv/docs/src/misc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -544,4 +544,8 @@ API
process priority, the result will equal one of the `UV_PRIORITY`
constants, and not necessarily the exact value of `priority`.

.. note::
On Windows, setting `PRIORITY_HIGHEST` will only work for elevated user,
for others it will be silently reduced to `PRIORITY_HIGH`.

.. versionadded:: 1.23.0
10 changes: 5 additions & 5 deletions deps/uv/docs/src/signal.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ Reception of some signals is emulated on Windows:
program is given approximately 10 seconds to perform cleanup. After that
Windows will unconditionally terminate it.

Watchers for other signals can be successfully created, but these signals
are never received. These signals are: `SIGILL`, `SIGABRT`, `SIGFPE`, `SIGSEGV`,
`SIGTERM` and `SIGKILL.`
* Watchers for other signals can be successfully created, but these signals
are never received. These signals are: `SIGILL`, `SIGABRT`, `SIGFPE`, `SIGSEGV`,
`SIGTERM` and `SIGKILL.`

Calls to raise() or abort() to programmatically raise a signal are
not detected by libuv; these will not trigger a signal watcher.
* Calls to raise() or abort() to programmatically raise a signal are
not detected by libuv; these will not trigger a signal watcher.

.. note::
On Linux SIGRT0 and SIGRT1 (signals 32 and 33) are used by the NPTL pthreads library to
Expand Down
2 changes: 1 addition & 1 deletion deps/uv/docs/src/stream.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Data types
`nread` might be 0, which does *not* indicate an error or EOF. This
is equivalent to ``EAGAIN`` or ``EWOULDBLOCK`` under ``read(2)``.

The callee is responsible for stopping closing the stream when an error happens
The callee is responsible for stopping/closing the stream when an error happens
by calling :c:func:`uv_read_stop` or :c:func:`uv_close`. Trying to read
from the stream again is undefined.

Expand Down
4 changes: 2 additions & 2 deletions deps/uv/docs/src/tcp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,13 @@ API

.. c:function:: int uv_tcp_getsockname(const uv_tcp_t* handle, struct sockaddr* name, int* namelen)

Get the current address to which the handle is bound. `addr` must point to
Get the current address to which the handle is bound. `name` must point to
a valid and big enough chunk of memory, ``struct sockaddr_storage`` is
recommended for IPv4 and IPv6 support.

.. c:function:: int uv_tcp_getpeername(const uv_tcp_t* handle, struct sockaddr* name, int* namelen)

Get the address of the peer connected to the handle. `addr` must point to
Get the address of the peer connected to the handle. `name` must point to
a valid and big enough chunk of memory, ``struct sockaddr_storage`` is
recommended for IPv4 and IPv6 support.

Expand Down
Loading