Concurrency refactor #35
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ve old files into older for cleanup
elkanatovey
added a commit
that referenced
this pull request
Dec 14, 2022
* src: create new directory structure for system * util: create util folder for global includes and include guards. debug asserts now work properly * src: continue work making grpc dependency * src: continue work making grpc dependency * dependdencies: move location of seal download * bin: make executables go into sensible folders * cmake: finish grpc cmake * cmake: revert seal cmake * cmake: stop auto generation of dependency targets * Update cmake.yml * WIP: Query expander (#17) QueryExpander separated from sealpir and can be used as a component to expand specific parts of queries. * cmake: change names of executables in tests for usability purposes * internals: change name of folder for restructure * Update cmake.yml set to ubuntu 22 for compiler puroses * Working-Frievalds (#18) math_utils, test, internals create 2 dim tensor container, implement freivalds algorithm and container, freivalds mostly runs in ptext space by use of split-ptx structure. add proto messages, not integrate with rest of project yet. tests added for freivalds. Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> * fx: wrong verify on ctx-ctx mat-mul * Minor maintainence (#22) * FetchDependencies.cmake: add option for precompiled libraries * README: update readme for new compile options * grpc: update grpc version * .github: create caching version of test * services-wip * adding op order test to evaluator wrapper. * small refactor * adding asserts * added comments * fix: forgot to switch back to bgv * associative test passing * fix: trivial_ciphertext into const func * transformer uses resize to ensure correct size of given matrix * matrix multiplication using matrices * working frievalds test * frievald's unit-test inspect failure with C != A*B * added benchmarks to ptx-ctx multiplication * fx: timing function * benchmark fix * fx: mat-mul with ptx-ctx, frievalds with ctx-ctx. * working on concurrent matrix-mult * adding channel and waitgroup * feat: threaded mat-mul for ptx-ctx * fx: using ntt in mult ptx-ctx matrices * refactor in mat-mult.cpp * parallel ntt transform of matrices * defined RPC messages. refined RPC services * relocated manager, worker services. * minimal working cmake to add services package. should later on split into sub packages * wip: setting up basic worker service * adding test/services package * fx: including services in the cmake of test * testing connection to basic service * wip: worker service * wip: worker receive stream * wip: worker * worker- adding distribicom message creation funcs * wip: worker receives complicated messages. unit tests sends multi-messages * refactoring worker.cpp * refactoring the stream code * fx build * wip: worker-service receive * rfactor * feat: added possibility to send a galoiskey to the worker * clang fix * wip: understanding how to create multi-service server * WIP * fx * WIP * fx * fx Co-authored-by: Jonathan Weiss <[email protected]> * fx: workers-unit test * add scripts used for lab work and fix yaml (#29) * scripts: add scripts used for faster development in aqua environment * github: edit workflow so doesnt try to run dependency tests * workflows fix (#30) * scripts: add scripts used for faster development in aqua environment * github: edit workflow so doesnt try to run dependency tests * Update fast_test.yml * 25 client service (#32) * cmake: make easier to find include files of each target * client_service: start functions * Services (#33) services package: - Worker -manager subscribing - WIP: work-distribution, first connection made (manager to worker)/ - WIP: thread safe DB-handle. - refactored worker, and created utils.hpp. math_util package: - channel read with timeouts. Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Jonathan Weiss <[email protected]> * Manager subscriber (#34) * service package: - threadsafe db. - heavy refactoring and adding utils.hpp. - connection between worker-manager half-complete. no distribution logic as of yet. - server.hpp improvements. - channel can now wwait with a timeout. - fix in matrix.h * read-with-timeout Co-authored-by: Jonathan Weiss <[email protected]> * Update fast_test.yml * Concurrency refactor (#35) * concurrency: -refactor cuncurrency into it's own target and folder, move old files into older for cleanup * src: -refactor cmake files and includes to more consistent naming scheme for including headers. -rename matrix_multiplier to matrix_operations -rename multiplication_utils namespace to math_utils nmespace * upgrading remote-compiler to support callgrind visualisations * WIP: Worker (#37) In this PR we are connecting one side of the main-server to the worker. - manager offers API for waiting to X workers to register. - streaming galois keys from manager to worker. - receiving galois keys on the worker side. - additional constants. * Client service (#36) * Create devcontainer.json * client_service: queries, callback with answer, decryption and round notification implemented * marshal: add marshaling for ptx and ctx vectors , pir responses, and queries into protobuf messages * client_service: test works, add ele sizes to configs Co-authored-by: elkanatovey <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> * server_service: start work on db modifying ptxs for between round work * server_service: start work on db modifying ptxs for between round work * server: add rpcs, make implement server from grpc * server: put in stubs for client connections, start work on bookeeping for work distribution * server_service: continue work * server: ledger now more organised * server: minor refactor * squashed branch: - replaced waitGroup with std:latch/ - mvp reciving worker's local processings. - matrix asserting correct access to its data. * server: commented out unimplemented code * removed unneeded scope * renamed namespace from pir_primitives to services * fix: shared lock on write * renamed clientQueryLedger * manager.cpp: start galois key mapping server.cpp: create mapping logic * manager.cpp: write galois mapping distribute work * server main-loop binary supported * manager: add db sending functionality marshal: add more marshaling caps + test * manager: add query sending functionality * manager: minor change to db responsibilities * manager: halfway done integration * fix: ipv6 support in extract ip * manager.cpp: continue work on integration * manager: mostly done integration * manager: fix bug and factor out old db constructs server: factor out old db constructs proto: minor update to unsigned integer types for consistency * proto: pass ip address in message body from worker * math_utils/evaluator_wrapper: add method for ptx embedding of ctxs * Async stream (#55) * async server sending using streams. * worker identification using credentials (secret given to server upon register). Co-authored-by: Jonathan Weiss <[email protected]> * evaluator_wrapper: add compose to plaintext, add const on add_plain * evaluator_wrapper: compose_to_plaintext writes as ntt form * Server service (#56) * server_service: add fuctionality for work distribution * evaluator_wrapper: add functionality for second stage pir Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Jonathan <[email protected]> * fix: galois_keys generation * Squashed commit of the following: commit df44f799c26ed4085374ff54fdd4a5b9f94eb7fd Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 09:34:03 2022 +0200 async mat-mult ops - mult with vector from left or right. - mult with matrices of different elements (ctxs-ptx, ptx-ctx, ctx-ctx). commit 867da10ccba23e1ea809cb19dd6a29efab0c64c6 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:51:56 2022 +0200 refactor: naming mult the same for ptx mult vs split-ptx mult commit 5c3e0e9766b7e9712da51c69d5b6f8174b44d211 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:17:56 2022 +0200 refactor commit 772c0f8dfae95e77ea4067b99c711a4eda435d6a Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:15:44 2022 +0200 dropped code used only in tests of itself. commit 578eb077b0f1f7e4142ae2b181241f38a98cc6e3 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:07:37 2022 +0200 removed commented out code commit 7741ddfd1a90884488fe0f2540077876b663ad87 Author: Jonathan Weiss <[email protected]> Date: Wed Dec 7 18:18:54 2022 +0200 feat: - ctx-ctx matmul is threaded. - removed unused code commit 24d9ea3e7e45587ae4eb5f57f167ba697af4c81b Author: Jonathan Weiss <[email protected]> Date: Wed Dec 7 18:13:46 2022 +0200 generic thread pool utilised commit 3c40703 Author: Jonathan <[email protected]> Date: Wed Dec 7 15:42:38 2022 +0200 Async stream (#55) * async server sending using streams. * worker identification using credentials (secret given to server upon register). Co-authored-by: Jonathan Weiss <[email protected]> commit 0ed00c7 Merge: 956e12a f448803 Author: elkanatovey <[email protected]> Date: Wed Dec 7 15:17:15 2022 +0200 Merge remote-tracking branch 'origin/server_service' into server_service commit 956e12a Author: elkanatovey <[email protected]> Date: Wed Dec 7 15:16:37 2022 +0200 math_utils/evaluator_wrapper: add method for ptx embedding of ctxs commit f448803 Author: elkana <[email protected]> Date: Tue Dec 6 23:17:11 2022 +0200 proto: pass ip address in message body from worker commit e54a9d2 Author: elkanatovey <[email protected]> Date: Tue Dec 6 18:03:54 2022 +0200 manager: fix bug and factor out old db constructs server: factor out old db constructs proto: minor update to unsigned integer types for consistency commit ecda72a Author: elkanatovey <[email protected]> Date: Tue Dec 6 17:37:36 2022 +0200 manager: mostly done integration commit 1a96c9e Author: elkana <[email protected]> Date: Tue Dec 6 12:45:37 2022 +0200 manager.cpp: continue work on integration commit e48447a Author: Jonathan Weiss <[email protected]> Date: Tue Dec 6 08:35:53 2022 +0200 fix: ipv6 support in extract ip commit 598ebc6 Author: elkanatovey <[email protected]> Date: Mon Dec 5 20:50:01 2022 +0200 manager: halfway done integration commit aee2ef2 Author: elkanatovey <[email protected]> Date: Mon Dec 5 18:12:11 2022 +0200 manager: minor change to db responsibilities commit 8c079f0 Author: elkanatovey <[email protected]> Date: Mon Dec 5 16:01:48 2022 +0200 manager: add query sending functionality commit 25ec9f2 Author: elkanatovey <[email protected]> Date: Mon Dec 5 14:24:30 2022 +0200 manager: add db sending functionality marshal: add more marshaling caps + test commit f901016 Author: Jonathan Weiss <[email protected]> Date: Mon Dec 5 09:48:53 2022 +0200 server main-loop binary supported commit c6ba581 Author: elkana <[email protected]> Date: Sun Dec 4 22:37:03 2022 +0200 manager.cpp: write galois mapping distribute work commit 5122396 Author: elkana <[email protected]> Date: Sun Dec 4 20:40:08 2022 +0200 manager.cpp: start galois key mapping server.cpp: create mapping logic commit 19059de Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:58:53 2022 +0200 renamed clientQueryLedger commit 1928551 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:53:19 2022 +0200 fix: shared lock on write commit e0267a1 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:29:57 2022 +0200 renamed namespace from pir_primitives to services commit 24c569f Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:27:53 2022 +0200 removed unneeded scope commit 39a3665 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:24:27 2022 +0200 server: commented out unimplemented code commit 30e15bc Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:17:56 2022 +0200 squashed branch: - replaced waitGroup with std:latch/ - mvp reciving worker's local processings. - matrix asserting correct access to its data. commit 8750fbf Author: elkanatovey <[email protected]> Date: Thu Dec 1 18:02:25 2022 +0200 server: minor refactor commit 0696f1a Author: elkanatovey <[email protected]> Date: Thu Dec 1 15:37:54 2022 +0200 server: ledger now more organised commit 558423a Author: elkanatovey <[email protected]> Date: Thu Dec 1 13:47:02 2022 +0200 server_service: continue work commit 7649d7f Author: elkanatovey <[email protected]> Date: Wed Nov 30 18:43:11 2022 +0200 server: put in stubs for client connections, start work on bookeeping for work distribution commit 80f53c6 Author: elkanatovey <[email protected]> Date: Wed Nov 30 14:22:34 2022 +0200 server: add rpcs, make implement server from grpc commit e90160c Author: elkanatovey <[email protected]> Date: Tue Nov 29 16:53:02 2022 +0200 server_service: start work on db modifying ptxs for between round work commit e76c080 Author: elkanatovey <[email protected]> Date: Tue Nov 29 16:49:16 2022 +0200 server_service: start work on db modifying ptxs for between round work * added some comments * promise struct * new test * changed def for async mat-mult * fx: removed unneeded check * template fixing * fix: concurrency issues * added test * small refactor to the creation of a FullServer * concurrency package: - fix: bug in channel. - add: threadpool * thread pool * promise set option * matops support receiving an existing threadpool * expander receives threadpool * misplaced constructor in tpp * server fix: construction error, async expander operative * bug fix: forgot to pass promise to thread * on new epoch management receives the client db, creates an EpochData instance and stores everything related to the current epoch there * avoid move * Second stage pir (#57) * manager: fix location of commas in test funcs * evaluator: add ptx decomposition mult * evaluator:add test for ptx embedding * evaluator:add ntt transform for EmbeddedCiphertext * server.cpp: minor change for debug mode * fix: query_expander construction * Create fast_test_debug * Update fast_test_debug * Before processings (#59) * channel throws if write on closed * manager sets up a threadpool * manager sets up random vector each epoch * Squashed commit of the following: commit 6284db7 Merge: 06d979b b75a7d2 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 11 14:20:26 2022 +0200 Merge branch 'dev' of github.com:elkanatovey/distribicom into dev commit 06d979b Author: Jonathan Weiss <[email protected]> Date: Sun Dec 11 14:20:17 2022 +0200 fix: query_expander construction commit b75a7d2 Author: Elkana Tovey <[email protected]> Date: Sun Dec 11 14:03:55 2022 +0200 Second stage pir (#57) * manager: fix location of commas in test funcs * evaluator: add ptx decomposition mult * evaluator:add test for ptx embedding * evaluator:add ntt transform for EmbeddedCiphertext * server.cpp: minor change for debug mode * code-style fix * new epoch async prepares for frievald. * server.cpp: minor change for debug mode * added test to async query expansion * verify the query_x_rand_vec in debug mod * refactor locking in distribute-work * client_context.hpp: start refactor of clientDB * fx: safelatch introduced * refactor safelatch in its own file * refactor safelatch in its own file * client_context.hpp: refactor client setting manager.hpp: start adding partial work to dbs * client_context.hpp: minor fix for cmake * convert client db mutex to pointer * move ClientDB into manager * matrix_operations.tpp: fix multiplication bug for regular plaintext case evaluator_wrapper.cpp: create method for calculating expansion ratio worker_test.cpp: update according to evaluator wrapper * query_expander.cpp: add version of expansion that returns query as col vector * server: factor db into manager matrix_operations: add sync version of scalar dot product manager: add sync versions of stage two, freivalds still has bugs * removed promise-hell * improved async usage * server.cpp: add partial answer struct to client manager.cpp: add method to store partial work, start calculate final answer worker_test: update per mods to client struct * refactor: queries_dim2 are now not promised * fx: ntt form db multiply with query_vec * ensuring worker-manager ntt forms match * ntt-forms match * matrix_operations.hpp: fix for scalar dot product dims * worker_test.cpp update test for more general query nums. manager.hpp: add comment * ntt query-dim-2 * ledger: added promise for verify_worker * async-verification * integrating with current server * fx: removed comments * duct-tape without sending responses * todos Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: elkana <[email protected]> * fix: using random to generate the worker credentials * worker-main * src/server.cpp add creation of clients * src/server.cpp add creation of clients * src/server.cpp add creation of clients * src/server.cpp add creation of clients * fx: worker's binary wouldn't stop running. * Update matrix_operations.hpp Co-authored-by: elkanatovey <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> Co-authored-by: Jonathan Weiss <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.