Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* src: create new directory structure for system * util: create util folder for global includes and include guards. debug asserts now work properly * src: continue work making grpc dependency * src: continue work making grpc dependency * dependdencies: move location of seal download * bin: make executables go into sensible folders * cmake: finish grpc cmake * cmake: revert seal cmake * cmake: stop auto generation of dependency targets * Update cmake.yml * WIP: Query expander (#17) QueryExpander separated from sealpir and can be used as a component to expand specific parts of queries. * cmake: change names of executables in tests for usability purposes * internals: change name of folder for restructure * Update cmake.yml set to ubuntu 22 for compiler puroses * Working-Frievalds (#18) math_utils, test, internals create 2 dim tensor container, implement freivalds algorithm and container, freivalds mostly runs in ptext space by use of split-ptx structure. add proto messages, not integrate with rest of project yet. tests added for freivalds. Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> * fx: wrong verify on ctx-ctx mat-mul * Minor maintainence (#22) * FetchDependencies.cmake: add option for precompiled libraries * README: update readme for new compile options * grpc: update grpc version * .github: create caching version of test * services-wip * adding op order test to evaluator wrapper. * small refactor * adding asserts * added comments * fix: forgot to switch back to bgv * associative test passing * fix: trivial_ciphertext into const func * transformer uses resize to ensure correct size of given matrix * matrix multiplication using matrices * working frievalds test * frievald's unit-test inspect failure with C != A*B * added benchmarks to ptx-ctx multiplication * fx: timing function * benchmark fix * fx: mat-mul with ptx-ctx, frievalds with ctx-ctx. * working on concurrent matrix-mult * adding channel and waitgroup * feat: threaded mat-mul for ptx-ctx * fx: using ntt in mult ptx-ctx matrices * refactor in mat-mult.cpp * parallel ntt transform of matrices * defined RPC messages. refined RPC services * relocated manager, worker services. * minimal working cmake to add services package. should later on split into sub packages * wip: setting up basic worker service * adding test/services package * fx: including services in the cmake of test * testing connection to basic service * wip: worker service * wip: worker receive stream * wip: worker * worker- adding distribicom message creation funcs * wip: worker receives complicated messages. unit tests sends multi-messages * refactoring worker.cpp * refactoring the stream code * fx build * wip: worker-service receive * rfactor * feat: added possibility to send a galoiskey to the worker * clang fix * wip: understanding how to create multi-service server * WIP * fx * WIP * fx * fx Co-authored-by: Jonathan Weiss <[email protected]> * fx: workers-unit test * add scripts used for lab work and fix yaml (#29) * scripts: add scripts used for faster development in aqua environment * github: edit workflow so doesnt try to run dependency tests * workflows fix (#30) * scripts: add scripts used for faster development in aqua environment * github: edit workflow so doesnt try to run dependency tests * Update fast_test.yml * 25 client service (#32) * cmake: make easier to find include files of each target * client_service: start functions * Services (#33) services package: - Worker -manager subscribing - WIP: work-distribution, first connection made (manager to worker)/ - WIP: thread safe DB-handle. - refactored worker, and created utils.hpp. math_util package: - channel read with timeouts. Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Jonathan Weiss <[email protected]> * Manager subscriber (#34) * service package: - threadsafe db. - heavy refactoring and adding utils.hpp. - connection between worker-manager half-complete. no distribution logic as of yet. - server.hpp improvements. - channel can now wwait with a timeout. - fix in matrix.h * read-with-timeout Co-authored-by: Jonathan Weiss <[email protected]> * Update fast_test.yml * Concurrency refactor (#35) * concurrency: -refactor cuncurrency into it's own target and folder, move old files into older for cleanup * src: -refactor cmake files and includes to more consistent naming scheme for including headers. -rename matrix_multiplier to matrix_operations -rename multiplication_utils namespace to math_utils nmespace * upgrading remote-compiler to support callgrind visualisations * WIP: Worker (#37) In this PR we are connecting one side of the main-server to the worker. - manager offers API for waiting to X workers to register. - streaming galois keys from manager to worker. - receiving galois keys on the worker side. - additional constants. * Client service (#36) * Create devcontainer.json * client_service: queries, callback with answer, decryption and round notification implemented * marshal: add marshaling for ptx and ctx vectors , pir responses, and queries into protobuf messages * client_service: test works, add ele sizes to configs Co-authored-by: elkanatovey <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> * server_service: start work on db modifying ptxs for between round work * server_service: start work on db modifying ptxs for between round work * server: add rpcs, make implement server from grpc * server: put in stubs for client connections, start work on bookeeping for work distribution * server_service: continue work * server: ledger now more organised * server: minor refactor * squashed branch: - replaced waitGroup with std:latch/ - mvp reciving worker's local processings. - matrix asserting correct access to its data. * server: commented out unimplemented code * removed unneeded scope * renamed namespace from pir_primitives to services * fix: shared lock on write * renamed clientQueryLedger * manager.cpp: start galois key mapping server.cpp: create mapping logic * manager.cpp: write galois mapping distribute work * server main-loop binary supported * manager: add db sending functionality marshal: add more marshaling caps + test * manager: add query sending functionality * manager: minor change to db responsibilities * manager: halfway done integration * fix: ipv6 support in extract ip * manager.cpp: continue work on integration * manager: mostly done integration * manager: fix bug and factor out old db constructs server: factor out old db constructs proto: minor update to unsigned integer types for consistency * proto: pass ip address in message body from worker * math_utils/evaluator_wrapper: add method for ptx embedding of ctxs * Async stream (#55) * async server sending using streams. * worker identification using credentials (secret given to server upon register). Co-authored-by: Jonathan Weiss <[email protected]> * evaluator_wrapper: add compose to plaintext, add const on add_plain * evaluator_wrapper: compose_to_plaintext writes as ntt form * Server service (#56) * server_service: add fuctionality for work distribution * evaluator_wrapper: add functionality for second stage pir Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: Jonathan <[email protected]> * fix: galois_keys generation * Squashed commit of the following: commit df44f799c26ed4085374ff54fdd4a5b9f94eb7fd Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 09:34:03 2022 +0200 async mat-mult ops - mult with vector from left or right. - mult with matrices of different elements (ctxs-ptx, ptx-ctx, ctx-ctx). commit 867da10ccba23e1ea809cb19dd6a29efab0c64c6 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:51:56 2022 +0200 refactor: naming mult the same for ptx mult vs split-ptx mult commit 5c3e0e9766b7e9712da51c69d5b6f8174b44d211 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:17:56 2022 +0200 refactor commit 772c0f8dfae95e77ea4067b99c711a4eda435d6a Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:15:44 2022 +0200 dropped code used only in tests of itself. commit 578eb077b0f1f7e4142ae2b181241f38a98cc6e3 Author: Jonathan Weiss <[email protected]> Date: Thu Dec 8 08:07:37 2022 +0200 removed commented out code commit 7741ddfd1a90884488fe0f2540077876b663ad87 Author: Jonathan Weiss <[email protected]> Date: Wed Dec 7 18:18:54 2022 +0200 feat: - ctx-ctx matmul is threaded. - removed unused code commit 24d9ea3e7e45587ae4eb5f57f167ba697af4c81b Author: Jonathan Weiss <[email protected]> Date: Wed Dec 7 18:13:46 2022 +0200 generic thread pool utilised commit 3c40703 Author: Jonathan <[email protected]> Date: Wed Dec 7 15:42:38 2022 +0200 Async stream (#55) * async server sending using streams. * worker identification using credentials (secret given to server upon register). Co-authored-by: Jonathan Weiss <[email protected]> commit 0ed00c7 Merge: 956e12a f448803 Author: elkanatovey <[email protected]> Date: Wed Dec 7 15:17:15 2022 +0200 Merge remote-tracking branch 'origin/server_service' into server_service commit 956e12a Author: elkanatovey <[email protected]> Date: Wed Dec 7 15:16:37 2022 +0200 math_utils/evaluator_wrapper: add method for ptx embedding of ctxs commit f448803 Author: elkana <[email protected]> Date: Tue Dec 6 23:17:11 2022 +0200 proto: pass ip address in message body from worker commit e54a9d2 Author: elkanatovey <[email protected]> Date: Tue Dec 6 18:03:54 2022 +0200 manager: fix bug and factor out old db constructs server: factor out old db constructs proto: minor update to unsigned integer types for consistency commit ecda72a Author: elkanatovey <[email protected]> Date: Tue Dec 6 17:37:36 2022 +0200 manager: mostly done integration commit 1a96c9e Author: elkana <[email protected]> Date: Tue Dec 6 12:45:37 2022 +0200 manager.cpp: continue work on integration commit e48447a Author: Jonathan Weiss <[email protected]> Date: Tue Dec 6 08:35:53 2022 +0200 fix: ipv6 support in extract ip commit 598ebc6 Author: elkanatovey <[email protected]> Date: Mon Dec 5 20:50:01 2022 +0200 manager: halfway done integration commit aee2ef2 Author: elkanatovey <[email protected]> Date: Mon Dec 5 18:12:11 2022 +0200 manager: minor change to db responsibilities commit 8c079f0 Author: elkanatovey <[email protected]> Date: Mon Dec 5 16:01:48 2022 +0200 manager: add query sending functionality commit 25ec9f2 Author: elkanatovey <[email protected]> Date: Mon Dec 5 14:24:30 2022 +0200 manager: add db sending functionality marshal: add more marshaling caps + test commit f901016 Author: Jonathan Weiss <[email protected]> Date: Mon Dec 5 09:48:53 2022 +0200 server main-loop binary supported commit c6ba581 Author: elkana <[email protected]> Date: Sun Dec 4 22:37:03 2022 +0200 manager.cpp: write galois mapping distribute work commit 5122396 Author: elkana <[email protected]> Date: Sun Dec 4 20:40:08 2022 +0200 manager.cpp: start galois key mapping server.cpp: create mapping logic commit 19059de Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:58:53 2022 +0200 renamed clientQueryLedger commit 1928551 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:53:19 2022 +0200 fix: shared lock on write commit e0267a1 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:29:57 2022 +0200 renamed namespace from pir_primitives to services commit 24c569f Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:27:53 2022 +0200 removed unneeded scope commit 39a3665 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:24:27 2022 +0200 server: commented out unimplemented code commit 30e15bc Author: Jonathan Weiss <[email protected]> Date: Sun Dec 4 11:17:56 2022 +0200 squashed branch: - replaced waitGroup with std:latch/ - mvp reciving worker's local processings. - matrix asserting correct access to its data. commit 8750fbf Author: elkanatovey <[email protected]> Date: Thu Dec 1 18:02:25 2022 +0200 server: minor refactor commit 0696f1a Author: elkanatovey <[email protected]> Date: Thu Dec 1 15:37:54 2022 +0200 server: ledger now more organised commit 558423a Author: elkanatovey <[email protected]> Date: Thu Dec 1 13:47:02 2022 +0200 server_service: continue work commit 7649d7f Author: elkanatovey <[email protected]> Date: Wed Nov 30 18:43:11 2022 +0200 server: put in stubs for client connections, start work on bookeeping for work distribution commit 80f53c6 Author: elkanatovey <[email protected]> Date: Wed Nov 30 14:22:34 2022 +0200 server: add rpcs, make implement server from grpc commit e90160c Author: elkanatovey <[email protected]> Date: Tue Nov 29 16:53:02 2022 +0200 server_service: start work on db modifying ptxs for between round work commit e76c080 Author: elkanatovey <[email protected]> Date: Tue Nov 29 16:49:16 2022 +0200 server_service: start work on db modifying ptxs for between round work * added some comments * promise struct * new test * changed def for async mat-mult * fx: removed unneeded check * template fixing * fix: concurrency issues * added test * small refactor to the creation of a FullServer * concurrency package: - fix: bug in channel. - add: threadpool * thread pool * promise set option * matops support receiving an existing threadpool * expander receives threadpool * misplaced constructor in tpp * server fix: construction error, async expander operative * bug fix: forgot to pass promise to thread * on new epoch management receives the client db, creates an EpochData instance and stores everything related to the current epoch there * avoid move * Second stage pir (#57) * manager: fix location of commas in test funcs * evaluator: add ptx decomposition mult * evaluator:add test for ptx embedding * evaluator:add ntt transform for EmbeddedCiphertext * server.cpp: minor change for debug mode * fix: query_expander construction * Create fast_test_debug * Update fast_test_debug * Before processings (#59) * channel throws if write on closed * manager sets up a threadpool * manager sets up random vector each epoch * Squashed commit of the following: commit 6284db7 Merge: 06d979b b75a7d2 Author: Jonathan Weiss <[email protected]> Date: Sun Dec 11 14:20:26 2022 +0200 Merge branch 'dev' of github.com:elkanatovey/distribicom into dev commit 06d979b Author: Jonathan Weiss <[email protected]> Date: Sun Dec 11 14:20:17 2022 +0200 fix: query_expander construction commit b75a7d2 Author: Elkana Tovey <[email protected]> Date: Sun Dec 11 14:03:55 2022 +0200 Second stage pir (#57) * manager: fix location of commas in test funcs * evaluator: add ptx decomposition mult * evaluator:add test for ptx embedding * evaluator:add ntt transform for EmbeddedCiphertext * server.cpp: minor change for debug mode * code-style fix * new epoch async prepares for frievald. * server.cpp: minor change for debug mode * added test to async query expansion * verify the query_x_rand_vec in debug mod * refactor locking in distribute-work * client_context.hpp: start refactor of clientDB * fx: safelatch introduced * refactor safelatch in its own file * refactor safelatch in its own file * client_context.hpp: refactor client setting manager.hpp: start adding partial work to dbs * client_context.hpp: minor fix for cmake * convert client db mutex to pointer * move ClientDB into manager * matrix_operations.tpp: fix multiplication bug for regular plaintext case evaluator_wrapper.cpp: create method for calculating expansion ratio worker_test.cpp: update according to evaluator wrapper * query_expander.cpp: add version of expansion that returns query as col vector * server: factor db into manager matrix_operations: add sync version of scalar dot product manager: add sync versions of stage two, freivalds still has bugs * removed promise-hell * improved async usage * server.cpp: add partial answer struct to client manager.cpp: add method to store partial work, start calculate final answer worker_test: update per mods to client struct * refactor: queries_dim2 are now not promised * fx: ntt form db multiply with query_vec * ensuring worker-manager ntt forms match * ntt-forms match * matrix_operations.hpp: fix for scalar dot product dims * worker_test.cpp update test for more general query nums. manager.hpp: add comment * ntt query-dim-2 * ledger: added promise for verify_worker * async-verification * integrating with current server * fx: removed comments * duct-tape without sending responses * todos Co-authored-by: Jonathan Weiss <[email protected]> Co-authored-by: elkana <[email protected]> * fix: using random to generate the worker credentials * worker-main * src/server.cpp add creation of clients * src/server.cpp add creation of clients * src/server.cpp add creation of clients * src/server.cpp add creation of clients * fx: worker's binary wouldn't stop running. * Update matrix_operations.hpp Co-authored-by: elkanatovey <[email protected]> Co-authored-by: Elkana Tovey <[email protected]> Co-authored-by: Jonathan Weiss <[email protected]>
- Loading branch information
4 people
authored
Dec 14, 2022
1 parent
177ebed
commit b4bce21