From 14da3c96312537f9ca63a79fc1f9ea0d848d4d4c Mon Sep 17 00:00:00 2001 From: Mick Date: Mon, 20 Dec 2021 09:16:05 +0100 Subject: [PATCH 1/3] Adds efficiency of REL steps to README --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 55d5458..19d10ec 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,19 @@ The remainder of the tutorials are optional and for users who wish to e.g. train 6. [REL as systemd service](https://github.com/informagi/REL/tree/master/tutorials/06_systemd_instructions.md) 7. [Notes on using custom models](https://github.com/informagi/REL/tree/master/tutorials/07_custom_models.md) +# Efficiency of REL +We measured the efficiency of REL on a per-document basis. We ran our API with 50 documents from AIDA-B with > 200 words, which is 323 (± 105) words and 42 (± 19) mentions per document. The results are added to the table below. + +| Model | Time MD | Time ED | +| ------ | --------------- | -- | +| With GPU | 0.44±0.22 | 0.24±0.08 | +| Without GPU | 2.41±1.24| 0.18±0.09| + +As our package has changed overtime, we refer to one of our [earlier commits](https://github.com/informagi/REL/tree/a0a93487ecc640a72f33ffe015a7a34dff8f054f) for reproducing the results in the table above. To reproduce the results above, perform the following steps: +1. Start the server. As can be seen in `server.py`, we added [checkpoints in our server calls](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/REL/server.py#L82) to measure time taken per call. +3. Once the server is started, run the [efficiency test](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_test.py). Do not forget to update the `base_url` in the respective script. +4. Finally, process the [efficiency results](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_results.py). + # Cite If you are using REL, please cite the following paper: From 97ebcd7093533e77175dcd93ca20c91f17ca1a14 Mon Sep 17 00:00:00 2001 From: Mick Date: Mon, 20 Dec 2021 13:00:41 +0100 Subject: [PATCH 2/3] Elaborates on base_url meaning and refers to tutorial --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 19d10ec..ccdb292 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ We measured the efficiency of REL on a per-document basis. We ran our API with 5 As our package has changed overtime, we refer to one of our [earlier commits](https://github.com/informagi/REL/tree/a0a93487ecc640a72f33ffe015a7a34dff8f054f) for reproducing the results in the table above. To reproduce the results above, perform the following steps: 1. Start the server. As can be seen in `server.py`, we added [checkpoints in our server calls](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/REL/server.py#L82) to measure time taken per call. -3. Once the server is started, run the [efficiency test](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_test.py). Do not forget to update the `base_url` in the respective script. +3. Once the server is started, run the [efficiency test](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_test.py). Do not forget to update the `base_url` in the respective script to the directory where all data is stored (see our [tutorial on how to get started](https://github.com/informagi/REL/blob/master/tutorials/01_How_to_get_started.md) 4. Finally, process the [efficiency results](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_results.py). # Cite From d624c8809079316251338d3d3f612c25c6522f20 Mon Sep 17 00:00:00 2001 From: Mick Date: Mon, 20 Dec 2021 13:02:18 +0100 Subject: [PATCH 3/3] Small update efficiency of REL readme --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ccdb292..790268a 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,7 @@ We measured the efficiency of REL on a per-document basis. We ran our API with 5 As our package has changed overtime, we refer to one of our [earlier commits](https://github.com/informagi/REL/tree/a0a93487ecc640a72f33ffe015a7a34dff8f054f) for reproducing the results in the table above. To reproduce the results above, perform the following steps: 1. Start the server. As can be seen in `server.py`, we added [checkpoints in our server calls](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/REL/server.py#L82) to measure time taken per call. -3. Once the server is started, run the [efficiency test](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_test.py). Do not forget to update the `base_url` in the respective script to the directory where all data is stored (see our [tutorial on how to get started](https://github.com/informagi/REL/blob/master/tutorials/01_How_to_get_started.md) +3. Once the server is started, run the [efficiency test](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_test.py). Do not forget to update the `base_url` to specify where the data is located in the filesystem. This directory refers to where all project-related data is stored (see our [tutorial on how to get started](https://github.com/informagi/REL/blob/master/tutorials/01_How_to_get_started.md) 4. Finally, process the [efficiency results](https://github.com/informagi/REL/blob/a0a93487ecc640a72f33ffe015a7a34dff8f054f/scripts/efficiency_results.py). # Cite