Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize source refresh and ES indexation process #176

Merged
merged 50 commits into from
Aug 29, 2023
Merged

Optimize source refresh and ES indexation process #176

merged 50 commits into from
Aug 29, 2023

Conversation

LePetitTim
Copy link
Collaborator

No description provided.

@LePetitTim LePetitTim requested a review from submarcos April 11, 2023 16:07
@LePetitTim LePetitTim force-pushed the perf_change_savepoint_index_es branch from 92998f6 to 8b91571 Compare April 11, 2023 16:12
@codecov
Copy link

codecov bot commented Apr 11, 2023 •
edited
Loading

Codecov Report

Merging #176 (6a28707) into master (dea73df) will increase coverage by 1.71%.
Report is 2 commits behind head on master.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #176      +/-   ##
==========================================
+ Coverage   93.32%   95.04%   +1.71%     
==========================================
  Files          78       79       +1     
  Lines        2758     2827      +69     
==========================================
+ Hits         2574     2687     +113     
+ Misses        184      140      -44
Flag Coverage Δ
backend-unittests 95.04% <100.00%> (+1.71%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
project/geosource/elasticsearch/index.py 81.39% <ø> (+45.83%) ⬆️
project/geosource/exceptions.py 100.00% <100.00%> (ø)
project/geosource/geostore_callbacks.py 100.00% <100.00%> (+3.44%) ⬆️
project/geosource/models.py 98.71% <100.00%> (+1.82%) ⬆️
project/geosource/serializers.py 85.21% <100.00%> (+6.82%) ⬆️
project/geosource/tasks.py 92.15% <100.00%> (-3.59%) ⬇️

@submarcos submarcos changed the title Perf change savepoint index es Optimize source refresh and ES indexation process Apr 12, 2023
@submarcos submarcos added Bug Something isn't working enhancement New feature or request labels Apr 12, 2023
@submarcos submarcos linked an issue Apr 12, 2023 that may be closed by this pull request
Improve source refresh process #112
Closed
@LePetitTim LePetitTim force-pushed the perf_change_savepoint_index_es branch 3 times, most recently from c96aaab to 3a162b2 Compare April 12, 2023 15:40
@LePetitTim LePetitTim force-pushed the perf_change_savepoint_index_es branch 3 times, most recently from 967c78f to cd6d449 Compare April 25, 2023 09:16
CFarcy
CFarcy previously approved these changes May 4, 2023
View reviewed changes
@LePetitTim LePetitTim force-pushed the perf_change_savepoint_index_es branch from cd6d449 to 5c477ca Compare May 10, 2023 13:01
@LePetitTim LePetitTim force-pushed the perf_change_savepoint_index_es branch from a845f1a to 8d3e2a6 Compare May 24, 2023 09:56
submarcos
submarcos reviewed May 24, 2023
View reviewed changes
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/models.py Outdated Show resolved Hide resolved
@CFarcy CFarcy force-pushed the perf_change_savepoint_index_es branch from 68b6818 to 103f9bc Compare May 25, 2023 13:55
@CFarcy CFarcy force-pushed the perf_change_savepoint_index_es branch from 228c2f7 to f5d68ed Compare June 6, 2023 09:59
@CFarcy CFarcy force-pushed the perf_change_savepoint_index_es branch 2 times, most recently from 25f4ee4 to 9e7ebdf Compare July 10, 2023 13:18
submarcos
submarcos reviewed Jul 10, 2023
View reviewed changes
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/models.py Outdated Show resolved Hide resolved
project/geosource/serializers.py Outdated Show resolved Hide resolved
@CFarcy CFarcy dismissed their stale review July 10, 2023 15:57

I reviewed this PR before adding some code to it. It is not relevant anymore

Cédric Farcy and others added 23 commits August 28, 2023 16:12
@submarcos
Update project/geosource/serializers.py
ead7a7b 
Co-authored-by: Jean-Etienne Castagnede <[email protected]>
@submarcos
Update project/geosource/models.py
fb49b35 
Co-authored-by: Jean-Etienne Castagnede <[email protected]>
Small code adjustement
cd3aac6
Isort
64a0794
Add translation for report status message
cded24e
Remove unused variables
58d1e5f
Isort
77385ec
Update migration to make it work with reel data
0b0930b 
* Remove then "re-create" the report field on the Source model to avoid conflict with existing data
Force report creation at refresh start
4d2f39a
Force save report after reset
ec235e8
Uncatched exception by get_records set reporting in error state
752e085
Report wrong user info for csvsource
0c5710c
Black
0ea5a61
Updating tests
37a0c92
Fix postgis refresh failing
e5a6a4b
Rework source status
b608952 
* Api return only status code instead of text
* wmts source does not have a refresh status
Fix report not set to warning status correctly
b538976
Improve source reporting
97cde85 
* remove unnecessary reporting
* improve message readability
* add total count to the report
Improve test coverage
b545543
Add ES patch to tests
daa042f
Black
f115142
More SourceException tests
ed1785f
Test refresh status for wmtssource
1464bec
@CFarcy CFarcy force-pushed the perf_change_savepoint_index_es branch from affcb2b to 38ffeef Compare August 28, 2023 14:12
@CFarcy CFarcy force-pushed the perf_change_savepoint_index_es branch from 38ffeef to 4d6fa86 Compare August 28, 2023 14:30
@submarcos submarcos merged commit de519db into master Aug 29, 2023
7 checks passed
@submarcos submarcos deleted the perf_change_savepoint_index_es branch August 29, 2023 09:05
@submarcos submarcos linked an issue Sep 1, 2023 that may be closed by this pull request
Improve source refresh status in admin #111
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@submarcos submarcos submarcos left review comments

@CFarcy CFarcy CFarcy left review comments

Assignees
No one assigned
Labels
Bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve source refresh process Improve source refresh status in admin
3 participants
@LePetitTim @CFarcy @submarcos