Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collector service #734

Merged
merged 64 commits into from
Aug 31, 2023
Merged

Collector service #734

merged 64 commits into from
Aug 31, 2023

Conversation

mike0sv
Copy link
Collaborator

@mike0sv mike0sv commented Aug 23, 2023 •
edited
Loading

For testing:

  • go to examples/integrations/collector_service
  • run evidently ui to run UI service
  • run python app.py to run collector service
  • run python example_report.py to configure collector, workspace and report and start sending data

Additional thoughts/todos:

  • Snapshot trigger is just "every N seconds", we might want to support others like cron or "every N events" (implement other triggers)
  • report needs to run once on configuration step because metric generators should be called, and they are called only on run.
  • We should move code to evidently.collector or evidently.ui.collector. Also add cli command to run it and probably add an option to add collector on top of UI service (eg evidently ui --collector will add routes at /api/collector)

Changes to previous attempt:

  • multiple collector configs: you can use one app to create multiple snapshots to multiple projects
  • collectors can be pre-configured via file or after service started with API
  • CollectorClient (not fully implemented yet)
  • Reference should be placed as a parquet file, but also can be set up/changed via API
  • trigger object (for now only interval implemented. we also need cron/buffer size triggers)
  • storage object (for now only in-memory implemented)

mike0sv and others added 30 commits May 11, 2023 17:47
@mike0sv
report serialization prototype
a229cd7
@mike0sv
WIP
fd05a76
@mike0sv
additional pydantic validators
de233a3
@mike0sv
it's alive
b8af055
@mike0sv
use metric directly
124afb0
@mike0sv
tests WIP
ef311a4
@mike0sv
tests WIP
0d86212
@mike0sv
fix mypy
7e63dea
@mike0sv
fix tests and silence mypy :(
836a700
@mike0sv
Merge branch 'main' into feature/result-profiles-pydantic
34196a1 
# Conflicts:
#	src/evidently/metrics/classification_performance/base_classification_metric.py
#	src/evidently/metrics/data_drift/column_drift_metric.py
#	src/evidently/metrics/data_integrity/column_summary_metric.py
@mike0sv
fix tests after merge
90a02a6
@mike0sv
inner suite payload
3804dfa
@mike0sv
fix metrics presets
1f05d41
@mike0sv
fix tests
8b02a55
@mike0sv
add test presets tests
88cd717
@mike0sv
fix example tests
e157815
@mike0sv
fix comments
06d5fee
@mike0sv
unused imports
e0e2d26
@mike0sv
make them private
5de221c
@mike0sv
Monitors POC
27ccdea
@mike0sv
fix show
8ff61a6
@mike0sv
Merge branch 'main' into feature/result-profiles-pydantic
97331f3 
# Conflicts:
#	examples/data_stories/default_stattest_adult.ipynb
#	examples/how_to_questions/how_to_run_drift_report_for_text_data.ipynb
#	examples/how_to_questions/how_to_use_aggregated_data_option.ipynb
#	src/evidently/metrics/classification_performance/pr_table_metric.py
#	src/evidently/metrics/data_integrity/dataset_summary_metric.py
@mike0sv
merge
d336170
@mike0sv
Merge branch 'feature/result-profiles-pydantic' into feature/monitors…
7692b7c 
…-poc

# Conflicts:
#	src/evidently/report/report.py
@mike0sv
rename
2791cfc
@mike0sv
backend POC
a5364e9
@Liraim
UI copy
f3641e5
@mike0sv
add id to reports and projects
bb1a983
@Liraim
Merge remote-tracking branch 'origin/feature/service_backend' into fe…
cce2f50 
…ature/evidently_service_ui
@Liraim
First version of UI refactoring and separating with service and stand…
8c252cd 
…alone ui.
mike0sv and others added 27 commits June 20, 2023 12:41
@mike0sv
Rename dashboard -> panel, save project metadata (#638)
2edcec9
@Liraim
Update ui files.
c61187b
@Liraim
Fix dashboard info for reports and test suites.
3d9fb29
@Liraim
Add download functionality.
651b973
@mike0sv
Better UI panels config (#642)
7cdbc43
@Liraim
Update test suites names.
1d1dd73 
Add download for test suites.
@mike0sv
add report tags (#643)
7f5701a
@mike0sv
add local workspace and project API (#645)
5971074
@mike0sv
add remote workspace and project API
946bf3c
@mike0sv
lint
4a1c60f
@mike0sv
lint
e645190
@mike0sv
lint
82355bb
@mike0sv
fix testresult exception field
45e9348
@mike0sv
collector service example
cdf84e3
@mike0sv
smol tweaks
6fb2a5e
@mike0sv
Merge branch 'main' into feature/collector_example_ui
1c30a63 
# Conflicts:
#	examples/sample_notebooks/evidently_metrics.ipynb
#	src/evidently/base_metric.py
#	src/evidently/core.py
#	src/evidently/descriptors/oov_words_percentage_descriptor.py
#	src/evidently/descriptors/trigger_words_presence_descriptor.py
#	src/evidently/experimental/report_set.py
#	src/evidently/features/OOV_words_percentage_feature.py
#	src/evidently/features/generated_features.py
#	src/evidently/features/non_letter_character_percentage_feature.py
#	src/evidently/features/text_length_feature.py
#	src/evidently/features/trigger_words_presence_feature.py
#	src/evidently/metric_results.py
#	src/evidently/metrics/classification_performance/base_classification_metric.py
#	src/evidently/metrics/classification_performance/classification_dummy_metric.py
#	src/evidently/metrics/classification_performance/classification_quality_metric.py
#	src/evidently/metrics/classification_performance/confusion_matrix_metric.py
#	src/evidently/metrics/classification_performance/quality_by_class_metric.py
#	src/evidently/metrics/data_drift/data_drift_table.py
#	src/evidently/metrics/data_drift/embeddings_drift.py
#	src/evidently/metrics/data_drift/text_domain_classifier_drift_metric.py
#	src/evidently/metrics/data_integrity/column_missing_values_metric.py
#	src/evidently/metrics/data_integrity/column_regexp_metric.py
#	src/evidently/metrics/data_integrity/column_summary_metric.py
#	src/evidently/metrics/data_integrity/dataset_summary_metric.py
#	src/evidently/metrics/data_quality/column_correlations_metric.py
#	src/evidently/metrics/data_quality/column_distribution_metric.py
#	src/evidently/metrics/data_quality/column_quantile_metric.py
#	src/evidently/metrics/regression_performance/objects.py
#	src/evidently/metrics/regression_performance/utils.py
#	src/evidently/pydantic_utils.py
#	src/evidently/report/report.py
#	src/evidently/suite/base_suite.py
#	src/evidently/test_suite/test_suite.py
#	src/evidently/tests/classification_performance_tests.py
#	src/evidently/tests/data_drift_tests.py
#	src/evidently/tests/data_integrity_tests.py
#	src/evidently/utils/numpy_encoder.py
#	tests/metric_preset/test_all_metrics_presets.py
#	tests/metrics/data_interity/test_dataset_summary_metric.py
#	tests/test_suite/test_test_suite_loading.py
#	ui/lib/App.tsx
#	ui/lib/api/Api.ts
#	ui/lib/api/LocalApi.ts
#	ui/lib/contexts/ApiContext.tsx
#	ui/service/package.json
#	ui/service/public/favicon-16x16.png
#	ui/service/public/favicon-32x32.png
#	ui/service/public/favicon.ico
#	ui/service/public/index.html
#	ui/service/src/Components/ProjectData.tsx
#	ui/service/src/Components/ProjectList.tsx
#	ui/service/src/Components/Reports.tsx
#	ui/service/src/Components/ServiceHeader.tsx
#	ui/service/src/Components/ServiceMainPage.tsx
#	ui/service/src/Components/TestSuites.tsx
#	ui/service/src/api/RemoteApi.ts
#	ui/service/src/index.tsx
@mike0sv
remove old stuff
c839b3d
@mike0sv
remove old stuff
0725ee8
@mike0sv
refactor
48caaed
@mike0sv
move collector inside
e8eae3b
@mike0sv
merge client logic
e6abcae
@mike0sv
lint and reqs
037dcdd
@mike0sv
lint and reqs
132543d
@mike0sv
add tests
0e50a3a
@mike0sv
fix exmaple and lint
9895e5d
@mike0sv
typos and requirements
e75ac6d
@mike0sv
Merge branch 'main' into feature/collector_service
0c739a9 
# Conflicts:
#	requirements.dev.txt
@emeli-dral emeli-dral merged commit 1771783 into main Aug 31, 2023
18 checks passed
@emeli-dral emeli-dral deleted the feature/collector_service branch August 31, 2023 19:29
@magdalenakuhn17 magdalenakuhn17 mentioned this pull request Nov 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mike0sv @emeli-dral @Liraim