Test Suites with Individual Tests and Test Presets

Implemented new interfaces to test data and models in a batch: Test Suite.

Implemented the following Individual tests:

• TestNumberOfColumns()
• TestNumberOfRows()
• TestColumnNANShare()
• TestShareOfOutRangeValues()
• TestNumberOfOutListValues()
• TestMeanInNSigmas()
• TestMostCommonValueShare()
• TestNumberOfConstantColumns()
• TestNumberOfDuplicatedColumns()
• TestNumberOfDuplicatedRows()
• TestHighlyCorrelatedFeatures()
• TestTargetFeaturesCorrelations()
• TestShareOfDriftedFeatures()
• TestValueDrfit()
• TestColumnsType()

Implemented the following test presets:

• Data Quality. This preset is focused on the data quality issues like duplicate rows or null values.  
• Data Stability. This preset identifies the changes in the data or differences between the batches.
• Data Drift. This one compares feature distributions using statistical tests and distance metrics.  
• NoTargetPerformance. This preset combines several checks to run when there are model predictions, there are no actuals or ground truth labels. This includes checking for prediction drift and some of the data quality and stability checks.

v0.1.51.dev0

Updates:

• Updated DataDriftTab: added target and prediction rows in DataDrift Table widget
• Updated CatTargetDriftTab: added additional widgets for probabilistic cases in both binary and multiclasss probabilistic classification, particularly widget for label drift and class probability distributions.

Fixes:

• #233
• fixed previes in DataDrift Table widget. Now histogram previews for refernce and current data share an x-axis. This means that bins order in refernce and current histograms is the same, it makes visual distribution comparion esier.

Automatic Stattest Selection

Release scope:

1. Stat test auto selection algorithm update: https://docs.evidentlyai.com/reports/data-drift#how-it-works

For small data with <= 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): two-sample Kolmogorov-Smirnov test.
• For categorical features or numerical features with n_unique <= 5: chi-squared test.
• For binary categorical features (n_unique <= 2), we use the proportion difference test for independent samples based on Z-score.
  All tests use a 0.95 confidence level by default.

For larger data with > 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): Wasserstein Distance.
• For categorical features or numerical with n_unique <= 5): Jensen–Shannon divergence.
  All tests use a threshold = 0.1 by default.

2. Added options for setting custom statistical test for Categorical and Numerical Target Drift Dashboard/Profile:
   cat_target_stattest_func: Defines a custom statistical test to detect target drift in CatTargetDrift.
num_target_stattest_func: Defines a custom statistical test to detect target drift in NumTargetDrift.

3. Added options for setting custom threshold for drift detection for Categorical and Numerical Target Drift Dashboard/Profile:
   cat_target_threshold: Optional[float] = None
num_target_threshold: Optional[float] = None
   These thresholds highly depends on selected stattest, generally it is either threshold for p_value or threshold for a distance.

Fixes:
#207

Statistical Tests for Categorical Features

StatTests
The following statistical tests now can be used for both numerical and categorical features:

• 'jensenshannon'
• 'kl_div'
• 'psi

Grafana monitoring example

• Updated the example to be used with several ML models
• Added monitors for NumTargetDrift, CatTargetDrift

Dashboard Customisation: Colour Scheme and Statistical Test

Colour Scheme
Support for custom colours in the Dashboards:

• primary_color
• secondary_color
• current_data_color
• reference_data_color
• color_sequence
• fill_color
• zero_line_color
• non_visible_color
• underestimation_color
• overestimation_color
• majority_color

Statistical Test:
Support for user implemented statistical tests
Support for custom statistical tests in Dashboards and Profiles
Available tests:

• 'ks'
• 'z'
• 'chisquare'
• 'jensenshannon'
• 'kl_div'
• 'psi'
• 'wasserstein'
  more info: docs

Fixes:
#193

Custom Text Comments in Dashboards

Custom Text Comments in Dashboards

• Added type type="text” for BaseWidgetInfo (for text widgets implementation)
• Markdown syntax is supported

see the example: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/text_widget_usage_iris.ipynb

Data Quality Dashboard Optimisation

• Data Quality Dashboard: add dataset overview widget
• Data Quality Dashboard: add correlations widget
• Speeded uploading via preview plots optimisation
• Paging in Data Quality feature table widget

DataQuality Dashboard and Profile

• DataQualityTab() is now available for Dashboards! The Tab helps to observe data columns, explore their properties and compare datasets.
• DataQualityProfileSection() is available for Profiles as well.
• ColumnMapping update: added task parameter to specify the type of machine learning problem. This parameter is used by DataQualityAnalyzer and some data quality widgets to calculate metrics and visuals with the respect to the task.

Support for strings as a categorical features

• Added monitors for NumTargetDrift, CatTargetDrift, ClassificationPerformance, ProbClassificationPerformance
• Fixed RegressionPerformance monitor
• Supported strings as a categorical features in DataDrift and CatTargetDrift dashboards
• Supported boolean features in DataDrift dashboard

QualityMetricsOptions

Analyzers Refactoring: analyzer result became a structured object instead of a dictionary for all Analyzers

The following Quality Metrics Options are added:

• conf_interval_n_sigmas (the width of confidence intervals ): int = DEFAULT_CONF_INTERVAL_SIZE
• classification_treshold (the threshold for true labels): float = DEFAULT_CLASSIFICATION_TRESHOLD
• cut_quantile (cut the data by right, left and two-sided quantiles): Union[None, Tuple[str, float], Dict[str, Tuple[str, float]]] = None