- Fixed division by zero error for diff metrics.
- Depends on
numpy>=1.16,<1.20.
- Created tfma.StandardExtracts with helper methods for common keys.
- Updated StandardMetricInputs to extend from the tfma.StandardExtracts.
- Created set of StandardMetricInputsPreprocessors for filtering extracts.
- Introduced a
padding_optionsconfig to ModelSpec to configure whether and how to pad the prediction and label tensors expected by the model's metrics.
- Fixed issue with metric computation deduplication logic.
- Depends on
apache-beam[gcp]>=2.27,<3. - Depends on
pyarrow>=1,<3. - Depends on
tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,<3. - Depends on
tensorflow-metadata>=0.27.0,<0.28.0. - Depends on
tfx-bsl>=0.27.0,<0.28.0.
- N/A
- N/A
- Added support for aggregating feature attributions using special metrics
that extend from
tfma.metrics.AttributionMetric(e.g.tfma.metrics.TotalAttributions,tfma.metrics.TotalAbsoluteAttributions). To use make use of these metrics a custom extractor that add attributions to thetfma.Extractsunder the key nametfma.ATTRIBUTIONS_KEYmust be manually created. - Added support for feature transformations using TFT and other preprocessing functions.
- Add support for rubber stamping (first run without a valid baseline model) when validating a model. The change threshold is ignored only when the model is rubber stamped, otherwise, an error is thrown.
- Fix the bug that Fairness Indicator UI metric list won't refresh if the input eval result changed.
- Add support for missing_thresholds failure to validations result.
- Updated to set min/max value for precision/recall plot to 0 and 1.
- Fix issue with MinLabelPosition not being sorted by predictions.
- Updated NDCG to ignore non-positive gains.
- Fix bug where an example could be aggregated more than once in a single slice if the same slice key were generated from more than one SlicingSpec.
- Add threshold support for confidence interval type metrics based on its unsampled_value.
- Depends on
apache-beam[gcp]>=2.25,!=2.26.*,<3. - Depends on
tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.4.*,<3. - Depends on
tensorflow-metadata>=0.26.0,<0.27.0. - Depends on
tfx-bsl>=0.26.0,<0.27.0.
- Changed MultiClassConfusionMatrix threshold check to prediction > threshold instead of prediction >= threshold.
- Changed default handling of materialize in default_extractors to False.
- Separated
tfma.extractors.BatchedInputExtractorintotfma.extractors.FeaturesExtractor,tfma.extractors.LabelsExtractor, andtfma.extractors.ExampleWeightsExtractor. - Change the thresholding to be inclusive, i.e. model passes when value is >= or <= to the threshold rather than > or <.
- N/A
-
Added support for reading and writing metrics, plots and validation results using Apache Parquet.
-
Updated the FI indicator slicing selection UI.
-
Fixed the problem that slices are refreshed when user selected a new baseline.
-
Add support for slicing on ragged and multidimensional data.
-
Load TFMA correctly in JupyterLabs even if Facets has loaded first.
-
Added support for aggregating metrics using top k values.
-
Added support for padding labels and predictions with -1 to align a batch of inputs for use in tf-ranking metrics computations.
-
Added support for fractional labels.
-
Add metric definitions as tooltips in the Fairness Inidicators metric selector UI
-
Added support for specifying label_key to use with MinLabelPosition metric.
-
From this release TFMA will also be hosting nightly packages on https://pypi-nightly.tensorflow.org. To install the nightly package use the following command:
pip install -i https://pypi-nightly.tensorflow.org/simple tensorflow-model-analysisNote: These nightly packages are unstable and breakages are likely to happen. The fix could often take a week or more depending on the complexity involved for the wheels to be available on the PyPI cloud service. You can always use the stable version of TFMA available on PyPI by running the command
pip install tensorflow-model-analysis.
- Fix incorrect calculation with MinLabelPosition when used with weighted examples.
- Fix issue with using NDCG metric without binarization settings.
- Fix incorrect computation when example weight is set to zero.
- Depends on
apache-beam[gcp]>=2.25,<3. - Depends on
tensorflow-metadata>=0.25.0,<0.26.0. - Depends on
tfx-bsl>=0.25.0,<0.26.0.
AggregationOptionsare now independent ofBinarizeOptions. In order to computeAggregationOptions.macro_averageorAggregationOptions.weighted_macro_average,AggregationOptions.class_weightsmust now be configured. IfAggregationOptions.class_weightsare provided, any missing keys now default to 0.0 instead of 1.0.- In the UI, aggregation based metrics will now be prefixed with 'micro_', 'macro_', or 'weighted_macro_' depending on the aggregation type.
tfma.extractors.FeatureExtractor,tfma.extractors.PredictExtractor,tfma.extractors.InputExtractor, andtfma.evaluators.MetricsAndPlotsEvaluatorare deprecated and may be replaced with newer versions in upcoming releases.
- N/A
- Depends on
apache-beam[gcp]>=2.24,<3. - Depends on
tfx-bsl>=0.24.0,<0.25.0.
- N/A
- N/A
- N/A
- Added an extra requirement group
all. As a result, barebone TFMA does not requiretensorflowjs,prompt-toolkitandipythonany more. - Added an extra requirement group
allthat specifies all the extra dependencies TFMA needs. Usepip install tensorflow-model-analysis[all]to pull in those dependencies.
- N/A
- N/A
- N/A
- Fix Jupyter lab issue with missing data-base-url.
- N/A
- N/A
- Use TFXIO and batched extractors by default in TFMA.
- Updated the type hint of FilterOutSlices.
- Fix issue with precision@k and recall@k giving incorrect values when negative thresholds are used (i.e. keras defaults).
- Fix issue with MultiClassConfusionMatrixPlot being overridden by MultiClassConfusionMatrix metrics.
- Made the Fairness Indicators UI thresholds drop down list sorted.
- Fix the bug that Sort menu is not hidden when there is no model comparison.
- Depends on
absl-py>=0.9,<0.11. - Depends on
ipython>=7,<8. - Depends on
pandas>=1.0,<2. - Depends on
protobuf>=3.9.2,<4. - Depends on
tensorflow-metadata>=0.24.0,<0.25.0. - Depends on
tfx-bsl>=0.24.0,<0.25.0.
- Query based metrics evaluations that make use of
MetricsSpecs.query_keyare now passedtfma.Extractswith leaf values that are of typenp.ndarraycontaining an additional dimension representing the values matched by the query (e.g. if the labels and predictions were previously 1D arrays, they will now be 2D arrays where the first dimension's size is equal to the number of examples matching the query key). Previously a list oftfma.Extractswas passed instead. This allows user's to now add custom metrics based ontf.keras.metrics.Metricas well astf.metrics.Metric(any previous customizations based ontf.metrics.Metricwill need to be updated). As part of this change thetfma.metrics.NDCG,tfma.metrics.MinValuePosition, andtfma.metrics.QueryStatisticshave been updated. - Renamed
ConfusionMatrixMetric.computetoConfusionMatrixMetric.resultfor consistency with other APIs.
- Deprecating Py3.5 support.
- Changed default confidence interval method from POISSON_BOOTSTRAP to JACKKNIFE. This should significantly improve confidence interval evaluation performance by as much as 10x in runtime and CPU resource usage.
- Added support for additional confusion matrix metrics (FDR, FOR, PT, TS, BA, F1 score, MCC, FM, Informedness, Markedness, etc). See https://en.wikipedia.org/wiki/Confusion_matrix for full list of metrics now supported.
- Change the number of partitions used by the JACKKNIFE confidence interval methodology from 100 to 20. This will reduce the quality of the confidence intervals but support computing confidence intervals on slices with fewer examples.
- Added
tfma.metrics.MultiClassConfusionMatrixAtThresholds. - Refactoring code to compute
tfma.metrics.MultiClassConfusionMatrixPlotusing derived computations. - Provide support for evaluating TFJS models.
- Added support for labels passed as SparseTensorValues.
- Stopped requiring
avro-python3. - Fix NoneType error when passing BinarizeOptions to tfma.metrics.default_multi_class_classification_specs.
- Fix issue with custom metrics contained in modules ending in tf.keras.metric.
- Changed the BoundedValue.value to be the unsampled metric value rather than the sample average.
- Add
EvalResult.get_metric_names(). - Added errors for missing slices during metrics validation.
- Added support for customizing confusion matrix based metrics in keras.
- Made BatchedInputExtractor externally visible.
- Updated tfma.load_eval_results API to return empty results instead of throwing an error when evaluation results are missing for a model_name.
- Fixed an issue in Fairness Indicators UI where omitted slices error message was being displayed even if no slice was omitted.
- Fix issue with slice_spec.is_slice_applicable not working for float, int, etc types that are encoded as strings.
- Wrap long strings in table cells in Fairness Indicators UI.
- Depends on
apache-beam[gcp]>=2.23,<3. - Depends on
pyarrow>=0.17,<0.18. - Depends on
scipy>=1.4.1,<2 - Depends on
tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,<3. - Depends on
tensorflow-metadata>=0.23,<0.24. - Depends on
tfx-bsl>=0.23,<0.24.
- Rename EvalResult.get_slices() to EvalResult.get_slice_names().
- Note: We plan to remove Python 3.5 support after this release.
- Added analyze_raw_data(), an API for evaluating TFMA metrics on Pandas DataFrames.
- Previously metrics would only be computed for combinations of keys that
produced different metric values (e.g.
ExampleCountwill be the same for all models, outputs, classes, etc, so only one metric key was used). Now a metric key will be returned for each combination associated with theMetricSpecdefinition even if the values will be the same. Support for model independent metrics has also been removed. This means by default multiple ExampleCount metrics will be created when multiple models are used (one per model). - Fixed issue with label_key and prediction_key settings not working with TF based metrics.
- Fairness Indicators UI
- Thresholds are now sorted in ascending order.
- Barchart can now be sorted by either slice or eval.
- Added support for slicing on any value extracted from the inputs (e.g. raw labels).
- Added support for filtering extracts based on sub-keys.
- Added beam counters to track the feature slices being used for evaluation.
- Adding KeyError when analyze_raw_data is run without a valid label_key or prediction_key within the provided Pandas DataFrame.
- Added documentation for
tfma.analyze_raw_data,tfma.view.SlicedMetrics, andtfma.view.SlicedPlots. - Unchecked Metric thresholds now block the model validation.
- Added support for per slice threshold settings.
- Added support for sharding metrics and plots outputs.
- Updated load_eval_result to support filtering plots by model name. Added support for loading multiple models at same output path using load_eval_results.
- Fix typo in jupyter widgets breaking TimeSeriesView and PlotViewer.
- Add
tfma.slicer.stringify_slice_key(). - Deprecated external use of tfma.slicer.SingleSliceSpec (tfma.SlicingSpec should be used instead).
- Updated tfma.default_eval_shared_model and tfma.default_extractors to better support custom model types.
- Depends on 'tensorflow-metadata>=0.22.2,<0.23'
- Changed to treat CLASSIFY_OUTPUT_SCORES involving 2 values as a multi-class classification prediction instead of converting to binary classification.
- Refactored confidence interval methodology field. The old path under
Options.confidence_interval_methodologyis now atOptions.confidence_intervals.methodology. - Removed model_load_time_callback from ModelLoader construct_fn (timing is now handled by load). Removed access to shared_handle from ModelLoader.
- Depends on
pyarrow>=0.16,<0.17.
- Added support for jackknife-based confidence intervals.
- Add EvalResult.get_metrics(), which extracts slice metrics in dictionary format from EvalResults.
- Adds TFMD
Schemaas an available argument to computations callbacks.
- Version is now available under
tfma.version.VERSIONortfma.__version__. - Add auto slicing utilities for significance testing.
- Fixed error when a metric and loss with the same classname are used.
- Adding two new ratios (false discovery rate and false omission rate) in Fairness Indicators.
MetricValues can now contain both a debug message and a value (rather than one or the other).- Fix issue with displaying ConfusionMatrixPlot in colab.
CalibrationPlotnow infersleftandrightvalues from schema, when available. This makes the calibration plot useful to regression users.- Fix issue with metrics not being computed properly when mixed with specs containing micro-aggregation computations.
- Remove batched keys. Instead use the same keys for batched and unbatched extract.
- Adding support to visualize Fairness Indicators in Fairness Indicators
TensorBoard Plugin by providing remote evalution path in query parameter:
<tensorboard_url>#fairness_indicators& p.fairness_indicators.evaluation_output_path=<evaluation_path>. - Fixed invalid metrics calculations for serving models using the classification API with binary outputs.
- Moved config writing code to extend from tfma.writer.Writer and made it a member of default_writers.
- Updated tfma.ExtractEvaluateAndWriteResults to accept Extracts as input in addition to serialize bytes and Arrow RecordBatches.
- Depends on
apache-beam[gcp]>=2.20,<3. - Depends on
pyarrow>=0.16,<1. - Depends on
tensorflow>=1.15,!=2.0.*,<3. - Depends on
tensorflow-metadata>=0.22,<0.23. - Depends on
tfx-bsl>=0.22,<0.23.
- Remove desired_batch_size as an option. Large batch failures can be handled via serially processing the failed batch which also acts as a deterent from scaling up batch sizes further. Batch size can be handled via BEAM batch size tuning.
- Deprecating Py2 support.
- Integrate TFXIO in TFMA. Use batched input and predict extractor in V2. Results in ~40% reduction in CPU seconds over existing TFMA v2 (InputExtractor + PredictExtractorV2). Modify TFMA public API to take an optional tensor adapter config as input.
- Populate confidence_interval field in addition to bounded_value when confidence intervals is enabled.
- Only requires
avro-python3>=1.8.1,!=1.9.2.*,<2.0.0on Python 3.5 + MacOS - Fix bug in SensitivitySpecificityBase derived metrics: guarantee well defined behaviour when the constraint lies between feasible points (see updated docstrings).
- Now publish NPM under
tensorflow_model_analysisfor UI components.
- Depends on 'tfx-bsl>=0.21.3,<0.22',
- Depends on 'tensorflow>=1.15,<3',
- Depends on 'apache-beam[gcp]>=2.17,<3',
- Rollback populating TDistributionValue metric when confidence intervals is enabled in V2.
- Added support for creating metrics specs from tf.keras.losses.
- Added evaluation comparison feature to the Fairness Indicators UI in Colab.
- Added better defaults handling for eval config so that a single model spec can be used for both candidate and baseline.
- Added support to provide output file format in load_eval_result API.
- Fixed issue with keras metrics saved with the model not being calculated unless a keras metric was added to the config.
- Depends on
pandas>=0.24,<2. - Depends on
pyarrow>=0.15,<1. - Depends on 'tfx-bsl>=0.21.3,<0.23',
- Depends on 'tensorflow>=1.15,!=2.0.*,<3',
- Depends on 'apache-beam[gcp]>=2.17,<2.18',
- Added support for model validation using either value threshold or diff threshold.
- Added a writer to output model validation result (ValidationResult).
- Added support for multi-model evaluation using EvalSavedModels.
- Added support for inserting model_names by default to metrics_specs.
- Added support for selecting custom model format evals in config.
- Fixed issue with model_name not being set in keras metrics.
- Populate TDistributionValue metric when confidence intervals is enabled in V2.
- Rename the writer MetricsAndPlotsWriter to MetricsPlotsAndValidationsWriter.
- Adding SciPy dependency for both Python2 and Python3
- Increased table and tooltip font in Fairness Indicators.
tfma.BinarizeOptions.class_ids,tfma.BinarizeOptions.k_list,tfma.BinarizeOptions.top_k_list, andtfma.Options.disabled_outputsare now wrapped in an additional proto message.
- Adding a TFLite predict extractor to enable obtaining inferences from TFLite models.
- Adding support to compute deterministic confidence intervals using a seed value in tfma.run_model_analysis API for testing or experimental purposes.
- Fixed calculation of
tfma.metrics.CoefficientOfDiscriminationandtfma.metrics.RelativeCoefficientOfDiscrimination.
- Renaming k_anonymization_count field name to min_slice_size.
- Added
tfma.metrics.MinLabelPositionandtfma.metrics.QueryStatisticsfor use with V2 metrics API. - Added
tfma.metrics.CoefficientOfDiscriminationandtfma.metrics.RelativeCoefficientOfDiscriminationfor use with V2 metrics API. - Added support for using
tf.keras.metrics.*metrics with V2 metrics API. - Added support for default V2 MetricSpecs and creating specs from
tf.kera.metrics.*andtfma.metrics.*metric classes. - Added new MetricsAndPlotsEvaluator based on V2 infrastructure. Note this evaluator also supports query-based metrics.
- Add support for micro_average, macro_average, and weighted_macro_average metrics.
- Added support for running V2 extractors and evaluators. V2 extractors will
be used whenever the default_eval_saved_model is created using a non-eval
tag (e.g.
tf.saved_model.SERVING). The V2 evaluator will be used whenever atfma.EvalConfigis used containingmetrics_specs. - Added support for
tfma.metrics.SquaredPearsonCorrelationfor use with V2 metrics API. - Improved support for TPU autoscaling and handling batch_size related scaling.
- Added support for
tfma.metrics.Specificity,tfma.metrics.FallOut, andtfma.metrics.MissRatefor use with V2 metrics API. RenamedAUCPlottoConfusionMatrixPlot,MultiClassConfusionMatrixAtThresholdstoMultiClassConfusionMatrixPlotandMultiLabelConfusionMatrixAtThresholdstoMultiLabelConfusionMatrixPlot. - Added Jupyter support to Fairness Indicators. Currently does not support WIT integration.
- Added fairness indicators metrics
tfma.addons.fairness.metrics.FairnessIndicators. - Updated documentation for new metrics infrastructure and newly supported models (keras, etc).
- Added support for model diff metrics. Users need to turn on "is_baseline" in the corresponding ModelSpec.
- Fixed error in
tfma-multi-class-confusion-matrix-at-thresholdswith default classNames value. - Fairness Indicators
- Compute ratio metrics with safe division.
- Remove "post_export_metrics" from metric names.
- Move threshold dropdown selector to a metric-by-metric basis, allowing different metrics to be inspected with different thresholds. Don't show thresholds for metrics that do not support them.
- Slices are now displayed in alphabetic order.
- Adding an option to "Select all" metrics in UI.
- Added auto slice key extractor based on statistics.
- Depends on 'tensorflow-metadata>=0.21,<0.22'.
- Made InputProcessor externally visible.
- Updated proto config to remove input/output data specs in favor of passing them directly to the run_eval.
- Fixed the bug that Fairness Indicator will skip metrics with NaN value.
- Updated vulcanized_tfma.js with UI changes in addons/fairness_indicators.
- Updated to use tf.io.gfile for reading config files (fixes issue with reading from GCS/HDFS in 0.15.0 and 0.15.1 releases).
- Added support for defaulting to using class IDs when classes are not present in outputs for multi-class metrics (for use in keras model_to_estimator).
- Added example count metrics (
tfma.metrics.ExampleCountandtfma.metrics.WeightedExampleCount) for use with V2 metrics API. - Added calibration metrics (
tfma.metrics.MeanLabel,tfma.metrics.MeanPrediction, andtfma.metrics.Calibration) for use with V2 metrics API. - Added
tfma.metrics.ConfusionMatrixAtThresholdsfor use with V2 metrics API. - Added
tfma.metrics.CalibrationPlotandtfma.metrics.AUCPlotfor use with V2 metrics API. - Added multi_class / multi_label plots (
tfma.metrics.MultiClassConfusionMatrixAtThresholds,tfma.metrics.MultiLabelConfusionMatrixAtThresholds) for use with V2 metrics API. - Added
tfma.metrics.NDCGmetric for use with V2 metrics API. - Added
calibrationas a post export metric.
- Depends on
tensorflow>=1.15,<3.0.- Starting from 1.15, package
tensorflowcomes with GPU support. Users won't need to choose betweentensorflowandtensorflow-gpu. - Caveat:
tensorflow2.0.0 is an exception and does not have GPU support. Iftensorflow-gpu2.0.0 is installed before installingtensorflow_model_analysis, it will be replaced withtensorflow2.0.0. Re-installtensorflow-gpu2.0.0 if needed.
- Starting from 1.15, package
- Added V2 of PredictExtractor that uses TF 2.0 signature APIs and supports keras models (note: keras model evaluation not fully supported yet).
tfma.run_model_analysis,tfma.default_extractors,tfma.default_evaluators, andtfma.default_writersnow allow settings to be passed as anEvalConfig.tfma.run_model_analysis,tfma.default_extractors,tfma.default_evaluators, andtfma.default_writersnow allow multiple models to be passed (note: multi-model support not fully implemented yet).- Added InputExtractor for extracting labels, features, and example weights from tf.Examples.
- Added Fairness Indicator as an addon.
- Enabled TF 2.0 support using compat.v1.
- Added support for slicing on native dicts of features in addition to FPL types.
- For multi-output and / or multi-class models, please provide output_name and / or class_id to tfma.view.render_plot.
- Replaced dependency on
tensorflow-transformwithtfx-bsl. If running with latest master,tfx-bslmust also be latest master. - Depends on
tfx-bsl>=0.15,<0.16. - Slicing now supports conversion between int/floats and strings.
- Depends on
apache-beam[gcp]>=2.16,<3. - Depends on
six==1.12.
- tfma.EvalResult.slicing_metrics now contains nested dictionaries of output, class id and then metric names.
- Update config serialization to use JSON instead of pickling and reformat config to include input_data_specs, model_specs, output_data_specs, and metrics_specs.
- Requires pre-installed TensorFlow >=1.15,<3.
- Added documentation on architecture.
- Added an
adapt_to_remove_metricsfunction totfma.exporterwhich can be used to remove metrics incompatible with TFMA (e.g.py_funcor streaming metrics) before exporting the TFMA EvalSavedModel. - Added support for passing sparse int64 tensors to precision/recall@k.
- Added support for binarization of multiclass metrics that use labels of the from (N) in addition to (N, 1).
- Added support for using iterators with EvalInputReceiver.
- Improved performance of confidence interval computations by modifying the pipeline shape.
- Added QueryBasedMetricsEvaluator which supports computing query-based metrics (e.g. normalized discounted cumulative gain).
- Added support for merging metrics produced by different evaluators.
- Added support for blacklisting specified features from fetches.
- Added functionality to the FeatureExtractor to specify the features dict as a possible destination.
- Added support for label vocabularies for binary and multi-class estimators that support the new ALL_CLASSES prediction output.
- Move example parsing in aggregation into the graph for performance improvements in both standard and model_agnostic evaluation modes.
- Created separate ModelLoader type for loading the EvalSavedModel.
- Upgraded codebase for TF 2.0 compatibility.
- Make metrics-related operations thread-safe by wrapping them with locks. This eliminates race conditions that were previously possible in multi-threaded runners which could result in incorrect metric values.
- More flexible
FanoutSlices. - Limit the number of sampling buckets to 20.
- Improved performance in Confidence Interval computation.
- Refactored poisson bootstrap code to be re-usable in other evaluators.
- Refactored k-anonymity code to be re-usable in other evaluators.
- Fixed slicer feature string value handling in Python3.
- Added support for example weight keys for multi-output models.
- Added option to set the desired batch size when calling run_model_analysis.
- Changed TFRecord compression type from UNCOMPRESSED to AUTO.
- Depends on
apache-beam[gcp]>=2.14,<3. - Depends on
numpy>=1.16,<2. - Depends on
protobuf>=3.7,<4. - Depends on
scipy==1.1.0. - Added support to change k_anonymization_count value via EvalConfig.
- Removed uses of deprecated tf.contrib packages (where possible).
tfma.default_writersnow requires theeval_saved_modelto be passed as an argument.- Requires pre-installed TensorFlow >=1.14,<2.
- Added support for squared pearson correlation (R squared) post export metric.
- Added support for mean absolute error post export metric.
- Added support for mean squared error and root mean squared error post export metric.
- Added support for not computing metrics for slices with less than a given number of examples.
- Cast / convert labels for precision / recall at K so that they work even if the label and the classes Tensors have different types, as long as the types are compatible.
- Post export metrics will now also search for prediction keys prefixed by metric_tag if it is specified.
- Added support for precision/recall @ k using canned estimators provided label vocab not used.
- Preserve unicode type of slice keys when serialising to and deserialising from disk, instead of always converting them to bytes.
- Use
__slots__in accumulators.
- Expose Python 3 types in the code (this will break Python 2 compatibility)
- Python 3.5 is supported.
- Added support for fetching additional tensors at prediction time besides features, predictions, and labels (predict now returns FetchedTensorValues type).
- Removed internal usages of encoding.NODE_SUFFIX indirection within dicts in the eval_saved_model module (encoding.NODE_SUFFIX is still used in FeaturesPredictionLabels)
- Predictions are now returned as tensors (vs dicts) when "predictions" is the only output (this is consistent with how features and labels work).
- Depends on
apache-beam[gcp]>=2.11,<3. - Depends on
protobuf>=3.7,<4. - Depends on
scipy==1.1.0. - Add support for multiple plots in a single evaluation.
- Add support for changeable confidence levels.
- Post export metrics for precision_recall_at_k were split into separate fuctions: precision_at_k and recall_at_k.
- Requires pre-installed TensorFlow >=1.13,<2.
- Python 3.5 readiness complete (all tests pass). Full Python 3.5 compatibility is expected to be available with the next version of Model Analysis (after Apache Beam 2.11 is released).
- Added support for customizing the pipeline (via extractors, evaluators, and writers). See architecture for more details.
- Added support for excluding the default metrics from the saved model graph during evaluation.
- Added a mechanism for performing evaluations via post_export_metrics without access to a Tensorflow EvalSavedModel.
- Added support for computing metrics with confidence intervals using the Poisson bootstrap technique. To use, set the num_bootstrap_samples to a number greater than 1--20 is recommended for confidence intervals.
- Fixed bugs where TFMA was incorrectly modifying elements in DoFns, which violates the Beam API.
- Fixed correctness issue stemming from TFMA incorrectly relying on evaluation ordering that TF doesn't guarantee.
- We now store feature and label Tensor information in SignatureDef inputs instead of Collections in anticipation of Collections being deprecated in TF 2.0.
- Add support for fractional labels in AUC, AUPRC and confusion matrix at
thresholds. Previously the labels were being passed directly to TensorFlow,
which would cast them to
bool, which meant that all non-zero labels were treated as positive examples. Now we treat a fractional labellin[0, 1]as two examples, a positive example with weightland a negative example with weight1 - l. - Depends on
numpy>=1.14.5,<2. - Depends on
scipy==0.19.1. - Depends on
protobuf==3.7.0rc2. - Chicago Taxi example is moved to tfx repo (https://github.com/tensorflow/tfx/tree/master/tfx/examples/chicago_taxi)
- Moved tfma.SingleSliceSpec to tfma.slicer.SingleSliceSpec.
- We now support unsupervised models which have
model_fns that do not take alabelsargument. - Improved performance by using
make_callableinstead of repeatedsession.runcalls. - Improved performance by better choice of default "combine" batch size.
- We now support passing in custom extractors in the model_eval_lib API.
- Added support for models which have multiple examples per raw input (e.g.
input is a compressed example which expands to multiple examples when parsed
by the model). For such models, you must specify an
example_refparameter to yourEvalInputReceiver. This 1-D integer Tensor should be batch aligned with features, predictions and labels and each element in it is an index in the raw input tensor to identify which input each feature / prediction / label came from. Seeeval_saved_model/example_trainers/fake_multi_examples_per_input_estimator.pyfor an example. - Added support for metrics with string
value_ops. - Added support for metrics whose
value_ops return multidimensional arrays. - We now support including your serving graph in the EvalSavedModel. You can
do this by passing a
serving_input_receiver_fntoexport_eval_savedmodelor any of the TFMA Exporters. - We now support customizing prediction and label keys for post_export_metrics.
- Depends on
apache-beam[gcp]>=2.8,<3. - Depends on
tensorflow-transform>=0.11,<1. - Requires pre-installed TensorFlow >=1.11,<2.
- Factor our utility functions for adding sliceable "meta-features" to FPL.
- Added public API docs
- Add an extractor to add sliceable "meta-features" to FPL.
- Potentially improved performance by fanning out large slices.
- Add support for assets_extra in
tfma.exporter.FinalExporter. - Add a light-weight library that includes only the export-related modules for
TFMA for use in your Trainer. See docstring in
tensorflow_model_analysis/export_only/__init__.pyfor usage. - Update
EvalInputReceiverso the TFMA collections written to the graph only contain the results of the last call if multiple calls toEvalInputReceiverare made. - We now finalize the graph after it's loaded and post-export metrics are added, potentially improving performance.
- Fix a bug in post-export PrecisionRecallAtK where labels with only 1 dimension were not correctly handled.
- Fix an issue where we were not correctly wrapping SparseTensors for
featuresandlabelsintf.identity, which could cause TFMA to encounter TensorFlow issue #17568 if there were control dependencies on thesefeaturesorlabels. - We now correctly preserve
dtypeswhen splitting and concatenating SparseTensors internally. The failure to do so previously could result in unexpectedly large memory usage if string values were involved due to the inefficient pickling of NumPy string arrays with a large number of elements.
- Requires pre-installed TensorFlow >=1.11,<2.
- We now require that
EvalInputReceiver,export_eval_savedmodel,make_export_strategy,make_final_exporter,FinalExporterandLatestExporterbe called with keyword arguments only. - Removed
check_metric_compatibilityfromEvalSavedModel. - We now enforce that the
receiver_tensorsdictionary forEvalInputReceivercontains exactly one key namedexamples. - Post-export metrics have now been moved up one level to
tfma.post_export_metrics. They should now be accessed viatfma.post_export_metrics.aucinstead oftfma.post_export_metrics.post_export_metrics.aucas they were before. - Separated extraction from evaluation.
EvaluteAndWriteResultsis now calledExtractEvaluateAndWriteResults. - Added
EvalSharedModeltype to encapsulatemodel_pathandadd_metrics_callbacksalong with a handle to a shared model instance.
- Improved performance especially when slicing across many features and/or feature values.
- Depends on
tensorflow-transform>=0.9,<1. - Requires pre-installed TensorFlow >=1.9,<2.
- Depends on
apache-beam[gcp]>=2.6,<3. - Updated ExampleCount to use the batch dimension as the example count. It also now tries a few fallbacks if none of the standard keys are found in the predictions dictionary: the first key in sorted order in the predictions dictionary, or failing that, the first key in sorted order in the labels dictionary, or failing that, it defaults to zero.
- Fix bug where we were mutating an element in a DoFn - this is prohibited in the Beam model and can cause subtle bugs.
- Fix bug where we were creating a separate Shared handle for each stage in Evaluate, resulting in no sharing of the model across stages.
- Requires pre-installed TensorFlow >=1.10,<2.
- Add a TFMA unit test library for unit testing your the exported model and associated metrics computations.
- Add
tfma.export.make_export_strategywhich is analogous totf.contrib.learn.make_export_strategy. - Add
tfma.exporter.FinalExporterandtfma.exporter.LatestExporterwhich are analogous totf.estimator.FinalExporterandtf.estimator.LastExporter. - Add
tfma.export.build_parsing_eval_input_receiver_fnwhich is analogous totf.estimator.export.build_parsing_serving_input_receiver_fn. - Add integration testing for DNN-based estimators.
- Add new post export metrics:
- AUC (
tfma.post_export_metrics.post_export_metrics.auc) - Precision/Recall at K
(
tfma.post_export_metrics.post_export_metrics.precision_recall_at_k) - Confusion matrix at thresholds
(
tfma.post_export_metrics.post_export_metrics.confusion_matrix_at_thresholds)
- AUC (
- Peak memory usage for large DataFlow jobs should be lower with a fix in when we compact batches of metrics during the combine phase of metrics computation.
- Remove batch size override in
chicago_taxiexample. - Added dependency on
protobuf>=3.6.0<4for protocol buffers. - Updated SparseTensor code to work with SparseTensors of any dimension. Previously on SparseTensors with dimension 2 (batch_size x values) were supported in the features dictionary.
- Updated code to work with SparseTensors and dense Tensors of variable lengths across batches.
- EvalSavedModels produced by TFMA 0.6.0 will not be compatible with later
versions due to the following changes:
- EvalSavedModels are now written out with a custom "eval_saved_model" tag, as opposed to the "serving" tag before.
- EvalSavedModels now include version metadata about the TFMA version that they were exported with.
- Metrics and plot outputs now are converted into proto and serialized. Metrics and plots produced by TFMA 0.6.0 will not be compatible with later versions.
- Requires pre-installed TensorFlow >=1.9,<2.
- TFMA now uses the TensorFlow Estimator functionality for exporting models of different modes behind the scenes. There are no user-facing changes API-wise, but EvalSavedModels produced by earlier versions of TFMA will not be compatible with this version of TFMA.
- tf.contrib.learn Estimators are no longer supported by TFMA. Only tf.estimator Estimators are supported.
- Metrics and plot outputs now include version metadata about the TFMA version that they were exported with. Metrics and plots produced by earlier versions of TFMA will not be compatible with this version of TFMA.
- Initial release of TensorFlow Model Analysis.