Adding transform extensions #1460

sfilipi · 2018-10-30T22:24:09Z

Addresses part of #1318 adding xtensions for all text related estimators.

… 1318

…alizer

shmoradims · 2018-10-31T14:01:33Z

src/Microsoft.ML.Data/Transforms/TransformsExtensionsCatalog.cs

+        /// <param name="columnsToDrop">The array of column names to drop.</param>
+        public static ColumnSelectingEstimator DropColumns(this TransformsCatalog catalog, params string[] columnsToDrop)
+        {
+            return new ColumnSelectingEstimator(CatalogUtils.GetEnvironment(catalog), null, columnsToDrop);


null, columnsToDrop [](start = 86, length = 19)

it would be more readable to use named arguments here and above, b/c ColumnSelectingEstimator takes so many optional arguments. #Resolved

shmoradims · 2018-10-31T14:03:43Z

src/Microsoft.ML.ImageAnalytics/ExtensionsCatalog.cs

+        /// Converts the images to grayscale.
+        /// </summary>
+        /// <param name="catalog">The transform's catalog.</param>
+        /// <param name="columns">The name of the columns containing the image paths(first item of the tuple), and the name of the resulting output column (second item of the tuple).</param>


sulting output column (second item of the tuple). [](start = 133, length = 57)

nit: What is our max line length guideline? I keep scrolling left/right to see the whole thing :) #Resolved

shmoradims · 2018-10-31T14:07:06Z

src/Microsoft.ML.ImageAnalytics/ExtensionsCatalog.cs

+        /// </summary>
+        /// <param name="catalog">The transform's catalog.</param>
+        /// <param name="columns">The name of the columns containing the image paths(first item of the tuple), and the name of the resulting output column (second item of the tuple).</param>
+        public static ImageGrayscalingEstimator Grayscale(this TransformsCatalog catalog, params (string input, string output)[] columns)


(string input, string output)[] columns [](start = 96, length = 40)

how come some extensions are multi input/output with tuples, some are single input/output, and some are multi input/output with ColumnInfo? #Pending

yeah, one per constructor; they follow the ctor signatures.

In reply to: 229707525 [](ancestors = 229707525)

shmoradims · 2018-10-31T14:12:54Z

src/Microsoft.ML.Transforms/CategoricalsCatalog.cs

+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, params KeyToBinaryVectorTransform.ColumnInfo[] columns)
+        => new KeyToBinaryVectorMappingEstimator(CatalogUtils.GetEnvironment(catalog), columns);
+
+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, string name, string source = null)


string name, string source = null [](start = 125, length = 33)

Did we change the naming convention from name/source to inputColumn/outputColumn? Or name/source is still valid?

Also they're sometimes called input/output.

Also I noticed, that some some of the extensions have output = null, as optional and some don't.

If it's too much for this PR, please add an issue for reviewing the full API surface for parameter consistency and mention all the different examples.
#Resolved

We are going with inputColumn outputColumn. I'll scan for inconsistencies. Thanks for pointing out.

In reply to: 229710031 [](ancestors = 229710031)

So there are a number of places where we have source->name usage (outside of this PR). The issue should address all instances right?

shmoradims · 2018-10-31T14:18:07Z

test/Microsoft.ML.OnnxTransformTest/OnnxTransformTests.cs

@@ -82,7 +82,7 @@ void TestSimpleCase()
            var xyData = new List<TestDataXY> { new TestDataXY() { A = new float[inputSize] } };
            var stringData = new List<TestDataDifferntType> { new TestDataDifferntType() { data_0 = new string[inputSize] } };
            var sizeData = new List<TestDataSize> { new TestDataSize() { data_0 = new float[2] } };
-            var pipe = new OnnxEstimator(Env, modelFile, "data_0", "softmaxout_1");
+            var pipe = new OnnxConvertingEstimator(Env, modelFile, "data_0", "softmaxout_1");


Converting [](start = 31, length = 10)

OnnxScoringEstimator could be a better name. #Resolved

shmoradims

singlis · 2018-10-31T16:18:21Z

src/Microsoft.ML.Data/Transforms/TransformsExtensionsCatalog.cs

+        }
+
+        /// <summary>
+        /// Constructs the Select Columns Estimator.


Columns Selecting Estimator #Resolved

Zruty0 · 2018-10-31T16:36:13Z

src/Microsoft.ML.Data/Transforms/CategoricalsCatalog.cs

+        /// </summary>
+        /// <param name="catalog">The categorical transform's catalog.</param>
+        /// <param name="columns">The input column to map back to vectors.</param>
+        public static KeyToVectorMappingEstimator ToVector(this TransformsCatalog.CategoricalTransforms catalog,


CategoricalTransforms [](start = 82, length = 21)

Actually I was expecting ToKey, ToVector, ToBinaryVector and ToValue to be in Conversions, not Categorical. In any case, let's not shorten it too much: MapKeyToVector, MapValueToKey etc. would be better I think #Pending

would the respective Estimator live in the Conversions namespace?

In reply to: 229773742 [](ancestors = 229773742)

Personally I would vote to keep them in the Categorical, if the opposite operation (Term) is there.

In reply to: 229779270 [](ancestors = 229779270,229773742)

Zruty0 · 2018-10-31T16:48:16Z

src/Microsoft.ML.Transforms/ExtensionsCatalog.cs

+        /// </summary>
+        /// <param name="catalog">The categorical transform's catalog.</param>
+        /// <param name="columns">The names of the input columns of the transformation and the corresponding names for the output columns.</param>
+        public static MissingValueIndicatorEstimator IndicateMissingValues(this TransformsCatalog catalog, params (string inputColumn, string outputColumn)[] columns)


Indicate [](start = 53, length = 8)

'indicate' seems weird. I don't have a better suggestion though... #Resolved

that should be fine. sklearn uses the same terminology: http://scikit-learn.org/stable/modules/generated/sklearn.impute.MissingIndicator.html#sklearn.impute.MissingIndicator

In reply to: 229778482 [](ancestors = 229778482)

Zruty0 · 2018-10-31T16:48:42Z

src/Microsoft.ML.Transforms/CategoricalsCatalog.cs

+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, params KeyToBinaryVectorTransform.ColumnInfo[] columns)
+        => new KeyToBinaryVectorMappingEstimator(CatalogUtils.GetEnvironment(catalog), columns);
+
+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, string name, string source = null)


ToBinaryVector [](start = 56, length = 14)

Summary #Resolved

Zruty0 · 2018-10-31T16:49:36Z

src/Microsoft.ML.Transforms/TextCatalog.cs

+        /// <param name="catalog">The text-related transform's catalog.</param>
+        /// <param name="columns">Pairs of columns to run the tokenization on.</param>
+        /// <param name="separators">The separators to use (uses space character by default).</param>
+        public static WordTokenizingEstimator TokenizeWord(this TransformsCatalog.TextTransforms catalog,


TokenizeWord [](start = 46, length = 12)

TokenizeWords #Resolved

Zruty0 · 2018-10-31T16:50:33Z

public static class ColumnCopyingCatalog

why not merge all the static catalog classes into one? I think we agreed to have one per assembly #Pending

Refers to: src/Microsoft.ML.Data/Transforms/TransformsExtensionsCatalog.cs:14 in 0b736b2. [](commit_id = 0b736b2, deletion_comment = False)

Zruty0

sfilipi · 2018-10-31T16:55:23Z

public static class ColumnCopyingCatalog

I did one per category in the two assemblies that gather the most of the transforms. That ok?

In reply to: 434763291 [](ancestors = 434763291)

Refers to: src/Microsoft.ML.Data/Transforms/TransformsExtensionsCatalog.cs:14 in 0b736b2. [](commit_id = 0b736b2, deletion_comment = False)

singlis · 2018-10-31T17:17:34Z

src/Microsoft.ML.ImageAnalytics/ExtensionsCatalog.cs

+        /// <param name="colors">The color schema as defined in <see cref="ImagePixelExtractorTransform.ColorBits"/>.</param>
+        /// <param name="interleave"></param>
+        public static ImagePixelExtractingEstimator ExtractPixels(this TransformsCatalog catalog, string inputColumn, string outputColumn,
+        ImagePixelExtractorTransform.ColorBits colors = ImagePixelExtractorTransform.ColorBits.Rgb, bool interleave = false)


singlis · 2018-10-31T17:17:51Z

src/Microsoft.ML.ImageAnalytics/ExtensionsCatalog.cs

+        /// <param name="cropAnchor">Where to place the anchor, to start cropping. Options defined in <see cref="ImageResizerTransform.Anchor"/></param>
+        /// <returns></returns>
+        public static ImageResizingEstimator Resize(this TransformsCatalog catalog, string inputColumn, string outputColumn,
+        int imageWidth, int imageHeight, ImageResizerTransform.ResizingKind resizing = ImageResizerTransform.ResizingKind.IsoCrop, ImageResizerTransform.Anchor cropAnchor = ImageResizerTransform.Anchor.Center)


tabbing here as well?

singlis · 2018-10-31T17:18:10Z

src/Microsoft.ML.ImageAnalytics/ImageGrayscaleTransform.cs

+        /// </summary>
+        /// <param name="env">The estimator's local <see cref="IHostEnvironment"/>.</param>
+        /// <param name="columns">The name of the columns containing the image paths(first item of the tuple), and the name of the resulting output column (second item of the tuple).</param>
+


delete line

singlis · 2018-10-31T17:21:27Z

src/Microsoft.ML.Transforms/CategoricalsCatalog.cs

+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, params KeyToBinaryVectorTransform.ColumnInfo[] columns)
+        => new KeyToBinaryVectorMappingEstimator(CatalogUtils.GetEnvironment(catalog), columns);
+
+        public static KeyToBinaryVectorMappingEstimator ToBinaryVector(this TransformsCatalog.CategoricalTransforms catalog, string name, string source = null)


So there are a number of places where we have source->name usage (outside of this PR). The issue should address all instances right?

singlis · 2018-10-31T17:26:39Z

src/Microsoft.ML.Data/Transforms/KeyToVectorTransform.cs

@@ -84,7 +84,7 @@ public sealed class Arguments

            [Argument(ArgumentType.AtMostOnce,
                HelpText = "Whether to combine multiple indicator vectors into a single bag vector instead of concatenating them. This is only relevant when the input is a vector.")]
-            public bool Bag = KeyToVectorEstimator.Defaults.Bag;
+            public bool Bag = KeyToVectorMappingEstimator.Defaults.Bag;


This is only part of the work right? I.e. the transforms will also be updated to KeyToVectorMappingTransform?

adding all extensions for the Text related transformation

bd7c85d

sfilipi requested review from Zruty0, Ivanidzo4ka, artidoro and shmoradims October 30, 2018 22:24

sfilipi added 6 commits October 30, 2018 22:30

Merge branch 'master' into addingTransformXtensions

d27a084

adding keyToVector extensions and renaming the estimatorto conform to…

470230e

… 1318

Adding SelectColumns xtensions. Renaming SelectEstimator and Textnorm…

6bd31dc

…alizer

KeyToBinary extensions

7494b19

Adding extensions for the Image transforms. Some renaming

017286c

regenerate csharp api

0b736b2

sfilipi changed the title ~~Adding all extensions for the Text related transformation~~ Adding transform extensions Oct 31, 2018

sfilipi self-assigned this Oct 31, 2018

sfilipi added the API Issues pertaining the friendly API label Oct 31, 2018

sfilipi requested a review from singlis October 31, 2018 07:33

shmoradims reviewed Oct 31, 2018

View reviewed changes

shmoradims approved these changes Oct 31, 2018

View reviewed changes

singlis reviewed Oct 31, 2018

View reviewed changes

Zruty0 reviewed Oct 31, 2018

View reviewed changes

Zruty0 approved these changes Oct 31, 2018

View reviewed changes

Addressing PR comments.

849c54a

singlis reviewed Oct 31, 2018

View reviewed changes

merging from master

e299b80

singlis approved these changes Oct 31, 2018

View reviewed changes

sfilipi merged commit 9d33efe into dotnet:master Oct 31, 2018

sfilipi deleted the addingTransformXtensions branch October 31, 2018 18:05

ghost locked as resolved and limited conversation to collaborators Mar 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding transform extensions #1460

Adding transform extensions #1460

sfilipi commented Oct 30, 2018

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

sfilipi Oct 31, 2018

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

sfilipi Oct 31, 2018 •

edited

Loading

singlis Oct 31, 2018

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims left a comment

singlis Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

sfilipi Oct 31, 2018

sfilipi Oct 31, 2018

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 commented Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 left a comment

sfilipi commented Oct 31, 2018

singlis Oct 31, 2018

singlis Oct 31, 2018

singlis Oct 31, 2018

singlis Oct 31, 2018

singlis Oct 31, 2018

Adding transform extensions #1460

Adding transform extensions #1460

Conversation

sfilipi commented Oct 30, 2018

shmoradims Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

sfilipi Oct 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims left a comment

Choose a reason for hiding this comment

singlis Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Zruty0 Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Zruty0 Oct 31, 2018 • edited by sfilipi Loading

Choose a reason for hiding this comment

Zruty0 commented Oct 31, 2018 • edited by sfilipi Loading

Zruty0 left a comment

Choose a reason for hiding this comment

sfilipi commented Oct 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

sfilipi Oct 31, 2018 •

edited

Loading

shmoradims Oct 31, 2018 •

edited by sfilipi

Loading

singlis Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 Oct 31, 2018 •

edited by sfilipi

Loading

Zruty0 commented Oct 31, 2018 •

edited by sfilipi

Loading