I want to convert a text classification sample from ML 0.4 to ML 0.11 and cannot find equivalent code for calling Transforms.Text.FeaturizeText

[bpietroiu]

System information

• OS version/distro: Win10
• .NET Version (eg., dotnet --info): 4.7.1

Issue

• What did you do?
  I want to convert a text classification sample from ML 0.4 to ML 0.11

• What happened?
  I cannot find equivalent parameters for TextFeaturizer, there are no CharFeatureExtractor and WordFeatureExtractor NgramExtractors in 0.11 Transforms.Text.FeaturizeText arguments

• What did you expect?

Source code / logs

0.4 version

                new TextFeaturizer("Features", "TextColumn")
                {
                    KeepDiacritics = false,
                    KeepPunctuations = false,
                    OutputTokens = true,
                    Language = TextFeaturizingEstimatorLanguage.English,
                    VectorNormalizer = TextFeaturizingEstimatorTextNormKind.L2,
                    TextCase = TextNormalizingEstimatorCaseNormalizationMode.Lower,
                    CharFeatureExtractor = new NGramNgramExtractor() {NgramLength = 3, AllLengths = false},
                    WordFeatureExtractor = new NGramNgramExtractor() {NgramLength = 3, AllLengths = true}
                }, 

0.11 version

var transform mlContext.Transforms.Text.FeaturizeText(outputColumnName: "TextColumnFeaturized",
                        options: new TextFeaturizingEstimator.Options()
                        {
                            KeepDiacritics = false,
                            KeepPunctuations = false,
                            OutputTokens = true,
                            TextLanguage = TextFeaturizingEstimator.Language.English,
                            VectorNormalizer = TextFeaturizingEstimator.TextNormKind.L2,
                            TextCase = TextNormalizingEstimator.CaseNormalizationMode.Lower,
                            UseWordExtractor = true,
                            UseCharExtractor = true
                        }, inputColumnNames: new []{ "TextColumn" });

I see that CharFeatureExtractor and WordFeatureExtractor parameters are gone, and instead two we have boolean properties, UseWordExtractor and UseCharExtractor.

The models trained using the above code and the same train data perform differently, namly the 0.4 version has a better classsification performace.

How do I configure a pipeline to achieve the same results as in 0.4 version?

Thank you!

[singlis]

Hi @bpietroiu,

You are correct -- there is no way to do this in 0.11. We have since filed a bug #2802, along with a fix that was merged a few days ago (#2911). This will be available in our next release and should provide the same functionality that you are looking for.

[antoniovs1029]

Hi, so it seems this problem was fixed in a previous release. Please, feel free to reopen the issue if you're still having problems with this. Thanks!