Convert LdaTransform to IEstimator/ITransformer API #1410

abgoswam · 2018-10-27T03:44:13Z

This PR converts LdaTransform to the IEstimator/ITransformer API paradigm (a work item related to #754)

LdaTransform is now subclass of OneToOneTransformerBase
Deleted the previous wrapped version.
Added pigsty extensions
Added onFit delegate to return LdaState info to user

Created separate issue #1411 for providing a convenient way of providing LDA topic summary info to the user

…digm

Ivanidzo4ka · 2018-10-29T17:56:23Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

@@ -17,12 +17,19 @@
 using Microsoft.ML.Runtime.Internal.Utilities;
 using Microsoft.ML.Runtime.Model;
 using Microsoft.ML.Runtime.TextAnalytics;
+using Microsoft.ML.Core.Data;


Ctrl+R+G (aka sort using) #Closed

Ivanidzo4ka · 2018-10-31T17:52:29Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+
+            // The type T asociated with the delegate will be the actual value type once #863 goes in.
+            // However, until such time as #863 goes in, it would be too awkward to attempt to extract the metadata.
+            // For now construct the useless object then pass it into the delegate.


this comment doesn't seems right. #Resolved

Ivanidzo4ka · 2018-10-31T18:18:25Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+    "Latent Dirichlet Allocation Transform", LdaTransform.LoaderSignature)]
+
+[assembly: LoadableClass(typeof(IRowMapper), typeof(LdaTransform), null, typeof(SignatureLoadRowMapper),
+    "Latent Dirichlet Allocation Transform", LdaTransform.LoaderSignature)]

 namespace Microsoft.ML.Runtime.TextAnalytics


Runtime [](start = 23, length = 7)

Can you merge with master, I'm sure Senja changed this namespace to something better. #Resolved

Ivanidzo4ka · 2018-10-31T18:20:39Z

test/Microsoft.ML.TestFramework/DataPipe/TestDataPipe.cs

@@ -646,7 +646,7 @@ public void TestLdaTransformEmptyDocumentException()

            try
            {
-                var lda = new LdaTransform(Env, args, srcView);
+                var lda = LdaTransform.Create(Env, args, srcView);


LdaTransform [](start = 26, length = 12)

Can you switch to estimator call?
Basically var lda = LdaEstimator(env, cols).Fit(srcView).Transform(scrView); #Resolved

Ivanidzo4ka · 2018-10-31T18:22:50Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        /// <param name="numSummaryTermPerTopic">The number of words to summarize the topic</param>
+        /// <param name="numBurninIterations">The number of burn-in iterations</param>
+        /// <param name="resetRandomGenerator">Reset the random number generator for each document</param>
+        /// <param name="onFit">Called upon fitting with the learnt enumeration on the dataset.</param>


Sprinkle it with dots at the end of sentence. #Resolved

abgoswam · 2018-11-05T20:25:44Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+                Contracts.Assert(0 <= iinfo && iinfo < _parent.ColumnPairs.Length);
+                disposer = null;
+
+                var test = TestType(_srcTypes[iinfo]);


TestType [](start = 27, length = 8)

@ivan. Should we put this check here v/s putting it in the Mapper constructor ? Does it make a difference ? #Closed

It make difference at what point it will fail. I would prefer to test types in constructor.

In reply to: 230899143 [](ancestors = 230899143)

I prefer not to have TestType idiom please

In reply to: 232105072 [](ancestors = 232105072,230899143)

if column type in incorrect we now throw ExceptSchemaMismatch

In reply to: 232437787 [](ancestors = 232437787,232105072,230899143)

wschin · 2018-11-06T22:50:42Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        private struct Config
+        {
+            public readonly int NumTopic;
+            public readonly Single AlphaSum;


Suggested change

public readonly Single AlphaSum;

public readonly float AlphaSum;

``` #Resolved

wschin · 2018-11-06T22:50:53Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        {
+            public readonly int NumTopic;
+            public readonly Single AlphaSum;
+            public readonly Single Beta;


Suggested change

public readonly Single Beta;

public readonly float Beta;

``` #Resolved

artidoro · 2018-11-06T22:51:07Z

test/Microsoft.ML.StaticPipelineTesting/StaticPipeTests.cs

@@ -22,6 +22,7 @@
 using System.Text;
 using Xunit;
 using Xunit.Abstractions;
+using static Microsoft.ML.Transforms.Text.LdaTransform;


Sort the usings. #Resolved

Control+R Control+G (remove and sort Using) gives this order itself

In reply to: 231324047 [](ancestors = 231324047)

wschin · 2018-11-06T22:53:20Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        }
+
+        /// <include file='doc.xml' path='doc/members/member[@name="LightLDA"]/*' />
+        /// <param name="input">The column to apply to.</param>


Could you describe what this transform is doing briefly? #Resolved

wschin · 2018-11-06T22:53:29Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        /// <param name="onFit">Called upon fitting with the learnt enumeration on the dataset.</param>
+        public static Vector<float> ToLdaTopicVector(this Vector<float> input,
+            int numTopic = LdaEstimator.Defaults.NumTopic,
+            Single alphaSum = LdaEstimator.Defaults.AlphaSum,


Suggested change

Single alphaSum = LdaEstimator.Defaults.AlphaSum,

float alphaSum = LdaEstimator.Defaults.AlphaSum,

``` #Resolved

wschin · 2018-11-06T22:53:39Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+        public static Vector<float> ToLdaTopicVector(this Vector<float> input,
+            int numTopic = LdaEstimator.Defaults.NumTopic,
+            Single alphaSum = LdaEstimator.Defaults.AlphaSum,
+            Single beta = LdaEstimator.Defaults.Beta,


Suggested change

Single beta = LdaEstimator.Defaults.Beta,

float beta = LdaEstimator.Defaults.Beta,

``` #Resolved

artidoro · 2018-11-07T18:48:08Z

Could you add mlcontext extensions?
You can follow this PR as an example: #1524. The whitening PR also does that. #Resolved

artidoro · 2018-11-07T18:53:05Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

@@ -155,8 +160,10 @@ public bool TryUnparse(StringBuilder sb)
            }
        }

-        private sealed class ColInfoEx
+        public sealed class ColumnInfo


Why not combine this with transformInfo? You can just add this public constructor and change the name to columnInfo, while keeping the other methods internal/private. #Resolved

I don't see any reason why you can't delete TransformInfo class and use ColumnInfo instead.

In reply to: 231632966 [](ancestors = 231632966)

fixed

In reply to: 232104746 [](ancestors = 232104746,231632966)

artidoro · 2018-11-07T18:59:32Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

-                    getSrc(ref src);
-                    lda.Output(in src, ref dst, numBurninIter, reset);
-                };
+            return new LdaTransform(_host, input, _columns);


fixed

In reply to: 231635318 [](ancestors = 231635318)

artidoro · 2018-11-07T19:12:15Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+            }
+            using (var ch = Host.Start("Train"))
+            {
+                Train(ch, input, _ldas);


Train [](start = 16, length = 5)

Pete suggested that we make training a private static method in the transform, and that we have a constructor for the transform that will accept the trained parameters as an input.

This should make it easier to allow initialization of the transform using parameters coming from an external training routine. It is not something that we are exposing now, however.

This allows the transform to only keep the fields storing trained parameters. For example _types should no longer be needed as a field in the transform. #Closed

I have implemented something like this for whitening.

In reply to: 231639657 [](ancestors = 231639657)

fixed. thanks for giving the context about this approach.

In reply to: 231640188 [](ancestors = 231640188,231639657)

artidoro · 2018-11-07T19:16:38Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs


-                NumThread = item.NumThreads ?? args.NumThreads ?? 0;
-                ectx.CheckUserArg(NumThread >= 0, nameof(item.NumThreads), "Must be positive or zero.");
+        public sealed class LdaState : IDisposable


public [](start = 8, length = 6)

Can this be made internal/private? #Closed

This looks like some 'LDA guts' and should under no circumstances be exposed to the user.

In reply to: 231641222 [](ancestors = 231641222)

Made this internal. Instead added a class LdaTopicSummary to provide topic summary information back to users (when needed)

In reply to: 232437742 [](ancestors = 232437742,231641222)

… fields in LdaTopicSummary

abgoswam · 2018-11-15T20:57:31Z

@Zruty0 I have addressed your PR comments. Could you kindly take a look ? #Resolved

…otnet#1411)

shauheen · 2018-11-19T18:06:11Z

ping @Zruty0 #Resolved

Ivanidzo4ka · 2018-11-19T20:31:09Z

src/Microsoft.ML.Transforms/Text/LdaStaticExtensions.cs

+using System;
+using System.Collections.Generic;
+
+namespace Microsoft.ML.Transforms.Text


Microsoft.ML.Transforms.Text [](start = 10, length = 28)

Pete will ask you to put static extensions into Microsoft.ML.StaticPipe; #Closed

I sure will

In reply to: 234771139 [](ancestors = 234771139)

Zruty0 · 2018-11-20T03:09:36Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

-                ectx.CheckUserArg(NumBurninIter >= 0, nameof(item.NumBurninIterations), "Must be non-negative.");
+                Contracts.CheckValue(input, nameof(input));
+                Contracts.CheckValueOrNull(output);
+                Contracts.CheckUserArg(numTopic > 0, nameof(numTopic), "Must be positive.");


CheckUserArg [](start = 26, length = 12)

Unless parsing command line, use CheckParam instead of CheckUserArg #Resolved

Zruty0 · 2018-11-20T03:12:18Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

-            ctx.CheckAtModel(GetVersionInfo());
-            h.CheckValue(input, nameof(input));
+            // For each topic, provide information about the (item, score) pairs.
+            public readonly Dictionary<int, List<Tuple<int, float>>> ItemScoresPerTopic;


List<Tuple<int, float> [](start = 44, length = 22)

Tuple<int, float> -> (int Item, float Score) (use named tuple instead of tuples)
List -> `ImmutableArray
#Resolved

Zruty0 · 2018-11-20T03:13:08Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

-            ctx.CheckAtModel(GetVersionInfo());
-            h.CheckValue(input, nameof(input));
+            // For each topic, provide information about the (item, score) pairs.
+            public readonly Dictionary<int, List<Tuple<int, float>>> ItemScoresPerTopic;


What is the int key in this dictionary? Is it a sequential integer? If yes, make it into ImmutableArray. If it's not a sequential integer, describe what it is, and make it into an IReadOnlyDictionary #Resolved

made it ImmutableArray since its a sequential integer

In reply to: 234857025 [](ancestors = 234857025)

Zruty0

Zruty0 · 2018-11-20T03:18:53Z

Approval conditional on fixing all the remaining open comments :)

Ivanidzo4ka · 2018-11-20T20:43:15Z

src/Microsoft.ML.Transforms/TextCatalog.cs

+
+        /// <summary>
+        /// Initializes a new instance of <see cref="LatentDirichletAllocationEstimator"/>.
+        /// </summary>


please use summary form method above.
It should be user friendly. #Resolved

fixed

In reply to: 235162180 [](ancestors = 235162180)

Ivanidzo4ka · 2018-11-20T20:46:08Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs


-                ResetRandomGenerator = item.ResetRandomGenerator ?? args.ResetRandomGenerator;
+            internal ColumnInfo(Column item, Arguments args)


internal ColumnInfo(Column item, Arguments args) [](start = 12, length = 48)

you can just call constructor above, and that way you don't need to duplicate code.
internal ColumnInfo(Column item, Arguments args): this( item.Source, item.Name, args.NumTopic, ....)
#Resolved

fixed

In reply to: 235162967 [](ancestors = 235162967)

Ivanidzo4ka · 2018-11-20T20:48:22Z

src/Microsoft.ML.Transforms/Text/LdaTransform.cs

+            }
+        }
+
+        public const string LoaderSignature = "LdaTransform";


public [](start = 8, length = 6)

internal #Resolved

fixed

In reply to: 235163681 [](ancestors = 235163681)

Ivanidzo4ka · 2018-11-20T20:58:48Z

test/Microsoft.ML.Tests/Transformers/TextFeaturizerTests.cs


            // The following call fails because of the following issue
            // https://github.com/dotnet/machinelearning/issues/969
+            // In this test it manifests because of the WordBagEstimator in the estimator chain


What would happen if you uncomment this?
I've recently convert WordBag to estimator so it should work properly.

If I uncomment this, I get an error
Schema mismatch for input column 'text': expected Text, got R4 Parameter name: inputSchema
The WordBagWorkout test still comments out the TestEstimatorCore (line #186 above) .

In reply to: 235166598 [](ancestors = 235166598)

Ivanidzo4ka

…ctor

abgoswam added 6 commits October 24, 2018 01:04

preparing to convert LDATransform to the IEstimator/ITransformer para…

acd8964

…digm

Added ColumnInfo and TransformInfo

cd2f20c

existing tests pass after refactor

c289ed1

fix re-arragement of arguments in entrypoints

8dfb527

refactor + cleanup of TopicSummary

bcb3b0d

enabled OnFit to return LdaState

1d39408

abgoswam requested review from Ivanidzo4ka and artidoro October 27, 2018 03:46

Ivanidzo4ka requested review from Zruty0, sfilipi and singlis October 29, 2018 17:55

Ivanidzo4ka reviewed Oct 29, 2018

View reviewed changes

Ivanidzo4ka reviewed Oct 31, 2018

View reviewed changes

abgoswam added 2 commits November 1, 2018 16:06

fixed merge conflicts

9ff60f7

fix build issues after merge; fix review comments

ad36d2f

abgoswam commented Nov 5, 2018

View reviewed changes

wschin reviewed Nov 6, 2018

View reviewed changes

artidoro reviewed Nov 6, 2018

View reviewed changes

wschin reviewed Nov 6, 2018

View reviewed changes

artidoro reviewed Nov 7, 2018

View reviewed changes

abgoswam added 2 commits November 15, 2018 07:47

review comments - 15; schema changes

5397de5

review comments - 15; provide better user-facing description. renamed…

edd60af

… fields in LdaTopicSummary

abgoswam added 2 commits November 15, 2018 21:18

review comments - 16; fix build break

b073038

review comments - 16; include words in LdaSummary (this also resolves d…

49da3ee

…otnet#1411)

abgoswam mentioned this pull request Nov 16, 2018

Convenient way of providing LDA topic summary #1411

Closed

abgoswam added 3 commits November 16, 2018 22:58

fixing merge conflicts

0724290

fixing merge conflicts

5073baa

fix build break because of manifest changes

d1481f8

Ivanidzo4ka reviewed Nov 19, 2018

View reviewed changes

Zruty0 reviewed Nov 20, 2018

View reviewed changes

Zruty0 approved these changes Nov 20, 2018

View reviewed changes

abgoswam added 4 commits November 20, 2018 16:46

Merge branch 'master' into abgoswam/LDA_conversion

65125d4

updated to latest interface changes

b869d7f

review comments - 17; named tuple, namespace change, fix CheckParams

62955a8

review comments - 18; ImmutableArray

40333a7

Ivanidzo4ka reviewed Nov 20, 2018

View reviewed changes

Ivanidzo4ka approved these changes Nov 20, 2018

View reviewed changes

review comments - 19; update summary text; remove dup code in constru…

850856b

…ctor

shauheen merged commit 1a9e7aa into dotnet:master Nov 20, 2018

abgoswam mentioned this pull request Nov 29, 2018

Add LDA example to Microsoft.ML.Samples #1782

Merged

abgoswam deleted the abgoswam/LDA_conversion branch January 13, 2019 18:12

ghost locked as resolved and limited conversation to collaborators Mar 27, 2022

	public readonly Single AlphaSum;
	public readonly float AlphaSum;
	``` #Resolved

	public readonly Single Beta;
	public readonly float Beta;
	``` #Resolved

	Single alphaSum = LdaEstimator.Defaults.AlphaSum,
	float alphaSum = LdaEstimator.Defaults.AlphaSum,
	``` #Resolved

	Single beta = LdaEstimator.Defaults.Beta,
	float beta = LdaEstimator.Defaults.Beta,
	``` #Resolved


		ResetRandomGenerator = item.ResetRandomGenerator ?? args.ResetRandomGenerator;
		internal ColumnInfo(Column item, Arguments args)

Convert LdaTransform to IEstimator/ITransformer API #1410

Convert LdaTransform to IEstimator/ITransformer API #1410

Conversation

abgoswam commented Oct 27, 2018 • edited Loading

Ivanidzo4ka Oct 29, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

Ivanidzo4ka Oct 31, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Oct 31, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Oct 31, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Ivanidzo4ka Oct 31, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

abgoswam Nov 5, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

wschin Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

artidoro Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wschin Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

wschin Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

wschin Nov 6, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

artidoro commented Nov 7, 2018 • edited by abgoswam Loading

artidoro Nov 7, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artidoro Nov 7, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artidoro Nov 7, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artidoro Nov 7, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam Nov 11, 2018 • edited Loading

Choose a reason for hiding this comment

abgoswam commented Nov 15, 2018 • edited Loading

shauheen commented Nov 19, 2018 • edited by abgoswam Loading

Ivanidzo4ka Nov 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Zruty0 Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Zruty0 Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zruty0 left a comment

Choose a reason for hiding this comment

Zruty0 commented Nov 20, 2018

Ivanidzo4ka Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Nov 20, 2018 • edited by abgoswam Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

abgoswam commented Oct 27, 2018 •

edited

Loading

Ivanidzo4ka Oct 29, 2018 •

edited by Zruty0

Loading

Ivanidzo4ka Oct 31, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Oct 31, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Oct 31, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Oct 31, 2018 •

edited by abgoswam

Loading

abgoswam Nov 5, 2018 •

edited by Zruty0

Loading

wschin Nov 6, 2018 •

edited by abgoswam

Loading

wschin Nov 6, 2018 •

edited by abgoswam

Loading

artidoro Nov 6, 2018 •

edited by abgoswam

Loading

wschin Nov 6, 2018 •

edited by abgoswam

Loading

wschin Nov 6, 2018 •

edited by abgoswam

Loading

wschin Nov 6, 2018 •

edited by abgoswam

Loading

artidoro commented Nov 7, 2018 •

edited by abgoswam

Loading

artidoro Nov 7, 2018 •

edited by abgoswam

Loading

artidoro Nov 7, 2018 •

edited by Zruty0

Loading

artidoro Nov 7, 2018 •

edited by Zruty0

Loading

artidoro Nov 7, 2018 •

edited by Zruty0

Loading

abgoswam Nov 11, 2018 •

edited

Loading

abgoswam commented Nov 15, 2018 •

edited

Loading

shauheen commented Nov 19, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Nov 19, 2018 •

edited

Loading

Zruty0 Nov 20, 2018 •

edited by abgoswam

Loading

Zruty0 Nov 20, 2018 •

edited by abgoswam

Loading

Zruty0 Nov 20, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Nov 20, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Nov 20, 2018 •

edited by abgoswam

Loading

Ivanidzo4ka Nov 20, 2018 •

edited by abgoswam

Loading