RFC: Create Metadata For Aggregation Store

[aklish]

Overview

Elide will export metadata (as elide models) for every table it manages via the Aggregation Data Store.

Supporting Types

Elide & Navi will start with the following set of primitive types:

• Time
• Integer
• Decimal
• Money
• Text
• Coordinate
• Boolean

public enum ValueType {
    TIME,
    INTEGER,
    DECIMAL,
    MONEY,
    TEXT,
    COORDINATE,
    BOOLEAN
}

Tables

Tables are not physical tables but represent what can logically be projected by Navi.

@Include(rootLevel = true, type = "table")
@Data
@ToString
@Builder
public class Table {
    @Id
    private String id;

    private String name;

    private String description;

    private String category;

    @Exclude
    private boolean hidden = false;

    private CardinalitySize cardinality;

    @OneToMany
    @ToString.Exclude
    private Set<Metric> metrics;

    @OneToMany
    @ToString.Exclude
    private Set<Dimension> dimensions;

    @OneToMany
    @ToString.Exclude
    private Set<TimeDimension> timeDimensions;

    @ToString.Exclude
    private Set<String> tags;
}

Tables support a set of tags that can either be used:

1. by elide for tooling purposes.
2. defined by the client

Tags include a namespace in the form namespace.tag. The namespace elide is reserved.

Columns

Columns are the base type for metrics and dimensions inside tables. Columns can be projected, filtered, and sorted on.

@Data
@ToString
@Builder
@Include
public abstract class Column {
    @Id
    private String id;

    private String name;

    private String description;

    @ToOne
    private Table table;

    //Defaults to self. Point to the terminal source column
    @ToOne
    private Column sourceColumn;

    @Exclude
    private boolean hidden = false;

    private String category;

    private ValueType valueType;

    @ToString.Exclude
    private Set<String> tags;
}

Columns support a set of tags that can either be used:

1. by elide for tooling purposes.
2. defined by the client

Tags include a namespace in the form namespace.tag. The namespace elide is reserved.

Metrics

Metrics are functions that are computed over a group of related rows.

@EqualsAndHashCode(callSuper = true)
@Include(type = "metric")
@Data
public class Metric extends Column {
    private Format defaultFormat;

    @ManyToOne
    @ToString.Exclude
    private MetricFunction metricFunction;
}

@Include(rootLevel = true, type = "metricFunction")
@Data
@ToString
@AllArgsConstructor
public class MetricFunction {
    @Id
    private String name;

    private String longName;

    private String description;

    @OneToMany
    private Set<FunctionArgument> arguments;
}

Metric functions can take zero or more arguments.

@Include(type = "functionArgument")
@Data
@ToString
public class FunctionArgument {
    @Id
    private String id;

    private String name;

    private String description;

    private ValueType type;

    private String subType;
}

Dimensions

Dimensions are columns that can be grouped, filtered, projected, and sorted. Dimensions can be sourced from joins to other tables. The metadata about where the dimension originates is surfaced to enable type ahead search.

@EqualsAndHashCode(callSuper = true)
@Include(type = "dimension")
@Data
public class Dimension extends Column {

}

Time dimensions are dimensions that support one or more time grain and a timezone. The grains and timezone can be selected at query time by the API client.

@EqualsAndHashCode(callSuper = true)
@Include(type = "timeDimension")
@Data
public class TimeDimension extends Dimension {

    Set<TimeGrain> supportedGrains;

    private TimeZone timezone;
}

public enum TimeGrain {

    DAY(Period.ofDays(1)),
    WEEK(Period.ofWeeks(1)),
    MONTH(Period.ofMonths(1)),
    YEAR(Period.ofYears(1))
    ;

    private final Period period;

    TimeGrain(final Period period) {
        this.period = period;
    }
}

[michael-mclawhorn]

Because Metric extends column, it's DataType can, structurally be a relationship. I'm not sure that's a problem, since it should probably never come up. But it's worth noting.

Do we want to address that there might be Metric column types that are different from dimension column types? I'm thinking of metrics that return something other than Number. (I'm thinking numbers with units particularly, but numbers with error bounds could be packaged this way)

Basically I'm on board with everything here.

[michael-mclawhorn]

Update: based on talking the @jkusa we'd like to add a category field to MetricFunction.

Also per discussion with @aklish and @jkusa:
Column needs to have an @id distinct from it's name, or it's name needs to embed the table it's connected to.

We may want to provide the same convention for MetricFunction

[chczy]

Merged.

[mtgraham]

Should MetricFunction also have id and name instead of name and longName?

[hellohanchen]

Currently Table is using name as Id, which is the jsonApiAlias of table's class. That's already unique, I don't think we need another name field.

[mtgraham]

@hellohanchen I think they just renamed name to id and longName to name. So now name will contain a display/formatted name of the table.

[hellohanchen]

@mattgraham1995 Thanks Matt.

Another question, why do we need sourceColumn in Column, how are we gonna to process that?