Columns Selection DSL Overhaul (Discussion ended)

[Jolanrensen]

Hi everyone!
While working on the Columns Selection DSL, which is an integral part of the library, I found many inconsistencies and took it upon myself to fix that. I documented almost every function and reworked all overloads, but I still have doubts about some.
So, I collected these concerns in a helpful Kotlin Notebook for you to experience and judge! The notebook imports a proposal version of DataFrame based on the extensive PR, so you can experience the changes firsthand.
The data used for the example can be found here and to view and run the notebook, I recommend the Kotlin Notebook Plugin for IntelliJ.
You can also use Datalore or the Kotlin Jupyter kernel, but then you'd miss out on the new and extensive KDocs, so I wouldn't recommend that.

Please leave your feedback below in the corresponding thread below. I'm curious to hear your feedback! Also, feel free to leave any questions below as well if my notebook isn't clear enough in explaining the issues.

[Jolanrensen]

New: ColumnSet / SingleColumn Split

[Jolanrensen]

New: Streamlined behavior for functions on SingleColumn<DataRow<*>> v/s ColumnSet

[Jolanrensen]

Question: Split plain DSL calls from SingleColumn functions?

[Jolanrensen]

Probably: yes. This allows functions like first to be called firstChild on SingleColumns and first in the normal DSL. Also would allow for overloads to not exist in the DSL, but still on SingleColumns.

[zaleslaw]

yes

[Jolanrensen]

done

[Jolanrensen]

Question: KProperty<DataRow<*>>.function() or KProperty<*>.function()?

[Jolanrensen]

Type safety is good, but explicit conversion might be much, especially since it's not needed for String/Columnpath.

[zaleslaw]

I prefer KProperty<>.function() for every day routine

I'm a big fan of this approach

df.select { SomeType::created.function3() // :) SomeType::filtering.function3() // :) SomeType::name.function3() // :( }

[Jolanrensen]

@zaleslaw that would create runtime exceptions if a group-function is called on a non-group property, similar to with the String-api, making it unsafer. Do you think that's worth it?

[Jolanrensen]

Question: All, Cols, Children overlap and differences

[zaleslaw]

I don't use this API

// all same df.select { all() } df.select { cols() } df.select { children() } // same but debatable name? df.select { filter { true } }

but love this

// all same
df.select { created.all() }
df.select { created.cols() }
df.select { created.children() }
df.select { created.filter { true } }

But it's ok to keep, the behaviour should be the same

[Jolanrensen]

Other idea: rename cols/colsOf on single columns (and strings, kproperties, etc) to children/childrenOf

[Jolanrensen]

Question: String/ColumnPath, allow group calls directly on it or only on SingleColumn?

[Jolanrensen]

Might just drop the get overloads. Making sure all names have something referring to "children" in it probably makes it clear enough to keep the overloads.

[zaleslaw]

yes, more useful functions everywhere

From the other hand, I really don't understand the concept where we call columns via String names.

If this works, what's the problem to be more Pytonic everywhere:)

[Jolanrensen]

@zaleslaw if you call columns on a String, you're assuming there's a column group with that string name and you're filtering its children.

[Jolanrensen]

Question: cols {} vs filter {}

[Jolanrensen]

might rename singleColumn instances to filterChildren as well.

[zaleslaw]

For filtering columns, I suggest use only one operator like cols{}, filter confused me here, I'm starting to think about data filtering

[Jolanrensen]

Yeah I think I agree, I might just keep ColumnSet.filter and drop the others.

[Jolanrensen]

Question: What to do with Except?

[Jolanrensen]

Make except retain structure, both on singleColumns and columnSets (and remove it in the plain dsl), and allExcept unpack the column group (on plain dsl and singleColumns) is probably the best way forward.

[zaleslaw]

Probably minus or overloading of - is better here. Thinking about columns like about List in Kotlin

[Jolanrensen]

but what should minus do? keep the structure of the group and remove the specified column or unpack the group and take all except the specified column?

[Jolanrensen]

Question: If SingleColum.except wouldn't change structure, shouldn't select {} as well?

[zaleslaw]

I'd like this // then what should this do? Select items and link with JUST target inside? // We could even make a take {} function for this for example

[Jolanrensen]

Question: What to do about Select/Cols overlap?

[Jolanrensen]

Might just provide the overloads. It's clear to the users what it would do, so it won't hurt to add. We can skip the index functions though, since they don't exist for DataFrame.select()

[zaleslaw]

Only this looks not pretty for me, no big deal hierarchy of selects, but sometimes for reading, I want to see something like "innerSelect" or "nestedSelect"

section.select { "link"[{ it.name.startsWith("target") }] }

[Jolanrensen]

New: Recursively (with Question)

[Jolanrensen]

Recursively can be misleading in some cases (the .rec() abbreviation as well). A better name that describes the behavior "traversing all columns including column groups and those inside column groups" would be awesome.
Also, since .recursively() is a post-modifier, it might be unclear about how far "back" it has influence. This could be solved by providing a scope {}-like function, but that has other downsides and needs te be properly named and tested.
Still, exploring the options would be good.

Anyone any ideas how other frameworks handle this?

[Jolanrensen]

.includingNested()?

[Jolanrensen]

.includingNestedColumns()/.inclNestCols()?

[Jolanrensen]

.atAllDepths()/.atAllLevels() is also an option

[zaleslaw]

You missed {} vs []

[Jolanrensen]

No, that's a subsection of Question: What to do about Select/Cols overlap?

[jbaron]

Personally I find that if the dataframe API is too different from the standard Kotlin collection API's, it is hard to remember the usage patterns (especially when not using it every day).

In my mental picture, a dataframe is just a columnar storage. So I would like to have normal Kotlin collection familiar access patterns over these columns.

The following code-snippet shows how a dataframe looks in my head, and the kind of column access patterns I would expect:

data class Col<T>(val name: String, val rows: List<T>)

class DataFrame(vararg val cols: Col<*>)

val df = DataFrame(
        Col("name", listOf("Peter", "John")),
        Col("age", listOf(50, 25))
)

// Some access patterns
df.cols.filterIsInstance<Col<Int>>().forEach { println(it.rows) }
df.cols.filter { it.name.contains('a') }.forEach { println(it.name) }

val floatColumns = df.cols.filterIsInstance<Col<Float>>()
floatColumns.forEach { it.rows.max() }

BTW, I only use dataframes now and then, so for sure I'm not a power user.

[Jolanrensen]

Well, you can retrieve the columns from a DataFrame into a List, so then you can simply use the standard library functions as you wish:

df.columns().filterIsInstance<ValueColumn<Long>>()

[jbaron]

Indeed that is what I typically use to select a column. What I wanted to state (poorly so I guess), is that offering too many convenience functions may reduce maintainability.

For example the following looks very nice section.select { colsOf<String>() except name } , but at the same time requires extra learning over a similar plain Kotlin collections approach: section.cols.filterIsInstance<Column<String>>().filter { it.name != "name" }

[Jolanrensen]

I agree, but that's kinda the definition of a DSL. It's a very readable mini-language, which, by definition, is different than the Kotlin stdlib. It's one of DataFrame's strengths to be able to write statements that are understandable for people who might not even be programmers. So that's why we're supporting both approaches.

Plus, the DSL allows for auto-generated accessors (https://kotlin.github.io/dataframe/extensionpropertiesapi.html) with proper scoping, typing, etc. This is more difficult in the non-DSL approach.

But, of course, if you prefer the Kotlin collections approach, you don't have to relearn your behavior, since DataFrame supports that too :).