Improving the Describe > One Variable > Summarise dialogue #7123

Ivanluv · 2022-01-13T13:39:45Z

fixes #7081
@africanmathsinitiative/developers this is ready for review
@rdstern In this PR I have added the functionality to control the orientation, i.e. choosing whether the variables or the summaries go down in the summary table

rdstern · 2022-01-14T10:44:12Z

@Ivanluv well done. That looks great for a start.
a) Omit the Treat Columns as Factor checkbox, but always do that, so the 2 other controls are always visible.
b) Change the Label for the first from Summary to Summaries. Default is for it to be checked.
c) Now a puzzle that might be more difficult to resolve, namely the receiver can take any type of variable. So could be factor, or ordered factor or date, etc. Currently it gives an error for dates or factors or character variables. Daters should be ok (in the long run) to solve, and also need to be possible in the general tabulation. I am happy for most summaries to be missing when they don't apply. But I would like to be able to cope with those that do. I hope you may have ideas on what can be done in mmtable2, e.g. dates, and what might be impossible.
d) Could you also check on different types of output - this applies to the general tabulation as well. Currently it is html. @Patowhiz is looking into whether that may be possible in the output window. I wonder whether there might be other options, particularly simple ones that could go into the current output window.
e) Oh I forgot - I think only half the corrections have been included - so the answers are wrong!

Here they are from the new dialogue - with the totals line included - so those are double the true values. But the columns referred to are incorrect!

rdstern · 2022-01-20T17:15:57Z

@Ivanluv it now seems to get the correct summaries, but in the wrong order.

This was the dialogue:

a) You can see that Blue Sky remains at the top in the results, but its summary is at the bottom!
b) The summary for R-Instat is in second place (towards the top - but it should be second from the bottom.
c) The labelling has become alphabetical.

And, may need @lilyclements help - it doesn't work for factor variables. etc. I would like it to do something for variables of any type. Not sure what to do here?

rdstern

Comments given for changes

Ivanluv · 2022-01-24T06:18:17Z

@lilyclements any suggestions here

lilyclements · 2022-01-26T17:01:27Z

@Ivanluv have you tried updating your branch? I am not getting this error if I try it on the version on PR #7107

Updating branch

Ivanluv · 2022-01-27T06:48:40Z

And, may need @lilyclements help - it doesn't work for factor variables. etc. I would like it to do something for variables of any type. Not sure what to do here?

@lilyclements I have updated this branch and its now working okay

Ivanluv · 2022-01-27T06:49:30Z

And, may need @lilyclements help - it doesn't work for factor variables. etc. I would like it to do something for variables of any type. Not sure what to do here?

@lilyclements do you have any ideas how we could fix this?

rdstern

@Ivanluv this is now working fine. Now I tried with other types of data (not-numeric) and it doesn't like that.
Now I thought you might need help from @lilyclements for this. But then I tried with our summary system and that handles all types of data perfectly - even character variables. And dates come out fine as dates in the summary. So I wonder if that can provide the code to be used before going into mmtable?
I treid with dodoma data, so there is a date variable and also month as a factor. Then I made month into character. It just gives NA when it can't work out the summary!

On the actual dialogue could you have the Variables as rows checked by default.

Ivanluv · 2022-01-27T11:11:41Z

@rdstern by the "summary system" do you mean summary tables?

rdstern · 2022-01-27T11:39:42Z

No, I meant Prepare > Data: Reshape > Column Summaries. The first receiver is Variable(s) to Summarise and it allows all types of variable - as does your Describe > One Variable > Summarise. And it also has the same sub-dialogue as yours, and copes with any summary for any type! Just what we need in your One Variable dialogue!

Ivanluv · 2022-01-27T13:44:08Z

@rdstern should we have the results stored as it is being done in the Describe > One Variable > Summarise?

rdstern · 2022-01-27T16:37:25Z

@Ivanluv you ask - " should we have the results stored as it is being done in the Describe > One Variable > Summarise?" I assume you mean as is done in the Prepare > One Variable > Column Summaries"

I would not, at least for now. Once we are able to do sensible summaries for variables of all types, then I would like to merge and then reflect and consider.

What is done here is a great update, and very useful. I am more concerned here with the tables of results. And they are able to use any of the summaries we have in our system. There will be more to do, but that can come later. We need it to stay simple, but could usefully add the options for the missing values - now that we have them in a package.
Then I am waiting for Patrick to sort out html for the output window, which will be great.

Then (as you say) there will be 2 further directions:
a) More flexibility on the output. We could add the presentation options from the Summary tables work. But let's finish that on the summary tables dialogue first. Then this dialogue could allow users to practice with very simple structures.
b) We could permit the save - as you say.

We also apply these ideas to the 2-way and 3-way tables - if we have the 3-way - I think so!

Then in simple graphs and tables (which IS descriptive statistics after all) we have a package to be envied!
This will be good for both teaching and also doing!

And I continue while I am thinking what's needed. I use this, because one-variable summaries is soooooo simple. Who would have thought you could do so much with it!

In the presentation of the results we can choose the order of the summaries in the general case, but not here. And, even in the general case, we can't yet choose the labels for those summaries - but we will in the General Summaries. Then the challenge in this dialogue is to make it as attractive and flexible as possible - but not complicated. Here it must stay also simple for the user.

rdstern

@Ivanluv not sure what has happened here? This now seems like the old dialogue again?

Ivanluv · 2022-01-28T11:55:11Z

@rdstern Am still making the changes for the dialogue.

rdstern

Hi @Ivanluv I assume these changes are "work in progress?", but I had a quick look.
a) The options are now visible all the time. Should not be, for the default where the summary is used.
b) The default of ticking the Variables as rows is still to be done.
c) The summary gives an error now.

Ivanluv · 2022-01-31T13:21:53Z

@rdstern could you have a look

lilyclements · 2022-02-14T13:25:08Z

@Ivanluv if the user selects to modify/choose the summary statistics to display, you are currently using $calculate_summary not $summary_table.

$summary_table is a wrapper for $calculate_summary therefore you do not need to use $pivot_longer with $summary_table as that is all incorporated into the code.
Additionally, the alphabetical order would be sorted with $summary_table since this is already written in.
Is there a reason to not use summary_table?

In PR #7230, the code is fixed to use date variables in the summary functions. Once this is merged in, then update your master and hopefully that will work! (Or you can merge it into this PR if you think that's more appropriate).

lilyclements · 2022-02-14T15:40:39Z

@Ivanluv I'm not sure if this will fix the proportions problem. If it is still a problem after changing the function then I can look into it then

lilyclements · 2022-03-08T12:57:25Z

@Ivanluv how is this work going?

Ivanluv · 2022-03-08T14:14:08Z

@lilyclements am waiting for PR #7230 to be merged

Summary table dates

Ivanluv · 2022-03-14T12:36:49Z

@rdstern @lilyclements I have merged PR #7230 into this and made the changes as suggested by @lilyclements i.e using the summary_table function instead of the $calculate_summary function. There is however a challenge when trying to find summaries not supported by certain data types eg getting the sum of factor columns as below.

After the error the table won't be produced even if one of the selected columns supports the summary

rdstern · 2022-03-16T12:24:04Z

It is looking nice

Note sensible summaries for date variables. And I made month into an ordered factor, so it can get the min and max! That's nice to explain to people!

But we do need to solve the problem that @Ivanluv mentioned, namely that it can't cope with some summary statistics - like sum, or mean, for a factor. When values are missing, and that option isn't given, then that cell is blank. We need the same for those statistics for a factor variable, etc.

One small change if @Ivanluv can't solve the current error, is to move the Missing option to below the other 2 checkboxes. Then it will be in the same relative position as in the summary dialogue.
But we can't merge until we can present tables of results for datasets (like the simple survey) that mix factors and variates.

lilyclements · 2022-03-24T14:04:29Z

So the sum/mean for factor is a problem in calculate_summary function. The same error reported by @Ivanluv can occur elsewhere in R-Instat. For example, in the "Column Summaries" dialog -

I've tried to work on getting this into the calculate_summary function, but think I need assistance from @dannyparsons

lilyclements · 2022-03-29T15:41:10Z

instat/static/InstatObject/R/Backend_Components/summary_functions.R

@@ -1418,28 +1423,29 @@ DataBook$set("public", "summary_table", function(data_name, columns_to_summarise
      for (facts in power_sets_summary) {
        if (length(facts) == 0) facts <- c()
        summary_margins_df <- data_book$get_data_frame(data_name = data_name) %>%


Can you change data_book to say self? Realised I missed these earlier.

lilyclements · 2022-03-29T15:41:20Z

instat/static/InstatObject/R/Backend_Components/summary_functions.R

-          dplyr::select(c(factors, columns_to_summarise)) %>%
-          tidyr::pivot_longer(cols = columns_to_summarise)
+          dplyr::select(c(tidyselect::all_of(factors), tidyselect::all_of(columns_to_summarise))) %>%
+          tidyr::pivot_longer(cols = columns_to_summarise, values_transform = list(value = as.character))
        data_book$import_data(data_tables = list(summary_margins_df = summary_margins_df))


lilyclements · 2022-03-29T15:41:25Z

instat/static/InstatObject/R/Backend_Components/summary_functions.R

-          dplyr::select(c(factors, columns_to_summarise)) %>%
-          tidyr::pivot_longer(cols = columns_to_summarise)
+          dplyr::select(c(tidyselect::all_of(factors), tidyselect::all_of(columns_to_summarise))) %>%
+          tidyr::pivot_longer(cols = columns_to_summarise, values_transform = list(value = as.character))
        data_book$import_data(data_tables = list(summary_margins_df = summary_margins_df))
        summary_margins[[length(summary_margins) + 1]] <- data_book$calculate_summary(data_name = "summary_margins_df", columns_to_summarise = "value", summaries = summaries, factors = facts, store_results = FALSE, drop = drop, na.rm = na.rm, return_output = TRUE, weights = weights, result_names = result_names, percentage_type = percentage_type, perc_total_columns = perc_total_columns, perc_total_factors = perc_total_factors, perc_total_filter = perc_total_filter, perc_decimal = perc_decimal, margin_name = margin_name, additional_filter = additional_filter, perc_return_all = FALSE, ...)


lilyclements · 2022-03-29T15:41:32Z

instat/static/InstatObject/R/Backend_Components/summary_functions.R

-          dplyr::select(c(factors, columns_to_summarise)) %>%
-          tidyr::pivot_longer(cols = columns_to_summarise)
+          dplyr::select(c(tidyselect::all_of(factors), tidyselect::all_of(columns_to_summarise))) %>%
+          tidyr::pivot_longer(cols = columns_to_summarise, values_transform = list(value = as.character))
        data_book$import_data(data_tables = list(summary_margins_df = summary_margins_df))
        summary_margins[[length(summary_margins) + 1]] <- data_book$calculate_summary(data_name = "summary_margins_df", columns_to_summarise = "value", summaries = summaries, factors = facts, store_results = FALSE, drop = drop, na.rm = na.rm, return_output = TRUE, weights = weights, result_names = result_names, percentage_type = percentage_type, perc_total_columns = perc_total_columns, perc_total_factors = perc_total_factors, perc_total_filter = perc_total_filter, perc_decimal = perc_decimal, margin_name = margin_name, additional_filter = additional_filter, perc_return_all = FALSE, ...)
        data_book$delete_dataframes(data_names = "summary_margins_df")


finally here. Thanks!

lilyclements · 2022-03-29T15:52:38Z

@rdstern @lilyclements I have merged PR #7230 into this and made the changes as suggested by @lilyclements i.e using the summary_table function instead of the $calculate_summary function. There is however a challenge when trying to find summaries not supported by certain data types eg getting the sum of factor columns as below.

@Ivanluv @rdstern I have now fixed this bug in PR #7338

Calculation summary edits

Ivanluv · 2022-03-31T09:08:18Z

@lilyclements I have pulled you changes from PR #7338 and am getting the following error

lilyclements

@Ivanluv is village or size a character variable? If so then the following should fix it. If not, then the following will fix another potential bug so can you implement the following anyway:

lilyclements · 2022-03-31T09:39:36Z

instat/static/InstatObject/R/Backend_Components/calculations.R

+                                                   property = "class")
+      # note: important that there *is* a space between | for str_detect function
+      if (any(stringr::str_detect("factor | Date", col_data_type))){
+        curr_data_list[[c_data_label]] <- curr_data_list[[c_data_label]] %>%


@Ivanluv can you change “factor | Date” to
“factor | character | Date”

it is important to have a space between the | too

@lilyclements the bug is now fixed

rdstern · 2022-04-01T20:45:31Z

@Ivanluv was this supposed to work now? I tried with a minimum for a factor and it said that isn't allowed.

rdstern · 2022-04-07T06:42:55Z

@Ivanluv and @lilyclements I am hoping this will be included in the Version 0.7.5 update? That's in 2 weeks time. OK?

lilyclements · 2022-04-08T09:35:14Z

@Ivanluv was this supposed to work now? I tried with a minimum for a factor and it said that isn't allowed.

@rdstern which combinations do we not want to work?

Factors and characters:
minimum/maximum
mean
median
sum
range
SD
variance

Ordered factors:
mean
median
sum
range
SD
variance

Dates:
sum
SD/var?

I have not yet put in that min/max (amongst others in the list above) shouldn't work for factors or characters, but that is very quick to add. I'll add any other combinations when you've responded to this so I know if there's any I'm missing here. Thanks!

rdstern · 2022-04-08T09:51:49Z

@lilyclements and @Ivanluv I'd prefer to look at this the other way round, namely what is sensible for different types. I am just keen that we have an option to mix the data types in this dialogue, rather than having to think carefully what to put in the the receiver.
So numeric allows everything. Logical and date are special cases of numeric
a) Ordinary factors (or character) only allow number of values, including number of unique values, number of missing, etc.
b) Ordered factors allow max and min (and they should therefore allow range - perhaps we don't yet use the range function?
c) Ordered factors currently don't allow anything else. I actually don't wee why they can't allow median and other percentiles, because they are ordered. But let's stick with what R does for now.
d) So only the numeric types allow means, sd, skewness and everything else. And for now percentiles, and proportions, etc.

Ivanluv · 2022-04-12T12:44:38Z

@lilyclements what is you suggestion on how to proceed ?

Updating Branch

ivanluv added 6 commits January 13, 2022 11:28

Modifying the summary function

e34f118

Adding Header functions

a888ac0

Adding the mmtable Operator

b3490e1

Adding and removing header functions

615c591

Linking the display checkboxes

5a00bb1

Minor corrections

1aa84b3

Minor changes

331bdcf

rdstern requested changes Jan 21, 2022

View reviewed changes

Merge pull request #244 from africanmathsinitiative/master

cb42fc8

Updating branch

rdstern requested changes Jan 27, 2022

View reviewed changes

Adding the calculate summary function

73dbad1

rdstern reviewed Jan 28, 2022

View reviewed changes

ivanluv added 2 commits January 28, 2022 14:55

Additional changes

7bfb2f8

adding the stacking functions

8acf381

rdstern requested changes Jan 30, 2022

View reviewed changes

ivanluv added 3 commits January 31, 2022 11:30

linking of the checkboxes to the custom radi button

1b29ce7

Adding Summary functins

5ad9996

adding the mmtable operator

f3e14fd

Update summary_functions.R

1f7aac6

Ivanluv and others added 2 commits March 9, 2022 16:15

Merge pull request #260 from lilyclements/summary_table_dates

7ca6f38

Summary table dates

Replacing the calculate function with the summary_table function

a177ff6

Changing the position of the omit missing checkbox

f1ac662

lilyclements reviewed Mar 29, 2022

View reviewed changes

Update calculations.R

63e1d9d

lilyclements mentioned this pull request Mar 29, 2022

Calculations to give NA if an incompatible summary statistic is chosen #7338

Closed

Merge pull request #266 from lilyclements/calculation_summary-edits

501b5be

Calculation summary edits

lilyclements reviewed Mar 31, 2022

View reviewed changes

Minor changes

0774bb7

N-thony mentioned this pull request Mar 31, 2022

Improved Options by Context > Prepare > Merge Additional Data dialog #7347

Merged

Merge pull request #275 from africanmathsinitiative/master

5e9acc2

Updating Branch

Ivanluv mentioned this pull request Apr 25, 2022

Improved the Describe > One Variable > Summarise dialog #7410

Merged

lilyclements mentioned this pull request Apr 25, 2022

Add NA for calculations with incompatible summaries #7401

Closed

Ivanluv closed this Apr 25, 2022

Ivanluv deleted the One_Variable#7081 branch July 6, 2022 08:24

Improving the Describe > One Variable > Summarise dialogue #7123

Improving the Describe > One Variable > Summarise dialogue #7123

Conversation

Ivanluv commented Jan 13, 2022

rdstern commented Jan 14, 2022 • edited Loading

rdstern commented Jan 20, 2022

rdstern left a comment

Choose a reason for hiding this comment

Ivanluv commented Jan 24, 2022

lilyclements commented Jan 26, 2022 • edited Loading

Ivanluv commented Jan 27, 2022

Ivanluv commented Jan 27, 2022 • edited Loading

rdstern left a comment

Choose a reason for hiding this comment

Ivanluv commented Jan 27, 2022

rdstern commented Jan 27, 2022

Ivanluv commented Jan 27, 2022

rdstern commented Jan 27, 2022 • edited Loading

rdstern left a comment

Choose a reason for hiding this comment

Ivanluv commented Jan 28, 2022

rdstern left a comment

Choose a reason for hiding this comment

Ivanluv commented Jan 31, 2022

lilyclements commented Feb 14, 2022

lilyclements commented Feb 14, 2022

lilyclements commented Mar 8, 2022

Ivanluv commented Mar 8, 2022

Ivanluv commented Mar 14, 2022

rdstern commented Mar 16, 2022

lilyclements commented Mar 24, 2022 • edited Loading

lilyclements Mar 29, 2022

Choose a reason for hiding this comment

lilyclements Mar 29, 2022

Choose a reason for hiding this comment

lilyclements Mar 29, 2022

Choose a reason for hiding this comment

lilyclements Mar 29, 2022

Choose a reason for hiding this comment

lilyclements commented Mar 29, 2022

Ivanluv commented Mar 31, 2022

lilyclements left a comment

Choose a reason for hiding this comment

lilyclements Mar 31, 2022

Choose a reason for hiding this comment

Ivanluv Mar 31, 2022

Choose a reason for hiding this comment

Ivanluv Mar 31, 2022

Choose a reason for hiding this comment

rdstern commented Apr 1, 2022

rdstern commented Apr 7, 2022

lilyclements commented Apr 8, 2022 • edited Loading

rdstern commented Apr 8, 2022

Ivanluv commented Apr 12, 2022

rdstern commented Jan 14, 2022 •

edited

Loading

lilyclements commented Jan 26, 2022 •

edited

Loading

Ivanluv commented Jan 27, 2022 •

edited

Loading

rdstern commented Jan 27, 2022 •

edited

Loading

lilyclements commented Mar 24, 2022 •

edited

Loading

lilyclements commented Apr 8, 2022 •

edited

Loading