Data sets for data science presentation work

[rdstern]

I would like us to organise and summarise a collection of interesting data sets. They should be documented and will become freely available. Some will go into the R-Instat library. Others will be housed apart and referenced. Some are already available and we then report where they are.

1. MICS survey data. They are on the MICS site and there is a separate discussion topic on them. These are survey data available in SPSS files, together with full analyses also available.
2. IPUMS data. Please could someone look for specific examples and check these data read nicely into R-Instat. Good to report. Can start with data from Kenya.
3. Kenya surveys from Kenya National Bureau of Statistics. Please could someone look here first. Then choose some surveys that are of interest and report.
4. Education: I think James recent course, where mastery tests were used, has interesting data and an important set of problems to investigate. Please check with James.
5. Procurement/Corruption: We already have the world bank data in the R-Instat library. Danny will send more examples. It will be good to have information on how the data were obtained, because I think this involved "scraping" the web. We now have data discussed in Procurement data for R-Instat work #6629.
6. Climatic data from Kenya: James/Wycklife's work. Includes data recovery as a component.
7. Climatic data from 14 countries. MSc essay being written about these data from Cameroon.
8. Satellite and reanalysis data: We have an interesting example from Germany (4 stations etc.) Perhaps there is more - Discuss with Danny.
9. Agriculture data: Fuma Gaskiya data - discuss with David. Also consider the games data here.
10. dslabs data: As used as examples in data science teaching by Harvard. Compare examples with chapters in the on-line book, which is here. Also good discussion on alternatives to bar charts here.
11. Streamflow data to come from discussions with John Gathenya.
12. Health/medical data: Discuss with Lily.

In particular we have data on each of our main areas. And discuss these as categories.

[rdstern]

In the list above I would now like to proceed with four of the data types from the above. Namely:
a) MICS data
b) Procurement (corruption) data
c) IPUMS data
d) Kenya Open data

These four sets are all such that they can't go directly in the R=-Instat library. But they can be referred to, in the library.

Your tasks in clude downloading and analysing the data. But your deliverables are as follows:

a) Documents on the data to make it as easy as possible for others to get access, download and then process the data using R-Instat.
b) Suggestions on how this should look once fully implemented - so the developers can improve R-Instat in ways that correspond to your needs.
c) So, if the resources you are preparing include pdf files, then how should an R-Instat user get access to this information. One way could be via File > Import from Library. Then there could be a directory for MICS with your overall pdf file about the surveys in general. And (within that) directories for each country perhaps. If so, then we should be able to read pdf files from the library? We can't yet. Can this facility be added to R-Instat, or is this not what is envisaged? Perhaps use the Help menu instead?
d) It would also be good to have access to the information from the R_Instat web page. That could be perhaps among the general improvements to the web site. Could that also be discussed during the spring?
e) The MICS data is currently in SPSS files. The procurement data is in csv and in Stata files. I like that, partly because we want people to be able to continue using those tools, if thety are happy with them.
f) So we need to be able to read those files. In principle we can. The new (not yet merged) version of the File > Import Data Frames dialogue reads multiple files of a given type. In practice the import of even a single stata (.dta) procurement file crashed R-Instat - though it read other stat files easily. That needs to be resolved - perhaps RStudio can read them, or this is from a different version of stata and ... (By the way, that's why we are also now starting from real data - you often miss issues if not.)
g) Then, in the analyses please concentrate on aspects that link directly to using resources in R-Instat and identifying needs. In particular most of these datasets have labelled data and we don't yet either manage, or analyse the labels part very well. You should know about labelled data (both variable and value labels) and what is available, and what can be improved in this regard. This relates both to managing the labels and also using them in analyses. In preparing data how much more should we do on labelled data is a question for the sprint. These data sets provide good examples and we may want to export to SPSS (say). How does that work without labels? It may be easy?
h) Then, on analyses, we have a few commands that make use of labels. In particular the Describe > One Variable > Frequencies, and Two Variable and Three Variable FRequencies do this. I suggest this provides good reasons for keeping these dialogues and hence for checking they are as good as possible.
i) So, please get examples of these data to work with next week. They are wide, so should help as we build up the facilities to cope better with wide data sets.
j) Also use the Simple Describe commands, report and suggest deficiencies and possible improvements.

[Wycklife]

@rdstern Thank you for sharing the tasks. I have documented about (MICS and IPUMS) as a word document i.e. how to download the data sets and the process of getting them into R-Instat. I suggest we add a new section on wiki called data types. From here we can document all the information for each data type and I think it will be easy to access them. May be if we can also upload the data sets and the reports as per the data type.

[rdstern]

@Wycklife can you add your word document into your message above?

[Wycklife]

@rdstern Yes I can, I thought it was still under discussion. I will add them.

[rdstern]

Please could someone investigate "the Islands" quickly? This is a statistical "game" a very impressive game from Australia by Michael Bulmer that provides data for epidemiology etc. Here is a blog about it - as often with imaginative teaching of statistics, it is from New Zealand. I would like just a brief description, and then a sample of data from a survey on these fictitious islands.

I hope this might interest @Cal-Insta who is starting teaching now. And/or @jkmusyoka and @shadrackkibet for possible use in Maseno, or even the AIMS Cameroon course. Could it become a week 3 case study? Perhaps @berylwaswa or @Kelvin-23 would wish to take it up - check with James first. Don't spend too much time at this stage - but I would like some data for R-Instat!

Aha - I have been to Michael Bulmer's home page. That gives a sample data set, so I am ok for now. It is here
Table1-2.zip

It would still be nice to have more information about the simulation "game" itself. I couldn't see easily how to play the game to get my own data. Is there an instruction guide anywhere?