diff --git a/Assignment 5.Rmd b/Assignment 5.Rmd index 8838dc9..231b745 100644 --- a/Assignment 5.Rmd +++ b/Assignment 5.Rmd @@ -8,15 +8,18 @@ For this assignment we will be using data from the Assistments Intelligent Tutor #Install & call libraries ```{r} -install.packages("party", "rpart") - +#install.packages("party", "rpart","rattle", "RColorBrewer") +#install.packages("party", "rpart") library(rpart) library(party) +library(rattle) +library(RColorBrewer) + ``` ## Part I ```{r} -D1 <- +D1 <- read.csv("~/R/HUDK/assignment5/intelligent_tutor.csv") ``` ##Classification Tree @@ -31,6 +34,8 @@ printcp(c.tree) #Plot the tree post(c.tree, file = "tree.ps", title = "Session Completion Action: 1 - Ask teacher, 2 - Start new session, 3 - Give up") +fancyRpartPlot(c.tree, caption = NULL) + ``` ## Part II @@ -40,22 +45,26 @@ We want to see if we can build a decision tree to help teachers decide which stu #Visualize our outcome variable "score" ```{r} - +hist(D1$score) ``` #Create a categorical outcome variable based on student score to advise the teacher using an "ifelse" statement + +It seems that the distribution of score is "tri-modal". That is, there are 3 groups of students gathered around 3 different score values. We will reccoment "advice" for those students in the middle range, with seems to be bewwwn .3 and .6. + ```{r} -D1$advice <- +D1$advice <- as.factor(ifelse(.3 + + + + + + + + + + + + + + + +Assignment 5 - Decision Trees + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + +

For this assignment we will be using data from the Assistments Intelligent Tutoring system. This system gives students hints based on how they perform on math problems.

+

#Install & call libraries

+
#install.packages("party", "rpart","rattle", "RColorBrewer")
+#install.packages("party", "rpart")
+library(rpart)
+library(party)
+
## Loading required package: grid
+
## Loading required package: mvtnorm
+
## Loading required package: modeltools
+
## Loading required package: stats4
+
## Loading required package: strucchange
+
## Loading required package: zoo
+
## 
+## Attaching package: 'zoo'
+
## The following objects are masked from 'package:base':
+## 
+##     as.Date, as.Date.numeric
+
## Loading required package: sandwich
+
library(rattle)
+
## Rattle: A free graphical interface for data science with R.
+## Version 5.2.0 Copyright (c) 2006-2018 Togaware Pty Ltd.
+## Type 'rattle()' to shake, rattle, and roll your data.
+
library(RColorBrewer)
+
+

Part I

+
D1 <- read.csv("~/R/HUDK/assignment5/intelligent_tutor.csv")
+

##Classification Tree First we will build a classification tree to predict which students ask a teacher for help, which start a new session, or which give up, based on whether or not the student completed a session (D1\(complete) and whether or not they asked for hints (D1\)hint.y).

+
c.tree <- rpart(action ~ hint.y + complete, method="class", data=D1) #Notice the standard R notion for a formula X ~ Y
+
+#Look at the error of this tree
+printcp(c.tree)
+
## 
+## Classification tree:
+## rpart(formula = action ~ hint.y + complete, data = D1, method = "class")
+## 
+## Variables actually used in tree construction:
+## [1] complete hint.y  
+## 
+## Root node error: 250/378 = 0.66138
+## 
+## n= 378 
+## 
+##      CP nsplit rel error xerror     xstd
+## 1 0.052      0     1.000  1.116 0.034193
+## 2 0.012      1     0.948  1.040 0.036036
+## 3 0.010      2     0.936  1.012 0.036587
+
#Plot the tree
+post(c.tree, file = "tree.ps", title = "Session Completion Action: 1 - Ask teacher, 2 - Start new session, 3 - Give up")
+
+fancyRpartPlot(c.tree, caption = NULL)
+

## Part II

+

#Regression Tree

+

We want to see if we can build a decision tree to help teachers decide which students to follow up with, based on students’ performance in Assistments. We will create three groups (“teacher should intervene”, “teacher should monitor student progress” and “no action”) based on students’ previous use of the system and how many hints they use. To do this we will be building a decision tree using the “party” package. The party package builds decision trees based on a set of statistical stopping rules.

+

#Visualize our outcome variable “score”

+
hist(D1$score)
+

+

#Create a categorical outcome variable based on student score to advise the teacher using an “ifelse” statement

+

It seems that the distribution of score is “tri-modal”. That is, there are 3 groups of students gathered around 3 different score values. We will reccoment “advice” for those students in the middle range, with seems to be bewwwn .3 and .6.

+
D1$advice <- as.factor(ifelse(.3 <D1$score & D1$score <= .6, 1, 0))
+

#Build a decision tree that predicts “advice” based on how many problems students have answered before, the percentage of those problems they got correct and how many hints they required

+
score_ctree <- rpart(advice ~ prior_prob_count + prior_percent_correct + hints, method="class", data=D1)
+

#Plot tree

+
fancyRpartPlot(score_ctree, caption = NULL)
+

+

Please interpret the tree, which two behaviors do you think the teacher should most closely pay attemtion to?

+

#Test Tree Upload the data “intelligent_tutor_new.csv”. This is a data set of a differnt sample of students doing the same problems in the same system. We can use the tree we built for the previous data set to try to predict the “advice” we should give the teacher about these new students.

+

ANSWER:

+
    +

  1. Students that ask for any hints. The first behavior teachers should pay attention to is students that ask for any hints. This category includes 42% of the population, and among these, 48% should receive advice.

    2. +

  2. Students whose percentage of prior correct answers is below .6 (among students that asked for hints).Among those students that asked for hints, paying attention to their prior correct answer percentage provides adittional information for teachers about which students are likely to benefit from receiving advice. Among students with prior correct answer rate below .6 AND who have asked for at least one hint, 60% should receive advice. This “dual” category (“at leas 1 hint” + “correct answer rate below .6”) includes 37.7% of the total students who should receive advice ( = .22*.6/.35)

    3. +
+
#Upload new data
+
+D2 <- read.csv("~/R/HUDK/assignment5/intelligent_tutor_new.csv")
+D2$advice <- ifelse(.3 <D2$score & D2$score <= .6, 1, 0)
+
+#Generate predicted advice using the predict() command for new students based on tree generated from old students
+
+D2$prediction <- predict(score_ctree, D2, type = "class")
+
+
+

Part III

+

Compare the predicted advice with the actual advice that these students recieved. What is the difference between the observed and predicted results?

+
#Upload new data
+
+D2 <- read.csv("~/R/HUDK/assignment5/intelligent_tutor_new.csv")
+D2$advice <- ifelse(D2$score == 1, 0, 1) # Anna told me that for this variable, students coded as "1" should not receive advice, while students coded "0" should (I find this actually very weired since everyone in the dataset has score =1)
+
+#Generate predicted advice using the predict() command for new students based on tree generated from old students
+
+D2$prediction <- predict(score_ctree, D2, type = "class")
+
+# let's build a two-way percentages table to see what we predicted right and wrong
+mytable <- table(as.factor(D2$advice),as.factor(D2$prediction))
+prop.table(mytable)
+
##    
+##        0    1
+##   0 0.73 0.27
+

It seems that the new data is very different from our training data, in the sense that in the training data we had 35% of students needing advice, while in the new data 0% of students needed advice. On the other hand, student inputs seem to be more stable (prior_percent_correct, hints, prior_prob_count), leading to a percentage of predicted “in-need-of-adcive-students” of 27%. Overall, our recommendation was “correct” for the 73% of students for whom we did not recommend advice. This is not that much of a good model though, when we compare to simply recommending the “most usual” state (no advice) - if we had done that, we would have got 100% correct recommendations.

+
+

To Submit Your Assignment

+

Please submit your assignment by first “knitting” your RMarkdown document into an html file and then commit, push and pull request both the RMarkdown file and the html file.

+
+
+ + + + +
+ + + + + + + + + + + + + + +