hackathon3 - solution.Rmd

---
title: "Hackathon 3 - clustering"
author: "Charles Lang"
date: "10/29/2020"
output: html_document
---

# Solution
```{r}
library(dplyr)
library(tidyr)


#Difference between anaylsis for advice and analysis for automation/production
#Predicting the future vs. predicting as characterization

D1 <- read.csv("engagement-data.csv", header = TRUE)
D2 <- read.csv("student-level.csv", header = TRUE)

#Look at variable types
#Get a fell for the data - structure, meaning of variables, ranges

# Restrucre and combine

D1b <- unite(D1, modality, c("modality", "week"), sep = "")
D1b <- spread(D1b, modality, measure)

D3 <- full_join(D1b,D2, by = "id")

#Visualize Engagement
summary(D1b$forum1)
hist(D1b$forum4, breaks = 100)

summary(D1b$game4)
hist(D1b$game2, breaks = 10)

summary(D1b$video1)

# Negative video watching recoded to 0
D1b$video1 <- ifelse(D1b$video1 < 0, 0, D1b$video1)
D1b$video2 <- ifelse(D1b$video2 < 0, 0, D1b$video2)
D1b$video6 <- ifelse(D1b$video6 < 0, 0, D1b$video6)

hist(D1b$video1, breaks = 100)

summary(D1b$logins1)
hist(D1b$logins1)

D4 <- select(D3, c(2,8,12,18))
pairs(D4)


# Visualize demographics
plot(D3$gender)

plot(D3$gender, D3$forum2)

table(D3$gender, D3$forum2)

plot(D3$gender, D3$video1)

#chisq.test(D3$gender, D3$video1)

plot(D3$parent.ed)

# Visualize Performance

hist(D3$exam)

plot(D3$exam, D3$forum1)

plot(D3$exam, D3$video1)

D4 <- select(D3, c(2,8,12,18,34))

pairs(D4)

D5 <- select(D3, c(24:34))

pairs(D5)

#I can be pretty confident in a theory that there are four groups, low/low, high/high, low/high, high/low

```

# k-means

Benefits

* Works with widely any number of variables
* Very fast
* Will find clusters

Gotchas

* Must be on the same scale
* Need to provide cluster number
* Assumes there are clusters to find - it will find clusters regardless of whether there are any or not
* Does not work on some distributional shapes
* Need uniform scale (uniform variance) - larger scale will swamp smaller scale
* Doesn’t work on categorical data of more than two categories (and the scale may be difficult to interpret)
* Can get stuck on local minima (need to run iterations)
* Too easy

```{r}

D5 <- select(D3, 2:34) #remove id variable!
D5 <- scale(D5)
D5 <- data.frame(D5)

fit <- kmeans(D5, 4)

D6 <- data.frame(D3, fit$cluster)

plot(D6$forum1, D6$assignment1, col = D6$fit.cluster)
```