courseNotes.rmd

---
title: "Programming for Data Analysis and Statistics with R"
subtitle: "Course Notes"
author: "Diren Senger"
output:
  bookdown::gitbook:
    lib_dir: "book_assets"
code_folding: hide
number_sections: TRUE
---
<!-- html_document: -->
<!--     toc: true -->
<!--     theme: readable -->
<!--     highlight: textmate -->
<!--     toc_depth: 3 -->
<!--     toc_float: TRUE -->
    
<script>
function myFunction(id) {
    var x = document.getElementById(id);
    if (x.style.display === "none") {
        x.style.display = "block";
    } else {
        x.style.display = "none";
    }
}
</script>


<style>
.question {
background-color: lightblue;
}
</style>

# index

This guide was created for the course "Programming for Data Analysis and Statistics with R" at the Informatica Feminale 2019 and extended for the same course in 2020.

# Installation Notes


![Creating a Project, Step 1](images/project1.png)


  If you haven't installed R and RStudio yet, use following instructions for the installation:
  
1. Install R: 
  - On Windows: https://www.dummies.com/programming/r/how-to-install-r/
  - For Linux and Mac: https://cran.r-project.org/doc/FAQ/R-FAQ.html#How-can-R-be-installed_003f



2. Install the free RStudio Desktop Version: https://www.rstudio.com/products/rstudio/#Desktop


Do you know how you can install new R packages? You can read about it here: https://www.dummies.com/programming/r/how-to-install-load-and-unload-packages-in-r/

# RStudio Programming Environment

## Creating a new project

In many statistics classes they teach you, that you should set a working directory:

```{r eval=F}
setwd("~/Documents/")
```

It can be helpful to create a project instead. All files within the project's directory will automatically have the correct working directory. In this way you can create new files and start coding easily.


1. Click File -> New Project...
  Even when you choose "Existing Directory" you will still be able to create a new directory
![Creating a Project, Step 1](images/project1.png)
2. Enter the diretory path straight away, or click on browse.
![Creating a Project, Step 2](images/project2.png)
3. Choose an existing directory or create a new one
![Creating a Project, Step 3](images/project3.png)

## RStudio screen

- In the upper left hand corner you have the code editor. This is where you can type your scripts.
- In the bottom left hand corner you have the console. Here you will see the results of all commands you are running.
- In the upper right hand corner you can see all objects you created, e.g. data frames, functions, strings, ...
- In the bottom left hand corner you will be able to see the files in your project, your plots and more information e.g. about functions
![RStudio screen](images/rstudio.png)

You can use "Ctrl" followed by a number to jump to a specific window. If you want to get into the terminal use "Shift+Alt+t".
![RStudio screen](images/rstudioNumbers.png)


## R Workflow

- You type your scripts in the code editor, so that you can save them.
- If you want to run the whole script, you can click on 'Source' at the tope of the code editor or press "Ctrl + Alt + R".
- Use "Ctrl + Alt + B/E" to run from the beginning to the current line or from the current line to the end.
- If you want to run a single line, you can click on 'Run' or press "Control + Enter"
- If you want to run a few lines, highlight the lines you want to run and use 'Run'/  "Control + Enter"
- If you want to use Autocomplete, you can press "Control + Space"
- If you want to read about an existing method, you can type "?\<Name of the method\>" to get to the documentation


# R Data types
## Character
```{r}
firstname <- "Diren"
surname <- "Senger"
name <- paste(firstname, surname)
name
```
## Numeric
```{r}
a <- 1
b <- 2
c <- a + b
c
d <- b*c/a
d
e <- d^2
e
f <- as.integer(3.14)
f
```
## Logical
```{r}
a <- TRUE
b <- FALSE

# or
a|b

# and
a&b

# not
!a

4 == 5

4 < 5
4>=4

is.na

as.numeric(a)
```

## Vectors
```{r}
# a simple numeric vector
myvector <- c(1,4,6)
myvector
# a vector of type character
myfamily <- c("Paula", "Ali", "Emma")
myfamily
# get the first element of a vector
myfamily[1]
# apply function to all items:
myvector + 1
paste(myfamily, "Meier")
# concatenate vectors:
longervector <- c(myvector, 5:7)
longervector
# create a sequence
odd <- seq(from = 1, to = 10, by = 2)
odd
# create a boring sequence
boring <- rep(1, 10)
boring
```


## Factors

```{r}
fac <- factor(c("good", "better", "best"))
levels(fac)
nlevels(fac)
```

## List of lists
You can create a list of lists (a list of vectors):
```{r}
myList <- list(smallLetters=letters[1:10], 
               capitalLetters=LETTERS[1:10], 
               numbers=1:5)
myList
```

If you want to choose lists, you can do it with a single bracket.
```{r}
# Accessing multiple elements
myList[1:2]
```

If you want to choose single elements, you need double brackets:
```{r}
# Accessing single elements
myList[2][2]
myList[[2]][2]
myList[[1:2]]
```



## Matrices
```{r}
m <- matrix(data=1:12, nrow = 3, ncol = 4)
m
```

Element wise operations:

```{r}
m
# Element-wise operations
m * 0.5
```


```{r}
m
# Element-wise operations
m * c(1,2,3)
```

matrix/ vector multiplication

```{r}
m
# Matrix multiplication
m %*% c(1,2,3,4)
```

Good explanation of Matrix multiplication:
[Eli Bendersky's website](https://eli.thegreenplace.net/2015/visualizing-matrix-multiplication-as-a-linear-combination/)



## Data frames

```{r}
# Creating a data frame
family.frame <- data.frame(index = 1:3, firstname = myfamily, surname = rep("Meier", 3))
```
```{r}
library(knitr)
kable(family.frame)
```


# Preparing your data for analysis

## Reading the data

Usually, you would use the "csv" format to read data into R. Sometimes you have other formats, e.g. "txt". Optimally, you have exactly one line for the header.
Make sure to choose the correct separator. You also have to tell R whether your file has a header or not.
```{r error=T}
df <- read.table("beedata.csv")
```

That did not work. Oh, yes, the separator!

```{r}
df <- read.table("beedata.csv", sep = ",")
```
```{r echo=F}
library(knitr)
kable(head(df, n = 5))
```

Looks ok, but we forgot the heading:

```{r}
df <- read.table("beedata.csv", sep = ",", header = T)
```
```{r  echo=F}
library(knitr)
kable(head(df, n = 5))
```

That looks good!

Alternatively, we can use "read.csv". In this case, the default parameters work for us.
```{r}
df <- read.csv("beedata.csv")
```

Take care, sometimes you don't get an error, but your data is screwed up anyway:
![sample Data](images/sampleData.png)
```{r}
df <- read.csv("sampleData.csv", sep = " ")
```
```{r  echo=F}
library(knitr)
kable(head(df, n = 5))
```

Use the head function to inspect your data.

## Transforming entire columns

The columns/ variables of a dataframe basically behave like entries in a named list. Each element of this list is a vector of a specific type.
You can inspect the structure of a dataframe using the function "str".
```{r}
df <- read.csv("beedata.csv")
str(df)
```

E.g. converting kg to g:
```{r}
df$weight_g <- df$weight_kg*1000
```

Marking high temperature values:
```{r}
df$highTemp <- df$t_i_1>25
```
```{r  echo=F}
library(knitr)
kable(head(df[,-(5:9)], n = 5))
```

Dealing with the timestamp (nanoseconds to seconds)
```{r}
df$time <- as.POSIXct(df$time/1000000000, origin="1970-01-01")
```
```{r  echo=F}
library(knitr)
kable(head(df, n = 5))
```

## Subsetting data

You can use squared brackets and boolean expressions to choose lines and columns: df[\<expression to choose lines\>,\<expression to choose columns\>]

Select and plot data from hive 4:

```{r}
df.4 <- df[df$hive==4,]
plot(df.4$time, df.4$t_o, ylim=c(0,40))
```
Select only the first 1000 lines:
```{r}
df.4.sub <- df.4[1:1000,]
plot(df.4.sub$time, df.4.sub$t_o, ylim=c(0,40))
```
Delete columns: / Choose only some columns

```{r}
names(df)
df.some <- df[, c(1,2,10)]
```
```{r  echo=F}
library(knitr)
kable(head(df.some, n = 5))
```

You can also use: 
```{r}
df.same <- df[, c("time", "hive", "weight_kg")]
```
```{r  echo=F}
library(knitr)
kable(head(df.same, n = 5))
```
or 
```{r}
df.or <- df[, -c(3:9,11)]
```
```{r  echo=F}
library(knitr)
kable(head(df.or, n = 5))
```

## Useful functions
- summary
- head
- tail
- sum
- mean



# Conditions - If and Else

## If

![](images/if.png){width=75%}

Simple if:
```{r}
a <- 19
if(a >= 18){
  print("Yes, you are allowed to see this content.")
}
```

if and else:

![](images/ifelse.png){width=75%}


```{r}
goodWeather <- TRUE
if (goodWeather){
  print("Let's go to the beach!")
} else{
  print("Let's eat pizza.")
}

```


Else-if: 

![](images/elseif.png){width=75%}


```{r}

do.you.want.this <- "chocolate"
#do.you.want.this <- "cookies"
#do.you.want.this <- "carrots"
if (do.you.want.this == "chocolate"){
  print("Yes, I love chocolate!")
} else if (do.you.want.this == "cookies"){
  print("Yes, if they are with chocolate.")
} else {
  print("Hm, not sure. Do you have chocolate?")
}
```


```{r include=F}
library(lubridate)
```


```{r}
library(lubridate)
birthday <- as.POSIXct("2019-08-06 10:50:18")
what.do.you.want.to.know <- "year"
#what.do.you.want.to.know <- "month"
#what.do.you.want.to.know <- "chocolate"
if (what.do.you.want.to.know == "year"){
  print(year(birthday))
} else if (what.do.you.want.to.know == "month"){
  print(month(birthday))
} else {
  print("Sorry, what do you want to know?")
}
```


# For and While

## For


```{r}
ages <- c(31, 29, 5)
total.age <- 0
for(i in ages){
  total.age <- total.age + i
}
total.age
```

```{r}
myfamily <- list(Paula=31, Ali=29, Emma=5)
age <- 0
for (i in 1:length(myfamily)){
  age <- age + myfamily[[i]]
}
age
```

apply it to columns:

```{r}
for (i in 1:ncol(df)){
  name <- names(df)[i]
  colclass <- class(df[,i])
  print(paste(name, ":", colclass))
}
```

transform long code ...

```{r eval=F}
df <- read.csv("beedata.csv")
print("mean of h:")
mean(df$h)
print("mean of t_i_1:")
mean(df$t_i_1)
print("mean of weight_kg:")
mean(df$weight_kg)
print("missing h values:")
sum(is.na(df$h))
print("missing t_i_1 values:")
sum(is.na(df$h))
print("missing weight_kg values:")
sum(is.na(df$h))
```

... into shorter code

```{r eval=F}
to.analyse <- c("h", "t_i_1", "weight_kg")
for(i in 1:length(to.analyse)){
  print(paste("mean of", to.analyse[i], ":"))
  print(mean(to.analyse[i]))
  print(paste("missing", to.analyse[i], "values:"))
  print(sum(is.na(df[,to.analyse[i]])))
}
```



![](images/for.png){width=75%}



## While
```{r}
a <- 2
while (a<1000){
  a <- a^2
  print(a)
}
```


![](images/while.png){width=75%}

## Do-while
```{r}
a <- 10
repeat{
  a <- a - sqrt(a)
  print(a)
  if(sqrt(a) > a){
    break
  }
}
```

bad example for Do-while:
```{r error=TRUE}
a <- 0.5
repeat{
  a <- a - sqrt(a)
  print(a)
  if(sqrt(a) > a){
    break
  }
}
```


![](images/do-while.png){width=75%}

## For vs. While
- if you do not know the number of iterations needed, use while
- else, always use for!
- it is easier to create a while-loop that never stops, than creating a for-loop that never stops.


# Writing own functions

The basic syntax to write functions looks like this:
```{r}
myPlus <- function(a, b){
  return(a + b)
}
myPlus(1,2)
```
You can use default parameters:
```{r}
myPlus <- function(a=1, b=1){
  return(a + b)
}
myPlus(1,2)
myPlus()
```
If you want to return multiple values, you can use a named list:
```{r}
dataframe.info <- function(df){
  cells.count <- ncol(df)*nrow(df)
  return(list(columns=ncol(df), rows=nrow(df), cells= cells.count))
}
dataframe.info(df)
```

# Debugging

Here is a buggy version of the function “funny.words.”


```{r}
funny.words <- function(s = "summer", count = 3){
  s = "summer"
  s1 <- ""
  for(i in 1:2){
    for(i in 1:count){
      print(substr(s, i, i))
      s1 <- paste(s1, substr(s, i, i), sep = "")
    }
  }
  return(s1)
}
funny.words()

funny.words("banana", 5)
```


Since this code consists of a function and a loop, we cannot run line by line to find the mistakes. However, we can create breakpoints by clicking next to the line numbering (left).

![](images/debugging.jpeg)

Then we can click on “source”" to execute all code until the first breakpoint.

![](images/source.jpeg)

In the console, there will be a new set of buttons:

![](images/buttons.jpeg)


- If you click on Next, R will execute the next line. (e.g. line 6)
- If you click on the second button, R will step in the current function call, so it will basically jump into an other function. (e.g. into the print function)
- If you click on the third button, R will execute the rest of the current function or loop. (e.g. line 6 and 7)
- If you click on “continue”“, R will run until we come across the next breakpoint. (e.g. in the next round of the loop or in line 10)
- If you click on “Stop”, R will exit the debug mode.


# Piping

Some people complain about all the brackets in the R syntax. With using pipes you can get rid of them. Pipes might also reflect your workflow more naturally than the usual syntax.

You can find the standard pipe operator in multiple packages, but you will get more options when using the magrittr packages.

Here is a simple example

```{r}
library(magrittr)
x <- 9
# Calculate the square-root of x
sqrt(x)

# Calculate it using pipes
x %>% sqrt

```


In case you want to update x, you can use:

```{r}
x <- 9
# Calculate the square-root of x and update x
x <- sqrt(x)
x

# Calculate it using pipes
x <- 9
x %<>% sqrt
x
```

This is an example with two functions and additional arguments:

```{r}

nrow(subset(df, hive==4))

df %>% subset(hive==4) %>% nrow

```

# Plotting

## Standard library

```{r}
# subset data
df.4 <- df[df$hive==4,]
# plot temperature outside
plot(df.4$time, df.4$t_o, ylim=c(0,40),type = 'p', pch=4)
# choose colours
cl <- rainbow(5)
# choose colums
cols <- 4:8
# plot each column
for (i in 1:5){
    lines(df.4$time, df.4[,cols[i]],col = cl[i],type = 'p', pch=4, ylim=c(0,40))
}
# add legend
legend("topright", legend=c(1, 2, 3, 4, 5, "outside"),
       col=c(cl, "black"), pch = 4, lty = 0, cex=0.8)
```

### Points or Lines? (type)

- “p”: Points
- “l”: Lines
- “b”: Both

### Line type (lty)

![http://www.sthda.com/english/wiki/line-types-in-r-lty](images/linetypes.png)

Source: [http://www.sthda.com/english/wiki/line-types-in-r-lty](http://www.sthda.com/english/wiki/line-types-in-r-lty)
### Point types (pch)
![http://www.sthda.com/english/wiki/r-plot-pch-symbols-the-different-point-shapes-available-in-r](images/dottypes.png)

Source: [http://www.sthda.com/english/wiki/r-plot-pch-symbols-the-different-point-shapes-available-in-r](http://www.sthda.com/english/wiki/r-plot-pch-symbols-the-different-point-shapes-available-in-r)

## ggplot

```{r}
# subset data
df.4 <- df[df$hive==4,]
# choose columns
df.4.cols <- df.4[,c(1,4:9)]
# reshape data
library(reshape)
mdf <- melt(df.4.cols, id=c("time")) 
# plot data
library(ggplot2)
ggplot(data = mdf, aes(x=time, y=value)) + geom_line(aes(colour=variable)) + ylim(c(0, 40))
```

# R-Markdown

R-Markdown is a great way to document your code and to write reports about your results.

## Output formats

The three most common output options for R-Markdown are html, pdf and word documents. If you want to create pdf documents, tex needs to be installed. 

![](images/rmarkdown.jpeg)

Depending on your choice, your document will have a different output parameter in the header:

```{r eval=F}
# html
---
title: "R Markdown Example"
author: "Somebody"
date: "August 14, 2019"
output: html_document
---
```

```{r eval=F}

# word
---
title: "R Markdown Example"
author: "Somebody"
date: "August 14, 2019"
output: word_document
---

```


There are many formatting options which you can specify in the header. This might be useful options for html:

```{r eval=F}

---
title: "R Markdown Example"
author: "Somebody"
date: "August 14, 2019"
output: 
  html_document:
    toc: true # print a table of content
    toc_depth: 2 # maximal depth of the table of content
    number_sections: true # automatically number the sections
    code_folding: hide # have buttons to show/ hide code
    theme: united # specify a theme
    toc_float: true # generate a navigation bar

---
```


## Document structure

If you create a new R-Markdown document, R will automatically create a sample document. You can see that “#” can be used to create headers:

```{r eval=F}
# Header 1

## Header 2

### Header 3
```



## Code blocks

For code blocks you have many options, too. The two most important are:

- eval: If set to true, the code will be executed
- echo: If set to true, the output will be printed/ plotted

## Lists

```{r eval=F}
* unordered list
    + sub-item 1
    + sub-item 2
        - sub-sub-item 1
* item 2

    
1. ordered list
2. item 2
    i) sub-item 1

```

* unordered list
    + sub-item 1
    + sub-item 2
        - sub-sub-item 1
* item 2

    
1. ordered list
2. item 2
    i) sub-item 1



## Insert an image

```{r eval=F}
![Some description](path to image)
```

## Further documentation

A good documentation of R-Markdown can be found here: [https://bookdown.org/yihui/rmarkdown/](https://bookdown.org/yihui/rmarkdown/)

And here is a cheat sheet: 
[https://www.rstudio.com/resources/cheatsheets/#rmarkdown](https://www.rstudio.com/resources/cheatsheets/#rmarkdown)

There is a lot more you can do with R-Markdown, see:
[https://rmarkdown.rstudio.com/gallery.html](https://rmarkdown.rstudio.com/gallery.html)


# Short statistical breaks

## Dinosours

[https://www.autodeskresearch.com/publications/samestats](https://www.autodeskresearch.com/publications/samestats)

## Stars
![Stars](images/stars.jpeg)
I don't know the source, I hope you forgive me.

## Correlation
[https://www.tylervigen.com/spurious-correlations](https://www.tylervigen.com/spurious-correlations)

## Beginners Book
[https://devhumor.com/content/uploads/images/July2020/Best-Programming-Book-for-Beginners-Comic---Art-of-Searching-on-Google.png](https://devhumor.com/content/uploads/images/July2020/Best-Programming-Book-for-Beginners-Comic---Art-of-Searching-on-Google.png)

<!-- # Linear Model -->
<!-- sub <- df[df$t_o<50 & df$t_o>-10,] -->
<!-- plot(sub$h, sub$t_o) -->
<!-- mod <-lm(sub$h~ sub$t_o) -->
<!-- summary(mod) -->


<!-- wir, kolja, micha und ralf -->