Data Analysis Patterns

Reading in multiple files

Suppose you have files with semantic names, like sampleA_rep01.tsv, sampleA_rep02.tsv, ..., sampleC_rep01.tsv. You want to load in and combine all data, and extract relevant metadata into columns. How do you do this? Tidyverse to the rescue:

### example setup:
DIR <- 'path/to/data' # change to directory you can write files to.
# filenames to make example work:
files <- c('sampleA_rep01.tsv', 'sampleA_rep02.tsv','sampleB_rep01.tsv', 
           'sampleB_rep02.tsv', 'sampleC_rep01.tsv', 'sampleC_rep02.tsv')

# write test files for example (iris a bunch of times)
walk(files, ~ write_tsv(iris, file.path(DIR, .)))

### Pattern:
# grab all files programmatically: 
input_files <- list.files(DIR, 
                          pattern='sample.*\\.tsv', full.names=TRUE)

# data loading pattern:
all_data <- tibble(file=input_files) %>% 
   # read data in (note: in general, best to 
   # pass col_names and col_types to map)
   mutate(data=map(file, read_tsv)) %>% 
   # get the file basename (no path); if 
   # your metadata is in the path, change accordingly!
   mutate(basename=basename(file)) %>% 
   # extract out the metadata from the base filename
   extract(basename, into=c('sample', 'rep'), 
           regex='sample([^_]+)_rep([^_]+)\\.tsv') %>% 
   unnest(data)  # optional, depends on what you need.

Before the unnest(), the data looks like:

# A tibble: 6 x 4
  file              data               sample rep
* <chr>             <list>             <chr>  <chr>
1 sampleA_rep01.tsv <tibble [150 × 5]> A      01
2 sampleA_rep02.tsv <tibble [150 × 5]> A      02
3 sampleB_rep01.tsv <tibble [150 × 5]> B      01
4 sampleB_rep02.tsv <tibble [150 × 5]> B      02
5 sampleC_rep01.tsv <tibble [150 × 5]> C      01
6 sampleC_rep02.tsv <tibble [150 × 5]> C      02