diff --git a/RNA-seq/01-qc_trim_quant.nb.html b/RNA-seq/01-qc_trim_quant.nb.html new file mode 100644 index 00000000..238a7cfd --- /dev/null +++ b/RNA-seq/01-qc_trim_quant.nb.html @@ -0,0 +1,591 @@ + + + + + + + + + + + + + + + +Introduction to bulk RNA-seq data processing + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Navigate the terminal interface
    +
    • +
  • Organize an analysis project
    • +
  • Apply FastQC for quality control analysis of Illumina sequencing data
    • +
  • Preprocess sequencing reads with fastp
    • +
  • Quantify RNA-seq expression with Salmon
    • +
+
+

We will first learn how to process RNA-seq data at the command line using samples that were assayed with paired-end sequencing.

+

These samples come from a project (PRJNA178120) that includes 8 samples from normal gastric tissue, gastric cancer cell lines and primary gastric tumor cell cultures.

+

Here we will perform quality control checks, trimming, and estimate the transcript abundances for a single sample, SRR585570.

+

+

Later, we will use the full dataset (n = 8) to explore how to summarize estimates to the gene level and do some exploratory data analyses with data the course directors have processed ahead of time.

+
+

We’ll first want to set our working directory to the top-level of the RNA-seq folder.

+

Copy and paste the text in the code blocks below into your Terminal window in RStudio. It should be in the lower left hand corner as a tab next to Console.

+

Set current directory to the top-level of the RNA-seq module:

+
cd ~/training-modules/RNA-seq
+

Here ~/ refers to your home directory on the RStudio Server, which is the base folder in which your files live, including most of the materials for training. This is also the default working directory when you open a new RStudio session. A home directory is specific to you as a user on the RStudio Server; each user has their own folder to store their files.

+

Because these steps are computationally time intensive, we’ve prepared a script to start running things. Once we start running the script, we will give a short lecture to introduce this module and then walk through and explain each of the individual steps that the script is executing.

+

Enter the following in the Terminal to start running the script:

+
bash scripts/run_SRR585570.sh
+

Note: Don’t worry if the Salmon step does not complete by the time we move on to the next notebook. This is a time and resource intensive step, so we have prepared the required output in case we need it.

+
+

Input files

+

The raw data FASTQ files (fastq.gz) for this sample, SRR585570, are in data/gastric-cancer/fastq/SRR585570. The first two directories, data/ and gastric-cancer/, tell us that these files are data and which experiment or dataset these data are from. We’ll be working with an additional dataset later in the module, so this latter distinction will become more important. The third directory, fastq/, tells us that this is where we will be storing fastq files.

+

The final directory, SRR585570, is specific to the sample we are working with. The use of the SRR585570 folder might seem unnecessary because we are only processing a single sample here, but keeping files for individual samples in their own folder helps keep things organized for multi-sample workflows. (You can peek ahead and look at the data/NB-cell/quant folder for such an example.)

+

There is no “one size fits all” approach for project organization. It’s most important that it’s consistent, easy for you and others to find the files you need quickly, and minimizes the likelihood for errors (e.g., writing over files accidentally).

+
+
+
+

Quality control with FastQC

+

+

The first thing our script does is use FastQC for quality control in command line mode. Here’s a link to the FastQC documentation: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/

+

Let’s take a look at some example reports from the authors of FastQC:

+ +

FastQC runs a series of quality checks on sequencing data and provides an HTML report. As the authors point out in the docs:

+
+

It is important to stress that although the analysis results appear to give a pass/fail result, these evaluations must be taken in the context of what you expect from your library.

+
+

The documentation for individual modules/analyses in FastQC is a great resource!

+

To save time, our script only runs one FASTQ file for SRR585570 with the following commands:

+
mkdir -p QC/gastric-cancer/fastqc/SRR585570
+

mkdir allows us to create a new folder in the QC/gastric-cancer/fastqc directory specifically to hold the report information that will be generated by FastQC for this sample. The -p allows us to create parent directories and will prevent an error if the directory we specify already exists.

+
# In the interest of time, we'll run one of the fastq files through FastQC
+fastqc data/gastric-cancer/fastq/SRR585570/SRR585570_1.fastq.gz \
+    -o QC/gastric-cancer/fastqc/SRR585570
+
+

-o

+

The -o flag allows us to specify where the output of FastQC is saved. Note that this is saved in a separate place than the raw data files and in a directory specifically for quality control information.

+

For comparison to the report for SRR585570_1.fastq.gz we generate with our script, we’ve prepared a FastQC report for one of the sets of reads for another sample in the experiment. It can be found at QC/gastric-cancer/fastqc/SRR585574/SRR585574_1_fastqc.html.

+

Let’s look at the reports for both samples.

+
+
+
+

Preprocessing with fastp

+

+

We use fastp to preprocess the FASTQ files (Chen et al. Bioinformatics. 2018.). Note that fastp has quality control functionality and many different options for preprocessing (see all options on GitHub), most of which we will not cover. Here, we focus on adapter trimming, quality filtering, and length filtering.

+

Below, we discuss the commands we used in the script.

+
+

Create output directories

+
# Create a directory to hold the trimmed fastq files
+mkdir -p data/gastric-cancer/fastq-trimmed/SRR585570
+# Create a directory to hold the QC output from fastp
+mkdir -p QC/gastric_cancer/fastp/SRR585570
+

As we’ll cover below, fastp essentially has two kinds of output: trimmed and filtered FASTQ files (data) and reports (quality control).

+
+
+

fastp

+
# Run the adapter and quality trimming step -- also produces QC report
+fastp -i data/gastric-cancer/fastq/SRR585570/SRR585570_1.fastq.gz \
+    -I data/gastric-cancer/fastq/SRR585570/SRR585570_2.fastq.gz \
+    -o data/gastric-cancer/fastq-trimmed/SRR585570/SRR585570_fastp_1.fastq.gz \
+    -O data/gastric-cancer/fastq-trimmed/SRR585570/SRR585570_fastp_2.fastq.gz \
+    --qualified_quality_phred 15 \
+    --length_required 20 \
+    --report_title "SRR585570" \
+    --json QC/gastric-cancer/fastp/SRR585570/SRR585570_fastp.json \
+    --html QC/gastric-cancer/fastp/SRR585570/SRR585570_fastp.html
+

Below, we’ll walk through the arguments/options we used to run fastp. By default, fastp performs adapter trimming, which you can read more about here. For paired-end data like the data we have for SRR585570, adapters can be detected automatically without specifying an adapter sequence.

+
+

Input: -i and -I

+

These arguments specify the read1 input and read2 (sometimes called left and right) input, respectively.

+
+
+

fastq output: -o and -O

+

These arguments specify the read1 output and read2 output, respectively. Note that the output is being placed in data/gastric-cancer/fastq-trimmed/SRR585570/, so the processed FASTQ files will be kept separate from from the original files. It is generally good practice to treat your “raw” data and its directories as fixed and separate from any processing and analysis that you do, to prevent accidentally modification of those original files. And in the event that you accidentally do modify the originals, you know exactly which files and directories to reset.

+
+
+

--qualified_quality_phred

+

Phred scores are the quality information included in a FASTQ file and the values indicate the chances that a base is called incorrectly. Let’s look at a screenshot of the Per Base Sequence Quality module from FastQC bad Illumina example we linked to above.

+
+ +

per-base-quality-screenshot

+
+

Anything below 20, where a Phred score of 20 represents a 1 in 100 chance that the call is incorrect, is considered poor quality by FastQC. Using --qualified_quality_phred 15 (which is the default), means scores >= 15 are considered “qualified.” Using the default parameters as we do here, reads will be filtered out if >40% of the bases are unqualified. You can read more about the quality filtering functionality of fastp here. The Salmon documentation notes that, given the way we run salmon quant, quantification may be more sensitive to calls that are likely to be erroneous (of low quality) and, therefore, quality trimming may be important.

+

Trimming, in contrast to filtering, refers to removing low quality base calls from the (typically 3’) end of reads. A recent paper from the Salmon authors (Srivastava et al. 2020) notes that trimming did not affect mapping rates from random publicly available human bulk (paired-end) RNA-seq samples (they used TrimGalore). fastp does have the functionality to perform trimming using a sliding window, which must be enabled. We are not using it here.

+

Note that there are two kinds of encoding for Phred scores: Phred 33 encoding and Phred 64 encoding. FastQC guessed that the file for SRR585570 uses Sanger/Illumina 1.9 encoding (Phred 33). If we had Phred 64 data, we’d use the --phred64 flag. You can read a little bit more about the encoding here.

+
+
+

--length_required

+

Trimming reads may result in short reads, which may affect gene expression estimates (Williams et al. BMC Bioinformatics. 2016.). Using --length_required 20 means that reads shorter than 20bp will be discarded (similar to what was used in Srivastava et al. above).

+
+
+

--report_title

+

When we look at the HTML report, it’s helpful to quickly identify what sample the report is for. Using --report title "SRR585570" means that the report will be titled “SRR585570” rather than the default (“fastp report”).

+
+
+

--json and --html

+

With these options, we’re specifying where the JSON and HTML reports will be saved (in the QC/gastric-cancer/fastp/ directory we created) and what the filenames will be. Including the sample name in the filenames again may help us with project organization.

+

If we look at QC/gastric-cancer/fastp/SRR585570_fastp.json or the top of the HTML report, we can see that fastp reports certain metrics before and after filtering, which can be very useful in making analysis decisions.

+
+
+
+
+

Quantification with Salmon

+

+

We’ll use Salmon for quantifying transcript expression (documentation). Salmon (Patro, et al. Nature Methods. 2017.) is fast and requires very little memory, which makes it a great choice for running on your laptop during training. We can use the output for downstream analyses like differential expression analysis and clustering. We use Salmon in mapping mode, with mapping validation enabled, using the following command:

+
# We perform quantification on the files that have been trimmed
+# and use the index generated with -k 23, as this may "improve sensitivity"
+# per the Salmon documentation
+salmon quant -i index/Homo_sapiens/short_index \
+    -l A \
+    -1 data/gastric-cancer/fastq-trimmed/SRR585570/SRR585570_fastp_1.fastq.gz \
+    -2 data/gastric-cancer/fastq-trimmed/SRR585570/SRR585570_fastp_2.fastq.gz \
+    -o data/gastric-cancer/salmon_quant/SRR585570 \
+    --validateMappings --rangeFactorizationBins 4 \
+    --gcBias --seqBias \
+    --threads 4
+

Below, we’ll walk through the arguments/options we used to run salmon quant.

+
+

Transcriptome index: -i

+

Salmon requires a set of transcripts (what we want to quantify) in the form of a transcriptome index built with salmon index. Building an index can take a while (but you only have to do it once!), so we’ve built the one we use today ahead of time. Before we use it, we’ll take a moment to give a bit of background.

+

You can see how we obtained this index and others on GitHub. Note that we used Homo sapiens GRCh38, Ensembl release 95. It is important to keep track of what build, resource, and files were used and putting our shell scripts on GitHub allows us to do that.

+

The salmon index command has a parameter -k which sets the k-mer length. The index we used was built with -k 23 and can be found here:

+
index/Homo_sapiens/short_index
+

Using a smaller value for k than the default (k = 31) is appropriate for shorter reads and may improve sensitivity when using --validateMappings according to the Salmon documentation.

+
+
+

-l

+

We use -l A to allow Salmon to automatically infer the library type based on a subset of reads, but you can also provide the library type to Salmon with this argument.

+
+
+

Input: -1 and -2

+

These data are paired-end, we use -1 and -2 to specify read1 and read2, respectively.

+
+
+

-o

+

Output directory, salmon quant should create this for us if it doesn’t exist yet.

+
+
+

--validateMappings and --rangeFactorizationBins

+

Using --validateMappings enables mapping validation, where Salmon checks its mappings using traditional alignment. This helps prevent “spurious mappings” where a read maps to a target but does not arise from it (see documentation for flag and the release notes for v0.10.0 where this was introduced).

+

When enabling mapping validation with --validateMappings, setting --rangeFactorizationBins 4 can improve quantification for certain classes of transcripts (docs).

+
+
+

--gcBias

+

With this option enabled, Salmon will attempt to correct for fragment GC-bias. Regions with high or low GC content tend to be underrepresented in sequencing data.

+

It should be noted that this is only appropriate for use with paired-end reads, as fragment length can not be inferred from single-end reads (see this GitHub issue).

+
+
+

--seqBias

+

With this option enabled, Salmon will attempt to correct for the bias that occurs when using random hexamer priming (preferential sequencing of reads when certain motifs appear at the beginning).

+
+
+

--threads

+

The --threads argument controls the number of threads that are available to Salmon during quantification. This in essence controls how much of the mapping can occur in parallel. If you had access to a computer with many cores, you could increase the number of threads to make quantification go faster.

+

Navigate to data/gastric-cancer/salmon_quant/SRR585571/aux_info and open meta_info.json. Look for a field called percent_mapped – what value does this sample have?

+ +
+
+ +
LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIGJ1bGsgUk5BLXNlcSBkYXRhIHByb2Nlc3NpbmciCmF1dGhvcjogQ0NETCBmb3IgQUxTRgpkYXRlOiAyMDIxCm91dHB1dDogICAKICBodG1sX25vdGVib29rOiAKICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OiB0cnVlCi0tLQoKIyMgT2JqZWN0aXZlcwoKVGhpcyBub3RlYm9vayB3aWxsIGRlbW9uc3RyYXRlIGhvdyB0bzoKCi0gTmF2aWdhdGUgdGhlIHRlcm1pbmFsIGludGVyZmFjZSAgCi0gT3JnYW5pemUgYW4gYW5hbHlzaXMgcHJvamVjdCAKLSBBcHBseSBbRmFzdFFDXShodHRwczovL3d3dy5iaW9pbmZvcm1hdGljcy5iYWJyYWhhbS5hYy51ay9wcm9qZWN0cy9mYXN0cWMvKSBmb3IgcXVhbGl0eSBjb250cm9sIGFuYWx5c2lzIG9mIElsbHVtaW5hIHNlcXVlbmNpbmcgZGF0YQotIFByZXByb2Nlc3Mgc2VxdWVuY2luZyByZWFkcyB3aXRoIFtmYXN0cF0oaHR0cHM6Ly9naXRodWIuY29tL09wZW5HZW5lL2Zhc3RwKQotIFF1YW50aWZ5IFJOQS1zZXEgZXhwcmVzc2lvbiB3aXRoIFtTYWxtb25dKGh0dHBzOi8vY29tYmluZS1sYWIuZ2l0aHViLmlvL3NhbG1vbi8pCgotLS0KCldlIHdpbGwgZmlyc3QgbGVhcm4gaG93IHRvIHByb2Nlc3MgUk5BLXNlcSBkYXRhIGF0IHRoZSBjb21tYW5kIGxpbmUgdXNpbmcgc2FtcGxlcyB0aGF0IHdlcmUgYXNzYXllZCB3aXRoIHBhaXJlZC1lbmQgc2VxdWVuY2luZy4KClRoZXNlIHNhbXBsZXMgY29tZSBmcm9tIGEgcHJvamVjdCAoW2BQUkpOQTE3ODEyMGBdKGh0dHBzOi8vd3d3LmViaS5hYy51ay9lbmEvZGF0YS92aWV3L1BSSk5BMTc4MTIwKSkgdGhhdCBpbmNsdWRlcyA4IHNhbXBsZXMgZnJvbSBub3JtYWwgZ2FzdHJpYyB0aXNzdWUsIGdhc3RyaWMgY2FuY2VyIGNlbGwgbGluZXMgYW5kIHByaW1hcnkgZ2FzdHJpYyB0dW1vciBjZWxsIGN1bHR1cmVzLgoKSGVyZSB3ZSB3aWxsIHBlcmZvcm0gcXVhbGl0eSBjb250cm9sIGNoZWNrcywgdHJpbW1pbmcsIGFuZCBlc3RpbWF0ZSB0aGUgdHJhbnNjcmlwdCBhYnVuZGFuY2VzIGZvciBhIHNpbmdsZSBzYW1wbGUsIFNSUjU4NTU3MC4KCiFbXShkaWFncmFtcy9ybmEtc2VxXzEucG5nKQoKTGF0ZXIsIHdlIHdpbGwgdXNlIHRoZSBmdWxsIGRhdGFzZXQgKG4gPSA4KSB0byBleHBsb3JlIGhvdyB0byBzdW1tYXJpemUgZXN0aW1hdGVzIHRvIHRoZSBnZW5lIGxldmVsIGFuZCBkbyBzb21lIGV4cGxvcmF0b3J5IGRhdGEgYW5hbHlzZXMgd2l0aCBkYXRhIHRoZSBjb3Vyc2UgZGlyZWN0b3JzIGhhdmUgcHJvY2Vzc2VkIGFoZWFkIG9mIHRpbWUuCgotLS0KCldlJ2xsIGZpcnN0IHdhbnQgdG8gc2V0IG91ciB3b3JraW5nIGRpcmVjdG9yeSB0byB0aGUgdG9wLWxldmVsIG9mIHRoZSBSTkEtc2VxIGZvbGRlci4KCkNvcHkgYW5kIHBhc3RlIHRoZSB0ZXh0IGluIHRoZSBjb2RlIGJsb2NrcyBiZWxvdyBpbnRvIHlvdXIgYFRlcm1pbmFsYCB3aW5kb3cgaW4gUlN0dWRpby4KSXQgc2hvdWxkIGJlIGluIHRoZSBsb3dlciBsZWZ0IGhhbmQgY29ybmVyIGFzIGEgdGFiIG5leHQgdG8gYENvbnNvbGVgLgoKKipTZXQgY3VycmVudCBkaXJlY3RvcnkgdG8gdGhlIHRvcC1sZXZlbCBvZiB0aGUgUk5BLXNlcSBtb2R1bGU6KioKCmBgYGJhc2gKY2Qgfi90cmFpbmluZy1tb2R1bGVzL1JOQS1zZXEKYGBgCgpIZXJlIGB+L2AgcmVmZXJzIHRvIHlvdXIgX2hvbWUgZGlyZWN0b3J5XyBvbiB0aGUgUlN0dWRpbyBTZXJ2ZXIsIHdoaWNoIGlzIHRoZSBiYXNlIGZvbGRlciBpbiB3aGljaCB5b3VyIGZpbGVzIGxpdmUsIGluY2x1ZGluZyBtb3N0IG9mIHRoZSBtYXRlcmlhbHMgZm9yIHRyYWluaW5nLgpUaGlzIGlzIGFsc28gdGhlIGRlZmF1bHQgd29ya2luZyBkaXJlY3Rvcnkgd2hlbiB5b3Ugb3BlbiBhIG5ldyBSU3R1ZGlvIHNlc3Npb24uCkEgaG9tZSBkaXJlY3RvcnkgaXMgc3BlY2lmaWMgdG8geW91IGFzIGEgdXNlciBvbiB0aGUgUlN0dWRpbyBTZXJ2ZXI7IGVhY2ggdXNlciBoYXMgdGhlaXIgb3duIGZvbGRlciB0byBzdG9yZSB0aGVpciBmaWxlcy4KCioqQmVjYXVzZSB0aGVzZSBzdGVwcyBhcmUgY29tcHV0YXRpb25hbGx5IHRpbWUgaW50ZW5zaXZlLCB3ZSd2ZSBwcmVwYXJlZCBhIHNjcmlwdCB0byBzdGFydCBydW5uaW5nIHRoaW5ncy4qKgpPbmNlIHdlIHN0YXJ0IHJ1bm5pbmcgdGhlIHNjcmlwdCwgd2Ugd2lsbCBnaXZlIGEgc2hvcnQgbGVjdHVyZSB0byBpbnRyb2R1Y2UgdGhpcyBtb2R1bGUgYW5kIHRoZW4gd2FsayB0aHJvdWdoIGFuZCBleHBsYWluIGVhY2ggb2YgdGhlIGluZGl2aWR1YWwgc3RlcHMgdGhhdCB0aGUgc2NyaXB0IGlzIGV4ZWN1dGluZy4KCioqRW50ZXIgdGhlIGZvbGxvd2luZyBpbiB0aGUgVGVybWluYWwgdG8gc3RhcnQgcnVubmluZyB0aGUgc2NyaXB0OioqCgpgYGBiYXNoCmJhc2ggc2NyaXB0cy9ydW5fU1JSNTg1NTcwLnNoCmBgYAoKX05vdGU6IERvbid0IHdvcnJ5IGlmIHRoZSBTYWxtb24gc3RlcCBkb2VzIG5vdCBjb21wbGV0ZSBieSB0aGUgdGltZSB3ZSBtb3ZlIG9uIHRvIHRoZSBuZXh0IG5vdGVib29rLgpUaGlzIGlzIGEgdGltZSBhbmQgcmVzb3VyY2UgaW50ZW5zaXZlIHN0ZXAsIHNvIHdlIGhhdmUgcHJlcGFyZWQgdGhlIHJlcXVpcmVkIG91dHB1dCBpbiBjYXNlIHdlIG5lZWQgaXQuXwoKIyMjIElucHV0IGZpbGVzCgpUaGUgcmF3IGRhdGEgRkFTVFEgZmlsZXMgKGBmYXN0cS5nemApIGZvciB0aGlzIHNhbXBsZSwgU1JSNTg1NTcwLCBhcmUgaW4gYGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEvU1JSNTg1NTcwYC4KVGhlIGZpcnN0IHR3byBkaXJlY3RvcmllcywgYGRhdGEvYCBhbmQgYGdhc3RyaWMtY2FuY2VyL2AsIHRlbGwgdXMgdGhhdCB0aGVzZSBmaWxlcyBhcmUgZGF0YSBhbmQgd2hpY2ggZXhwZXJpbWVudCBvciBkYXRhc2V0IHRoZXNlIGRhdGEgYXJlIGZyb20uCldlJ2xsIGJlIHdvcmtpbmcgd2l0aCBhbiBhZGRpdGlvbmFsIGRhdGFzZXQgbGF0ZXIgaW4gdGhlIG1vZHVsZSwgc28gdGhpcyBsYXR0ZXIgZGlzdGluY3Rpb24gd2lsbCBiZWNvbWUgbW9yZSBpbXBvcnRhbnQuClRoZSB0aGlyZCBkaXJlY3RvcnksIGBmYXN0cS9gLCB0ZWxscyB1cyB0aGF0IHRoaXMgaXMgd2hlcmUgd2Ugd2lsbCBiZSBzdG9yaW5nIGZhc3RxIGZpbGVzLgoKVGhlIGZpbmFsIGRpcmVjdG9yeSwgYFNSUjU4NTU3MGAsIGlzIHNwZWNpZmljIHRvIHRoZSBzYW1wbGUgd2UgYXJlIHdvcmtpbmcgd2l0aC4KVGhlIHVzZSBvZiB0aGUgYFNSUjU4NTU3MGAgZm9sZGVyIG1pZ2h0IHNlZW0gdW5uZWNlc3NhcnkgYmVjYXVzZSB3ZSBhcmUgb25seSBwcm9jZXNzaW5nIGEgc2luZ2xlIHNhbXBsZSBoZXJlLCBidXQga2VlcGluZyBmaWxlcyBmb3IgaW5kaXZpZHVhbCBzYW1wbGVzIGluIHRoZWlyIG93biBmb2xkZXIgaGVscHMga2VlcCB0aGluZ3Mgb3JnYW5pemVkIGZvciBtdWx0aS1zYW1wbGUgd29ya2Zsb3dzLgooWW91IGNhbiBwZWVrIGFoZWFkIGFuZCBsb29rIGF0IHRoZSBgZGF0YS9OQi1jZWxsL3F1YW50YCBmb2xkZXIgZm9yIHN1Y2ggYW4gZXhhbXBsZS4pCgpUaGVyZSBpcyBubyAib25lIHNpemUgZml0cyBhbGwiIGFwcHJvYWNoIGZvciBwcm9qZWN0IG9yZ2FuaXphdGlvbi4KSXQncyBtb3N0IGltcG9ydGFudCB0aGF0IGl0J3MgY29uc2lzdGVudCwgZWFzeSBmb3IgeW91IGFuZCBvdGhlcnMgdG8gZmluZCB0aGUgZmlsZXMgeW91IG5lZWQgcXVpY2tseSwgYW5kIG1pbmltaXplcyB0aGUgbGlrZWxpaG9vZCBmb3IgZXJyb3JzIChlLmcuLCB3cml0aW5nIG92ZXIgZmlsZXMgYWNjaWRlbnRhbGx5KS4KCiMjIFF1YWxpdHkgY29udHJvbCB3aXRoIEZhc3RRQwoKIVtdKGRpYWdyYW1zL3JuYS1zZXFfMi5wbmcpCgpUaGUgZmlyc3QgdGhpbmcgb3VyIHNjcmlwdCBkb2VzIGlzIHVzZSBbRmFzdFFDXShodHRwczovL3d3dy5iaW9pbmZvcm1hdGljcy5iYWJyYWhhbS5hYy51ay9wcm9qZWN0cy9mYXN0cWMvKSBmb3IgcXVhbGl0eSBjb250cm9sIGluIGNvbW1hbmQgbGluZSBtb2RlLgpIZXJlJ3MgYSBsaW5rIHRvIHRoZSBGYXN0UUMgZG9jdW1lbnRhdGlvbjogaHR0cHM6Ly93d3cuYmlvaW5mb3JtYXRpY3MuYmFicmFoYW0uYWMudWsvcHJvamVjdHMvZmFzdHFjL0hlbHAvCgoqKkxldCdzIHRha2UgYSBsb29rIGF0IHNvbWUgZXhhbXBsZSByZXBvcnRzIGZyb20gdGhlIGF1dGhvcnMgb2YgRmFzdFFDOioqCgoqIFtHb29kIElsbHVtaW5hIGRhdGEgZXhhbXBsZSByZXBvcnRdKGh0dHBzOi8vd3d3LmJpb2luZm9ybWF0aWNzLmJhYnJhaGFtLmFjLnVrL3Byb2plY3RzL2Zhc3RxYy9nb29kX3NlcXVlbmNlX3Nob3J0X2Zhc3RxYy5odG1sKSBmcm9tIEZhc3RRQwoqIFtCYWQgSWxsdW1pbmEgZGF0YSBleGFtcGxlIHJlcG9ydF0oaHR0cHM6Ly93d3cuYmlvaW5mb3JtYXRpY3MuYmFicmFoYW0uYWMudWsvcHJvamVjdHMvZmFzdHFjL2JhZF9zZXF1ZW5jZV9mYXN0cWMuaHRtbCkgZnJvbSBGYXN0UUMKCkZhc3RRQyBydW5zIGEgc2VyaWVzIG9mIHF1YWxpdHkgY2hlY2tzIG9uIHNlcXVlbmNpbmcgZGF0YSBhbmQgcHJvdmlkZXMgYW4gSFRNTCByZXBvcnQuIEFzIHRoZSBhdXRob3JzIHBvaW50IG91dCBpbiB0aGUgW2RvY3NdKGh0dHBzOi8vd3d3LmJpb2luZm9ybWF0aWNzLmJhYnJhaGFtLmFjLnVrL3Byb2plY3RzL2Zhc3RxYy9IZWxwLzIlMjBCYXNpYyUyME9wZXJhdGlvbnMvMi4yJTIwRXZhbHVhdGluZyUyMFJlc3VsdHMuaHRtbCk6Cgo+IEl0IGlzIGltcG9ydGFudCB0byBzdHJlc3MgdGhhdCBhbHRob3VnaCB0aGUgYW5hbHlzaXMgcmVzdWx0cyBhcHBlYXIgdG8gZ2l2ZSBhIHBhc3MvZmFpbCByZXN1bHQsIHRoZXNlIGV2YWx1YXRpb25zIG11c3QgYmUgdGFrZW4gaW4gdGhlIGNvbnRleHQgb2Ygd2hhdCB5b3UgZXhwZWN0IGZyb20geW91ciBsaWJyYXJ5LgoKVGhlIFtkb2N1bWVudGF0aW9uIGZvciBpbmRpdmlkdWFsIG1vZHVsZXMvYW5hbHlzZXNdKGh0dHBzOi8vd3d3LmJpb2luZm9ybWF0aWNzLmJhYnJhaGFtLmFjLnVrL3Byb2plY3RzL2Zhc3RxYy9IZWxwLzMlMjBBbmFseXNpcyUyME1vZHVsZXMvKSBpbiBGYXN0UUMgaXMgYSBncmVhdCByZXNvdXJjZSEKClRvIHNhdmUgdGltZSwgb3VyIHNjcmlwdCBvbmx5IHJ1bnMgb25lIEZBU1RRIGZpbGUgZm9yIFNSUjU4NTU3MCB3aXRoIHRoZSBmb2xsb3dpbmcgY29tbWFuZHM6CgpgYGBiYXNoCm1rZGlyIC1wIFFDL2dhc3RyaWMtY2FuY2VyL2Zhc3RxYy9TUlI1ODU1NzAKYGBgCgpgbWtkaXJgIGFsbG93cyB1cyB0byBjcmVhdGUgYSBuZXcgZm9sZGVyIGluIHRoZSBgUUMvZ2FzdHJpYy1jYW5jZXIvZmFzdHFjYCBkaXJlY3Rvcnkgc3BlY2lmaWNhbGx5IHRvIGhvbGQgdGhlIHJlcG9ydCBpbmZvcm1hdGlvbiB0aGF0IHdpbGwgYmUgZ2VuZXJhdGVkIGJ5IEZhc3RRQyBfZm9yIHRoaXMgc2FtcGxlXy4KVGhlIGAtcGAgYWxsb3dzIHVzIHRvIGNyZWF0ZSBfcGFyZW50XyBkaXJlY3RvcmllcyBhbmQgd2lsbCBwcmV2ZW50IGFuIGVycm9yIGlmIHRoZSBkaXJlY3Rvcnkgd2Ugc3BlY2lmeSBhbHJlYWR5IGV4aXN0cy4KCmBgYGJhc2gKIyBJbiB0aGUgaW50ZXJlc3Qgb2YgdGltZSwgd2UnbGwgcnVuIG9uZSBvZiB0aGUgZmFzdHEgZmlsZXMgdGhyb3VnaCBGYXN0UUMKZmFzdHFjIGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEvU1JSNTg1NTcwL1NSUjU4NTU3MF8xLmZhc3RxLmd6IFwKCS1vIFFDL2dhc3RyaWMtY2FuY2VyL2Zhc3RxYy9TUlI1ODU1NzAKYGBgCgojIyMjIGAtb2AKClRoZSBgLW9gIGZsYWcgYWxsb3dzIHVzIHRvIHNwZWNpZnkgd2hlcmUgdGhlIG91dHB1dCBvZiBGYXN0UUMgaXMgc2F2ZWQuCk5vdGUgdGhhdCB0aGlzIGlzIHNhdmVkIGluIGEgc2VwYXJhdGUgcGxhY2UgdGhhbiB0aGUgcmF3IGRhdGEgZmlsZXMgYW5kIGluIGEgZGlyZWN0b3J5IHNwZWNpZmljYWxseSBmb3IgcXVhbGl0eSBjb250cm9sIGluZm9ybWF0aW9uLgoKRm9yIGNvbXBhcmlzb24gdG8gdGhlIHJlcG9ydCBmb3IgYFNSUjU4NTU3MF8xLmZhc3RxLmd6YCB3ZSBnZW5lcmF0ZSB3aXRoIG91ciBzY3JpcHQsIHdlJ3ZlIHByZXBhcmVkIGEgRmFzdFFDIHJlcG9ydCBmb3Igb25lIG9mIHRoZSBzZXRzIG9mIHJlYWRzIGZvciBhbm90aGVyIHNhbXBsZSBpbiB0aGUgZXhwZXJpbWVudC4KSXQgY2FuIGJlIGZvdW5kIGF0IGBRQy9nYXN0cmljLWNhbmNlci9mYXN0cWMvU1JSNTg1NTc0L1NSUjU4NTU3NF8xX2Zhc3RxYy5odG1sYC4KCioqTGV0J3MgbG9vayBhdCB0aGUgcmVwb3J0cyBmb3IgYm90aCBzYW1wbGVzLioqCgojIyBQcmVwcm9jZXNzaW5nIHdpdGggZmFzdHAKCiFbXShkaWFncmFtcy9ybmEtc2VxXzMucG5nKQoKV2UgdXNlIFtmYXN0cF0oaHR0cHM6Ly9naXRodWIuY29tL09wZW5HZW5lL2Zhc3RwKSB0byBwcmVwcm9jZXNzIHRoZSBGQVNUUSBmaWxlcyAoW0NoZW4gZXQgYWwuIF9CaW9pbmZvcm1hdGljcy5fIDIwMTguXShodHRwczovL2RvaS5vcmcvMTAuMTA5My9iaW9pbmZvcm1hdGljcy9idHk1NjApKS4KTm90ZSB0aGF0IGZhc3RwIGhhcyBxdWFsaXR5IGNvbnRyb2wgZnVuY3Rpb25hbGl0eSBhbmQgbWFueSBkaWZmZXJlbnQgb3B0aW9ucyBmb3IgcHJlcHJvY2Vzc2luZyAoc2VlIFthbGwgb3B0aW9ucyBvbiBHaXRIdWJdKGh0dHBzOi8vZ2l0aHViLmNvbS9PcGVuR2VuZS9mYXN0cC9ibG9iL21hc3Rlci9SRUFETUUubWQjYWxsLW9wdGlvbnMpKSwgbW9zdCBvZiB3aGljaCB3ZSB3aWxsIG5vdCBjb3Zlci4KSGVyZSwgd2UgZm9jdXMgb24gYWRhcHRlciB0cmltbWluZywgcXVhbGl0eSBmaWx0ZXJpbmcsIGFuZCBsZW5ndGggZmlsdGVyaW5nLgoKQmVsb3csIHdlIGRpc2N1c3MgdGhlIGNvbW1hbmRzIHdlIHVzZWQgaW4gdGhlIHNjcmlwdC4KCiMjIyBDcmVhdGUgb3V0cHV0IGRpcmVjdG9yaWVzCgpgYGBiYXNoCiMgQ3JlYXRlIGEgZGlyZWN0b3J5IHRvIGhvbGQgdGhlIHRyaW1tZWQgZmFzdHEgZmlsZXMKbWtkaXIgLXAgZGF0YS9nYXN0cmljLWNhbmNlci9mYXN0cS10cmltbWVkL1NSUjU4NTU3MAojIENyZWF0ZSBhIGRpcmVjdG9yeSB0byBob2xkIHRoZSBRQyBvdXRwdXQgZnJvbSBmYXN0cApta2RpciAtcCBRQy9nYXN0cmljX2NhbmNlci9mYXN0cC9TUlI1ODU1NzAKYGBgCgpBcyB3ZSdsbCBjb3ZlciBiZWxvdywgZmFzdHAgZXNzZW50aWFsbHkgaGFzIHR3byBraW5kcyBvZiBvdXRwdXQ6IHRyaW1tZWQgYW5kIGZpbHRlcmVkIEZBU1RRIGZpbGVzIChkYXRhKSBhbmQgcmVwb3J0cyAocXVhbGl0eSBjb250cm9sKS4KCiMjIyBmYXN0cAoKYGBgYmFzaAojIFJ1biB0aGUgYWRhcHRlciBhbmQgcXVhbGl0eSB0cmltbWluZyBzdGVwIC0tIGFsc28gcHJvZHVjZXMgUUMgcmVwb3J0CmZhc3RwIC1pIGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEvU1JSNTg1NTcwL1NSUjU4NTU3MF8xLmZhc3RxLmd6IFwKICAgIC1JIGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEvU1JSNTg1NTcwL1NSUjU4NTU3MF8yLmZhc3RxLmd6IFwKICAgIC1vIGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEtdHJpbW1lZC9TUlI1ODU1NzAvU1JSNTg1NTcwX2Zhc3RwXzEuZmFzdHEuZ3ogXAogICAgLU8gZGF0YS9nYXN0cmljLWNhbmNlci9mYXN0cS10cmltbWVkL1NSUjU4NTU3MC9TUlI1ODU1NzBfZmFzdHBfMi5mYXN0cS5neiBcCiAgICAtLXF1YWxpZmllZF9xdWFsaXR5X3BocmVkIDE1IFwKICAgIC0tbGVuZ3RoX3JlcXVpcmVkIDIwIFwKICAgIC0tcmVwb3J0X3RpdGxlICJTUlI1ODU1NzAiIFwKICAgIC0tanNvbiBRQy9nYXN0cmljLWNhbmNlci9mYXN0cC9TUlI1ODU1NzAvU1JSNTg1NTcwX2Zhc3RwLmpzb24gXAogICAgLS1odG1sIFFDL2dhc3RyaWMtY2FuY2VyL2Zhc3RwL1NSUjU4NTU3MC9TUlI1ODU1NzBfZmFzdHAuaHRtbApgYGAKCkJlbG93LCB3ZSdsbCB3YWxrIHRocm91Z2ggdGhlIGFyZ3VtZW50cy9vcHRpb25zIHdlIHVzZWQgdG8gcnVuIGBmYXN0cGAuCkJ5IGRlZmF1bHQsIGZhc3RwIHBlcmZvcm1zIGFkYXB0ZXIgdHJpbW1pbmcsIHdoaWNoIHlvdSBjYW4gcmVhZCBtb3JlIGFib3V0IFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vT3BlbkdlbmUvZmFzdHAjYWRhcHRlcnMpLgpGb3IgcGFpcmVkLWVuZCBkYXRhIGxpa2UgdGhlIGRhdGEgd2UgaGF2ZSBmb3IgU1JSNTg1NTcwLCBhZGFwdGVycyBjYW4gYmUgZGV0ZWN0ZWQgYXV0b21hdGljYWxseSB3aXRob3V0IHNwZWNpZnlpbmcgYW4gYWRhcHRlciBzZXF1ZW5jZS4KCiMjIyMgSW5wdXQ6IGAtaWAgYW5kIGAtSWAKClRoZXNlIGFyZ3VtZW50cyBzcGVjaWZ5IHRoZSByZWFkMSBpbnB1dCBhbmQgcmVhZDIgKHNvbWV0aW1lcyBjYWxsZWQgbGVmdCBhbmQgcmlnaHQpIGlucHV0LCByZXNwZWN0aXZlbHkuCgojIyMjIGZhc3RxIG91dHB1dDogYC1vYCBhbmQgYC1PYAoKVGhlc2UgYXJndW1lbnRzIHNwZWNpZnkgdGhlIHJlYWQxIG91dHB1dCBhbmQgcmVhZDIgb3V0cHV0LCByZXNwZWN0aXZlbHkuCk5vdGUgdGhhdCB0aGUgb3V0cHV0IGlzIGJlaW5nIHBsYWNlZCBpbiBgZGF0YS9nYXN0cmljLWNhbmNlci9mYXN0cS10cmltbWVkL1NSUjU4NTU3MC9gLCBzbyB0aGUgcHJvY2Vzc2VkIEZBU1RRIGZpbGVzIHdpbGwgYmUga2VwdCBzZXBhcmF0ZSBmcm9tIGZyb20gdGhlIG9yaWdpbmFsIGZpbGVzLgpJdCBpcyBnZW5lcmFsbHkgZ29vZCBwcmFjdGljZSB0byB0cmVhdCB5b3VyICJyYXciIGRhdGEgYW5kIGl0cyBkaXJlY3RvcmllcyBhcyBmaXhlZCBhbmQgc2VwYXJhdGUgZnJvbSBhbnkgcHJvY2Vzc2luZyBhbmQgYW5hbHlzaXMgdGhhdCB5b3UgZG8sIHRvIHByZXZlbnQgYWNjaWRlbnRhbGx5IG1vZGlmaWNhdGlvbiBvZiB0aG9zZSBvcmlnaW5hbCBmaWxlcy4gCkFuZCBpbiB0aGUgZXZlbnQgdGhhdCB5b3UgYWNjaWRlbnRhbGx5IGRvIG1vZGlmeSB0aGUgb3JpZ2luYWxzLCB5b3Uga25vdyBleGFjdGx5IHdoaWNoIGZpbGVzIGFuZCBkaXJlY3RvcmllcyB0byByZXNldC4KCiMjIyMgYC0tcXVhbGlmaWVkX3F1YWxpdHlfcGhyZWRgCgpbUGhyZWQgc2NvcmVzXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9QaHJlZF9xdWFsaXR5X3Njb3JlKSBhcmUgdGhlIHF1YWxpdHkgaW5mb3JtYXRpb24gaW5jbHVkZWQgaW4gYSBGQVNUUSBmaWxlIGFuZCB0aGUgdmFsdWVzIGluZGljYXRlIHRoZSBjaGFuY2VzIHRoYXQgYSBiYXNlIGlzIGNhbGxlZCBpbmNvcnJlY3RseS4gTGV0J3MgbG9vayBhdCBhIHNjcmVlbnNob3Qgb2YgdGhlIFBlciBCYXNlIFNlcXVlbmNlIFF1YWxpdHkgbW9kdWxlIGZyb20gW0Zhc3RRQyBiYWQgSWxsdW1pbmEgZXhhbXBsZV0oaHR0cHM6Ly93d3cuYmlvaW5mb3JtYXRpY3MuYmFicmFoYW0uYWMudWsvcHJvamVjdHMvZmFzdHFjL2JhZF9zZXF1ZW5jZV9mYXN0cWMuaHRtbCkgd2UgbGlua2VkIHRvIGFib3ZlLgoKIVtwZXItYmFzZS1xdWFsaXR5LXNjcmVlbnNob3RdKGh0dHBzOi8vdXNlci1pbWFnZXMuZ2l0aHVidXNlcmNvbnRlbnQuY29tLzE5NTM0MjA1LzU5MjkyMzE2LWQ2MjkyNTAwLThjNGEtMTFlOS05NjdiLTEzMmJkOGM1NDU3Ny5wbmcpCgpBbnl0aGluZyBiZWxvdyAyMCwgd2hlcmUgYSBQaHJlZCBzY29yZSBvZiAyMCByZXByZXNlbnRzIGEgMSBpbiAxMDAgY2hhbmNlIHRoYXQgdGhlIGNhbGwgaXMgaW5jb3JyZWN0LCBpcyBjb25zaWRlcmVkIHBvb3IgcXVhbGl0eSBieSBGYXN0UUMuClVzaW5nIGAtLXF1YWxpZmllZF9xdWFsaXR5X3BocmVkIDE1YCAod2hpY2ggaXMgdGhlIGRlZmF1bHQpLCBtZWFucyBzY29yZXMgPj0gMTUgYXJlIGNvbnNpZGVyZWQgInF1YWxpZmllZC4iClVzaW5nIHRoZSBkZWZhdWx0IHBhcmFtZXRlcnMgYXMgd2UgZG8gaGVyZSwgcmVhZHMgd2lsbCBiZSBmaWx0ZXJlZCBvdXQgaWYgPjQwJSBvZiB0aGUgYmFzZXMgYXJlIHVucXVhbGlmaWVkLgpZb3UgY2FuIHJlYWQgbW9yZSBhYm91dCB0aGUgKipxdWFsaXR5IGZpbHRlcmluZyoqIGZ1bmN0aW9uYWxpdHkgb2YgZmFzdHAgW2hlcmVdKGh0dHBzOi8vZ2l0aHViLmNvbS9PcGVuR2VuZS9mYXN0cC9ibG9iL21hc3Rlci9SRUFETUUubWQjcXVhbGl0eS1maWx0ZXIpLgpUaGUgU2FsbW9uIGRvY3VtZW50YXRpb24gbm90ZXMgdGhhdCwgZ2l2ZW4gdGhlIHdheSB3ZSBydW4gYHNhbG1vbiBxdWFudGAsIHF1YW50aWZpY2F0aW9uIG1heSBiZSBtb3JlIHNlbnNpdGl2ZSB0byBjYWxscyB0aGF0IGFyZSBsaWtlbHkgdG8gYmUgZXJyb25lb3VzIChvZiBsb3cgcXVhbGl0eSkgYW5kLCB0aGVyZWZvcmUsIHF1YWxpdHkgdHJpbW1pbmcgbWF5IGJlIGltcG9ydGFudC4KClRyaW1taW5nLCBpbiBjb250cmFzdCB0byBmaWx0ZXJpbmcsIHJlZmVycyB0byByZW1vdmluZyBsb3cgcXVhbGl0eSBiYXNlIGNhbGxzIGZyb20gdGhlICh0eXBpY2FsbHkgMycpIGVuZCBvZiByZWFkcy4KQSByZWNlbnQgcGFwZXIgZnJvbSB0aGUgU2FsbW9uIGF1dGhvcnMgKFtTcml2YXN0YXZhIF9ldCBhbC5fIDIwMjBdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2L3MxMzA1OS0wMjAtMDIxNTEtOCkpIG5vdGVzIHRoYXQgdHJpbW1pbmcgZGlkIG5vdCBhZmZlY3QgbWFwcGluZyByYXRlcyBmcm9tIHJhbmRvbSBwdWJsaWNseSBhdmFpbGFibGUgaHVtYW4gYnVsayAocGFpcmVkLWVuZCkgUk5BLXNlcSBzYW1wbGVzICh0aGV5IHVzZWQgW1RyaW1HYWxvcmVdKGh0dHBzOi8vZ2l0aHViLmNvbS9GZWxpeEtydWVnZXIvVHJpbUdhbG9yZSkpLgpmYXN0cCBkb2VzIGhhdmUgW3RoZSBmdW5jdGlvbmFsaXR5XShodHRwczovL2dpdGh1Yi5jb20vT3BlbkdlbmUvZmFzdHAjcGVyLXJlYWQtY3V0dGluZy1ieS1xdWFsaXR5LXNjb3JlKSB0byBwZXJmb3JtIHRyaW1taW5nIHVzaW5nIGEgc2xpZGluZyB3aW5kb3csIHdoaWNoIG11c3QgYmUgZW5hYmxlZC4KV2UgYXJlIG5vdCB1c2luZyBpdCBoZXJlLgoKX05vdGUgdGhhdCB0aGVyZSBhcmUgdHdvIGtpbmRzIG9mIGVuY29kaW5nIGZvciBQaHJlZCBzY29yZXM6IFBocmVkIDMzIGVuY29kaW5nIGFuZCBQaHJlZCA2NCBlbmNvZGluZy4KRmFzdFFDIGd1ZXNzZWQgdGhhdCB0aGUgZmlsZSBmb3IgU1JSNTg1NTcwIHVzZXMgU2FuZ2VyL0lsbHVtaW5hIDEuOSBlbmNvZGluZyAoUGhyZWQgMzMpLgpJZiB3ZSBoYWQgUGhyZWQgNjQgZGF0YSwgd2UnZCB1c2UgdGhlIGAtLXBocmVkNjRgIGZsYWcuCllvdSBjYW4gcmVhZCBhIGxpdHRsZSBiaXQgbW9yZSBhYm91dCB0aGUgZW5jb2RpbmcgW2hlcmVdKGh0dHA6Ly9yZXNvdXJjZXMucWlhZ2VuYmlvaW5mb3JtYXRpY3MuY29tL21hbnVhbHMvY2xjZ2Vub21pY3N3b3JrYmVuY2gvNzAwL1F1YWxpdHlfc2NvcmVzX2luX0lsbHVtaW5hX3BsYXRmb3JtLmh0bWwpLl8KCiMjIyMgYC0tbGVuZ3RoX3JlcXVpcmVkYAoKVHJpbW1pbmcgcmVhZHMgbWF5IHJlc3VsdCBpbiBzaG9ydCByZWFkcywgd2hpY2ggbWF5IGFmZmVjdCBnZW5lIGV4cHJlc3Npb24gZXN0aW1hdGVzIChbV2lsbGlhbXMgZXQgYWwuIF9CTUMgQmlvaW5mb3JtYXRpY3MuXyAyMDE2Ll0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvczEyODU5LTAxNi0wOTU2LTIpKS4KVXNpbmcgYC0tbGVuZ3RoX3JlcXVpcmVkIDIwYCBtZWFucyB0aGF0IHJlYWRzIHNob3J0ZXIgdGhhbiAyMGJwIHdpbGwgYmUgZGlzY2FyZGVkIChzaW1pbGFyIHRvIHdoYXQgd2FzIHVzZWQgaW4gU3JpdmFzdGF2YSBldCBhbC4gYWJvdmUpLgoKIyMjIyBgLS1yZXBvcnRfdGl0bGVgCgpXaGVuIHdlIGxvb2sgYXQgdGhlIEhUTUwgcmVwb3J0LCBpdCdzIGhlbHBmdWwgdG8gcXVpY2tseSBpZGVudGlmeSB3aGF0IHNhbXBsZSB0aGUgcmVwb3J0IGlzIGZvci4gVXNpbmcgYC0tcmVwb3J0IHRpdGxlICJTUlI1ODU1NzAiYCBtZWFucyB0aGF0IHRoZSByZXBvcnQgd2lsbCBiZSB0aXRsZWQgIlNSUjU4NTU3MCIgcmF0aGVyIHRoYW4gdGhlIGRlZmF1bHQgKCJmYXN0cCByZXBvcnQiKS4KCiMjIyMgYC0tanNvbmAgYW5kIGAtLWh0bWxgCgpXaXRoIHRoZXNlIG9wdGlvbnMsIHdlJ3JlIHNwZWNpZnlpbmcgd2hlcmUgdGhlIFtKU09OXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9KU09OKSBhbmQgSFRNTCByZXBvcnRzIHdpbGwgYmUgc2F2ZWQgKGluIHRoZSBgUUMvZ2FzdHJpYy1jYW5jZXIvZmFzdHAvYCBkaXJlY3Rvcnkgd2UgY3JlYXRlZCkgYW5kIHdoYXQgdGhlIGZpbGVuYW1lcyB3aWxsIGJlLgpJbmNsdWRpbmcgdGhlIHNhbXBsZSBuYW1lIGluIHRoZSBmaWxlbmFtZXMgYWdhaW4gbWF5IGhlbHAgdXMgd2l0aCBwcm9qZWN0IG9yZ2FuaXphdGlvbi4KCioqSWYgd2UgbG9vayBhdCoqIGBRQy9nYXN0cmljLWNhbmNlci9mYXN0cC9TUlI1ODU1NzBfZmFzdHAuanNvbmAgKipvciB0aGUgdG9wIG9mIHRoZSBIVE1MIHJlcG9ydCwgd2UgY2FuIHNlZSB0aGF0IGZhc3RwIHJlcG9ydHMgY2VydGFpbiBtZXRyaWNzIGJlZm9yZSBhbmQgYWZ0ZXIgZmlsdGVyaW5nLCB3aGljaCBjYW4gYmUgdmVyeSB1c2VmdWwgaW4gbWFraW5nIGFuYWx5c2lzIGRlY2lzaW9ucy4qKgoKIyMgUXVhbnRpZmljYXRpb24gd2l0aCBTYWxtb24KCiFbXShkaWFncmFtcy9ybmEtc2VxXzQucG5nKQoKV2UnbGwgdXNlIFtTYWxtb25dKGh0dHBzOi8vY29tYmluZS1sYWIuZ2l0aHViLmlvL3NhbG1vbi8pIGZvciBxdWFudGlmeWluZyB0cmFuc2NyaXB0IGV4cHJlc3Npb24gKFtkb2N1bWVudGF0aW9uXShodHRwOi8vc2FsbW9uLnJlYWR0aGVkb2NzLmlvL2VuL2xhdGVzdC8pKS4KU2FsbW9uIChbUGF0cm8sIGV0IGFsLiBfTmF0dXJlIE1ldGhvZHMuXyAyMDE3Ll0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwMzgvbm1ldGguNDE5NykpIGlzIGZhc3QgYW5kIHJlcXVpcmVzIHZlcnkgbGl0dGxlIG1lbW9yeSwgd2hpY2ggbWFrZXMgaXQgYSBncmVhdCBjaG9pY2UgZm9yIHJ1bm5pbmcgb24geW91ciBsYXB0b3AgZHVyaW5nIHRyYWluaW5nLgpXZSBjYW4gdXNlIHRoZSBvdXRwdXQgZm9yIGRvd25zdHJlYW0gYW5hbHlzZXMgbGlrZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyBhbmQgY2x1c3RlcmluZy4KV2UgdXNlIFNhbG1vbiBpbiBtYXBwaW5nIG1vZGUsIHdpdGggbWFwcGluZyB2YWxpZGF0aW9uIGVuYWJsZWQsIHVzaW5nIHRoZSBmb2xsb3dpbmcgY29tbWFuZDoKCmBgYGJhc2gKIyBXZSBwZXJmb3JtIHF1YW50aWZpY2F0aW9uIG9uIHRoZSBmaWxlcyB0aGF0IGhhdmUgYmVlbiB0cmltbWVkCiMgYW5kIHVzZSB0aGUgaW5kZXggZ2VuZXJhdGVkIHdpdGggLWsgMjMsIGFzIHRoaXMgbWF5ICJpbXByb3ZlIHNlbnNpdGl2aXR5IgojIHBlciB0aGUgU2FsbW9uIGRvY3VtZW50YXRpb24Kc2FsbW9uIHF1YW50IC1pIGluZGV4L0hvbW9fc2FwaWVucy9zaG9ydF9pbmRleCBcCiAgICAtbCBBIFwKICAgIC0xIGRhdGEvZ2FzdHJpYy1jYW5jZXIvZmFzdHEtdHJpbW1lZC9TUlI1ODU1NzAvU1JSNTg1NTcwX2Zhc3RwXzEuZmFzdHEuZ3ogXAogICAgLTIgZGF0YS9nYXN0cmljLWNhbmNlci9mYXN0cS10cmltbWVkL1NSUjU4NTU3MC9TUlI1ODU1NzBfZmFzdHBfMi5mYXN0cS5neiBcCiAgICAtbyBkYXRhL2dhc3RyaWMtY2FuY2VyL3NhbG1vbl9xdWFudC9TUlI1ODU1NzAgXAogICAgLS12YWxpZGF0ZU1hcHBpbmdzIC0tcmFuZ2VGYWN0b3JpemF0aW9uQmlucyA0IFwKICAgIC0tZ2NCaWFzIC0tc2VxQmlhcyBcCiAgICAtLXRocmVhZHMgNApgYGAKCkJlbG93LCB3ZSdsbCB3YWxrIHRocm91Z2ggdGhlIGFyZ3VtZW50cy9vcHRpb25zIHdlIHVzZWQgdG8gcnVuIGBzYWxtb24gcXVhbnRgLgoKIyMjIyBUcmFuc2NyaXB0b21lIGluZGV4OiBgLWlgCgpTYWxtb24gcmVxdWlyZXMgYSBzZXQgb2YgdHJhbnNjcmlwdHMgKHdoYXQgd2Ugd2FudCB0byBxdWFudGlmeSkgaW4gdGhlIGZvcm0gb2YgYSB0cmFuc2NyaXB0b21lIGluZGV4IGJ1aWx0IHdpdGggYHNhbG1vbiBpbmRleGAuCkJ1aWxkaW5nIGFuIGluZGV4IGNhbiB0YWtlIGEgd2hpbGUgKGJ1dCB5b3Ugb25seSBoYXZlIHRvIGRvIGl0IG9uY2UhKSwgc28gd2UndmUgYnVpbHQgdGhlIG9uZSB3ZSB1c2UgdG9kYXkgYWhlYWQgb2YgdGltZS4KQmVmb3JlIHdlIHVzZSBpdCwgd2UnbGwgdGFrZSBhIG1vbWVudCB0byBnaXZlIGEgYml0IG9mIGJhY2tncm91bmQuCgpZb3UgY2FuIHNlZSBob3cgd2Ugb2J0YWluZWQgdGhpcyBpbmRleCBhbmQgb3RoZXJzIFtvbiBHaXRIdWJdKGh0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL3RyYWluaW5nLXR4b21lLXByZXApLgpOb3RlIHRoYXQgd2UgdXNlZCBIb21vIHNhcGllbnMgR1JDaDM4LCBFbnNlbWJsIHJlbGVhc2UgOTUuCkl0IGlzIGltcG9ydGFudCB0byBrZWVwIHRyYWNrIG9mIHdoYXQgYnVpbGQsIHJlc291cmNlLCBhbmQgZmlsZXMgd2VyZSB1c2VkIGFuZCBwdXR0aW5nIG91ciBzaGVsbCBzY3JpcHRzIG9uIEdpdEh1YiBhbGxvd3MgdXMgdG8gZG8gdGhhdC4gIAoKVGhlIGBzYWxtb24gaW5kZXhgIGNvbW1hbmQgaGFzIGEgcGFyYW1ldGVyIGAta2Agd2hpY2ggc2V0cyB0aGUgay1tZXIgbGVuZ3RoLgpUaGUgaW5kZXggd2UgdXNlZCB3YXMgYnVpbHQgd2l0aCBgLWsgMjNgIGFuZCBjYW4gYmUgZm91bmQgaGVyZToKCmBgYAppbmRleC9Ib21vX3NhcGllbnMvc2hvcnRfaW5kZXgKYGBgCgpVc2luZyBhIHNtYWxsZXIgdmFsdWUgZm9yIF9rXyB0aGFuIHRoZSBkZWZhdWx0IChfa18gPSAzMSkgaXMgYXBwcm9wcmlhdGUgZm9yIHNob3J0ZXIgcmVhZHMgYW5kIG1heSBpbXByb3ZlIHNlbnNpdGl2aXR5IHdoZW4gdXNpbmcgYC0tdmFsaWRhdGVNYXBwaW5nc2AgYWNjb3JkaW5nIHRvIHRoZSBbU2FsbW9uIGRvY3VtZW50YXRpb25dKGh0dHBzOi8vc2FsbW9uLnJlYWR0aGVkb2NzLmlvL2VuL2xhdGVzdC9zYWxtb24uaHRtbCNwcmVwYXJpbmctdHJhbnNjcmlwdG9tZS1pbmRpY2VzLW1hcHBpbmctYmFzZWQtbW9kZSkuCgojIyMjIGAtbGAKCldlIHVzZSBgLWwgQWAgdG8gYWxsb3cgU2FsbW9uIHRvIGF1dG9tYXRpY2FsbHkgaW5mZXIgdGhlIGxpYnJhcnkgdHlwZSBiYXNlZCBvbiBhIHN1YnNldCBvZiByZWFkcywgYnV0IHlvdSBjYW4gYWxzbyBwcm92aWRlIHRoZSBbbGlicmFyeSB0eXBlXShodHRwOi8vc2FsbW9uLnJlYWR0aGVkb2NzLmlvL2VuL2xhdGVzdC9zYWxtb24uaHRtbCN3aGF0LXMtdGhpcy1saWJ0eXBlKSB0byBTYWxtb24gd2l0aCB0aGlzIGFyZ3VtZW50LgoKIyMjIyBJbnB1dDogYC0xYCBhbmQgYC0yYAoKVGhlc2UgZGF0YSBhcmUgcGFpcmVkLWVuZCwgd2UgdXNlIGAtMWAgYW5kIGAtMmAgdG8gc3BlY2lmeSByZWFkMSBhbmQgcmVhZDIsIHJlc3BlY3RpdmVseS4KCiMjIyMgYC1vYAoKT3V0cHV0IGRpcmVjdG9yeSwgYHNhbG1vbiBxdWFudGAgc2hvdWxkIGNyZWF0ZSB0aGlzIGZvciB1cyBpZiBpdCBkb2Vzbid0IGV4aXN0IHlldC4KCiMjIyMgYC0tdmFsaWRhdGVNYXBwaW5nc2AgYW5kIGAtLXJhbmdlRmFjdG9yaXphdGlvbkJpbnNgCgpVc2luZyBgLS12YWxpZGF0ZU1hcHBpbmdzYCBlbmFibGVzIG1hcHBpbmcgdmFsaWRhdGlvbiwgd2hlcmUgU2FsbW9uIGNoZWNrcyBpdHMgbWFwcGluZ3MgdXNpbmcgdHJhZGl0aW9uYWwgYWxpZ25tZW50LgpUaGlzIGhlbHBzIHByZXZlbnQgInNwdXJpb3VzIG1hcHBpbmdzIiB3aGVyZSBhIHJlYWQgbWFwcyB0byBhIHRhcmdldCBidXQgZG9lcyBub3QgYXJpc2UgZnJvbSBpdCAoc2VlIFtkb2N1bWVudGF0aW9uIGZvciBmbGFnXShodHRwczovL3NhbG1vbi5yZWFkdGhlZG9jcy5pby9lbi9sYXRlc3Qvc2FsbW9uLmh0bWwjdmFsaWRhdGVtYXBwaW5ncykgYW5kIHRoZSBbcmVsZWFzZSBub3RlcyBmb3IgYHYwLjEwLjBgXShodHRwczovL2dpdGh1Yi5jb20vQ09NQklORS1sYWIvc2FsbW9uL3JlbGVhc2VzL3RhZy92MC4xMC4wKSB3aGVyZSB0aGlzIHdhcyBpbnRyb2R1Y2VkKS4KCldoZW4gZW5hYmxpbmcgbWFwcGluZyB2YWxpZGF0aW9uIHdpdGggYC0tdmFsaWRhdGVNYXBwaW5nc2AsIHNldHRpbmcgYC0tcmFuZ2VGYWN0b3JpemF0aW9uQmlucyA0YCBjYW4gaW1wcm92ZSBxdWFudGlmaWNhdGlvbiBmb3IgY2VydGFpbiBjbGFzc2VzIG9mIHRyYW5zY3JpcHRzIChbZG9jc10oaHR0cHM6Ly9zYWxtb24ucmVhZHRoZWRvY3MuaW8vZW4vbGF0ZXN0L3NhbG1vbi5odG1sI3JhbmdlZmFjdG9yaXphdGlvbmJpbnMpKS4KCiMjIyMgYC0tZ2NCaWFzYAoKV2l0aCB0aGlzIG9wdGlvbiBlbmFibGVkLCBTYWxtb24gd2lsbCBhdHRlbXB0IHRvIGNvcnJlY3QgZm9yIGZyYWdtZW50IEdDLWJpYXMuClJlZ2lvbnMgd2l0aCBoaWdoIG9yIGxvdyBHQyBjb250ZW50IHRlbmQgdG8gYmUgdW5kZXJyZXByZXNlbnRlZCBpbiBzZXF1ZW5jaW5nIGRhdGEuCgpJdCBzaG91bGQgYmUgbm90ZWQgdGhhdCB0aGlzIGlzIG9ubHkgYXBwcm9wcmlhdGUgZm9yIHVzZSB3aXRoIHBhaXJlZC1lbmQgcmVhZHMsIGFzIGZyYWdtZW50IGxlbmd0aCBjYW4gbm90IGJlIGluZmVycmVkIGZyb20gc2luZ2xlLWVuZCByZWFkcyAoc2VlIFt0aGlzIEdpdEh1YiBpc3N1ZV0oaHR0cHM6Ly9naXRodWIuY29tL0NPTUJJTkUtbGFiL3NhbG1vbi9pc3N1ZXMvODMpKS4KCiMjIyMgYC0tc2VxQmlhc2AKCldpdGggdGhpcyBvcHRpb24gZW5hYmxlZCwgU2FsbW9uIHdpbGwgYXR0ZW1wdCB0byBjb3JyZWN0IGZvciB0aGUgYmlhcyB0aGF0IG9jY3VycyB3aGVuIHVzaW5nIHJhbmRvbSBoZXhhbWVyIHByaW1pbmcgKHByZWZlcmVudGlhbCBzZXF1ZW5jaW5nIG9mIHJlYWRzIHdoZW4gY2VydGFpbiBtb3RpZnMgYXBwZWFyIGF0IHRoZSBiZWdpbm5pbmcpLgoKIyMjIyBgLS10aHJlYWRzYAoKVGhlIGAtLXRocmVhZHNgIGFyZ3VtZW50IGNvbnRyb2xzIHRoZSBudW1iZXIgb2YgdGhyZWFkcyB0aGF0IGFyZSBhdmFpbGFibGUgdG8gU2FsbW9uIGR1cmluZyBxdWFudGlmaWNhdGlvbi4KVGhpcyBpbiBlc3NlbmNlIGNvbnRyb2xzIGhvdyBtdWNoIG9mIHRoZSBtYXBwaW5nIGNhbiBvY2N1ciBpbiBwYXJhbGxlbC4KSWYgeW91IGhhZCBhY2Nlc3MgdG8gYSBjb21wdXRlciB3aXRoIG1hbnkgY29yZXMsIHlvdSBjb3VsZCBpbmNyZWFzZSB0aGUgbnVtYmVyIG9mIHRocmVhZHMgdG8gbWFrZSBxdWFudGlmaWNhdGlvbiBnbyBmYXN0ZXIuCgoqKk5hdmlnYXRlIHRvKiogYGRhdGEvZ2FzdHJpYy1jYW5jZXIvc2FsbW9uX3F1YW50L1NSUjU4NTU3MS9hdXhfaW5mb2AgKiphbmQgb3BlbioqIGBtZXRhX2luZm8uanNvbmAqKi4KTG9vayBmb3IgYSBmaWVsZCBjYWxsZWQqKiBgcGVyY2VudF9tYXBwZWRgICoqLS0gd2hhdCB2YWx1ZSBkb2VzIHRoaXMgc2FtcGxlIGhhdmU/KioK
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/RNA-seq/02-gastric_cancer_tximeta-live.Rmd b/RNA-seq/02-gastric_cancer_tximeta-live.Rmd index 0c3cecc8..826509e8 100644 --- a/RNA-seq/02-gastric_cancer_tximeta-live.Rmd +++ b/RNA-seq/02-gastric_cancer_tximeta-live.Rmd @@ -1,12 +1,22 @@ --- title: "Gastric cancer: gene-level summarization with `tximeta`" +author: CCDL for ALSF +date: 2021 output: html_notebook: toc: true toc_float: true --- -**CCDL 2021** +## Objectives + +This notebook will demonstrate how to: + +- Import RNA-seq expression quantification output using `tximeta` +- Summarize transcript-level expression to the gene level +- Interrogate and extract data from a `SummarizedExperiment` object + +--- In this notebook, we'll import the transcript expression quantification output from `salmon quant` using the [`tximeta`](https://bioconductor.org/packages/release/bioc/html/tximeta.html) package. `tximeta` is in part a wrapper around another package, [`tximport`](https://bioconductor.org/packages/release/bioc/html/tximport.html), which imports transcript expression data and summarizes it to the gene level. diff --git a/RNA-seq/02-gastric_cancer_tximeta.nb.html b/RNA-seq/02-gastric_cancer_tximeta.nb.html new file mode 100644 index 00000000..c876baa4 --- /dev/null +++ b/RNA-seq/02-gastric_cancer_tximeta.nb.html @@ -0,0 +1,1039 @@ + + + + + + + + + + + + + + + +Gastric cancer: gene-level summarization with tximeta + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Import RNA-seq expression quantification output using tximeta
    • +
  • Summarize transcript-level expression to the gene level
    • +
  • Interrogate and extract data from a SummarizedExperiment object
    • +
+
+

In this notebook, we’ll import the transcript expression quantification output from salmon quant using the tximeta package. tximeta is in part a wrapper around another package, tximport, which imports transcript expression data and summarizes it to the gene level. Working at the gene rather than transcript level has a number of potential advantages for interpretability, efficiency, and reduction of false positives (Soneson et al. 2016). tximeta eases some of the burden of import by automatically identifying the correct set of annotation data to append to many data sets (Love et al. 2020).

+

+

For more information about tximeta, see this excellent vignette from Love et al.

+
+
+

Libraries and functions

+ + + +
# Load magrittr for the pipe
+library(magrittr)
+
+# Load the tximeta package
+library(tximeta)
+
+# Load the SummarizedExperiment package
+library(SummarizedExperiment)
+ + +
Loading required package: MatrixGenerics
+ + +
Loading required package: matrixStats
+ + +

+Attaching package: 'MatrixGenerics'
+ + +
The following objects are masked from 'package:matrixStats':
+
+    colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
+    colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
+    colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
+    colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
+    colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
+    colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
+    colWeightedMeans, colWeightedMedians, colWeightedSds,
+    colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
+    rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
+    rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
+    rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
+    rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
+    rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
+    rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
+    rowWeightedSds, rowWeightedVars
+ + +
Loading required package: GenomicRanges
+ + +
Loading required package: stats4
+ + +
Loading required package: BiocGenerics
+ + +
Loading required package: parallel
+ + +

+Attaching package: 'BiocGenerics'
+ + +
The following objects are masked from 'package:parallel':
+
+    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
+    clusterExport, clusterMap, parApply, parCapply, parLapply,
+    parLapplyLB, parRapply, parSapply, parSapplyLB
+ + +
The following objects are masked from 'package:stats':
+
+    IQR, mad, sd, var, xtabs
+ + +
The following objects are masked from 'package:base':
+
+    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
+    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
+    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
+    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
+    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
+    union, unique, unsplit, which.max, which.min
+ + +
Loading required package: S4Vectors
+ + +

+Attaching package: 'S4Vectors'
+ + +
The following object is masked from 'package:base':
+
+    expand.grid
+ + +
Loading required package: IRanges
+ + +
Loading required package: GenomeInfoDb
+ + +
Loading required package: Biobase
+ + +
Welcome to Bioconductor
+
+    Vignettes contain introductory material; view with
+    'browseVignettes()'. To cite Bioconductor, see
+    'citation("Biobase")', and for packages 'citation("pkgname")'.
+ + +

+Attaching package: 'Biobase'
+ + +
The following object is masked from 'package:MatrixGenerics':
+
+    rowMedians
+ + +
The following objects are masked from 'package:matrixStats':
+
+    anyMissing, rowMedians
+ + + +
+
+

Directories and files

+ + + +
# directory where the data are located
+data_dir <- file.path("data", "gastric-cancer")
+
+# directory where the quant files are located, each sample is its own
+# directory
+quant_dir <- file.path(data_dir, "salmon_quant")
+
+# create a directory to hold the tximeta results if it doesn't exist yet
+txi_dir <- file.path(data_dir, "txi")
+if (!dir.exists(txi_dir)) {
+  dir.create(txi_dir, recursive = TRUE)
+}
+ + + +

We’ll need the quant.sf files for all the samples in an experiment which we have stored in quant_dir.

+ + + +
# the quant files themselves
+sf_files <- list.files(quant_dir, recursive = TRUE, full.names = TRUE,
+                       pattern = "quant.sf")
+ + + + + + +
# sample metadata file
+meta_file <- file.path(data_dir, "gastric-cancer_metadata.tsv")
+ + + +

Output

+ + + +
# Name the output gastric-cancer_tximeta.RDS and use the directory created 
+# above as the rest of the path
+txi_out_file <- file.path(txi_dir, "gastric-cancer_tximeta.RDS")
+ + + +
+
+

File names

+

All output files from salmon quant we’ll use with tximeta are named quant.sf. Unfortunately, this means that the file names themselves do not have any information about the sample they come from!

+ + + +
# Let's look at the full path for the quant.sf files
+sf_files
+ + +
[1] "data/gastric-cancer/salmon_quant/SRR585570/quant.sf"
+[2] "data/gastric-cancer/salmon_quant/SRR585571/quant.sf"
+[3] "data/gastric-cancer/salmon_quant/SRR585572/quant.sf"
+[4] "data/gastric-cancer/salmon_quant/SRR585573/quant.sf"
+[5] "data/gastric-cancer/salmon_quant/SRR585574/quant.sf"
+[6] "data/gastric-cancer/salmon_quant/SRR585575/quant.sf"
+[7] "data/gastric-cancer/salmon_quant/SRR585576/quant.sf"
+[8] "data/gastric-cancer/salmon_quant/SRR585577/quant.sf"
+ + +
data/gastric-cancer/salmon_quant/SRR585570/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585571/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585572/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585573/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585574/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585575/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585576/quant.sf
+
+data/gastric-cancer/salmon_quant/SRR585577/quant.sf
+ + + +

Let’s extract the sample names from the file paths using the stringr package.

+

Notice how the file path is separated by /. If we were to split up this character string by /, the second to last item is the sample names (because we used them as directory names for the salmon output). This is exactly what stringr::word() allows us to do: split up the file paths by / and extract the sample names.

+ + + +
sample_names <- stringr::word(sf_files, -2, sep = "/")
+sample_names
+ + +
[1] "SRR585570" "SRR585571" "SRR585572" "SRR585573" "SRR585574" "SRR585575"
+[7] "SRR585576" "SRR585577"
+ + +
SRR585570
+
+SRR585571
+
+SRR585572
+
+SRR585573
+
+SRR585574
+
+SRR585575
+
+SRR585576
+
+SRR585577
+ + + +
+
+

Set up metadata

+

tximeta needs a data frame with at least these two columns: - a files column with the file paths to the quant.sf files - a names column with the sample names

+ + + +
coldata <- data.frame(files = sf_files, 
+                      names = sample_names)
+ + + +

We have more information about these samples stored in the metadata file that we will also want stored in coldata. Let’s read in the sample metadata from the TSV file.

+ + + +
# Read in the sample metadata TSV file and have a look
+sample_meta_df <- readr::read_tsv(meta_file)
+ + +

+── Column specification ────────────────────────────────────────────────────────
+cols(
+  tissue = col_character(),
+  srr_accession = col_character(),
+  title = col_character()
+)
+ + +
sample_meta_df
+ +
+ +
+ + +

We’ll want this information to be added to the coldata, which we can do by using a join function to match up the rows between the two data frames and combine them.

+ + + +
coldata <- coldata %>%
+  dplyr::inner_join(sample_meta_df, by = c("names" = "srr_accession"))
+
+coldata
+ +
+ +
+ + +
+
+

Import expression data with tximeta

+

Using the coldata data frame that we set up, we can now run the tximeta() to import our expression data while automatically finding and associating the transcript annotations that were used when we performed the quantification.

+

The first time you run tximeta() you may get a message about storing downloaded transcriptome data in a cache directory so that it can retrieve the data more quickly the next time. We recommend you use the cache, and accept the default location.

+ + + +
txi_data <- tximeta(coldata)
+ + +
importing quantifications
+ + +
reading in files with read_tsv
+ + +
1 2 3 4 5 6 7 8 
+found matching transcriptome:
+[ Ensembl - Homo sapiens - release 95 ]
+useHub=TRUE: checking for EnsDb via 'AnnotationHub'
+using temporary cache /tmp/RtmpCAqF9W/BiocFileCache
+snapshotDate(): 2020-10-27
+found matching EnsDb via 'AnnotationHub'
+downloading 1 resources
+retrieving 1 resource
+loading from cache
+require("ensembldb")
+generating transcript ranges
+ + + +

*tximeta currently works easily for most human and mouse datasets, but requires a few more steps for other species.

+
+
+

Summarize to gene

+

We’ll summarize to the gene level using the summarizeToGene() function.

+ + + +
# Summarize to the gene level
+gene_summarized <- summarizeToGene(txi_data)
+ + +
loading existing EnsDb created: 2021-03-16 18:26:32
+ + +
obtaining transcript-to-gene mapping from database
+ + +
generating gene ranges
+ + +
summarizing abundance
+ + +
summarizing counts
+ + +
summarizing length
+ + + +

We can use the class function to see what type of object gene_summarized is.

+ + + +
# Check what type of object `gene_summarized` is
+class(gene_summarized)
+ + +
[1] "RangedSummarizedExperiment"
+attr(,"package")
+[1] "SummarizedExperiment"
+ + +
RangedSummarizedExperiment
+ + + +

This tells us that gene_summarized is an object called a SummarizedExperiment which can be handled by functions from the package of the same name. We more specifically have a RangedSummarizedExperiment which is a more specific type of SummarizedExperiment.

+

SummarizedExperiment objects have this general structure:

+
+ +

SummarizedExperiment

+
+

This figure is from this handy vignette about SummarizedExperiment objects.

+

As shown in the diagram, we can use some of the functions provided by the SummarizedExperiment package to extract data from our gene_summarized object. For example, calling rowData() on our object shows all the gene information that tximeta set up!

+ + + +
# rowData() shows us our gene annotation
+rowData(gene_summarized)
+ + +
DataFrame with 37788 rows and 9 columns
+                        gene_id   gene_name           gene_biotype
+                    <character> <character>            <character>
+ENSG00000000003 ENSG00000000003      TSPAN6         protein_coding
+ENSG00000000005 ENSG00000000005        TNMD         protein_coding
+ENSG00000000419 ENSG00000000419        DPM1         protein_coding
+ENSG00000000457 ENSG00000000457       SCYL3         protein_coding
+ENSG00000000460 ENSG00000000460    C1orf112         protein_coding
+...                         ...         ...                    ...
+ENSG00000285978 ENSG00000285978  AC113348.2         protein_coding
+ENSG00000285982 ENSG00000285982  AC012213.5         protein_coding
+ENSG00000285986 ENSG00000285986  BX248415.1 unprocessed_pseudogene
+ENSG00000285990 ENSG00000285990  AL589743.7 transcribed_unproces..
+                seq_coord_system            description    gene_id_version
+                     <character>            <character>        <character>
+ENSG00000000003       chromosome tetraspanin 6 [Sourc.. ENSG00000000003.14
+ENSG00000000005       chromosome tenomodulin [Source:..  ENSG00000000005.5
+ENSG00000000419       chromosome dolichyl-phosphate m.. ENSG00000000419.12
+ENSG00000000457       chromosome SCY1 like pseudokina.. ENSG00000000457.13
+ENSG00000000460       chromosome chromosome 1 open re.. ENSG00000000460.16
+...                          ...                    ...                ...
+ENSG00000285978       chromosome       novel transcript  ENSG00000285978.1
+ENSG00000285982       chromosome          novel protein  ENSG00000285982.1
+ENSG00000285986       chromosome complement factor H ..  ENSG00000285986.1
+ENSG00000285990       chromosome neurobeachin (NBEA) ..  ENSG00000285990.1
+                     symbol entrezid
+                <character>   <list>
+ENSG00000000003      TSPAN6     7105
+ENSG00000000005        TNMD    64102
+ENSG00000000419        DPM1     8813
+ENSG00000000457       SCYL3    57147
+ENSG00000000460    C1orf112    55732
+...                     ...      ...
+ENSG00000285978  AC113348.2       NA
+ENSG00000285982  AC012213.5       NA
+ENSG00000285986  BX248415.1       NA
+ENSG00000285990  AL589743.7       NA
+                                                             tx_ids
+                                                    <CharacterList>
+ENSG00000000003 ENST00000373020,ENST00000494424,ENST00000496771,...
+ENSG00000000005                     ENST00000373031,ENST00000485971
+ENSG00000000419 ENST00000371582,ENST00000371584,ENST00000371588,...
+ENSG00000000457 ENST00000367770,ENST00000367771,ENST00000367772,...
+ENSG00000000460 ENST00000286031,ENST00000359326,ENST00000413811,...
+...                                                             ...
+ENSG00000285978                                     ENST00000638723
+ENSG00000285982                                     ENST00000649416
+ENSG00000285986                                     ENST00000649395
+ENSG00000285990                                     ENST00000649331
+ [ reached getOption("max.print") -- omitted 1 rows ]
+ + + +

The assay slot in SummarizedExperiments holds data from the experiment. In this case, it will include our gene-level expression information stored as a gene x sample matrix.

+

Multiple assays can be stored in an SummarizedExperiment and we can use the assayNames() function to see what assays are included in gene_summarized.

+ + + +
assayNames(gene_summarized)
+ + +
[1] "counts"    "abundance" "length"
+ + +
counts
+
+abundance
+
+length
+ + + +

If we want to extract an assay’s data, we can use assay() function and specify the name of the assay we want to extract.

+ + + +
counts_mat <- assay(gene_summarized, "counts")
+ + + +

We can use the class function to see what type of object the assay() function returns.

+ + + +
# Check what type of object `counts_mat` is
+class(counts_mat)
+ + +
[1] "matrix" "array"
+ + +
matrix
+
+array
+ + + +

Alternatively, we could extract the TPM data – called abundance from gene_summarized.

+ + + +
# Let's look at the first few rows of the gene-level TPM
+head(assay(gene_summarized, "abundance"))
+ + +
                SRR585570 SRR585571  SRR585572 SRR585573 SRR585574 SRR585575
+ENSG00000000003 25.032603 18.896831  12.288181 26.911045 22.088410 17.168736
+ENSG00000000005  0.121595  0.000000   0.000000  0.000000  0.000000  0.000000
+ENSG00000000419 26.679297 20.771196 103.246348 69.495297 66.335181 77.471536
+ENSG00000000457  5.655858  2.921236   6.511128  5.107480  5.106009  3.845323
+ENSG00000000460  1.757408  2.933740   1.354462  2.195826  6.341131 16.792151
+ENSG00000000938  1.692637  2.807437   0.078625  0.000000  0.028502  0.000000
+                SRR585576 SRR585577
+ENSG00000000003 17.974009 29.513266
+ENSG00000000005  0.000000  0.000000
+ENSG00000000419 44.036543 35.660794
+ENSG00000000457  4.452530  3.346821
+ENSG00000000460  5.089307  7.927036
+ENSG00000000938  0.000000  0.000000
+ + + +
+
+

Save to file

+

We could use readr::write_tsv to save counts_mat only but gene_summarized has a lot of information stored here beyond the counts, so we may want to save all of this to a RDS object.

+ + + +
# Write `gene_summarized` to RDS object
+readr::write_rds(gene_summarized, file = txi_out_file)
+ + + +

We’ll import this with the DESeq2 package in the next notebook.

+
+
+

Session Info

+

Record session info for reproducibility & provenance purposes.

+ + + +
sessionInfo()
+ + +
R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
+[8] methods   base     
+
+other attached packages:
+ [1] ensembldb_2.14.0            AnnotationFilter_1.14.0    
+ [3] GenomicFeatures_1.42.1      AnnotationDbi_1.52.0       
+ [5] SummarizedExperiment_1.20.0 Biobase_2.50.0             
+ [7] GenomicRanges_1.42.0        GenomeInfoDb_1.26.2        
+ [9] IRanges_2.24.1              S4Vectors_0.28.1           
+[11] BiocGenerics_0.36.0         MatrixGenerics_1.2.0       
+[13] matrixStats_0.57.0          tximeta_1.8.4              
+[15] magrittr_2.0.1              optparse_1.6.6             
+
+loaded via a namespace (and not attached):
+ [1] httr_1.4.2                    jsonlite_1.7.2               
+ [3] bit64_4.0.5                   AnnotationHub_2.22.0         
+ [5] shiny_1.6.0                   assertthat_0.2.1             
+ [7] interactiveDisplayBase_1.28.0 askpass_1.1                  
+ [9] BiocManager_1.30.10           BiocFileCache_1.14.0         
+[11] blob_1.2.1                    GenomeInfoDbData_1.2.4       
+[13] Rsamtools_2.6.0               yaml_2.2.1                   
+[15] progress_1.2.2                BiocVersion_3.12.0           
+[17] lattice_0.20-41               pillar_1.4.7                 
+[19] RSQLite_2.2.3                 glue_1.4.2                   
+[21] digest_0.6.27                 promises_1.1.1               
+[23] XVector_0.30.0                Matrix_1.3-2                 
+[25] htmltools_0.5.1.1             httpuv_1.5.5                 
+[27] XML_3.99-0.5                  pkgconfig_2.0.3              
+[29] biomaRt_2.46.3                zlibbioc_1.36.0              
+[31] purrr_0.3.4                   xtable_1.8-4                 
+[33] getopt_1.20.3                 later_1.1.0.1                
+[35] BiocParallel_1.24.1           tibble_3.0.5                 
+[37] openssl_1.4.3                 generics_0.1.0               
+[39] ellipsis_0.3.1                withr_2.4.0                  
+[41] cachem_1.0.1                  lazyeval_0.2.2               
+[43] cli_2.2.0                     crayon_1.3.4                 
+[45] mime_0.9                      ps_1.5.0                     
+[47] memoise_1.1.0                 evaluate_0.14                
+[49] fansi_0.4.2                   xml2_1.3.2                   
+[51] tools_4.0.3                   prettyunits_1.1.1            
+[53] hms_1.0.0                     lifecycle_0.2.0              
+[55] stringr_1.4.0                 DelayedArray_0.16.2          
+[57] Biostrings_2.58.0             compiler_4.0.3               
+[59] rlang_0.4.10                  grid_4.0.3                   
+[61] RCurl_1.98-1.2                rstudioapi_0.13              
+[63] tximport_1.18.0               rappdirs_0.3.1               
+[65] bitops_1.0-6                  rmarkdown_2.6                
+[67] DBI_1.1.1                     curl_4.3                     
+[69] R6_2.5.0                      GenomicAlignments_1.26.0     
+[71] knitr_1.30                    dplyr_1.0.3                  
+[73] rtracklayer_1.50.0            fastmap_1.1.0                
+[75] bit_4.0.4                     ProtGenerics_1.22.0          
+[77] readr_1.4.0                   stringi_1.5.3                
+[79] Rcpp_1.0.6                    vctrs_0.3.6                  
+[81] dbplyr_2.0.0                  tidyselect_1.1.0             
+[83] xfun_0.20
+ + +
+ +
LS0tCnRpdGxlOiAiR2FzdHJpYyBjYW5jZXI6IGdlbmUtbGV2ZWwgc3VtbWFyaXphdGlvbiB3aXRoIGB0eGltZXRhYCIKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjEKb3V0cHV0OiAgIAogIGh0bWxfbm90ZWJvb2s6IAogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBJbXBvcnQgUk5BLXNlcSBleHByZXNzaW9uIHF1YW50aWZpY2F0aW9uIG91dHB1dCB1c2luZyBgdHhpbWV0YWAKLSBTdW1tYXJpemUgdHJhbnNjcmlwdC1sZXZlbCBleHByZXNzaW9uIHRvIHRoZSBnZW5lIGxldmVsCi0gSW50ZXJyb2dhdGUgYW5kIGV4dHJhY3QgZGF0YSBmcm9tIGEgYFN1bW1hcml6ZWRFeHBlcmltZW50YCBvYmplY3QgCgotLS0KCkluIHRoaXMgbm90ZWJvb2ssIHdlJ2xsIGltcG9ydCB0aGUgdHJhbnNjcmlwdCBleHByZXNzaW9uIHF1YW50aWZpY2F0aW9uIG91dHB1dCBmcm9tIGBzYWxtb24gcXVhbnRgIHVzaW5nIHRoZSBbYHR4aW1ldGFgXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL2h0bWwvdHhpbWV0YS5odG1sKSBwYWNrYWdlLgpgdHhpbWV0YWAgaXMgaW4gcGFydCBhIHdyYXBwZXIgYXJvdW5kIGFub3RoZXIgcGFja2FnZSwgW2B0eGltcG9ydGBdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC90eGltcG9ydC5odG1sKSwgd2hpY2ggaW1wb3J0cyB0cmFuc2NyaXB0IGV4cHJlc3Npb24gZGF0YSBhbmQgc3VtbWFyaXplcyBpdCB0byB0aGUgZ2VuZSBsZXZlbC4KV29ya2luZyBhdCB0aGUgZ2VuZSByYXRoZXIgdGhhbiB0cmFuc2NyaXB0IGxldmVsIGhhcyBhIG51bWJlciBvZiBwb3RlbnRpYWwgYWR2YW50YWdlcyBmb3IgaW50ZXJwcmV0YWJpbGl0eSwgZWZmaWNpZW5jeSwgYW5kIHJlZHVjdGlvbiBvZiBmYWxzZSBwb3NpdGl2ZXMgKFtTb25lc29uIF9ldCBhbC5fIDIwMTZdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMjY4OC9mMTAwMHJlc2VhcmNoLjc1NjMuMikpLgpgdHhpbWV0YWAgZWFzZXMgc29tZSBvZiB0aGUgYnVyZGVuIG9mIGltcG9ydCBieSBhdXRvbWF0aWNhbGx5IGlkZW50aWZ5aW5nIHRoZSBjb3JyZWN0IHNldCBvZiBhbm5vdGF0aW9uIGRhdGEgdG8gYXBwZW5kIHRvIG1hbnkgZGF0YSBzZXRzIChbTG92ZSBfZXQgYWwuXyAyMDIwXShodHRwczovL2pvdXJuYWxzLnBsb3Mub3JnL3Bsb3Njb21wYmlvbC9hcnRpY2xlP2lkPTEwLjEzNzEvam91cm5hbC5wY2JpLjEwMDc2NjQpKS4KCiFbXShkaWFncmFtcy9ybmEtc2VxXzUucG5nKQoKRm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgYHR4aW1ldGFgLCBzZWUgW3RoaXMgZXhjZWxsZW50IHZpZ25ldHRlXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL3R4aW1ldGEvaW5zdC9kb2MvdHhpbWV0YS5odG1sKSBmcm9tIExvdmUgX2V0IGFsXy4KCiMjIExpYnJhcmllcyBhbmQgZnVuY3Rpb25zCgpgYGB7ciBsaWJyYXJ5fQojIExvYWQgbWFncml0dHIgZm9yIHRoZSBwaXBlCmxpYnJhcnkobWFncml0dHIpCgojIExvYWQgdGhlIHR4aW1ldGEgcGFja2FnZQpsaWJyYXJ5KHR4aW1ldGEpCgojIExvYWQgdGhlIFN1bW1hcml6ZWRFeHBlcmltZW50IHBhY2thZ2UKbGlicmFyeShTdW1tYXJpemVkRXhwZXJpbWVudCkKYGBgCgojIyBEaXJlY3RvcmllcyBhbmQgZmlsZXMKCmBgYHtyIGRpcmVjdG9yaWVzLCBsaXZlID0gVFJVRX0KIyBkaXJlY3Rvcnkgd2hlcmUgdGhlIGRhdGEgYXJlIGxvY2F0ZWQKZGF0YV9kaXIgPC0gZmlsZS5wYXRoKCJkYXRhIiwgImdhc3RyaWMtY2FuY2VyIikKCiMgZGlyZWN0b3J5IHdoZXJlIHRoZSBxdWFudCBmaWxlcyBhcmUgbG9jYXRlZCwgZWFjaCBzYW1wbGUgaXMgaXRzIG93bgojIGRpcmVjdG9yeQpxdWFudF9kaXIgPC0gZmlsZS5wYXRoKGRhdGFfZGlyLCAic2FsbW9uX3F1YW50IikKCiMgY3JlYXRlIGEgZGlyZWN0b3J5IHRvIGhvbGQgdGhlIHR4aW1ldGEgcmVzdWx0cyBpZiBpdCBkb2Vzbid0IGV4aXN0IHlldAp0eGlfZGlyIDwtIGZpbGUucGF0aChkYXRhX2RpciwgInR4aSIpCmlmICghZGlyLmV4aXN0cyh0eGlfZGlyKSkgewogIGRpci5jcmVhdGUodHhpX2RpciwgcmVjdXJzaXZlID0gVFJVRSkKfQpgYGAKCldlJ2xsIG5lZWQgdGhlIGBxdWFudC5zZmAgZmlsZXMgZm9yIGFsbCB0aGUgc2FtcGxlcyBpbiBhbiBleHBlcmltZW50IHdoaWNoIHdlIGhhdmUgc3RvcmVkIGluIGBxdWFudF9kaXJgLiAKCmBgYHtyIGlucHV0LW5hbWVzfQojIHRoZSBxdWFudCBmaWxlcyB0aGVtc2VsdmVzCnNmX2ZpbGVzIDwtIGxpc3QuZmlsZXMocXVhbnRfZGlyLCByZWN1cnNpdmUgPSBUUlVFLCBmdWxsLm5hbWVzID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgICBwYXR0ZXJuID0gInF1YW50LnNmIikKYGBgCgpgYGB7ciBtZXRhZGF0YS1maWxlfQojIHNhbXBsZSBtZXRhZGF0YSBmaWxlCm1ldGFfZmlsZSA8LSBmaWxlLnBhdGgoZGF0YV9kaXIsICJnYXN0cmljLWNhbmNlcl9tZXRhZGF0YS50c3YiKQpgYGAKCioqT3V0cHV0KioKCmBgYHtyIG91dHB1dC1uYW1lcywgbGl2ZSA9IFRSVUV9CiMgTmFtZSB0aGUgb3V0cHV0IGdhc3RyaWMtY2FuY2VyX3R4aW1ldGEuUkRTIGFuZCB1c2UgdGhlIGRpcmVjdG9yeSBjcmVhdGVkIAojIGFib3ZlIGFzIHRoZSByZXN0IG9mIHRoZSBwYXRoCnR4aV9vdXRfZmlsZSA8LSBmaWxlLnBhdGgodHhpX2RpciwgImdhc3RyaWMtY2FuY2VyX3R4aW1ldGEuUkRTIikKYGBgCgojIyBGaWxlIG5hbWVzCgpBbGwgb3V0cHV0IGZpbGVzIGZyb20gYHNhbG1vbiBxdWFudGAgd2UnbGwgdXNlIHdpdGggYHR4aW1ldGFgIGFyZSBuYW1lZCBgcXVhbnQuc2ZgLgpVbmZvcnR1bmF0ZWx5LCB0aGlzIG1lYW5zIHRoYXQgdGhlIGZpbGUgbmFtZXMgdGhlbXNlbHZlcyBkbyBub3QgaGF2ZSBhbnkgaW5mb3JtYXRpb24gYWJvdXQgdGhlIHNhbXBsZSB0aGV5IGNvbWUgZnJvbSEKCmBgYHtyIHNmX2ZpbGVzLCBsaXZlID0gVFJVRX0KIyBMZXQncyBsb29rIGF0IHRoZSBmdWxsIHBhdGggZm9yIHRoZSBxdWFudC5zZiBmaWxlcwpzZl9maWxlcwpgYGAKCkxldCdzIGV4dHJhY3QgdGhlIF9zYW1wbGVfIG5hbWVzIGZyb20gdGhlICoqZmlsZSBwYXRocyoqIHVzaW5nIHRoZSBgc3RyaW5ncmAgcGFja2FnZS4KCk5vdGljZSBob3cgdGhlIGZpbGUgcGF0aCBpcyBzZXBhcmF0ZWQgYnkgYC9gLgpJZiB3ZSB3ZXJlIHRvIHNwbGl0IHVwIHRoaXMgY2hhcmFjdGVyIHN0cmluZyBieSBgL2AsIHRoZSBzZWNvbmQgdG8gbGFzdCBpdGVtIGlzIHRoZSBzYW1wbGUgbmFtZXMgKGJlY2F1c2Ugd2UgdXNlZCB0aGVtIGFzIGRpcmVjdG9yeSBuYW1lcyBmb3IgdGhlIGBzYWxtb25gIG91dHB1dCkuClRoaXMgaXMgZXhhY3RseSB3aGF0IGBzdHJpbmdyOjp3b3JkKClgIGFsbG93cyB1cyB0byBkbzogc3BsaXQgdXAgdGhlIGZpbGUgcGF0aHMgYnkgYC9gIGFuZCBleHRyYWN0IHRoZSBzYW1wbGUgbmFtZXMuCgpgYGB7ciBzYW1wbGVfbmFtZXN9CnNhbXBsZV9uYW1lcyA8LSBzdHJpbmdyOjp3b3JkKHNmX2ZpbGVzLCAtMiwgc2VwID0gIi8iKQpzYW1wbGVfbmFtZXMKYGBgCgojIyBTZXQgdXAgbWV0YWRhdGEKCmB0eGltZXRhYCBuZWVkcyBhIGRhdGEgZnJhbWUgd2l0aCBhdCBsZWFzdCB0aGVzZSB0d28gY29sdW1uczogCi0gYSBgZmlsZXNgIGNvbHVtbiAgd2l0aCB0aGUgZmlsZSBwYXRocyB0byB0aGUgcXVhbnQuc2YgZmlsZXMKLSBhIGBuYW1lc2AgY29sdW1uIHdpdGggdGhlIHNhbXBsZSBuYW1lcwoKYGBge3IgbmFtZXNfc2ZfZmlsZXN9CmNvbGRhdGEgPC0gZGF0YS5mcmFtZShmaWxlcyA9IHNmX2ZpbGVzLCAKICAgICAgICAgICAgICAgICAgICAgIG5hbWVzID0gc2FtcGxlX25hbWVzKQpgYGAKCldlIGhhdmUgbW9yZSBpbmZvcm1hdGlvbiBhYm91dCB0aGVzZSBzYW1wbGVzIHN0b3JlZCBpbiB0aGUgbWV0YWRhdGEgZmlsZSB0aGF0IHdlIHdpbGwgYWxzbyB3YW50IHN0b3JlZCBpbiBgY29sZGF0YWAuIApMZXQncyByZWFkIGluIHRoZSBzYW1wbGUgbWV0YWRhdGEgZnJvbSB0aGUgVFNWIGZpbGUuCgpgYGB7ciBzYW1wbGVfbWV0YV9kZiwgbGl2ZSA9IFRSVUV9CiMgUmVhZCBpbiB0aGUgc2FtcGxlIG1ldGFkYXRhIFRTViBmaWxlIGFuZCBoYXZlIGEgbG9vawpzYW1wbGVfbWV0YV9kZiA8LSByZWFkcjo6cmVhZF90c3YobWV0YV9maWxlKQpzYW1wbGVfbWV0YV9kZgpgYGAKCldlJ2xsIHdhbnQgdGhpcyBpbmZvcm1hdGlvbiB0byBiZSBhZGRlZCB0byB0aGUgYGNvbGRhdGFgLCB3aGljaCB3ZSBjYW4gZG8gYnkgdXNpbmcgYSBqb2luIGZ1bmN0aW9uIHRvIG1hdGNoIHVwIHRoZSByb3dzIGJldHdlZW4gdGhlIHR3byBkYXRhIGZyYW1lcyBhbmQgY29tYmluZSB0aGVtLiAKCmBgYHtyIGpvaW4tc2FtcGxlX21ldGFfZGZ9CmNvbGRhdGEgPC0gY29sZGF0YSAlPiUKICBkcGx5cjo6aW5uZXJfam9pbihzYW1wbGVfbWV0YV9kZiwgYnkgPSBjKCJuYW1lcyIgPSAic3JyX2FjY2Vzc2lvbiIpKQoKY29sZGF0YQpgYGAKCiMjIEltcG9ydCBleHByZXNzaW9uIGRhdGEgd2l0aCBgdHhpbWV0YWAKClVzaW5nIHRoZSBgY29sZGF0YWAgZGF0YSBmcmFtZSB0aGF0IHdlIHNldCB1cCwgd2UgY2FuIG5vdyBydW4gdGhlIGB0eGltZXRhKClgIHRvIGltcG9ydCBvdXIgZXhwcmVzc2lvbiBkYXRhIHdoaWxlIGF1dG9tYXRpY2FsbHkgZmluZGluZyBhbmQgYXNzb2NpYXRpbmcgdGhlIHRyYW5zY3JpcHQgYW5ub3RhdGlvbnMgdGhhdCB3ZXJlIHVzZWQgd2hlbiB3ZSBwZXJmb3JtZWQgdGhlIHF1YW50aWZpY2F0aW9uLgoKVGhlIGZpcnN0IHRpbWUgeW91IHJ1biBgdHhpbWV0YSgpYCB5b3UgbWF5IGdldCBhIG1lc3NhZ2UgYWJvdXQgc3RvcmluZyBkb3dubG9hZGVkIHRyYW5zY3JpcHRvbWUgZGF0YSBpbiBhIGNhY2hlIGRpcmVjdG9yeSBzbyB0aGF0IGl0IGNhbiByZXRyaWV2ZSB0aGUgZGF0YSBtb3JlIHF1aWNrbHkgdGhlIG5leHQgdGltZS4KV2UgcmVjb21tZW5kIHlvdSB1c2UgdGhlIGNhY2hlLCBhbmQgYWNjZXB0IHRoZSBkZWZhdWx0IGxvY2F0aW9uLgoKYGBge3IgdHhpbWV0YSwgbGl2ZSA9IFRSVUV9CnR4aV9kYXRhIDwtIHR4aW1ldGEoY29sZGF0YSkKYGBgCgoqdHhpbWV0YSBjdXJyZW50bHkgd29ya3MgZWFzaWx5IGZvciBtb3N0IGh1bWFuIGFuZCBtb3VzZSBkYXRhc2V0cywgYnV0IHJlcXVpcmVzIGEgW2ZldyBtb3JlIHN0ZXBzIGZvciBvdGhlciBzcGVjaWVzXShodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy90eGltZXRhL2luc3QvZG9jL3R4aW1ldGEuaHRtbCNXaGF0X2lmX2NoZWNrc3VtX2lzbiVFMiU4MCU5OXRfa25vd24pLgoKIyMgU3VtbWFyaXplIHRvIGdlbmUKCldlJ2xsIHN1bW1hcml6ZSB0byB0aGUgZ2VuZSBsZXZlbCB1c2luZyB0aGUgYHN1bW1hcml6ZVRvR2VuZSgpYCBmdW5jdGlvbi4KCmBgYHtyIHN1bW1hcml6ZS1nZW5lfQojIFN1bW1hcml6ZSB0byB0aGUgZ2VuZSBsZXZlbApnZW5lX3N1bW1hcml6ZWQgPC0gc3VtbWFyaXplVG9HZW5lKHR4aV9kYXRhKSAKYGBgCgpXZSBjYW4gdXNlIHRoZSBgY2xhc3NgIGZ1bmN0aW9uIHRvIHNlZSB3aGF0IHR5cGUgb2Ygb2JqZWN0IGBnZW5lX3N1bW1hcml6ZWRgIGlzLgoKYGBge3IgY2xhc3MsIGxpdmUgPSBUUlVFfQojIENoZWNrIHdoYXQgdHlwZSBvZiBvYmplY3QgYGdlbmVfc3VtbWFyaXplZGAgaXMKY2xhc3MoZ2VuZV9zdW1tYXJpemVkKQpgYGAKClRoaXMgdGVsbHMgdXMgdGhhdCBgZ2VuZV9zdW1tYXJpemVkYCBpcyBhbiBvYmplY3QgY2FsbGVkIGEgW2BTdW1tYXJpemVkRXhwZXJpbWVudGBdKGh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9TdW1tYXJpemVkRXhwZXJpbWVudC92ZXJzaW9ucy8xLjIuMy90b3BpY3MvU3VtbWFyaXplZEV4cGVyaW1lbnQtY2xhc3MpIHdoaWNoIGNhbiBiZSBoYW5kbGVkIGJ5IGZ1bmN0aW9ucyBmcm9tIHRoZSBwYWNrYWdlIG9mIHRoZSBzYW1lIG5hbWUuIApXZSBtb3JlIHNwZWNpZmljYWxseSBoYXZlIGEgYFJhbmdlZFN1bW1hcml6ZWRFeHBlcmltZW50YCB3aGljaCBpcyBhIG1vcmUgc3BlY2lmaWMgdHlwZSBvZiBgU3VtbWFyaXplZEV4cGVyaW1lbnRgLgoKYFN1bW1hcml6ZWRFeHBlcmltZW50YCBvYmplY3RzIGhhdmUgdGhpcyBnZW5lcmFsIHN0cnVjdHVyZTogIAoKIVtTdW1tYXJpemVkRXhwZXJpbWVudF0oZGlhZ3JhbXMvU3VtbWFyaXplRXhwZXJpbWVudC1zdHJ1Y3R1cmUucG5nKQoKVGhpcyBmaWd1cmUgaXMgZnJvbSB0aGlzIGhhbmR5IHZpZ25ldHRlIGFib3V0IFtgU3VtbWFyaXplZEV4cGVyaW1lbnRgIG9iamVjdHNdKGh0dHBzOi8vd3d3LmJpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvU3VtbWFyaXplZEV4cGVyaW1lbnQvaW5zdC9kb2MvU3VtbWFyaXplZEV4cGVyaW1lbnQuaHRtbCkuCgpBcyBzaG93biBpbiB0aGUgZGlhZ3JhbSwgd2UgY2FuIHVzZSBzb21lIG9mIHRoZSBmdW5jdGlvbnMgcHJvdmlkZWQgYnkgdGhlIGBTdW1tYXJpemVkRXhwZXJpbWVudGAgcGFja2FnZSB0byBleHRyYWN0IGRhdGEgZnJvbSBvdXIgYGdlbmVfc3VtbWFyaXplZGAgb2JqZWN0LiAKRm9yIGV4YW1wbGUsIGNhbGxpbmcgYHJvd0RhdGEoKWAgb24gb3VyIG9iamVjdCBzaG93cyBhbGwgdGhlIGdlbmUgaW5mb3JtYXRpb24gdGhhdCBgdHhpbWV0YWAgc2V0IHVwIQoKYGBge3J9CiMgcm93RGF0YSgpIHNob3dzIHVzIG91ciBnZW5lIGFubm90YXRpb24Kcm93RGF0YShnZW5lX3N1bW1hcml6ZWQpCmBgYAoKVGhlIGBhc3NheWAgc2xvdCBpbiBgU3VtbWFyaXplZEV4cGVyaW1lbnRgcyBob2xkcyBkYXRhIGZyb20gdGhlIGV4cGVyaW1lbnQuIApJbiB0aGlzIGNhc2UsIGl0IHdpbGwgaW5jbHVkZSBvdXIgZ2VuZS1sZXZlbCBleHByZXNzaW9uIGluZm9ybWF0aW9uIHN0b3JlZCBhcyBhIGdlbmUgeCBzYW1wbGUgbWF0cml4LiAKCk11bHRpcGxlIGBhc3NheXNgIGNhbiBiZSBzdG9yZWQgaW4gYW4gYFN1bW1hcml6ZWRFeHBlcmltZW50YCBhbmQgd2UgY2FuIHVzZSB0aGUgYGFzc2F5TmFtZXMoKWAgZnVuY3Rpb24gdG8gc2VlIHdoYXQgYXNzYXlzIGFyZSBpbmNsdWRlZCBpbiBgZ2VuZV9zdW1tYXJpemVkYC4gCgpgYGB7ciBhc3NheS1uYW1lcywgbGl2ZSA9IFRSVUV9CmFzc2F5TmFtZXMoZ2VuZV9zdW1tYXJpemVkKQpgYGAKCklmIHdlIHdhbnQgdG8gZXh0cmFjdCBhbiBgYXNzYXlgJ3MgZGF0YSwgd2UgY2FuIHVzZSBgYXNzYXkoKWAgZnVuY3Rpb24gYW5kIHNwZWNpZnkgdGhlIG5hbWUgb2YgdGhlIGFzc2F5IHdlIHdhbnQgdG8gZXh0cmFjdC4KCmBgYHtyIGFzc2F5LWNvdW50cywgbGl2ZSA9IFRSVUV9CmNvdW50c19tYXQgPC0gYXNzYXkoZ2VuZV9zdW1tYXJpemVkLCAiY291bnRzIikKYGBgCgpXZSBjYW4gdXNlIHRoZSBgY2xhc3NgIGZ1bmN0aW9uIHRvIHNlZSB3aGF0IHR5cGUgb2Ygb2JqZWN0IHRoZSBgYXNzYXkoKWAgZnVuY3Rpb24gcmV0dXJucy4KCmBgYHtyIGNsYXNzLWNvdW50cywgbGl2ZSA9IFRSVUV9CiMgQ2hlY2sgd2hhdCB0eXBlIG9mIG9iamVjdCBgY291bnRzX21hdGAgaXMKY2xhc3MoY291bnRzX21hdCkKYGBgCgpBbHRlcm5hdGl2ZWx5LCB3ZSBjb3VsZCBleHRyYWN0IHRoZSBUUE0gZGF0YSAtLSBjYWxsZWQgYGFidW5kYW5jZWAgZnJvbSBgZ2VuZV9zdW1tYXJpemVkYC4KCmBgYHtyIGFzc2F5LWFidW5kYW5jZSwgbGl2ZSA9IFRSVUV9CiMgTGV0J3MgbG9vayBhdCB0aGUgZmlyc3QgZmV3IHJvd3Mgb2YgdGhlIGdlbmUtbGV2ZWwgVFBNCmhlYWQoYXNzYXkoZ2VuZV9zdW1tYXJpemVkLCAiYWJ1bmRhbmNlIikpCmBgYAoKIyMgU2F2ZSB0byBmaWxlCgpXZSBjb3VsZCB1c2UgYHJlYWRyOjp3cml0ZV90c3ZgIHRvIHNhdmUgYGNvdW50c19tYXRgIG9ubHkgYnV0IGBnZW5lX3N1bW1hcml6ZWRgIGhhcyBhIGxvdCBvZiBpbmZvcm1hdGlvbiBzdG9yZWQgaGVyZSBiZXlvbmQgdGhlIGNvdW50cywgc28gd2UgbWF5IHdhbnQgdG8gc2F2ZSBhbGwgb2YgdGhpcyB0byBhIFJEUyBvYmplY3QuIAoKYGBge3Igd3JpdGUtdHhpLCBsaXZlID0gVFJVRX0KIyBXcml0ZSBgZ2VuZV9zdW1tYXJpemVkYCB0byBSRFMgb2JqZWN0CnJlYWRyOjp3cml0ZV9yZHMoZ2VuZV9zdW1tYXJpemVkLCBmaWxlID0gdHhpX291dF9maWxlKQpgYGAKCldlJ2xsIGltcG9ydCB0aGlzIHdpdGggdGhlIGBERVNlcTJgIHBhY2thZ2UgaW4gdGhlIG5leHQgbm90ZWJvb2suCgojIyBTZXNzaW9uIEluZm8KClJlY29yZCBzZXNzaW9uIGluZm8gZm9yIHJlcHJvZHVjaWJpbGl0eSAmIHByb3ZlbmFuY2UgcHVycG9zZXMuCgpgYGB7ciBzZXNzaW9uaW5mb30Kc2Vzc2lvbkluZm8oKQpgYGAK
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/RNA-seq/03-gastric_cancer_exploratory-live.Rmd b/RNA-seq/03-gastric_cancer_exploratory-live.Rmd index 40ccc83f..7fb1ccaf 100644 --- a/RNA-seq/03-gastric_cancer_exploratory-live.Rmd +++ b/RNA-seq/03-gastric_cancer_exploratory-live.Rmd @@ -1,12 +1,22 @@ --- title: "Gastric cancer: exploratory analysis" +author: CCDL for ALSF +date: 2021 output: html_notebook: toc: true toc_float: true --- -**CCDL 2021** +## Objectives + +This notebook will demonstrate how to: + +- Create a `DESeq2` data set from a `SummarizedExperiment` +- Transform RNA-seq count data with a Variance Stabilizing Transformation +- Create PCA plots to explore structure among RNA-seq samples + +--- In this notebook, we'll import the gastric cancer data and do some exploratory analyses and visual inspection. @@ -14,10 +24,8 @@ We'll use the [`DESeq2`](https://bioconductor.org/packages/release/bioc/html/DES ![](diagrams/rna-seq_6.png) -`DESeq2` also has an -[excellent vignette](https://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html) -from Love, Anders, and Huber from which this is adapted -(see also: [Love, Anders, and Huber. _Genome Biology_. 2014.](https://doi.org/10.1186/s13059-014-0550-8)). +`DESeq2` also has an [excellent vignette](https://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html) +from Love, Anders, and Huber from which this is adapted (see also: [Love, Anders, and Huber. _Genome Biology_. 2014.](https://doi.org/10.1186/s13059-014-0550-8)). ## Libraries and functions @@ -75,8 +83,7 @@ First, let's read in the data we processed with `tximeta`. ### Set up DESeq2 object -We use the tissue of origin in the design formula because that will allow us -to model this variable of interest. +We use the tissue of origin in the design formula because that will allow us to model this variable of interest. ```{r ddset} ddset <- DESeqDataSet(gene_summarized, @@ -85,27 +92,33 @@ ddset <- DESeqDataSet(gene_summarized, ### Variance stabilizing transformation -Before visualizing the data, we'll transform it such that it is on a `log2` -scale for large counts and library size is taken into account with the `DESeq2` -function for variance stabilizing transformation. -See [this section of -the `DESeq2` vignette](http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#data-transformations-and-visualization) -for more on this topic. +Raw count data is not usually suitable for the algorithms we use for dimensionality reduction, clustering, or heatmaps. +To improve this, we will transform the count data to create an expression measure that is better suited for these analyses. +The core transformation will map the expression to a log2 scale, while accounting for some of the expected variation among samples and genes. + +Since different samples are usually sequenced to different depths, we want to transform our RNA-seq count data to make different samples more directly comparable. +We also want to deal with the fact that genes with low counts are also likely to have higher variance (on the log2 scale), as that could bias our clustering. +To handle both of these considerations, we can calculate a Variance Stabilizing Transformation of the count data, and work with that transformed data for our analysis. + +See [this section of the `DESeq2` vignette](http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#data-transformations-and-visualization) for more on this topic. ```{r vst} vst_data <- vst(ddset) ``` -### Principal components analysis +### Principal component analysis + +Principal component analysis (PCA) is a dimensionality reduction technique that allows us to identify the largest components of variation in a complex dataset. +Our expression data can be thought of as mapping each sample in a multidimensional space defined by the expression level of each gene. +The expression of many of those genes are correlated, so we can often get a better, simpler picture of the data by combining the information from those correlated genes. + +PCA rotates and transforms this space so that each axis is now a combination of multiple correlated genes, ordered so the first axes capture the most variation from the data. +These new axes are the "principal components." +If we look at the first few components, we can often get a nice overview of relationships among the samples in the data. -Principal components analysis (PCA) is a dimensionality reduction technique -that captures the main sources of variation in our data in the first two -principal components (PC1 and PC2). -Visualizing PC1 and PC2 can give us insight into how different variables (e.g., -tissue source) affect our dataset and help us spot any technical effects -(more on that below). +The `plotPCA()` function we will use from the `DESeq2` package calculates and plots the first two principal components (PC1 and PC2). +Visualizing PC1 and PC2 can give us insight into how different variables (e.g., tissue source) affect our dataset and help us spot any technical effects (more on that below). -`DESeq2` has built-in functionality for performing PCA. ```{r plotPCA, live = TRUE} # DESeq2 built in function is called plotPCA and we want to color points by diff --git a/RNA-seq/03-gastric_cancer_exploratory.nb.html b/RNA-seq/03-gastric_cancer_exploratory.nb.html new file mode 100644 index 00000000..bea05544 --- /dev/null +++ b/RNA-seq/03-gastric_cancer_exploratory.nb.html @@ -0,0 +1,828 @@ + + + + + + + + + + + + + + + +Gastric cancer: exploratory analysis + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Create a DESeq2 data set from a SummarizedExperiment
    • +
  • Transform RNA-seq count data with a Variance Stabilizing Transformation
    • +
  • Create PCA plots to explore structure among RNA-seq samples
    • +
+
+

In this notebook, we’ll import the gastric cancer data and do some exploratory analyses and visual inspection. We’ll use the DESeq2 package for this.

+

+

DESeq2 also has an excellent vignette from Love, Anders, and Huber from which this is adapted (see also: Love, Anders, and Huber. Genome Biology. 2014.).

+
+
+

Libraries and functions

+ + + +
# Load the DESeq2 library
+library(DESeq2)
+ + +
Loading required package: S4Vectors
+ + +
Loading required package: stats4
+ + +
Loading required package: BiocGenerics
+ + +
Loading required package: parallel
+ + +

+Attaching package: 'BiocGenerics'
+ + +
The following objects are masked from 'package:parallel':
+
+    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
+    clusterExport, clusterMap, parApply, parCapply, parLapply,
+    parLapplyLB, parRapply, parSapply, parSapplyLB
+ + +
The following objects are masked from 'package:stats':
+
+    IQR, mad, sd, var, xtabs
+ + +
The following objects are masked from 'package:base':
+
+    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
+    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
+    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
+    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
+    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
+    union, unique, unsplit, which.max, which.min
+ + +

+Attaching package: 'S4Vectors'
+ + +
The following object is masked from 'package:base':
+
+    expand.grid
+ + +
Loading required package: IRanges
+ + +
Loading required package: GenomicRanges
+ + +
Loading required package: GenomeInfoDb
+ + +
Loading required package: SummarizedExperiment
+ + +
Loading required package: MatrixGenerics
+ + +
Loading required package: matrixStats
+ + +

+Attaching package: 'MatrixGenerics'
+ + +
The following objects are masked from 'package:matrixStats':
+
+    colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
+    colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
+    colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
+    colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
+    colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
+    colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
+    colWeightedMeans, colWeightedMedians, colWeightedSds,
+    colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
+    rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
+    rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
+    rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
+    rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
+    rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
+    rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
+    rowWeightedSds, rowWeightedVars
+ + +
Loading required package: Biobase
+ + +
Welcome to Bioconductor
+
+    Vignettes contain introductory material; view with
+    'browseVignettes()'. To cite Bioconductor, see
+    'citation("Biobase")', and for packages 'citation("pkgname")'.
+ + +

+Attaching package: 'Biobase'
+ + +
The following object is masked from 'package:MatrixGenerics':
+
+    rowMedians
+ + +
The following objects are masked from 'package:matrixStats':
+
+    anyMissing, rowMedians
+ + + + + + +
# magrittr for the pipe
+library(magrittr)
+ + + +
+
+

Directories and files

+ + + +
# Main data directory
+data_dir <- file.path("data", "gastric-cancer")
+
+# directory with the tximeta processed data
+txi_dir <- file.path(data_dir, "txi")
+txi_file <- file.path(txi_dir, "gastric-cancer_tximeta.RDS")
+ + + + + + +
# sample metadata file
+meta_file <- file.path(data_dir, "gastric-cancer_metadata.tsv")
+ + + +

We’ll create a directory to hold our plots.

+ + + +
# Create a plots directory if it does not exist yet
+plots_dir <- file.path("plots", "gastric-cancer")
+if (!dir.exists(plots_dir)) {
+  dir.create(plots_dir, recursive = TRUE)
+}
+ + + +

Output

+ + + +
# We will save a PDF copy of the PCA plot to the plots directory
+# and name the file "gastric-cancer_PC_scatter.pdf"
+pca_plot_file <- file.path(plots_dir, "gastric-cancer_PC_scatter.pdf")
+ + + +
+
+

DESeq2

+
+

Creating a DESeq2 dataset from a tximeta object

+

First, let’s read in the data we processed with tximeta.

+ + + +
# Read in the RDS file we created in the last notebook
+gene_summarized <- readr::read_rds(txi_file)
+ + + +
+
+

Set up DESeq2 object

+

We use the tissue of origin in the design formula because that will allow us to model this variable of interest.

+ + + +
ddset <- DESeqDataSet(gene_summarized,
+                      design = ~ tissue)
+ + +
using counts and average transcript lengths from tximeta
+ + +
Warning in DESeqDataSet(gene_summarized, design = ~tissue): some variables in
+design formula are characters, converting to factors
+ + + +
+
+

Variance stabilizing transformation

+

Raw count data is not usually suitable for the algorithms we use for dimensionality reduction, clustering, or heatmaps. To improve this, we will transform the count data to create an expression measure that is better suited for these analyses. The core transformation will map the expression to a log2 scale, while accounting for some of the expected variation among samples and genes.

+

Since different samples are usually sequenced to different depths, we want to transform our RNA-seq count data to make different samples more directly comparable. We also want to deal with the fact that genes with low counts are also likely to have higher variance (on the log2 scale), as that could bias our clustering. To handle both of these considerations, we can calculate a Variance Stabilizing Transformation of the count data, and work with that transformed data for our analysis.

+

See this section of the DESeq2 vignette for more on this topic.

+ + + +
vst_data <- vst(ddset)
+ + +
using 'avgTxLength' from assays(dds), correcting for library size
+ + + +
+
+

Principal component analysis

+

Principal component analysis (PCA) is a dimensionality reduction technique that allows us to identify the largest components of variation in a complex dataset. Our expression data can be thought of as mapping each sample in a multidimensional space defined by the expression level of each gene. The expression of many of those genes are correlated, so we can often get a better, simpler picture of the data by combining the information from those correlated genes.

+

PCA rotates and transforms this space so that each axis is now a combination of multiple correlated genes, ordered so the first axes capture the most variation from the data. These new axes are the “principal components.” If we look at the first few components, we can often get a nice overview of relationships among the samples in the data.

+

The plotPCA() function we will use from the DESeq2 package calculates and plots the first two principal components (PC1 and PC2). Visualizing PC1 and PC2 can give us insight into how different variables (e.g., tissue source) affect our dataset and help us spot any technical effects (more on that below).

+ + + +
# DESeq2 built in function is called plotPCA and we want to color points by
+# tissue
+plotPCA(vst_data, intgroup = "tissue")
+ + +

+ + + +

Save the most recent plot to file with ggsave from ggplot2

+ + + +
# Save the PDF file 
+ggplot2::ggsave(pca_plot_file, plot = ggplot2::last_plot())
+ + +
Saving 7 x 5 in image
+ + + +
+
+
+

A note on technical effects

+

We don’t have batch information (i.e., when the samples were run) for this particular experiment, but let’s imagine that SRR585574 and SRR585576 were run separately from all other samples. We’ll add this as a new “toy” column in the sample data (colData).

+ + + +
# Extract colData
+sample_info <- colData(vst_data)
+
+# Print out preview
+sample_info
+ + +
DataFrame with 8 rows and 3 columns
+                names                   tissue                  title
+          <character>                 <factor>            <character>
+SRR585570   SRR585570 gastric_normal           Gastric normal (CGC-..
+SRR585571   SRR585571 gastric_normal           Gastric normal (CGC-..
+SRR585572   SRR585572 primary_gastric_tumor    Primary gastric tumo..
+SRR585573   SRR585573 primary_gastric_tumor    Primary gastric tumo..
+SRR585574   SRR585574 primary_gastric_tumor    Primary gastric tumo..
+SRR585575   SRR585575 gastric_cancer_cell_line                 SNU484
+SRR585576   SRR585576 gastric_cancer_cell_line                 SNU601
+SRR585577   SRR585577 gastric_cancer_cell_line                 SNU668
+ + + +

Now we can add a new column with toy batch information and re-store the colData().

+ + + +
# Add batch information
+sample_info$batch <- c("batch1", "batch1", "batch1", "batch1", "batch2", 
+                       "batch1", "batch2", "batch1")
+ + + +

If this batch information were real we would have included it with the sample metadata when we made the original SummarizedExperiment object with tximeta. We would then include it in the model stored in our DESeq2 object using the design argument (design = ~ tissue + batch) and we would re-run the DESeqDataSet() and vst() steps we did above. Here we will take a bit of a shortcut and add it directly to the colData() for our vst()-transformed data.

+ + + +
# Add coldata() with batch info to vst_data
+colData(vst_data) <- sample_info
+ + + + + + +
# PCA plot - tissue *and* batch
+# We want plotPCA to return the data so we can have more control about the plot
+pca_data <- plotPCA(vst_data, 
+                    intgroup = c("tissue", "batch"), 
+                    returnData = TRUE)
+ + + + + + +
# Here we are setting up the percent variance that we are extracting from the `pca_data` object
+percent_var <- round(100 * attr(pca_data, "percentVar"))
+ + + +

Let’s use ggplot to visualize the first two principal components.

+ + + +
# Color points by "batch" and use shape to indicate the tissue of origin
+ggplot2::ggplot(pca_data, ggplot2::aes(PC1, PC2, 
+                                       color = batch, 
+                                       shape = tissue)) +
+  ggplot2::geom_point(size = 3) +
+  ggplot2::xlab(paste0("PC1: ", percent_var[1],"% variance")) +
+  ggplot2::ylab(paste0("PC2: ", percent_var[2],"% variance")) + 
+  ggplot2::coord_fixed()
+ + +

+ + + +
+
+

Session Info

+

Record session info for reproducibility & provenance purposes.

+ + + +
sessionInfo()
+ + +
R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
+[8] methods   base     
+
+other attached packages:
+ [1] magrittr_2.0.1              DESeq2_1.30.1              
+ [3] SummarizedExperiment_1.20.0 Biobase_2.50.0             
+ [5] MatrixGenerics_1.2.0        matrixStats_0.57.0         
+ [7] GenomicRanges_1.42.0        GenomeInfoDb_1.26.2        
+ [9] IRanges_2.24.1              S4Vectors_0.28.1           
+[11] BiocGenerics_0.36.0         optparse_1.6.6             
+
+loaded via a namespace (and not attached):
+ [1] locfit_1.5-9.4         Rcpp_1.0.6             lattice_0.20-41       
+ [4] assertthat_0.2.1       digest_0.6.27          R6_2.5.0              
+ [7] RSQLite_2.2.3          evaluate_0.14          httr_1.4.2            
+[10] ggplot2_3.3.3          pillar_1.4.7           zlibbioc_1.36.0       
+[13] rlang_0.4.10           annotate_1.68.0        blob_1.2.1            
+[16] Matrix_1.3-2           rmarkdown_2.6          labeling_0.4.2        
+[19] splines_4.0.3          BiocParallel_1.24.1    readr_1.4.0           
+[22] geneplotter_1.68.0     stringr_1.4.0          RCurl_1.98-1.2        
+[25] bit_4.0.4              munsell_0.5.0          DelayedArray_0.16.2   
+[28] compiler_4.0.3         xfun_0.20              pkgconfig_2.0.3       
+[31] htmltools_0.5.1.1      tidyselect_1.1.0       tibble_3.0.5          
+[34] GenomeInfoDbData_1.2.4 XML_3.99-0.5           crayon_1.3.4          
+[37] dplyr_1.0.3            bitops_1.0-6           grid_4.0.3            
+[40] jsonlite_1.7.2         xtable_1.8-4           gtable_0.3.0          
+[43] lifecycle_0.2.0        DBI_1.1.1              scales_1.1.1          
+[46] stringi_1.5.3          farver_2.0.3           XVector_0.30.0        
+[49] genefilter_1.72.1      getopt_1.20.3          ellipsis_0.3.1        
+[52] vctrs_0.3.6            generics_0.1.0         RColorBrewer_1.1-2    
+[55] tools_4.0.3            bit64_4.0.5            glue_1.4.2            
+[58] purrr_0.3.4            hms_1.0.0              survival_3.2-7        
+[61] yaml_2.2.1             AnnotationDbi_1.52.0   colorspace_2.0-0      
+[64] memoise_1.1.0          knitr_1.30
+ + +
+ +
LS0tCnRpdGxlOiAiR2FzdHJpYyBjYW5jZXI6IGV4cGxvcmF0b3J5IGFuYWx5c2lzIgphdXRob3I6IENDREwgZm9yIEFMU0YKZGF0ZTogMjAyMQpvdXRwdXQ6ICAgCiAgaHRtbF9ub3RlYm9vazogCiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQotLS0KCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86CgotIENyZWF0ZSBhIGBERVNlcTJgIGRhdGEgc2V0IGZyb20gYSBgU3VtbWFyaXplZEV4cGVyaW1lbnRgCi0gVHJhbnNmb3JtIFJOQS1zZXEgY291bnQgZGF0YSB3aXRoIGEgVmFyaWFuY2UgU3RhYmlsaXppbmcgVHJhbnNmb3JtYXRpb24gCi0gQ3JlYXRlIFBDQSBwbG90cyB0byBleHBsb3JlIHN0cnVjdHVyZSBhbW9uZyBSTkEtc2VxIHNhbXBsZXMKCi0tLQoKSW4gdGhpcyBub3RlYm9vaywgd2UnbGwgaW1wb3J0IHRoZSBnYXN0cmljIGNhbmNlciBkYXRhIGFuZCBkbyBzb21lIGV4cGxvcmF0b3J5CmFuYWx5c2VzIGFuZCB2aXN1YWwgaW5zcGVjdGlvbi4KV2UnbGwgdXNlIHRoZSBbYERFU2VxMmBdKGh0dHBzOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC9ERVNlcTIuaHRtbCkgcGFja2FnZSBmb3IgdGhpcy4KCiFbXShkaWFncmFtcy9ybmEtc2VxXzYucG5nKQoKYERFU2VxMmAgYWxzbyBoYXMgYW4gW2V4Y2VsbGVudCB2aWduZXR0ZV0oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sKSAKZnJvbSBMb3ZlLCBBbmRlcnMsIGFuZCBIdWJlciBmcm9tIHdoaWNoIHRoaXMgaXMgYWRhcHRlZCAoc2VlIGFsc286IFtMb3ZlLCBBbmRlcnMsIGFuZCBIdWJlci4gX0dlbm9tZSBCaW9sb2d5Xy4gMjAxNC5dKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2L3MxMzA1OS0wMTQtMDU1MC04KSkuCgojIyBMaWJyYXJpZXMgYW5kIGZ1bmN0aW9ucwoKYGBge3IgbGlicmFyeSwgbGl2ZSA9IFRSVUV9CiMgTG9hZCB0aGUgREVTZXEyIGxpYnJhcnkKbGlicmFyeShERVNlcTIpCmBgYAoKYGBge3IgbWFncml0dHJ9CiMgbWFncml0dHIgZm9yIHRoZSBwaXBlCmxpYnJhcnkobWFncml0dHIpCmBgYAoKIyMgRGlyZWN0b3JpZXMgYW5kIGZpbGVzCgpgYGB7ciBpbnB1dC1maWxlc30KIyBNYWluIGRhdGEgZGlyZWN0b3J5CmRhdGFfZGlyIDwtIGZpbGUucGF0aCgiZGF0YSIsICJnYXN0cmljLWNhbmNlciIpCgojIGRpcmVjdG9yeSB3aXRoIHRoZSB0eGltZXRhIHByb2Nlc3NlZCBkYXRhCnR4aV9kaXIgPC0gZmlsZS5wYXRoKGRhdGFfZGlyLCAidHhpIikKdHhpX2ZpbGUgPC0gZmlsZS5wYXRoKHR4aV9kaXIsICJnYXN0cmljLWNhbmNlcl90eGltZXRhLlJEUyIpCmBgYAoKYGBge3IgbWV0YWRhdGEtZmlsZX0KIyBzYW1wbGUgbWV0YWRhdGEgZmlsZQptZXRhX2ZpbGUgPC0gZmlsZS5wYXRoKGRhdGFfZGlyLCAiZ2FzdHJpYy1jYW5jZXJfbWV0YWRhdGEudHN2IikKYGBgCgpXZSdsbCBjcmVhdGUgYSBkaXJlY3RvcnkgdG8gaG9sZCBvdXIgcGxvdHMuCgpgYGB7ciBwbG90cy1kaXIsIGxpdmUgPSBUUlVFfQojIENyZWF0ZSBhIHBsb3RzIGRpcmVjdG9yeSBpZiBpdCBkb2VzIG5vdCBleGlzdCB5ZXQKcGxvdHNfZGlyIDwtIGZpbGUucGF0aCgicGxvdHMiLCAiZ2FzdHJpYy1jYW5jZXIiKQppZiAoIWRpci5leGlzdHMocGxvdHNfZGlyKSkgewogIGRpci5jcmVhdGUocGxvdHNfZGlyLCByZWN1cnNpdmUgPSBUUlVFKQp9CmBgYAoKKipPdXRwdXQqKgoKYGBge3Igb3V0cHV0LWZpbGVzLCBsaXZlID0gVFJVRX0KIyBXZSB3aWxsIHNhdmUgYSBQREYgY29weSBvZiB0aGUgUENBIHBsb3QgdG8gdGhlIHBsb3RzIGRpcmVjdG9yeQojIGFuZCBuYW1lIHRoZSBmaWxlICJnYXN0cmljLWNhbmNlcl9QQ19zY2F0dGVyLnBkZiIKcGNhX3Bsb3RfZmlsZSA8LSBmaWxlLnBhdGgocGxvdHNfZGlyLCAiZ2FzdHJpYy1jYW5jZXJfUENfc2NhdHRlci5wZGYiKQpgYGAKCiMjIERFU2VxMgoKIyMjIENyZWF0aW5nIGEgREVTZXEyIGRhdGFzZXQgZnJvbSBhIHR4aW1ldGEgb2JqZWN0CgpGaXJzdCwgbGV0J3MgcmVhZCBpbiB0aGUgZGF0YSB3ZSBwcm9jZXNzZWQgd2l0aCBgdHhpbWV0YWAuCgpgYGB7ciByZWFkLXJkcywgbGl2ZSA9IFRSVUV9CiMgUmVhZCBpbiB0aGUgUkRTIGZpbGUgd2UgY3JlYXRlZCBpbiB0aGUgbGFzdCBub3RlYm9vawpnZW5lX3N1bW1hcml6ZWQgPC0gcmVhZHI6OnJlYWRfcmRzKHR4aV9maWxlKQpgYGAKCiMjIyBTZXQgdXAgREVTZXEyIG9iamVjdAoKV2UgdXNlIHRoZSB0aXNzdWUgb2Ygb3JpZ2luIGluIHRoZSBkZXNpZ24gZm9ybXVsYSBiZWNhdXNlIHRoYXQgd2lsbCBhbGxvdyB1cyB0byBtb2RlbCB0aGlzIHZhcmlhYmxlIG9mIGludGVyZXN0LgoKYGBge3IgZGRzZXR9CmRkc2V0IDwtIERFU2VxRGF0YVNldChnZW5lX3N1bW1hcml6ZWQsCiAgICAgICAgICAgICAgICAgICAgICBkZXNpZ24gPSB+IHRpc3N1ZSkKYGBgCgojIyMgVmFyaWFuY2Ugc3RhYmlsaXppbmcgdHJhbnNmb3JtYXRpb24KClJhdyBjb3VudCBkYXRhIGlzIG5vdCB1c3VhbGx5IHN1aXRhYmxlIGZvciB0aGUgYWxnb3JpdGhtcyB3ZSB1c2UgZm9yIGRpbWVuc2lvbmFsaXR5IHJlZHVjdGlvbiwgY2x1c3RlcmluZywgb3IgaGVhdG1hcHMuIApUbyBpbXByb3ZlIHRoaXMsIHdlIHdpbGwgdHJhbnNmb3JtIHRoZSBjb3VudCBkYXRhIHRvIGNyZWF0ZSBhbiBleHByZXNzaW9uIG1lYXN1cmUgdGhhdCBpcyBiZXR0ZXIgc3VpdGVkIGZvciB0aGVzZSBhbmFseXNlcy4gClRoZSBjb3JlIHRyYW5zZm9ybWF0aW9uIHdpbGwgbWFwIHRoZSBleHByZXNzaW9uIHRvIGEgbG9nMiBzY2FsZSwgd2hpbGUgYWNjb3VudGluZyBmb3Igc29tZSBvZiB0aGUgZXhwZWN0ZWQgdmFyaWF0aW9uIGFtb25nIHNhbXBsZXMgYW5kIGdlbmVzLgoKU2luY2UgZGlmZmVyZW50IHNhbXBsZXMgYXJlIHVzdWFsbHkgc2VxdWVuY2VkIHRvIGRpZmZlcmVudCBkZXB0aHMsIHdlIHdhbnQgdG8gdHJhbnNmb3JtIG91ciBSTkEtc2VxIGNvdW50IGRhdGEgdG8gbWFrZSBkaWZmZXJlbnQgc2FtcGxlcyBtb3JlIGRpcmVjdGx5IGNvbXBhcmFibGUuIApXZSBhbHNvIHdhbnQgdG8gZGVhbCB3aXRoIHRoZSBmYWN0IHRoYXQgZ2VuZXMgd2l0aCBsb3cgY291bnRzIGFyZSBhbHNvIGxpa2VseSB0byBoYXZlIGhpZ2hlciB2YXJpYW5jZSAob24gdGhlIGxvZzIgc2NhbGUpLCBhcyB0aGF0IGNvdWxkIGJpYXMgb3VyIGNsdXN0ZXJpbmcuClRvIGhhbmRsZSBib3RoIG9mIHRoZXNlIGNvbnNpZGVyYXRpb25zLCB3ZSBjYW4gY2FsY3VsYXRlIGEgVmFyaWFuY2UgU3RhYmlsaXppbmcgVHJhbnNmb3JtYXRpb24gb2YgdGhlIGNvdW50IGRhdGEsIGFuZCB3b3JrIHdpdGggdGhhdCB0cmFuc2Zvcm1lZCBkYXRhIGZvciBvdXIgYW5hbHlzaXMuCgpTZWUgW3RoaXMgc2VjdGlvbiBvZiB0aGUgYERFU2VxMmAgdmlnbmV0dGVdKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNkYXRhLXRyYW5zZm9ybWF0aW9ucy1hbmQtdmlzdWFsaXphdGlvbikgZm9yIG1vcmUgb24gdGhpcyB0b3BpYy4gCgpgYGB7ciB2c3R9CnZzdF9kYXRhIDwtIHZzdChkZHNldCkKYGBgCgojIyMgUHJpbmNpcGFsIGNvbXBvbmVudCBhbmFseXNpcwoKUHJpbmNpcGFsIGNvbXBvbmVudCBhbmFseXNpcyAoUENBKSBpcyBhIGRpbWVuc2lvbmFsaXR5IHJlZHVjdGlvbiB0ZWNobmlxdWUgdGhhdCBhbGxvd3MgdXMgdG8gaWRlbnRpZnkgdGhlIGxhcmdlc3QgY29tcG9uZW50cyBvZiB2YXJpYXRpb24gaW4gYSBjb21wbGV4IGRhdGFzZXQuCk91ciBleHByZXNzaW9uIGRhdGEgY2FuIGJlIHRob3VnaHQgb2YgYXMgbWFwcGluZyBlYWNoIHNhbXBsZSBpbiBhIG11bHRpZGltZW5zaW9uYWwgc3BhY2UgZGVmaW5lZCBieSB0aGUgZXhwcmVzc2lvbiBsZXZlbCBvZiBlYWNoIGdlbmUuClRoZSBleHByZXNzaW9uIG9mIG1hbnkgb2YgdGhvc2UgZ2VuZXMgYXJlIGNvcnJlbGF0ZWQsIHNvIHdlIGNhbiBvZnRlbiBnZXQgYSBiZXR0ZXIsIHNpbXBsZXIgcGljdHVyZSBvZiB0aGUgZGF0YSBieSBjb21iaW5pbmcgdGhlIGluZm9ybWF0aW9uIGZyb20gdGhvc2UgY29ycmVsYXRlZCBnZW5lcy4KClBDQSByb3RhdGVzIGFuZCB0cmFuc2Zvcm1zIHRoaXMgc3BhY2Ugc28gdGhhdCBlYWNoIGF4aXMgaXMgbm93IGEgY29tYmluYXRpb24gb2YgbXVsdGlwbGUgY29ycmVsYXRlZCBnZW5lcywgb3JkZXJlZCBzbyB0aGUgZmlyc3QgYXhlcyBjYXB0dXJlIHRoZSBtb3N0IHZhcmlhdGlvbiBmcm9tIHRoZSBkYXRhLiAKVGhlc2UgbmV3IGF4ZXMgYXJlIHRoZSAicHJpbmNpcGFsIGNvbXBvbmVudHMuIgpJZiB3ZSBsb29rIGF0IHRoZSBmaXJzdCBmZXcgY29tcG9uZW50cywgd2UgY2FuIG9mdGVuIGdldCBhIG5pY2Ugb3ZlcnZpZXcgb2YgcmVsYXRpb25zaGlwcyBhbW9uZyB0aGUgc2FtcGxlcyBpbiB0aGUgZGF0YS4KClRoZSBgcGxvdFBDQSgpYCBmdW5jdGlvbiB3ZSB3aWxsIHVzZSBmcm9tIHRoZSBgREVTZXEyYCBwYWNrYWdlIGNhbGN1bGF0ZXMgYW5kIHBsb3RzIHRoZSBmaXJzdCB0d28gcHJpbmNpcGFsIGNvbXBvbmVudHMgKFBDMSBhbmQgUEMyKS4KVmlzdWFsaXppbmcgUEMxIGFuZCBQQzIgY2FuIGdpdmUgdXMgaW5zaWdodCBpbnRvIGhvdyBkaWZmZXJlbnQgdmFyaWFibGVzIChlLmcuLCB0aXNzdWUgc291cmNlKSBhZmZlY3Qgb3VyIGRhdGFzZXQgYW5kIGhlbHAgdXMgc3BvdCBhbnkgdGVjaG5pY2FsIGVmZmVjdHMgKG1vcmUgb24gdGhhdCBiZWxvdykuCgoKYGBge3IgcGxvdFBDQSwgbGl2ZSA9IFRSVUV9CiMgREVTZXEyIGJ1aWx0IGluIGZ1bmN0aW9uIGlzIGNhbGxlZCBwbG90UENBIGFuZCB3ZSB3YW50IHRvIGNvbG9yIHBvaW50cyBieQojIHRpc3N1ZQpwbG90UENBKHZzdF9kYXRhLCBpbnRncm91cCA9ICJ0aXNzdWUiKQpgYGAKClNhdmUgdGhlIG1vc3QgcmVjZW50IHBsb3QgdG8gZmlsZSB3aXRoIGBnZ3NhdmVgIGZyb20gYGdncGxvdDJgCgpgYGB7ciBzYXZlLXBkZn0KIyBTYXZlIHRoZSBQREYgZmlsZSAKZ2dwbG90Mjo6Z2dzYXZlKHBjYV9wbG90X2ZpbGUsIHBsb3QgPSBnZ3Bsb3QyOjpsYXN0X3Bsb3QoKSkKYGBgCgojIyBBIG5vdGUgb24gdGVjaG5pY2FsIGVmZmVjdHMKCldlIGRvbid0IGhhdmUgYmF0Y2ggaW5mb3JtYXRpb24gKGkuZS4sIHdoZW4gdGhlIHNhbXBsZXMgd2VyZSBydW4pIGZvciB0aGlzIHBhcnRpY3VsYXIgZXhwZXJpbWVudCwgYnV0IGxldCdzIGltYWdpbmUgdGhhdCBgU1JSNTg1NTc0YCBhbmQgYFNSUjU4NTU3NmAgd2VyZSBydW4gc2VwYXJhdGVseSBmcm9tIGFsbCBvdGhlciBzYW1wbGVzLgpXZSdsbCBhZGQgdGhpcyBhcyBhIG5ldyAidG95IiBjb2x1bW4gaW4gdGhlIHNhbXBsZSBkYXRhIChgY29sRGF0YWApLgoKYGBge3IgZXh0cmFjdC1zYW1wbGUsIGxpdmUgPSBUUlVFfQojIEV4dHJhY3QgY29sRGF0YQpzYW1wbGVfaW5mbyA8LSBjb2xEYXRhKHZzdF9kYXRhKQoKIyBQcmludCBvdXQgcHJldmlldwpzYW1wbGVfaW5mbwpgYGAKCk5vdyB3ZSBjYW4gYWRkIGEgbmV3IGNvbHVtbiB3aXRoIHRveSBiYXRjaCBpbmZvcm1hdGlvbiBhbmQgcmUtc3RvcmUgdGhlIGBjb2xEYXRhKClgLiAKCmBgYHtyIGFkZC1iYXRjaH0KIyBBZGQgYmF0Y2ggaW5mb3JtYXRpb24Kc2FtcGxlX2luZm8kYmF0Y2ggPC0gYygiYmF0Y2gxIiwgImJhdGNoMSIsICJiYXRjaDEiLCAiYmF0Y2gxIiwgImJhdGNoMiIsIAogICAgICAgICAgICAgICAgICAgICAgICJiYXRjaDEiLCAiYmF0Y2gyIiwgImJhdGNoMSIpCmBgYAogICAgICAgICAgICAgICAgICAgICAgICAgICAgCklmIHRoaXMgYmF0Y2ggaW5mb3JtYXRpb24gd2VyZSByZWFsIHdlIHdvdWxkIGhhdmUgaW5jbHVkZWQgaXQgd2l0aCB0aGUgc2FtcGxlIG1ldGFkYXRhIHdoZW4gd2UgbWFkZSB0aGUgb3JpZ2luYWwgYFN1bW1hcml6ZWRFeHBlcmltZW50YCBvYmplY3Qgd2l0aCBgdHhpbWV0YWAuCldlIHdvdWxkIHRoZW4gaW5jbHVkZSBpdCBpbiB0aGUgbW9kZWwgc3RvcmVkIGluIG91ciBERVNlcTIgb2JqZWN0IHVzaW5nIHRoZSBgZGVzaWduYCBhcmd1bWVudCAoYGRlc2lnbiA9IH4gdGlzc3VlICsgYmF0Y2hgKSBhbmQgd2Ugd291bGQgcmUtcnVuIHRoZSBgREVTZXFEYXRhU2V0KClgIGFuZCBgdnN0KClgIHN0ZXBzIHdlIGRpZCBhYm92ZS4gCkhlcmUgd2Ugd2lsbCB0YWtlIGEgYml0IG9mIGEgc2hvcnRjdXQgYW5kIGFkZCBpdCBkaXJlY3RseSB0byB0aGUgYGNvbERhdGEoKWAgZm9yIG91ciBgdnN0KClgLXRyYW5zZm9ybWVkIGRhdGEuCgpgYGB7ciBjb2xkYXRhLXZzdCwgbGl2ZSA9IFRSVUV9CiMgQWRkIGNvbGRhdGEoKSB3aXRoIGJhdGNoIGluZm8gdG8gdnN0X2RhdGEKY29sRGF0YSh2c3RfZGF0YSkgPC0gc2FtcGxlX2luZm8KYGBgCgpgYGB7ciBwbG90UENBLTIsIGxpdmUgPSBUUlVFfQojIFBDQSBwbG90IC0gdGlzc3VlICphbmQqIGJhdGNoCiMgV2Ugd2FudCBwbG90UENBIHRvIHJldHVybiB0aGUgZGF0YSBzbyB3ZSBjYW4gaGF2ZSBtb3JlIGNvbnRyb2wgYWJvdXQgdGhlIHBsb3QKcGNhX2RhdGEgPC0gcGxvdFBDQSh2c3RfZGF0YSwgCiAgICAgICAgICAgICAgICAgICAgaW50Z3JvdXAgPSBjKCJ0aXNzdWUiLCAiYmF0Y2giKSwgCiAgICAgICAgICAgICAgICAgICAgcmV0dXJuRGF0YSA9IFRSVUUpCmBgYAoKYGBge3IgcGVyY2VudF92YXJ9CiMgSGVyZSB3ZSBhcmUgc2V0dGluZyB1cCB0aGUgcGVyY2VudCB2YXJpYW5jZSB0aGF0IHdlIGFyZSBleHRyYWN0aW5nIGZyb20gdGhlIGBwY2FfZGF0YWAgb2JqZWN0CnBlcmNlbnRfdmFyIDwtIHJvdW5kKDEwMCAqIGF0dHIocGNhX2RhdGEsICJwZXJjZW50VmFyIikpCmBgYAoKTGV0J3MgdXNlIGdncGxvdCB0byB2aXN1YWxpemUgdGhlIGZpcnN0IHR3byBwcmluY2lwYWwgY29tcG9uZW50cy4gCgpgYGB7ciBjb2xvci1ieS1iYXRjaCwgbGl2ZSA9IFRSVUV9CiMgQ29sb3IgcG9pbnRzIGJ5ICJiYXRjaCIgYW5kIHVzZSBzaGFwZSB0byBpbmRpY2F0ZSB0aGUgdGlzc3VlIG9mIG9yaWdpbgpnZ3Bsb3QyOjpnZ3Bsb3QocGNhX2RhdGEsIGdncGxvdDI6OmFlcyhQQzEsIFBDMiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbG9yID0gYmF0Y2gsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzaGFwZSA9IHRpc3N1ZSkpICsKICBnZ3Bsb3QyOjpnZW9tX3BvaW50KHNpemUgPSAzKSArCiAgZ2dwbG90Mjo6eGxhYihwYXN0ZTAoIlBDMTogIiwgcGVyY2VudF92YXJbMV0sIiUgdmFyaWFuY2UiKSkgKwogIGdncGxvdDI6OnlsYWIocGFzdGUwKCJQQzI6ICIsIHBlcmNlbnRfdmFyWzJdLCIlIHZhcmlhbmNlIikpICsgCiAgZ2dwbG90Mjo6Y29vcmRfZml4ZWQoKQpgYGAKCiMjIFNlc3Npb24gSW5mbwoKUmVjb3JkIHNlc3Npb24gaW5mbyBmb3IgcmVwcm9kdWNpYmlsaXR5ICYgcHJvdmVuYW5jZSBwdXJwb3Nlcy4KCmBgYHtyIHNlc3Npb25pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/RNA-seq/04-nb_cell_line_tximeta.nb.html b/RNA-seq/04-nb_cell_line_tximeta.nb.html new file mode 100644 index 00000000..0d7ec20c --- /dev/null +++ b/RNA-seq/04-nb_cell_line_tximeta.nb.html @@ -0,0 +1,424 @@ + + + + + + + + + + + + + + + +Processing a Neuroblastoma cell line data set + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +

In this section, we’ll be working with RNA-seq data from neuroblastoma (NB) cell lines from Harenza, et al. (2017)

+

The course directors have already processed the raw data using salmon quant and the quant.sf files for each sample can be found in data/NB-cell/salmon_quant/<SAMPLE>.

+

+

In the gastric cancer example, we imported Salmon-processed data with tximeta to then use with DESeq2. We will also use DESeq2 for these analyses, specifically for differential expression analysis.

+

In order to prepare the NB cell line data for differential expression analysis, we will modify the gastric cancer tximeta notebook (02-gastric_cancer_tximeta-live.Rmd) and save this new notebook as nb_cell_tximeta.Rmd:

+
    +

  • To create a new notebook, select File > New File > R Notebook. The new notebook should appear in your Source Pane in RStudio. Save the new notebook, using Ctrl+S (Cmd+S on Mac) or File > Save, in the training-modules/RNA-seq directory with the name nb_cell_line_tximeta.Rmd. You can add a new chunk by clicking the Insert Chunk button on the toolbar or by pressing Cmd+Option+I.

    • +

  • Alter the code from 02-gastric_cancer_tximeta-live.Rmd to use the NB cell line data. The quant.sf files for each sample can be found in data/NB-cell/salmon_quant/<SAMPLE>.

    • +

  • Save the tximeta output as data/NB-cell/txi/NB-cell_tximeta.RDS. Note that data/NB-cell/txi/ is a new directory.

    • +
+ + +
LS0tCnRpdGxlOiAiUHJvY2Vzc2luZyBhIE5ldXJvYmxhc3RvbWEgY2VsbCBsaW5lIGRhdGEgc2V0IgphdXRob3I6IENDREwgZm9yIEFMU0YKZGF0ZTogMjAyMQpvdXRwdXQ6ICAgCiAgaHRtbF9ub3RlYm9vazogCiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQotLS0KCkluIHRoaXMgc2VjdGlvbiwgd2UnbGwgYmUgd29ya2luZyB3aXRoIFJOQS1zZXEgZGF0YSBmcm9tIG5ldXJvYmxhc3RvbWEgKE5CKSBjZWxsIGxpbmVzIGZyb20KW0hhcmVuemEsIF9ldCBhbC5fICgyMDE3KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjEwMzgvc2RhdGEuMjAxNy4zMykKClRoZSBjb3Vyc2UgZGlyZWN0b3JzIGhhdmUgYWxyZWFkeSBwcm9jZXNzZWQgdGhlIHJhdyBkYXRhIHVzaW5nIGBzYWxtb24gcXVhbnRgIGFuZCB0aGUgYHF1YW50LnNmYCBmaWxlcyBmb3IgZWFjaCBzYW1wbGUgY2FuIGJlIGZvdW5kIGluIGBkYXRhL05CLWNlbGwvc2FsbW9uX3F1YW50LzxTQU1QTEU+YC4KCiFbXShkaWFncmFtcy9ybmEtc2VxXzUucG5nKQoKSW4gdGhlIGdhc3RyaWMgY2FuY2VyIGV4YW1wbGUsIHdlIGltcG9ydGVkIFNhbG1vbi1wcm9jZXNzZWQgZGF0YSB3aXRoIGB0eGltZXRhYCB0byB0aGVuIHVzZSB3aXRoIGBERVNlcTJgLgpXZSB3aWxsIGFsc28gdXNlIGBERVNlcTJgIGZvciB0aGVzZSBhbmFseXNlcywgc3BlY2lmaWNhbGx5IGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcy4KCkluIG9yZGVyIHRvIHByZXBhcmUgdGhlIE5CIGNlbGwgbGluZSBkYXRhIGZvciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcywgd2Ugd2lsbCBtb2RpZnkgdGhlIGdhc3RyaWMgY2FuY2VyIHR4aW1ldGEgbm90ZWJvb2sgKGAwMi1nYXN0cmljX2NhbmNlcl90eGltZXRhLWxpdmUuUm1kYCkgYW5kIHNhdmUgdGhpcyBuZXcgbm90ZWJvb2sgYXMgYG5iX2NlbGxfdHhpbWV0YS5SbWRgOgoKKiBUbyBjcmVhdGUgYSBuZXcgbm90ZWJvb2ssIHNlbGVjdCBgRmlsZWAgPiBgTmV3IEZpbGVgID4gYFIgTm90ZWJvb2tgLgpUaGUgbmV3IG5vdGVib29rIHNob3VsZCBhcHBlYXIgaW4geW91ciBTb3VyY2UgUGFuZSBpbiBSU3R1ZGlvLgpTYXZlIHRoZSBuZXcgbm90ZWJvb2ssIHVzaW5nIEN0cmwrUyAoQ21kK1Mgb24gTWFjKSBvciBgRmlsZWAgPiBgU2F2ZWAsIGluIHRoZSBgdHJhaW5pbmctbW9kdWxlcy9STkEtc2VxYCBkaXJlY3Rvcnkgd2l0aCB0aGUgbmFtZSBgbmJfY2VsbF9saW5lX3R4aW1ldGEuUm1kYC4KWW91IGNhbiBhZGQgYSBuZXcgY2h1bmsgYnkgY2xpY2tpbmcgdGhlICpJbnNlcnQgQ2h1bmsqIGJ1dHRvbiBvbiB0aGUgdG9vbGJhciBvciBieSBwcmVzc2luZyAqQ21kK09wdGlvbitJKi4KCiogQWx0ZXIgdGhlIGNvZGUgZnJvbSBgMDItZ2FzdHJpY19jYW5jZXJfdHhpbWV0YS1saXZlLlJtZGAgdG8gdXNlIHRoZSBOQiBjZWxsIGxpbmUgZGF0YS4KVGhlIGBxdWFudC5zZmAgZmlsZXMgZm9yIGVhY2ggc2FtcGxlIGNhbiBiZSBmb3VuZCBpbiBgZGF0YS9OQi1jZWxsL3NhbG1vbl9xdWFudC88U0FNUExFPmAuCgoqIFNhdmUgdGhlIGB0eGltZXRhYCBvdXRwdXQgYXMgYGRhdGEvTkItY2VsbC90eGkvTkItY2VsbF90eGltZXRhLlJEU2AuIE5vdGUgdGhhdCBgZGF0YS9OQi1jZWxsL3R4aS9gIGlzIGEgbmV3IGRpcmVjdG9yeS4K
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/RNA-seq/05-nb_cell_line_DESeq2-live.Rmd b/RNA-seq/05-nb_cell_line_DESeq2-live.Rmd index 8f34822b..40def28a 100644 --- a/RNA-seq/05-nb_cell_line_DESeq2-live.Rmd +++ b/RNA-seq/05-nb_cell_line_DESeq2-live.Rmd @@ -1,12 +1,23 @@ --- title: "Neuroblastoma Cell Line: Differential expression analysis with DESeq2" +author: CCDL for ALSF +date: 2021 output: html_notebook: toc: true toc_float: true --- -**CCDL 2021** + +## Objectives + +This notebook will demonstrate how to: + +- Perform differential expression analysis with `DESeq2` +- Apply a shrinkage algorithm to improve estimates of expression changes +- Draw a volcano plot with the `EnhancedVolcano` package + +--- In this notebook, we'll perform an analysis to identify the genes that are differentially expressed in _MYCN_ amplified vs. nonamplified neuroblastoma cell lines. @@ -135,9 +146,9 @@ gene_summarized$status <- relevel(gene_summarized$status, ref = "Nonamplified") ``` -### Differential expression +## Differential expression analysis -#### Filtering low-expressed genes +### Filtering low-expressed genes Genes that have very low counts are not likely to yield reliable differential expression results, so we will do some light [pre-filtering](http://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#pre-filtering). We will keep only genes with total counts of at least 10 across all samples. @@ -148,7 +159,7 @@ ddset <- ddset[genes_to_keep, ] ``` -#### Differential expression analysis +### The `DESeq()` function We'll now use the wrapper function `DESeq()` to perform our differential expression analysis. As mentioned earlier, this performs a number of steps, including an [outlier removal procedure](https://bioconductor.org/packages/release/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#approach-to-count-outliers). @@ -179,7 +190,7 @@ summary(deseq_results, alpha = 0.05) ``` -#### Shrinking log2 fold change estimates +### Shrinking log2 fold change estimates The estimates of log2 fold change calculated by `DESeq()` are not corrected for expression level. This means that when counts are small, we are likely to end up with some large fold change values that overestimate the true extent of the change between conditions. @@ -262,7 +273,7 @@ readr::write_tsv(deseq_df, file = deseq_df_file) ``` -#### Volcano Plot +## Making a Volcano Plot With these shrunken effect sizes, we will draw a volcano plot, using the [`EnhancedVolcano` package](https://github.com/kevinblighe/EnhancedVolcano) to make it a bit easier. This package automatically color codes the points by cutoffs for both significance and fold change and labels many of the significant genes (subject to spacing). diff --git a/RNA-seq/05-nb_cell_line_DESeq2.nb.html b/RNA-seq/05-nb_cell_line_DESeq2.nb.html new file mode 100644 index 00000000..c36d5826 --- /dev/null +++ b/RNA-seq/05-nb_cell_line_DESeq2.nb.html @@ -0,0 +1,1159 @@ + + + + + + + + + + + + + + + +Neuroblastoma Cell Line: Differential expression analysis with DESeq2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Perform differential expression analysis with DESeq2
    • +
  • Apply a shrinkage algorithm to improve estimates of expression changes
    • +
  • Draw a volcano plot with the EnhancedVolcano package
    • +
+
+

In this notebook, we’ll perform an analysis to identify the genes that are differentially expressed in MYCN amplified vs. nonamplified neuroblastoma cell lines.

+

These RNA-seq data are from Harenza, et al. (2017).

+

More information about DESeq2 can be found in the excellent vignette from Love, Anders, and Huber from which this is adapted (see also: Love, et al. (2014)).

+

DESeq2 takes unnormalized counts or estimated counts and does the following:

+ +

+
+
+

Libraries and functions

+ + + +
# magrittr pipe
+library(magrittr)
+
+# Load the DESeq2 library
+library(DESeq2)
+ + +
Loading required package: S4Vectors
+ + +
Loading required package: stats4
+ + +
Loading required package: BiocGenerics
+ + +
Loading required package: parallel
+ + +

+Attaching package: 'BiocGenerics'
+ + +
The following objects are masked from 'package:parallel':
+
+    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
+    clusterExport, clusterMap, parApply, parCapply, parLapply,
+    parLapplyLB, parRapply, parSapply, parSapplyLB
+ + +
The following objects are masked from 'package:stats':
+
+    IQR, mad, sd, var, xtabs
+ + +
The following objects are masked from 'package:base':
+
+    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
+    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
+    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
+    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
+    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
+    union, unique, unsplit, which.max, which.min
+ + +

+Attaching package: 'S4Vectors'
+ + +
The following object is masked from 'package:base':
+
+    expand.grid
+ + +
Loading required package: IRanges
+ + +
Loading required package: GenomicRanges
+ + +
Loading required package: GenomeInfoDb
+ + +
Loading required package: SummarizedExperiment
+ + +
Loading required package: MatrixGenerics
+ + +
Loading required package: matrixStats
+ + +

+Attaching package: 'MatrixGenerics'
+ + +
The following objects are masked from 'package:matrixStats':
+
+    colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
+    colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
+    colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
+    colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
+    colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
+    colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
+    colWeightedMeans, colWeightedMedians, colWeightedSds,
+    colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
+    rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
+    rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
+    rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
+    rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
+    rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
+    rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
+    rowWeightedSds, rowWeightedVars
+ + +
Loading required package: Biobase
+ + +
Welcome to Bioconductor
+
+    Vignettes contain introductory material; view with
+    'browseVignettes()'. To cite Bioconductor, see
+    'citation("Biobase")', and for packages 'citation("pkgname")'.
+ + +

+Attaching package: 'Biobase'
+ + +
The following object is masked from 'package:MatrixGenerics':
+
+    rowMedians
+ + +
The following objects are masked from 'package:matrixStats':
+
+    anyMissing, rowMedians
+ + +
# We will be making fancy volcano plots
+library(EnhancedVolcano)
+ + +
Loading required package: ggplot2
+ + +
Loading required package: ggrepel
+ + +
Registered S3 methods overwritten by 'ggalt':
+  method                  from   
+  grid.draw.absoluteGrob  ggplot2
+  grobHeight.absoluteGrob ggplot2
+  grobWidth.absoluteGrob  ggplot2
+  grobX.absoluteGrob      ggplot2
+  grobY.absoluteGrob      ggplot2
+ + + +
+
+

Directories and files

+

Input

+ + + +
# directory with the tximeta processed data
+txi_dir <- file.path("data", "NB-cell", "txi")
+txi_file <- file.path(txi_dir, "NB-cell_tximeta.RDS")
+ + + +

Output

+

We’ll create a results directory to hold our results.

+ + + +
# Create a results directory if it doesn't already exist
+results_dir <- file.path("results", "NB-cell")
+if (!dir.exists(results_dir)) {
+  dir.create(results_dir, recursive = TRUE)
+}
+ + + +

We will also need a directory to store our plots.

+ + + +
# Create a plots directory if it doesn't already exist
+plots_dir <- file.path("plots", "NB-cell")
+if (!dir.exists(plots_dir)) {
+  dir.create(plots_dir, recursive = TRUE)
+}
+ + + + + + +
# RDS for the output of DESeq analysis
+deseq_file <- file.path(results_dir, 
+                        "NB-cell_DESeq_amplified_v_nonamplified.RDS")
+
+# DESeq2 results table
+deseq_df_file <- file.path(results_dir,
+                           "NB-cell_DESeq_amplified_v_nonamplified_results.tsv")
+
+# PNG of the volcano plot
+volcano_file <- file.path(plots_dir, "NB-cell_volcano.png")
+ + + +
+
+

DESeq2

+
+

Creating a DESeq2 dataset from tximeta object

+

First, let’s read in the data we processed with tximeta.

+
+

Preparation

+ + + +
# Read in the RDS file we created in the last notebook
+gene_summarized <- readr::read_rds(txi_file)
+ + + +

We’re most interested in MYCN amplification, which we had stored in the status column of the sample metadata of gene_summarized. While the sample metadata is stored internally in the colData slot, the SummarizedExperiment object makes it easy for us to access it as if it were just a column of a data frame, using the familiar $ syntax.

+ + + +
gene_summarized$status
+ + +
 [1] "Amplified"    "Amplified"    "Amplified"    "Amplified"    "Amplified"   
+ [6] "Amplified"    "Amplified"    "Amplified"    "Nonamplified" "Nonamplified"
+[11] "Amplified"    "Amplified"    "Amplified"    "Nonamplified" "Amplified"   
+[16] "Amplified"    "Amplified"    "Amplified"    "Nonamplified" "Amplified"   
+[21] "Amplified"    "Amplified"    "Nonamplified" "Amplified"    "Nonamplified"
+[26] "Amplified"    "Amplified"    "Amplified"    "Nonamplified" "Nonamplified"
+[31] "Nonamplified" "Amplified"    "Amplified"    "Amplified"    "Nonamplified"
+[36] "Nonamplified" "Amplified"    "Amplified"    "Nonamplified"
+ + +
Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Amplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Nonamplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+
+Nonamplified
+
+Amplified
+
+Amplified
+
+Nonamplified
+ + + +

This is stored as a character type, but to give a bit more information to DESeq, we will convert this to a factor.

+ + + +
gene_summarized$status <- as.factor(gene_summarized$status)
+ + + +

We’ll want to use the “Nonamplified” samples as our reference. Let’s look at the levels of status.

+ + + +
levels(gene_summarized$status)
+ + +
[1] "Amplified"    "Nonamplified"
+ + +
Amplified
+
+Nonamplified
+ + + +

We can see that these are in alphabetical order, so “Amplified” samples would be the reference. We can use the relevel() function to remedy this.

+ + + +
gene_summarized$status <- relevel(gene_summarized$status, ref = "Nonamplified")
+ + + + + + +
# Check what the levels are now
+levels(gene_summarized$status)
+ + +
[1] "Nonamplified" "Amplified"
+ + +
Nonamplified
+
+Amplified
+ + + +
+
+

DESeq Dataset creation

+ + + +
# Create a DESeq2 dataset from `gene_summarized` 
+# remember that `status` is the variable of interest here
+ddset <- DESeqDataSet(gene_summarized,
+                      design = ~ status)
+ + +
using counts and average transcript lengths from tximeta
+ + + +
+
+
+
+

Differential expression analysis

+
+

Filtering low-expressed genes

+

Genes that have very low counts are not likely to yield reliable differential expression results, so we will do some light pre-filtering. We will keep only genes with total counts of at least 10 across all samples.

+ + + +
genes_to_keep <- rowSums(counts(ddset)) >= 10
+ddset <- ddset[genes_to_keep, ]
+ + + +
+
+

The DESeq() function

+

We’ll now use the wrapper function DESeq() to perform our differential expression analysis. As mentioned earlier, this performs a number of steps, including an outlier removal procedure. For this particular dataset, there is a pretty large number of outliers, which can be a bit of a red flag, but we will proceed for now.

+ + + +
deseq_object <- DESeq(ddset)
+ + +
estimating size factors
+ + +
using 'avgTxLength' from assays(dds), correcting for library size
+ + +
estimating dispersions
+ + +
gene-wise dispersion estimates
+ + +
mean-dispersion relationship
+ + +
final dispersion estimates
+ + +
fitting model and testing
+ + +
-- replacing outliers and refitting for 2895 genes
+-- DESeq argument 'minReplicatesForReplace' = 7 
+-- original counts are preserved in counts(dds)
+ + +
estimating dispersions
+ + +
fitting model and testing
+ + + +

Let’s save this to our results file.

+ + + +
# Save the results as an RDS
+readr::write_rds(deseq_object, file = deseq_file)
+ + + +

Now we will have a look at the results table.

+ + + +
deseq_results <- results(deseq_object)
+deseq_results
+ + +
log2 fold change (MLE): status Amplified vs Nonamplified 
+Wald test p-value: status Amplified vs Nonamplified 
+DataFrame with 24912 rows and 6 columns
+                   baseMean log2FoldChange     lfcSE      stat    pvalue
+                  <numeric>      <numeric> <numeric> <numeric> <numeric>
+ENSG00000000003 1148.278399       0.921536  0.424309  2.171849  0.029867
+ENSG00000000005    0.627406       1.672285  2.247996  0.743900  0.456937
+ENSG00000000419 1680.109464      -0.176649  0.215485 -0.819775  0.412344
+ENSG00000000457  962.907631      -0.257752  0.166387 -1.549110  0.121355
+ENSG00000000460 1595.937423      -0.133821  0.197230 -0.678504  0.497452
+...                     ...            ...       ...       ...       ...
+ENSG00000285976  1874.02776      0.0285397  0.183730  0.155335  0.876557
+ENSG00000285978     1.40743     -1.1452465  0.874165 -1.310103  0.190161
+ENSG00000285982    90.93868      0.1131803  0.493040  0.229556  0.818437
+ENSG00000285990    13.77859      0.3673226  0.456293  0.805015  0.420811
+ENSG00000285991    17.07491      0.0709553  0.333191  0.212957  0.831361
+                     padj
+                <numeric>
+ENSG00000000003  0.133479
+ENSG00000000005        NA
+ENSG00000000419  0.656626
+ENSG00000000457  0.326981
+ENSG00000000460  0.721065
+...                   ...
+ENSG00000285976  0.946696
+ENSG00000285978  0.427379
+ENSG00000285982  0.918545
+ENSG00000285990  0.662821
+ENSG00000285991  0.926078
+ + + +

How many genes were differentially expressed (FDR < 0.05)?

+ + + +
summary(deseq_results, alpha = 0.05)
+ + +

+out of 24799 with nonzero total read count
+adjusted p-value < 0.05
+LFC > 0 (up)       : 1071, 4.3%
+LFC < 0 (down)     : 1798, 7.3%
+outliers [1]       : 0, 0%
+low counts [2]     : 3478, 14%
+(mean count < 1)
+[1] see 'cooksCutoff' argument of ?results
+[2] see 'independentFiltering' argument of ?results
+ + + +
+
+

Shrinking log2 fold change estimates

+

The estimates of log2 fold change calculated by DESeq() are not corrected for expression level. This means that when counts are small, we are likely to end up with some large fold change values that overestimate the true extent of the change between conditions.

+

We can correct this by applying a “shrinkage” procedure, which will adjust large values with small counts downward, while preserving values with larger counts, which are likely to be more accurate.

+

To do this, we will use the lfcShrink() function, but first we need to know the name and/or position of the “coefficient” that was calculated by DESeq(), which we can do with the resultsNames() function

+ + + +
# get the deseq coefficient names:
+resultsNames(deseq_object)
+ + +
[1] "Intercept"                        "status_Amplified_vs_Nonamplified"
+ + +
Intercept
+
+status_Amplified_vs_Nonamplified
+ + + +

We are interested in the status coefficient, which is in position 2.

+

There are a few options for the shrinkage estimation. The default is apeglm (Zhu et al. 2018), but we have found that this can be sensitive to extreme outliers, which are definitely a factor in this data set. So for this data set we will be using ashr (Stephens 2017)

+ + + +
# calculate shrunken log2 fold change estimates
+deseq_shrunken <- lfcShrink(deseq_object,
+                            # the coefficient we want to reestimate
+                            coef = 2, 
+                            # We will use `ashr` for estimation
+                            type = "ashr"
+                            )
+ + +
using 'ashr' for LFC shrinkage. If used in published research, please cite:
+    Stephens, M. (2016) False discovery rates: a new deal. Biostatistics, 18:2.
+    https://doi.org/10.1093/biostatistics/kxw041
+ + + +

Let’s compare the log2 fold change estimates from the two results tables by creating a plot.

+

First we will combine the results into a new data frame.

+ + + +
comparison_df <- data.frame(
+  lfc_original = deseq_results$log2FoldChange,
+  lfc_shrunken = deseq_shrunken$log2FoldChange,
+  logmean = log10(deseq_results$baseMean)
+  )
+ + + +

Now we can plot the original and shrunken log2 fold change values to see what happened after shrinkage.

+ + + +
ggplot(comparison_df, 
+       aes(x = lfc_original, 
+           y = lfc_shrunken, 
+           color = logmean)) +
+  geom_point(alpha = 0.1) +
+  theme_bw() +
+  scale_color_viridis_c() + 
+  coord_cartesian(xlim = c(-10,10), ylim = c(-10,10)) # zoom in on the middle
+ + +

+ + + +

We will now do a bit of manipulation to store the results in a data frame and add the gene symbols.

+ + + +
# this is of class DESeqResults -- we want a data frame
+deseq_df <- deseq_shrunken %>%
+  # convert to a data frame
+  as.data.frame() %>%
+  # the gene ids were stored as row names -- let's them a column for easy display
+  tibble::rownames_to_column(var = "gene_id") %>%
+  # add on the gene symbols from the original deseq object
+  dplyr::mutate(gene_symbol = rowData(deseq_object)$gene_name)
+ + + +

Let’s print out the results table, sorted by log2 fold change. The highest values should be genes more expressed in the MYCN amplified cell lines.

+ + + +
# Print the table sorted by log2FoldChange 
+deseq_df %>% 
+  dplyr::arrange(dplyr::desc(log2FoldChange))
+ +
+ +
+ + +

Now let’s write the full results table to a file.

+ + + +
readr::write_tsv(deseq_df, file = deseq_df_file)
+ + + +
+
+
+

Making a Volcano Plot

+

With these shrunken effect sizes, we will draw a volcano plot, using the EnhancedVolcano package to make it a bit easier. This package automatically color codes the points by cutoffs for both significance and fold change and labels many of the significant genes (subject to spacing). EnhancedVolcano has many, many options, which is a good thing if you don’t like all of it’s default settings. Even better, it outputs a ggplot2 object, so if we want to customize it further, we can do that with the same ggplot2 commands we have used before.

+ + + +
EnhancedVolcano(deseq_df,
+                x = 'log2FoldChange', # fold change statistic to plot
+                y = 'pvalue', # significance values
+                lab = deseq_df$gene_symbol, # labels for points
+                pCutoff = 1e-05, # The p value cutoff we will use (default)
+                FCcutoff = 1, # The fold change cutoff (default)
+                title = NULL, # no title
+                subtitle = NULL, # or subtitle
+                caption = NULL, # or caption
+                labSize = 3  # smaller labels
+                ) +
+  # change the overall theme
+  theme_classic() +
+  # move the legend to the bottom
+  theme(legend.position = "bottom")
+ + +

+ + + +

We will save this plot to a file as well:

+ + + +
ggsave(volcano_file, plot = last_plot())
+ + +
Saving 7 x 5 in image
+ + + +
+
+

Session Info

+

Record session info for reproducibility & provenance purposes.

+ + + +
sessionInfo()
+ + +
R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
+[8] methods   base     
+
+other attached packages:
+ [1] EnhancedVolcano_1.8.0       ggrepel_0.9.1              
+ [3] ggplot2_3.3.3               DESeq2_1.30.1              
+ [5] SummarizedExperiment_1.20.0 Biobase_2.50.0             
+ [7] MatrixGenerics_1.2.0        matrixStats_0.57.0         
+ [9] GenomicRanges_1.42.0        GenomeInfoDb_1.26.2        
+[11] IRanges_2.24.1              S4Vectors_0.28.1           
+[13] BiocGenerics_0.36.0         magrittr_2.0.1             
+[15] optparse_1.6.6             
+
+loaded via a namespace (and not attached):
+ [1] bitops_1.0-6           bit64_4.0.5            ash_1.0-15            
+ [4] RColorBrewer_1.1-2     httr_1.4.2             tools_4.0.3           
+ [7] irlba_2.3.3            R6_2.5.0               KernSmooth_2.23-18    
+[10] vipor_0.4.5            DBI_1.1.1              colorspace_2.0-0      
+[13] withr_2.4.0            tidyselect_1.1.0       ggrastr_0.2.1         
+[16] ggalt_0.4.0            bit_4.0.4              compiler_4.0.3        
+[19] extrafontdb_1.0        DelayedArray_0.16.2    labeling_0.4.2        
+[22] scales_1.1.1           SQUAREM_2021.1         proj4_1.0-10          
+[25] readr_1.4.0            genefilter_1.72.1      mixsqp_0.3-43         
+[28] stringr_1.4.0          digest_0.6.27          rmarkdown_2.6         
+[31] XVector_0.30.0         pkgconfig_2.0.3        htmltools_0.5.1.1     
+[34] extrafont_0.17         invgamma_1.1           maps_3.3.0            
+[37] rlang_0.4.10           RSQLite_2.2.3          farver_2.0.3          
+[40] generics_0.1.0         jsonlite_1.7.2         BiocParallel_1.24.1   
+[43] dplyr_1.0.3            RCurl_1.98-1.2         GenomeInfoDbData_1.2.4
+[46] Matrix_1.3-2           Rcpp_1.0.6             ggbeeswarm_0.6.0      
+[49] munsell_0.5.0          lifecycle_0.2.0        stringi_1.5.3         
+[52] yaml_2.2.1             MASS_7.3-53            zlibbioc_1.36.0       
+[55] grid_4.0.3             blob_1.2.1             crayon_1.3.4          
+[58] lattice_0.20-41        splines_4.0.3          annotate_1.68.0       
+[61] hms_1.0.0              locfit_1.5-9.4         knitr_1.30            
+[64] pillar_1.4.7           geneplotter_1.68.0     XML_3.99-0.5          
+[67] glue_1.4.2             evaluate_0.14          vctrs_0.3.6           
+[70] Rttf2pt1_1.3.8         gtable_0.3.0           getopt_1.20.3         
+[73] purrr_0.3.4            assertthat_0.2.1       ashr_2.2-47           
+[76] xfun_0.20              xtable_1.8-4           viridisLite_0.3.0     
+[79] survival_3.2-7         truncnorm_1.0-8        tibble_3.0.5          
+[82] AnnotationDbi_1.52.0   beeswarm_0.2.3         memoise_1.1.0         
+[85] ellipsis_0.3.1
+ + +
+ +
LS0tCnRpdGxlOiAiTmV1cm9ibGFzdG9tYSBDZWxsIExpbmU6IERpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIHdpdGggREVTZXEyIgphdXRob3I6IENDREwgZm9yIEFMU0YKZGF0ZTogMjAyMQpvdXRwdXQ6ICAgCiAgaHRtbF9ub3RlYm9vazogCiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQotLS0KCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBQZXJmb3JtIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIHdpdGggYERFU2VxMmAKLSBBcHBseSBhIHNocmlua2FnZSBhbGdvcml0aG0gdG8gaW1wcm92ZSBlc3RpbWF0ZXMgb2YgZXhwcmVzc2lvbiBjaGFuZ2VzCi0gRHJhdyBhIHZvbGNhbm8gcGxvdCB3aXRoIHRoZSBgRW5oYW5jZWRWb2xjYW5vYCBwYWNrYWdlCgotLS0KCkluIHRoaXMgbm90ZWJvb2ssIHdlJ2xsIHBlcmZvcm0gYW4gYW5hbHlzaXMgdG8gaWRlbnRpZnkgdGhlIGdlbmVzIHRoYXQgYXJlIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBpbiBfTVlDTl8gYW1wbGlmaWVkIHZzLiBub25hbXBsaWZpZWQgbmV1cm9ibGFzdG9tYSBjZWxsIGxpbmVzLiAKClRoZXNlIFJOQS1zZXEgZGF0YSBhcmUgZnJvbSBbSGFyZW56YSwgX2V0IGFsLl8gKDIwMTcpXShodHRwczovL2RvaS5vcmcvMTAuMTAzOC9zZGF0YS4yMDE3LjMzKS4KCk1vcmUgaW5mb3JtYXRpb24gYWJvdXQgREVTZXEyIGNhbiBiZSBmb3VuZCBpbiB0aGUgW2V4Y2VsbGVudCB2aWduZXR0ZV0oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sKSBmcm9tIExvdmUsIEFuZGVycywgYW5kIEh1YmVyIGZyb20gd2hpY2ggdGhpcyBpcyBhZGFwdGVkIChzZWUgYWxzbzogW0xvdmUsIF9ldCBhbC5fICgyMDE0KV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvczEzMDU5LTAxNC0wNTUwLTgpKS4KCkRFU2VxMiB0YWtlcyB1bm5vcm1hbGl6ZWQgY291bnRzIG9yIGVzdGltYXRlZCBjb3VudHMgYW5kIGRvZXMgdGhlIGZvbGxvd2luZzoKCiogW0VzdGltYXRlcyBzaXplIGZhY3RvcnNdKGh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9ERVNlcTIvdmVyc2lvbnMvMS4xMi4zL3RvcGljcy9lc3RpbWF0ZVNpemVGYWN0b3JzKQoqIFtFc3RpbWF0ZXMgZGlzcGVyc2lvbl0oaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL0RFU2VxMi92ZXJzaW9ucy8xLjEyLjMvdG9waWNzL2VzdGltYXRlRGlzcGVyc2lvbnMpIAoqIE5lZ2F0aXZlIGJpbm9taWFsIGdlbmVyYWxpemVkIGxpbmVhciBtb2RlbCBmaXR0aW5nIGFuZCBbV2FsZCBzdGF0aXN0aWNzXShodHRwczovL3d3dy5yZG9jdW1lbnRhdGlvbi5vcmcvcGFja2FnZXMvREVTZXEyL3ZlcnNpb25zLzEuMTIuMy90b3BpY3MvbmJpbm9tV2FsZFRlc3QpCgohW10oZGlhZ3JhbXMvcm5hLXNlcV82LnBuZykKCiMjIExpYnJhcmllcyBhbmQgZnVuY3Rpb25zCgpgYGB7ciBsaWJyYXJ5fQojIG1hZ3JpdHRyIHBpcGUKbGlicmFyeShtYWdyaXR0cikKCiMgTG9hZCB0aGUgREVTZXEyIGxpYnJhcnkKbGlicmFyeShERVNlcTIpCgojIFdlIHdpbGwgYmUgbWFraW5nIGZhbmN5IHZvbGNhbm8gcGxvdHMKbGlicmFyeShFbmhhbmNlZFZvbGNhbm8pCmBgYAoKIyMgRGlyZWN0b3JpZXMgYW5kIGZpbGVzCgoqKklucHV0KioKCmBgYHtyIGlucHV0LWZpbGVzfQojIGRpcmVjdG9yeSB3aXRoIHRoZSB0eGltZXRhIHByb2Nlc3NlZCBkYXRhCnR4aV9kaXIgPC0gZmlsZS5wYXRoKCJkYXRhIiwgIk5CLWNlbGwiLCAidHhpIikKdHhpX2ZpbGUgPC0gZmlsZS5wYXRoKHR4aV9kaXIsICJOQi1jZWxsX3R4aW1ldGEuUkRTIikKYGBgCgoKKipPdXRwdXQqKgoKV2UnbGwgY3JlYXRlIGEgcmVzdWx0cyBkaXJlY3RvcnkgdG8gaG9sZCBvdXIgcmVzdWx0cy4KCmBgYHtyIHJlc3VsdHMtZGlyfQojIENyZWF0ZSBhIHJlc3VsdHMgZGlyZWN0b3J5IGlmIGl0IGRvZXNuJ3QgYWxyZWFkeSBleGlzdApyZXN1bHRzX2RpciA8LSBmaWxlLnBhdGgoInJlc3VsdHMiLCAiTkItY2VsbCIpCmlmICghZGlyLmV4aXN0cyhyZXN1bHRzX2RpcikpIHsKICBkaXIuY3JlYXRlKHJlc3VsdHNfZGlyLCByZWN1cnNpdmUgPSBUUlVFKQp9CmBgYAoKV2Ugd2lsbCBhbHNvIG5lZWQgYSBkaXJlY3RvcnkgdG8gc3RvcmUgb3VyIHBsb3RzLgoKYGBge3IgcGxvdHMtZGlyLCBsaXZlID0gVFJVRX0KIyBDcmVhdGUgYSBwbG90cyBkaXJlY3RvcnkgaWYgaXQgZG9lc24ndCBhbHJlYWR5IGV4aXN0CnBsb3RzX2RpciA8LSBmaWxlLnBhdGgoInBsb3RzIiwgIk5CLWNlbGwiKQppZiAoIWRpci5leGlzdHMocGxvdHNfZGlyKSkgewogIGRpci5jcmVhdGUocGxvdHNfZGlyLCByZWN1cnNpdmUgPSBUUlVFKQp9CmBgYAoKCmBgYHtyIG91dHB1dC1maWxlc30KIyBSRFMgZm9yIHRoZSBvdXRwdXQgb2YgREVTZXEgYW5hbHlzaXMKZGVzZXFfZmlsZSA8LSBmaWxlLnBhdGgocmVzdWx0c19kaXIsIAogICAgICAgICAgICAgICAgICAgICAgICAiTkItY2VsbF9ERVNlcV9hbXBsaWZpZWRfdl9ub25hbXBsaWZpZWQuUkRTIikKCiMgREVTZXEyIHJlc3VsdHMgdGFibGUKZGVzZXFfZGZfZmlsZSA8LSBmaWxlLnBhdGgocmVzdWx0c19kaXIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICJOQi1jZWxsX0RFU2VxX2FtcGxpZmllZF92X25vbmFtcGxpZmllZF9yZXN1bHRzLnRzdiIpCgojIFBORyBvZiB0aGUgdm9sY2FubyBwbG90CnZvbGNhbm9fZmlsZSA8LSBmaWxlLnBhdGgocGxvdHNfZGlyLCAiTkItY2VsbF92b2xjYW5vLnBuZyIpCmBgYAoKIyMgREVTZXEyCgojIyMgQ3JlYXRpbmcgYSBERVNlcTIgZGF0YXNldCBmcm9tIHR4aW1ldGEgb2JqZWN0CgpGaXJzdCwgbGV0J3MgcmVhZCBpbiB0aGUgZGF0YSB3ZSBwcm9jZXNzZWQgd2l0aCBgdHhpbWV0YWAuCgojIyMjIFByZXBhcmF0aW9uCgpgYGB7ciByZWFkX3JkcywgbGl2ZSA9IFRSVUV9CiMgUmVhZCBpbiB0aGUgUkRTIGZpbGUgd2UgY3JlYXRlZCBpbiB0aGUgbGFzdCBub3RlYm9vawpnZW5lX3N1bW1hcml6ZWQgPC0gcmVhZHI6OnJlYWRfcmRzKHR4aV9maWxlKQpgYGAKCldlJ3JlIG1vc3QgaW50ZXJlc3RlZCBpbiBfTVlDTl8gYW1wbGlmaWNhdGlvbiwgd2hpY2ggd2UgaGFkIHN0b3JlZCBpbiB0aGUgYHN0YXR1c2AgY29sdW1uIG9mIHRoZSBzYW1wbGUgbWV0YWRhdGEgb2YgYGdlbmVfc3VtbWFyaXplZGAuCldoaWxlIHRoZSBzYW1wbGUgbWV0YWRhdGEgaXMgc3RvcmVkIGludGVybmFsbHkgaW4gdGhlIGBjb2xEYXRhYCBzbG90LCB0aGUgYFN1bW1hcml6ZWRFeHBlcmltZW50YCBvYmplY3QgbWFrZXMgaXQgZWFzeSBmb3IgdXMgdG8gYWNjZXNzIGl0IGFzIGlmIGl0IHdlcmUganVzdCBhIGNvbHVtbiBvZiBhIGRhdGEgZnJhbWUsIHVzaW5nIHRoZSBmYW1pbGlhciBgJGAgc3ludGF4LgoKCmBgYHtyIFN0YXR1cywgbGl2ZSA9IFRSVUV9CmdlbmVfc3VtbWFyaXplZCRzdGF0dXMKYGBgCgpUaGlzIGlzIHN0b3JlZCBhcyBhIGBjaGFyYWN0ZXJgIHR5cGUsIGJ1dCB0byBnaXZlIGEgYml0IG1vcmUgaW5mb3JtYXRpb24gdG8gYERFU2VxYCwgd2Ugd2lsbCBjb252ZXJ0IHRoaXMgdG8gYSBgZmFjdG9yYC4KCmBgYHtyIHN0YXR1c19mYWN0b3IsIGxpdmUgPSBUUlVFfQpnZW5lX3N1bW1hcml6ZWQkc3RhdHVzIDwtIGFzLmZhY3RvcihnZW5lX3N1bW1hcml6ZWQkc3RhdHVzKQpgYGAKCldlJ2xsIHdhbnQgdG8gdXNlIHRoZSAiTm9uYW1wbGlmaWVkIiBzYW1wbGVzIGFzIG91ciBfcmVmZXJlbmNlXy4KTGV0J3MgbG9vayBhdCB0aGUgYGxldmVsc2Agb2YgYHN0YXR1c2AuCgpgYGB7ciBsZXZlbHN9CmxldmVscyhnZW5lX3N1bW1hcml6ZWQkc3RhdHVzKQpgYGAKCldlIGNhbiBzZWUgdGhhdCB0aGVzZSBhcmUgaW4gYWxwaGFiZXRpY2FsIG9yZGVyLCBzbyAiQW1wbGlmaWVkIiBzYW1wbGVzIHdvdWxkIGJlIHRoZSByZWZlcmVuY2UuIApXZSBjYW4gdXNlIHRoZSBgcmVsZXZlbCgpYCBmdW5jdGlvbiB0byByZW1lZHkgdGhpcy4KCmBgYHtyIHJlbGV2ZWx9CmdlbmVfc3VtbWFyaXplZCRzdGF0dXMgPC0gcmVsZXZlbChnZW5lX3N1bW1hcml6ZWQkc3RhdHVzLCByZWYgPSAiTm9uYW1wbGlmaWVkIikKYGBgCgpgYGB7ciBjaGVjay1sZXZlbHMsIGxpdmUgPSBUUlVFfQojIENoZWNrIHdoYXQgdGhlIGxldmVscyBhcmUgbm93CmxldmVscyhnZW5lX3N1bW1hcml6ZWQkc3RhdHVzKQpgYGAKCgoKIyMjIyBERVNlcSBEYXRhc2V0IGNyZWF0aW9uCgpgYGB7ciBkZHNldCwgbGl2ZSA9IFRSVUV9CiMgQ3JlYXRlIGEgREVTZXEyIGRhdGFzZXQgZnJvbSBgZ2VuZV9zdW1tYXJpemVkYCAKIyByZW1lbWJlciB0aGF0IGBzdGF0dXNgIGlzIHRoZSB2YXJpYWJsZSBvZiBpbnRlcmVzdCBoZXJlCmRkc2V0IDwtIERFU2VxRGF0YVNldChnZW5lX3N1bW1hcml6ZWQsCiAgICAgICAgICAgICAgICAgICAgICBkZXNpZ24gPSB+IHN0YXR1cykKYGBgCgojIyBEaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcwoKIyMjIEZpbHRlcmluZyBsb3ctZXhwcmVzc2VkIGdlbmVzCgpHZW5lcyB0aGF0IGhhdmUgdmVyeSBsb3cgY291bnRzIGFyZSBub3QgbGlrZWx5IHRvIHlpZWxkIHJlbGlhYmxlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIHJlc3VsdHMsIHNvIHdlIHdpbGwgZG8gc29tZSBsaWdodCBbcHJlLWZpbHRlcmluZ10oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjcHJlLWZpbHRlcmluZykuIApXZSB3aWxsIGtlZXAgb25seSBnZW5lcyB3aXRoIHRvdGFsIGNvdW50cyBvZiBhdCBsZWFzdCAxMCBhY3Jvc3MgYWxsIHNhbXBsZXMuCgpgYGB7ciBmaWx0ZXJfZGRzZXR9CmdlbmVzX3RvX2tlZXAgPC0gcm93U3Vtcyhjb3VudHMoZGRzZXQpKSA+PSAxMApkZHNldCA8LSBkZHNldFtnZW5lc190b19rZWVwLCBdCmBgYAoKCiMjIyBUaGUgYERFU2VxKClgIGZ1bmN0aW9uCgpXZSdsbCBub3cgdXNlIHRoZSB3cmFwcGVyIGZ1bmN0aW9uIGBERVNlcSgpYCB0byBwZXJmb3JtIG91ciBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcy4KQXMgbWVudGlvbmVkIGVhcmxpZXIsIHRoaXMgcGVyZm9ybXMgYSBudW1iZXIgb2Ygc3RlcHMsIGluY2x1ZGluZyBhbiBbb3V0bGllciByZW1vdmFsIHByb2NlZHVyZV0oaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI2FwcHJvYWNoLXRvLWNvdW50LW91dGxpZXJzKS4gCkZvciB0aGlzIHBhcnRpY3VsYXIgZGF0YXNldCwgdGhlcmUgaXMgYSBwcmV0dHkgbGFyZ2UgbnVtYmVyIG9mIG91dGxpZXJzLCB3aGljaCBjYW4gYmUgYSBiaXQgb2YgYSByZWQgZmxhZywgYnV0IHdlIHdpbGwgcHJvY2VlZCBmb3Igbm93LgoKYGBge3IgREVTZXF9CmRlc2VxX29iamVjdCA8LSBERVNlcShkZHNldCkKYGBgCgpMZXQncyBzYXZlIHRoaXMgdG8gb3VyIHJlc3VsdHMgZmlsZS4KCmBgYHtyIHdyaXRlX3JkcywgbGl2ZSA9IFRSVUV9CiMgU2F2ZSB0aGUgcmVzdWx0cyBhcyBhbiBSRFMKcmVhZHI6OndyaXRlX3JkcyhkZXNlcV9vYmplY3QsIGZpbGUgPSBkZXNlcV9maWxlKQpgYGAKCk5vdyB3ZSB3aWxsIGhhdmUgYSBsb29rIGF0IHRoZSByZXN1bHRzIHRhYmxlLgoKYGBge3IgZGVzZXFfcmVzdWx0c30KZGVzZXFfcmVzdWx0cyA8LSByZXN1bHRzKGRlc2VxX29iamVjdCkKZGVzZXFfcmVzdWx0cwpgYGAKCkhvdyBtYW55IGdlbmVzIHdlcmUgZGlmZmVyZW50aWFsbHkgZXhwcmVzc2VkIChGRFIgPCAwLjA1KT8KCmBgYHtyIHJlc3VsdHNfc3VtbWFyeX0Kc3VtbWFyeShkZXNlcV9yZXN1bHRzLCBhbHBoYSA9IDAuMDUpCmBgYAoKCiMjIyBTaHJpbmtpbmcgbG9nMiBmb2xkIGNoYW5nZSBlc3RpbWF0ZXMKClRoZSBlc3RpbWF0ZXMgb2YgbG9nMiBmb2xkIGNoYW5nZSBjYWxjdWxhdGVkIGJ5IGBERVNlcSgpYCBhcmUgbm90IGNvcnJlY3RlZCBmb3IgZXhwcmVzc2lvbiBsZXZlbC4KVGhpcyBtZWFucyB0aGF0IHdoZW4gY291bnRzIGFyZSBzbWFsbCwgd2UgYXJlIGxpa2VseSB0byBlbmQgdXAgd2l0aCBzb21lIGxhcmdlIGZvbGQgY2hhbmdlIHZhbHVlcyB0aGF0IG92ZXJlc3RpbWF0ZSB0aGUgdHJ1ZSBleHRlbnQgb2YgdGhlIGNoYW5nZSBiZXR3ZWVuIGNvbmRpdGlvbnMuCgpXZSBjYW4gY29ycmVjdCB0aGlzIGJ5IGFwcGx5aW5nIGEgInNocmlua2FnZSIgcHJvY2VkdXJlLCB3aGljaCB3aWxsIGFkanVzdCBsYXJnZSB2YWx1ZXMgd2l0aCBzbWFsbCBjb3VudHMgZG93bndhcmQsIHdoaWxlIHByZXNlcnZpbmcgdmFsdWVzIHdpdGggbGFyZ2VyIGNvdW50cywgd2hpY2ggYXJlIGxpa2VseSB0byBiZSBtb3JlIGFjY3VyYXRlLgoKVG8gZG8gdGhpcywgd2Ugd2lsbCB1c2UgdGhlIGBsZmNTaHJpbmsoKWAgZnVuY3Rpb24sIGJ1dCBmaXJzdCB3ZSBuZWVkIHRvIGtub3cgdGhlIG5hbWUgYW5kL29yIHBvc2l0aW9uIG9mIHRoZSAiY29lZmZpY2llbnQiIHRoYXQgd2FzIGNhbGN1bGF0ZWQgYnkgYERFU2VxKClgLCB3aGljaCB3ZSBjYW4gZG8gd2l0aCB0aGUgYHJlc3VsdHNOYW1lcygpYCBmdW5jdGlvbgoKYGBge3IgZGVzZXFfY29lZn0KIyBnZXQgdGhlIGRlc2VxIGNvZWZmaWNpZW50IG5hbWVzOgpyZXN1bHRzTmFtZXMoZGVzZXFfb2JqZWN0KQpgYGAKCldlIGFyZSBpbnRlcmVzdGVkIGluIHRoZSBgc3RhdHVzYCBjb2VmZmljaWVudCwgd2hpY2ggaXMgaW4gcG9zaXRpb24gMi4KClRoZXJlIGFyZSBhIGZldyBvcHRpb25zIGZvciB0aGUgc2hyaW5rYWdlIGVzdGltYXRpb24uIApUaGUgZGVmYXVsdCBpcyBgYXBlZ2xtYCAoW1podSBfZXQgYWwuXyAyMDE4XShodHRwczovL2RvaS5vcmcvMTAuMTA5My9iaW9pbmZvcm1hdGljcy9idHk4OTUpKSwgYnV0IHdlIGhhdmUgZm91bmQgdGhhdCB0aGlzIGNhbiBiZSBzZW5zaXRpdmUgdG8gZXh0cmVtZSBvdXRsaWVycywgd2hpY2ggYXJlIGRlZmluaXRlbHkgYSBmYWN0b3IgaW4gdGhpcyBkYXRhIHNldC4KU28gZm9yIHRoaXMgZGF0YSBzZXQgd2Ugd2lsbCBiZSB1c2luZyBgYXNocmAgKFtTdGVwaGVucyAyMDE3XShodHRwczovL2RvaS5vcmcvMTAuMTA5My9iaW9zdGF0aXN0aWNzL2t4dzA0MSkpIAoKYGBge3IgbGZjX3Nocmlua30KIyBjYWxjdWxhdGUgc2hydW5rZW4gbG9nMiBmb2xkIGNoYW5nZSBlc3RpbWF0ZXMKZGVzZXFfc2hydW5rZW4gPC0gbGZjU2hyaW5rKGRlc2VxX29iamVjdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgdGhlIGNvZWZmaWNpZW50IHdlIHdhbnQgdG8gcmVlc3RpbWF0ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgY29lZiA9IDIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBXZSB3aWxsIHVzZSBgYXNocmAgZm9yIGVzdGltYXRpb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR5cGUgPSAiYXNociIKICAgICAgICAgICAgICAgICAgICAgICAgICAgICkKYGBgCgpMZXQncyBjb21wYXJlIHRoZSBsb2cyIGZvbGQgY2hhbmdlIGVzdGltYXRlcyBmcm9tIHRoZSB0d28gcmVzdWx0cyB0YWJsZXMgYnkgY3JlYXRpbmcgYSBwbG90LiAKCkZpcnN0IHdlIHdpbGwgY29tYmluZSB0aGUgcmVzdWx0cyBpbnRvIGEgbmV3IGRhdGEgZnJhbWUuCgpgYGB7ciBjb21wYXJlX3Nocmlua30KY29tcGFyaXNvbl9kZiA8LSBkYXRhLmZyYW1lKAogIGxmY19vcmlnaW5hbCA9IGRlc2VxX3Jlc3VsdHMkbG9nMkZvbGRDaGFuZ2UsCiAgbGZjX3NocnVua2VuID0gZGVzZXFfc2hydW5rZW4kbG9nMkZvbGRDaGFuZ2UsCiAgbG9nbWVhbiA9IGxvZzEwKGRlc2VxX3Jlc3VsdHMkYmFzZU1lYW4pCiAgKQpgYGAKCk5vdyB3ZSBjYW4gcGxvdCB0aGUgb3JpZ2luYWwgYW5kIHNocnVua2VuIGxvZzIgZm9sZCBjaGFuZ2UgdmFsdWVzIHRvIHNlZSB3aGF0IGhhcHBlbmVkIGFmdGVyIHNocmlua2FnZS4KCmBgYHtyIHBsb3RfY29tcGFyaXNvbn0KZ2dwbG90KGNvbXBhcmlzb25fZGYsIAogICAgICAgYWVzKHggPSBsZmNfb3JpZ2luYWwsIAogICAgICAgICAgIHkgPSBsZmNfc2hydW5rZW4sIAogICAgICAgICAgIGNvbG9yID0gbG9nbWVhbikpICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4xKSArCiAgdGhlbWVfYncoKSArCiAgc2NhbGVfY29sb3JfdmlyaWRpc19jKCkgKyAKICBjb29yZF9jYXJ0ZXNpYW4oeGxpbSA9IGMoLTEwLDEwKSwgeWxpbSA9IGMoLTEwLDEwKSkgIyB6b29tIGluIG9uIHRoZSBtaWRkbGUKYGBgCgpXZSB3aWxsIG5vdyBkbyBhIGJpdCBvZiBtYW5pcHVsYXRpb24gdG8gc3RvcmUgdGhlIHJlc3VsdHMgaW4gYSBkYXRhIGZyYW1lIGFuZCBhZGQgdGhlIGdlbmUgc3ltYm9scy4KCmBgYHtyIHJlc3VsdHNfZGF0YWZyYW1lfQojIHRoaXMgaXMgb2YgY2xhc3MgREVTZXFSZXN1bHRzIC0tIHdlIHdhbnQgYSBkYXRhIGZyYW1lCmRlc2VxX2RmIDwtIGRlc2VxX3NocnVua2VuICU+JQogICMgY29udmVydCB0byBhIGRhdGEgZnJhbWUKICBhcy5kYXRhLmZyYW1lKCkgJT4lCiAgIyB0aGUgZ2VuZSBpZHMgd2VyZSBzdG9yZWQgYXMgcm93IG5hbWVzIC0tIGxldCdzIHRoZW0gYSBjb2x1bW4gZm9yIGVhc3kgZGlzcGxheQogIHRpYmJsZTo6cm93bmFtZXNfdG9fY29sdW1uKHZhciA9ICJnZW5lX2lkIikgJT4lCiAgIyBhZGQgb24gdGhlIGdlbmUgc3ltYm9scyBmcm9tIHRoZSBvcmlnaW5hbCBkZXNlcSBvYmplY3QKICBkcGx5cjo6bXV0YXRlKGdlbmVfc3ltYm9sID0gcm93RGF0YShkZXNlcV9vYmplY3QpJGdlbmVfbmFtZSkKYGBgCgpMZXQncyBwcmludCBvdXQgdGhlIHJlc3VsdHMgdGFibGUsIHNvcnRlZCBieSBsb2cyIGZvbGQgY2hhbmdlLiAKVGhlIGhpZ2hlc3QgdmFsdWVzIHNob3VsZCBiZSBnZW5lcyBtb3JlIGV4cHJlc3NlZCBpbiB0aGUgX01ZQ05fIGFtcGxpZmllZCBjZWxsIGxpbmVzLgoKYGBge3Igc29ydGVkX3RhYmxlLCBsaXZlID0gVFJVRX0KIyBQcmludCB0aGUgdGFibGUgc29ydGVkIGJ5IGxvZzJGb2xkQ2hhbmdlIApkZXNlcV9kZiAlPiUgCiAgZHBseXI6OmFycmFuZ2UoZHBseXI6OmRlc2MobG9nMkZvbGRDaGFuZ2UpKQpgYGAKCk5vdyBsZXQncyB3cml0ZSB0aGUgZnVsbCByZXN1bHRzIHRhYmxlIHRvIGEgZmlsZS4KCmBgYHtyIHdyaXRlX3Rzdn0KcmVhZHI6OndyaXRlX3RzdihkZXNlcV9kZiwgZmlsZSA9IGRlc2VxX2RmX2ZpbGUpCmBgYAoKCiMjIE1ha2luZyBhIFZvbGNhbm8gUGxvdCAKCldpdGggdGhlc2Ugc2hydW5rZW4gZWZmZWN0IHNpemVzLCB3ZSB3aWxsIGRyYXcgYSB2b2xjYW5vIHBsb3QsIHVzaW5nIHRoZSBbYEVuaGFuY2VkVm9sY2Fub2AgcGFja2FnZV0oaHR0cHM6Ly9naXRodWIuY29tL2tldmluYmxpZ2hlL0VuaGFuY2VkVm9sY2FubykgdG8gbWFrZSBpdCBhIGJpdCBlYXNpZXIuClRoaXMgcGFja2FnZSBhdXRvbWF0aWNhbGx5IGNvbG9yIGNvZGVzIHRoZSBwb2ludHMgYnkgY3V0b2ZmcyBmb3IgYm90aCBzaWduaWZpY2FuY2UgYW5kIGZvbGQgY2hhbmdlIGFuZCBsYWJlbHMgbWFueSBvZiB0aGUgc2lnbmlmaWNhbnQgZ2VuZXMgKHN1YmplY3QgdG8gc3BhY2luZykuCmBFbmhhbmNlZFZvbGNhbm9gIGhhcyBtYW55LCBtYW55IG9wdGlvbnMsIHdoaWNoIGlzIGEgZ29vZCB0aGluZyBpZiB5b3UgZG9uJ3QgbGlrZSBhbGwgb2YgaXQncyBkZWZhdWx0IHNldHRpbmdzLgpFdmVuIGJldHRlciwgaXQgb3V0cHV0cyBhIGBnZ3Bsb3QyYCBvYmplY3QsIHNvIGlmIHdlIHdhbnQgdG8gY3VzdG9taXplIGl0IGZ1cnRoZXIsIHdlIGNhbiBkbyB0aGF0IHdpdGggdGhlIHNhbWUgYGdncGxvdDJgIGNvbW1hbmRzIHdlIGhhdmUgdXNlZCBiZWZvcmUuCgpgYGB7ciB2b2xjYW5vfQpFbmhhbmNlZFZvbGNhbm8oZGVzZXFfZGYsCiAgICAgICAgICAgICAgICB4ID0gJ2xvZzJGb2xkQ2hhbmdlJywgIyBmb2xkIGNoYW5nZSBzdGF0aXN0aWMgdG8gcGxvdAogICAgICAgICAgICAgICAgeSA9ICdwdmFsdWUnLCAjIHNpZ25pZmljYW5jZSB2YWx1ZXMKICAgICAgICAgICAgICAgIGxhYiA9IGRlc2VxX2RmJGdlbmVfc3ltYm9sLCAjIGxhYmVscyBmb3IgcG9pbnRzCiAgICAgICAgICAgICAgICBwQ3V0b2ZmID0gMWUtMDUsICMgVGhlIHAgdmFsdWUgY3V0b2ZmIHdlIHdpbGwgdXNlIChkZWZhdWx0KQogICAgICAgICAgICAgICAgRkNjdXRvZmYgPSAxLCAjIFRoZSBmb2xkIGNoYW5nZSBjdXRvZmYgKGRlZmF1bHQpCiAgICAgICAgICAgICAgICB0aXRsZSA9IE5VTEwsICMgbm8gdGl0bGUKICAgICAgICAgICAgICAgIHN1YnRpdGxlID0gTlVMTCwgIyBvciBzdWJ0aXRsZQogICAgICAgICAgICAgICAgY2FwdGlvbiA9IE5VTEwsICMgb3IgY2FwdGlvbgogICAgICAgICAgICAgICAgbGFiU2l6ZSA9IDMgICMgc21hbGxlciBsYWJlbHMKICAgICAgICAgICAgICAgICkgKwogICMgY2hhbmdlIHRoZSBvdmVyYWxsIHRoZW1lCiAgdGhlbWVfY2xhc3NpYygpICsKICAjIG1vdmUgdGhlIGxlZ2VuZCB0byB0aGUgYm90dG9tCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gImJvdHRvbSIpCmBgYAoKV2Ugd2lsbCBzYXZlIHRoaXMgcGxvdCB0byBhIGZpbGUgYXMgd2VsbDoKCmBgYHtyIHNhdmVfcGxvdH0KZ2dzYXZlKHZvbGNhbm9fZmlsZSwgcGxvdCA9IGxhc3RfcGxvdCgpKQpgYGAKCgojIyBTZXNzaW9uIEluZm8KClJlY29yZCBzZXNzaW9uIGluZm8gZm9yIHJlcHJvZHVjaWJpbGl0eSAmIHByb3ZlbmFuY2UgcHVycG9zZXMuCgpgYGB7ciBzZXNzaW9uaW5mb30Kc2Vzc2lvbkluZm8oKQpgYGAK
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/RNA-seq/06-openpbta_heatmap-live.Rmd b/RNA-seq/06-openpbta_heatmap-live.Rmd new file mode 100644 index 00000000..40788420 --- /dev/null +++ b/RNA-seq/06-openpbta_heatmap-live.Rmd @@ -0,0 +1,380 @@ +--- +title: "OpenPBTA: Create a heatmap" +author: CCDL for ALSF +date: 2021 +output: + html_notebook: + toc: true + toc_float: true +--- + + +## Objectives + +This notebook will demonstrate how to: + +- Load tabular expression data and prepare it for use with `DESeq2` +- Create an annotated heatmap of gene expression variation using the `ComplexHeatmap` package +- Customize a PCA plot and compare the output with hierarchical clustering of the same data + +--- + +In this notebook, we cluster RNA-seq data from the Open Pediatric Brain Tumor Atlas (OpenPBTA) project and create a heatmap. +OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children's Hospital of Philadelphia conducted openly on GitHub. + +You can read more about the project [here](https://github.com/alexslemonade/openpbta-analysis/#openpbta-analysis). + +We've downloaded some of the publicly available expression data from the project and selected a subset with the most common disease types for analysis here. + +We'll use a package called [`ComplexHeatmap`](https://www.bioconductor.org/packages/release/bioc/html/ComplexHeatmap.html) to make our heatmap. +This package allows us to annotate our heatmap with sample information, and will also perform clustering as part of generating the heatmap. +It is highly flexible and opinionated - the data structures we pass `ComplexHeatmap` functions often need to be _just right_. +See the [`ComplexHeatmap` Complete Reference](https://jokergoo.github.io/ComplexHeatmap-reference/book/) for more information. + +## Set up + +### Libraries + +```{r load_libraries} +# We will manipulate RNASeq data with DESeq2 at the start +library(DESeq2) + +# Then we'll be doing a bit of data wrangling with the Tidyverse +library(tidyverse) + +# ComplexHeatmap is the package we'll use for making a heatmap +# It will do the hierarchical clustering for us as well +library(ComplexHeatmap) +``` + +### Directories and files + +We have stored the data we'll use in this notebook in `data/open-pbta`. + +```{r directories, live = TRUE} + +# We'll store the heatmap in plots/open-pbta - create directory if it doesn't exist yet + +``` + +#### Input files + +```{r input_files} +# The metadata describing the samples +histologies_file <- file.path(data_dir, "pbta-histologies-subset.tsv") + +# The RNA-seq counts table +rnaseq_file = file.path(data_dir, "pbta-rsem-expected_count-subset.rds") +``` + +#### Output files + +```{r png_output, live = TRUE} + +``` + +## Read in and prepare data + +### Metadata + +Let's read in the metadata file file and take a look at the data. + +```{r read_rds, live = TRUE} + +``` + +Use the chunk below to explore the metadata data frame. + +```{r explore_histologies_df, live = TRUE, eval = FALSE} + +``` + +We'll use the disease labels in a column called `short_histology` when we label the heatmap. +Let's count how many samples are assigned each `short_histology` label using the Tidyverse. + +```{r count_histologies} +histology_count_df <- histologies_df %>% + # Count how many samples are in each short_histology and name the column + # with that number n + count(short_histology) %>% + # Sort from largest number of samples to smallest number of samples in a + # histology + arrange(desc(n)) + +histology_count_df +``` + +### RNA-seq data + +Read in the expression count matrix (stored as a data frame). + +```{r read_in_rnaseq, live = TRUE} +# Read in and examine the RNA-seq data + +``` + +### Convert and round + +The count data we have consists mostly of integers, but because of the estimation procedure that RSEM uses, some counts are fractional. +`DESeq2`, which we will be using below expects all integers, so we will round all of the count data. +This is easiest if we first convert from a data frame to a matrix. + +```{r convert_round, live = TRUE} + + # move gene_id to the rownames + + # convert to a matrix and round + +``` + +### Variance Stabilizing Transformation + +Raw counts are not usually suitable for the algorithms we use for clustering and heatmap display, so we will use the `vst()` function from the `DESeq2` package to transform our data. + +Since we are starting from a matrix, not a `SummarizedExperiment` as we did previously, we will need to provide the sample information ourselves. +Just to be sure nothing is out of order, we will check that the identifiers for the sample information stored in `histologies_df` matches the columns of our matrix. + +```{r check-order} +all.equal(histologies_df$Kids_First_Biospecimen_ID, + colnames(rnaseq_mat)) +``` + +Now we can make our matrix into a `DESeq2` dataset, adding on the sample information from `histologies_df`. +Unlike when we were performing differential expression analysis, we won't provide an experimental design at this stage. + +```{r make-DESEq} +ddset <- DESeqDataSetFromMatrix(rnaseq_mat, + colData = histologies_df, + design = ~ 1) # don't store an experimental design +``` + +We will again remove low count genes, as they are not likely to be informative. + +```{r trim} +genes_to_keep <- rowSums(counts(ddset)) >= 10 +ddset <- ddset[genes_to_keep, ] +``` + +Now we can apply the variance stabilizing transformation, saving the results in a new object. + +```{r vst, live=TRUE} +# apply variance stabilizing transformation + +``` + +This object stores information about the transformation that was applied, but for now, we will only need the matrix of transformed data, which we can extract with `assay()`. + +```{r extract_expr, live = TRUE} +# extract transformed data + +``` + +What are the dimensions of this transformed RNA-seq data matrix? + +```{r rnaseq_dim, live = TRUE} + +``` + +Almost 50k genes would be hard to visualize on a single heatmap! +If we are making a heatmap to get an idea of the structure in our data, it can be helpful to subset to high variance genes. +This is because the genes that *don't* vary are not likely to contribute much to the overall patterns we are interested in. + +First, we'll calculate the variance for each gene using `rowVars()` from the `matrixStats` package and then take the genes in the top 10%. + +```{r high_var_genes} +# Calculate variance from the expression data +gene_variance <- matrixStats::rowVars(expr_mat) + +# Find the value that we'll use as a threshold to filter the top 10% +variance_threshold <- quantile(gene_variance, 0.9) + +# Row indices of high variance genes +high_variance_index <- which(gene_variance > variance_threshold) + +# What does a row index look like? +head(high_variance_index) +``` + +```{r high_var_mat, live = TRUE} +# Get a matrix that is subset to just the high variance genes + +``` + +## Heatmap + +### Annotation + +First, we'll set up the metadata that we want to use to label samples in the heatmap. +In `ComplexHeatmap` terminology, this is called annotation, or `HeatmapAnnotation`, specifically. + +```{r sample_annotation_df} +sample_annotation_df <- histologies_df %>% + # Select only the columns that we'll use + select(Kids_First_Biospecimen_ID, + short_histology, + composition, + tumor_descriptor) + +# Let's examine these columns +sample_annotation_df +``` +`ComplexHeatmap` is going to want the data frame we provide to have the sample identifiers as row names, so let's set that up. + +```{r annotation_as_df, live = TRUE} + +``` + +To specify the colors in our annotation bar, we need to create a list of named vectors. +The names of the list need to *exactly* match the column names in `sample_annotation_df` and the names in each vector need to *exactly* match the values in those columns. + +```{r color_palettes} +# The Okabe Ito palette is recommended for those with color vision deficiencies +histology_colors <- palette.colors(palette = "Okabe-Ito")[2:5] +# `palette.colors()` returns a named vector, which can cause trouble +histology_colors <- unname(histology_colors) + +# annotation color list for ComplexHeatMap +sample_annotation_colors <- list( + short_histology = c( + "LGAT" = histology_colors[[1]], + "Ependymoma" = histology_colors[[2]], + "HGAT" = histology_colors[[3]], + "Medulloblastoma" = histology_colors[[4]] + ), + composition = c( + "Solid Tissue" = "#A0A0A0", # light grey + "Derived Cell Line" = "#000000" # black + ), + tumor_descriptor = c( + "Initial CNS Tumor" = "#3333FF", + "Progressive" = "#FFFF99", + "Recurrence" = "#CCCCFF", + "Second Malignancy" = "#000033", + "Unavailable" = "#FFFFFF" # white for missing data + ) +) +``` + +We need to create a special type of object with `HeatmapAnnotation` using the annotation data frame and the list of color vectors. +We will also make the annotation labels a bit nicer to look at than the raw columns names. + +```{r create_annotation, live = TRUE} + +``` + +### Values for display + +We will z-score the expression values for display. +This is sometimes called a standardized score. +Some heatmap plotting packages will do this for you, but `ComplexHeatmap` does not. +It's calculated for each value in a row (gene) by subtracting the gene's mean and dividing by the gene's standard deviation. +This will result in every row having a mean of 0 and a standard deviation of 1. + +```{r zscore} +zscores_mat <- + (high_var_mat - rowMeans(high_var_mat)) / matrixStats::rowSds(high_var_mat) +``` + +Since all genes have the same z-score variance (by definition!), if you're filtering to high variance values, it's important to do that _prior_ to standardizing. + +### Heatmap itself! + +Okay, now we're ready to make a heatmap complete with annotation bars. + +```{r make_heatmap} +Heatmap(zscores_mat, + # The distance metric used for clustering the rows + # This is different from the default (Euclidean) + clustering_distance_rows = "pearson", + # Linkage method for row clustering + # This is different from the default (complete) + clustering_method_rows = "average", + # Distance metric for columns + clustering_distance_columns = "pearson", + # Linkage for columns + clustering_method_columns = "average", + show_row_names = FALSE, + show_column_names = FALSE, + # Add annotation bars to the top of the heatmap + top_annotation = column_annotation, + # This will be used as the label for the color bar + # of the cells of the heatmap itself + name = "z-score") +``` + +`ComplexHeatmap` gives a few warning here, but nothing to be concerned about. +One is complaining that the color scale may be truncated relative to our data. +The other warns that our very large heatmap itself is being drawn at a lower resolution to speed things up. + +We have to create the heatmap _twice_ -- once for display in the notebook and once to save to a PNG file. +For the second drawing we will deal with the raster warning and produce a higher resolution output. + +```{r save_heatmap_png, live = TRUE} +# Open PNG plot device + +# Heatmap, again! + +# Shut down current graphics device + +``` + +### Notes on clustering + +You probably noticed that there were some arguments that we defined that were related to the clustering of samples and genes when we built the heatmaps. +The clustering determines the arrangement of genes and samples in the heatmap, making it so genes or samples with similar expression patterns are grouped. +If we didn't do that, we would have a heatmap that looked basically like random static! + +The kind of clustering we performed was [hierarchical clustering](https://en.wikipedia.org/wiki/Hierarchical_clustering). +This is an _agglomerative_ method of clustering - each sample starts in its own cluster and then at each step of the algorithm the two most similar clusters are joined until there's only a single cluster left. + +The arguments we chose for `clustering_distance_*` and `clustering_method_*` told `ComplexHeatmap` to use 1 minus the correlation between samples as the distance measure, and to average samples when they were merged into a group. +This method is also known as [UPGMA (unweighted pair group method with arithmetic mean)](https://en.wikipedia.org/wiki/UPGMA). + +The result of this clustering process is a tree (dendrogram), which is shown at the top for samples and to the left for genes. +It is important to note that the linear arrangement of samples in the tree is somewhat arbitrary, so we have to be careful not to overinterpret it. +We can rotate around any branch of the tree in our visualization with no change to the tree topology itself. +In fact, the samples at opposite ends of the heatmap could be right next to one another with the right set of rotations! + + +### PCA as an alternative to clustering + +You probably noticed that the hierarchical clustering we performed didn't perfectly separate out the different cancer subtypes we were looking at. + +What would have happened if we had used PCA to visualize the relationships among samples? +Would that have made it easier to separate disease types? +Let's try it! + +We will use the VST transformed data here too, and the `plotPCA()` function from `DESeq2`. + +```{r expr-pca } +# Use plotPCA, but return the data for custom plotting +pca_df <- plotPCA(vst_data, + ntop = 5000, # use the top 5000 genes by variance + intgroup = "short_histology", + returnData = TRUE) + + +ggplot(pca_df, aes(PC1, PC2, color = short_histology)) + + geom_point() + + theme_bw() + + scale_color_manual( + values = c("LGAT" = histology_colors[[1]], + "Ependymoma" = histology_colors[[2]], + "HGAT" = histology_colors[[3]], + "Medulloblastoma" = histology_colors[[4]]) + ) + + labs(color = "Histology") +``` + +We can see that neither method perfectly separates the different disease types. +With PCA, if we didn't color the points, it might even be difficult to identify distinct clusters in the data at all by eye. +(We could look at more than two PCs, which might help out, but we would need to use a different function for our calculation, since `plotPCA()` only returns the first two.) + +For nice comparison of the relative advantages of these two methods (with a little mention of some further directions), we recommend [blog post by Soneson](https://www.rna-seqblog.com/a-comparison-between-pca-and-hierarchical-clustering/). + + +## Session Info + +```{r session_info} +sessionInfo() +``` diff --git a/RNA-seq/06-openpbta_heatmap.nb.html b/RNA-seq/06-openpbta_heatmap.nb.html new file mode 100644 index 00000000..dc76cb75 --- /dev/null +++ b/RNA-seq/06-openpbta_heatmap.nb.html @@ -0,0 +1,1118 @@ + + + + + + + + + + + + + + + +OpenPBTA: Create a heatmap + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+
+
+
+
+ +
+ + + + + + + + +
+

Objectives

+

This notebook will demonstrate how to:

+
    +
  • Load tabular expression data and prepare it for use with DESeq2
    • +
  • Create an annotated heatmap of gene expression variation using the ComplexHeatmap package
    • +
  • Customize a PCA plot and compare the output with hierarchical clustering of the same data
    • +
+
+

In this notebook, we cluster RNA-seq data from the Open Pediatric Brain Tumor Atlas (OpenPBTA) project and create a heatmap. OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

+

You can read more about the project here.

+

We’ve downloaded some of the publicly available expression data from the project and selected a subset with the most common disease types for analysis here.

+

We’ll use a package called ComplexHeatmap to make our heatmap. This package allows us to annotate our heatmap with sample information, and will also perform clustering as part of generating the heatmap. It is highly flexible and opinionated - the data structures we pass ComplexHeatmap functions often need to be just right. See the ComplexHeatmap Complete Reference for more information.

+
+
+

Set up

+
+

Libraries

+ + + +
# We will manipulate RNASeq data with DESeq2 at the start
+library(DESeq2)
+ + +
Loading required package: S4Vectors
+ + +
Loading required package: stats4
+ + +
Loading required package: BiocGenerics
+ + +
Loading required package: parallel
+ + +

+Attaching package: 'BiocGenerics'
+ + +
The following objects are masked from 'package:parallel':
+
+    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
+    clusterExport, clusterMap, parApply, parCapply, parLapply,
+    parLapplyLB, parRapply, parSapply, parSapplyLB
+ + +
The following objects are masked from 'package:stats':
+
+    IQR, mad, sd, var, xtabs
+ + +
The following objects are masked from 'package:base':
+
+    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
+    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
+    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
+    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
+    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
+    union, unique, unsplit, which.max, which.min
+ + +

+Attaching package: 'S4Vectors'
+ + +
The following object is masked from 'package:base':
+
+    expand.grid
+ + +
Loading required package: IRanges
+ + +
Loading required package: GenomicRanges
+ + +
Loading required package: GenomeInfoDb
+ + +
Loading required package: SummarizedExperiment
+ + +
Loading required package: MatrixGenerics
+ + +
Loading required package: matrixStats
+ + +

+Attaching package: 'MatrixGenerics'
+ + +
The following objects are masked from 'package:matrixStats':
+
+    colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
+    colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
+    colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
+    colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
+    colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
+    colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
+    colWeightedMeans, colWeightedMedians, colWeightedSds,
+    colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
+    rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
+    rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
+    rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
+    rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
+    rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
+    rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
+    rowWeightedSds, rowWeightedVars
+ + +
Loading required package: Biobase
+ + +
Welcome to Bioconductor
+
+    Vignettes contain introductory material; view with
+    'browseVignettes()'. To cite Bioconductor, see
+    'citation("Biobase")', and for packages 'citation("pkgname")'.
+ + +

+Attaching package: 'Biobase'
+ + +
The following object is masked from 'package:MatrixGenerics':
+
+    rowMedians
+ + +
The following objects are masked from 'package:matrixStats':
+
+    anyMissing, rowMedians
+ + +
# Then we'll be doing a bit of data wrangling with the Tidyverse
+library(tidyverse)
+ + +
── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
+ + +
✔ ggplot2 3.3.3     ✔ purrr   0.3.4
+✔ tibble  3.0.5     ✔ dplyr   1.0.3
+✔ tidyr   1.1.2     ✔ stringr 1.4.0
+✔ readr   1.4.0     ✔ forcats 0.5.0
+ + +
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
+✖ dplyr::collapse()   masks IRanges::collapse()
+✖ dplyr::combine()    masks Biobase::combine(), BiocGenerics::combine()
+✖ dplyr::count()      masks matrixStats::count()
+✖ dplyr::desc()       masks IRanges::desc()
+✖ tidyr::expand()     masks S4Vectors::expand()
+✖ dplyr::filter()     masks stats::filter()
+✖ dplyr::first()      masks S4Vectors::first()
+✖ dplyr::lag()        masks stats::lag()
+✖ ggplot2::Position() masks BiocGenerics::Position(), base::Position()
+✖ purrr::reduce()     masks GenomicRanges::reduce(), IRanges::reduce()
+✖ dplyr::rename()     masks S4Vectors::rename()
+✖ dplyr::slice()      masks IRanges::slice()
+ + +
# ComplexHeatmap is the package we'll use for making a heatmap
+# It will do the hierarchical clustering for us as well
+library(ComplexHeatmap)
+ + +
Loading required package: grid
+ + +
========================================
+ComplexHeatmap version 2.6.2
+Bioconductor page: http://bioconductor.org/packages/ComplexHeatmap/
+Github page: https://github.com/jokergoo/ComplexHeatmap
+Documentation: http://jokergoo.github.io/ComplexHeatmap-reference
+
+If you use it in published research, please cite:
+Gu, Z. Complex heatmaps reveal patterns and correlations in multidimensional 
+  genomic data. Bioinformatics 2016.
+
+This message can be suppressed by:
+  suppressPackageStartupMessages(library(ComplexHeatmap))
+========================================
+ + + +
+
+

Directories and files

+

We have stored the data we’ll use in this notebook in data/open-pbta.

+ + + +
data_dir <- file.path("data", "open-pbta")
+
+# We'll store the heatmap in plots/open-pbta - create directory if it doesn't exist yet
+plots_dir <- file.path("plots", "open-pbta")
+if (!dir.exists(plots_dir)) {
+  dir.create(plots_dir, recursive = TRUE)
+}
+ + + +
+

Input files

+ + + +
# The metadata describing the samples
+histologies_file <- file.path(data_dir, "pbta-histologies-subset.tsv")
+
+# The RNA-seq counts table
+rnaseq_file = file.path(data_dir, "pbta-rsem-expected_count-subset.rds")
+ + + +
+
+

Output files

+ + + +
heatmap_file <- file.path(plots_dir, 
+                          "common_histologies_high_variance_heatmap.png")
+ + + +
+
+
+
+

Read in and prepare data

+
+

Metadata

+

Let’s read in the metadata file file and take a look at the data.

+ + + +
histologies_df <- read_tsv(histologies_file)
+ + +

+── Column specification ────────────────────────────────────────────────────────
+cols(
+  .default = col_character(),
+  OS_days = col_double(),
+  age_last_update_days = col_double()
+)
+ℹ Use `spec()` for the full column specifications.
+ + + +

Use the chunk below to explore the metadata data frame.

+ + + +
View(histologies_df)
+ + + +

We’ll use the disease labels in a column called short_histology when we label the heatmap. Let’s count how many samples are assigned each short_histology label using the Tidyverse.

+ + + +
histology_count_df <- histologies_df %>%
+  # Count how many samples are in each short_histology and name the column 
+  # with that number n
+  count(short_histology) %>%
+  # Sort from largest number of samples to smallest number of samples in a 
+  # histology
+  arrange(desc(n))
+
+histology_count_df
+ +
+ +
+ + +
+
+

RNA-seq data

+

Read in the expression count matrix (stored as a data frame).

+ + + +
# Read in and examine the RNA-seq data
+rnaseq_exp <- read_rds(rnaseq_file)
+ + + +
+
+

Convert and round

+

The count data we have consists mostly of integers, but because of the estimation procedure that RSEM uses, some counts are fractional. DESeq2, which we will be using below expects all integers, so we will round all of the count data. This is easiest if we first convert from a data frame to a matrix.

+ + + +
rnaseq_mat <- rnaseq_exp %>%
+  # move gene_id to the rownames
+  tibble::column_to_rownames("gene_id") %>%
+  # convert to a matrix and round
+  as.matrix() %>%
+  round()
+ + + +
+
+

Variance Stabilizing Transformation

+

Raw counts are not usually suitable for the algorithms we use for clustering and heatmap display, so we will use the vst() function from the DESeq2 package to transform our data.

+

Since we are starting from a matrix, not a SummarizedExperiment as we did previously, we will need to provide the sample information ourselves. Just to be sure nothing is out of order, we will check that the identifiers for the sample information stored in histologies_df matches the columns of our matrix.

+ + + +
all.equal(histologies_df$Kids_First_Biospecimen_ID,
+          colnames(rnaseq_mat))
+ + +
[1] TRUE
+ + + +

Now we can make our matrix into a DESeq2 dataset, adding on the sample information from histologies_df. Unlike when we were performing differential expression analysis, we won’t provide an experimental design at this stage.

+ + + +
ddset <- DESeqDataSetFromMatrix(rnaseq_mat,
+                                colData = histologies_df,
+                                design = ~ 1) # don't store an experimental design
+ + +
converting counts to integer mode
+ + + +

We will again remove low count genes, as they are not likely to be informative.

+ + + +
genes_to_keep <- rowSums(counts(ddset)) >= 10
+ddset <- ddset[genes_to_keep, ]
+ + + +

Now we can apply the variance stabilizing transformation, saving the results in a new object.

+ + + +
# apply variance stabilizing transformation
+vst_data <- vst(ddset, blind = TRUE)
+ + + +

This object stores information about the transformation that was applied, but for now, we will only need the matrix of transformed data, which we can extract with assay().

+ + + +
# extract transformed data
+expr_mat <- assay(vst_data)
+ + + +

What are the dimensions of this transformed RNA-seq data matrix?

+ + + +
dim(expr_mat)
+ + +
[1] 48663   607
+ + + +

Almost 50k genes would be hard to visualize on a single heatmap! If we are making a heatmap to get an idea of the structure in our data, it can be helpful to subset to high variance genes. This is because the genes that don’t vary are not likely to contribute much to the overall patterns we are interested in.

+

First, we’ll calculate the variance for each gene using rowVars() from the matrixStats package and then take the genes in the top 10%.

+ + + +
# Calculate variance from the expression data 
+gene_variance <- matrixStats::rowVars(expr_mat)
+
+# Find the value that we'll use as a threshold to filter the top 10%
+variance_threshold <- quantile(gene_variance, 0.9)
+
+# Row indices of high variance genes
+high_variance_index <- which(gene_variance > variance_threshold)
+
+# What does a row index look like?
+head(high_variance_index)
+ + +
[1]  7 15 24 25 26 28
+ + + + + + +
# Get a matrix that is subset to just the high variance genes
+high_var_mat <- expr_mat[high_variance_index, ]
+ + + +
+
+
+

Heatmap

+
+

Annotation

+

First, we’ll set up the metadata that we want to use to label samples in the heatmap. In ComplexHeatmap terminology, this is called annotation, or HeatmapAnnotation, specifically.

+ + + +
sample_annotation_df <- histologies_df %>%
+  # Select only the columns that we'll use
+  select(Kids_First_Biospecimen_ID,
+         short_histology,
+         composition,
+         tumor_descriptor)
+
+# Let's examine these columns
+sample_annotation_df
+ +
+ +
+ + +

ComplexHeatmap is going to want the data frame we provide to have the sample identifiers as row names, so let’s set that up.

+ + + +
sample_annotation_df <- sample_annotation_df %>%
+  tibble::column_to_rownames("Kids_First_Biospecimen_ID")
+ + + +

To specify the colors in our annotation bar, we need to create a list of named vectors. The names of the list need to exactly match the column names in sample_annotation_df and the names in each vector need to exactly match the values in those columns.

+ + + +
# The Okabe Ito palette is recommended for those with color vision deficiencies  
+histology_colors <- palette.colors(palette = "Okabe-Ito")[2:5]
+# `palette.colors()` returns a named vector, which can cause trouble
+histology_colors <- unname(histology_colors)
+
+# annotation color list for ComplexHeatMap
+sample_annotation_colors <- list(
+  short_histology = c(
+    "LGAT" = histology_colors[[1]],
+    "Ependymoma" = histology_colors[[2]],
+    "HGAT" = histology_colors[[3]],
+    "Medulloblastoma" = histology_colors[[4]]
+  ),
+  composition = c(
+    "Solid Tissue" = "#A0A0A0",  # light grey
+    "Derived Cell Line" = "#000000"  # black
+  ),
+  tumor_descriptor = c(
+    "Initial CNS Tumor" = "#3333FF",
+    "Progressive" = "#FFFF99",
+    "Recurrence" = "#CCCCFF",
+    "Second Malignancy" = "#000033",
+    "Unavailable" = "#FFFFFF"  # white for missing data
+  )
+)
+ + + +

We need to create a special type of object with HeatmapAnnotation using the annotation data frame and the list of color vectors. We will also make the annotation labels a bit nicer to look at than the raw columns names.

+ + + +
column_annotation <- HeatmapAnnotation(
+  df = sample_annotation_df,
+  col = sample_annotation_colors,
+  annotation_label = c("Histology", "Composition", "Tumor descriptor")
+)
+ + + +
+
+

Values for display

+

We will z-score the expression values for display. This is sometimes called a standardized score. Some heatmap plotting packages will do this for you, but ComplexHeatmap does not. It’s calculated for each value in a row (gene) by subtracting the gene’s mean and dividing by the gene’s standard deviation. This will result in every row having a mean of 0 and a standard deviation of 1.

+ + + +
zscores_mat <- 
+  (high_var_mat - rowMeans(high_var_mat)) / matrixStats::rowSds(high_var_mat)
+ + + +

Since all genes have the same z-score variance (by definition!), if you’re filtering to high variance values, it’s important to do that prior to standardizing.

+
+
+

Heatmap itself!

+

Okay, now we’re ready to make a heatmap complete with annotation bars.

+ + + +
Heatmap(zscores_mat,
+        # The distance metric used for clustering the rows
+        # This is different from the default (Euclidean)
+        clustering_distance_rows = "pearson",
+        # Linkage method for row clustering
+        # This is different from the default (complete)
+        clustering_method_rows = "average",
+        # Distance metric for columns
+        clustering_distance_columns = "pearson",
+        # Linkage for columns
+        clustering_method_columns = "average",
+        show_row_names = FALSE,
+        show_column_names = FALSE,
+        # Add annotation bars to the top of the heatmap
+        top_annotation = column_annotation,
+        # This will be used as the label for the color bar
+        # of the cells of the heatmap itself
+        name = "z-score")
+ + +
The automatically generated colors map from the minus and plus 99^th of
+the absolute values in the matrix. There are outliers in the matrix
+whose patterns might be hidden by this color mapping. You can manually
+set the color to `col` argument.
+
+Use `suppressMessages()` to turn off this message.
+ + +
`use_raster` is automatically set to TRUE for a matrix with more than
+2000 rows. You can control `use_raster` argument by explicitly setting
+TRUE/FALSE to it.
+
+Set `ht_opt$message = FALSE` to turn off this message.
+ + +

+ + + +

ComplexHeatmap gives a few warning here, but nothing to be concerned about. One is complaining that the color scale may be truncated relative to our data. The other warns that our very large heatmap itself is being drawn at a lower resolution to speed things up.

+

We have to create the heatmap twice – once for display in the notebook and once to save to a PNG file. For the second drawing we will deal with the raster warning and produce a higher resolution output.

+ + + +
# Open PNG plot device
+png(filename = heatmap_file, 
+    width = 11, 
+    height = 7, 
+    units = "in", 
+    res = 300)
+# Heatmap, again!
+Heatmap(zscores_mat,
+        clustering_distance_rows = "pearson",
+        clustering_method_rows = "average",
+        clustering_distance_columns = "pearson",
+        clustering_method_columns = "average",
+        show_row_names = FALSE,
+        show_column_names = FALSE,
+        top_annotation = column_annotation,
+        name = "z-score",
+        use_raster = FALSE) # higher resolution for output (be careful with PDF output!)
+ + +
The automatically generated colors map from the minus and plus 99^th of
+the absolute values in the matrix. There are outliers in the matrix
+whose patterns might be hidden by this color mapping. You can manually
+set the color to `col` argument.
+
+Use `suppressMessages()` to turn off this message.
+ + +
# Shut down current graphics device
+dev.off()
+ + +
png 
+  2
+ + + +
+
+

Notes on clustering

+

You probably noticed that there were some arguments that we defined that were related to the clustering of samples and genes when we built the heatmaps. The clustering determines the arrangement of genes and samples in the heatmap, making it so genes or samples with similar expression patterns are grouped. If we didn’t do that, we would have a heatmap that looked basically like random static!

+

The kind of clustering we performed was hierarchical clustering. This is an agglomerative method of clustering - each sample starts in its own cluster and then at each step of the algorithm the two most similar clusters are joined until there’s only a single cluster left.

+

The arguments we chose for clustering_distance_* and clustering_method_* told ComplexHeatmap to use 1 minus the correlation between samples as the distance measure, and to average samples when they were merged into a group. This method is also known as UPGMA (unweighted pair group method with arithmetic mean).

+

The result of this clustering process is a tree (dendrogram), which is shown at the top for samples and to the left for genes. It is important to note that the linear arrangement of samples in the tree is somewhat arbitrary, so we have to be careful not to overinterpret it. We can rotate around any branch of the tree in our visualization with no change to the tree topology itself. In fact, the samples at opposite ends of the heatmap could be right next to one another with the right set of rotations!

+
+
+

PCA as an alternative to clustering

+

You probably noticed that the hierarchical clustering we performed didn’t perfectly separate out the different cancer subtypes we were looking at.

+

What would have happened if we had used PCA to visualize the relationships among samples? Would that have made it easier to separate disease types? Let’s try it!

+

We will use the VST transformed data here too, and the plotPCA() function from DESeq2.

+ + + +
# Use plotPCA, but return the data for custom plotting
+pca_df <- plotPCA(vst_data,
+                  ntop = 5000, # use the top 5000 genes by variance
+                  intgroup = "short_histology",
+                  returnData = TRUE) 
+
+
+ggplot(pca_df, aes(PC1, PC2, color = short_histology)) + 
+  geom_point() +
+  theme_bw() + 
+  scale_color_manual(
+    values = c("LGAT" = histology_colors[[1]],
+               "Ependymoma" = histology_colors[[2]],
+               "HGAT" = histology_colors[[3]],
+               "Medulloblastoma" = histology_colors[[4]])
+  ) + 
+  labs(color = "Histology")
+ + +

+ + + +

We can see that neither method perfectly separates the different disease types. With PCA, if we didn’t color the points, it might even be difficult to identify distinct clusters in the data at all by eye. (We could look at more than two PCs, which might help out, but we would need to use a different function for our calculation, since plotPCA() only returns the first two.)

+

For nice comparison of the relative advantages of these two methods (with a little mention of some further directions), we recommend blog post by Soneson.

+
+
+
+

Session Info

+ + + +
sessionInfo()
+ + +
R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+ [1] grid      parallel  stats4    stats     graphics  grDevices utils    
+ [8] datasets  methods   base     
+
+other attached packages:
+ [1] ComplexHeatmap_2.6.2        forcats_0.5.0              
+ [3] stringr_1.4.0               dplyr_1.0.3                
+ [5] purrr_0.3.4                 readr_1.4.0                
+ [7] tidyr_1.1.2                 tibble_3.0.5               
+ [9] ggplot2_3.3.3               tidyverse_1.3.0            
+[11] DESeq2_1.30.1               SummarizedExperiment_1.20.0
+[13] Biobase_2.50.0              MatrixGenerics_1.2.0       
+[15] matrixStats_0.57.0          GenomicRanges_1.42.0       
+[17] GenomeInfoDb_1.26.2         IRanges_2.24.1             
+[19] S4Vectors_0.28.1            BiocGenerics_0.36.0        
+[21] optparse_1.6.6             
+
+loaded via a namespace (and not attached):
+ [1] bitops_1.0-6           fs_1.5.0               lubridate_1.7.9.2     
+ [4] bit64_4.0.5            RColorBrewer_1.1-2     httr_1.4.2            
+ [7] tools_4.0.3            backports_1.2.1        R6_2.5.0              
+[10] DBI_1.1.1              colorspace_2.0-0       GetoptLong_1.0.5      
+[13] withr_2.4.0            tidyselect_1.1.0       bit_4.0.4             
+[16] compiler_4.0.3         cli_2.2.0              rvest_0.3.6           
+[19] Cairo_1.5-12.2         xml2_1.3.2             DelayedArray_0.16.2   
+[22] labeling_0.4.2         scales_1.1.1           genefilter_1.72.1     
+[25] digest_0.6.27          rmarkdown_2.6          XVector_0.30.0        
+[28] pkgconfig_2.0.3        htmltools_0.5.1.1      dbplyr_2.0.0          
+[31] GlobalOptions_0.1.2    rlang_0.4.10           readxl_1.3.1          
+[34] rstudioapi_0.13        RSQLite_2.2.3          farver_2.0.3          
+[37] shape_1.4.5            generics_0.1.0         jsonlite_1.7.2        
+[40] BiocParallel_1.24.1    RCurl_1.98-1.2         magrittr_2.0.1        
+[43] GenomeInfoDbData_1.2.4 Matrix_1.3-2           Rcpp_1.0.6            
+[46] munsell_0.5.0          fansi_0.4.2            lifecycle_0.2.0       
+[49] stringi_1.5.3          yaml_2.2.1             zlibbioc_1.36.0       
+[52] blob_1.2.1             crayon_1.3.4           lattice_0.20-41       
+[55] haven_2.3.1            splines_4.0.3          annotate_1.68.0       
+[58] circlize_0.4.12        hms_1.0.0              magick_2.6.0          
+[61] locfit_1.5-9.4         ps_1.5.0               knitr_1.30            
+[64] pillar_1.4.7           rjson_0.2.20           geneplotter_1.68.0    
+[67] reprex_0.3.0           XML_3.99-0.5           glue_1.4.2            
+[70] evaluate_0.14          modelr_0.1.8           png_0.1-7             
+[73] vctrs_0.3.6            cellranger_1.1.0       gtable_0.3.0          
+[76] getopt_1.20.3          clue_0.3-58            assertthat_0.2.1      
+[79] xfun_0.20              xtable_1.8-4           broom_0.7.3           
+[82] survival_3.2-7         AnnotationDbi_1.52.0   memoise_1.1.0         
+[85] cluster_2.1.0          ellipsis_0.3.1
+ + +
+ +
LS0tCnRpdGxlOiAiT3BlblBCVEE6IENyZWF0ZSBhIGhlYXRtYXAiCmF1dGhvcjogQ0NETCBmb3IgQUxTRgpkYXRlOiAyMDIxCm91dHB1dDogICAKICBodG1sX25vdGVib29rOiAKICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OiB0cnVlCi0tLQoKCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86CgotIExvYWQgdGFidWxhciBleHByZXNzaW9uIGRhdGEgYW5kIHByZXBhcmUgaXQgZm9yIHVzZSB3aXRoIGBERVNlcTJgCi0gQ3JlYXRlIGFuIGFubm90YXRlZCBoZWF0bWFwIG9mIGdlbmUgZXhwcmVzc2lvbiB2YXJpYXRpb24gdXNpbmcgdGhlIGBDb21wbGV4SGVhdG1hcGAgcGFja2FnZQotIEN1c3RvbWl6ZSBhIFBDQSBwbG90IGFuZCBjb21wYXJlIHRoZSBvdXRwdXQgd2l0aCBoaWVyYXJjaGljYWwgY2x1c3RlcmluZyBvZiB0aGUgc2FtZSBkYXRhIAoKLS0tCgpJbiB0aGlzIG5vdGVib29rLCB3ZSBjbHVzdGVyIFJOQS1zZXEgZGF0YSBmcm9tIHRoZSBPcGVuIFBlZGlhdHJpYyBCcmFpbiBUdW1vciBBdGxhcyAoT3BlblBCVEEpIHByb2plY3QgYW5kIGNyZWF0ZSBhIGhlYXRtYXAuCk9wZW5QQlRBIGlzIGEgY29sbGFib3JhdGl2ZSBwcm9qZWN0IG9yZ2FuaXplZCBieSB0aGUgQ0NETCBhbmQgdGhlIENlbnRlciBmb3IgRGF0YS1Ecml2ZW4gRGlzY292ZXJ5IGluIEJpb21lZGljaW5lIChEM2IpIGF0IHRoZSBDaGlsZHJlbidzIEhvc3BpdGFsIG9mIFBoaWxhZGVscGhpYSBjb25kdWN0ZWQgb3Blbmx5IG9uIEdpdEh1Yi4KCllvdSBjYW4gcmVhZCBtb3JlIGFib3V0IHRoZSBwcm9qZWN0IFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vYWxleHNsZW1vbmFkZS9vcGVucGJ0YS1hbmFseXNpcy8jb3BlbnBidGEtYW5hbHlzaXMpLgoKV2UndmUgZG93bmxvYWRlZCBzb21lIG9mIHRoZSBwdWJsaWNseSBhdmFpbGFibGUgZXhwcmVzc2lvbiBkYXRhIGZyb20gdGhlIHByb2plY3QgYW5kIHNlbGVjdGVkIGEgc3Vic2V0IHdpdGggdGhlIG1vc3QgY29tbW9uIGRpc2Vhc2UgdHlwZXMgZm9yIGFuYWx5c2lzIGhlcmUuCgpXZSdsbCB1c2UgYSBwYWNrYWdlIGNhbGxlZCBbYENvbXBsZXhIZWF0bWFwYF0oaHR0cHM6Ly93d3cuYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9yZWxlYXNlL2Jpb2MvaHRtbC9Db21wbGV4SGVhdG1hcC5odG1sKSB0byBtYWtlIG91ciBoZWF0bWFwLgpUaGlzIHBhY2thZ2UgYWxsb3dzIHVzIHRvIGFubm90YXRlIG91ciBoZWF0bWFwIHdpdGggc2FtcGxlIGluZm9ybWF0aW9uLCBhbmQgd2lsbCBhbHNvIHBlcmZvcm0gY2x1c3RlcmluZyBhcyBwYXJ0IG9mIGdlbmVyYXRpbmcgdGhlIGhlYXRtYXAuCkl0IGlzIGhpZ2hseSBmbGV4aWJsZSBhbmQgb3BpbmlvbmF0ZWQgLSB0aGUgZGF0YSBzdHJ1Y3R1cmVzIHdlIHBhc3MgYENvbXBsZXhIZWF0bWFwYCBmdW5jdGlvbnMgb2Z0ZW4gbmVlZCB0byBiZSBfanVzdCByaWdodF8uClNlZSB0aGUgW2BDb21wbGV4SGVhdG1hcGAgQ29tcGxldGUgUmVmZXJlbmNlXShodHRwczovL2pva2VyZ29vLmdpdGh1Yi5pby9Db21wbGV4SGVhdG1hcC1yZWZlcmVuY2UvYm9vay8pIGZvciBtb3JlIGluZm9ybWF0aW9uLgoKIyMgU2V0IHVwCgojIyMgTGlicmFyaWVzCgpgYGB7ciBsb2FkX2xpYnJhcmllc30KIyBXZSB3aWxsIG1hbmlwdWxhdGUgUk5BU2VxIGRhdGEgd2l0aCBERVNlcTIgYXQgdGhlIHN0YXJ0CmxpYnJhcnkoREVTZXEyKQoKIyBUaGVuIHdlJ2xsIGJlIGRvaW5nIGEgYml0IG9mIGRhdGEgd3JhbmdsaW5nIHdpdGggdGhlIFRpZHl2ZXJzZQpsaWJyYXJ5KHRpZHl2ZXJzZSkKCiMgQ29tcGxleEhlYXRtYXAgaXMgdGhlIHBhY2thZ2Ugd2UnbGwgdXNlIGZvciBtYWtpbmcgYSBoZWF0bWFwCiMgSXQgd2lsbCBkbyB0aGUgaGllcmFyY2hpY2FsIGNsdXN0ZXJpbmcgZm9yIHVzIGFzIHdlbGwKbGlicmFyeShDb21wbGV4SGVhdG1hcCkKYGBgCgojIyMgRGlyZWN0b3JpZXMgYW5kIGZpbGVzCgpXZSBoYXZlIHN0b3JlZCB0aGUgZGF0YSB3ZSdsbCB1c2UgaW4gdGhpcyBub3RlYm9vayBpbiBgZGF0YS9vcGVuLXBidGFgLiAKCmBgYHtyIGRpcmVjdG9yaWVzLCBsaXZlID0gVFJVRX0KZGF0YV9kaXIgPC0gZmlsZS5wYXRoKCJkYXRhIiwgIm9wZW4tcGJ0YSIpCgojIFdlJ2xsIHN0b3JlIHRoZSBoZWF0bWFwIGluIHBsb3RzL29wZW4tcGJ0YSAtIGNyZWF0ZSBkaXJlY3RvcnkgaWYgaXQgZG9lc24ndCBleGlzdCB5ZXQKcGxvdHNfZGlyIDwtIGZpbGUucGF0aCgicGxvdHMiLCAib3Blbi1wYnRhIikKaWYgKCFkaXIuZXhpc3RzKHBsb3RzX2RpcikpIHsKICBkaXIuY3JlYXRlKHBsb3RzX2RpciwgcmVjdXJzaXZlID0gVFJVRSkKfQpgYGAKCiMjIyMgSW5wdXQgZmlsZXMKCmBgYHtyIGlucHV0X2ZpbGVzfQojIFRoZSBtZXRhZGF0YSBkZXNjcmliaW5nIHRoZSBzYW1wbGVzCmhpc3RvbG9naWVzX2ZpbGUgPC0gZmlsZS5wYXRoKGRhdGFfZGlyLCAicGJ0YS1oaXN0b2xvZ2llcy1zdWJzZXQudHN2IikKCiMgVGhlIFJOQS1zZXEgY291bnRzIHRhYmxlCnJuYXNlcV9maWxlID0gZmlsZS5wYXRoKGRhdGFfZGlyLCAicGJ0YS1yc2VtLWV4cGVjdGVkX2NvdW50LXN1YnNldC5yZHMiKQpgYGAKCiMjIyMgT3V0cHV0IGZpbGVzCgpgYGB7ciBwbmdfb3V0cHV0LCBsaXZlID0gVFJVRX0KaGVhdG1hcF9maWxlIDwtIGZpbGUucGF0aChwbG90c19kaXIsIAogICAgICAgICAgICAgICAgICAgICAgICAgICJjb21tb25faGlzdG9sb2dpZXNfaGlnaF92YXJpYW5jZV9oZWF0bWFwLnBuZyIpCmBgYAoKIyMgUmVhZCBpbiBhbmQgcHJlcGFyZSBkYXRhCgojIyMgTWV0YWRhdGEgCgpMZXQncyByZWFkIGluIHRoZSBtZXRhZGF0YSBmaWxlIGZpbGUgYW5kIHRha2UgYSBsb29rIGF0IHRoZSBkYXRhLgoKYGBge3IgcmVhZF9yZHMsIGxpdmUgPSBUUlVFfQpoaXN0b2xvZ2llc19kZiA8LSByZWFkX3RzdihoaXN0b2xvZ2llc19maWxlKSAgCmBgYAoKVXNlIHRoZSBjaHVuayBiZWxvdyB0byBleHBsb3JlIHRoZSBtZXRhZGF0YSBkYXRhIGZyYW1lLgoKYGBge3IgZXhwbG9yZV9oaXN0b2xvZ2llc19kZiwgbGl2ZSA9IFRSVUUsIGV2YWwgPSBGQUxTRX0KVmlldyhoaXN0b2xvZ2llc19kZikKYGBgCgpXZSdsbCB1c2UgdGhlIGRpc2Vhc2UgbGFiZWxzIGluIGEgY29sdW1uIGNhbGxlZCBgc2hvcnRfaGlzdG9sb2d5YCB3aGVuIHdlIGxhYmVsIHRoZSBoZWF0bWFwLgpMZXQncyBjb3VudCBob3cgbWFueSBzYW1wbGVzIGFyZSBhc3NpZ25lZCBlYWNoIGBzaG9ydF9oaXN0b2xvZ3lgIGxhYmVsIHVzaW5nIHRoZSBUaWR5dmVyc2UuCgpgYGB7ciBjb3VudF9oaXN0b2xvZ2llc30KaGlzdG9sb2d5X2NvdW50X2RmIDwtIGhpc3RvbG9naWVzX2RmICU+JQogICMgQ291bnQgaG93IG1hbnkgc2FtcGxlcyBhcmUgaW4gZWFjaCBzaG9ydF9oaXN0b2xvZ3kgYW5kIG5hbWUgdGhlIGNvbHVtbiAKICAjIHdpdGggdGhhdCBudW1iZXIgbgogIGNvdW50KHNob3J0X2hpc3RvbG9neSkgJT4lCiAgIyBTb3J0IGZyb20gbGFyZ2VzdCBudW1iZXIgb2Ygc2FtcGxlcyB0byBzbWFsbGVzdCBudW1iZXIgb2Ygc2FtcGxlcyBpbiBhIAogICMgaGlzdG9sb2d5CiAgYXJyYW5nZShkZXNjKG4pKQoKaGlzdG9sb2d5X2NvdW50X2RmCmBgYAoKIyMjIFJOQS1zZXEgZGF0YQoKUmVhZCBpbiB0aGUgZXhwcmVzc2lvbiBjb3VudCBtYXRyaXggKHN0b3JlZCBhcyBhIGRhdGEgZnJhbWUpLgoKYGBge3IgcmVhZF9pbl9ybmFzZXEsIGxpdmUgPSBUUlVFfQojIFJlYWQgaW4gYW5kIGV4YW1pbmUgdGhlIFJOQS1zZXEgZGF0YQpybmFzZXFfZXhwIDwtIHJlYWRfcmRzKHJuYXNlcV9maWxlKQpgYGAKCiMjIyBDb252ZXJ0IGFuZCByb3VuZAoKVGhlIGNvdW50IGRhdGEgd2UgaGF2ZSBjb25zaXN0cyBtb3N0bHkgb2YgaW50ZWdlcnMsIGJ1dCBiZWNhdXNlIG9mIHRoZSBlc3RpbWF0aW9uIHByb2NlZHVyZSB0aGF0IFJTRU0gdXNlcywgc29tZSBjb3VudHMgYXJlIGZyYWN0aW9uYWwuIApgREVTZXEyYCwgd2hpY2ggd2Ugd2lsbCBiZSB1c2luZyBiZWxvdyBleHBlY3RzIGFsbCBpbnRlZ2Vycywgc28gd2Ugd2lsbCByb3VuZCBhbGwgb2YgdGhlIGNvdW50IGRhdGEuIApUaGlzIGlzIGVhc2llc3QgaWYgd2UgZmlyc3QgY29udmVydCBmcm9tIGEgZGF0YSBmcmFtZSB0byBhIG1hdHJpeC4KCmBgYHtyIGNvbnZlcnRfcm91bmQsIGxpdmUgPSBUUlVFfQpybmFzZXFfbWF0IDwtIHJuYXNlcV9leHAgJT4lCiAgIyBtb3ZlIGdlbmVfaWQgdG8gdGhlIHJvd25hbWVzCiAgdGliYmxlOjpjb2x1bW5fdG9fcm93bmFtZXMoImdlbmVfaWQiKSAlPiUKICAjIGNvbnZlcnQgdG8gYSBtYXRyaXggYW5kIHJvdW5kCiAgYXMubWF0cml4KCkgJT4lCiAgcm91bmQoKQpgYGAKCiMjIyBWYXJpYW5jZSBTdGFiaWxpemluZyBUcmFuc2Zvcm1hdGlvbgoKUmF3IGNvdW50cyBhcmUgbm90IHVzdWFsbHkgc3VpdGFibGUgZm9yIHRoZSBhbGdvcml0aG1zIHdlIHVzZSBmb3IgY2x1c3RlcmluZyBhbmQgaGVhdG1hcCBkaXNwbGF5LCBzbyB3ZSB3aWxsIHVzZSB0aGUgYHZzdCgpYCBmdW5jdGlvbiBmcm9tIHRoZSBgREVTZXEyYCBwYWNrYWdlIHRvIHRyYW5zZm9ybSBvdXIgZGF0YS4KClNpbmNlIHdlIGFyZSBzdGFydGluZyBmcm9tIGEgbWF0cml4LCBub3QgYSBgU3VtbWFyaXplZEV4cGVyaW1lbnRgIGFzIHdlIGRpZCBwcmV2aW91c2x5LCB3ZSB3aWxsIG5lZWQgdG8gcHJvdmlkZSB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIG91cnNlbHZlcy4KSnVzdCB0byBiZSBzdXJlIG5vdGhpbmcgaXMgb3V0IG9mIG9yZGVyLCB3ZSB3aWxsIGNoZWNrIHRoYXQgdGhlIGlkZW50aWZpZXJzIGZvciB0aGUgc2FtcGxlIGluZm9ybWF0aW9uIHN0b3JlZCBpbiBgaGlzdG9sb2dpZXNfZGZgIG1hdGNoZXMgdGhlIGNvbHVtbnMgb2Ygb3VyIG1hdHJpeC4KCmBgYHtyIGNoZWNrLW9yZGVyfQphbGwuZXF1YWwoaGlzdG9sb2dpZXNfZGYkS2lkc19GaXJzdF9CaW9zcGVjaW1lbl9JRCwKICAgICAgICAgIGNvbG5hbWVzKHJuYXNlcV9tYXQpKQpgYGAKCk5vdyB3ZSBjYW4gbWFrZSBvdXIgbWF0cml4IGludG8gYSBgREVTZXEyYCBkYXRhc2V0LCBhZGRpbmcgb24gdGhlIHNhbXBsZSBpbmZvcm1hdGlvbiBmcm9tIGBoaXN0b2xvZ2llc19kZmAuIApVbmxpa2Ugd2hlbiB3ZSB3ZXJlIHBlcmZvcm1pbmcgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMsIHdlIHdvbid0IHByb3ZpZGUgYW4gZXhwZXJpbWVudGFsIGRlc2lnbiBhdCB0aGlzIHN0YWdlLgoKYGBge3IgbWFrZS1ERVNFcX0KZGRzZXQgPC0gREVTZXFEYXRhU2V0RnJvbU1hdHJpeChybmFzZXFfbWF0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbERhdGEgPSBoaXN0b2xvZ2llc19kZiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkZXNpZ24gPSB+IDEpICMgZG9uJ3Qgc3RvcmUgYW4gZXhwZXJpbWVudGFsIGRlc2lnbgpgYGAKCldlIHdpbGwgYWdhaW4gcmVtb3ZlIGxvdyBjb3VudCBnZW5lcywgYXMgdGhleSBhcmUgbm90IGxpa2VseSB0byBiZSBpbmZvcm1hdGl2ZS4gCgpgYGB7ciB0cmltfQpnZW5lc190b19rZWVwIDwtIHJvd1N1bXMoY291bnRzKGRkc2V0KSkgPj0gMTAKZGRzZXQgPC0gZGRzZXRbZ2VuZXNfdG9fa2VlcCwgXQpgYGAKCk5vdyB3ZSBjYW4gYXBwbHkgdGhlIHZhcmlhbmNlIHN0YWJpbGl6aW5nIHRyYW5zZm9ybWF0aW9uLCBzYXZpbmcgdGhlIHJlc3VsdHMgaW4gYSBuZXcgb2JqZWN0LgoKYGBge3IgdnN0LCBsaXZlPVRSVUV9CiMgYXBwbHkgdmFyaWFuY2Ugc3RhYmlsaXppbmcgdHJhbnNmb3JtYXRpb24KdnN0X2RhdGEgPC0gdnN0KGRkc2V0LCBibGluZCA9IFRSVUUpCmBgYAoKVGhpcyBvYmplY3Qgc3RvcmVzIGluZm9ybWF0aW9uIGFib3V0IHRoZSB0cmFuc2Zvcm1hdGlvbiB0aGF0IHdhcyBhcHBsaWVkLCBidXQgZm9yIG5vdywgd2Ugd2lsbCBvbmx5IG5lZWQgdGhlIG1hdHJpeCBvZiB0cmFuc2Zvcm1lZCBkYXRhLCB3aGljaCB3ZSBjYW4gZXh0cmFjdCB3aXRoIGBhc3NheSgpYC4KCmBgYHtyIGV4dHJhY3RfZXhwciwgbGl2ZSA9IFRSVUV9CiMgZXh0cmFjdCB0cmFuc2Zvcm1lZCBkYXRhCmV4cHJfbWF0IDwtIGFzc2F5KHZzdF9kYXRhKQpgYGAKCldoYXQgYXJlIHRoZSBkaW1lbnNpb25zIG9mIHRoaXMgdHJhbnNmb3JtZWQgUk5BLXNlcSBkYXRhIG1hdHJpeD8KCmBgYHtyIHJuYXNlcV9kaW0sIGxpdmUgPSBUUlVFfQpkaW0oZXhwcl9tYXQpCmBgYAoKQWxtb3N0IDUwayBnZW5lcyB3b3VsZCBiZSBoYXJkIHRvIHZpc3VhbGl6ZSBvbiBhIHNpbmdsZSBoZWF0bWFwIQpJZiB3ZSBhcmUgbWFraW5nIGEgaGVhdG1hcCB0byBnZXQgYW4gaWRlYSBvZiB0aGUgc3RydWN0dXJlIGluIG91ciBkYXRhLCBpdCBjYW4gYmUgaGVscGZ1bCB0byBzdWJzZXQgdG8gaGlnaCB2YXJpYW5jZSBnZW5lcy4KVGhpcyBpcyBiZWNhdXNlIHRoZSBnZW5lcyB0aGF0ICpkb24ndCogdmFyeSBhcmUgbm90IGxpa2VseSB0byBjb250cmlidXRlIG11Y2ggdG8gdGhlIG92ZXJhbGwgcGF0dGVybnMgd2UgYXJlIGludGVyZXN0ZWQgaW4uCgpGaXJzdCwgd2UnbGwgY2FsY3VsYXRlIHRoZSB2YXJpYW5jZSBmb3IgZWFjaCBnZW5lIHVzaW5nIGByb3dWYXJzKClgIGZyb20gdGhlIGBtYXRyaXhTdGF0c2AgcGFja2FnZSBhbmQgdGhlbiB0YWtlIHRoZSBnZW5lcyBpbiB0aGUgdG9wIDEwJS4KCmBgYHtyIGhpZ2hfdmFyX2dlbmVzfQojIENhbGN1bGF0ZSB2YXJpYW5jZSBmcm9tIHRoZSBleHByZXNzaW9uIGRhdGEgCmdlbmVfdmFyaWFuY2UgPC0gbWF0cml4U3RhdHM6OnJvd1ZhcnMoZXhwcl9tYXQpCgojIEZpbmQgdGhlIHZhbHVlIHRoYXQgd2UnbGwgdXNlIGFzIGEgdGhyZXNob2xkIHRvIGZpbHRlciB0aGUgdG9wIDEwJQp2YXJpYW5jZV90aHJlc2hvbGQgPC0gcXVhbnRpbGUoZ2VuZV92YXJpYW5jZSwgMC45KQoKIyBSb3cgaW5kaWNlcyBvZiBoaWdoIHZhcmlhbmNlIGdlbmVzCmhpZ2hfdmFyaWFuY2VfaW5kZXggPC0gd2hpY2goZ2VuZV92YXJpYW5jZSA+IHZhcmlhbmNlX3RocmVzaG9sZCkKCiMgV2hhdCBkb2VzIGEgcm93IGluZGV4IGxvb2sgbGlrZT8KaGVhZChoaWdoX3ZhcmlhbmNlX2luZGV4KQpgYGAKCmBgYHtyIGhpZ2hfdmFyX21hdCwgbGl2ZSA9IFRSVUV9CiMgR2V0IGEgbWF0cml4IHRoYXQgaXMgc3Vic2V0IHRvIGp1c3QgdGhlIGhpZ2ggdmFyaWFuY2UgZ2VuZXMKaGlnaF92YXJfbWF0IDwtIGV4cHJfbWF0W2hpZ2hfdmFyaWFuY2VfaW5kZXgsIF0KYGBgCgojIyBIZWF0bWFwCgojIyMgQW5ub3RhdGlvbgoKRmlyc3QsIHdlJ2xsIHNldCB1cCB0aGUgbWV0YWRhdGEgdGhhdCB3ZSB3YW50IHRvIHVzZSB0byBsYWJlbCBzYW1wbGVzIGluIHRoZSBoZWF0bWFwLgpJbiBgQ29tcGxleEhlYXRtYXBgIHRlcm1pbm9sb2d5LCB0aGlzIGlzIGNhbGxlZCBhbm5vdGF0aW9uLCBvciBgSGVhdG1hcEFubm90YXRpb25gLCBzcGVjaWZpY2FsbHkuCgpgYGB7ciBzYW1wbGVfYW5ub3RhdGlvbl9kZn0Kc2FtcGxlX2Fubm90YXRpb25fZGYgPC0gaGlzdG9sb2dpZXNfZGYgJT4lCiAgIyBTZWxlY3Qgb25seSB0aGUgY29sdW1ucyB0aGF0IHdlJ2xsIHVzZQogIHNlbGVjdChLaWRzX0ZpcnN0X0Jpb3NwZWNpbWVuX0lELAogICAgICAgICBzaG9ydF9oaXN0b2xvZ3ksCiAgICAgICAgIGNvbXBvc2l0aW9uLAogICAgICAgICB0dW1vcl9kZXNjcmlwdG9yKQoKIyBMZXQncyBleGFtaW5lIHRoZXNlIGNvbHVtbnMKc2FtcGxlX2Fubm90YXRpb25fZGYKYGBgCmBDb21wbGV4SGVhdG1hcGAgaXMgZ29pbmcgdG8gd2FudCB0aGUgZGF0YSBmcmFtZSB3ZSBwcm92aWRlIHRvIGhhdmUgdGhlIHNhbXBsZSBpZGVudGlmaWVycyBhcyByb3cgbmFtZXMsIHNvIGxldCdzIHNldCB0aGF0IHVwLiAKCmBgYHtyIGFubm90YXRpb25fYXNfZGYsIGxpdmUgPSBUUlVFfQpzYW1wbGVfYW5ub3RhdGlvbl9kZiA8LSBzYW1wbGVfYW5ub3RhdGlvbl9kZiAlPiUKICB0aWJibGU6OmNvbHVtbl90b19yb3duYW1lcygiS2lkc19GaXJzdF9CaW9zcGVjaW1lbl9JRCIpCmBgYAoKVG8gc3BlY2lmeSB0aGUgY29sb3JzIGluIG91ciBhbm5vdGF0aW9uIGJhciwgd2UgbmVlZCB0byBjcmVhdGUgYSBsaXN0IG9mIG5hbWVkIHZlY3RvcnMuClRoZSBuYW1lcyBvZiB0aGUgbGlzdCBuZWVkIHRvICpleGFjdGx5KiBtYXRjaCB0aGUgY29sdW1uIG5hbWVzIGluIGBzYW1wbGVfYW5ub3RhdGlvbl9kZmAgYW5kIHRoZSBuYW1lcyBpbiBlYWNoIHZlY3RvciBuZWVkIHRvICpleGFjdGx5KiBtYXRjaCB0aGUgdmFsdWVzIGluIHRob3NlIGNvbHVtbnMuCgpgYGB7ciBjb2xvcl9wYWxldHRlc30KIyBUaGUgT2thYmUgSXRvIHBhbGV0dGUgaXMgcmVjb21tZW5kZWQgZm9yIHRob3NlIHdpdGggY29sb3IgdmlzaW9uIGRlZmljaWVuY2llcyAgCmhpc3RvbG9neV9jb2xvcnMgPC0gcGFsZXR0ZS5jb2xvcnMocGFsZXR0ZSA9ICJPa2FiZS1JdG8iKVsyOjVdCiMgYHBhbGV0dGUuY29sb3JzKClgIHJldHVybnMgYSBuYW1lZCB2ZWN0b3IsIHdoaWNoIGNhbiBjYXVzZSB0cm91YmxlCmhpc3RvbG9neV9jb2xvcnMgPC0gdW5uYW1lKGhpc3RvbG9neV9jb2xvcnMpCgojIGFubm90YXRpb24gY29sb3IgbGlzdCBmb3IgQ29tcGxleEhlYXRNYXAKc2FtcGxlX2Fubm90YXRpb25fY29sb3JzIDwtIGxpc3QoCiAgc2hvcnRfaGlzdG9sb2d5ID0gYygKICAgICJMR0FUIiA9IGhpc3RvbG9neV9jb2xvcnNbWzFdXSwKICAgICJFcGVuZHltb21hIiA9IGhpc3RvbG9neV9jb2xvcnNbWzJdXSwKICAgICJIR0FUIiA9IGhpc3RvbG9neV9jb2xvcnNbWzNdXSwKICAgICJNZWR1bGxvYmxhc3RvbWEiID0gaGlzdG9sb2d5X2NvbG9yc1tbNF1dCiAgKSwKICBjb21wb3NpdGlvbiA9IGMoCiAgICAiU29saWQgVGlzc3VlIiA9ICIjQTBBMEEwIiwgICMgbGlnaHQgZ3JleQogICAgIkRlcml2ZWQgQ2VsbCBMaW5lIiA9ICIjMDAwMDAwIiAgIyBibGFjawogICksCiAgdHVtb3JfZGVzY3JpcHRvciA9IGMoCiAgICAiSW5pdGlhbCBDTlMgVHVtb3IiID0gIiMzMzMzRkYiLAogICAgIlByb2dyZXNzaXZlIiA9ICIjRkZGRjk5IiwKICAgICJSZWN1cnJlbmNlIiA9ICIjQ0NDQ0ZGIiwKICAgICJTZWNvbmQgTWFsaWduYW5jeSIgPSAiIzAwMDAzMyIsCiAgICAiVW5hdmFpbGFibGUiID0gIiNGRkZGRkYiICAjIHdoaXRlIGZvciBtaXNzaW5nIGRhdGEKICApCikKYGBgCgpXZSBuZWVkIHRvIGNyZWF0ZSBhIHNwZWNpYWwgdHlwZSBvZiBvYmplY3Qgd2l0aCBgSGVhdG1hcEFubm90YXRpb25gIHVzaW5nIHRoZSBhbm5vdGF0aW9uIGRhdGEgZnJhbWUgYW5kIHRoZSBsaXN0IG9mIGNvbG9yIHZlY3RvcnMuCldlIHdpbGwgYWxzbyBtYWtlIHRoZSBhbm5vdGF0aW9uIGxhYmVscyBhIGJpdCBuaWNlciB0byBsb29rIGF0IHRoYW4gdGhlIHJhdyBjb2x1bW5zIG5hbWVzLgoKYGBge3IgY3JlYXRlX2Fubm90YXRpb24sIGxpdmUgPSBUUlVFfQpjb2x1bW5fYW5ub3RhdGlvbiA8LSBIZWF0bWFwQW5ub3RhdGlvbigKICBkZiA9IHNhbXBsZV9hbm5vdGF0aW9uX2RmLAogIGNvbCA9IHNhbXBsZV9hbm5vdGF0aW9uX2NvbG9ycywKICBhbm5vdGF0aW9uX2xhYmVsID0gYygiSGlzdG9sb2d5IiwgIkNvbXBvc2l0aW9uIiwgIlR1bW9yIGRlc2NyaXB0b3IiKQopCmBgYAoKIyMjIFZhbHVlcyBmb3IgZGlzcGxheQoKV2Ugd2lsbCB6LXNjb3JlIHRoZSBleHByZXNzaW9uIHZhbHVlcyBmb3IgZGlzcGxheS4KVGhpcyBpcyBzb21ldGltZXMgY2FsbGVkIGEgc3RhbmRhcmRpemVkIHNjb3JlLgpTb21lIGhlYXRtYXAgcGxvdHRpbmcgcGFja2FnZXMgd2lsbCBkbyB0aGlzIGZvciB5b3UsIGJ1dCBgQ29tcGxleEhlYXRtYXBgIGRvZXMgbm90LgpJdCdzIGNhbGN1bGF0ZWQgZm9yIGVhY2ggdmFsdWUgaW4gYSByb3cgKGdlbmUpIGJ5IHN1YnRyYWN0aW5nIHRoZSBnZW5lJ3MgbWVhbiBhbmQgZGl2aWRpbmcgYnkgdGhlIGdlbmUncyBzdGFuZGFyZCBkZXZpYXRpb24uClRoaXMgd2lsbCByZXN1bHQgaW4gZXZlcnkgcm93IGhhdmluZyBhIG1lYW4gb2YgMCBhbmQgYSBzdGFuZGFyZCBkZXZpYXRpb24gb2YgMS4KCmBgYHtyIHpzY29yZX0KenNjb3Jlc19tYXQgPC0gCiAgKGhpZ2hfdmFyX21hdCAtIHJvd01lYW5zKGhpZ2hfdmFyX21hdCkpIC8gbWF0cml4U3RhdHM6OnJvd1NkcyhoaWdoX3Zhcl9tYXQpCmBgYAoKU2luY2UgYWxsIGdlbmVzIGhhdmUgdGhlIHNhbWUgei1zY29yZSB2YXJpYW5jZSAoYnkgZGVmaW5pdGlvbiEpLCBpZiB5b3UncmUgZmlsdGVyaW5nIHRvIGhpZ2ggdmFyaWFuY2UgdmFsdWVzLCBpdCdzIGltcG9ydGFudCB0byBkbyB0aGF0IF9wcmlvcl8gdG8gc3RhbmRhcmRpemluZy4KCiMjIyBIZWF0bWFwIGl0c2VsZiEKCk9rYXksIG5vdyB3ZSdyZSByZWFkeSB0byBtYWtlIGEgaGVhdG1hcCBjb21wbGV0ZSB3aXRoIGFubm90YXRpb24gYmFycy4KCmBgYHtyIG1ha2VfaGVhdG1hcH0KSGVhdG1hcCh6c2NvcmVzX21hdCwKICAgICAgICAjIFRoZSBkaXN0YW5jZSBtZXRyaWMgdXNlZCBmb3IgY2x1c3RlcmluZyB0aGUgcm93cwogICAgICAgICMgVGhpcyBpcyBkaWZmZXJlbnQgZnJvbSB0aGUgZGVmYXVsdCAoRXVjbGlkZWFuKQogICAgICAgIGNsdXN0ZXJpbmdfZGlzdGFuY2Vfcm93cyA9ICJwZWFyc29uIiwKICAgICAgICAjIExpbmthZ2UgbWV0aG9kIGZvciByb3cgY2x1c3RlcmluZwogICAgICAgICMgVGhpcyBpcyBkaWZmZXJlbnQgZnJvbSB0aGUgZGVmYXVsdCAoY29tcGxldGUpCiAgICAgICAgY2x1c3RlcmluZ19tZXRob2Rfcm93cyA9ICJhdmVyYWdlIiwKICAgICAgICAjIERpc3RhbmNlIG1ldHJpYyBmb3IgY29sdW1ucwogICAgICAgIGNsdXN0ZXJpbmdfZGlzdGFuY2VfY29sdW1ucyA9ICJwZWFyc29uIiwKICAgICAgICAjIExpbmthZ2UgZm9yIGNvbHVtbnMKICAgICAgICBjbHVzdGVyaW5nX21ldGhvZF9jb2x1bW5zID0gImF2ZXJhZ2UiLAogICAgICAgIHNob3dfcm93X25hbWVzID0gRkFMU0UsCiAgICAgICAgc2hvd19jb2x1bW5fbmFtZXMgPSBGQUxTRSwKICAgICAgICAjIEFkZCBhbm5vdGF0aW9uIGJhcnMgdG8gdGhlIHRvcCBvZiB0aGUgaGVhdG1hcAogICAgICAgIHRvcF9hbm5vdGF0aW9uID0gY29sdW1uX2Fubm90YXRpb24sCiAgICAgICAgIyBUaGlzIHdpbGwgYmUgdXNlZCBhcyB0aGUgbGFiZWwgZm9yIHRoZSBjb2xvciBiYXIKICAgICAgICAjIG9mIHRoZSBjZWxscyBvZiB0aGUgaGVhdG1hcCBpdHNlbGYKICAgICAgICBuYW1lID0gInotc2NvcmUiKQpgYGAKCmBDb21wbGV4SGVhdG1hcGAgZ2l2ZXMgYSBmZXcgd2FybmluZyBoZXJlLCBidXQgbm90aGluZyB0byBiZSBjb25jZXJuZWQgYWJvdXQuIApPbmUgaXMgY29tcGxhaW5pbmcgdGhhdCB0aGUgY29sb3Igc2NhbGUgbWF5IGJlIHRydW5jYXRlZCByZWxhdGl2ZSB0byBvdXIgZGF0YS4KVGhlIG90aGVyIHdhcm5zIHRoYXQgb3VyIHZlcnkgbGFyZ2UgaGVhdG1hcCBpdHNlbGYgaXMgYmVpbmcgZHJhd24gYXQgYSBsb3dlciByZXNvbHV0aW9uIHRvIHNwZWVkIHRoaW5ncyB1cC4KCldlIGhhdmUgdG8gY3JlYXRlIHRoZSBoZWF0bWFwIF90d2ljZV8gLS0gb25jZSBmb3IgZGlzcGxheSBpbiB0aGUgbm90ZWJvb2sgYW5kIG9uY2UgdG8gc2F2ZSB0byBhIFBORyBmaWxlLgpGb3IgdGhlIHNlY29uZCBkcmF3aW5nIHdlIHdpbGwgZGVhbCB3aXRoIHRoZSByYXN0ZXIgd2FybmluZyBhbmQgcHJvZHVjZSBhIGhpZ2hlciByZXNvbHV0aW9uIG91dHB1dC4KCmBgYHtyIHNhdmVfaGVhdG1hcF9wbmcsIGxpdmUgPSBUUlVFfQojIE9wZW4gUE5HIHBsb3QgZGV2aWNlCnBuZyhmaWxlbmFtZSA9IGhlYXRtYXBfZmlsZSwgCiAgICB3aWR0aCA9IDExLCAKICAgIGhlaWdodCA9IDcsIAogICAgdW5pdHMgPSAiaW4iLCAKICAgIHJlcyA9IDMwMCkKIyBIZWF0bWFwLCBhZ2FpbiEKSGVhdG1hcCh6c2NvcmVzX21hdCwKICAgICAgICBjbHVzdGVyaW5nX2Rpc3RhbmNlX3Jvd3MgPSAicGVhcnNvbiIsCiAgICAgICAgY2x1c3RlcmluZ19tZXRob2Rfcm93cyA9ICJhdmVyYWdlIiwKICAgICAgICBjbHVzdGVyaW5nX2Rpc3RhbmNlX2NvbHVtbnMgPSAicGVhcnNvbiIsCiAgICAgICAgY2x1c3RlcmluZ19tZXRob2RfY29sdW1ucyA9ICJhdmVyYWdlIiwKICAgICAgICBzaG93X3Jvd19uYW1lcyA9IEZBTFNFLAogICAgICAgIHNob3dfY29sdW1uX25hbWVzID0gRkFMU0UsCiAgICAgICAgdG9wX2Fubm90YXRpb24gPSBjb2x1bW5fYW5ub3RhdGlvbiwKICAgICAgICBuYW1lID0gInotc2NvcmUiLAogICAgICAgIHVzZV9yYXN0ZXIgPSBGQUxTRSkgIyBoaWdoZXIgcmVzb2x1dGlvbiBmb3Igb3V0cHV0IChiZSBjYXJlZnVsIHdpdGggUERGIG91dHB1dCEpCiMgU2h1dCBkb3duIGN1cnJlbnQgZ3JhcGhpY3MgZGV2aWNlCmRldi5vZmYoKQpgYGAKCiMjIyBOb3RlcyBvbiBjbHVzdGVyaW5nCgpZb3UgcHJvYmFibHkgbm90aWNlZCB0aGF0IHRoZXJlIHdlcmUgc29tZSBhcmd1bWVudHMgdGhhdCB3ZSBkZWZpbmVkIHRoYXQgd2VyZSByZWxhdGVkIHRvIHRoZSBjbHVzdGVyaW5nIG9mIHNhbXBsZXMgYW5kIGdlbmVzIHdoZW4gd2UgYnVpbHQgdGhlIGhlYXRtYXBzLgpUaGUgY2x1c3RlcmluZyBkZXRlcm1pbmVzIHRoZSBhcnJhbmdlbWVudCBvZiBnZW5lcyBhbmQgc2FtcGxlcyBpbiB0aGUgaGVhdG1hcCwgbWFraW5nIGl0IHNvIGdlbmVzIG9yIHNhbXBsZXMgd2l0aCBzaW1pbGFyIGV4cHJlc3Npb24gcGF0dGVybnMgYXJlIGdyb3VwZWQuCklmIHdlIGRpZG4ndCBkbyB0aGF0LCB3ZSB3b3VsZCBoYXZlIGEgaGVhdG1hcCB0aGF0IGxvb2tlZCBiYXNpY2FsbHkgbGlrZSByYW5kb20gc3RhdGljIQoKVGhlIGtpbmQgb2YgY2x1c3RlcmluZyB3ZSBwZXJmb3JtZWQgd2FzIFtoaWVyYXJjaGljYWwgY2x1c3RlcmluZ10oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvSGllcmFyY2hpY2FsX2NsdXN0ZXJpbmcpLgpUaGlzIGlzIGFuIF9hZ2dsb21lcmF0aXZlXyBtZXRob2Qgb2YgY2x1c3RlcmluZyAtIGVhY2ggc2FtcGxlIHN0YXJ0cyBpbiBpdHMgb3duIGNsdXN0ZXIgYW5kIHRoZW4gYXQgZWFjaCBzdGVwIG9mIHRoZSBhbGdvcml0aG0gdGhlIHR3byBtb3N0IHNpbWlsYXIgY2x1c3RlcnMgYXJlIGpvaW5lZCB1bnRpbCB0aGVyZSdzIG9ubHkgYSBzaW5nbGUgY2x1c3RlciBsZWZ0LgoKVGhlIGFyZ3VtZW50cyB3ZSBjaG9zZSBmb3IgYGNsdXN0ZXJpbmdfZGlzdGFuY2VfKmAgYW5kIGBjbHVzdGVyaW5nX21ldGhvZF8qYCB0b2xkIGBDb21wbGV4SGVhdG1hcGAgdG8gdXNlIDEgbWludXMgdGhlIGNvcnJlbGF0aW9uIGJldHdlZW4gc2FtcGxlcyBhcyB0aGUgZGlzdGFuY2UgbWVhc3VyZSwgYW5kIHRvIGF2ZXJhZ2Ugc2FtcGxlcyB3aGVuIHRoZXkgd2VyZSBtZXJnZWQgaW50byBhIGdyb3VwLgpUaGlzIG1ldGhvZCBpcyBhbHNvIGtub3duIGFzIFtVUEdNQSAodW53ZWlnaHRlZCBwYWlyIGdyb3VwIG1ldGhvZCB3aXRoIGFyaXRobWV0aWMgbWVhbildKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL1VQR01BKS4KClRoZSByZXN1bHQgb2YgdGhpcyBjbHVzdGVyaW5nIHByb2Nlc3MgaXMgYSB0cmVlIChkZW5kcm9ncmFtKSwgd2hpY2ggaXMgc2hvd24gYXQgdGhlIHRvcCBmb3Igc2FtcGxlcyBhbmQgdG8gdGhlIGxlZnQgZm9yIGdlbmVzLgpJdCBpcyBpbXBvcnRhbnQgdG8gbm90ZSB0aGF0IHRoZSBsaW5lYXIgYXJyYW5nZW1lbnQgb2Ygc2FtcGxlcyBpbiB0aGUgdHJlZSBpcyBzb21ld2hhdCBhcmJpdHJhcnksIHNvIHdlIGhhdmUgdG8gYmUgY2FyZWZ1bCBub3QgdG8gb3ZlcmludGVycHJldCBpdC4KV2UgY2FuIHJvdGF0ZSBhcm91bmQgYW55IGJyYW5jaCBvZiB0aGUgdHJlZSBpbiBvdXIgdmlzdWFsaXphdGlvbiB3aXRoIG5vIGNoYW5nZSB0byB0aGUgdHJlZSB0b3BvbG9neSBpdHNlbGYuCkluIGZhY3QsIHRoZSBzYW1wbGVzIGF0IG9wcG9zaXRlIGVuZHMgb2YgdGhlIGhlYXRtYXAgY291bGQgYmUgcmlnaHQgbmV4dCB0byBvbmUgYW5vdGhlciB3aXRoIHRoZSByaWdodCBzZXQgb2Ygcm90YXRpb25zIQoKCiMjIyBQQ0EgYXMgYW4gYWx0ZXJuYXRpdmUgdG8gY2x1c3RlcmluZwoKWW91IHByb2JhYmx5IG5vdGljZWQgdGhhdCB0aGUgaGllcmFyY2hpY2FsIGNsdXN0ZXJpbmcgd2UgcGVyZm9ybWVkIGRpZG4ndCBwZXJmZWN0bHkgc2VwYXJhdGUgb3V0IHRoZSBkaWZmZXJlbnQgY2FuY2VyIHN1YnR5cGVzIHdlIHdlcmUgbG9va2luZyBhdC4gCgpXaGF0IHdvdWxkIGhhdmUgaGFwcGVuZWQgaWYgd2UgaGFkIHVzZWQgUENBIHRvIHZpc3VhbGl6ZSB0aGUgcmVsYXRpb25zaGlwcyBhbW9uZyBzYW1wbGVzPyAKV291bGQgdGhhdCBoYXZlIG1hZGUgaXQgZWFzaWVyIHRvIHNlcGFyYXRlIGRpc2Vhc2UgdHlwZXM/IApMZXQncyB0cnkgaXQhCgpXZSB3aWxsIHVzZSB0aGUgVlNUIHRyYW5zZm9ybWVkIGRhdGEgaGVyZSB0b28sIGFuZCB0aGUgYHBsb3RQQ0EoKWAgZnVuY3Rpb24gZnJvbSBgREVTZXEyYC4KCmBgYHtyIGV4cHItcGNhIH0KIyBVc2UgcGxvdFBDQSwgYnV0IHJldHVybiB0aGUgZGF0YSBmb3IgY3VzdG9tIHBsb3R0aW5nCnBjYV9kZiA8LSBwbG90UENBKHZzdF9kYXRhLAogICAgICAgICAgICAgICAgICBudG9wID0gNTAwMCwgIyB1c2UgdGhlIHRvcCA1MDAwIGdlbmVzIGJ5IHZhcmlhbmNlCiAgICAgICAgICAgICAgICAgIGludGdyb3VwID0gInNob3J0X2hpc3RvbG9neSIsCiAgICAgICAgICAgICAgICAgIHJldHVybkRhdGEgPSBUUlVFKSAKCgpnZ3Bsb3QocGNhX2RmLCBhZXMoUEMxLCBQQzIsIGNvbG9yID0gc2hvcnRfaGlzdG9sb2d5KSkgKyAKICBnZW9tX3BvaW50KCkgKwogIHRoZW1lX2J3KCkgKyAKICBzY2FsZV9jb2xvcl9tYW51YWwoCiAgICB2YWx1ZXMgPSBjKCJMR0FUIiA9IGhpc3RvbG9neV9jb2xvcnNbWzFdXSwKICAgICAgICAgICAgICAgIkVwZW5keW1vbWEiID0gaGlzdG9sb2d5X2NvbG9yc1tbMl1dLAogICAgICAgICAgICAgICAiSEdBVCIgPSBoaXN0b2xvZ3lfY29sb3JzW1szXV0sCiAgICAgICAgICAgICAgICJNZWR1bGxvYmxhc3RvbWEiID0gaGlzdG9sb2d5X2NvbG9yc1tbNF1dKQogICkgKyAKICBsYWJzKGNvbG9yID0gIkhpc3RvbG9neSIpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IG5laXRoZXIgbWV0aG9kIHBlcmZlY3RseSBzZXBhcmF0ZXMgdGhlIGRpZmZlcmVudCBkaXNlYXNlIHR5cGVzLgpXaXRoIFBDQSwgaWYgd2UgZGlkbid0IGNvbG9yIHRoZSBwb2ludHMsIGl0IG1pZ2h0IGV2ZW4gYmUgZGlmZmljdWx0IHRvIGlkZW50aWZ5IGRpc3RpbmN0IGNsdXN0ZXJzIGluIHRoZSBkYXRhIGF0IGFsbCBieSBleWUuIAooV2UgY291bGQgbG9vayBhdCBtb3JlIHRoYW4gdHdvIFBDcywgd2hpY2ggbWlnaHQgaGVscCBvdXQsIGJ1dCB3ZSB3b3VsZCBuZWVkIHRvIHVzZSBhIGRpZmZlcmVudCBmdW5jdGlvbiBmb3Igb3VyIGNhbGN1bGF0aW9uLCBzaW5jZSBgcGxvdFBDQSgpYCBvbmx5IHJldHVybnMgdGhlIGZpcnN0IHR3by4pCgpGb3IgbmljZSBjb21wYXJpc29uIG9mIHRoZSByZWxhdGl2ZSBhZHZhbnRhZ2VzIG9mIHRoZXNlIHR3byBtZXRob2RzICh3aXRoIGEgbGl0dGxlIG1lbnRpb24gb2Ygc29tZSBmdXJ0aGVyIGRpcmVjdGlvbnMpLCB3ZSByZWNvbW1lbmQgW2Jsb2cgcG9zdCBieSBTb25lc29uXShodHRwczovL3d3dy5ybmEtc2VxYmxvZy5jb20vYS1jb21wYXJpc29uLWJldHdlZW4tcGNhLWFuZC1oaWVyYXJjaGljYWwtY2x1c3RlcmluZy8pLgoKCiMjIFNlc3Npb24gSW5mbwoKYGBge3Igc2Vzc2lvbl9pbmZvfQpzZXNzaW9uSW5mbygpCmBgYAo=
+ + +
+
+ +
+ + + + + + + + + + + + + + + + + diff --git a/intro-to-R-tidyverse/01-intro_to_base_R-live.Rmd b/intro-to-R-tidyverse/01-intro_to_base_R-live.Rmd index 38f62d94..42390164 100644 --- a/intro-to-R-tidyverse/01-intro_to_base_R-live.Rmd +++ b/intro-to-R-tidyverse/01-intro_to_base_R-live.Rmd @@ -1,29 +1,34 @@ --- title: "Introduction to R and RStudio" -author: "Originally authored by Stephanie J. Spielman,
adapted for the CCDL workshop by Candace Savonen" -date: "2020" +author: Originally authored by Stephanie J. Spielman,
adapted by CCDL for ALSF +date: 2021 output: html_notebook: toc: true toc_float: true --- -**CCDL 2020** - ## Objectives -+ Navigate the RStudio environment -+ Use R for simple calculations, both mathematical and logical -+ Understand and apply base R functions -+ Understand, define, and use R data types, including vector manipulation and indexing -+ Understand the anatomy of a data frame + +This notebook will demonstrate how to: + +- Navigate the RStudio environment +- Use R for simple calculations, both mathematical and logical +- Define and use variables in base R +- Understand and apply base R functions +- Understand, define, and use R data types, including vector manipulation and indexing +- Understand the anatomy of a data frame + +--- #### *More resources for learning R* -+ [Swirl, an interactive tutorial](https://swirlstats.com/) -+ [R Markdown](http://rmarkdown.rstudio.com) -+ [R for Data Science](https://r4ds.had.co.nz/) -+ [Tutorial on R, RStudio and R Markdown](https://ismayc.github.io/rbasics-book/) -+ [Handy R cheatsheets](https://www.rstudio.com/resources/cheatsheets/) -+ [R Notebooks tutorial](https://bookdown.org/yihui/rmarkdown/) + +- [Swirl, an interactive tutorial](https://swirlstats.com/) +- [R Markdown](http://rmarkdown.rstudio.com) +- [R for Data Science](https://r4ds.had.co.nz/) +- [Tutorial on R, RStudio and R Markdown](https://ismayc.github.io/rbasics-book/) +- [Handy R cheatsheets](https://www.rstudio.com/resources/cheatsheets/) +- [R Notebooks tutorial](https://bookdown.org/yihui/rmarkdown/) ## What is R? diff --git a/intro-to-R-tidyverse/01-intro_to_base_R.nb.html b/intro-to-R-tidyverse/01-intro_to_base_R.nb.html index b541bec7..0dbdbd3a 100644 --- a/intro-to-R-tidyverse/01-intro_to_base_R.nb.html +++ b/intro-to-R-tidyverse/01-intro_to_base_R.nb.html @@ -333,6 +333,7 @@

Objectives

  • Understand the anatomy of a data frame
    • +

    More resources for learning R

    @@ -440,33 +447,42 @@

    Defining and using variables

    To define a variable, we use the assignment operator which looks like an arrow: <-, for example x <- 7 takes the value on the right-hand side of the operator and assigns it to the variable name on the left-hand side.

    - + 
    # Define a variable x to equal 7, and print out the value of x
 x <- 7
 
 # We can have R repeat back to us what `x` is by just using `x`
 x
    + +
    [1] 7
    +

    Some features of variables, considering the example x <- 7: Every variable has a name, a value, and a type. This variable’s name is x, its value is 7, and its type is numeric (7 is a number!). Re-defining a variable will overwrite the value.

    - + 
    x <- 5.5
 
 x
    + +
    [1] 5.5
    +

    We can modify an existing variable by reassigning it to its same name. Here we’ll add 2 to x and reassign the result back to x.

    - + 
    x <- x + 2
 
 x
    + +
    [1] 7.5
    + @@ -480,13 +496,20 @@

    Comments

    Comments in R code are indicated with pound signs (aka hashtags, octothorps). R will ignore any text in a line after the pound sign, so you can put whatever text you like there.

    - -
    22/7 # not quite pi
-
-# If we need a better approximation of pi, we can use Euler's formula
+
+
    22/7 # not quite pi
    
+
+
+
    [1] 3.142857
    
+
+
+
    # If we need a better approximation of pi, we can use Euler's formula
 # This uses atan(), which calculates arctangent.
 20 * atan(1/7) + 8 * atan(3/79) 
    
 
+
+
    [1] 3.141593
    
+
 
 
 
    Help out Future You by adding lots of comments! Future You next week thinks Today You is an idiot, and the only way you can convince Future You that Today You is reasonably competent is by adding comments in your code explaining why Today You is actually not so bad.
    
@@ -503,42 +526,57 @@ 
    Functions
    
 
    Functions also return values for us to use. In the case of log(), the returned value is the log’d value the function computed.
    
 
 
-
+
 
    log(73)
    
 
+
+
    [1] 4.290459
    
+
 
 
 
    Here we can specify an argument of base to calculate log base 3.
    
 
 
-
+
 
    log(81, base = 3)
    
 
+
+
    [1] 4
    
+
 
 
 
    If we don’t specify the argument names, it assumes they are in the order that log defines them. See ?log to see more about its arguments.
    
 
 
-
+
 
    log(8, 2)
    
 
+
+
    [1] 3
    
+
 
 
 
    We can switch the order if we specify the argument names.
    
 
 
-
+
 
    log(base = 10, x = 4342)
    
 
+
+
    [1] 3.63769
    
+
 
 
 
    We can also provide variables as arguments in the same way as the raw values.
    
 
 
-
+
 
    meaning <- 42
 log(meaning)
    
 
+
+
    [1] 3.73767
    
+
 
 
 
@@ -549,7 +587,7 @@ 
    Variable Types
    
 
    Variable types in R can sometimes be coerced (converted) from one type to another.
    
 
 
-
+
 
    # Define a variable with a number
 x <- 15
    
 
@@ -558,32 +596,50 @@ 
    Variable Types
    
 
    The function class() will tell us the variable’s type.
    
 
 
-
+
 
    class(x)
    
 
+
+
    [1] "numeric"
    
+
+
+
    numeric
    
+
 
 
 
    Let’s coerce it to a character.
    
 
 
-
+
 
    x <- as.character(x)
 class(x)
    
 
+
+
    [1] "character"
    
+
+
+
    character
    
+
 
 
 
    See it now has quotes around it? It’s now a character and will behave as such
    
 
 
-
+
 
    x
    
 
+
+
    [1] "15"
    
+
+
+
    15
    
+
 
 
 
    Use this chunk to try to perform calculations with x, now that it is a character, what happens?
    
 
 
-
+
 
    # Try to perform calculations on `x`
    
 
 
@@ -591,7 +647,7 @@ 
    Variable Types
    
 
    But we can’t coerce everything:
    
 
 
-
+
 
    # Let's create a character variable
 x <- "look at my character variable"
    
 
@@ -600,17 +656,23 @@ 
    Variable Types
    
 
    Let’s try making this a numeric variable:
    
 
 
-
+
 
    x <- as.numeric(x)
    
 
+
+
    Warning: NAs introduced by coercion
    
+
 
 
 
    Print out x.
    
 
 
-
+
 
    x
    
 
+
+
    [1] NA
    
+
 
 
 
    R is telling us it doesn’t know how to convert this to a numeric variable, so it has returned NA instead.
    
@@ -669,7 +731,7 @@ 
    Vectors
    
 
    You will have noticed that all your computations tend to pop up with a [1] preceding them in R’s output. This is because, in fact, all (ok mostly all) variables are by default vectors, and our answers are the first (in these cases only) value in the vector. As vectors get longer, new index indicators will appear at the start of new lines.
    
 
 
-
+
 
    # This is actually an vector that has one item in it.
 x <- 7
    
 
@@ -677,70 +739,94 @@ 
    Vectors
    
 
 
 
-
+
 
    # The length() functions tells us how long an vector is:
 length(x)
    
 
+
+
    [1] 1
    
+
 
 
 
    We can define vectors with the function c(), which stands for “combine”. This function takes a comma-separated set of values to place in the vector, and returns the vector itself:
    
 
 
-
+
 
    my_numeric_vector <- c(1, 1, 2, 3, 5, 8, 13, 21)
 my_numeric_vector
    
 
+
+
    [1]  1  1  2  3  5  8 13 21
    
+
 
 
 
    We can build on vectors in place by redefining them
    
 
 
-
+
 
    # add the next two Fibonacci numbers to the series.
 my_numeric_vector <- c(my_numeric_vector, 34, 55)
 my_numeric_vector
    
 
+
+
     [1]  1  1  2  3  5  8 13 21 34 55
    
+
 
 
 
    We can pull out specific items from an vector using a process called indexing, which uses brackets [] to specify the position of an item.
    
 
 
-
+
 
    # Grab the fourth value from my_numeric_vector
 # This gives us an vector of length 1 
 my_numeric_vector[4]
    
 
+
+
    [1] 3
    
+
 
 
 
    Colons are also a nice way to quickly make ordered numeric vectors Use a colon to specify an inclusive range of indices This will return an vector with 2, 3, 4, and 5.
    
 
 
-
+
 
    my_numeric_vector[2:5]
    
 
+
+
    [1] 1 2 3 5
    
+
 
 
 
    One major benefit of vectors is the concept of vectorization, where R by default performs operations on the entire vector at once. For example, we can get the log of all numbers 1-2 with a single, simple call, and more!
    
 
 
-
+
 
    values_1_to_20 <- 1:20
    
 
 
 
-
+
 
    # calculate the log of values_1_to_20
 log(values_1_to_20)
    
 
+
+
     [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 1.7917595 1.9459101
+ [8] 2.0794415 2.1972246 2.3025851 2.3978953 2.4849066 2.5649494 2.6390573
+[15] 2.7080502 2.7725887 2.8332133 2.8903718 2.9444390 2.9957323
    
+
 
 
 
    Finally, we can apply logical expressions to vectors, just as we can do for single values. The output here is a logical vector telling us whether each value in example_vector is TRUE or FALSE
    
 
 
-
+
 
    # Which values are <= 3?
 values_1_to_20 <= 3
    
 
+
+
     [1]  TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
+[13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
    
+
 
 
 
    There are several key functions which can be used on vectors containing numeric values, some of which are below.
    
@@ -753,11 +839,14 @@ 
    Vectors
    
 
    We can try out these functions on the vector values_1_to_20 we’ve created.
    
 
 
-
-
    mean(values_1_to_20)
-
-# Try out some of the other functions we've listed above 
-
    
+
+
    mean(values_1_to_20)
    
+
+
+
    [1] 10.5
    
+
+
+
    # Try out some of the other functions we've listed above 
    
 
 
 
@@ -771,28 +860,37 @@ 
    The %in% logical operator
    
 
    %in% is useful for determining whether a given item(s) are in an vector.
    
 
 
-
+
 
    # is `7` in our vector? 
 7 %in% values_1_to_20
    
 
+
+
    [1] TRUE
    
+
 
 
 
 
-
+
 
    # is `50` in our vector? 
 50 %in% values_1_to_20
    
 
+
+
    [1] FALSE
    
+
 
 
 
    We can test a vector of values being within another vector of values.
    
 
 
-
+
 
    question_values <- c(1:3, 7, 50)
 # Are these values in our vector?
 question_values %in% values_1_to_20
    
 
+
+
    [1]  TRUE  TRUE  TRUE  TRUE FALSE
    
+
 
 
 
@@ -802,7 +900,7 @@ 
    Data frames
    
 
    Data frames are one of the most useful tools for data analysis in R. They are tables which consist of rows and columns, much like a spreadsheet. Each column is a variable which behaves as a vector, and each row is an observation. We will begin our exploration with dataset of measurements from three penguin species measured, which we can find in the palmerpenguins package. We’ll talk more about packages soon! To use this dataset, we will load it from the palmerpenguins package using a :: (more on this later) and assign it to a variable named penguins in our current environment.
    
 
 
-
+
 
    penguins <- palmerpenguins::penguins
    
 
 
@@ -823,65 +921,129 @@ 
    Exploring data frames
    
 
    This provides summary statistics for each column:
    
 
 
-
+
 
    summary(penguins)
    
 
+
+
          species          island    bill_length_mm  bill_depth_mm  
+ Adelie   :152   Biscoe   :168   Min.   :32.10   Min.   :13.10  
+ Chinstrap: 68   Dream    :124   1st Qu.:39.23   1st Qu.:15.60  
+ Gentoo   :124   Torgersen: 52   Median :44.45   Median :17.30  
+                                 Mean   :43.92   Mean   :17.15  
+                                 3rd Qu.:48.50   3rd Qu.:18.70  
+                                 Max.   :59.60   Max.   :21.50  
+                                 NA's   :2       NA's   :2      
+ flipper_length_mm  body_mass_g       sex           year     
+ Min.   :172.0     Min.   :2700   female:165   Min.   :2007  
+ 1st Qu.:190.0     1st Qu.:3550   male  :168   1st Qu.:2007  
+ Median :197.0     Median :4050   NA's  : 11   Median :2008  
+ Mean   :200.9     Mean   :4202                Mean   :2008  
+ 3rd Qu.:213.0     3rd Qu.:4750                3rd Qu.:2009  
+ Max.   :231.0     Max.   :6300                Max.   :2009  
+ NA's   :2         NA's   :2                                 
    
+
 
 
 
    This provides a short view of the structure and contents of the data frame.
    
 
 
-
+
 
    str(penguins)
    
 
+
+
    tibble [344 × 8] (S3: tbl_df/tbl/data.frame)
+ $ species          : Factor w/ 3 levels "Adelie","Chinstrap",..: 1 1 1 1 1 1 1 1 1 1 ...
+ $ island           : Factor w/ 3 levels "Biscoe","Dream",..: 3 3 3 3 3 3 3 3 3 3 ...
+ $ bill_length_mm   : num [1:344] 39.1 39.5 40.3 NA 36.7 39.3 38.9 39.2 34.1 42 ...
+ $ bill_depth_mm    : num [1:344] 18.7 17.4 18 NA 19.3 20.6 17.8 19.6 18.1 20.2 ...
+ $ flipper_length_mm: int [1:344] 181 186 195 NA 193 190 181 195 193 190 ...
+ $ body_mass_g      : int [1:344] 3750 3800 3250 NA 3450 3650 3625 4675 3475 4250 ...
+ $ sex              : Factor w/ 2 levels "female","male": 2 1 1 NA 1 2 1 2 NA NA ...
+ $ year             : int [1:344] 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 ...
    
+
 
 
 
    You’ll notice that the column species is a factor: This is a special type of character variable that represents distinct categories known as “levels”. We have learned here that there are three levels in the species column: Adelie, Chinstrap, and Gentoo. We might want to explore individual columns of the data frame more in-depth. We can examine individual columns using the dollar sign $ to select one by name:
    
 
 
-
+
 
    # Extract bill_length_mm as a vector
-penguins$bill_length_mm
-
-# indexing operators can be used too
+penguins$bill_length_mm
    
+
+
+
      [1] 39.1 39.5 40.3   NA 36.7 39.3 38.9 39.2 34.1 42.0 37.8 37.8 41.1 38.6 34.6
+ [16] 36.6 38.7 42.5 34.4 46.0 37.8 37.7 35.9 38.2 38.8 35.3 40.6 40.5 37.9 40.5
+ [31] 39.5 37.2 39.5 40.9 36.4 39.2 38.8 42.2 37.6 39.8 36.5 40.8 36.0 44.1 37.0
+ [46] 39.6 41.1 37.5 36.0 42.3 39.6 40.1 35.0 42.0 34.5 41.4 39.0 40.6 36.5 37.6
+ [61] 35.7 41.3 37.6 41.1 36.4 41.6 35.5 41.1 35.9 41.8 33.5 39.7 39.6 45.8 35.5
+ [76] 42.8 40.9 37.2 36.2 42.1 34.6 42.9 36.7 35.1 37.3 41.3 36.3 36.9 38.3 38.9
+ [91] 35.7 41.1 34.0 39.6 36.2 40.8 38.1 40.3 33.1 43.2
+ [ reached getOption("max.print") -- omitted 244 entries ]
    
+
+
+
    # indexing operators can be used too
 penguins$bill_depth_mm[1:10]
    
 
+
+
     [1] 18.7 17.4 18.0   NA 19.3 20.6 17.8 19.6 18.1 20.2
    
+
 
 
 
    We can perform our regular vector operations on columns directly.
    
 
 
-
+
 
    # calculate the mean of the bill_length_mm column
 mean(penguins$bill_length_mm,
      na.rm = TRUE) # remove missing values before calculating the mean
    
 
+
+
    [1] 43.92193
    
+
 
 
 
    We can also calculate the full summary statistics for a single column directly.
    
 
 
-
+
 
    # show a summary of the bill_length_mm column
 summary(penguins$bill_length_mm)
    
 
+
+
       Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
+  32.10   39.23   44.45   43.92   48.50   59.60       2 
    
+
 
 
 
    Extract Species as a vector and subset it to see a preview.
    
 
 
-
+
 
    # get the first 10 values of the Species column
 penguins$species[1:10]
    
 
+
+
     [1] Adelie Adelie Adelie Adelie Adelie Adelie Adelie Adelie Adelie Adelie
+Levels: Adelie Chinstrap Gentoo
    
+
 
 
 
    And view its levels with the levels() function.
    
 
 
-
+
 
    levels(penguins$species)
    
 
+
+
    [1] "Adelie"    "Chinstrap" "Gentoo"   
    
+
+
+
    Adelie
+
+Chinstrap
+
+Gentoo
    
+
 
 
 
@@ -892,7 +1054,7 @@ 
    Files and directories
    
 
    Here we will use a function, read_tsv() from the readr package. Before we are able to use the function, we have to load the package using library().
    
 
 
-
+
 
    library(readr)
    
 
 
@@ -900,15 +1062,21 @@ 
    Files and directories
    
 
    file.path() creates a properly formatted file path by adding a path separator (/ on Mac and Linux operating system, which is the operating system that our RStudio Server runs on) between separate folders or directories. Because file path separators can differ between your computer and the computer of someone who wants to use your code, we use file.path() instead of typing out "data/gene_results_GSE44971.tsv". Each argument to file.path() is a directory or file name. You’ll notice each argument is in quotes, we specify data first because the file, gene_results_GSE44971.tsv is in the data folder.
    
 
 
-
+
 
    file.path("data", "gene_results_GSE44971.tsv")
    
 
+
+
    [1] "data/gene_results_GSE44971.tsv"
    
+
+
+
    data/gene_results_GSE44971.tsv
    
+
 
 
 
    We can store this file path as a variable in our environment.
    
 
 
-
+
 
    gene_file_path <- file.path("data", "gene_results_GSE44971.tsv")
    
 
 
@@ -916,19 +1084,38 @@ 
    Files and directories
    
 
    Now we are ready to use read_tsv() to read the file into R. The resulting data frame will be stored in a variable named stats_df. Note the <- which is responsible for saving this to our global environment.
    
 
 
-
+
 
    # read in the file `gene_results_GSE44971.tsv` from the data directory
 stats_df <- read_tsv(gene_file_path)
    
 
+
+
    
+── Column specification ────────────────────────────────────────────────────────
+cols(
+  ensembl_id = col_character(),
+  gene_symbol = col_character(),
+  contrast = col_character(),
+  log_fold_change = col_double(),
+  avg_expression = col_double(),
+  t_statistic = col_double(),
+  p_value = col_double(),
+  adj_p_value = col_double()
+)
    
+
 
 
 
    Take a look at your environment panel to see what stats_df looks like. We can also print out a preview of the stats_df data frame here.
    
 
 
-
+
 
    # display stats_df
 stats_df
    
 
+
    
+
+
    
 
 
 
    
@@ -936,14 +1123,49 @@ 
    Session Info
    
 
    At the end of every notebook, you will see us print out sessionInfo. This aids in the reproducibility of your code by showing exactly what packages and versions were being used the last time the notebook was run.
    
 
 
-
+
 
    sessionInfo()
    
 
+
+
    R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] stats     graphics  grDevices utils     datasets  methods   base     
+
+other attached packages:
+[1] readr_1.4.0    optparse_1.6.6
+
+loaded via a namespace (and not attached):
+ [1] rstudioapi_0.13      knitr_1.30           magrittr_2.0.1      
+ [4] hms_1.0.0            getopt_1.20.3        R6_2.5.0            
+ [7] rlang_0.4.10         fansi_0.4.2          stringr_1.4.0       
+[10] tools_4.0.3          palmerpenguins_0.1.0 xfun_0.20           
+[13] cli_2.2.0            htmltools_0.5.1.1    ellipsis_0.3.1      
+[16] yaml_2.2.1           digest_0.6.27        assertthat_0.2.1    
+[19] tibble_3.0.5         lifecycle_0.2.0      crayon_1.3.4        
+[22] ps_1.5.0             vctrs_0.3.6          glue_1.4.2          
+[25] evaluate_0.14        rmarkdown_2.6        stringi_1.5.3       
+[28] compiler_4.0.3       pillar_1.4.7         jsonlite_1.7.2      
+[31] pkgconfig_2.0.3     
    
+
 
 
    
 
 
-
    LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIFIgYW5kIFJTdHVkaW8iCmF1dGhvcjogT3JpZ2luYWxseSBhdXRob3JlZCBieSBTdGVwaGFuaWUgSi4gU3BpZWxtYW4sPGJyPmFkYXB0ZWQgYnkgQ0NETCBmb3IgQUxTRgpkYXRlOiAnMjAyMScKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6CiAgICB0b2M6IHRydWUKICAgIHRvY19mbG9hdDogdHJ1ZQotLS0KCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86ICAKCi0gTmF2aWdhdGUgdGhlIFJTdHVkaW8gZW52aXJvbm1lbnQgIAotIFVzZSBSIGZvciBzaW1wbGUgY2FsY3VsYXRpb25zLCBib3RoIG1hdGhlbWF0aWNhbCBhbmQgbG9naWNhbCAgCi0gRGVmaW5lIGFuZCB1c2UgdmFyaWFibGVzIGluIGJhc2UgUiAgCi0gVW5kZXJzdGFuZCBhbmQgYXBwbHkgYmFzZSBSIGZ1bmN0aW9ucyAgIAotIFVuZGVyc3RhbmQsIGRlZmluZSwgYW5kIHVzZSBSIGRhdGEgdHlwZXMsIGluY2x1ZGluZyB2ZWN0b3IgbWFuaXB1bGF0aW9uIGFuZCBpbmRleGluZyAgCi0gVW5kZXJzdGFuZCB0aGUgYW5hdG9teSBvZiBhIGRhdGEgZnJhbWUgIAoKIyMjIyAqTW9yZSByZXNvdXJjZXMgZm9yIGxlYXJuaW5nIFIqIAoKLSBbU3dpcmwsIGFuIGludGVyYWN0aXZlIHR1dG9yaWFsXShodHRwczovL3N3aXJsc3RhdHMuY29tLykgIAotIFtSIE1hcmtkb3duXShodHRwOi8vcm1hcmtkb3duLnJzdHVkaW8uY29tKSAgCi0gW1IgZm9yIERhdGEgU2NpZW5jZV0oaHR0cHM6Ly9yNGRzLmhhZC5jby5uei8pICAKLSBbVHV0b3JpYWwgb24gUiwgUlN0dWRpbyBhbmQgUiBNYXJrZG93bl0oaHR0cHM6Ly9pc21heWMuZ2l0aHViLmlvL3JiYXNpY3MtYm9vay8pICAKLSBbSGFuZHkgUiBjaGVhdHNoZWV0c10oaHR0cHM6Ly93d3cucnN0dWRpby5jb20vcmVzb3VyY2VzL2NoZWF0c2hlZXRzLykgIAotIFtSIE5vdGVib29rcyB0dXRvcmlhbF0oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duLykgIAoKIyMgV2hhdCBpcyBSPwoKKipSKiogaXMgYSBzdGF0aXN0aWNhbCBjb21wdXRpbmcgbGFuZ3VhZ2UgdGhhdCBpcyBfb3BlbiBzb3VyY2VfLCBtZWFuaW5nIHRoZSB1bmRlcmx5aW5nIGNvZGUgZm9yIHRoZSBsYW5ndWFnZSBpcyBmcmVlbHkgYXZhaWxhYmxlIHRvIGFueW9uZS4gCllvdSBkbyBub3QgbmVlZCBhIHNwZWNpYWwgbGljZW5zZSBvciBzZXQgb2YgcGVybWlzc2lvbnMgdG8gdXNlIGFuZCBkZXZlbG9wIGNvZGUgaW4gUi4gCgpSIGl0c2VsZiBpcyBhbiBfaW50ZXJwcmV0ZWQgY29tcHV0ZXIgbGFuZ3VhZ2VfIGFuZCBjb21lcyB3aXRoIGZ1bmN0aW9uYWxpdHkgdGhhdCBjb21lcyBidW5kbGVkIHdpdGggdGhlIGxhbmd1YWdlIGl0c2VsZiwga25vd24gYXMgKioiYmFzZSBSIioqLgpCdXQgdGhlcmUgaXMgYWxzbyByaWNoIGFkZGl0aW9uYWwgZnVuY3Rpb25hbGl0eSBwcm92aWRlZCBieSAqKmV4dGVybmFsIHBhY2thZ2VzKiosIG9yIGxpYnJhcmllcyBvZiBjb2RlIHRoYXQgYXNzaXN0IGluIGFjY29tcGxpc2hpbmcgY2VydGFpbiB0YXNrcyBhbmQgY2FuIGJlIGZyZWVseSBkb3dubG9hZGVkIGFuZCBsb2FkZWQgZm9yIHVzZS4gCgpJbiB0aGUgbmV4dCBub3RlYm9vayBhbmQgc3Vic2VxdWVudCBtb2R1bGVzLCB3ZSB3aWxsIGJlIHVzaW5nIGEgc3VpdGUgb2YgcGFja2FnZXMgY29sbGVjdGl2ZWx5IGtub3duIGFzIFsqKlRoZSBUaWR5dmVyc2UqKl0oaHR0cDovL3RpZHl2ZXJzZS5vcmcpLiAKVGhlIGB0aWR5dmVyc2VgIGlzIGdlYXJlZCB0b3dhcmRzIGludHVpdGl2ZSBkYXRhIHNjaWVuY2UgYXBwbGljYXRpb25zIHRoYXQgZm9sbG93IGEgc2hhcmVkIGRhdGEgcGhpbG9zb3BoeS4KQnV0IHRoZXJlIGFyZSBzdGlsbCBtYW55IGNvcmUgZmVhdHVyZXMgb2YgYmFzZSBSIHdoaWNoIGFyZSBpbXBvcnRhbnQgdG8gYmUgYXdhcmUgb2YsIGFuZCB3ZSB3aWxsIGJlIHVzaW5nIGNvbmNlcHRzIGZyb20gYm90aCBiYXNlIFIgYW5kIHRoZSB0aWR5dmVyc2UgaW4gb3VyIGFuYWx5c2VzLCBhcyB3ZWxsIGFzIHRhc2sgc3BlY2lmaWMgcGFja2FnZXMgZm9yIGFuYWx5c2VzIHN1Y2ggYXMgZ2VuZSBleHByZXNzaW9uLiAKCiMjIyBXaGF0IGlzIFJTdHVkaW8/CgpSU3R1ZGlvIGlzIGEgX2dyYXBoaWNhbCBlbnZpcm9ubWVudF8gKCJpbnRlZ3JhdGVkIGRldmVsb3BtZW50IGVudmlyb25tZW50IiBvciBJREUpIGZvciB3cml0aW5nIGFuZCBkZXZlbG9waW5nIFIgY29kZS4gUlN0dWRpbyBpcyBOT1QgYSBzZXBhcmF0ZSBwcm9ncmFtbWluZyBsYW5ndWFnZSAtIGl0IGlzIGFuIGludGVyZmFjZSB3ZSB1c2UgdG8gZmFjaWxpdGF0ZSBSIHByb2dyYW1taW5nLiAKSW4gb3RoZXIgd29yZHMsIHlvdSBjYW4gcHJvZ3JhbSBpbiBSIHdpdGhvdXQgUlN0dWRpbywgYnV0IHlvdSBjYW4ndCB1c2UgdGhlIFJTdHVkaW8gZW52aXJvbm1lbnQgd2l0aG91dCBSLgoKRm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgUlN0dWRpbyB0aGFuIHlvdSBldmVyIHdhbnRlZCB0byBrbm93LCBzZWUgdGhpcyBbUlN0dWRpbyBJREUgQ2hlYXRzaGVldF0oaHR0cHM6Ly9kMzN3dWJyZmtpMGw2OC5jbG91ZGZyb250Lm5ldC8wZGMwZWZjY2ZhZjYzOGRlOGJlNGNlNDNlMmJhYjhhOTE1NDFkM2QyL2M0ZmRjL3dwLWNvbnRlbnQvdXBsb2Fkcy8yMDE4LzA4L3JzdHVkaW8taWRlLnBuZykuCgojIyBUaGUgUlN0dWRpbyBFbnZpcm9ubWVudAoKVGhlIFJTdHVkaW8gZW52aXJvbm1lbnQgaGFzIGZvdXIgbWFpbiAqKnBhbmVzKiosIGVhY2ggb2Ygd2hpY2ggbWF5IGhhdmUgYSBudW1iZXIgb2YgdGFicyB0aGF0IGRpc3BsYXkgZGlmZmVyZW50IGluZm9ybWF0aW9uIG9yIGZ1bmN0aW9uYWxpdHkuICh0aGVpciBzcGVjaWZpYyBsb2NhdGlvbiBjYW4gYmUgY2hhbmdlZCB1bmRlciBUb29scyAtPiBHbG9iYWwgT3B0aW9ucyAtPiBQYW5lIExheW91dCkuCiFbUlN0dWRpbyBBcHBlYXJhbmNlXShzY3JlZW5zaG90cy9yc3R1ZGlvLXBhbmVzLnBuZykgCgoxLiBUaGUgKipFZGl0b3IqKiBwYW5lIGlzIHdoZXJlIHlvdSBjYW4gd3JpdGUgUiBzY3JpcHRzIGFuZCBvdGhlciBkb2N1bWVudHMuIEVhY2ggdGFiIGhlcmUgaXMgaXRzIG93biBkb2N1bWVudC4KVGhpcyBpcyB5b3VyIF90ZXh0IGVkaXRvcl8sIHdoaWNoIHdpbGwgYWxsb3cgeW91IHRvIHNhdmUgeW91ciBSIGNvZGUgZm9yIGZ1dHVyZSB1c2UuIApOb3RlIHRoYXQgY2hhbmdlIGNvZGUgaGVyZSB3aWxsIG5vdCBydW4gYXV0b21hdGljYWxseSB1bnRpbCB5b3UgcnVuIGl0LiAKCjIuIFRoZSAqKkNvbnNvbGUqKiBwYW5lIGlzIHdoZXJlIHlvdSBjYW4gX2ludGVyYWN0aXZlbHlfIHJ1biBSIGNvZGUuIAogICsgVGhlcmUgaXMgYWxzbyBhICoqVGVybWluYWwqKiB0YWIgaGVyZSB3aGljaCBjYW4gYmUgdXNlZCBmb3IgcnVubmluZyBwcm9ncmFtcyBvdXRzaWRlIFIgb24geW91ciBjb21wdXRlcgogIAozLiBUaGUgKipFbnZpcm9ubWVudCoqIHBhbmUgcHJpbWFyaWx5IGRpc3BsYXlzIHRoZSB2YXJpYWJsZXMsIHNvbWV0aW1lcyBrbm93biBhcyBfb2JqZWN0c18gdGhhdCBhcmUgZGVmaW5lZCBkdXJpbmcgYSBnaXZlbiBSIHNlc3Npb24sIGFuZCB3aGF0IGRhdGEgb3IgdmFsdWVzIHRoZXkgbWlnaHQgaG9sZC4KCjQuIFRoZSAqKkhlbHAgdmlld2VyKiogcGFuZSBoYXMgc2V2ZXJhbCB0YWJzIGFsbCBvZiB3aGljaCBhcmUgcHJldHR5IGltcG9ydGFudDoKICAgICsgVGhlICoqRmlsZXMqKiB0YWIgc2hvd3MgdGhlIHN0cnVjdHVyZSBhbmQgY29udGVudHMgb2YgZmlsZXMgYW5kIGZvbGRlcnMgKGFsc28ga25vd24gYXMgZGlyZWN0b3JpZXMpIG9uIHlvdXIgY29tcHV0ZXIuCiAgICArIFRoZSAqKlBsb3RzKiogdGFiIHdpbGwgcmV2ZWFsIHBsb3RzIHdoZW4geW91IG1ha2UgdGhlbQogICAgKyBUaGUgKipQYWNrYWdlcyoqIHRhYiBzaG93cyB3aGljaCBpbnN0YWxsZWQgcGFja2FnZXMgaGF2ZSBiZWVuIGxvYWRlZCBpbnRvIHlvdXIgUiBzZXNzaW9uCiAgICArIFRoZSAqKkhlbHAqKiB0YWIgd2lsbCBzaG93IHRoZSBoZWxwIHBhZ2Ugd2hlbiB5b3UgbG9vayB1cCBhIGZ1bmN0aW9uCiAgICArIFRoZSAqKlZpZXdlcioqIHBhbmUgd2lsbCByZXZlYWwgY29tcGlsZWQgUiBNYXJrZG93biBkb2N1bWVudHMgCgojIyBCYXNpYyBDYWxjdWxhdGlvbnMKCiMjIyBNYXRoZW1hdGljYWwgb3BlcmF0b3JzCgpUaGUgbW9zdCBiYXNpYyB1c2Ugb2YgUiBpcyBhcyBhIHJlZ3VsYXIgY2FsY3VsYXRvcjoKCnwgT3BlcmF0aW9uIHwgU3ltYm9sIHwKfC0tLS0tLS0tLS0tfC0tLS0tLS0tfAp8IEFkZCAgfCBgK2AgfCAKfCBTdWJ0cmFjdCAgfCBgLWAgfCAKfCBNdWx0aXBseSAgfCBgKmAgfCAKfCBEaXZpZGUgIHwgYC9gIHwgCnwgRXhwb25lbnRpYXRlIHwgYF5gIG9yIGAqKmAgfCAKCkZvciBleGFtcGxlLCB3ZSBjYW4gZG8gc29tZSBzaW1wbGUgbXVsdGlwbGljYXRpb24gbGlrZSB0aGlzLiAKV2hlbiB5b3UgZXhlY3V0ZSBjb2RlIHdpdGhpbiB0aGUgbm90ZWJvb2ssIHRoZSByZXN1bHRzIGFwcGVhciBiZW5lYXRoIHRoZSBjb2RlLiAKVHJ5IGV4ZWN1dGluZyB0aGlzIGNodW5rIGJ5IGNsaWNraW5nIHRoZSAqUnVuKiBidXR0b24gd2l0aGluIHRoZSBjaHVuayBvciBieSAKcGxhY2luZyB5b3VyIGN1cnNvciBpbnNpZGUgaXQgYW5kIHByZXNzaW5nICpDbWQrU2hpZnQrRW50ZXIqLiAKCmBgYHtyIGNhbGN1bGF0b3J9CjUgKiA2CmBgYAoKVXNlIHRoZSBjb25zb2xlIHRvIGNhbGN1bGF0ZSBvdGhlciBleHByZXNzaW9ucy4gU3RhbmRhcmQgb3JkZXIgb2Ygb3BlcmF0aW9ucyBhcHBsaWVzIChtb3N0bHkpLCBhbmQgIHlvdSBjYW4gdXNlIHBhcmVudGhlc2VzIGAoKWAgYXMgeW91IG1pZ2h0IGV4cGVjdCAoYnV0IG5vdCBicmFja2V0cyBgW11gIG9yIGJyYWNlc2B7fWAsIHdoaWNoIGhhdmUgc3BlY2lhbCBtZWFuaW5ncykuIE5vdGUgaG93ZXZlciwgdGhhdCB5b3UgbXVzdCAqKmFsd2F5cyoqIHNwZWNpZnkgbXVsdGlwbGljYXRpb24gd2l0aCBgKmA7IGltcGxpY2l0IG11bHRpcGxpY2F0aW9uIHN1Y2ggYXMgYDEwKDMgKyA0KWAgb3IgYDEweGAgd2lsbCBub3Qgd29yayBhbmQgd2lsbCBnZW5lcmF0ZSBhbiBlcnJvciwgb3Igd29yc2UuCgpgYGB7ciBleHByZXNzaW9ucywgbGl2ZSA9IFRSVUV9CjEwICogKDMgKyA0KV4yCmBgYAoKCiMjIyBEZWZpbmluZyBhbmQgdXNpbmcgdmFyaWFibGVzIAoKVG8gZGVmaW5lIGEgdmFyaWFibGUsIHdlIHVzZSB0aGUgX2Fzc2lnbm1lbnQgb3BlcmF0b3JfIHdoaWNoIGxvb2tzIGxpa2UgYW4gYXJyb3c6IGA8LWAsIGZvciBleGFtcGxlIGB4IDwtIDdgIHRha2VzIHRoZSB2YWx1ZSBvbiB0aGUgcmlnaHQtaGFuZCBzaWRlIG9mIHRoZSBvcGVyYXRvciBhbmQgYXNzaWducyBpdCB0byB0aGUgdmFyaWFibGUgbmFtZSBvbiB0aGUgbGVmdC1oYW5kIHNpZGUuIAoKYGBge3IgdmFyLWRlZmluZSwgbGl2ZSA9IFRSVUV9CiMgRGVmaW5lIGEgdmFyaWFibGUgeCB0byBlcXVhbCA3LCBhbmQgcHJpbnQgb3V0IHRoZSB2YWx1ZSBvZiB4CnggPC0gNwoKIyBXZSBjYW4gaGF2ZSBSIHJlcGVhdCBiYWNrIHRvIHVzIHdoYXQgYHhgIGlzIGJ5IGp1c3QgdXNpbmcgYHhgCngKYGBgCgpTb21lIGZlYXR1cmVzIG9mIHZhcmlhYmxlcywgY29uc2lkZXJpbmcgdGhlIGV4YW1wbGUgYHggPC0gN2A6CkV2ZXJ5IHZhcmlhYmxlIGhhcyBhICoqbmFtZSoqLCBhICoqdmFsdWUqKiwgYW5kIGEgKip0eXBlKiouIApUaGlzIHZhcmlhYmxlJ3MgbmFtZSBpcyBgeGAsIGl0cyB2YWx1ZSBpcyBgN2AsIGFuZCBpdHMgdHlwZSBpcyBgbnVtZXJpY2AgKDcgaXMgYSBudW1iZXIhKS4KUmUtZGVmaW5pbmcgYSB2YXJpYWJsZSB3aWxsIG92ZXJ3cml0ZSB0aGUgdmFsdWUuCgpgYGB7ciB2YXItcmVkZWZpbmV9CnggPC0gNS41Cgp4CmBgYAoKV2UgY2FuIG1vZGlmeSBhbiBleGlzdGluZyB2YXJpYWJsZSBieSByZWFzc2lnbmluZyBpdCB0byBpdHMgc2FtZSBuYW1lLiAKSGVyZSB3ZSdsbCBhZGQgYDJgIHRvIGB4YCBhbmQgcmVhc3NpZ24gdGhlIHJlc3VsdCBiYWNrIHRvIGB4YC4gCgpgYGB7ciB2YXItbW9kaWZ5LCBsaXZlID0gVFJVRX0KeCA8LSB4ICsgMgoKeApgYGAKCiMjIyBWYXJpYWJsZSBuYW1pbmcgbm90ZToKQXMgYmVzdCB5b3UgY2FuLCBpdCBpcyBhIGdvb2QgaWRlYSB0byBtYWtlIHlvdXIgdmFyaWFibGUgbmFtZXMgaW5mb3JtYXRpdmUgKGUuZy4gYHhgIGRvZXNuJ3QgbWVhbiBhbnl0aGluZywgYnV0IGBzYW5kd2ljaF9wcmljZWAgaXMgbWVhbmluZ2Z1bC4uLiBpZiB3ZSdyZSB0YWxraW5nIGFib3V0IHRoZSBjb3N0IG9mIHNhbmR3aWNoZXMsIHRoYXQgaXMuLikuIAoKIyMjIENvbW1lbnRzCgpBcmd1YWJseSB0aGUgX19tb3N0IGltcG9ydGFudF9fIGFzcGVjdCBvZiB5b3VyIGNvZGluZyBpcyBjb21tZW50czogU21hbGwgcGllY2VzIG9mIGV4cGxhbmF0b3J5IHRleHQgeW91IGxlYXZlIGluIHlvdXIgY29kZSB0byBleHBsYWluIHdoYXQgdGhlIGNvZGUgaXMgZG9pbmcgYW5kL29yIGxlYXZlIG5vdGVzIHRvIHlvdXJzZWxmIG9yIG90aGVycy4gCkNvbW1lbnRzIGFyZSBpbnZhbHVhYmxlIGZvciBjb21tdW5pY2F0aW5nIHlvdXIgY29kZSB0byBvdGhlcnMsIGJ1dCB0aGV5IGFyZSBtb3N0IGltcG9ydGFudCBmb3IgKipGdXR1cmUgWW91KiouIApGdXR1cmUgWW91IGNvbWVzIGludG8gZXhpc3RlbmNlIGFib3V0IG9uZSBzZWNvbmQgYWZ0ZXIgeW91IHdyaXRlIGNvZGUsIGFuZCBoYXMgbm8gaWRlYSB3aGF0IG9uIGVhcnRoIFBhc3QgWW91IHdhcyB0aGlua2luZy4gCgpDb21tZW50cyBpbiBSIGNvZGUgYXJlIGluZGljYXRlZCB3aXRoIHBvdW5kIHNpZ25zICgqYWthKiBoYXNodGFncywgb2N0b3Rob3JwcykuIFIgd2lsbCBfaWdub3JlXyBhbnkgdGV4dCBpbiBhIGxpbmUgYWZ0ZXIgdGhlIHBvdW5kIHNpZ24sIHNvIHlvdSBjYW4gcHV0IHdoYXRldmVyIHRleHQgeW91IGxpa2UgdGhlcmUuCgpgYGB7ciBjb21tZW50c30KMjIvNyAjIG5vdCBxdWl0ZSBwaQoKIyBJZiB3ZSBuZWVkIGEgYmV0dGVyIGFwcHJveGltYXRpb24gb2YgcGksIHdlIGNhbiB1c2UgRXVsZXIncyBmb3JtdWxhCiMgVGhpcyB1c2VzIGF0YW4oKSwgd2hpY2ggY2FsY3VsYXRlcyBhcmN0YW5nZW50LgoyMCAqIGF0YW4oMS83KSArIDggKiBhdGFuKDMvNzkpIApgYGAKCkhlbHAgb3V0IEZ1dHVyZSBZb3UgYnkgYWRkaW5nIGxvdHMgb2YgY29tbWVudHMhIApGdXR1cmUgWW91IG5leHQgd2VlayB0aGlua3MgVG9kYXkgWW91IGlzIGFuIGlkaW90LCBhbmQgdGhlIG9ubHkgd2F5IHlvdSBjYW4gY29udmluY2UgRnV0dXJlIFlvdSB0aGF0IFRvZGF5IFlvdSBpcyByZWFzb25hYmx5IGNvbXBldGVudCBpcyBieSBhZGRpbmcgY29tbWVudHMgaW4geW91ciBjb2RlIGV4cGxhaW5pbmcgd2h5IFRvZGF5IFlvdSBpcyBhY3R1YWxseSBub3Qgc28gYmFkLgoKIyMgRnVuY3Rpb25zCldlIGNhbiB1c2UgcHJlLWJ1aWx0IGNvbXB1dGF0aW9uIG1ldGhvZHMgY2FsbGVkICJmdW5jdGlvbnMiIGZvciBvdGhlciBvcGVyYXRpb25zLiAKRnVuY3Rpb25zIGhhdmUgdGhlIGZvbGxvd2luZyBmb3JtYXQsIHdoZXJlIHRoZSBfYXJndW1lbnRfIGlzIHRoZSBpbmZvcm1hdGlvbiB3ZSBhcmUgcHJvdmlkaW5nIHRvIHRoZSBmdW5jdGlvbiBmb3IgaXQgdG8gcnVuLiAKQW4gZXhhbXBsZSBvZiB0aGlzIHdhcyB0aGUgYGF0YW4oKWAgZnVuY3Rpb24gdXNlZCBhYm92ZS4KCmBgYHIKZnVuY3Rpb25fbmFtZShhcmd1bWVudCkKYGBgCgpUbyBsZWFybiBhYm91dCBmdW5jdGlvbnMsIHdlJ2xsIGV4YW1pbmUgb25lIGNhbGxlZCBgbG9nKClgIGZpcnN0LiAKClRvIGtub3cgd2hhdCBhIGZ1bmN0aW9uIGRvZXMgYW5kIGhvdyB0byB1c2UgaXQsIHVzZSB0aGUgcXVlc3Rpb24gbWFyayB3aGljaCB3aWxsIHJldmVhbCBkb2N1bWVudGF0aW9uIGluIHRoZSAqKmhlbHAgcGFuZSoqOiBgP2xvZ2AKIVtyaGVscF0oc2NyZWVuc2hvdHMvcmhlbHAtbG9nLnBuZykgCgpUaGUgZG9jdW1lbnRhdGlvbiB0ZWxscyB1cyB0aGF0IGBsb2coKWAgaXMgZGVyaXZlZCBmcm9tIGB7YmFzZX1gLCBtZWFuaW5nIGl0IGlzIGEgZnVuY3Rpb24gdGhhdCBpcyBwYXJ0IG9mIGJhc2UgUi4gCkl0IHByb3ZpZGVzIGEgYnJpZWYgZGVzY3JpcHRpb24gb2Ygd2hhdCB0aGUgZnVuY3Rpb24gZG9lcyBhbmQgc2hvd3Mgc2V2ZXJhbCBleGFtcGxlcyBvZiB0byBob3cgdXNlIGl0LgoKSW4gcGFydGljdWxhciwgZG9jdW1lbnRhdGlvbiB0ZWxscyBhYm91dCBob3cgd2hhdCBhcmd1bWVudChzKSB0byBwcm92aWRlOgorIFRoZSBmaXJzdCBfcmVxdWlyZWRfIGFyZ3VtZW50IGlzIHRoZSB2YWx1ZSB3ZSdkIGxpa2UgdG8gdGFrZSB0aGUgbG9nIG9mLCBieSBkZWZhdWx0IGl0cyBfbmF0dXJhbCBsb2dfCisgVGhlIHNlY29uZCBfb3B0aW9uYWxfIGFyZ3VtZW50IGNhbiBzcGVjaWZ5IGEgZGlmZmVyZW50IGJhc2UgcmF0aGVyIHRoYW4gdGhlIGRlZmF1bHQgYGVgLgoKRnVuY3Rpb25zIGFsc28gX3JldHVybl8gdmFsdWVzIGZvciB1cyB0byB1c2UuIApJbiB0aGUgY2FzZSBvZiBgbG9nKClgLCB0aGUgcmV0dXJuZWQgdmFsdWUgaXMgdGhlIGxvZydkIHZhbHVlIHRoZSBmdW5jdGlvbiBjb21wdXRlZC4KCmBgYHtyIGxvZ30KbG9nKDczKQpgYGAKCkhlcmUgd2UgY2FuIHNwZWNpZnkgYW4gX2FyZ3VtZW50XyBvZiBgYmFzZWAgdG8gY2FsY3VsYXRlIGxvZyBiYXNlIDMuIAoKYGBge3IgbG9nM30KbG9nKDgxLCBiYXNlID0gMykKYGBgCgpJZiB3ZSBkb24ndCBzcGVjaWZ5IHRoZSBfYXJndW1lbnRfIG5hbWVzLCBpdCBhc3N1bWVzIHRoZXkgYXJlIGluIHRoZSBvcmRlciB0aGF0IGBsb2dgIGRlZmluZXMgdGhlbS4gClNlZSBgP2xvZ2AgdG8gc2VlIG1vcmUgYWJvdXQgaXRzIGFyZ3VtZW50cy4gCgpgYGB7ciBsb2cyLCBsaXZlID0gVFJVRX0KbG9nKDgsIDIpCmBgYAoKV2UgY2FuIHN3aXRjaCB0aGUgb3JkZXIgaWYgd2Ugc3BlY2lmeSB0aGUgYXJndW1lbnQgbmFtZXMuIAoKYGBge3IgbG9nLW9yZGVyfQpsb2coYmFzZSA9IDEwLCB4ID0gNDM0MikKYGBgCgpXZSBjYW4gYWxzbyBwcm92aWRlIHZhcmlhYmxlcyBhcyBhcmd1bWVudHMgaW4gdGhlIHNhbWUgd2F5IGFzIHRoZSByYXcgdmFsdWVzLiAKCmBgYHtyIGxvZy12YXJpYWJsZX0KbWVhbmluZyA8LSA0Mgpsb2cobWVhbmluZykKYGBgCgojIyBXb3JraW5nIHdpdGggdmFyaWFibGVzCgojIyMgVmFyaWFibGUgVHlwZXMKClZhcmlhYmxlIHR5cGVzIGluIFIgY2FuIHNvbWV0aW1lcyBiZSBfY29lcmNlZF8gKGNvbnZlcnRlZCkgZnJvbSBvbmUgdHlwZSB0byBhbm90aGVyLgoKYGBge3J9CiMgRGVmaW5lIGEgdmFyaWFibGUgd2l0aCBhIG51bWJlcgp4IDwtIDE1CmBgYAoKVGhlIGZ1bmN0aW9uIGBjbGFzcygpYCB3aWxsIHRlbGwgdXMgdGhlIHZhcmlhYmxlJ3MgdHlwZS4KCmBgYHtyfQpjbGFzcyh4KQpgYGAKCkxldCdzIGNvZXJjZSBpdCB0byBhIGNoYXJhY3Rlci4gCgpgYGB7cn0KeCA8LSBhcy5jaGFyYWN0ZXIoeCkKY2xhc3MoeCkKYGBgCgpTZWUgaXQgbm93IGhhcyBxdW90ZXMgYXJvdW5kIGl0PyBJdCdzIG5vdyBhIGNoYXJhY3RlciBhbmQgd2lsbCBiZWhhdmUgYXMgc3VjaAoKYGBge3J9CngKYGBgCgpVc2UgdGhpcyBjaHVuayB0byB0cnkgdG8gcGVyZm9ybSBjYWxjdWxhdGlvbnMgd2l0aCBgeGAsIG5vdyB0aGF0IGl0IGlzIGEgY2hhcmFjdGVyLCB3aGF0IGhhcHBlbnM/IAoKYGBge3IgbGl2ZSA9IFRSVUV9CiMgVHJ5IHRvIHBlcmZvcm0gY2FsY3VsYXRpb25zIG9uIGB4YApgYGAKCkJ1dCB3ZSBjYW4ndCBjb2VyY2UgZXZlcnl0aGluZzoKCmBgYHtyfQojIExldCdzIGNyZWF0ZSBhIGNoYXJhY3RlciB2YXJpYWJsZQp4IDwtICJsb29rIGF0IG15IGNoYXJhY3RlciB2YXJpYWJsZSIKYGBgCgpMZXQncyB0cnkgbWFraW5nIHRoaXMgYSBudW1lcmljIHZhcmlhYmxlOgoKYGBge3IgY29lcmNlLWNoYXIsIGVycm9yPVRSVUV9CnggPC0gYXMubnVtZXJpYyh4KQpgYGAKClByaW50IG91dCBgeGAuCgpgYGB7cn0KeApgYGAKClIgaXMgdGVsbGluZyB1cyBpdCBkb2Vzbid0IGtub3cgaG93IHRvIGNvbnZlcnQgdGhpcyB0byBhIG51bWVyaWMgdmFyaWFibGUsIHNvIGl0IGhhcyByZXR1cm5lZCBgTkFgIGluc3RlYWQuCgpGb3IgcmVmZXJlbmNlLCBoZXJlJ3MgYSBzdW1tYXJ5IG9mIHNvbWUgb2YgdGhlIG1vc3QgaW1wb3J0YW50IHZhcmlhYmxlIHR5cGVzLiAKCnwgVmFyaWFibGUgVHlwZSB8IERlZmluaXRpb24gfCBFeGFtcGxlcyB8IENvZXJjaW9uIHwKfC0tLS0tLS0tLS0tLS0tLXwtLS0tLS0tLS0tLS18LS0tLS0tLS0tLXwgLS0tLS0tLS18CnwgYG51bWVyaWNgICAgICAgIHwgQW55IG51bWJlciB2YWx1ZSB8IGA1YDxicj5gNy41YCA8YnI+YC0xYHwgYGFzLm51bWVyaWMoKWAKfCBgaW50ZWdlcmAgICAgICAgfCBBbnkgX3dob2xlXyBudW1iZXIgdmFsdWUgKG5vIGRlY2ltYWxzKSB8IGA1YCA8YnI+IGAtMTAwYCB8IGBhcy5pbnRlZ2VyKClgCnxgY2hhcmFjdGVyYCAgICAgIHwgQW55IGNvbGxlY3Rpb24gb2YgY2hhcmFjdGVycyBkZWZpbmVkIHdpdGhpbiBfcXVvdGF0aW9uIG1hcmtzXy4gQWxzbyBrbm93biBhcyBhICJzdHJpbmciLiB8IGAiYSJgIChhIHNpbmdsZSBsZXR0ZXIpIDxicj5gInN0cmluZ29mbGV0dGVycyJgIChhIHdob2xlIGJ1bmNoIG9mIGNoYXJhY3RlcnMgcHV0IHRvZ2V0aGVyIGFzIG9uZSkgPGJyPiBgInN0cmluZyBvZiBsZXR0ZXJzIGFuZCBzcGFjZXMiYCA8YnI+IGAiNSJgIDxicj4gYCdzaW5nbGUgcXVvdGVzIGFyZSBhbHNvIGdvb2QnYCB8IGBhcy5jaGFyYWN0ZXIoKWAKfGBsb2dpY2FsYCAgICAgIHwgQSB2YWx1ZSBvZiBgVFJVRWAsIGBGQUxTRWAsIG9yIGBOQWAgfCBgVFJVRWAgPGJyPiBgRkFMU0VgIDxicj4gYE5BYCAobm90IGRlZmluZWQpIHwgYGFzLmxvZ2ljYWwoKWAgCnxgZmFjdG9yYCAgICAgICB8IEEgc3BlY2lhbCB0eXBlIG9mIHZhcmlhYmxlIHRoYXQgZGVub3RlcyBzcGVjaWZpYyBjYXRlZ29yaWVzIG9mIGEgY2F0ZWdvcmljYWwgdmFyaWFibGUgfCAoc3RheSB0dW5lZC4uKSB8IGBhcy5mYWN0b3IoKWAKCiMjIyBWZWN0b3JzCgpZb3Ugd2lsbCBoYXZlIG5vdGljZWQgdGhhdCBhbGwgeW91ciBjb21wdXRhdGlvbnMgdGVuZCB0byBwb3AgdXAgd2l0aCBhIGBbMV1gIHByZWNlZGluZyB0aGVtIGluIFIncyBvdXRwdXQuIApUaGlzIGlzIGJlY2F1c2UsIGluIGZhY3QsIGFsbCAob2sgbW9zdGx5IGFsbCkgdmFyaWFibGVzIGFyZSBfYnkgZGVmYXVsdF8gIHZlY3RvcnMsIGFuZCBvdXIgYW5zd2VycyBhcmUgdGhlIGZpcnN0IChpbiB0aGVzZSBjYXNlcyBvbmx5KSB2YWx1ZSBpbiB0aGUgdmVjdG9yLiAKQXMgdmVjdG9ycyBnZXQgbG9uZ2VyLCBuZXcgaW5kZXggaW5kaWNhdG9ycyB3aWxsIGFwcGVhciBhdCB0aGUgc3RhcnQgb2YgbmV3IGxpbmVzLiAKCmBgYHtyfQojIFRoaXMgaXMgYWN0dWFsbHkgYW4gdmVjdG9yIHRoYXQgaGFzIG9uZSBpdGVtIGluIGl0Lgp4IDwtIDcKYGBgCgpgYGB7ciB2ZWN0b3ItbGVuZ3RofQojIFRoZSBsZW5ndGgoKSBmdW5jdGlvbnMgdGVsbHMgdXMgaG93IGxvbmcgYW4gdmVjdG9yIGlzOgpsZW5ndGgoeCkKYGBgCgpXZSBjYW4gZGVmaW5lIHZlY3RvcnMgd2l0aCB0aGUgZnVuY3Rpb24gYGMoKWAsIHdoaWNoIHN0YW5kcyBmb3IgImNvbWJpbmUiLiAKVGhpcyBmdW5jdGlvbiB0YWtlcyBhIGNvbW1hLXNlcGFyYXRlZCBzZXQgb2YgdmFsdWVzIHRvIHBsYWNlIGluIHRoZSB2ZWN0b3IsIGFuZCByZXR1cm5zIHRoZSB2ZWN0b3IgaXRzZWxmOgoKYGBge3IgbWFrZS12ZWN0b3J9Cm15X251bWVyaWNfdmVjdG9yIDwtIGMoMSwgMSwgMiwgMywgNSwgOCwgMTMsIDIxKQpteV9udW1lcmljX3ZlY3RvcgpgYGAKCldlIGNhbiBidWlsZCBvbiB2ZWN0b3JzIGluIHBsYWNlIGJ5IHJlZGVmaW5pbmcgdGhlbQoKYGBge3IgZmliYm9uYWNjaSwgbGl2ZSA9IFRSVUV9CiMgYWRkIHRoZSBuZXh0IHR3byBGaWJvbmFjY2kgbnVtYmVycyB0byB0aGUgc2VyaWVzLgpteV9udW1lcmljX3ZlY3RvciA8LSBjKG15X251bWVyaWNfdmVjdG9yLCAzNCwgNTUpCm15X251bWVyaWNfdmVjdG9yCmBgYAoKV2UgY2FuIHB1bGwgb3V0IHNwZWNpZmljIGl0ZW1zIGZyb20gYW4gdmVjdG9yIHVzaW5nIGEgcHJvY2VzcyBjYWxsZWQgX2luZGV4aW5nXywgd2hpY2ggdXNlcyBicmFja2V0cyBgW11gIHRvIHNwZWNpZnkgdGhlIHBvc2l0aW9uIG9mIGFuIGl0ZW0uIAoKYGBge3Igc3Vic2V0MX0KIyBHcmFiIHRoZSBmb3VydGggdmFsdWUgZnJvbSBteV9udW1lcmljX3ZlY3RvcgojIFRoaXMgZ2l2ZXMgdXMgYW4gdmVjdG9yIG9mIGxlbmd0aCAxIApteV9udW1lcmljX3ZlY3Rvcls0XQpgYGAKCkNvbG9ucyBhcmUgYWxzbyBhIG5pY2Ugd2F5IHRvIHF1aWNrbHkgbWFrZSBvcmRlcmVkIG51bWVyaWMgdmVjdG9ycwpVc2UgYSBjb2xvbiB0byBzcGVjaWZ5IGFuIGluY2x1c2l2ZSByYW5nZSBvZiBpbmRpY2VzClRoaXMgd2lsbCByZXR1cm4gYW4gdmVjdG9yIHdpdGggMiwgMywgNCwgYW5kIDUuCgpgYGB7ciBzdWJzZXQtbWFueX0KbXlfbnVtZXJpY192ZWN0b3JbMjo1XQpgYGAKCk9uZSBtYWpvciBiZW5lZml0IG9mIHZlY3RvcnMgaXMgdGhlIGNvbmNlcHQgb2YgKip2ZWN0b3JpemF0aW9uKiosIHdoZXJlIFIgYnkgZGVmYXVsdCBwZXJmb3JtcyBvcGVyYXRpb25zIG9uIHRoZSBfZW50aXJlIHZlY3RvciBhdCBvbmNlXy4gCkZvciBleGFtcGxlLCB3ZSBjYW4gZ2V0IHRoZSBsb2cgb2YgYWxsIG51bWJlcnMgMS0yIHdpdGggYSBzaW5nbGUsIHNpbXBsZSBjYWxsLCBhbmQgbW9yZSEKCmBgYHtyIHZlY3Rvcml6ZX0KdmFsdWVzXzFfdG9fMjAgPC0gMToyMApgYGAKYGBge3IgdmVjdG9yaXplLWxvZywgbGl2ZSA9IFRSVUV9CiMgY2FsY3VsYXRlIHRoZSBsb2cgb2YgdmFsdWVzXzFfdG9fMjAKbG9nKHZhbHVlc18xX3RvXzIwKQpgYGAKCkZpbmFsbHksIHdlIGNhbiBhcHBseSBsb2dpY2FsIGV4cHJlc3Npb25zIHRvIHZlY3RvcnMsIGp1c3QgYXMgd2UgY2FuIGRvIGZvciBzaW5nbGUgdmFsdWVzLgpUaGUgb3V0cHV0IGhlcmUgaXMgYSBsb2dpY2FsIHZlY3RvciB0ZWxsaW5nIHVzIHdoZXRoZXIgZWFjaCB2YWx1ZSBpbiBleGFtcGxlX3ZlY3RvciBpcyBUUlVFIG9yIEZBTFNFCgpgYGB7ciB2ZWN0b3ItY29tcGFyZX0KIyBXaGljaCB2YWx1ZXMgYXJlIDw9IDM/CnZhbHVlc18xX3RvXzIwIDw9IDMKYGBgCgpUaGVyZSBhcmUgc2V2ZXJhbCBrZXkgZnVuY3Rpb25zIHdoaWNoIGNhbiBiZSB1c2VkIG9uIHZlY3RvcnMgY29udGFpbmluZyBudW1lcmljIHZhbHVlcywgc29tZSBvZiB3aGljaCBhcmUgYmVsb3cuCgorIGBtZWFuKClgOiBUaGUgYXZlcmFnZSB2YWx1ZSBpbiB0aGUgdmVjdG9yCisgYG1pbigpYDogVGhlIG1pbmltdW0gdmFsdWUgaW4gdGhlIHZlY3RvcgorIGBtYXgoKWA6IFRoZSBtYXhpbXVtIHZhbHVlIGluIHRoZSB2ZWN0b3IKKyBgc3VtKClgOiBUaGUgc3VtIG9mIGFsbCB2YWx1ZXMgaW4gdGhlIHZlY3RvcgoKV2UgY2FuIHRyeSBvdXQgdGhlc2UgZnVuY3Rpb25zIG9uIHRoZSB2ZWN0b3IgYHZhbHVlc18xX3RvXzIwYCB3ZSd2ZSBjcmVhdGVkLiAKCmBgYHtyIHZlY3Rvci1mdW5jc30KbWVhbih2YWx1ZXNfMV90b18yMCkKCiMgVHJ5IG91dCBzb21lIG9mIHRoZSBvdGhlciBmdW5jdGlvbnMgd2UndmUgbGlzdGVkIGFib3ZlIAoKYGBgCgojIyMgQSBub3RlIG9uIHZhcmlhYmxlIG5hbWluZwoKV2UgaGF2ZSBsZWFybmVkIGZ1bmN0aW9ucyBzdWNoIGFzIGBjYCwgYGxlbmd0aGAsIGBzdW1gLCBhbmQgZXRjLiAKSW1hZ2luZSBkZWZpbmluZyBhIHZhcmlhYmxlIGNhbGxlZCBgY2A6IFRoaXMgd2lsbCB3b3JrLCBidXQgaXQgd2lsbCBsZWFkIHRvIGEgCmxvdCBvZiB1bmludGVuZGVkIGJ1Z3MsIHNvIGl0cyBiZXN0IHRvIGF2b2lkIHRoaXMuIAoKIyMjIFRoZSBgJWluJWAgbG9naWNhbCBvcGVyYXRvciAKCmAlaW4lYCBpcyB1c2VmdWwgZm9yIGRldGVybWluaW5nIHdoZXRoZXIgYSBnaXZlbiBpdGVtKHMpIGFyZSBpbiBhbiB2ZWN0b3IuCgpgYGB7ciBpbi1vcGVyYXRvcn0KIyBpcyBgN2AgaW4gb3VyIHZlY3Rvcj8gCjcgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCmBgYHtyIGluMiwgbGl2ZSA9IFRSVUV9CiMgaXMgYDUwYCBpbiBvdXIgdmVjdG9yPyAKNTAgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCldlIGNhbiB0ZXN0IGEgdmVjdG9yIG9mIHZhbHVlcyBiZWluZyB3aXRoaW4gYW5vdGhlciB2ZWN0b3Igb2YgdmFsdWVzLiAKCmBgYHtyIHZlY3Rvci1pbiwgbGl2ZSA9IFRSVUV9CnF1ZXN0aW9uX3ZhbHVlcyA8LSBjKDE6MywgNywgNTApCiMgQXJlIHRoZXNlIHZhbHVlcyBpbiBvdXIgdmVjdG9yPwpxdWVzdGlvbl92YWx1ZXMgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCiMjIERhdGEgZnJhbWVzCgpfRGF0YSBmcmFtZXMgYXJlIG9uZSBvZiB0aGUgbW9zdCB1c2VmdWwgdG9vbHMgZm9yIGRhdGEgYW5hbHlzaXMgaW4gUi5fIApUaGV5IGFyZSB0YWJsZXMgd2hpY2ggY29uc2lzdCBvZiByb3dzIGFuZCBjb2x1bW5zLCBtdWNoIGxpa2UgYSBfc3ByZWFkc2hlZXRfLiAKRWFjaCBjb2x1bW4gaXMgYSB2YXJpYWJsZSB3aGljaCBiZWhhdmVzIGFzIGEgX3ZlY3Rvcl8sIGFuZCBlYWNoIHJvdyBpcyBhbiBvYnNlcnZhdGlvbi4gCldlIHdpbGwgYmVnaW4gb3VyIGV4cGxvcmF0aW9uIHdpdGggZGF0YXNldCBvZiBtZWFzdXJlbWVudHMgZnJvbSB0aHJlZSBwZW5ndWluIHNwZWNpZXMgbWVhc3VyZWQsIHdoaWNoIHdlIGNhbiBmaW5kIGluIHRoZSBbYHBhbG1lcnBlbmd1aW5zYCBwYWNrYWdlXShodHRwczovL2FsbGlzb25ob3JzdC5naXRodWIuaW8vcGFsbWVycGVuZ3VpbnMvKS4gCldlJ2xsIHRhbGsgbW9yZSBhYm91dCBwYWNrYWdlcyBzb29uIQpUbyB1c2UgdGhpcyBkYXRhc2V0LCB3ZSB3aWxsIGxvYWQgaXQgZnJvbSB0aGUgYHBhbG1lcnBlbmd1aW5zYCBwYWNrYWdlIHVzaW5nIGEgYDo6YCAobW9yZSBvbiB0aGlzIGxhdGVyKSBhbmQgYXNzaWduIGl0IHRvIGEgdmFyaWFibGUgbmFtZWQgYHBlbmd1aW5zYCBpbiBvdXIgY3VycmVudCBlbnZpcm9ubWVudC4KCmBgYHtyIHBlbmd1aW4tbGlicmFyeX0KcGVuZ3VpbnMgPC0gcGFsbWVycGVuZ3VpbnM6OnBlbmd1aW5zCmBgYAoKIVtkcmF3aW5ncyBvZiBwZW5ndWluIHNwZWNpZXNdKGRpYWdyYW1zL2x0ZXJfcGVuZ3VpbnMucG5nKSBBcnR3b3JrIGJ5IEBhbGxpc29uX2hvcnN0CgojIyMgRXhwbG9yaW5nIGRhdGEgZnJhbWVzCgpUaGUgZmlyc3Qgc3RlcCB0byB1c2luZyBhbnkgZGF0YSBpcyB0byBsb29rIGF0IGl0ISEhIApSU3R1ZGlvIGNvbnRhaW5zIGEgc3BlY2lhbCBmdW5jdGlvbiBgVmlldygpYCB3aGljaCBhbGxvd3MgeW91IHRvIGxpdGVyYWxseSB2aWV3IGEgdmFyaWFibGUuCllvdSBjYW4gY2xpY2sgb24gdGhlIG9iamVjdCBpbiB0aGUgZW52aXJvbm1lbnQgcGFuZS4gCgpTb21lIHVzZWZ1bCBmdW5jdGlvbnMgZm9yIGV4cGxvcmluZyBvdXIgZGF0YSBmcmFtZSBpbmNsdWRlOgoKKyBgaGVhZCgpYCB0byBzZWUgdGhlIGZpcnN0IDYgcm93cyBvZiBhIGRhdGEgZnJhbWUuIEFkZGl0aW9uYWwgYXJndW1lbnRzIHN1cHBsaWVkIGNhbiBjaGFuZ2UgdGhlIG51bWJlciBvZiByb3dzLgorIGB0YWlsKClgIHRvIHNlZSB0aGUgbGFzdCA2IHJvd3Mgb2YgYSBkYXRhIGZyYW1lLiBBZGRpdGlvbmFsIGFyZ3VtZW50cyBzdXBwbGllZCBjYW4gY2hhbmdlIHRoZSBudW1iZXIgb2Ygcm93cy4KKyBgbmFtZXMoKWAgdG8gc2VlIHRoZSBjb2x1bW4gbmFtZXMgb2YgdGhlIGRhdGEgZnJhbWUuCisgYG5yb3coKWAgdG8gc2VlIGhvdyBtYW55IHJvd3MgYXJlIGluIHRoZSBkYXRhIGZyYW1lCisgYG5jb2woKWAgdG8gc2VlIGhvdyBtYW55IGNvbHVtbnMgYXJlIGluIHRoZSBkYXRhIGZyYW1lLgoKV2UgY2FuIGFkZGl0aW9uYWxseSBleHBsb3JlIF9vdmVyYWxsIHByb3BlcnRpZXNfIG9mIHRoZSBkYXRhIGZyYW1lIHdpdGggdHdvIGRpZmZlcmVudCBmdW5jdGlvbnM6IGBzdW1tYXJ5KClgIGFuZCBgc3RyKClgLgoKVGhpcyBwcm92aWRlcyBzdW1tYXJ5IHN0YXRpc3RpY3MgZm9yIGVhY2ggY29sdW1uOgoKYGBge3IgcGVuZ3VpbnMtc3VtbWFyeX0Kc3VtbWFyeShwZW5ndWlucykKYGBgCgpUaGlzIHByb3ZpZGVzIGEgc2hvcnQgdmlldyBvZiB0aGUgKipzdHIqKnVjdHVyZSBhbmQgY29udGVudHMgb2YgdGhlIGRhdGEgZnJhbWUuCgpgYGB7ciBwZW5ndWlucy1zdHJ9CnN0cihwZW5ndWlucykKYGBgCgpZb3UnbGwgbm90aWNlIHRoYXQgdGhlIGNvbHVtbiBgc3BlY2llc2AgaXMgYSBfZmFjdG9yXzogVGhpcyBpcyBhIHNwZWNpYWwgdHlwZSBvZiBjaGFyYWN0ZXIgdmFyaWFibGUgdGhhdCByZXByZXNlbnRzIGRpc3RpbmN0IGNhdGVnb3JpZXMga25vd24gYXMgImxldmVscyIuIApXZSBoYXZlIGxlYXJuZWQgaGVyZSB0aGF0IHRoZXJlIGFyZSB0aHJlZSBsZXZlbHMgaW4gdGhlIGBzcGVjaWVzYCBjb2x1bW46IEFkZWxpZSwgQ2hpbnN0cmFwLCBhbmQgR2VudG9vLgpXZSBtaWdodCB3YW50IHRvIGV4cGxvcmUgaW5kaXZpZHVhbCBjb2x1bW5zIG9mIHRoZSBkYXRhIGZyYW1lIG1vcmUgaW4tZGVwdGguIApXZSBjYW4gZXhhbWluZSBpbmRpdmlkdWFsIGNvbHVtbnMgdXNpbmcgdGhlIGRvbGxhciBzaWduIGAkYCB0byBzZWxlY3Qgb25lIGJ5IG5hbWU6CgpgYGB7ciBwZW5ndWlucy1zdWJzZXR9CiMgRXh0cmFjdCBiaWxsX2xlbmd0aF9tbSBhcyBhIHZlY3RvcgpwZW5ndWlucyRiaWxsX2xlbmd0aF9tbQoKIyBpbmRleGluZyBvcGVyYXRvcnMgY2FuIGJlIHVzZWQgdG9vCnBlbmd1aW5zJGJpbGxfZGVwdGhfbW1bMToxMF0KYGBgCgpXZSBjYW4gcGVyZm9ybSBvdXIgcmVndWxhciB2ZWN0b3Igb3BlcmF0aW9ucyBvbiBjb2x1bW5zIGRpcmVjdGx5LgoKYGBge3IgcGVuZ3VpbnMtY29sLW1lYW4sIGxpdmUgPSBUUlVFfQojIGNhbGN1bGF0ZSB0aGUgbWVhbiBvZiB0aGUgYmlsbF9sZW5ndGhfbW0gY29sdW1uCm1lYW4ocGVuZ3VpbnMkYmlsbF9sZW5ndGhfbW0sCiAgICAgbmEucm0gPSBUUlVFKSAjIHJlbW92ZSBtaXNzaW5nIHZhbHVlcyBiZWZvcmUgY2FsY3VsYXRpbmcgdGhlIG1lYW4KYGBgCgpXZSBjYW4gYWxzbyBjYWxjdWxhdGUgdGhlIGZ1bGwgc3VtbWFyeSBzdGF0aXN0aWNzIGZvciBhIHNpbmdsZSBjb2x1bW4gZGlyZWN0bHkuIAoKYGBge3IgcGVuZ3VpbnMtY29sLXN1bW1hcnksIGxpdmUgPSBUUlVFfQojIHNob3cgYSBzdW1tYXJ5IG9mIHRoZSBiaWxsX2xlbmd0aF9tbSBjb2x1bW4Kc3VtbWFyeShwZW5ndWlucyRiaWxsX2xlbmd0aF9tbSkKYGBgCgpFeHRyYWN0IGBTcGVjaWVzYCBhcyBhIHZlY3RvciBhbmQgc3Vic2V0IGl0IHRvIHNlZSBhIHByZXZpZXcuCgpgYGB7ciBwZW5ndWlucy1jb2wtc3Vic2V0LCBsaXZlID0gVFJVRX0KIyBnZXQgdGhlIGZpcnN0IDEwIHZhbHVlcyBvZiB0aGUgU3BlY2llcyBjb2x1bW4KcGVuZ3VpbnMkc3BlY2llc1sxOjEwXQpgYGAKCkFuZCB2aWV3IGl0cyBfbGV2ZWxzXyB3aXRoIHRoZSBgbGV2ZWxzKClgIGZ1bmN0aW9uLgoKYGBge3IgcGVuZ3Vpbi1sZXZlbHN9CmxldmVscyhwZW5ndWlucyRzcGVjaWVzKQpgYGAKCiMjIEZpbGVzIGFuZCBkaXJlY3RvcmllcwoKSW4gbWFueSBzaXR1YXRpb25zLCB3ZSB3aWxsIGJlIHJlYWRpbmcgaW4gdGFidWxhciBkYXRhIGZyb20gYSBmaWxlIGFuZCB1c2luZyBpdCBhcyBhIGRhdGEgZnJhbWUuIApUbyBwcmFjdGljZSwgd2Ugd2lsbCByZWFkIGluIGEgZmlsZSB3ZSB3aWxsIGJlIHVzaW5nIGluIHRoZSBuZXh0IG5vdGVib29rIGFzIHdlbGwsIGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YCwgaW4gdGhlIGBkYXRhYCBmb2xkZXIuIApGaWxlIHBhdGhzIGFyZSByZWxhdGl2ZSB0byB0aGUgbG9jYXRpb24gd2hlcmUgdGhpcyBub3RlYm9vayBmaWxlICguUm1kKSBpcyBzYXZlZC4KCkhlcmUgd2Ugd2lsbCB1c2UgYSBmdW5jdGlvbiwgYHJlYWRfdHN2KClgIGZyb20gdGhlIGByZWFkcmAgcGFja2FnZS4KQmVmb3JlIHdlIGFyZSBhYmxlIHRvIHVzZSB0aGUgZnVuY3Rpb24sIHdlIGhhdmUgdG8gbG9hZCB0aGUgcGFja2FnZSB1c2luZyBgbGlicmFyeSgpYC4gCgpgYGB7ciByZWFkcn0KbGlicmFyeShyZWFkcikKYGBgCgpgZmlsZS5wYXRoKClgIGNyZWF0ZXMgYSBwcm9wZXJseSBmb3JtYXR0ZWQgZmlsZSBwYXRoIGJ5IGFkZGluZyBhIHBhdGggc2VwYXJhdG9yIChgL2Agb24gTWFjIGFuZCBMaW51eCBvcGVyYXRpbmcgc3lzdGVtLCB3aGljaCBpcyB0aGUgb3BlcmF0aW5nIHN5c3RlbSB0aGF0IG91ciBSU3R1ZGlvIFNlcnZlciBydW5zIG9uKSBiZXR3ZWVuIHNlcGFyYXRlIGZvbGRlcnMgb3IgZGlyZWN0b3JpZXMuCkJlY2F1c2UgZmlsZSBwYXRoIHNlcGFyYXRvcnMgY2FuIGRpZmZlciBiZXR3ZWVuIHlvdXIgY29tcHV0ZXIgYW5kIHRoZSBjb21wdXRlciBvZiBzb21lb25lIHdobyB3YW50cyB0byB1c2UgeW91ciBjb2RlLCB3ZSB1c2UgYGZpbGUucGF0aCgpYCBpbnN0ZWFkIG9mIHR5cGluZyBvdXQgYCJkYXRhL2dlbmVfcmVzdWx0c19HU0U0NDk3MS50c3YiYC4KRWFjaCBfYXJndW1lbnRfIHRvIGBmaWxlLnBhdGgoKWAgaXMgYSBkaXJlY3Rvcnkgb3IgZmlsZSBuYW1lLgpZb3UnbGwgbm90aWNlIGVhY2ggYXJndW1lbnQgaXMgaW4gcXVvdGVzLCB3ZSBzcGVjaWZ5IGBkYXRhYCBmaXJzdCBiZWNhdXNlIHRoZSBmaWxlLCBgZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdmAgaXMgaW4gdGhlIGBkYXRhYCBmb2xkZXIuIAoKYGBge3IgZmlsZS5wYXRofQpmaWxlLnBhdGgoImRhdGEiLCAiZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdiIpCmBgYAoKV2UgY2FuIHN0b3JlIHRoaXMgZmlsZSBwYXRoIGFzIGEgdmFyaWFibGUgaW4gb3VyIGVudmlyb25tZW50LiAKCmBgYHtyIGZpbGUucGF0aC12YXJpYWJsZX0KZ2VuZV9maWxlX3BhdGggPC0gZmlsZS5wYXRoKCJkYXRhIiwgImdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3YiKQpgYGAKCk5vdyB3ZSBhcmUgcmVhZHkgdG8gdXNlIGByZWFkX3RzdigpYCB0byByZWFkIHRoZSBmaWxlIGludG8gUi4KVGhlIHJlc3VsdGluZyBkYXRhIGZyYW1lIHdpbGwgYmUgc3RvcmVkIGluIGEgdmFyaWFibGUgbmFtZWQgYHN0YXRzX2RmYC4KTm90ZSB0aGUgYDwtYCB3aGljaCBpcyByZXNwb25zaWJsZSBmb3Igc2F2aW5nIHRoaXMgdG8gb3VyIGdsb2JhbCBlbnZpcm9ubWVudC4gCgpgYGB7ciByZWFkLXN0YXRzfQojIHJlYWQgaW4gdGhlIGZpbGUgYGdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3ZgIGZyb20gdGhlIGRhdGEgZGlyZWN0b3J5CnN0YXRzX2RmIDwtIHJlYWRfdHN2KGdlbmVfZmlsZV9wYXRoKQpgYGAKClRha2UgYSBsb29rIGF0IHlvdXIgZW52aXJvbm1lbnQgcGFuZWwgdG8gc2VlIHdoYXQgYHN0YXRzX2RmYCBsb29rcyBsaWtlLiAKV2UgY2FuIGFsc28gcHJpbnQgb3V0IGEgcHJldmlldyBvZiB0aGUgYHN0YXRzX2RmYCBkYXRhIGZyYW1lIGhlcmUuIAoKYGBge3Igc2hvdy1zdGF0cywgbGl2ZSA9IFRSVUV9CiMgZGlzcGxheSBzdGF0c19kZgpzdGF0c19kZgpgYGAKCiMjIyBTZXNzaW9uIEluZm8KCkF0IHRoZSBlbmQgb2YgZXZlcnkgbm90ZWJvb2ssIHlvdSB3aWxsIHNlZSB1cyBwcmludCBvdXQgYHNlc3Npb25JbmZvYC4gClRoaXMgYWlkcyBpbiB0aGUgcmVwcm9kdWNpYmlsaXR5IG9mIHlvdXIgY29kZSBieSBzaG93aW5nIGV4YWN0bHkgd2hhdCBwYWNrYWdlcyAKYW5kIHZlcnNpb25zIHdlcmUgYmVpbmcgdXNlZCB0aGUgbGFzdCB0aW1lIHRoZSBub3RlYm9vayB3YXMgcnVuLgoKYGBge3J9CnNlc3Npb25JbmZvKCkKYGBgCg==
    
+
    LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIFIgYW5kIFJTdHVkaW8iCmF1dGhvcjogT3JpZ2luYWxseSBhdXRob3JlZCBieSBTdGVwaGFuaWUgSi4gU3BpZWxtYW4sPGJyPmFkYXB0ZWQgYnkgQ0NETCBmb3IgQUxTRgpkYXRlOiAyMDIxCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOiAgCgotIE5hdmlnYXRlIHRoZSBSU3R1ZGlvIGVudmlyb25tZW50ICAKLSBVc2UgUiBmb3Igc2ltcGxlIGNhbGN1bGF0aW9ucywgYm90aCBtYXRoZW1hdGljYWwgYW5kIGxvZ2ljYWwgIAotIERlZmluZSBhbmQgdXNlIHZhcmlhYmxlcyBpbiBiYXNlIFIgIAotIFVuZGVyc3RhbmQgYW5kIGFwcGx5IGJhc2UgUiBmdW5jdGlvbnMgICAKLSBVbmRlcnN0YW5kLCBkZWZpbmUsIGFuZCB1c2UgUiBkYXRhIHR5cGVzLCBpbmNsdWRpbmcgdmVjdG9yIG1hbmlwdWxhdGlvbiBhbmQgaW5kZXhpbmcgIAotIFVuZGVyc3RhbmQgdGhlIGFuYXRvbXkgb2YgYSBkYXRhIGZyYW1lICAKCi0tLQoKIyMjIyAqTW9yZSByZXNvdXJjZXMgZm9yIGxlYXJuaW5nIFIqIAoKLSBbU3dpcmwsIGFuIGludGVyYWN0aXZlIHR1dG9yaWFsXShodHRwczovL3N3aXJsc3RhdHMuY29tLykgIAotIFtSIE1hcmtkb3duXShodHRwOi8vcm1hcmtkb3duLnJzdHVkaW8uY29tKSAgCi0gW1IgZm9yIERhdGEgU2NpZW5jZV0oaHR0cHM6Ly9yNGRzLmhhZC5jby5uei8pICAKLSBbVHV0b3JpYWwgb24gUiwgUlN0dWRpbyBhbmQgUiBNYXJrZG93bl0oaHR0cHM6Ly9pc21heWMuZ2l0aHViLmlvL3JiYXNpY3MtYm9vay8pICAKLSBbSGFuZHkgUiBjaGVhdHNoZWV0c10oaHR0cHM6Ly93d3cucnN0dWRpby5jb20vcmVzb3VyY2VzL2NoZWF0c2hlZXRzLykgIAotIFtSIE5vdGVib29rcyB0dXRvcmlhbF0oaHR0cHM6Ly9ib29rZG93bi5vcmcveWlodWkvcm1hcmtkb3duLykgIAoKIyMgV2hhdCBpcyBSPwoKKipSKiogaXMgYSBzdGF0aXN0aWNhbCBjb21wdXRpbmcgbGFuZ3VhZ2UgdGhhdCBpcyBfb3BlbiBzb3VyY2VfLCBtZWFuaW5nIHRoZSB1bmRlcmx5aW5nIGNvZGUgZm9yIHRoZSBsYW5ndWFnZSBpcyBmcmVlbHkgYXZhaWxhYmxlIHRvIGFueW9uZS4gCllvdSBkbyBub3QgbmVlZCBhIHNwZWNpYWwgbGljZW5zZSBvciBzZXQgb2YgcGVybWlzc2lvbnMgdG8gdXNlIGFuZCBkZXZlbG9wIGNvZGUgaW4gUi4gCgpSIGl0c2VsZiBpcyBhbiBfaW50ZXJwcmV0ZWQgY29tcHV0ZXIgbGFuZ3VhZ2VfIGFuZCBjb21lcyB3aXRoIGZ1bmN0aW9uYWxpdHkgdGhhdCBjb21lcyBidW5kbGVkIHdpdGggdGhlIGxhbmd1YWdlIGl0c2VsZiwga25vd24gYXMgKioiYmFzZSBSIioqLgpCdXQgdGhlcmUgaXMgYWxzbyByaWNoIGFkZGl0aW9uYWwgZnVuY3Rpb25hbGl0eSBwcm92aWRlZCBieSAqKmV4dGVybmFsIHBhY2thZ2VzKiosIG9yIGxpYnJhcmllcyBvZiBjb2RlIHRoYXQgYXNzaXN0IGluIGFjY29tcGxpc2hpbmcgY2VydGFpbiB0YXNrcyBhbmQgY2FuIGJlIGZyZWVseSBkb3dubG9hZGVkIGFuZCBsb2FkZWQgZm9yIHVzZS4gCgpJbiB0aGUgbmV4dCBub3RlYm9vayBhbmQgc3Vic2VxdWVudCBtb2R1bGVzLCB3ZSB3aWxsIGJlIHVzaW5nIGEgc3VpdGUgb2YgcGFja2FnZXMgY29sbGVjdGl2ZWx5IGtub3duIGFzIFsqKlRoZSBUaWR5dmVyc2UqKl0oaHR0cDovL3RpZHl2ZXJzZS5vcmcpLiAKVGhlIGB0aWR5dmVyc2VgIGlzIGdlYXJlZCB0b3dhcmRzIGludHVpdGl2ZSBkYXRhIHNjaWVuY2UgYXBwbGljYXRpb25zIHRoYXQgZm9sbG93IGEgc2hhcmVkIGRhdGEgcGhpbG9zb3BoeS4KQnV0IHRoZXJlIGFyZSBzdGlsbCBtYW55IGNvcmUgZmVhdHVyZXMgb2YgYmFzZSBSIHdoaWNoIGFyZSBpbXBvcnRhbnQgdG8gYmUgYXdhcmUgb2YsIGFuZCB3ZSB3aWxsIGJlIHVzaW5nIGNvbmNlcHRzIGZyb20gYm90aCBiYXNlIFIgYW5kIHRoZSB0aWR5dmVyc2UgaW4gb3VyIGFuYWx5c2VzLCBhcyB3ZWxsIGFzIHRhc2sgc3BlY2lmaWMgcGFja2FnZXMgZm9yIGFuYWx5c2VzIHN1Y2ggYXMgZ2VuZSBleHByZXNzaW9uLiAKCiMjIyBXaGF0IGlzIFJTdHVkaW8/CgpSU3R1ZGlvIGlzIGEgX2dyYXBoaWNhbCBlbnZpcm9ubWVudF8gKCJpbnRlZ3JhdGVkIGRldmVsb3BtZW50IGVudmlyb25tZW50IiBvciBJREUpIGZvciB3cml0aW5nIGFuZCBkZXZlbG9waW5nIFIgY29kZS4gUlN0dWRpbyBpcyBOT1QgYSBzZXBhcmF0ZSBwcm9ncmFtbWluZyBsYW5ndWFnZSAtIGl0IGlzIGFuIGludGVyZmFjZSB3ZSB1c2UgdG8gZmFjaWxpdGF0ZSBSIHByb2dyYW1taW5nLiAKSW4gb3RoZXIgd29yZHMsIHlvdSBjYW4gcHJvZ3JhbSBpbiBSIHdpdGhvdXQgUlN0dWRpbywgYnV0IHlvdSBjYW4ndCB1c2UgdGhlIFJTdHVkaW8gZW52aXJvbm1lbnQgd2l0aG91dCBSLgoKRm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgUlN0dWRpbyB0aGFuIHlvdSBldmVyIHdhbnRlZCB0byBrbm93LCBzZWUgdGhpcyBbUlN0dWRpbyBJREUgQ2hlYXRzaGVldF0oaHR0cHM6Ly9kMzN3dWJyZmtpMGw2OC5jbG91ZGZyb250Lm5ldC8wZGMwZWZjY2ZhZjYzOGRlOGJlNGNlNDNlMmJhYjhhOTE1NDFkM2QyL2M0ZmRjL3dwLWNvbnRlbnQvdXBsb2Fkcy8yMDE4LzA4L3JzdHVkaW8taWRlLnBuZykuCgojIyBUaGUgUlN0dWRpbyBFbnZpcm9ubWVudAoKVGhlIFJTdHVkaW8gZW52aXJvbm1lbnQgaGFzIGZvdXIgbWFpbiAqKnBhbmVzKiosIGVhY2ggb2Ygd2hpY2ggbWF5IGhhdmUgYSBudW1iZXIgb2YgdGFicyB0aGF0IGRpc3BsYXkgZGlmZmVyZW50IGluZm9ybWF0aW9uIG9yIGZ1bmN0aW9uYWxpdHkuICh0aGVpciBzcGVjaWZpYyBsb2NhdGlvbiBjYW4gYmUgY2hhbmdlZCB1bmRlciBUb29scyAtPiBHbG9iYWwgT3B0aW9ucyAtPiBQYW5lIExheW91dCkuCiFbUlN0dWRpbyBBcHBlYXJhbmNlXShzY3JlZW5zaG90cy9yc3R1ZGlvLXBhbmVzLnBuZykgCgoxLiBUaGUgKipFZGl0b3IqKiBwYW5lIGlzIHdoZXJlIHlvdSBjYW4gd3JpdGUgUiBzY3JpcHRzIGFuZCBvdGhlciBkb2N1bWVudHMuIEVhY2ggdGFiIGhlcmUgaXMgaXRzIG93biBkb2N1bWVudC4KVGhpcyBpcyB5b3VyIF90ZXh0IGVkaXRvcl8sIHdoaWNoIHdpbGwgYWxsb3cgeW91IHRvIHNhdmUgeW91ciBSIGNvZGUgZm9yIGZ1dHVyZSB1c2UuIApOb3RlIHRoYXQgY2hhbmdlIGNvZGUgaGVyZSB3aWxsIG5vdCBydW4gYXV0b21hdGljYWxseSB1bnRpbCB5b3UgcnVuIGl0LiAKCjIuIFRoZSAqKkNvbnNvbGUqKiBwYW5lIGlzIHdoZXJlIHlvdSBjYW4gX2ludGVyYWN0aXZlbHlfIHJ1biBSIGNvZGUuIAogICsgVGhlcmUgaXMgYWxzbyBhICoqVGVybWluYWwqKiB0YWIgaGVyZSB3aGljaCBjYW4gYmUgdXNlZCBmb3IgcnVubmluZyBwcm9ncmFtcyBvdXRzaWRlIFIgb24geW91ciBjb21wdXRlcgogIAozLiBUaGUgKipFbnZpcm9ubWVudCoqIHBhbmUgcHJpbWFyaWx5IGRpc3BsYXlzIHRoZSB2YXJpYWJsZXMsIHNvbWV0aW1lcyBrbm93biBhcyBfb2JqZWN0c18gdGhhdCBhcmUgZGVmaW5lZCBkdXJpbmcgYSBnaXZlbiBSIHNlc3Npb24sIGFuZCB3aGF0IGRhdGEgb3IgdmFsdWVzIHRoZXkgbWlnaHQgaG9sZC4KCjQuIFRoZSAqKkhlbHAgdmlld2VyKiogcGFuZSBoYXMgc2V2ZXJhbCB0YWJzIGFsbCBvZiB3aGljaCBhcmUgcHJldHR5IGltcG9ydGFudDoKICAgICsgVGhlICoqRmlsZXMqKiB0YWIgc2hvd3MgdGhlIHN0cnVjdHVyZSBhbmQgY29udGVudHMgb2YgZmlsZXMgYW5kIGZvbGRlcnMgKGFsc28ga25vd24gYXMgZGlyZWN0b3JpZXMpIG9uIHlvdXIgY29tcHV0ZXIuCiAgICArIFRoZSAqKlBsb3RzKiogdGFiIHdpbGwgcmV2ZWFsIHBsb3RzIHdoZW4geW91IG1ha2UgdGhlbQogICAgKyBUaGUgKipQYWNrYWdlcyoqIHRhYiBzaG93cyB3aGljaCBpbnN0YWxsZWQgcGFja2FnZXMgaGF2ZSBiZWVuIGxvYWRlZCBpbnRvIHlvdXIgUiBzZXNzaW9uCiAgICArIFRoZSAqKkhlbHAqKiB0YWIgd2lsbCBzaG93IHRoZSBoZWxwIHBhZ2Ugd2hlbiB5b3UgbG9vayB1cCBhIGZ1bmN0aW9uCiAgICArIFRoZSAqKlZpZXdlcioqIHBhbmUgd2lsbCByZXZlYWwgY29tcGlsZWQgUiBNYXJrZG93biBkb2N1bWVudHMgCgojIyBCYXNpYyBDYWxjdWxhdGlvbnMKCiMjIyBNYXRoZW1hdGljYWwgb3BlcmF0b3JzCgpUaGUgbW9zdCBiYXNpYyB1c2Ugb2YgUiBpcyBhcyBhIHJlZ3VsYXIgY2FsY3VsYXRvcjoKCnwgT3BlcmF0aW9uIHwgU3ltYm9sIHwKfC0tLS0tLS0tLS0tfC0tLS0tLS0tfAp8IEFkZCAgfCBgK2AgfCAKfCBTdWJ0cmFjdCAgfCBgLWAgfCAKfCBNdWx0aXBseSAgfCBgKmAgfCAKfCBEaXZpZGUgIHwgYC9gIHwgCnwgRXhwb25lbnRpYXRlIHwgYF5gIG9yIGAqKmAgfCAKCkZvciBleGFtcGxlLCB3ZSBjYW4gZG8gc29tZSBzaW1wbGUgbXVsdGlwbGljYXRpb24gbGlrZSB0aGlzLiAKV2hlbiB5b3UgZXhlY3V0ZSBjb2RlIHdpdGhpbiB0aGUgbm90ZWJvb2ssIHRoZSByZXN1bHRzIGFwcGVhciBiZW5lYXRoIHRoZSBjb2RlLiAKVHJ5IGV4ZWN1dGluZyB0aGlzIGNodW5rIGJ5IGNsaWNraW5nIHRoZSAqUnVuKiBidXR0b24gd2l0aGluIHRoZSBjaHVuayBvciBieSAKcGxhY2luZyB5b3VyIGN1cnNvciBpbnNpZGUgaXQgYW5kIHByZXNzaW5nICpDbWQrU2hpZnQrRW50ZXIqLiAKCmBgYHtyIGNhbGN1bGF0b3J9CjUgKiA2CmBgYAoKVXNlIHRoZSBjb25zb2xlIHRvIGNhbGN1bGF0ZSBvdGhlciBleHByZXNzaW9ucy4gU3RhbmRhcmQgb3JkZXIgb2Ygb3BlcmF0aW9ucyBhcHBsaWVzIChtb3N0bHkpLCBhbmQgIHlvdSBjYW4gdXNlIHBhcmVudGhlc2VzIGAoKWAgYXMgeW91IG1pZ2h0IGV4cGVjdCAoYnV0IG5vdCBicmFja2V0cyBgW11gIG9yIGJyYWNlc2B7fWAsIHdoaWNoIGhhdmUgc3BlY2lhbCBtZWFuaW5ncykuIE5vdGUgaG93ZXZlciwgdGhhdCB5b3UgbXVzdCAqKmFsd2F5cyoqIHNwZWNpZnkgbXVsdGlwbGljYXRpb24gd2l0aCBgKmA7IGltcGxpY2l0IG11bHRpcGxpY2F0aW9uIHN1Y2ggYXMgYDEwKDMgKyA0KWAgb3IgYDEweGAgd2lsbCBub3Qgd29yayBhbmQgd2lsbCBnZW5lcmF0ZSBhbiBlcnJvciwgb3Igd29yc2UuCgpgYGB7ciBleHByZXNzaW9ucywgbGl2ZSA9IFRSVUV9CjEwICogKDMgKyA0KV4yCmBgYAoKCiMjIyBEZWZpbmluZyBhbmQgdXNpbmcgdmFyaWFibGVzIAoKVG8gZGVmaW5lIGEgdmFyaWFibGUsIHdlIHVzZSB0aGUgX2Fzc2lnbm1lbnQgb3BlcmF0b3JfIHdoaWNoIGxvb2tzIGxpa2UgYW4gYXJyb3c6IGA8LWAsIGZvciBleGFtcGxlIGB4IDwtIDdgIHRha2VzIHRoZSB2YWx1ZSBvbiB0aGUgcmlnaHQtaGFuZCBzaWRlIG9mIHRoZSBvcGVyYXRvciBhbmQgYXNzaWducyBpdCB0byB0aGUgdmFyaWFibGUgbmFtZSBvbiB0aGUgbGVmdC1oYW5kIHNpZGUuIAoKYGBge3IgdmFyLWRlZmluZSwgbGl2ZSA9IFRSVUV9CiMgRGVmaW5lIGEgdmFyaWFibGUgeCB0byBlcXVhbCA3LCBhbmQgcHJpbnQgb3V0IHRoZSB2YWx1ZSBvZiB4CnggPC0gNwoKIyBXZSBjYW4gaGF2ZSBSIHJlcGVhdCBiYWNrIHRvIHVzIHdoYXQgYHhgIGlzIGJ5IGp1c3QgdXNpbmcgYHhgCngKYGBgCgpTb21lIGZlYXR1cmVzIG9mIHZhcmlhYmxlcywgY29uc2lkZXJpbmcgdGhlIGV4YW1wbGUgYHggPC0gN2A6CkV2ZXJ5IHZhcmlhYmxlIGhhcyBhICoqbmFtZSoqLCBhICoqdmFsdWUqKiwgYW5kIGEgKip0eXBlKiouIApUaGlzIHZhcmlhYmxlJ3MgbmFtZSBpcyBgeGAsIGl0cyB2YWx1ZSBpcyBgN2AsIGFuZCBpdHMgdHlwZSBpcyBgbnVtZXJpY2AgKDcgaXMgYSBudW1iZXIhKS4KUmUtZGVmaW5pbmcgYSB2YXJpYWJsZSB3aWxsIG92ZXJ3cml0ZSB0aGUgdmFsdWUuCgpgYGB7ciB2YXItcmVkZWZpbmV9CnggPC0gNS41Cgp4CmBgYAoKV2UgY2FuIG1vZGlmeSBhbiBleGlzdGluZyB2YXJpYWJsZSBieSByZWFzc2lnbmluZyBpdCB0byBpdHMgc2FtZSBuYW1lLiAKSGVyZSB3ZSdsbCBhZGQgYDJgIHRvIGB4YCBhbmQgcmVhc3NpZ24gdGhlIHJlc3VsdCBiYWNrIHRvIGB4YC4gCgpgYGB7ciB2YXItbW9kaWZ5LCBsaXZlID0gVFJVRX0KeCA8LSB4ICsgMgoKeApgYGAKCiMjIyBWYXJpYWJsZSBuYW1pbmcgbm90ZToKQXMgYmVzdCB5b3UgY2FuLCBpdCBpcyBhIGdvb2QgaWRlYSB0byBtYWtlIHlvdXIgdmFyaWFibGUgbmFtZXMgaW5mb3JtYXRpdmUgKGUuZy4gYHhgIGRvZXNuJ3QgbWVhbiBhbnl0aGluZywgYnV0IGBzYW5kd2ljaF9wcmljZWAgaXMgbWVhbmluZ2Z1bC4uLiBpZiB3ZSdyZSB0YWxraW5nIGFib3V0IHRoZSBjb3N0IG9mIHNhbmR3aWNoZXMsIHRoYXQgaXMuLikuIAoKIyMjIENvbW1lbnRzCgpBcmd1YWJseSB0aGUgX19tb3N0IGltcG9ydGFudF9fIGFzcGVjdCBvZiB5b3VyIGNvZGluZyBpcyBjb21tZW50czogU21hbGwgcGllY2VzIG9mIGV4cGxhbmF0b3J5IHRleHQgeW91IGxlYXZlIGluIHlvdXIgY29kZSB0byBleHBsYWluIHdoYXQgdGhlIGNvZGUgaXMgZG9pbmcgYW5kL29yIGxlYXZlIG5vdGVzIHRvIHlvdXJzZWxmIG9yIG90aGVycy4gCkNvbW1lbnRzIGFyZSBpbnZhbHVhYmxlIGZvciBjb21tdW5pY2F0aW5nIHlvdXIgY29kZSB0byBvdGhlcnMsIGJ1dCB0aGV5IGFyZSBtb3N0IGltcG9ydGFudCBmb3IgKipGdXR1cmUgWW91KiouIApGdXR1cmUgWW91IGNvbWVzIGludG8gZXhpc3RlbmNlIGFib3V0IG9uZSBzZWNvbmQgYWZ0ZXIgeW91IHdyaXRlIGNvZGUsIGFuZCBoYXMgbm8gaWRlYSB3aGF0IG9uIGVhcnRoIFBhc3QgWW91IHdhcyB0aGlua2luZy4gCgpDb21tZW50cyBpbiBSIGNvZGUgYXJlIGluZGljYXRlZCB3aXRoIHBvdW5kIHNpZ25zICgqYWthKiBoYXNodGFncywgb2N0b3Rob3JwcykuIFIgd2lsbCBfaWdub3JlXyBhbnkgdGV4dCBpbiBhIGxpbmUgYWZ0ZXIgdGhlIHBvdW5kIHNpZ24sIHNvIHlvdSBjYW4gcHV0IHdoYXRldmVyIHRleHQgeW91IGxpa2UgdGhlcmUuCgpgYGB7ciBjb21tZW50c30KMjIvNyAjIG5vdCBxdWl0ZSBwaQoKIyBJZiB3ZSBuZWVkIGEgYmV0dGVyIGFwcHJveGltYXRpb24gb2YgcGksIHdlIGNhbiB1c2UgRXVsZXIncyBmb3JtdWxhCiMgVGhpcyB1c2VzIGF0YW4oKSwgd2hpY2ggY2FsY3VsYXRlcyBhcmN0YW5nZW50LgoyMCAqIGF0YW4oMS83KSArIDggKiBhdGFuKDMvNzkpIApgYGAKCkhlbHAgb3V0IEZ1dHVyZSBZb3UgYnkgYWRkaW5nIGxvdHMgb2YgY29tbWVudHMhIApGdXR1cmUgWW91IG5leHQgd2VlayB0aGlua3MgVG9kYXkgWW91IGlzIGFuIGlkaW90LCBhbmQgdGhlIG9ubHkgd2F5IHlvdSBjYW4gY29udmluY2UgRnV0dXJlIFlvdSB0aGF0IFRvZGF5IFlvdSBpcyByZWFzb25hYmx5IGNvbXBldGVudCBpcyBieSBhZGRpbmcgY29tbWVudHMgaW4geW91ciBjb2RlIGV4cGxhaW5pbmcgd2h5IFRvZGF5IFlvdSBpcyBhY3R1YWxseSBub3Qgc28gYmFkLgoKIyMgRnVuY3Rpb25zCldlIGNhbiB1c2UgcHJlLWJ1aWx0IGNvbXB1dGF0aW9uIG1ldGhvZHMgY2FsbGVkICJmdW5jdGlvbnMiIGZvciBvdGhlciBvcGVyYXRpb25zLiAKRnVuY3Rpb25zIGhhdmUgdGhlIGZvbGxvd2luZyBmb3JtYXQsIHdoZXJlIHRoZSBfYXJndW1lbnRfIGlzIHRoZSBpbmZvcm1hdGlvbiB3ZSBhcmUgcHJvdmlkaW5nIHRvIHRoZSBmdW5jdGlvbiBmb3IgaXQgdG8gcnVuLiAKQW4gZXhhbXBsZSBvZiB0aGlzIHdhcyB0aGUgYGF0YW4oKWAgZnVuY3Rpb24gdXNlZCBhYm92ZS4KCmBgYHIKZnVuY3Rpb25fbmFtZShhcmd1bWVudCkKYGBgCgpUbyBsZWFybiBhYm91dCBmdW5jdGlvbnMsIHdlJ2xsIGV4YW1pbmUgb25lIGNhbGxlZCBgbG9nKClgIGZpcnN0LiAKClRvIGtub3cgd2hhdCBhIGZ1bmN0aW9uIGRvZXMgYW5kIGhvdyB0byB1c2UgaXQsIHVzZSB0aGUgcXVlc3Rpb24gbWFyayB3aGljaCB3aWxsIHJldmVhbCBkb2N1bWVudGF0aW9uIGluIHRoZSAqKmhlbHAgcGFuZSoqOiBgP2xvZ2AKIVtyaGVscF0oc2NyZWVuc2hvdHMvcmhlbHAtbG9nLnBuZykgCgpUaGUgZG9jdW1lbnRhdGlvbiB0ZWxscyB1cyB0aGF0IGBsb2coKWAgaXMgZGVyaXZlZCBmcm9tIGB7YmFzZX1gLCBtZWFuaW5nIGl0IGlzIGEgZnVuY3Rpb24gdGhhdCBpcyBwYXJ0IG9mIGJhc2UgUi4gCkl0IHByb3ZpZGVzIGEgYnJpZWYgZGVzY3JpcHRpb24gb2Ygd2hhdCB0aGUgZnVuY3Rpb24gZG9lcyBhbmQgc2hvd3Mgc2V2ZXJhbCBleGFtcGxlcyBvZiB0byBob3cgdXNlIGl0LgoKSW4gcGFydGljdWxhciwgZG9jdW1lbnRhdGlvbiB0ZWxscyBhYm91dCBob3cgd2hhdCBhcmd1bWVudChzKSB0byBwcm92aWRlOgorIFRoZSBmaXJzdCBfcmVxdWlyZWRfIGFyZ3VtZW50IGlzIHRoZSB2YWx1ZSB3ZSdkIGxpa2UgdG8gdGFrZSB0aGUgbG9nIG9mLCBieSBkZWZhdWx0IGl0cyBfbmF0dXJhbCBsb2dfCisgVGhlIHNlY29uZCBfb3B0aW9uYWxfIGFyZ3VtZW50IGNhbiBzcGVjaWZ5IGEgZGlmZmVyZW50IGJhc2UgcmF0aGVyIHRoYW4gdGhlIGRlZmF1bHQgYGVgLgoKRnVuY3Rpb25zIGFsc28gX3JldHVybl8gdmFsdWVzIGZvciB1cyB0byB1c2UuIApJbiB0aGUgY2FzZSBvZiBgbG9nKClgLCB0aGUgcmV0dXJuZWQgdmFsdWUgaXMgdGhlIGxvZydkIHZhbHVlIHRoZSBmdW5jdGlvbiBjb21wdXRlZC4KCmBgYHtyIGxvZ30KbG9nKDczKQpgYGAKCkhlcmUgd2UgY2FuIHNwZWNpZnkgYW4gX2FyZ3VtZW50XyBvZiBgYmFzZWAgdG8gY2FsY3VsYXRlIGxvZyBiYXNlIDMuIAoKYGBge3IgbG9nM30KbG9nKDgxLCBiYXNlID0gMykKYGBgCgpJZiB3ZSBkb24ndCBzcGVjaWZ5IHRoZSBfYXJndW1lbnRfIG5hbWVzLCBpdCBhc3N1bWVzIHRoZXkgYXJlIGluIHRoZSBvcmRlciB0aGF0IGBsb2dgIGRlZmluZXMgdGhlbS4gClNlZSBgP2xvZ2AgdG8gc2VlIG1vcmUgYWJvdXQgaXRzIGFyZ3VtZW50cy4gCgpgYGB7ciBsb2cyLCBsaXZlID0gVFJVRX0KbG9nKDgsIDIpCmBgYAoKV2UgY2FuIHN3aXRjaCB0aGUgb3JkZXIgaWYgd2Ugc3BlY2lmeSB0aGUgYXJndW1lbnQgbmFtZXMuIAoKYGBge3IgbG9nLW9yZGVyfQpsb2coYmFzZSA9IDEwLCB4ID0gNDM0MikKYGBgCgpXZSBjYW4gYWxzbyBwcm92aWRlIHZhcmlhYmxlcyBhcyBhcmd1bWVudHMgaW4gdGhlIHNhbWUgd2F5IGFzIHRoZSByYXcgdmFsdWVzLiAKCmBgYHtyIGxvZy12YXJpYWJsZX0KbWVhbmluZyA8LSA0Mgpsb2cobWVhbmluZykKYGBgCgojIyBXb3JraW5nIHdpdGggdmFyaWFibGVzCgojIyMgVmFyaWFibGUgVHlwZXMKClZhcmlhYmxlIHR5cGVzIGluIFIgY2FuIHNvbWV0aW1lcyBiZSBfY29lcmNlZF8gKGNvbnZlcnRlZCkgZnJvbSBvbmUgdHlwZSB0byBhbm90aGVyLgoKYGBge3J9CiMgRGVmaW5lIGEgdmFyaWFibGUgd2l0aCBhIG51bWJlcgp4IDwtIDE1CmBgYAoKVGhlIGZ1bmN0aW9uIGBjbGFzcygpYCB3aWxsIHRlbGwgdXMgdGhlIHZhcmlhYmxlJ3MgdHlwZS4KCmBgYHtyfQpjbGFzcyh4KQpgYGAKCkxldCdzIGNvZXJjZSBpdCB0byBhIGNoYXJhY3Rlci4gCgpgYGB7cn0KeCA8LSBhcy5jaGFyYWN0ZXIoeCkKY2xhc3MoeCkKYGBgCgpTZWUgaXQgbm93IGhhcyBxdW90ZXMgYXJvdW5kIGl0PyBJdCdzIG5vdyBhIGNoYXJhY3RlciBhbmQgd2lsbCBiZWhhdmUgYXMgc3VjaAoKYGBge3J9CngKYGBgCgpVc2UgdGhpcyBjaHVuayB0byB0cnkgdG8gcGVyZm9ybSBjYWxjdWxhdGlvbnMgd2l0aCBgeGAsIG5vdyB0aGF0IGl0IGlzIGEgY2hhcmFjdGVyLCB3aGF0IGhhcHBlbnM/IAoKYGBge3IgbGl2ZSA9IFRSVUV9CiMgVHJ5IHRvIHBlcmZvcm0gY2FsY3VsYXRpb25zIG9uIGB4YApgYGAKCkJ1dCB3ZSBjYW4ndCBjb2VyY2UgZXZlcnl0aGluZzoKCmBgYHtyfQojIExldCdzIGNyZWF0ZSBhIGNoYXJhY3RlciB2YXJpYWJsZQp4IDwtICJsb29rIGF0IG15IGNoYXJhY3RlciB2YXJpYWJsZSIKYGBgCgpMZXQncyB0cnkgbWFraW5nIHRoaXMgYSBudW1lcmljIHZhcmlhYmxlOgoKYGBge3IgY29lcmNlLWNoYXIsIGVycm9yPVRSVUV9CnggPC0gYXMubnVtZXJpYyh4KQpgYGAKClByaW50IG91dCBgeGAuCgpgYGB7cn0KeApgYGAKClIgaXMgdGVsbGluZyB1cyBpdCBkb2Vzbid0IGtub3cgaG93IHRvIGNvbnZlcnQgdGhpcyB0byBhIG51bWVyaWMgdmFyaWFibGUsIHNvIGl0IGhhcyByZXR1cm5lZCBgTkFgIGluc3RlYWQuCgpGb3IgcmVmZXJlbmNlLCBoZXJlJ3MgYSBzdW1tYXJ5IG9mIHNvbWUgb2YgdGhlIG1vc3QgaW1wb3J0YW50IHZhcmlhYmxlIHR5cGVzLiAKCnwgVmFyaWFibGUgVHlwZSB8IERlZmluaXRpb24gfCBFeGFtcGxlcyB8IENvZXJjaW9uIHwKfC0tLS0tLS0tLS0tLS0tLXwtLS0tLS0tLS0tLS18LS0tLS0tLS0tLXwgLS0tLS0tLS18CnwgYG51bWVyaWNgICAgICAgIHwgQW55IG51bWJlciB2YWx1ZSB8IGA1YDxicj5gNy41YCA8YnI+YC0xYHwgYGFzLm51bWVyaWMoKWAKfCBgaW50ZWdlcmAgICAgICAgfCBBbnkgX3dob2xlXyBudW1iZXIgdmFsdWUgKG5vIGRlY2ltYWxzKSB8IGA1YCA8YnI+IGAtMTAwYCB8IGBhcy5pbnRlZ2VyKClgCnxgY2hhcmFjdGVyYCAgICAgIHwgQW55IGNvbGxlY3Rpb24gb2YgY2hhcmFjdGVycyBkZWZpbmVkIHdpdGhpbiBfcXVvdGF0aW9uIG1hcmtzXy4gQWxzbyBrbm93biBhcyBhICJzdHJpbmciLiB8IGAiYSJgIChhIHNpbmdsZSBsZXR0ZXIpIDxicj5gInN0cmluZ29mbGV0dGVycyJgIChhIHdob2xlIGJ1bmNoIG9mIGNoYXJhY3RlcnMgcHV0IHRvZ2V0aGVyIGFzIG9uZSkgPGJyPiBgInN0cmluZyBvZiBsZXR0ZXJzIGFuZCBzcGFjZXMiYCA8YnI+IGAiNSJgIDxicj4gYCdzaW5nbGUgcXVvdGVzIGFyZSBhbHNvIGdvb2QnYCB8IGBhcy5jaGFyYWN0ZXIoKWAKfGBsb2dpY2FsYCAgICAgIHwgQSB2YWx1ZSBvZiBgVFJVRWAsIGBGQUxTRWAsIG9yIGBOQWAgfCBgVFJVRWAgPGJyPiBgRkFMU0VgIDxicj4gYE5BYCAobm90IGRlZmluZWQpIHwgYGFzLmxvZ2ljYWwoKWAgCnxgZmFjdG9yYCAgICAgICB8IEEgc3BlY2lhbCB0eXBlIG9mIHZhcmlhYmxlIHRoYXQgZGVub3RlcyBzcGVjaWZpYyBjYXRlZ29yaWVzIG9mIGEgY2F0ZWdvcmljYWwgdmFyaWFibGUgfCAoc3RheSB0dW5lZC4uKSB8IGBhcy5mYWN0b3IoKWAKCiMjIyBWZWN0b3JzCgpZb3Ugd2lsbCBoYXZlIG5vdGljZWQgdGhhdCBhbGwgeW91ciBjb21wdXRhdGlvbnMgdGVuZCB0byBwb3AgdXAgd2l0aCBhIGBbMV1gIHByZWNlZGluZyB0aGVtIGluIFIncyBvdXRwdXQuIApUaGlzIGlzIGJlY2F1c2UsIGluIGZhY3QsIGFsbCAob2sgbW9zdGx5IGFsbCkgdmFyaWFibGVzIGFyZSBfYnkgZGVmYXVsdF8gIHZlY3RvcnMsIGFuZCBvdXIgYW5zd2VycyBhcmUgdGhlIGZpcnN0IChpbiB0aGVzZSBjYXNlcyBvbmx5KSB2YWx1ZSBpbiB0aGUgdmVjdG9yLiAKQXMgdmVjdG9ycyBnZXQgbG9uZ2VyLCBuZXcgaW5kZXggaW5kaWNhdG9ycyB3aWxsIGFwcGVhciBhdCB0aGUgc3RhcnQgb2YgbmV3IGxpbmVzLiAKCmBgYHtyfQojIFRoaXMgaXMgYWN0dWFsbHkgYW4gdmVjdG9yIHRoYXQgaGFzIG9uZSBpdGVtIGluIGl0Lgp4IDwtIDcKYGBgCgpgYGB7ciB2ZWN0b3ItbGVuZ3RofQojIFRoZSBsZW5ndGgoKSBmdW5jdGlvbnMgdGVsbHMgdXMgaG93IGxvbmcgYW4gdmVjdG9yIGlzOgpsZW5ndGgoeCkKYGBgCgpXZSBjYW4gZGVmaW5lIHZlY3RvcnMgd2l0aCB0aGUgZnVuY3Rpb24gYGMoKWAsIHdoaWNoIHN0YW5kcyBmb3IgImNvbWJpbmUiLiAKVGhpcyBmdW5jdGlvbiB0YWtlcyBhIGNvbW1hLXNlcGFyYXRlZCBzZXQgb2YgdmFsdWVzIHRvIHBsYWNlIGluIHRoZSB2ZWN0b3IsIGFuZCByZXR1cm5zIHRoZSB2ZWN0b3IgaXRzZWxmOgoKYGBge3IgbWFrZS12ZWN0b3J9Cm15X251bWVyaWNfdmVjdG9yIDwtIGMoMSwgMSwgMiwgMywgNSwgOCwgMTMsIDIxKQpteV9udW1lcmljX3ZlY3RvcgpgYGAKCldlIGNhbiBidWlsZCBvbiB2ZWN0b3JzIGluIHBsYWNlIGJ5IHJlZGVmaW5pbmcgdGhlbQoKYGBge3IgZmliYm9uYWNjaSwgbGl2ZSA9IFRSVUV9CiMgYWRkIHRoZSBuZXh0IHR3byBGaWJvbmFjY2kgbnVtYmVycyB0byB0aGUgc2VyaWVzLgpteV9udW1lcmljX3ZlY3RvciA8LSBjKG15X251bWVyaWNfdmVjdG9yLCAzNCwgNTUpCm15X251bWVyaWNfdmVjdG9yCmBgYAoKV2UgY2FuIHB1bGwgb3V0IHNwZWNpZmljIGl0ZW1zIGZyb20gYW4gdmVjdG9yIHVzaW5nIGEgcHJvY2VzcyBjYWxsZWQgX2luZGV4aW5nXywgd2hpY2ggdXNlcyBicmFja2V0cyBgW11gIHRvIHNwZWNpZnkgdGhlIHBvc2l0aW9uIG9mIGFuIGl0ZW0uIAoKYGBge3Igc3Vic2V0MX0KIyBHcmFiIHRoZSBmb3VydGggdmFsdWUgZnJvbSBteV9udW1lcmljX3ZlY3RvcgojIFRoaXMgZ2l2ZXMgdXMgYW4gdmVjdG9yIG9mIGxlbmd0aCAxIApteV9udW1lcmljX3ZlY3Rvcls0XQpgYGAKCkNvbG9ucyBhcmUgYWxzbyBhIG5pY2Ugd2F5IHRvIHF1aWNrbHkgbWFrZSBvcmRlcmVkIG51bWVyaWMgdmVjdG9ycwpVc2UgYSBjb2xvbiB0byBzcGVjaWZ5IGFuIGluY2x1c2l2ZSByYW5nZSBvZiBpbmRpY2VzClRoaXMgd2lsbCByZXR1cm4gYW4gdmVjdG9yIHdpdGggMiwgMywgNCwgYW5kIDUuCgpgYGB7ciBzdWJzZXQtbWFueX0KbXlfbnVtZXJpY192ZWN0b3JbMjo1XQpgYGAKCk9uZSBtYWpvciBiZW5lZml0IG9mIHZlY3RvcnMgaXMgdGhlIGNvbmNlcHQgb2YgKip2ZWN0b3JpemF0aW9uKiosIHdoZXJlIFIgYnkgZGVmYXVsdCBwZXJmb3JtcyBvcGVyYXRpb25zIG9uIHRoZSBfZW50aXJlIHZlY3RvciBhdCBvbmNlXy4gCkZvciBleGFtcGxlLCB3ZSBjYW4gZ2V0IHRoZSBsb2cgb2YgYWxsIG51bWJlcnMgMS0yIHdpdGggYSBzaW5nbGUsIHNpbXBsZSBjYWxsLCBhbmQgbW9yZSEKCmBgYHtyIHZlY3Rvcml6ZX0KdmFsdWVzXzFfdG9fMjAgPC0gMToyMApgYGAKYGBge3IgdmVjdG9yaXplLWxvZywgbGl2ZSA9IFRSVUV9CiMgY2FsY3VsYXRlIHRoZSBsb2cgb2YgdmFsdWVzXzFfdG9fMjAKbG9nKHZhbHVlc18xX3RvXzIwKQpgYGAKCkZpbmFsbHksIHdlIGNhbiBhcHBseSBsb2dpY2FsIGV4cHJlc3Npb25zIHRvIHZlY3RvcnMsIGp1c3QgYXMgd2UgY2FuIGRvIGZvciBzaW5nbGUgdmFsdWVzLgpUaGUgb3V0cHV0IGhlcmUgaXMgYSBsb2dpY2FsIHZlY3RvciB0ZWxsaW5nIHVzIHdoZXRoZXIgZWFjaCB2YWx1ZSBpbiBleGFtcGxlX3ZlY3RvciBpcyBUUlVFIG9yIEZBTFNFCgpgYGB7ciB2ZWN0b3ItY29tcGFyZX0KIyBXaGljaCB2YWx1ZXMgYXJlIDw9IDM/CnZhbHVlc18xX3RvXzIwIDw9IDMKYGBgCgpUaGVyZSBhcmUgc2V2ZXJhbCBrZXkgZnVuY3Rpb25zIHdoaWNoIGNhbiBiZSB1c2VkIG9uIHZlY3RvcnMgY29udGFpbmluZyBudW1lcmljIHZhbHVlcywgc29tZSBvZiB3aGljaCBhcmUgYmVsb3cuCgorIGBtZWFuKClgOiBUaGUgYXZlcmFnZSB2YWx1ZSBpbiB0aGUgdmVjdG9yCisgYG1pbigpYDogVGhlIG1pbmltdW0gdmFsdWUgaW4gdGhlIHZlY3RvcgorIGBtYXgoKWA6IFRoZSBtYXhpbXVtIHZhbHVlIGluIHRoZSB2ZWN0b3IKKyBgc3VtKClgOiBUaGUgc3VtIG9mIGFsbCB2YWx1ZXMgaW4gdGhlIHZlY3RvcgoKV2UgY2FuIHRyeSBvdXQgdGhlc2UgZnVuY3Rpb25zIG9uIHRoZSB2ZWN0b3IgYHZhbHVlc18xX3RvXzIwYCB3ZSd2ZSBjcmVhdGVkLiAKCmBgYHtyIHZlY3Rvci1mdW5jc30KbWVhbih2YWx1ZXNfMV90b18yMCkKCiMgVHJ5IG91dCBzb21lIG9mIHRoZSBvdGhlciBmdW5jdGlvbnMgd2UndmUgbGlzdGVkIGFib3ZlIAoKYGBgCgojIyMgQSBub3RlIG9uIHZhcmlhYmxlIG5hbWluZwoKV2UgaGF2ZSBsZWFybmVkIGZ1bmN0aW9ucyBzdWNoIGFzIGBjYCwgYGxlbmd0aGAsIGBzdW1gLCBhbmQgZXRjLiAKSW1hZ2luZSBkZWZpbmluZyBhIHZhcmlhYmxlIGNhbGxlZCBgY2A6IFRoaXMgd2lsbCB3b3JrLCBidXQgaXQgd2lsbCBsZWFkIHRvIGEgCmxvdCBvZiB1bmludGVuZGVkIGJ1Z3MsIHNvIGl0cyBiZXN0IHRvIGF2b2lkIHRoaXMuIAoKIyMjIFRoZSBgJWluJWAgbG9naWNhbCBvcGVyYXRvciAKCmAlaW4lYCBpcyB1c2VmdWwgZm9yIGRldGVybWluaW5nIHdoZXRoZXIgYSBnaXZlbiBpdGVtKHMpIGFyZSBpbiBhbiB2ZWN0b3IuCgpgYGB7ciBpbi1vcGVyYXRvcn0KIyBpcyBgN2AgaW4gb3VyIHZlY3Rvcj8gCjcgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCmBgYHtyIGluMiwgbGl2ZSA9IFRSVUV9CiMgaXMgYDUwYCBpbiBvdXIgdmVjdG9yPyAKNTAgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCldlIGNhbiB0ZXN0IGEgdmVjdG9yIG9mIHZhbHVlcyBiZWluZyB3aXRoaW4gYW5vdGhlciB2ZWN0b3Igb2YgdmFsdWVzLiAKCmBgYHtyIHZlY3Rvci1pbiwgbGl2ZSA9IFRSVUV9CnF1ZXN0aW9uX3ZhbHVlcyA8LSBjKDE6MywgNywgNTApCiMgQXJlIHRoZXNlIHZhbHVlcyBpbiBvdXIgdmVjdG9yPwpxdWVzdGlvbl92YWx1ZXMgJWluJSB2YWx1ZXNfMV90b18yMApgYGAKCiMjIERhdGEgZnJhbWVzCgpfRGF0YSBmcmFtZXMgYXJlIG9uZSBvZiB0aGUgbW9zdCB1c2VmdWwgdG9vbHMgZm9yIGRhdGEgYW5hbHlzaXMgaW4gUi5fIApUaGV5IGFyZSB0YWJsZXMgd2hpY2ggY29uc2lzdCBvZiByb3dzIGFuZCBjb2x1bW5zLCBtdWNoIGxpa2UgYSBfc3ByZWFkc2hlZXRfLiAKRWFjaCBjb2x1bW4gaXMgYSB2YXJpYWJsZSB3aGljaCBiZWhhdmVzIGFzIGEgX3ZlY3Rvcl8sIGFuZCBlYWNoIHJvdyBpcyBhbiBvYnNlcnZhdGlvbi4gCldlIHdpbGwgYmVnaW4gb3VyIGV4cGxvcmF0aW9uIHdpdGggZGF0YXNldCBvZiBtZWFzdXJlbWVudHMgZnJvbSB0aHJlZSBwZW5ndWluIHNwZWNpZXMgbWVhc3VyZWQsIHdoaWNoIHdlIGNhbiBmaW5kIGluIHRoZSBbYHBhbG1lcnBlbmd1aW5zYCBwYWNrYWdlXShodHRwczovL2FsbGlzb25ob3JzdC5naXRodWIuaW8vcGFsbWVycGVuZ3VpbnMvKS4gCldlJ2xsIHRhbGsgbW9yZSBhYm91dCBwYWNrYWdlcyBzb29uIQpUbyB1c2UgdGhpcyBkYXRhc2V0LCB3ZSB3aWxsIGxvYWQgaXQgZnJvbSB0aGUgYHBhbG1lcnBlbmd1aW5zYCBwYWNrYWdlIHVzaW5nIGEgYDo6YCAobW9yZSBvbiB0aGlzIGxhdGVyKSBhbmQgYXNzaWduIGl0IHRvIGEgdmFyaWFibGUgbmFtZWQgYHBlbmd1aW5zYCBpbiBvdXIgY3VycmVudCBlbnZpcm9ubWVudC4KCmBgYHtyIHBlbmd1aW4tbGlicmFyeX0KcGVuZ3VpbnMgPC0gcGFsbWVycGVuZ3VpbnM6OnBlbmd1aW5zCmBgYAoKIVtkcmF3aW5ncyBvZiBwZW5ndWluIHNwZWNpZXNdKGRpYWdyYW1zL2x0ZXJfcGVuZ3VpbnMucG5nKSBBcnR3b3JrIGJ5IEBhbGxpc29uX2hvcnN0CgojIyMgRXhwbG9yaW5nIGRhdGEgZnJhbWVzCgpUaGUgZmlyc3Qgc3RlcCB0byB1c2luZyBhbnkgZGF0YSBpcyB0byBsb29rIGF0IGl0ISEhIApSU3R1ZGlvIGNvbnRhaW5zIGEgc3BlY2lhbCBmdW5jdGlvbiBgVmlldygpYCB3aGljaCBhbGxvd3MgeW91IHRvIGxpdGVyYWxseSB2aWV3IGEgdmFyaWFibGUuCllvdSBjYW4gY2xpY2sgb24gdGhlIG9iamVjdCBpbiB0aGUgZW52aXJvbm1lbnQgcGFuZS4gCgpTb21lIHVzZWZ1bCBmdW5jdGlvbnMgZm9yIGV4cGxvcmluZyBvdXIgZGF0YSBmcmFtZSBpbmNsdWRlOgoKKyBgaGVhZCgpYCB0byBzZWUgdGhlIGZpcnN0IDYgcm93cyBvZiBhIGRhdGEgZnJhbWUuIEFkZGl0aW9uYWwgYXJndW1lbnRzIHN1cHBsaWVkIGNhbiBjaGFuZ2UgdGhlIG51bWJlciBvZiByb3dzLgorIGB0YWlsKClgIHRvIHNlZSB0aGUgbGFzdCA2IHJvd3Mgb2YgYSBkYXRhIGZyYW1lLiBBZGRpdGlvbmFsIGFyZ3VtZW50cyBzdXBwbGllZCBjYW4gY2hhbmdlIHRoZSBudW1iZXIgb2Ygcm93cy4KKyBgbmFtZXMoKWAgdG8gc2VlIHRoZSBjb2x1bW4gbmFtZXMgb2YgdGhlIGRhdGEgZnJhbWUuCisgYG5yb3coKWAgdG8gc2VlIGhvdyBtYW55IHJvd3MgYXJlIGluIHRoZSBkYXRhIGZyYW1lCisgYG5jb2woKWAgdG8gc2VlIGhvdyBtYW55IGNvbHVtbnMgYXJlIGluIHRoZSBkYXRhIGZyYW1lLgoKV2UgY2FuIGFkZGl0aW9uYWxseSBleHBsb3JlIF9vdmVyYWxsIHByb3BlcnRpZXNfIG9mIHRoZSBkYXRhIGZyYW1lIHdpdGggdHdvIGRpZmZlcmVudCBmdW5jdGlvbnM6IGBzdW1tYXJ5KClgIGFuZCBgc3RyKClgLgoKVGhpcyBwcm92aWRlcyBzdW1tYXJ5IHN0YXRpc3RpY3MgZm9yIGVhY2ggY29sdW1uOgoKYGBge3IgcGVuZ3VpbnMtc3VtbWFyeX0Kc3VtbWFyeShwZW5ndWlucykKYGBgCgpUaGlzIHByb3ZpZGVzIGEgc2hvcnQgdmlldyBvZiB0aGUgKipzdHIqKnVjdHVyZSBhbmQgY29udGVudHMgb2YgdGhlIGRhdGEgZnJhbWUuCgpgYGB7ciBwZW5ndWlucy1zdHJ9CnN0cihwZW5ndWlucykKYGBgCgpZb3UnbGwgbm90aWNlIHRoYXQgdGhlIGNvbHVtbiBgc3BlY2llc2AgaXMgYSBfZmFjdG9yXzogVGhpcyBpcyBhIHNwZWNpYWwgdHlwZSBvZiBjaGFyYWN0ZXIgdmFyaWFibGUgdGhhdCByZXByZXNlbnRzIGRpc3RpbmN0IGNhdGVnb3JpZXMga25vd24gYXMgImxldmVscyIuIApXZSBoYXZlIGxlYXJuZWQgaGVyZSB0aGF0IHRoZXJlIGFyZSB0aHJlZSBsZXZlbHMgaW4gdGhlIGBzcGVjaWVzYCBjb2x1bW46IEFkZWxpZSwgQ2hpbnN0cmFwLCBhbmQgR2VudG9vLgpXZSBtaWdodCB3YW50IHRvIGV4cGxvcmUgaW5kaXZpZHVhbCBjb2x1bW5zIG9mIHRoZSBkYXRhIGZyYW1lIG1vcmUgaW4tZGVwdGguIApXZSBjYW4gZXhhbWluZSBpbmRpdmlkdWFsIGNvbHVtbnMgdXNpbmcgdGhlIGRvbGxhciBzaWduIGAkYCB0byBzZWxlY3Qgb25lIGJ5IG5hbWU6CgpgYGB7ciBwZW5ndWlucy1zdWJzZXR9CiMgRXh0cmFjdCBiaWxsX2xlbmd0aF9tbSBhcyBhIHZlY3RvcgpwZW5ndWlucyRiaWxsX2xlbmd0aF9tbQoKIyBpbmRleGluZyBvcGVyYXRvcnMgY2FuIGJlIHVzZWQgdG9vCnBlbmd1aW5zJGJpbGxfZGVwdGhfbW1bMToxMF0KYGBgCgpXZSBjYW4gcGVyZm9ybSBvdXIgcmVndWxhciB2ZWN0b3Igb3BlcmF0aW9ucyBvbiBjb2x1bW5zIGRpcmVjdGx5LgoKYGBge3IgcGVuZ3VpbnMtY29sLW1lYW4sIGxpdmUgPSBUUlVFfQojIGNhbGN1bGF0ZSB0aGUgbWVhbiBvZiB0aGUgYmlsbF9sZW5ndGhfbW0gY29sdW1uCm1lYW4ocGVuZ3VpbnMkYmlsbF9sZW5ndGhfbW0sCiAgICAgbmEucm0gPSBUUlVFKSAjIHJlbW92ZSBtaXNzaW5nIHZhbHVlcyBiZWZvcmUgY2FsY3VsYXRpbmcgdGhlIG1lYW4KYGBgCgpXZSBjYW4gYWxzbyBjYWxjdWxhdGUgdGhlIGZ1bGwgc3VtbWFyeSBzdGF0aXN0aWNzIGZvciBhIHNpbmdsZSBjb2x1bW4gZGlyZWN0bHkuIAoKYGBge3IgcGVuZ3VpbnMtY29sLXN1bW1hcnksIGxpdmUgPSBUUlVFfQojIHNob3cgYSBzdW1tYXJ5IG9mIHRoZSBiaWxsX2xlbmd0aF9tbSBjb2x1bW4Kc3VtbWFyeShwZW5ndWlucyRiaWxsX2xlbmd0aF9tbSkKYGBgCgpFeHRyYWN0IGBTcGVjaWVzYCBhcyBhIHZlY3RvciBhbmQgc3Vic2V0IGl0IHRvIHNlZSBhIHByZXZpZXcuCgpgYGB7ciBwZW5ndWlucy1jb2wtc3Vic2V0LCBsaXZlID0gVFJVRX0KIyBnZXQgdGhlIGZpcnN0IDEwIHZhbHVlcyBvZiB0aGUgU3BlY2llcyBjb2x1bW4KcGVuZ3VpbnMkc3BlY2llc1sxOjEwXQpgYGAKCkFuZCB2aWV3IGl0cyBfbGV2ZWxzXyB3aXRoIHRoZSBgbGV2ZWxzKClgIGZ1bmN0aW9uLgoKYGBge3IgcGVuZ3Vpbi1sZXZlbHN9CmxldmVscyhwZW5ndWlucyRzcGVjaWVzKQpgYGAKCiMjIEZpbGVzIGFuZCBkaXJlY3RvcmllcwoKSW4gbWFueSBzaXR1YXRpb25zLCB3ZSB3aWxsIGJlIHJlYWRpbmcgaW4gdGFidWxhciBkYXRhIGZyb20gYSBmaWxlIGFuZCB1c2luZyBpdCBhcyBhIGRhdGEgZnJhbWUuIApUbyBwcmFjdGljZSwgd2Ugd2lsbCByZWFkIGluIGEgZmlsZSB3ZSB3aWxsIGJlIHVzaW5nIGluIHRoZSBuZXh0IG5vdGVib29rIGFzIHdlbGwsIGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YCwgaW4gdGhlIGBkYXRhYCBmb2xkZXIuIApGaWxlIHBhdGhzIGFyZSByZWxhdGl2ZSB0byB0aGUgbG9jYXRpb24gd2hlcmUgdGhpcyBub3RlYm9vayBmaWxlICguUm1kKSBpcyBzYXZlZC4KCkhlcmUgd2Ugd2lsbCB1c2UgYSBmdW5jdGlvbiwgYHJlYWRfdHN2KClgIGZyb20gdGhlIGByZWFkcmAgcGFja2FnZS4KQmVmb3JlIHdlIGFyZSBhYmxlIHRvIHVzZSB0aGUgZnVuY3Rpb24sIHdlIGhhdmUgdG8gbG9hZCB0aGUgcGFja2FnZSB1c2luZyBgbGlicmFyeSgpYC4gCgpgYGB7ciByZWFkcn0KbGlicmFyeShyZWFkcikKYGBgCgpgZmlsZS5wYXRoKClgIGNyZWF0ZXMgYSBwcm9wZXJseSBmb3JtYXR0ZWQgZmlsZSBwYXRoIGJ5IGFkZGluZyBhIHBhdGggc2VwYXJhdG9yIChgL2Agb24gTWFjIGFuZCBMaW51eCBvcGVyYXRpbmcgc3lzdGVtLCB3aGljaCBpcyB0aGUgb3BlcmF0aW5nIHN5c3RlbSB0aGF0IG91ciBSU3R1ZGlvIFNlcnZlciBydW5zIG9uKSBiZXR3ZWVuIHNlcGFyYXRlIGZvbGRlcnMgb3IgZGlyZWN0b3JpZXMuCkJlY2F1c2UgZmlsZSBwYXRoIHNlcGFyYXRvcnMgY2FuIGRpZmZlciBiZXR3ZWVuIHlvdXIgY29tcHV0ZXIgYW5kIHRoZSBjb21wdXRlciBvZiBzb21lb25lIHdobyB3YW50cyB0byB1c2UgeW91ciBjb2RlLCB3ZSB1c2UgYGZpbGUucGF0aCgpYCBpbnN0ZWFkIG9mIHR5cGluZyBvdXQgYCJkYXRhL2dlbmVfcmVzdWx0c19HU0U0NDk3MS50c3YiYC4KRWFjaCBfYXJndW1lbnRfIHRvIGBmaWxlLnBhdGgoKWAgaXMgYSBkaXJlY3Rvcnkgb3IgZmlsZSBuYW1lLgpZb3UnbGwgbm90aWNlIGVhY2ggYXJndW1lbnQgaXMgaW4gcXVvdGVzLCB3ZSBzcGVjaWZ5IGBkYXRhYCBmaXJzdCBiZWNhdXNlIHRoZSBmaWxlLCBgZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdmAgaXMgaW4gdGhlIGBkYXRhYCBmb2xkZXIuIAoKYGBge3IgZmlsZS5wYXRofQpmaWxlLnBhdGgoImRhdGEiLCAiZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdiIpCmBgYAoKV2UgY2FuIHN0b3JlIHRoaXMgZmlsZSBwYXRoIGFzIGEgdmFyaWFibGUgaW4gb3VyIGVudmlyb25tZW50LiAKCmBgYHtyIGZpbGUucGF0aC12YXJpYWJsZX0KZ2VuZV9maWxlX3BhdGggPC0gZmlsZS5wYXRoKCJkYXRhIiwgImdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3YiKQpgYGAKCk5vdyB3ZSBhcmUgcmVhZHkgdG8gdXNlIGByZWFkX3RzdigpYCB0byByZWFkIHRoZSBmaWxlIGludG8gUi4KVGhlIHJlc3VsdGluZyBkYXRhIGZyYW1lIHdpbGwgYmUgc3RvcmVkIGluIGEgdmFyaWFibGUgbmFtZWQgYHN0YXRzX2RmYC4KTm90ZSB0aGUgYDwtYCB3aGljaCBpcyByZXNwb25zaWJsZSBmb3Igc2F2aW5nIHRoaXMgdG8gb3VyIGdsb2JhbCBlbnZpcm9ubWVudC4gCgpgYGB7ciByZWFkLXN0YXRzfQojIHJlYWQgaW4gdGhlIGZpbGUgYGdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3ZgIGZyb20gdGhlIGRhdGEgZGlyZWN0b3J5CnN0YXRzX2RmIDwtIHJlYWRfdHN2KGdlbmVfZmlsZV9wYXRoKQpgYGAKClRha2UgYSBsb29rIGF0IHlvdXIgZW52aXJvbm1lbnQgcGFuZWwgdG8gc2VlIHdoYXQgYHN0YXRzX2RmYCBsb29rcyBsaWtlLiAKV2UgY2FuIGFsc28gcHJpbnQgb3V0IGEgcHJldmlldyBvZiB0aGUgYHN0YXRzX2RmYCBkYXRhIGZyYW1lIGhlcmUuIAoKYGBge3Igc2hvdy1zdGF0cywgbGl2ZSA9IFRSVUV9CiMgZGlzcGxheSBzdGF0c19kZgpzdGF0c19kZgpgYGAKCiMjIyBTZXNzaW9uIEluZm8KCkF0IHRoZSBlbmQgb2YgZXZlcnkgbm90ZWJvb2ssIHlvdSB3aWxsIHNlZSB1cyBwcmludCBvdXQgYHNlc3Npb25JbmZvYC4gClRoaXMgYWlkcyBpbiB0aGUgcmVwcm9kdWNpYmlsaXR5IG9mIHlvdXIgY29kZSBieSBzaG93aW5nIGV4YWN0bHkgd2hhdCBwYWNrYWdlcyAKYW5kIHZlcnNpb25zIHdlcmUgYmVpbmcgdXNlZCB0aGUgbGFzdCB0aW1lIHRoZSBub3RlYm9vayB3YXMgcnVuLgoKYGBge3J9CnNlc3Npb25JbmZvKCkKYGBgCg==
    
 
 
 
diff --git a/intro-to-R-tidyverse/02-intro_to_ggplot2-live.Rmd b/intro-to-R-tidyverse/02-intro_to_ggplot2-live.Rmd
index 8084d96b..c011309e 100644
--- a/intro-to-R-tidyverse/02-intro_to_ggplot2-live.Rmd
+++ b/intro-to-R-tidyverse/02-intro_to_ggplot2-live.Rmd
@@ -1,14 +1,24 @@
 ---
 title: "Introduction to ggplot2"
+author: "CCDL for ALSF" 
+date: 2021
 output:   
   html_notebook: 
     toc: true
     toc_float: true
 ---
 
-**CCDL 2020**
 
-## Objective for this notebook analysis: 
+## Objectives
+
+This notebook will demonstrate how to: 
+
+- Load and use R packages  
+- Read in and perform simple manipulations of data frames 
+- Use `ggplot2` to plot and visualize data
+- Customize plots using features of `ggplot2`
+
+---
 
 We'll use a real gene expression dataset to get comfortable making visualizations using ggplot2. 
 We've [performed differential expression analyses](./scripts/00-setup-intro-to-R.R) on a pre-processed [astrocytoma microarray dataset](https://www.refine.bio/experiments/GSE44971/gene-expression-data-from-pilocytic-astrocytoma-tumour-samples-and-normal-cerebellum-controls).
@@ -20,10 +30,11 @@ We performed three sets of contrasts:
 3) An interaction of both `sex` and `tissue`. 
 
 **More ggplot2 resources:**  
-+ [handy cheatsheet for ggplot2](https://rstudio.com/wp-content/uploads/2015/03/ggplot2-cheatsheet.pdf)  
-+ [ggplot2 courses and books from Tidyverse](https://ggplot2.tidyverse.org/)  
-+ [ggplot2 data viz chapter of R for Data Science](https://r4ds.had.co.nz/data-visualisation.html)  
-+ [ggplot2 online tutorial](http://r-statistics.co/Complete-Ggplot2-Tutorial-Part1-With-R-Code.html)  
+
+- [handy cheatsheet for ggplot2](https://rstudio.com/wp-content/uploads/2015/03/ggplot2-cheatsheet.pdf)  
+- [ggplot2 courses and books from Tidyverse](https://ggplot2.tidyverse.org/)  
+- [ggplot2 data viz chapter of R for Data Science](https://r4ds.had.co.nz/data-visualisation.html)  
+- [ggplot2 online tutorial](http://r-statistics.co/Complete-Ggplot2-Tutorial-Part1-With-R-Code.html)  
 
 ## Set Up
 
@@ -217,7 +228,6 @@ For now, we will choose the value of 5.5 (that is close to a Bonferroni correcti
 
 ```
 
-We can also change the background and appearance of the plot as a whole by adding a `theme`.
 We can change the x and y labels by using `ylab` and `xlab` functions and add a
 title using `ggtitle`.
 
diff --git a/intro-to-R-tidyverse/02-intro_to_ggplot2.nb.html b/intro-to-R-tidyverse/02-intro_to_ggplot2.nb.html
index 315f553a..e83cc0f2 100644
--- a/intro-to-R-tidyverse/02-intro_to_ggplot2.nb.html
+++ b/intro-to-R-tidyverse/02-intro_to_ggplot2.nb.html
@@ -323,10 +323,11 @@ 
    Objectives
    
 
+
    
 
    We’ll use a real gene expression dataset to get comfortable making visualizations using ggplot2. We’ve performed differential expression analyses on a pre-processed astrocytoma microarray dataset. We’ll start by making a volcano plot of differential gene expression results from this experiment. We performed three sets of contrasts:
    
 
      
 
    1. sex category contrasting: Male vs Female
      
@@ -351,7 +352,7 @@ 
      Set Up
      
 
      We saved these results to a tab separated values (TSV) file called gene_results_GSE44971.tsv. It’s been saved to the data folder. File paths are relative to where this notebook file (.Rmd) is saved. So we can reference it later, let’s make a variable with our data directory name.
      
 
 
-
+
 
      data_dir <- "data"
      
 
 
@@ -359,7 +360,7 @@ 
      Set Up
      
 
      Let’s declare our output folder name as its own variable.
      
 
 
-
+
 
      plots_dir <- "plots"
      
 
 
@@ -367,7 +368,7 @@ 
      Set Up
      
 
      We can also create a directory if it doesn’t already exist.
      
 
 
-
+
 
      # The if statement here tests whether the plot directory exists and
 # only executes the  expressions between the braces if it does not.
 if (!dir.exists(plots_dir)) {
@@ -379,9 +380,23 @@ 
      Set Up
      
 
      In this notebook we will be using functions from the Tidyverse set of packages, so we need to load in those functions using library(). We could load the individual packages we need one at a time, but it is convenient for now to load them all with the tidyverse package, which groups many of them together. Keep a look out for where we tell you which individual package different functions come from.
      
 
 
-
+
 
      library(tidyverse)
      
 
+
+
      ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
      
+
+
+
      ✔ ggplot2 3.3.3     ✔ purrr   0.3.4
+✔ tibble  3.0.5     ✔ dplyr   1.0.3
+✔ tidyr   1.1.2     ✔ stringr 1.4.0
+✔ readr   1.4.0     ✔ forcats 0.5.0
      
+
+
+
      ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
+✖ dplyr::filter() masks stats::filter()
+✖ dplyr::lag()    masks stats::lag()
      
+
 
 
 
@@ -390,47 +405,101 @@ 
      Read in the differential expression analysis results file
      
 
      Here we are using a tidyverse function read_tsv() from the readr package. Like we did in the previous notebook, we will store the resulting data frame as stats_df.
      
 
 
-
+
 
      # read in the file `gene_results_GSE44971.tsv` from the data directory
 stats_df <- read_tsv(file.path(
   data_dir,
   "gene_results_GSE44971.tsv"
 ))
      
 
+
+
      
+── Column specification ────────────────────────────────────────────────────────
+cols(
+  ensembl_id = col_character(),
+  gene_symbol = col_character(),
+  contrast = col_character(),
+  log_fold_change = col_double(),
+  avg_expression = col_double(),
+  t_statistic = col_double(),
+  p_value = col_double(),
+  adj_p_value = col_double()
+)
      
+
 
 
 
      We can take a look at a column individually by using a $. Note we are using head() so the whole thing doesn’t print out.
      
 
 
-
+
 
      head(stats_df$contrast)
      
 
+
+
      [1] "male_female" "male_female" "male_female" "male_female" "male_female"
+[6] "male_female"
      
+
+
+
      male_female
+
+male_female
+
+male_female
+
+male_female
+
+male_female
+
+male_female
      
+
 
 
 
      If we want to see a specific set of values, we can use brackets with the indices of the values we’d like returned.
      
 
 
-
+
 
      stats_df$avg_expression[6:10]
      
 
+
+
      [1] 19.084011  8.453933  5.116563  6.345609 25.473133
      
+
 
 
 
      Let’s look at some basic statistics from the data set using summary()
      
 
 
-
+
 
      # summary of stats_df
 summary(stats_df)
      
 
+
+
        ensembl_id        gene_symbol          contrast         log_fold_change    
+ Length:6804        Length:6804        Length:6804        Min.   :-180.8118  
+ Class :character   Class :character   Class :character   1st Qu.:  -1.6703  
+ Mode  :character   Mode  :character   Mode  :character   Median :   0.1500  
+                                                          Mean   :   0.2608  
+                                                          3rd Qu.:   2.1049  
+                                                          Max.   : 129.3009  
+ avg_expression     t_statistic           p_value         adj_p_value     
+ Min.   :  5.003   Min.   :-32.84581   Min.   :0.00000   Min.   :0.00000  
+ 1st Qu.:  6.304   1st Qu.: -1.16444   1st Qu.:0.01309   1st Qu.:0.05657  
+ Median :  8.482   Median :  0.10619   Median :0.18919   Median :0.41354  
+ Mean   : 13.847   Mean   : -0.00819   Mean   :0.31223   Mean   :0.44833  
+ 3rd Qu.: 14.022   3rd Qu.:  1.46589   3rd Qu.:0.57634   3rd Qu.:0.82067  
+ Max.   :190.708   Max.   : 10.48302   Max.   :0.99979   Max.   :0.99988  
      
+
 
 
 
      The statistics for contrast are not very informative, so let’s do that again with just the contrast column after converting it to a factor
      
 
 
-
+
 
      # summary of `stats_df$contrast` as a factor
 summary(as.factor(stats_df$contrast))
      
 
+
+
      astrocytoma_normal        interaction        male_female 
+              2268               2268               2268 
      
+
 
 
 
@@ -439,7 +508,7 @@ 
      Set up the dataset
      
 
      Before we make our plot, we want to calculate a set of new values for each row; transformations of the raw statistics in our table. To do this we will use a function from the dplyr package called mutate() to make a new column of -log10 p values.
      
 
 
-
+
 
      # add a `neg_log10_p` column to the data frame
 stats_df <- mutate(stats_df, # data frame we'd like to add a variable to
   neg_log10_p = -log10(p_value) # column name and values
@@ -458,16 +527,21 @@ 
      Set up the dataset
      
 
      Now we can try out the filter() function.Notice that we are not assigning the results to a variable, so this filtered dataset will not be saved to the environment.
      
 
 
-
+
 
      # filter stats_df to "male_female" only
 filter(stats_df, contrast == "male_female")
      
 
+
      
+
+
      
 
 
 
      Now we can assign the results to a new data frame: male_female_df.
      
 
 
-
+
 
      # filter and save to male_female_df
 male_female_df <- filter(stats_df, contrast == "male_female")
      
 
@@ -479,7 +553,7 @@ 
      Plotting this data
      
 
      Let’s make a volcano plot with this data. First let’s take a look at only the tumor vs. normal comparison. Let’s save this as a separate data frame by assigning it a new name.
      
 
 
-
+
 
      tumor_normal_df <- filter(stats_df, contrast == "astrocytoma_normal")
      
 
 
@@ -492,7 +566,7 @@ 
      Plotting this data
      
 
    
 
 
-
+
 
    ggplot(
   tumor_normal_df, # This first argument is the data frame with the data we want to plot
   aes(
@@ -502,12 +576,15 @@ 
    Plotting this data
    
   ) # This is the column name of the data we want for the y-axis
 )
    
 
+
+
    
+
 
 
 
    You’ll notice this plot doesn’t have anything on it because we haven’t specified a plot type yet. To do that, we will add another ggplot layer with + which will specify exactly what we want to plot. A volcano plot is a special kind of scatter plot, so to make that we will want to plot individual points, which we can do with geom_point().
    
 
 
-
+
 
    # This first part is the same as before
 ggplot(
   tumor_normal_df,
@@ -519,6 +596,9 @@ 
    Plotting this data
    
   # Now we are adding on a layer to specify what kind of plot we want
   geom_point()
    
 
+
+
    
+
 
 
 
    Here’s a brief summary of ggplot2 structure. ggplot2 structure
    
@@ -527,7 +607,7 @@ 
    Adjust our ggplot
    
 
    Now that we have a base plot that shows our data, we can add layers on to it and adjust it. We can adjust the color of points using the color aesthetic.
    
 
 
-
+
 
    ggplot(
   tumor_normal_df,
   aes(
@@ -538,12 +618,15 @@ 
    Adjust our ggplot
    
 ) +
   geom_point()
    
 
+
+
    
+
 
 
 
    Because we have so many points overlapping one another, we will want to adjust the transparency, which we can do with an alpha argument.
    
 
 
-
+
 
    ggplot(
   tumor_normal_df,
   aes(
@@ -554,12 +637,15 @@ 
    Adjust our ggplot
    
 ) +
   geom_point(alpha = 0.2) # We are using the `alpha` argument to make our points transparent
    
 
+
+
    
+
 
 
 
    Notice that we added the alpha within the geom_point() function, not to the aes(). We did this because we want all of the points to have the same level of transparency, and it will not vary depending on any variable in the data. We can also change the background and appearance of the plot as a whole by adding a theme.
    
 
 
-
+
 
    ggplot(
   tumor_normal_df,
   aes(
@@ -571,12 +657,15 @@ 
    Adjust our ggplot
    
   geom_point(alpha = 0.2) +
   theme_bw() # Add on this set of appearance presets to make it pretty
    
 
+
+
    
+
 
 
 
    We are not limited to a single plotting layer. For example, if we want to add a horizontal line to indicate a significance cutoff, we can do that with geom_hline(). For now, we will choose the value of 5.5 (that is close to a Bonferroni correction) and add that to the plot.
    
 
 
-
+
 
    ggplot(
   tumor_normal_df,
   aes(
@@ -588,12 +677,15 @@ 
    Adjust our ggplot
    
   geom_point(alpha = 0.2) + 
   geom_hline(yintercept = 5.5, color = "darkgreen") # we can specify colors by names here
    
 
+
+
    
+
 
 
 
    We can change the x and y labels by using ylab and xlab functions and add a title using ggtitle.
    
 
 
-
+
 
    ggplot(
   tumor_normal_df,
   aes(
@@ -609,21 +701,23 @@ 
    Adjust our ggplot
    
   ylab("-log10 p value") + # Add a y label
   ggtitle("Astrocytoma Tumor vs Normal Cerebellum") # Add main title
    
 
+
+
    
+
 
 
 
    Use this chunk to make the same kind of plot as the previous chunk but instead plot the male female contrast data, that is stored in male_female_df.
    
 
 
-
-
    # Use this chunk to make the same kind of volcano plot, but with the male-female contrast data.
-
    
+
+
    # Use this chunk to make the same kind of volcano plot, but with the male-female contrast data.
    
 
 
 
 
    Turns out, we don’t have to plot each contrast separately, instead, we can use the original data frame that contains all three contrasts’ data, stats_df, and add a facet_wrap to make each contrast its own plot.
    
 
 
-
+
 
    ggplot(
   stats_df, # Switch to the bigger data frame with all three contrasts' data
   aes(
@@ -641,12 +735,15 @@ 
    Adjust our ggplot
    
   ylab("-log10 p value") +
   coord_cartesian(xlim = c(-25, 25)) # zoom in on the x-axis
    
 
+
+
    
+
 
 
 
    We can store the plot as an object in the global environment by using <- operator. Here we will call it volcano_plot.
    
 
 
-
+
 
    volcano_plot <- ggplot(
   stats_df, # We are calling this plot `volcano_plot`
   aes(
@@ -668,12 +765,15 @@ 
    Adjust our ggplot
    
 
    When we are happy with our plot, we can save the plot using ggsave.
    
 
 
-
+
 
    ggsave(
   plot = volcano_plot,
   filename = file.path(plots_dir, "volcano_plot.png")
 )
    
 
+
+
    Saving 7 x 5 in image
    
+
 
 
 
@@ -681,15 +781,55 @@ 
    Adjust our ggplot
    
 
    Session Info
    
 
 
-
+
 
    # Print out the versions and packages we are using in this session
 sessionInfo()
    
 
+
+
    R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] stats     graphics  grDevices utils     datasets  methods   base     
+
+other attached packages:
+ [1] forcats_0.5.0   stringr_1.4.0   dplyr_1.0.3     purrr_0.3.4    
+ [5] readr_1.4.0     tidyr_1.1.2     tibble_3.0.5    ggplot2_3.3.3  
+ [9] tidyverse_1.3.0 optparse_1.6.6 
+
+loaded via a namespace (and not attached):
+ [1] tidyselect_1.1.0  xfun_0.20         haven_2.3.1       colorspace_2.0-0 
+ [5] vctrs_0.3.6       generics_0.1.0    htmltools_0.5.1.1 getopt_1.20.3    
+ [9] yaml_2.2.1        rlang_0.4.10      pillar_1.4.7      glue_1.4.2       
+[13] withr_2.4.0       DBI_1.1.1         dbplyr_2.0.0      modelr_0.1.8     
+[17] readxl_1.3.1      lifecycle_0.2.0   munsell_0.5.0     gtable_0.3.0     
+[21] cellranger_1.1.0  rvest_0.3.6       evaluate_0.14     labeling_0.4.2   
+[25] knitr_1.30        ps_1.5.0          fansi_0.4.2       broom_0.7.3      
+[29] Rcpp_1.0.6        scales_1.1.1      backports_1.2.1   jsonlite_1.7.2   
+[33] farver_2.0.3      fs_1.5.0          hms_1.0.0         digest_0.6.27    
+[37] stringi_1.5.3     grid_4.0.3        cli_2.2.0         tools_4.0.3      
+[41] magrittr_2.0.1    crayon_1.3.4      pkgconfig_2.0.3   ellipsis_0.3.1   
+[45] xml2_1.3.2        reprex_0.3.0      lubridate_1.7.9.2 assertthat_0.2.1 
+[49] rmarkdown_2.6     httr_1.4.2        rstudioapi_0.13   R6_2.5.0         
+[53] compiler_4.0.3   
    
+
 
 
 
 
-
    LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIGdncGxvdDIiCmF1dGhvcjogIkNDREwgZm9yIEFMU0YiIApkYXRlOiAyMDIxCm91dHB1dDogICAKICBodG1sX25vdGVib29rOiAKICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OiB0cnVlCi0tLQoKCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86IAoKLSBMb2FkIGFuZCB1c2UgUiBwYWNrYWdlcyAgCi0gTWFuaXB1bGF0ZSBkYXRhIGZyYW1lcyAKLSBVc2UgZ2dwbG90IHRvIHBsb3QgYW5kIHZpc3VhbGl6ZSBkYXRhCi0gQ3VzdG9taXplIHBsb3RzIHVzaW5nIGZlYXR1cmVzIG9mIGdncGxvdAoKV2UnbGwgdXNlIGEgcmVhbCBnZW5lIGV4cHJlc3Npb24gZGF0YXNldCB0byBnZXQgY29tZm9ydGFibGUgbWFraW5nIHZpc3VhbGl6YXRpb25zIHVzaW5nIGdncGxvdDIuIApXZSd2ZSBbcGVyZm9ybWVkIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2VzXSguL3NjcmlwdHMvMDAtc2V0dXAtaW50cm8tdG8tUi5SKSBvbiBhIHByZS1wcm9jZXNzZWQgW2FzdHJvY3l0b21hIG1pY3JvYXJyYXkgZGF0YXNldF0oaHR0cHM6Ly93d3cucmVmaW5lLmJpby9leHBlcmltZW50cy9HU0U0NDk3MS9nZW5lLWV4cHJlc3Npb24tZGF0YS1mcm9tLXBpbG9jeXRpYy1hc3Ryb2N5dG9tYS10dW1vdXItc2FtcGxlcy1hbmQtbm9ybWFsLWNlcmViZWxsdW0tY29udHJvbHMpLgpXZSdsbCBzdGFydCBieSBtYWtpbmcgYSB2b2xjYW5vIHBsb3Qgb2YgZGlmZmVyZW50aWFsIGdlbmUgZXhwcmVzc2lvbiByZXN1bHRzIGZyb20gdGhpcyBleHBlcmltZW50LgpXZSBwZXJmb3JtZWQgdGhyZWUgc2V0cyBvZiBjb250cmFzdHM6ICAKCjEpIGBzZXhgIGNhdGVnb3J5IGNvbnRyYXN0aW5nOiBgTWFsZWAgdnMgYEZlbWFsZWAgIAoyKSBgdGlzc3VlYCBjYXRlZ29yeSBjb250cmFzdGluZyA6IGBQaWxvY3l0aWMgYXN0cm9jeXRvbWEgdHVtb3JgIHNhbXBsZXMgdnMgYG5vcm1hbCBjZXJlYmVsbHVtYCBzYW1wbGVzICAKMykgQW4gaW50ZXJhY3Rpb24gb2YgYm90aCBgc2V4YCBhbmQgYHRpc3N1ZWAuIAoKKipNb3JlIGdncGxvdDIgcmVzb3VyY2VzOioqICAKCi0gW2hhbmR5IGNoZWF0c2hlZXQgZm9yIGdncGxvdDJdKGh0dHBzOi8vcnN0dWRpby5jb20vd3AtY29udGVudC91cGxvYWRzLzIwMTUvMDMvZ2dwbG90Mi1jaGVhdHNoZWV0LnBkZikgIAotIFtnZ3Bsb3QyIGNvdXJzZXMgYW5kIGJvb2tzIGZyb20gVGlkeXZlcnNlXShodHRwczovL2dncGxvdDIudGlkeXZlcnNlLm9yZy8pICAKLSBbZ2dwbG90MiBkYXRhIHZpeiBjaGFwdGVyIG9mIFIgZm9yIERhdGEgU2NpZW5jZV0oaHR0cHM6Ly9yNGRzLmhhZC5jby5uei9kYXRhLXZpc3VhbGlzYXRpb24uaHRtbCkgIAotIFtnZ3Bsb3QyIG9ubGluZSB0dXRvcmlhbF0oaHR0cDovL3Itc3RhdGlzdGljcy5jby9Db21wbGV0ZS1HZ3Bsb3QyLVR1dG9yaWFsLVBhcnQxLVdpdGgtUi1Db2RlLmh0bWwpICAKCiMjIFNldCBVcAoKV2Ugc2F2ZWQgdGhlc2UgcmVzdWx0cyB0byBhIHRhYiBzZXBhcmF0ZWQgdmFsdWVzIChUU1YpIGZpbGUgY2FsbGVkIGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YC4KSXQncyBiZWVuIHNhdmVkIHRvIHRoZSBgZGF0YWAgZm9sZGVyLiAKRmlsZSBwYXRocyBhcmUgcmVsYXRpdmUgdG8gd2hlcmUgdGhpcyBub3RlYm9vayBmaWxlICguUm1kKSBpcyBzYXZlZC4KU28gd2UgY2FuIHJlZmVyZW5jZSBpdCBsYXRlciwgbGV0J3MgbWFrZSBhIHZhcmlhYmxlIHdpdGggb3VyIGRhdGEgZGlyZWN0b3J5IG5hbWUuIAoKYGBge3J9CmRhdGFfZGlyIDwtICJkYXRhIgpgYGAKCkxldCdzIGRlY2xhcmUgb3VyIG91dHB1dCBmb2xkZXIgbmFtZSBhcyBpdHMgb3duIHZhcmlhYmxlLiAKCmBgYHtyfQpwbG90c19kaXIgPC0gInBsb3RzIgpgYGAKCldlIGNhbiBhbHNvIGNyZWF0ZSBhIGRpcmVjdG9yeSBpZiBpdCBkb2Vzbid0IGFscmVhZHkgZXhpc3QuIAoKYGBge3IgY3JlYXRlaWZ9CiMgVGhlIGlmIHN0YXRlbWVudCBoZXJlIHRlc3RzIHdoZXRoZXIgdGhlIHBsb3QgZGlyZWN0b3J5IGV4aXN0cyBhbmQKIyBvbmx5IGV4ZWN1dGVzIHRoZSAgZXhwcmVzc2lvbnMgYmV0d2VlbiB0aGUgYnJhY2VzIGlmIGl0IGRvZXMgbm90LgppZiAoIWRpci5leGlzdHMocGxvdHNfZGlyKSkgewogIGRpci5jcmVhdGUocGxvdHNfZGlyKQp9CmBgYAoKSW4gdGhpcyBub3RlYm9vayB3ZSB3aWxsIGJlIHVzaW5nIGZ1bmN0aW9ucyBmcm9tIHRoZSBUaWR5dmVyc2Ugc2V0IG9mIHBhY2thZ2VzLCBzbyB3ZSBuZWVkIHRvIGxvYWQgaW4gdGhvc2UgZnVuY3Rpb25zIHVzaW5nIGBsaWJyYXJ5KClgLgpXZSBjb3VsZCBsb2FkIHRoZSBpbmRpdmlkdWFsIHBhY2thZ2VzIHdlIG5lZWQgb25lIGF0IGEgdGltZSwgYnV0IGl0IGlzIGNvbnZlbmllbnQgZm9yIG5vdyB0byBsb2FkIHRoZW0gYWxsIHdpdGggdGhlIGB0aWR5dmVyc2VgIHBhY2thZ2UsIHdoaWNoIGdyb3VwcyBtYW55IG9mIHRoZW0gdG9nZXRoZXIuCktlZXAgYSBsb29rIG91dCBmb3Igd2hlcmUgd2UgdGVsbCB5b3Ugd2hpY2ggaW5kaXZpZHVhbCBwYWNrYWdlIGRpZmZlcmVudCBmdW5jdGlvbnMgY29tZSBmcm9tLgoKYGBge3IgdGlkeXZlcnNlfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIyBSZWFkIGluIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyByZXN1bHRzIGZpbGUKCkhlcmUgd2UgYXJlIHVzaW5nIGEgYHRpZHl2ZXJzZWAgZnVuY3Rpb24gYHJlYWRfdHN2KClgIGZyb20gdGhlIGByZWFkcmAgcGFja2FnZS4KTGlrZSB3ZSBkaWQgaW4gdGhlIHByZXZpb3VzIG5vdGVib29rLCB3ZSB3aWxsIHN0b3JlIHRoZSByZXN1bHRpbmcgZGF0YSBmcmFtZSBhcyBgc3RhdHNfZGZgLgoKYGBge3IgcmVhZC1zdGF0c30KIyByZWFkIGluIHRoZSBmaWxlIGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YCBmcm9tIHRoZSBkYXRhIGRpcmVjdG9yeQpzdGF0c19kZiA8LSByZWFkX3RzdihmaWxlLnBhdGgoCiAgZGF0YV9kaXIsCiAgImdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3YiCikpCmBgYAoKV2UgY2FuIHRha2UgYSBsb29rIGF0IGEgY29sdW1uIGluZGl2aWR1YWxseSBieSB1c2luZyBhIGAkYC4gCk5vdGUgd2UgYXJlIHVzaW5nIGBoZWFkKClgIHNvIHRoZSB3aG9sZSB0aGluZyBkb2Vzbid0IHByaW50IG91dC4gCgpgYGB7ciBjb2x1bW59CmhlYWQoc3RhdHNfZGYkY29udHJhc3QpCmBgYAoKSWYgd2Ugd2FudCB0byBzZWUgYSBzcGVjaWZpYyBzZXQgb2YgdmFsdWVzLCB3ZSBjYW4gdXNlIGJyYWNrZXRzIHdpdGggdGhlIGluZGljZXMgb2YgdGhlIHZhbHVlcyB3ZSdkIGxpa2UgcmV0dXJuZWQuCgpgYGB7cn0Kc3RhdHNfZGYkYXZnX2V4cHJlc3Npb25bNjoxMF0KYGBgCgpMZXQncyBsb29rIGF0IHNvbWUgYmFzaWMgc3RhdGlzdGljcyBmcm9tIHRoZSBkYXRhIHNldCB1c2luZyBgc3VtbWFyeSgpYAoKYGBge3Igc3RhdHMtc3VtbWFyeSwgbGl2ZSA9IFRSVUV9CiMgc3VtbWFyeSBvZiBzdGF0c19kZgpzdW1tYXJ5KHN0YXRzX2RmKQpgYGAKClRoZSBzdGF0aXN0aWNzIGZvciBgY29udHJhc3RgIGFyZSBub3QgdmVyeSBpbmZvcm1hdGl2ZSwgc28gbGV0J3MgZG8gdGhhdCBhZ2FpbiB3aXRoIGp1c3QgdGhlIGBjb250cmFzdGAgY29sdW1uIGFmdGVyIGNvbnZlcnRpbmcgaXQgdG8gYSBgZmFjdG9yYApgYGB7ciBmYWN0b3Itc3VtbWFyeSwgbGl2ZSA9IFRSVUV9CiMgc3VtbWFyeSBvZiBgc3RhdHNfZGYkY29udHJhc3RgIGFzIGEgZmFjdG9yCnN1bW1hcnkoYXMuZmFjdG9yKHN0YXRzX2RmJGNvbnRyYXN0KSkKYGBgCgojIyBTZXQgdXAgdGhlIGRhdGFzZXQKCkJlZm9yZSB3ZSBtYWtlIG91ciBwbG90LCB3ZSB3YW50IHRvIGNhbGN1bGF0ZSBhIHNldCBvZiBuZXcgdmFsdWVzIGZvciBlYWNoIHJvdzsgdHJhbnNmb3JtYXRpb25zIG9mIHRoZSByYXcgc3RhdGlzdGljcyBpbiBvdXIgdGFibGUuClRvIGRvIHRoaXMgd2Ugd2lsbCB1c2UgYSBmdW5jdGlvbiBmcm9tIHRoZSBgZHBseXJgIHBhY2thZ2UgY2FsbGVkIGBtdXRhdGUoKWAgdG8gbWFrZSBhIG5ldyBjb2x1bW4gb2YgLWxvZzEwIHAgdmFsdWVzLgoKYGBge3IgbXV0YXRlfQojIGFkZCBhIGBuZWdfbG9nMTBfcGAgY29sdW1uIHRvIHRoZSBkYXRhIGZyYW1lCnN0YXRzX2RmIDwtIG11dGF0ZShzdGF0c19kZiwgIyBkYXRhIGZyYW1lIHdlJ2QgbGlrZSB0byBhZGQgYSB2YXJpYWJsZSB0bwogIG5lZ19sb2cxMF9wID0gLWxvZzEwKHBfdmFsdWUpICMgY29sdW1uIG5hbWUgYW5kIHZhbHVlcwopCmBgYAoKTGV0J3MgZmlsdGVyIHRvIG9ubHkgYG1hbGVfZmVtYWxlYCBjb250cmFzdCBkYXRhLiAKRmlyc3QgbGV0J3MgdHJ5IG91dCBhIGxvZ2ljYWwgZXhwcmVzc2lvbjogCgpgYGB7ciBldmFsID0gRkFMU0V9CnN0YXRzX2RmJGNvbnRyYXN0ID09ICJtYWxlX2ZlbWFsZSIKYGBgCgpOb3cgd2UgY2FuIHRyeSBvdXQgdGhlIGBmaWx0ZXIoKWAgZnVuY3Rpb24uTm90aWNlIHRoYXQgd2UgYXJlIG5vdCBhc3NpZ25pbmcgdGhlIHJlc3VsdHMgdG8gYSB2YXJpYWJsZSwgc28gdGhpcyBmaWx0ZXJlZCBkYXRhc2V0IHdpbGwgbm90IGJlIHNhdmVkIHRvIHRoZSBlbnZpcm9ubWVudC4KCmBgYHtyIGZpbHRlciwgbGl2ZSA9IFRSVUV9CiMgZmlsdGVyIHN0YXRzX2RmIHRvICJtYWxlX2ZlbWFsZSIgb25seQpmaWx0ZXIoc3RhdHNfZGYsIGNvbnRyYXN0ID09ICJtYWxlX2ZlbWFsZSIpCmBgYAoKTm93IHdlIGNhbiBhc3NpZ24gdGhlIHJlc3VsdHMgdG8gYSBuZXcgZGF0YSBmcmFtZTogYG1hbGVfZmVtYWxlX2RmYC4gCgpgYGB7ciBmaWx0ZXItc2F2ZSwgbGl2ZSA9IFRSVUV9CiMgZmlsdGVyIGFuZCBzYXZlIHRvIG1hbGVfZmVtYWxlX2RmCm1hbGVfZmVtYWxlX2RmIDwtIGZpbHRlcihzdGF0c19kZiwgY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIikKYGBgCgojIyBQbG90dGluZyB0aGlzIGRhdGEKCkxldCdzIG1ha2UgYSB2b2xjYW5vIHBsb3Qgd2l0aCB0aGlzIGRhdGEuIApGaXJzdCBsZXQncyB0YWtlIGEgbG9vayBhdCBvbmx5IHRoZSB0dW1vciB2cy4gbm9ybWFsIGNvbXBhcmlzb24uIApMZXQncyBzYXZlIHRoaXMgYXMgYSBzZXBhcmF0ZSBkYXRhIGZyYW1lIGJ5IGFzc2lnbmluZyBpdCBhIG5ldyBuYW1lLiAKCmBgYHtyIGZpbHRlci10dW1vcn0KdHVtb3Jfbm9ybWFsX2RmIDwtIGZpbHRlcihzdGF0c19kZiwgY29udHJhc3QgPT0gImFzdHJvY3l0b21hX25vcm1hbCIpCmBgYAoKVG8gbWFrZSB0aGlzIHBsb3Qgd2Ugd2lsbCBiZSB1c2luZyBmdW5jdGlvbnMgZnJvbSB0aGUgYGdncGxvdDJgIHBhY2thZ2UsIHRoZSBtYWluIHBsb3R0aW5nIHBhY2thZ2Ugb2YgdGhlIHRpZHl2ZXJzZS4KV2UgdXNlIHRoZSBmaXJzdCBmdW5jdGlvbiwgYGdncGxvdCgpYCB0byBkZWZpbmUgdGhlIGRhdGEgdGhhdCB3aWxsIGJlIHBsb3R0ZWQuCmBnZ3Bsb3QoKWAgdGFrZXMgdHdvIG1haW4gYXJndW1lbnRzOiAgCgoxLiBgZGF0YWAsIHdoaWNoIGlzIHRoZSBkYXRhIGZyYW1lIHRoYXQgY29udGFpbnMgdGhlIGRhdGEgd2Ugd2FudCB0byBwbG90LiAgCjIuIGBtYXBwaW5nYCwgd2hpY2ggaXMgYSBzcGVjaWFsIGxpc3QgbWFkZSB3aXRoIHRoZSBgYWVzKClgIGZ1bmN0aW9uIHRvIGRlc2NyaWJlIHdoaWNoIHZhbHVlcyB3aWxsIGJlIHVzZWQgZm9yIGVhY2ggKiphZXMqKnRoZXRpYyBjb21wb25lbnQgb2YgdGhlIHBsb3QsIHN1Y2ggYXMgdGhlIHggYW5kIHkgY29vcmRpbmF0ZXMgb2YgZWFjaCBwb2ludC4gCihJZiB5b3UgZmluZCBjYWxsaW5nIHRoaW5ncyBsaWtlIHRoZSB4IGFuZCB5IGNvb3JkaW5hdGVzICJhZXN0aGV0aWNzIiBjb25mdXNpbmcsIGRvbid0IHdvcnJ5LCB5b3UgYXJlIG5vdCBhbG9uZS4pICAKCmBgYHtyIGdncGxvdC1iYXNlfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLCAjIFRoaXMgZmlyc3QgYXJndW1lbnQgaXMgdGhlIGRhdGEgZnJhbWUgd2l0aCB0aGUgZGF0YSB3ZSB3YW50IHRvIHBsb3QKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLCAjIFRoaXMgaXMgdGhlIGNvbHVtbiBuYW1lIG9mIHRoZSB2YWx1ZXMgd2Ugd2FudCB0byB1c2UKICAgICMgZm9yIHRoZSB4IGNvb3JkaW5hdGVzCiAgICB5ID0gbmVnX2xvZzEwX3AKICApICMgVGhpcyBpcyB0aGUgY29sdW1uIG5hbWUgb2YgdGhlIGRhdGEgd2Ugd2FudCBmb3IgdGhlIHktYXhpcwopCmBgYAoKWW91J2xsIG5vdGljZSB0aGlzIHBsb3QgZG9lc24ndCBoYXZlIGFueXRoaW5nIG9uIGl0IGJlY2F1c2Ugd2UgaGF2ZW4ndCAKc3BlY2lmaWVkIGEgcGxvdCB0eXBlIHlldC4KVG8gZG8gdGhhdCwgd2Ugd2lsbCBhZGQgYW5vdGhlciBnZ3Bsb3QgbGF5ZXIgd2l0aCBgK2Agd2hpY2ggd2lsbCBzcGVjaWZ5IGV4YWN0bHkgd2hhdCB3ZSB3YW50IHRvIHBsb3QuCkEgdm9sY2FubyBwbG90IGlzIGEgc3BlY2lhbCBraW5kIG9mIHNjYXR0ZXIgcGxvdCwgc28gdG8gbWFrZSB0aGF0IHdlIHdpbGwgd2FudCB0byBwbG90IGluZGl2aWR1YWwgcG9pbnRzLCB3aGljaCB3ZSBjYW4gZG8gd2l0aCBgZ2VvbV9wb2ludCgpYC4KCmBgYHtyIGdncGxvdC1wb2ludHMsIGxpdmUgPSBUUlVFfQojIFRoaXMgZmlyc3QgcGFydCBpcyB0aGUgc2FtZSBhcyBiZWZvcmUKZ2dwbG90KAogIHR1bW9yX25vcm1hbF9kZiwKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wCiAgKQopICsKICAjIE5vdyB3ZSBhcmUgYWRkaW5nIG9uIGEgbGF5ZXIgdG8gc3BlY2lmeSB3aGF0IGtpbmQgb2YgcGxvdCB3ZSB3YW50CiAgZ2VvbV9wb2ludCgpCmBgYAoKSGVyZSdzIGEgYnJpZWYgc3VtbWFyeSBvZiBnZ3Bsb3QyIHN0cnVjdHVyZS4gCiFbZ2dwbG90MiBzdHJ1Y3R1cmVdKGRpYWdyYW1zL2dncGxvdF9zdHJ1Y3R1cmUucG5nKQoKIyMjIEFkanVzdCBvdXIgZ2dwbG90CgpOb3cgdGhhdCB3ZSBoYXZlIGEgYmFzZSBwbG90IHRoYXQgc2hvd3Mgb3VyIGRhdGEsIHdlIGNhbiBhZGQgbGF5ZXJzIG9uIHRvIGl0IGFuZCBhZGp1c3QgaXQuCldlIGNhbiBhZGp1c3QgdGhlIGNvbG9yIG9mIHBvaW50cyB1c2luZyB0aGUgYGNvbG9yYCBhZXN0aGV0aWMuCgpgYGB7ciBnZ3Bsb3QtY29sb3IsIGxpdmUgPSBUUlVFfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AsCiAgICBjb2xvciA9IGF2Z19leHByZXNzaW9uCiAgKSAjIFdlIGFkZGVkIHRoaXMgYXJndW1lbnQgdG8gY29sb3IgY29kZSB0aGUgcG9pbnRzIQopICsKICBnZW9tX3BvaW50KCkKYGBgCgpCZWNhdXNlIHdlIGhhdmUgc28gbWFueSBwb2ludHMgb3ZlcmxhcHBpbmcgb25lIGFub3RoZXIsIHdlIHdpbGwgd2FudCB0byBhZGp1c3QgCnRoZSB0cmFuc3BhcmVuY3ksIHdoaWNoIHdlIGNhbiBkbyB3aXRoIGFuIGBhbHBoYWAgYXJndW1lbnQuIAoKYGBge3IgZ2dwbG90LWFscGhhLCBsaXZlID0gVFJVRX0KZ2dwbG90KAogIHR1bW9yX25vcm1hbF9kZiwKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgIyBXZSBhcmUgdXNpbmcgdGhlIGBhbHBoYWAgYXJndW1lbnQgdG8gbWFrZSBvdXIgcG9pbnRzIHRyYW5zcGFyZW50CmBgYAoKTm90aWNlIHRoYXQgd2UgYWRkZWQgdGhlIGFscGhhIHdpdGhpbiB0aGUgYGdlb21fcG9pbnQoKWAgZnVuY3Rpb24sIG5vdCB0byB0aGUgYGFlcygpYC4gCldlIGRpZCB0aGlzIGJlY2F1c2Ugd2Ugd2FudCBhbGwgb2YgdGhlIHBvaW50cyB0byBoYXZlIHRoZSBzYW1lIGxldmVsIG9mIHRyYW5zcGFyZW5jeSwgYW5kIGl0IHdpbGwgbm90IHZhcnkgZGVwZW5kaW5nIG9uIGFueSB2YXJpYWJsZSBpbiB0aGUgZGF0YS4KV2UgY2FuIGFsc28gY2hhbmdlIHRoZSBiYWNrZ3JvdW5kIGFuZCBhcHBlYXJhbmNlIG9mIHRoZSBwbG90IGFzIGEgd2hvbGUgYnkgYWRkaW5nIGEgYHRoZW1lYC4KCmBgYHtyIGdncGxvdC10aGVtZX0KZ2dwbG90KAogIHR1bW9yX25vcm1hbF9kZiwKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgKwogIHRoZW1lX2J3KCkgIyBBZGQgb24gdGhpcyBzZXQgb2YgYXBwZWFyYW5jZSBwcmVzZXRzIHRvIG1ha2UgaXQgcHJldHR5CmBgYAoKV2UgYXJlIG5vdCBsaW1pdGVkIHRvIGEgc2luZ2xlIHBsb3R0aW5nIGxheWVyLiAKRm9yIGV4YW1wbGUsIGlmIHdlIHdhbnQgdG8gYWRkIGEgaG9yaXpvbnRhbCBsaW5lIHRvIGluZGljYXRlIGEgc2lnbmlmaWNhbmNlIGN1dG9mZiwgd2UgY2FuIGRvIHRoYXQgd2l0aCBgZ2VvbV9obGluZSgpYC4KRm9yIG5vdywgd2Ugd2lsbCBjaG9vc2UgdGhlIHZhbHVlIG9mIDUuNSAodGhhdCBpcyBjbG9zZSB0byBhIEJvbmZlcnJvbmkgY29ycmVjdGlvbikgYW5kIGFkZCB0aGF0IHRvIHRoZSBwbG90LgoKYGBge3IgZ2dwbG90LWhsaW5lLCBsaXZlID0gVFJVRX0KZ2dwbG90KAogIHR1bW9yX25vcm1hbF9kZiwKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgKyAKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSA1LjUsIGNvbG9yID0gImRhcmtncmVlbiIpICMgd2UgY2FuIHNwZWNpZnkgY29sb3JzIGJ5IG5hbWVzIGhlcmUKYGBgCgpXZSBjYW4gY2hhbmdlIHRoZSB4IGFuZCB5IGxhYmVscyBieSB1c2luZyBgeWxhYmAgYW5kIGB4bGFiYCBmdW5jdGlvbnMgYW5kIGFkZCBhCnRpdGxlIHVzaW5nIGBnZ3RpdGxlYC4KCmBgYHtyIGdncGxvdC1sYWJlbH0KZ2dwbG90KAogIHR1bW9yX25vcm1hbF9kZiwKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgKwogIGdlb21faGxpbmUoeWludGVyY2VwdCA9IDUuNSwgY29sb3IgPSAiZGFya2dyZWVuIikgKwogIHRoZW1lX2J3KCkgKwogIHhsYWIoImxvZzIgRm9sZCBDaGFuZ2UgVHVtb3IvTm9ybWFsIikgKyAjIEFkZCBhbiB4IGxhYmVsCiAgeWxhYigiLWxvZzEwIHAgdmFsdWUiKSArICMgQWRkIGEgeSBsYWJlbAogIGdndGl0bGUoIkFzdHJvY3l0b21hIFR1bW9yIHZzIE5vcm1hbCBDZXJlYmVsbHVtIikgIyBBZGQgbWFpbiB0aXRsZQpgYGAKClVzZSB0aGlzIGNodW5rIHRvIG1ha2UgdGhlIHNhbWUga2luZCBvZiBwbG90IGFzIHRoZSBwcmV2aW91cyBjaHVuayBidXQgaW5zdGVhZCBwbG90IHRoZSBtYWxlIGZlbWFsZSBjb250cmFzdCBkYXRhLCB0aGF0IGlzIHN0b3JlZCBpbiBgbWFsZV9mZW1hbGVfZGZgLiAKCmBgYHtyIG1mLXZvbGNhbm8sIGxpdmUgPSBUUlVFfQojIFVzZSB0aGlzIGNodW5rIHRvIG1ha2UgdGhlIHNhbWUga2luZCBvZiB2b2xjYW5vIHBsb3QsIGJ1dCB3aXRoIHRoZSBtYWxlLWZlbWFsZSBjb250cmFzdCBkYXRhLgoKYGBgCgpUdXJucyBvdXQsIHdlIGRvbid0IGhhdmUgdG8gcGxvdCBlYWNoIGNvbnRyYXN0IHNlcGFyYXRlbHksIGluc3RlYWQsIHdlIGNhbiB1c2UgdGhlIG9yaWdpbmFsIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyBhbGwgdGhyZWUgY29udHJhc3RzJyBkYXRhLCBgc3RhdHNfZGZgLCBhbmQgYWRkIGEgYGZhY2V0X3dyYXBgIHRvIG1ha2UgZWFjaCBjb250cmFzdCBpdHMgb3duIHBsb3QuIAoKYGBge3IgZ2dwbG90LWZhY2V0c30KZ2dwbG90KAogIHN0YXRzX2RmLCAjIFN3aXRjaCB0byB0aGUgYmlnZ2VyIGRhdGEgZnJhbWUgd2l0aCBhbGwgdGhyZWUgY29udHJhc3RzJyBkYXRhCiAgYWVzKAogICAgeCA9IGxvZ19mb2xkX2NoYW5nZSwKICAgIHkgPSBuZWdfbG9nMTBfcCwKICAgIGNvbG9yID0gYXZnX2V4cHJlc3Npb24KICApCikgKwogIGdlb21fcG9pbnQoYWxwaGEgPSAwLjIpICsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSA1LjUsIGNvbG9yID0gImRhcmtncmVlbiIpICsKICB0aGVtZV9idygpICsKICBmYWNldF93cmFwKH5jb250cmFzdCkgKwogIHhsYWIoImxvZzIgRm9sZCBDaGFuZ2UiKSArICMgTm93IHRoYXQgdGhpcyBpbmNsdWRlcyB0aGUgb3RoZXIgY29udHJhc3RzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgd2UnbGwgbWFrZSB0aGlzIGxhYmVsIG1vcmUgZ2VuZXJhbAogIHlsYWIoIi1sb2cxMCBwIHZhbHVlIikgKwogIGNvb3JkX2NhcnRlc2lhbih4bGltID0gYygtMjUsIDI1KSkgIyB6b29tIGluIG9uIHRoZSB4LWF4aXMKYGBgCgpXZSBjYW4gc3RvcmUgdGhlIHBsb3QgYXMgYW4gb2JqZWN0IGluIHRoZSBnbG9iYWwgZW52aXJvbm1lbnQgYnkgdXNpbmcgYDwtYCBvcGVyYXRvci4gCkhlcmUgd2Ugd2lsbCBjYWxsIGl0IGB2b2xjYW5vX3Bsb3RgLiAKCmBgYHtyIGdncGxvdC1zdG9yZS1vYmplY3R9CnZvbGNhbm9fcGxvdCA8LSBnZ3Bsb3QoCiAgc3RhdHNfZGYsICMgV2UgYXJlIGNhbGxpbmcgdGhpcyBwbG90IGB2b2xjYW5vX3Bsb3RgCiAgYWVzKAogICAgeCA9IGxvZ19mb2xkX2NoYW5nZSwKICAgIHkgPSBuZWdfbG9nMTBfcCwKICAgIGNvbG9yID0gYXZnX2V4cHJlc3Npb24KICApCikgKwogIGdlb21fcG9pbnQoYWxwaGEgPSAwLjIpICsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSA1LjUsIGNvbG9yID0gImRhcmtncmVlbiIpICsKICB0aGVtZV9idygpICsKICBmYWNldF93cmFwKH5jb250cmFzdCkgKwogIHhsYWIoImxvZzIgRm9sZCBDaGFuZ2UiKSArCiAgeWxhYigiLWxvZzEwIHAgdmFsdWUiKSArCiAgY29vcmRfY2FydGVzaWFuKHhsaW0gPSBjKC0yNSwgMjUpKQpgYGAKCldoZW4gd2UgYXJlIGhhcHB5IHdpdGggb3VyIHBsb3QsIHdlIGNhbiBzYXZlIHRoZSBwbG90IHVzaW5nIGBnZ3NhdmVgLiAKCmBgYHtyIGdnc2F2ZX0KZ2dzYXZlKAogIHBsb3QgPSB2b2xjYW5vX3Bsb3QsCiAgZmlsZW5hbWUgPSBmaWxlLnBhdGgocGxvdHNfZGlyLCAidm9sY2Fub19wbG90LnBuZyIpCikKYGBgCgojIyMgU2Vzc2lvbiBJbmZvCgpgYGB7cn0KIyBQcmludCBvdXQgdGhlIHZlcnNpb25zIGFuZCBwYWNrYWdlcyB3ZSBhcmUgdXNpbmcgaW4gdGhpcyBzZXNzaW9uCnNlc3Npb25JbmZvKCkKYGBgCg==
    
+
    LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIGdncGxvdDIiCmF1dGhvcjogIkNDREwgZm9yIEFMU0YiIApkYXRlOiAyMDIxCm91dHB1dDogICAKICBodG1sX25vdGVib29rOiAKICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OiB0cnVlCi0tLQoKCiMjIE9iamVjdGl2ZXMKClRoaXMgbm90ZWJvb2sgd2lsbCBkZW1vbnN0cmF0ZSBob3cgdG86IAoKLSBMb2FkIGFuZCB1c2UgUiBwYWNrYWdlcyAgCi0gUmVhZCBpbiBhbmQgcGVyZm9ybSBzaW1wbGUgbWFuaXB1bGF0aW9ucyBvZiBkYXRhIGZyYW1lcyAKLSBVc2UgYGdncGxvdDJgIHRvIHBsb3QgYW5kIHZpc3VhbGl6ZSBkYXRhCi0gQ3VzdG9taXplIHBsb3RzIHVzaW5nIGZlYXR1cmVzIG9mIGBnZ3Bsb3QyYAoKLS0tCgpXZSdsbCB1c2UgYSByZWFsIGdlbmUgZXhwcmVzc2lvbiBkYXRhc2V0IHRvIGdldCBjb21mb3J0YWJsZSBtYWtpbmcgdmlzdWFsaXphdGlvbnMgdXNpbmcgZ2dwbG90Mi4gCldlJ3ZlIFtwZXJmb3JtZWQgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzZXNdKC4vc2NyaXB0cy8wMC1zZXR1cC1pbnRyby10by1SLlIpIG9uIGEgcHJlLXByb2Nlc3NlZCBbYXN0cm9jeXRvbWEgbWljcm9hcnJheSBkYXRhc2V0XShodHRwczovL3d3dy5yZWZpbmUuYmlvL2V4cGVyaW1lbnRzL0dTRTQ0OTcxL2dlbmUtZXhwcmVzc2lvbi1kYXRhLWZyb20tcGlsb2N5dGljLWFzdHJvY3l0b21hLXR1bW91ci1zYW1wbGVzLWFuZC1ub3JtYWwtY2VyZWJlbGx1bS1jb250cm9scykuCldlJ2xsIHN0YXJ0IGJ5IG1ha2luZyBhIHZvbGNhbm8gcGxvdCBvZiBkaWZmZXJlbnRpYWwgZ2VuZSBleHByZXNzaW9uIHJlc3VsdHMgZnJvbSB0aGlzIGV4cGVyaW1lbnQuCldlIHBlcmZvcm1lZCB0aHJlZSBzZXRzIG9mIGNvbnRyYXN0czogIAoKMSkgYHNleGAgY2F0ZWdvcnkgY29udHJhc3Rpbmc6IGBNYWxlYCB2cyBgRmVtYWxlYCAgCjIpIGB0aXNzdWVgIGNhdGVnb3J5IGNvbnRyYXN0aW5nIDogYFBpbG9jeXRpYyBhc3Ryb2N5dG9tYSB0dW1vcmAgc2FtcGxlcyB2cyBgbm9ybWFsIGNlcmViZWxsdW1gIHNhbXBsZXMgIAozKSBBbiBpbnRlcmFjdGlvbiBvZiBib3RoIGBzZXhgIGFuZCBgdGlzc3VlYC4gCgoqKk1vcmUgZ2dwbG90MiByZXNvdXJjZXM6KiogIAoKLSBbaGFuZHkgY2hlYXRzaGVldCBmb3IgZ2dwbG90Ml0oaHR0cHM6Ly9yc3R1ZGlvLmNvbS93cC1jb250ZW50L3VwbG9hZHMvMjAxNS8wMy9nZ3Bsb3QyLWNoZWF0c2hlZXQucGRmKSAgCi0gW2dncGxvdDIgY291cnNlcyBhbmQgYm9va3MgZnJvbSBUaWR5dmVyc2VdKGh0dHBzOi8vZ2dwbG90Mi50aWR5dmVyc2Uub3JnLykgIAotIFtnZ3Bsb3QyIGRhdGEgdml6IGNoYXB0ZXIgb2YgUiBmb3IgRGF0YSBTY2llbmNlXShodHRwczovL3I0ZHMuaGFkLmNvLm56L2RhdGEtdmlzdWFsaXNhdGlvbi5odG1sKSAgCi0gW2dncGxvdDIgb25saW5lIHR1dG9yaWFsXShodHRwOi8vci1zdGF0aXN0aWNzLmNvL0NvbXBsZXRlLUdncGxvdDItVHV0b3JpYWwtUGFydDEtV2l0aC1SLUNvZGUuaHRtbCkgIAoKIyMgU2V0IFVwCgpXZSBzYXZlZCB0aGVzZSByZXN1bHRzIHRvIGEgdGFiIHNlcGFyYXRlZCB2YWx1ZXMgKFRTVikgZmlsZSBjYWxsZWQgYGdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3ZgLgpJdCdzIGJlZW4gc2F2ZWQgdG8gdGhlIGBkYXRhYCBmb2xkZXIuIApGaWxlIHBhdGhzIGFyZSByZWxhdGl2ZSB0byB3aGVyZSB0aGlzIG5vdGVib29rIGZpbGUgKC5SbWQpIGlzIHNhdmVkLgpTbyB3ZSBjYW4gcmVmZXJlbmNlIGl0IGxhdGVyLCBsZXQncyBtYWtlIGEgdmFyaWFibGUgd2l0aCBvdXIgZGF0YSBkaXJlY3RvcnkgbmFtZS4gCgpgYGB7cn0KZGF0YV9kaXIgPC0gImRhdGEiCmBgYAoKTGV0J3MgZGVjbGFyZSBvdXIgb3V0cHV0IGZvbGRlciBuYW1lIGFzIGl0cyBvd24gdmFyaWFibGUuIAoKYGBge3J9CnBsb3RzX2RpciA8LSAicGxvdHMiCmBgYAoKV2UgY2FuIGFsc28gY3JlYXRlIGEgZGlyZWN0b3J5IGlmIGl0IGRvZXNuJ3QgYWxyZWFkeSBleGlzdC4gCgpgYGB7ciBjcmVhdGVpZn0KIyBUaGUgaWYgc3RhdGVtZW50IGhlcmUgdGVzdHMgd2hldGhlciB0aGUgcGxvdCBkaXJlY3RvcnkgZXhpc3RzIGFuZAojIG9ubHkgZXhlY3V0ZXMgdGhlICBleHByZXNzaW9ucyBiZXR3ZWVuIHRoZSBicmFjZXMgaWYgaXQgZG9lcyBub3QuCmlmICghZGlyLmV4aXN0cyhwbG90c19kaXIpKSB7CiAgZGlyLmNyZWF0ZShwbG90c19kaXIpCn0KYGBgCgpJbiB0aGlzIG5vdGVib29rIHdlIHdpbGwgYmUgdXNpbmcgZnVuY3Rpb25zIGZyb20gdGhlIFRpZHl2ZXJzZSBzZXQgb2YgcGFja2FnZXMsIHNvIHdlIG5lZWQgdG8gbG9hZCBpbiB0aG9zZSBmdW5jdGlvbnMgdXNpbmcgYGxpYnJhcnkoKWAuCldlIGNvdWxkIGxvYWQgdGhlIGluZGl2aWR1YWwgcGFja2FnZXMgd2UgbmVlZCBvbmUgYXQgYSB0aW1lLCBidXQgaXQgaXMgY29udmVuaWVudCBmb3Igbm93IHRvIGxvYWQgdGhlbSBhbGwgd2l0aCB0aGUgYHRpZHl2ZXJzZWAgcGFja2FnZSwgd2hpY2ggZ3JvdXBzIG1hbnkgb2YgdGhlbSB0b2dldGhlci4KS2VlcCBhIGxvb2sgb3V0IGZvciB3aGVyZSB3ZSB0ZWxsIHlvdSB3aGljaCBpbmRpdmlkdWFsIHBhY2thZ2UgZGlmZmVyZW50IGZ1bmN0aW9ucyBjb21lIGZyb20uCgpgYGB7ciB0aWR5dmVyc2V9CmxpYnJhcnkodGlkeXZlcnNlKQpgYGAKCiMjIFJlYWQgaW4gdGhlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIHJlc3VsdHMgZmlsZQoKSGVyZSB3ZSBhcmUgdXNpbmcgYSBgdGlkeXZlcnNlYCBmdW5jdGlvbiBgcmVhZF90c3YoKWAgZnJvbSB0aGUgYHJlYWRyYCBwYWNrYWdlLgpMaWtlIHdlIGRpZCBpbiB0aGUgcHJldmlvdXMgbm90ZWJvb2ssIHdlIHdpbGwgc3RvcmUgdGhlIHJlc3VsdGluZyBkYXRhIGZyYW1lIGFzIGBzdGF0c19kZmAuCgpgYGB7ciByZWFkLXN0YXRzfQojIHJlYWQgaW4gdGhlIGZpbGUgYGdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3ZgIGZyb20gdGhlIGRhdGEgZGlyZWN0b3J5CnN0YXRzX2RmIDwtIHJlYWRfdHN2KGZpbGUucGF0aCgKICBkYXRhX2RpciwKICAiZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdiIKKSkKYGBgCgpXZSBjYW4gdGFrZSBhIGxvb2sgYXQgYSBjb2x1bW4gaW5kaXZpZHVhbGx5IGJ5IHVzaW5nIGEgYCRgLiAKTm90ZSB3ZSBhcmUgdXNpbmcgYGhlYWQoKWAgc28gdGhlIHdob2xlIHRoaW5nIGRvZXNuJ3QgcHJpbnQgb3V0LiAKCmBgYHtyIGNvbHVtbn0KaGVhZChzdGF0c19kZiRjb250cmFzdCkKYGBgCgpJZiB3ZSB3YW50IHRvIHNlZSBhIHNwZWNpZmljIHNldCBvZiB2YWx1ZXMsIHdlIGNhbiB1c2UgYnJhY2tldHMgd2l0aCB0aGUgaW5kaWNlcyBvZiB0aGUgdmFsdWVzIHdlJ2QgbGlrZSByZXR1cm5lZC4KCmBgYHtyfQpzdGF0c19kZiRhdmdfZXhwcmVzc2lvbls2OjEwXQpgYGAKCkxldCdzIGxvb2sgYXQgc29tZSBiYXNpYyBzdGF0aXN0aWNzIGZyb20gdGhlIGRhdGEgc2V0IHVzaW5nIGBzdW1tYXJ5KClgCgpgYGB7ciBzdGF0cy1zdW1tYXJ5LCBsaXZlID0gVFJVRX0KIyBzdW1tYXJ5IG9mIHN0YXRzX2RmCnN1bW1hcnkoc3RhdHNfZGYpCmBgYAoKVGhlIHN0YXRpc3RpY3MgZm9yIGBjb250cmFzdGAgYXJlIG5vdCB2ZXJ5IGluZm9ybWF0aXZlLCBzbyBsZXQncyBkbyB0aGF0IGFnYWluIHdpdGgganVzdCB0aGUgYGNvbnRyYXN0YCBjb2x1bW4gYWZ0ZXIgY29udmVydGluZyBpdCB0byBhIGBmYWN0b3JgCmBgYHtyIGZhY3Rvci1zdW1tYXJ5LCBsaXZlID0gVFJVRX0KIyBzdW1tYXJ5IG9mIGBzdGF0c19kZiRjb250cmFzdGAgYXMgYSBmYWN0b3IKc3VtbWFyeShhcy5mYWN0b3Ioc3RhdHNfZGYkY29udHJhc3QpKQpgYGAKCiMjIFNldCB1cCB0aGUgZGF0YXNldAoKQmVmb3JlIHdlIG1ha2Ugb3VyIHBsb3QsIHdlIHdhbnQgdG8gY2FsY3VsYXRlIGEgc2V0IG9mIG5ldyB2YWx1ZXMgZm9yIGVhY2ggcm93OyB0cmFuc2Zvcm1hdGlvbnMgb2YgdGhlIHJhdyBzdGF0aXN0aWNzIGluIG91ciB0YWJsZS4KVG8gZG8gdGhpcyB3ZSB3aWxsIHVzZSBhIGZ1bmN0aW9uIGZyb20gdGhlIGBkcGx5cmAgcGFja2FnZSBjYWxsZWQgYG11dGF0ZSgpYCB0byBtYWtlIGEgbmV3IGNvbHVtbiBvZiAtbG9nMTAgcCB2YWx1ZXMuCgpgYGB7ciBtdXRhdGV9CiMgYWRkIGEgYG5lZ19sb2cxMF9wYCBjb2x1bW4gdG8gdGhlIGRhdGEgZnJhbWUKc3RhdHNfZGYgPC0gbXV0YXRlKHN0YXRzX2RmLCAjIGRhdGEgZnJhbWUgd2UnZCBsaWtlIHRvIGFkZCBhIHZhcmlhYmxlIHRvCiAgbmVnX2xvZzEwX3AgPSAtbG9nMTAocF92YWx1ZSkgIyBjb2x1bW4gbmFtZSBhbmQgdmFsdWVzCikKYGBgCgpMZXQncyBmaWx0ZXIgdG8gb25seSBgbWFsZV9mZW1hbGVgIGNvbnRyYXN0IGRhdGEuIApGaXJzdCBsZXQncyB0cnkgb3V0IGEgbG9naWNhbCBleHByZXNzaW9uOiAKCmBgYHtyIGV2YWwgPSBGQUxTRX0Kc3RhdHNfZGYkY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIgpgYGAKCk5vdyB3ZSBjYW4gdHJ5IG91dCB0aGUgYGZpbHRlcigpYCBmdW5jdGlvbi5Ob3RpY2UgdGhhdCB3ZSBhcmUgbm90IGFzc2lnbmluZyB0aGUgcmVzdWx0cyB0byBhIHZhcmlhYmxlLCBzbyB0aGlzIGZpbHRlcmVkIGRhdGFzZXQgd2lsbCBub3QgYmUgc2F2ZWQgdG8gdGhlIGVudmlyb25tZW50LgoKYGBge3IgZmlsdGVyLCBsaXZlID0gVFJVRX0KIyBmaWx0ZXIgc3RhdHNfZGYgdG8gIm1hbGVfZmVtYWxlIiBvbmx5CmZpbHRlcihzdGF0c19kZiwgY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIikKYGBgCgpOb3cgd2UgY2FuIGFzc2lnbiB0aGUgcmVzdWx0cyB0byBhIG5ldyBkYXRhIGZyYW1lOiBgbWFsZV9mZW1hbGVfZGZgLiAKCmBgYHtyIGZpbHRlci1zYXZlLCBsaXZlID0gVFJVRX0KIyBmaWx0ZXIgYW5kIHNhdmUgdG8gbWFsZV9mZW1hbGVfZGYKbWFsZV9mZW1hbGVfZGYgPC0gZmlsdGVyKHN0YXRzX2RmLCBjb250cmFzdCA9PSAibWFsZV9mZW1hbGUiKQpgYGAKCiMjIFBsb3R0aW5nIHRoaXMgZGF0YQoKTGV0J3MgbWFrZSBhIHZvbGNhbm8gcGxvdCB3aXRoIHRoaXMgZGF0YS4gCkZpcnN0IGxldCdzIHRha2UgYSBsb29rIGF0IG9ubHkgdGhlIHR1bW9yIHZzLiBub3JtYWwgY29tcGFyaXNvbi4gCkxldCdzIHNhdmUgdGhpcyBhcyBhIHNlcGFyYXRlIGRhdGEgZnJhbWUgYnkgYXNzaWduaW5nIGl0IGEgbmV3IG5hbWUuIAoKYGBge3IgZmlsdGVyLXR1bW9yfQp0dW1vcl9ub3JtYWxfZGYgPC0gZmlsdGVyKHN0YXRzX2RmLCBjb250cmFzdCA9PSAiYXN0cm9jeXRvbWFfbm9ybWFsIikKYGBgCgpUbyBtYWtlIHRoaXMgcGxvdCB3ZSB3aWxsIGJlIHVzaW5nIGZ1bmN0aW9ucyBmcm9tIHRoZSBgZ2dwbG90MmAgcGFja2FnZSwgdGhlIG1haW4gcGxvdHRpbmcgcGFja2FnZSBvZiB0aGUgdGlkeXZlcnNlLgpXZSB1c2UgdGhlIGZpcnN0IGZ1bmN0aW9uLCBgZ2dwbG90KClgIHRvIGRlZmluZSB0aGUgZGF0YSB0aGF0IHdpbGwgYmUgcGxvdHRlZC4KYGdncGxvdCgpYCB0YWtlcyB0d28gbWFpbiBhcmd1bWVudHM6ICAKCjEuIGBkYXRhYCwgd2hpY2ggaXMgdGhlIGRhdGEgZnJhbWUgdGhhdCBjb250YWlucyB0aGUgZGF0YSB3ZSB3YW50IHRvIHBsb3QuICAKMi4gYG1hcHBpbmdgLCB3aGljaCBpcyBhIHNwZWNpYWwgbGlzdCBtYWRlIHdpdGggdGhlIGBhZXMoKWAgZnVuY3Rpb24gdG8gZGVzY3JpYmUgd2hpY2ggdmFsdWVzIHdpbGwgYmUgdXNlZCBmb3IgZWFjaCAqKmFlcyoqdGhldGljIGNvbXBvbmVudCBvZiB0aGUgcGxvdCwgc3VjaCBhcyB0aGUgeCBhbmQgeSBjb29yZGluYXRlcyBvZiBlYWNoIHBvaW50LiAKKElmIHlvdSBmaW5kIGNhbGxpbmcgdGhpbmdzIGxpa2UgdGhlIHggYW5kIHkgY29vcmRpbmF0ZXMgImFlc3RoZXRpY3MiIGNvbmZ1c2luZywgZG9uJ3Qgd29ycnksIHlvdSBhcmUgbm90IGFsb25lLikgIAoKYGBge3IgZ2dwbG90LWJhc2V9CmdncGxvdCgKICB0dW1vcl9ub3JtYWxfZGYsICMgVGhpcyBmaXJzdCBhcmd1bWVudCBpcyB0aGUgZGF0YSBmcmFtZSB3aXRoIHRoZSBkYXRhIHdlIHdhbnQgdG8gcGxvdAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsICMgVGhpcyBpcyB0aGUgY29sdW1uIG5hbWUgb2YgdGhlIHZhbHVlcyB3ZSB3YW50IHRvIHVzZQogICAgIyBmb3IgdGhlIHggY29vcmRpbmF0ZXMKICAgIHkgPSBuZWdfbG9nMTBfcAogICkgIyBUaGlzIGlzIHRoZSBjb2x1bW4gbmFtZSBvZiB0aGUgZGF0YSB3ZSB3YW50IGZvciB0aGUgeS1heGlzCikKYGBgCgpZb3UnbGwgbm90aWNlIHRoaXMgcGxvdCBkb2Vzbid0IGhhdmUgYW55dGhpbmcgb24gaXQgYmVjYXVzZSB3ZSBoYXZlbid0IApzcGVjaWZpZWQgYSBwbG90IHR5cGUgeWV0LgpUbyBkbyB0aGF0LCB3ZSB3aWxsIGFkZCBhbm90aGVyIGdncGxvdCBsYXllciB3aXRoIGArYCB3aGljaCB3aWxsIHNwZWNpZnkgZXhhY3RseSB3aGF0IHdlIHdhbnQgdG8gcGxvdC4KQSB2b2xjYW5vIHBsb3QgaXMgYSBzcGVjaWFsIGtpbmQgb2Ygc2NhdHRlciBwbG90LCBzbyB0byBtYWtlIHRoYXQgd2Ugd2lsbCB3YW50IHRvIHBsb3QgaW5kaXZpZHVhbCBwb2ludHMsIHdoaWNoIHdlIGNhbiBkbyB3aXRoIGBnZW9tX3BvaW50KClgLgoKYGBge3IgZ2dwbG90LXBvaW50cywgbGl2ZSA9IFRSVUV9CiMgVGhpcyBmaXJzdCBwYXJ0IGlzIHRoZSBzYW1lIGFzIGJlZm9yZQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AKICApCikgKwogICMgTm93IHdlIGFyZSBhZGRpbmcgb24gYSBsYXllciB0byBzcGVjaWZ5IHdoYXQga2luZCBvZiBwbG90IHdlIHdhbnQKICBnZW9tX3BvaW50KCkKYGBgCgpIZXJlJ3MgYSBicmllZiBzdW1tYXJ5IG9mIGdncGxvdDIgc3RydWN0dXJlLiAKIVtnZ3Bsb3QyIHN0cnVjdHVyZV0oZGlhZ3JhbXMvZ2dwbG90X3N0cnVjdHVyZS5wbmcpCgojIyMgQWRqdXN0IG91ciBnZ3Bsb3QKCk5vdyB0aGF0IHdlIGhhdmUgYSBiYXNlIHBsb3QgdGhhdCBzaG93cyBvdXIgZGF0YSwgd2UgY2FuIGFkZCBsYXllcnMgb24gdG8gaXQgYW5kIGFkanVzdCBpdC4KV2UgY2FuIGFkanVzdCB0aGUgY29sb3Igb2YgcG9pbnRzIHVzaW5nIHRoZSBgY29sb3JgIGFlc3RoZXRpYy4KCmBgYHtyIGdncGxvdC1jb2xvciwgbGl2ZSA9IFRSVUV9CmdncGxvdCgKICB0dW1vcl9ub3JtYWxfZGYsCiAgYWVzKAogICAgeCA9IGxvZ19mb2xkX2NoYW5nZSwKICAgIHkgPSBuZWdfbG9nMTBfcCwKICAgIGNvbG9yID0gYXZnX2V4cHJlc3Npb24KICApICMgV2UgYWRkZWQgdGhpcyBhcmd1bWVudCB0byBjb2xvciBjb2RlIHRoZSBwb2ludHMhCikgKwogIGdlb21fcG9pbnQoKQpgYGAKCkJlY2F1c2Ugd2UgaGF2ZSBzbyBtYW55IHBvaW50cyBvdmVybGFwcGluZyBvbmUgYW5vdGhlciwgd2Ugd2lsbCB3YW50IHRvIGFkanVzdCAKdGhlIHRyYW5zcGFyZW5jeSwgd2hpY2ggd2UgY2FuIGRvIHdpdGggYW4gYGFscGhhYCBhcmd1bWVudC4gCgpgYGB7ciBnZ3Bsb3QtYWxwaGEsIGxpdmUgPSBUUlVFfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AsCiAgICBjb2xvciA9IGF2Z19leHByZXNzaW9uCiAgKQopICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSAjIFdlIGFyZSB1c2luZyB0aGUgYGFscGhhYCBhcmd1bWVudCB0byBtYWtlIG91ciBwb2ludHMgdHJhbnNwYXJlbnQKYGBgCgpOb3RpY2UgdGhhdCB3ZSBhZGRlZCB0aGUgYWxwaGEgd2l0aGluIHRoZSBgZ2VvbV9wb2ludCgpYCBmdW5jdGlvbiwgbm90IHRvIHRoZSBgYWVzKClgLiAKV2UgZGlkIHRoaXMgYmVjYXVzZSB3ZSB3YW50IGFsbCBvZiB0aGUgcG9pbnRzIHRvIGhhdmUgdGhlIHNhbWUgbGV2ZWwgb2YgdHJhbnNwYXJlbmN5LCBhbmQgaXQgd2lsbCBub3QgdmFyeSBkZXBlbmRpbmcgb24gYW55IHZhcmlhYmxlIGluIHRoZSBkYXRhLgpXZSBjYW4gYWxzbyBjaGFuZ2UgdGhlIGJhY2tncm91bmQgYW5kIGFwcGVhcmFuY2Ugb2YgdGhlIHBsb3QgYXMgYSB3aG9sZSBieSBhZGRpbmcgYSBgdGhlbWVgLgoKYGBge3IgZ2dwbG90LXRoZW1lfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AsCiAgICBjb2xvciA9IGF2Z19leHByZXNzaW9uCiAgKQopICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSArCiAgdGhlbWVfYncoKSAjIEFkZCBvbiB0aGlzIHNldCBvZiBhcHBlYXJhbmNlIHByZXNldHMgdG8gbWFrZSBpdCBwcmV0dHkKYGBgCgpXZSBhcmUgbm90IGxpbWl0ZWQgdG8gYSBzaW5nbGUgcGxvdHRpbmcgbGF5ZXIuIApGb3IgZXhhbXBsZSwgaWYgd2Ugd2FudCB0byBhZGQgYSBob3Jpem9udGFsIGxpbmUgdG8gaW5kaWNhdGUgYSBzaWduaWZpY2FuY2UgY3V0b2ZmLCB3ZSBjYW4gZG8gdGhhdCB3aXRoIGBnZW9tX2hsaW5lKClgLgpGb3Igbm93LCB3ZSB3aWxsIGNob29zZSB0aGUgdmFsdWUgb2YgNS41ICh0aGF0IGlzIGNsb3NlIHRvIGEgQm9uZmVycm9uaSBjb3JyZWN0aW9uKSBhbmQgYWRkIHRoYXQgdG8gdGhlIHBsb3QuCgpgYGB7ciBnZ3Bsb3QtaGxpbmUsIGxpdmUgPSBUUlVFfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AsCiAgICBjb2xvciA9IGF2Z19leHByZXNzaW9uCiAgKQopICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSArIAogIGdlb21faGxpbmUoeWludGVyY2VwdCA9IDUuNSwgY29sb3IgPSAiZGFya2dyZWVuIikgIyB3ZSBjYW4gc3BlY2lmeSBjb2xvcnMgYnkgbmFtZXMgaGVyZQpgYGAKCldlIGNhbiBjaGFuZ2UgdGhlIHggYW5kIHkgbGFiZWxzIGJ5IHVzaW5nIGB5bGFiYCBhbmQgYHhsYWJgIGZ1bmN0aW9ucyBhbmQgYWRkIGEKdGl0bGUgdXNpbmcgYGdndGl0bGVgLgoKYGBge3IgZ2dwbG90LWxhYmVsfQpnZ3Bsb3QoCiAgdHVtb3Jfbm9ybWFsX2RmLAogIGFlcygKICAgIHggPSBsb2dfZm9sZF9jaGFuZ2UsCiAgICB5ID0gbmVnX2xvZzEwX3AsCiAgICBjb2xvciA9IGF2Z19leHByZXNzaW9uCiAgKQopICsKICBnZW9tX3BvaW50KGFscGhhID0gMC4yKSArCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gNS41LCBjb2xvciA9ICJkYXJrZ3JlZW4iKSArCiAgdGhlbWVfYncoKSArCiAgeGxhYigibG9nMiBGb2xkIENoYW5nZSBUdW1vci9Ob3JtYWwiKSArICMgQWRkIGFuIHggbGFiZWwKICB5bGFiKCItbG9nMTAgcCB2YWx1ZSIpICsgIyBBZGQgYSB5IGxhYmVsCiAgZ2d0aXRsZSgiQXN0cm9jeXRvbWEgVHVtb3IgdnMgTm9ybWFsIENlcmViZWxsdW0iKSAjIEFkZCBtYWluIHRpdGxlCmBgYAoKVXNlIHRoaXMgY2h1bmsgdG8gbWFrZSB0aGUgc2FtZSBraW5kIG9mIHBsb3QgYXMgdGhlIHByZXZpb3VzIGNodW5rIGJ1dCBpbnN0ZWFkIHBsb3QgdGhlIG1hbGUgZmVtYWxlIGNvbnRyYXN0IGRhdGEsIHRoYXQgaXMgc3RvcmVkIGluIGBtYWxlX2ZlbWFsZV9kZmAuIAoKYGBge3IgbWYtdm9sY2FubywgbGl2ZSA9IFRSVUV9CiMgVXNlIHRoaXMgY2h1bmsgdG8gbWFrZSB0aGUgc2FtZSBraW5kIG9mIHZvbGNhbm8gcGxvdCwgYnV0IHdpdGggdGhlIG1hbGUtZmVtYWxlIGNvbnRyYXN0IGRhdGEuCgpgYGAKClR1cm5zIG91dCwgd2UgZG9uJ3QgaGF2ZSB0byBwbG90IGVhY2ggY29udHJhc3Qgc2VwYXJhdGVseSwgaW5zdGVhZCwgd2UgY2FuIHVzZSB0aGUgb3JpZ2luYWwgZGF0YSBmcmFtZSB0aGF0IGNvbnRhaW5zIGFsbCB0aHJlZSBjb250cmFzdHMnIGRhdGEsIGBzdGF0c19kZmAsIGFuZCBhZGQgYSBgZmFjZXRfd3JhcGAgdG8gbWFrZSBlYWNoIGNvbnRyYXN0IGl0cyBvd24gcGxvdC4gCgpgYGB7ciBnZ3Bsb3QtZmFjZXRzfQpnZ3Bsb3QoCiAgc3RhdHNfZGYsICMgU3dpdGNoIHRvIHRoZSBiaWdnZXIgZGF0YSBmcmFtZSB3aXRoIGFsbCB0aHJlZSBjb250cmFzdHMnIGRhdGEKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgKwogIGdlb21faGxpbmUoeWludGVyY2VwdCA9IDUuNSwgY29sb3IgPSAiZGFya2dyZWVuIikgKwogIHRoZW1lX2J3KCkgKwogIGZhY2V0X3dyYXAofmNvbnRyYXN0KSArCiAgeGxhYigibG9nMiBGb2xkIENoYW5nZSIpICsgIyBOb3cgdGhhdCB0aGlzIGluY2x1ZGVzIHRoZSBvdGhlciBjb250cmFzdHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIyB3ZSdsbCBtYWtlIHRoaXMgbGFiZWwgbW9yZSBnZW5lcmFsCiAgeWxhYigiLWxvZzEwIHAgdmFsdWUiKSArCiAgY29vcmRfY2FydGVzaWFuKHhsaW0gPSBjKC0yNSwgMjUpKSAjIHpvb20gaW4gb24gdGhlIHgtYXhpcwpgYGAKCldlIGNhbiBzdG9yZSB0aGUgcGxvdCBhcyBhbiBvYmplY3QgaW4gdGhlIGdsb2JhbCBlbnZpcm9ubWVudCBieSB1c2luZyBgPC1gIG9wZXJhdG9yLiAKSGVyZSB3ZSB3aWxsIGNhbGwgaXQgYHZvbGNhbm9fcGxvdGAuIAoKYGBge3IgZ2dwbG90LXN0b3JlLW9iamVjdH0Kdm9sY2Fub19wbG90IDwtIGdncGxvdCgKICBzdGF0c19kZiwgIyBXZSBhcmUgY2FsbGluZyB0aGlzIHBsb3QgYHZvbGNhbm9fcGxvdGAKICBhZXMoCiAgICB4ID0gbG9nX2ZvbGRfY2hhbmdlLAogICAgeSA9IG5lZ19sb2cxMF9wLAogICAgY29sb3IgPSBhdmdfZXhwcmVzc2lvbgogICkKKSArCiAgZ2VvbV9wb2ludChhbHBoYSA9IDAuMikgKwogIGdlb21faGxpbmUoeWludGVyY2VwdCA9IDUuNSwgY29sb3IgPSAiZGFya2dyZWVuIikgKwogIHRoZW1lX2J3KCkgKwogIGZhY2V0X3dyYXAofmNvbnRyYXN0KSArCiAgeGxhYigibG9nMiBGb2xkIENoYW5nZSIpICsKICB5bGFiKCItbG9nMTAgcCB2YWx1ZSIpICsKICBjb29yZF9jYXJ0ZXNpYW4oeGxpbSA9IGMoLTI1LCAyNSkpCmBgYAoKV2hlbiB3ZSBhcmUgaGFwcHkgd2l0aCBvdXIgcGxvdCwgd2UgY2FuIHNhdmUgdGhlIHBsb3QgdXNpbmcgYGdnc2F2ZWAuIAoKYGBge3IgZ2dzYXZlfQpnZ3NhdmUoCiAgcGxvdCA9IHZvbGNhbm9fcGxvdCwKICBmaWxlbmFtZSA9IGZpbGUucGF0aChwbG90c19kaXIsICJ2b2xjYW5vX3Bsb3QucG5nIikKKQpgYGAKCiMjIyBTZXNzaW9uIEluZm8KCmBgYHtyfQojIFByaW50IG91dCB0aGUgdmVyc2lvbnMgYW5kIHBhY2thZ2VzIHdlIGFyZSB1c2luZyBpbiB0aGlzIHNlc3Npb24Kc2Vzc2lvbkluZm8oKQpgYGAK
    
 
 
 
diff --git a/intro-to-R-tidyverse/03-intro_to_tidyverse-live.Rmd b/intro-to-R-tidyverse/03-intro_to_tidyverse-live.Rmd
index 440e4856..91149b7a 100644
--- a/intro-to-R-tidyverse/03-intro_to_tidyverse-live.Rmd
+++ b/intro-to-R-tidyverse/03-intro_to_tidyverse-live.Rmd
@@ -1,5 +1,7 @@
 ---
 title: "Introduction to tidyverse"
+author: "CCDL for ALSF" 
+date: 2021
 output:   
   html_notebook: 
     toc: true
@@ -8,15 +10,24 @@ editor_options:
   chunk_output_type: inline
 ---
 
-**CCDL 2020**
 
-## Objective for this notebook analysis: 
+## Objectives
+
+This notebook will demonstrate how to:
+
+- Use functions from the tidyverse to read and write data frames
+- Implement and use tidyverse functions to wrangle data (i.e. filter, mutate, arrange, join)
+- Use `magrittr` pipes (`%>%`) to combine multiple operations 
+- Use the `apply()` function to apply functions across rows or columns of a matrix
+
+---
 
 We'll use the same gene expression dataset we used in the [previous notebook](./02-intro_to_ggplot2.Rmd).
 It is a pre-processed [astrocytoma microarray dataset](https://www.refine.bio/experiments/GSE44971/gene-expression-data-from-pilocytic-astrocytoma-tumour-samples-and-normal-cerebellum-controls) 
 that we performed a set of [differential expression analyses on](./scripts/00-setup-intro-to-R.R).  
 
-**More tidyverse resources:**  
+**More tidyverse resources:** 
+
 - [R for Data Science](https://r4ds.had.co.nz/)  
 - [tidyverse documentation](https://dplyr.tidyverse.org/)  
 - [Cheatsheet of tidyverse data transformation](https://github.com/rstudio/cheatsheets/raw/master/data-transformation.pdf)  
@@ -187,10 +198,10 @@ Use this chunk to explore what `gene_df` looks like.
 
 What information is contained in `gene_df`?
 
-## dplyr pipes
+## `magrittr` pipes
 
 One nifty feature of the tidyverse is pipes: `%>%`
-These handy things allows you to funnel the result of one expression to the next,
+These handy things, which come from the `magrittr` package, allow you to funnel the result of one expression to the next,
 making your code a little more streamlined.
 
 For example, the output from this:  
diff --git a/intro-to-R-tidyverse/03-intro_to_tidyverse.nb.html b/intro-to-R-tidyverse/03-intro_to_tidyverse.nb.html
index 938626aa..f87fbe44 100644
--- a/intro-to-R-tidyverse/03-intro_to_tidyverse.nb.html
+++ b/intro-to-R-tidyverse/03-intro_to_tidyverse.nb.html
@@ -321,11 +321,12 @@ 
    2021
    
 
    Objectives
    
 
    This notebook will demonstrate how to:
    
 
      
-
    • Use the tidyverse package to read and write data frames
      • 
-
    • Use dplyr pipes to manipulate data frames
      • 
-
    • Implement and use functions in the tidyverse package to wrangle data (i.e. filter, mutate, arrange, join)
      • 
-
    • Use the apply function to apply functions across entire data frames
      • 
+
    • Use functions from the tidyverse to read and write data frames
      • 
+
    • Implement and use tidyverse functions to wrangle data (i.e. filter, mutate, arrange, join)
      • 
+
    • Use magrittr pipes (%>%) to combine multiple operations
      • 
+
    • Use the apply() function to apply functions across rows or columns of a matrix
      • 
 
    
+
    
 
    We’ll use the same gene expression dataset we used in the previous notebook. It is a pre-processed astrocytoma microarray dataset that we performed a set of differential expression analyses on.
    
 
    More tidyverse resources:
    
 
      
@@ -344,306 +345,185 @@ 
      Set Up
      
 
      Our RStudio Server already has the tidyverse group of packages installed for you. But if you needed to install it or other packages available on CRAN, you do it using the install.packages() function like this: install.packages("tidyverse").
      
 
 
-
-
      ```r
-library(tidyverse)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-### Referencing a library's function with `::`
-
-Note that if we had not imported the tidyverse set of packages using `library()` 
-like above, and we wanted to use a tidyverse function like `read_tsv()`, we 
-would need to tell R what package to find this function in.
-To do this, we would use `::` to tell R to load in this function from the 
-`readr` package by using `readr::read_tsv()`.
-You will see this `::` method of referencing libraries within packages 
-throughout the course. 
-We like to use it in part to remove any ambiguity in which version of a 
-function we are using; it is not too uncommon for different packages to use the 
-same name for very different functions!
-
-## Managing directories
-
-Before we can import the data we need, we should double check where R is 
-looking for files, aka the current **working directory**. 
-We can do this by using the `getwd()` function, which will tell us what folder
-we are in. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBMZXQncyBjaGVjayB3aGF0IGRpcmVjdG9yeSB3ZSBhcmUgaW46XG5nZXR3ZCgpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Let's check what directory we are in:
+
+
      library(tidyverse)
      
+
+
+
      ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
      
+
+
+
      ✔ ggplot2 3.3.3     ✔ purrr   0.3.4
+✔ tibble  3.0.5     ✔ dplyr   1.0.3
+✔ tidyr   1.1.2     ✔ stringr 1.4.0
+✔ readr   1.4.0     ✔ forcats 0.5.0
      
+
+
+
      ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
+✖ dplyr::filter() masks stats::filter()
+✖ dplyr::lag()    masks stats::lag()
      
+
+
+
+
      
+
      Referencing a library’s function with ::
      
+
      Note that if we had not imported the tidyverse set of packages using library() like above, and we wanted to use a tidyverse function like read_tsv(), we would need to tell R what package to find this function in. To do this, we would use :: to tell R to load in this function from the readr package by using readr::read_tsv(). You will see this :: method of referencing libraries within packages throughout the course. We like to use it in part to remove any ambiguity in which version of a function we are using; it is not too uncommon for different packages to use the same name for very different functions!
      
+
      
+
+
      
+
      Managing directories
      
+
      Before we can import the data we need, we should double check where R is looking for files, aka the current working directory. We can do this by using the getwd() function, which will tell us what folder we are in.
      
+
+
+
+
      # Let's check what directory we are in:
 getwd()
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFxcL19fdy90cmFpbmluZy1tb2R1bGVzL3RyYWluaW5nLW1vZHVsZXMvaW50cm8tdG8tUi10aWR5dmVyc2VcXFxuL19fdy90cmFpbmluZy1tb2R1bGVzL3RyYWluaW5nLW1vZHVsZXMvaW50cm8tdG8tUi10aWR5dmVyc2VcbiJ9 -->
-
      
-
      [1] /__w/training-modules/training-modules/intro-to-R-tidyverse
      
-/__w/training-modules/training-modules/intro-to-R-tidyverse
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-For Rmd files, the working directory is wherever the file is located, but 
-commands executed in the console may have a different working directory.
-
-We will want to make a directory for our output and we will call this directory: 
-`results`. 
-But before we create the directory, we should check if it already exists. 
-We will show two ways that we can do this. 
-
-First, we can use the `dir()` function to have R list the files in our working 
-directory. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBMZXQncyBjaGVjayB3aGF0IGZpbGVzIGFyZSBoZXJlXG5kaXIoKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Let's check what files are here
+
+
+
      [1] "/__w/training-modules/training-modules/intro-to-R-tidyverse"
      
+
+
+
      /__w/training-modules/training-modules/intro-to-R-tidyverse
      
+
+
+
+
      For Rmd files, the working directory is wherever the file is located, but commands executed in the console may have a different working directory.
      
+
      We will want to make a directory for our output and we will call this directory: results. But before we create the directory, we should check if it already exists. We will show two ways that we can do this.
      
+
      First, we can use the dir() function to have R list the files in our working directory.
      
+
+
+
+
      # Let's check what files are here
 dir()
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiIFsxXSBcXDAwYS1yc3R1ZGlvX2d1aWRlLm1kXFwgICAgICAgICAgICAgICAgICAgXG4gWzJdIFxcMDBiLWRlYnVnZ2luZ19yZXNvdXJjZXMubWRcXCAgICAgICAgICAgICBcbiBbM10gXFwwMGMtZ29vZC1zY2llbnRpZmljLWNvZGluZy1wcmFjdGljZXMubWRcXFxuIFs0XSBcXDAxLWludHJvX3RvX2Jhc2VfUi1saXZlLlJtZFxcICAgICAgICAgICAgXG4gWzVdIFxcMDEtaW50cm9fdG9fYmFzZV9SLm5iLmh0bWxcXCAgICAgICAgICAgICBcbiBbNl0gXFwwMS1pbnRyb190b19iYXNlX1IuUm1kXFwgICAgICAgICAgICAgICAgIFxuIFs3XSBcXDAyLWludHJvX3RvX2dncGxvdDItbGl2ZS5SbWRcXCAgICAgICAgICAgXG4gWzhdIFxcMDItaW50cm9fdG9fZ2dwbG90Mi5uYi5odG1sXFwgICAgICAgICAgICBcbiBbOV0gXFwwMi1pbnRyb190b19nZ3Bsb3QyLlJtZFxcICAgICAgICAgICAgICAgIFxuWzEwXSBcXDAzLWludHJvX3RvX3RpZHl2ZXJzZS1saXZlLlJtZFxcICAgICAgICAgXG5bMTFdIFxcMDMtaW50cm9fdG9fdGlkeXZlcnNlLm5iLmh0bWxcXCAgICAgICAgICBcblsxMl0gXFwwMy1pbnRyb190b190aWR5dmVyc2UuUm1kXFwgICAgICAgICAgICAgIFxuWzEzXSBcXGRhdGFcXCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgXG5bMTRdIFxcZGlhZ3JhbXNcXCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBcblsxNV0gXFxleGVyY2lzZV8wMS1pbnRyb190b19iYXNlX1IuUm1kXFwgICAgICAgIFxuWzE2XSBcXGV4ZXJjaXNlXzAyLWludHJvX3RvX1IuUm1kXFwgICAgICAgICAgICAgXG5bMTddIFxcZXhlcmNpc2VfMDNhLWludHJvX3RvX3RpZHl2ZXJzZS5SbWRcXCAgICBcblsxOF0gXFxleGVyY2lzZV8wM2ItaW50cm9fdG9fdGlkeXZlcnNlLlJtZFxcICAgIFxuWzE5XSBcXHBsb3RzXFwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgXG5bMjBdIFxcUkVBRE1FLm1kXFwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBcblsyMV0gXFxyZXN1bHRzXFwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFxuWzIyXSBcXHNjcmVlbnNob3RzXFwgICAgICAgICAgICAgICAgICAgICAgICAgICAgXG5bMjNdIFxcc2NyaXB0c1xcICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBcbjAwYS1yc3R1ZGlvX2d1aWRlLm1kXG5cbjAwYi1kZWJ1Z2dpbmdfcmVzb3VyY2VzLm1kXG5cbjAwYy1nb29kLXNjaWVudGlmaWMtY29kaW5nLXByYWN0aWNlcy5tZFxuXG4wMS1pbnRyb190b19iYXNlX1ItbGl2ZS5SbWRcblxuMDEtaW50cm9fdG9fYmFzZV9SLm5iLmh0bWxcblxuMDEtaW50cm9fdG9fYmFzZV9SLlJtZFxuXG4wMi1pbnRyb190b19nZ3Bsb3QyLWxpdmUuUm1kXG5cbjAyLWludHJvX3RvX2dncGxvdDIubmIuaHRtbFxuXG4wMi1pbnRyb190b19nZ3Bsb3QyLlJtZFxuXG4wMy1pbnRyb190b190aWR5dmVyc2UtbGl2ZS5SbWRcblxuMDMtaW50cm9fdG9fdGlkeXZlcnNlLm5iLmh0bWxcblxuMDMtaW50cm9fdG9fdGlkeXZlcnNlLlJtZFxuXG5kYXRhXG5cbmRpYWdyYW1zXG5cbmV4ZXJjaXNlXzAxLWludHJvX3RvX2Jhc2VfUi5SbWRcblxuZXhlcmNpc2VfMDItaW50cm9fdG9fUi5SbWRcblxuZXhlcmNpc2VfMDNhLWludHJvX3RvX3RpZHl2ZXJzZS5SbWRcblxuZXhlcmNpc2VfMDNiLWludHJvX3RvX3RpZHl2ZXJzZS5SbWRcblxucGxvdHNcblxuUkVBRE1FLm1kXG5cbnJlc3VsdHNcblxuc2NyZWVuc2hvdHNcblxuc2NyaXB0c1xuIn0= -->
-
      
-
      [1] \00a-rstudio_guide.md 
      
-[2] \00b-debugging_resources.md 
      
-[3] \00c-good-scientific-coding-practices.md
      
-[4] \01-intro_to_base_R-live.Rmd 
      
-[5] \01-intro_to_base_R.nb.html 
      
-[6] \01-intro_to_base_R.Rmd 
      
-[7] \02-intro_to_ggplot2-live.Rmd 
      
-[8] \02-intro_to_ggplot2.nb.html 
      
-[9] \02-intro_to_ggplot2.Rmd 
      
-[10] \03-intro_to_tidyverse-live.Rmd 
      
-[11] \03-intro_to_tidyverse.nb.html 
      
-[12] \03-intro_to_tidyverse.Rmd 
      
-[13]  
      
-[14]  
      
-[15] _01-intro_to_base_R.Rmd 
      
-[16] _02-intro_to_R.Rmd 
      
-[17] _03a-intro_to_tidyverse.Rmd 
      
-[18] _03b-intro_to_tidyverse.Rmd 
      
-[19]  
      
-[20] .md 
      
-[21]  
      
-[22]  
      
-[23]  
      
-00a-rstudio_guide.md
      
-
      00b-debugging_resources.md
      
-
      00c-good-scientific-coding-practices.md
      
-
      01-intro_to_base_R-live.Rmd
      
-
      01-intro_to_base_R.nb.html
      
-
      01-intro_to_base_R.Rmd
      
-
      02-intro_to_ggplot2-live.Rmd
      
-
      02-intro_to_ggplot2.nb.html
      
-
      02-intro_to_ggplot2.Rmd
      
-
      03-intro_to_tidyverse-live.Rmd
      
-
      03-intro_to_tidyverse.nb.html
      
-
      03-intro_to_tidyverse.Rmd
      
-
      data
      
-
      diagrams
      
-
      exercise_01-intro_to_base_R.Rmd
      
-
      exercise_02-intro_to_R.Rmd
      
-
      exercise_03a-intro_to_tidyverse.Rmd
      
-
      exercise_03b-intro_to_tidyverse.Rmd
      
-
      plots
      
-
      README.md
      
-
      results
      
-
      screenshots
      
-
      scripts
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-This shows us there is no folder called "results" yet. 
-
-If we want to more pointedly look for "results" in our working directory we can 
-use the `dir.exists()` function.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBDaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzXG5kaXIuZXhpc3RzKFxccmVzdWx0c1xcKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Check if the results directory exists
-dir.exists(\results\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFRSVUVcbiJ9 -->
-
      
-
      [1] TRUE
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-If the above says `FALSE` that means we will need to create a `results` 
-directory using the function `dir.create()`.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBNYWtlIGEgZGlyZWN0b3J5IHdpdGhpbiB0aGUgd29ya2luZyBkaXJlY3RvcnkgY2FsbGVkICdyZXN1bHRzJ1xuZGlyLmNyZWF0ZShcXHJlc3VsdHNcXClcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-# Make a directory within the working directory called 'results'
-dir.create(\results\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-After creating the results directory above, let's re-run `dir.exists()` to see 
-if now it exists.
+
+
+
       [1] "00a-rstudio_guide.md"                   
+ [2] "00b-debugging_resources.md"             
+ [3] "00c-good-scientific-coding-practices.md"
+ [4] "01-intro_to_base_R-live.Rmd"            
+ [5] "01-intro_to_base_R.nb.html"             
+ [6] "01-intro_to_base_R.Rmd"                 
+ [7] "02-intro_to_ggplot2-live.Rmd"           
+ [8] "02-intro_to_ggplot2.nb.html"            
+ [9] "02-intro_to_ggplot2.Rmd"                
+[10] "03-intro_to_tidyverse-live.Rmd"         
+[11] "03-intro_to_tidyverse.nb.html"          
+[12] "03-intro_to_tidyverse.Rmd"              
+[13] "data"                                   
+[14] "diagrams"                               
+[15] "exercise_01-intro_to_base_R.Rmd"        
+[16] "exercise_02-intro_to_R.Rmd"             
+[17] "exercise_03a-intro_to_tidyverse.Rmd"    
+[18] "exercise_03b-intro_to_tidyverse.Rmd"    
+[19] "plots"                                  
+[20] "README.md"                              
+[21] "results"                                
+[22] "screenshots"                            
+[23] "scripts"                                
      
+
+
+
      00a-rstudio_guide.md
 
+00b-debugging_resources.md
 
-<!-- rnb-text-end -->
+00c-good-scientific-coding-practices.md
 
+01-intro_to_base_R-live.Rmd
 
-<!-- rnb-chunk-begin -->
+01-intro_to_base_R.nb.html
 
+01-intro_to_base_R.Rmd
 
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBSZS1jaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzXG5kaXIuZXhpc3RzKFxccmVzdWx0c1xcKVxuYGBgXG5gYGAifQ== -->
+02-intro_to_ggplot2-live.Rmd
 
-```r
-```r
-# Re-check if the results directory exists
-dir.exists(\results\)
      
-
      
-<!-- rnb-source-end -->
+02-intro_to_ggplot2.nb.html
 
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFRSVUVcbiJ9 -->
-
      
-
      [1] TRUE
      
-
      
+02-intro_to_ggplot2.Rmd
 
+03-intro_to_tidyverse-live.Rmd
 
-<!-- rnb-output-end -->
+03-intro_to_tidyverse.nb.html
 
-<!-- rnb-chunk-end -->
+03-intro_to_tidyverse.Rmd
 
+data
 
-<!-- rnb-text-begin -->
+diagrams
 
+exercise_01-intro_to_base_R.Rmd
 
-We can use the output of `dir.exists()` to automatically create or hold off on 
-creating a directory by putting this together in an `if` statement like below. 
-An `if` statement has two main parts:
-First, the test, which is an expression that will result in either `TRUE` or `FALSE`.
-This is put in parenthesis immediately after the `if`.
-The next part is the body, which is the commands that will be executed *if* the 
-test is `TRUE`.
-These are placed within a set of braces `{ }`.
-Note that we used an exclamation point in the test to signify that we want a 
-directory to be created only *if* `dir.exists(results)` is NOT equal to `TRUE`.
+exercise_02-intro_to_R.Rmd
 
+exercise_03a-intro_to_tidyverse.Rmd
 
-<!-- rnb-text-end -->
+exercise_03b-intro_to_tidyverse.Rmd
 
+plots
 
-<!-- rnb-chunk-begin -->
+README.md
 
+results
 
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBJZiAncmVzdWx0cycgZGlyZWN0b3J5IGRvZXNuJ3QgZXhpc3QuLi5cbmlmICghZGlyLmV4aXN0cyhcXHJlc3VsdHNcXCkpIHtcbiAgIyAuLi4gY3JlYXRlIGEgJ3Jlc3VsdHMnIGRpcmVjdG9yeVxuICBkaXIuY3JlYXRlKFxccmVzdWx0c1xcKVxufVxuYGBgXG5gYGAifQ== -->
+screenshots
 
-```r
-```r
-# If 'results' directory doesn't exist...
-if (!dir.exists(\results\)) {
+scripts
      
+
+
+
+
      This shows us there is no folder called “results” yet.
      
+
      If we want to more pointedly look for “results” in our working directory we can use the dir.exists() function.
      
+
+
+
+
      # Check if the results directory exists
+dir.exists("results")
      
+
+
+
      [1] TRUE
      
+
+
+
+
      If the above says FALSE that means we will need to create a results directory using the function dir.create().
      
+
+
+
+
      # Make a directory within the working directory called 'results'
+dir.create("results")
      
+
+
+
      Warning in dir.create("results"): 'results' already exists
      
+
+
+
+
      After creating the results directory above, let’s re-run dir.exists() to see if now it exists.
      
+
+
+
+
      # Re-check if the results directory exists
+dir.exists("results")
      
+
+
+
      [1] TRUE
      
+
+
+
+
      We can use the output of dir.exists() to automatically create or hold off on creating a directory by putting this together in an if statement like below. An if statement has two main parts: First, the test, which is an expression that will result in either TRUE or FALSE. This is put in parenthesis immediately after the if. The next part is the body, which is the commands that will be executed if the test is TRUE. These are placed within a set of braces { }. Note that we used an exclamation point in the test to signify that we want a directory to be created only if dir.exists(results) is NOT equal to TRUE.
      
+
+
+
+
      # If 'results' directory doesn't exist...
+if (!dir.exists("results")) {
   # ... create a 'results' directory
-  dir.create(\results\)
+  dir.create("results")
 }
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-The `dir.exists()` function will not work on files themselves.
-In that case, there is an analogous function called `file.exists()`.
-
-Try using the `file.exists()` function to see if the file 
-`gene_results_GSE44971.tsv` exists in the current directory.
-Use the code chunk we set up for you below. 
-Note that in our notebooks (and sometimes elsewhere), wherever you see a 
-`<FILL_IN_THE_BLANK>` like in the chunk below, that is meant for you to replace 
-(including the angle brackets) with the correct phrase before you run the chunk 
-(otherwise you will get an error).
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjpbIiMgUmVwbGFjZSB0aGUgPFBVVF9GSUxFX05BTUVfSEVSRT4gd2l0aCB0aGUgbmFtZSBvZiB0aGUgZmlsZSB5b3UgYXJlIGxvb2tpbmcgZm9yIiwiIyBSZW1lbWJlciB0byB1c2UgcXVvdGVzIHRvIG1ha2UgaXQgYSBjaGFyYWN0ZXIgc3RyaW5nIiwiZmlsZS5leGlzdHMoPFBVVF9GSUxFX05BTUVfSEVSRT4pIl19 -->
-
-```r
-# Replace the <PUT_FILE_NAME_HERE> with the name of the file you are looking for
+
+
+
+
      The dir.exists() function will not work on files themselves. In that case, there is an analogous function called file.exists().
      
+
      Try using the file.exists() function to see if the file gene_results_GSE44971.tsv exists in the current directory. Use the code chunk we set up for you below. Note that in our notebooks (and sometimes elsewhere), wherever you see a <FILL_IN_THE_BLANK> like in the chunk below, that is meant for you to replace (including the angle brackets) with the correct phrase before you run the chunk (otherwise you will get an error).
      
+
+
+
+
      # Replace the <PUT_FILE_NAME_HERE> with the name of the file you are looking for
 # Remember to use quotes to make it a character string
 file.exists(<PUT_FILE_NAME_HERE>)
      
 
@@ -655,362 +535,192 @@ 
      Read a TSV file
      
 
      Declare the name of the directory where we will read in the data.
      
 
 
-
-
      ```r
-data_dir <- \data\
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Although base R has functions to read in data files, the functions in the 
-`readr` package (part of the tidyverse) are faster and more straightforward 
-to use so we are going to use those here. 
-Because the file we are reading in is a TSV (tab separated values) file we will 
-be using the `read_tsv` function. 
-There are analogous functions for CSV (comma separated values) files 
-(`read_csv()`) and other files types.
-
-## Read in the differential expression analysis results file
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KFxuICBmaWxlLnBhdGgoZGF0YV9kaXIsXG4gICAgICAgICAgICBcXGdlbmVfcmVzdWx0c19HU0U0NDk3MS50c3ZcXClcbiAgKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-stats_df <- readr::read_tsv(
+
+
      data_dir <- "data"
      
+
+
+
+
      Although base R has functions to read in data files, the functions in the readr package (part of the tidyverse) are faster and more straightforward to use so we are going to use those here. Because the file we are reading in is a TSV (tab separated values) file we will be using the read_tsv function. There are analogous functions for CSV (comma separated values) files (read_csv()) and other files types.
      
+
      
+
+
      
+
      Read in the differential expression analysis results file
      
+
+
+
+
      stats_df <- readr::read_tsv(
   file.path(data_dir,
-            \gene_results_GSE44971.tsv\)
+            "gene_results_GSE44971.tsv")
   )
      
+
+
 
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Following the template of the previous chunk, use this chunk to read in the file
-`GSE44971.tsv` that is in the `data` folder and save it in the variable `gene_df`. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBVc2UgdGhpcyBjaHVuayB0byByZWFkIGluIGRhdGEgZnJvbSB0aGUgZmlsZSBgR1NFNDQ5NzEudHN2YFxuZ2VuZV9kZiA8LSByZWFkcjo6cmVhZF90c3YoXG4gIGZpbGUucGF0aChkYXRhX2RpcixcbiAgICAgICAgICAgIFxcR1NFNDQ5NzEudHN2XFwpXG4gIClcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-# Use this chunk to read in data from the file `GSE44971.tsv`
+── Column specification ────────────────────────────────────────────────────────
+cols(
+  ensembl_id = col_character(),
+  gene_symbol = col_character(),
+  contrast = col_character(),
+  log_fold_change = col_double(),
+  avg_expression = col_double(),
+  t_statistic = col_double(),
+  p_value = col_double(),
+  adj_p_value = col_double()
+)
      
+
+
+
+
      Following the template of the previous chunk, use this chunk to read in the file GSE44971.tsv that is in the data folder and save it in the variable gene_df.
      
+
+
+
+
      # Use this chunk to read in data from the file `GSE44971.tsv`
 gene_df <- readr::read_tsv(
   file.path(data_dir,
-            \GSE44971.tsv\)
+            "GSE44971.tsv")
   )
      
+
+
 
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Use this chunk to explore what `gene_df` looks like. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBFeHBsb3JlIGBnZW5lX2RmYFxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Explore `gene_df`
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What information is contained in `gene_df`?
-
-## dplyr pipes
-
-One nifty feature of the tidyverse is pipes: `%>%`
-These handy things allows you to funnel the result of one expression to the next,
-making your code a little more streamlined.
-
-For example, the output from this:  
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuZmlsdGVyKHN0YXRzX2RmLCBjb250cmFzdCA9PSBcXG1hbGVfZmVtYWxlXFwpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-filter(stats_df, contrast == \male_female\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-...is the same as the output from this:  
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgJT4lIGZpbHRlcihjb250cmFzdCA9PSBcXG1hbGVfZmVtYWxlXFwpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-stats_df %>% filter(contrast == \male_female\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-  
-This can make your code cleaner and easier to follow a series of related 
-commands. 
-Let's look at an example with our stats of of how the same 
-functions look with or without pipes:
-
-*Example 1:* without pipes: 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfYXJyYW5nZWQgPC0gYXJyYW5nZShzdGF0c19kZiwgdF9zdGF0aXN0aWMpXG5zdGF0c19maWx0ZXJlZCA8LSBmaWx0ZXIoc3RhdHNfYXJyYW5nZWQsIGF2Z19leHByZXNzaW9uID4gNTApXG5zdGF0c19ub3BpcGUgPC0gc2VsZWN0KHN0YXRzX2ZpbHRlcmVkLCBjb250cmFzdCwgbG9nX2ZvbGRfY2hhbmdlLCBwX3ZhbHVlKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-stats_arranged <- arrange(stats_df, t_statistic)
+── Column specification ────────────────────────────────────────────────────────
+cols(
+  .default = col_double(),
+  Gene = col_character()
+)
+ℹ Use `spec()` for the full column specifications.
      
+
+
+
+
      Use this chunk to explore what gene_df looks like.
      
+
+
+
+
      # Explore `gene_df`
      
+
+
+
+
      What information is contained in gene_df?
      
+
      
+
      
+
      magrittr pipes
      
+
      One nifty feature of the tidyverse is pipes: %>% These handy things, which come from the magrittr package, allow you to funnel the result of one expression to the next, making your code a little more streamlined.
      
+
      For example, the output from this:
      
+
+
+
+
      filter(stats_df, contrast == "male_female")
      
+
+
      
+
+
      
+
+
+
      …is the same as the output from this:
      
+
+
+
+
      stats_df %>% filter(contrast == "male_female")
      
+
+
      
+
+
      
+
+
+
      This can make your code cleaner and easier to follow a series of related commands. Let’s look at an example with our stats of of how the same functions look with or without pipes:
      
+
      Example 1: without pipes:
      
+
+
+
+
      stats_arranged <- arrange(stats_df, t_statistic)
 stats_filtered <- filter(stats_arranged, avg_expression > 50)
 stats_nopipe <- select(stats_filtered, contrast, log_fold_change, p_value)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-  
-UGH, we have to keep track of all of those different intermediate data frames 
-and type their names so many times here! 
-We could maybe streamline things by using the same variable name at each stage, 
-but even then there is a lot of extra typing, and it is easy to get confused 
-about what has been done where.
-It's annoying and makes it harder for people to read. 
-  
-*Example 2:* Same result as 1 but with pipes!
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBFeGFtcGxlIG9mIHRoZSBzYW1lIG1vZGlmaWNhdGlvbnMgYXMgYWJvdmUgYnV0IHdpdGggcGlwZXMhXG5zdGF0c19waXBlICA8LSBzdGF0c19kZiAlPiVcbiAgICAgICAgICAgICAgIGFycmFuZ2UodF9zdGF0aXN0aWMpICU+JVxuICAgICAgICAgICAgICAgZmlsdGVyKGF2Z19leHByZXNzaW9uID4gNTApICU+JVxuICAgICAgICAgICAgICAgc2VsZWN0KGNvbnRyYXN0LCBsb2dfZm9sZF9jaGFuZ2UsIHBfdmFsdWUpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Example of the same modifications as above but with pipes!
+
+
+
+
      UGH, we have to keep track of all of those different intermediate data frames and type their names so many times here! We could maybe streamline things by using the same variable name at each stage, but even then there is a lot of extra typing, and it is easy to get confused about what has been done where. It’s annoying and makes it harder for people to read.
      
+
      Example 2: Same result as 1 but with pipes!
      
+
+
+
+
      # Example of the same modifications as above but with pipes!
 stats_pipe  <- stats_df %>%
                arrange(t_statistic) %>%
                filter(avg_expression > 50) %>%
                select(contrast, log_fold_change, p_value)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What the `%>%` (pipe) is doing here is feeding the result of the expression on 
-its left into the first argument of the next function (to its right, or on the 
-next line here). 
-We can then skip that first argument (the data in these cases), and move right 
-on to the part we care about at that step: what we are arranging, filtering, or 
-selecting  in this case.
-
-Let's double check that these are the same by using the function, `all.equal()`. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuYWxsLmVxdWFsKHN0YXRzX25vcGlwZSwgc3RhdHNfcGlwZSlcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-all.equal(stats_nopipe, stats_pipe)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFRSVUVcbiJ9 -->
-
      
-
      [1] TRUE
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-`all.equal()` is letting us know that these two objects are the same. 
-
-Now that hopefully you are convinced that the tidyverse can help you make your 
-code neater and easier to use and read, let's go through some of the popular 
-tidyverse functions and so we can create pipelines like this. 
-
-## Common tidyverse functions
-
-Let's say we wanted to filter this gene expression dataset to particular sample
-groups.
-In order to do this, we would use the function `filter()` as well as a logic 
-statement (usually one that refers to a column or columns in the data frame).
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBIZXJlIGxldCdzIGZpbHRlciBzdGF0c19kZiB0byB0aGUgZ2VuZV9zeW1ib2wgXFxTTkNBXFxcbnN0YXRzX2RmICU+JSBcbiAgZmlsdGVyKGdlbmVfc3ltYm9sID09IFxcU05DQVxcKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Here let's filter stats_df to the gene_symbol \SNCA\
+
+
+
+
      What the %>% (pipe) is doing here is feeding the result of the expression on its left into the first argument of the next function (to its right, or on the next line here). We can then skip that first argument (the data in these cases), and move right on to the part we care about at that step: what we are arranging, filtering, or selecting in this case.
      
+
      Let’s double check that these are the same by using the function, all.equal().
      
+
+
+
+
      all.equal(stats_nopipe, stats_pipe)
      
+
+
+
      [1] TRUE
      
+
+
+
+
      all.equal() is letting us know that these two objects are the same.
      
+
      Now that hopefully you are convinced that the tidyverse can help you make your code neater and easier to use and read, let’s go through some of the popular tidyverse functions and so we can create pipelines like this.
      
+
      
+
      
+
      Common tidyverse functions
      
+
      Let’s say we wanted to filter this gene expression dataset to particular sample groups. In order to do this, we would use the function filter() as well as a logic statement (usually one that refers to a column or columns in the data frame).
      
+
+
+
+
      # Here let's filter stats_df to the gene_symbol "SNCA"
 stats_df %>% 
-  filter(gene_symbol == \SNCA\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-We can use `filter()` similarly for numeric statements.  
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBIZXJlIGxldCdzIGZpbHRlciB0aGUgZGF0YSB0byByb3dzIHdpdGggYXZlcmFnZSBleHByZXNzaW9uIHZhbHVlcyBhYm92ZSA1MFxuc3RhdHNfZGYgJT4lXG4gIGZpbHRlcihhdmdfZXhwcmVzc2lvbiA+IDUwKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Here let's filter the data to rows with average expression values above 50
+  filter(gene_symbol == "SNCA")
      
+
+
      
+
+
      
+
+
+
      We can use filter() similarly for numeric statements.
      
+
+
+
+
      # Here let's filter the data to rows with average expression values above 50
 stats_df %>%
   filter(avg_expression > 50)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-We can apply multiple filters at once, which will require all of them to be 
-satisfied for every row in the results:
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBmaWx0ZXIgdG8gaGlnaGx5IGV4cHJlc3NlZCBnZW5lcyB3aXRoIGNvbnRyYXN0IFxcbWFsZV9mZW1hbGVcXFxuc3RhdHNfZGYgJT4lXG4gIGZpbHRlcihjb250cmFzdCA9PSBcXG1hbGVfZmVtYWxlXFwsIFxuICAgICAgICAgYXZnX2V4cHJlc3Npb24gPiA1MClcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-# filter to highly expressed genes with contrast \male_female\
+
+
      
+
+
      
+
+
+
      We can apply multiple filters at once, which will require all of them to be satisfied for every row in the results:
      
+
+
+
+
      # filter to highly expressed genes with contrast "male_female"
 stats_df %>%
-  filter(contrast == \male_female\, 
+  filter(contrast == "male_female", 
          avg_expression > 50)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-When we are filtering, the `%in%` operator can come in handy if we have multiple
-items we would like to match.
-Let's take a look at what using `%in%` does.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjpbImdlbmVzX29mX2ludGVyZXN0IDwtIGMoXCJTTkNBXCIsIFwiQ0RLTjFBXCIpIiwic3RhdHNfZGYkZ2VuZV9zeW1ib2wgJWluJSBnZW5lc19vZl9pbnRlcmVzdCJdfQ== -->
-
-```r
-genes_of_interest <- c("SNCA", "CDKN1A")
+
+
      
+
+
      
+
+
+
      When we are filtering, the %in% operator can come in handy if we have multiple items we would like to match. Let’s take a look at what using %in% does.
      
+
+
+
+
      genes_of_interest <- c("SNCA", "CDKN1A")
 stats_df$gene_symbol %in% genes_of_interest
      
 
 
@@ -1018,627 +728,314 @@ 
      Read a TSV file
      
 
      %in% returns a logical vector that now we can use in dplyr::filter.
      
 
 
-
-
      ```r
-# filter to genes of interest
+
+
      # filter to genes of interest
 stats_df %>% 
-  filter(gene_symbol %in% c(\SNCA\, \CDKN1A\))
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Let's return to our first `filter()` and build on to it. 
-This time, let's keep only some of the columns from the data frame using the 
-`select()` function. 
-Let's also save this as a new data frame called `stats_filtered_df`.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBmaWx0ZXIgdG8gaGlnaGx5IGV4cHJlc3NlZCBcXG1hbGVfZmVtYWxlXFxcbiMgYW5kIHNlbGVjdCBnZW5lX3N5bWJvbCwgbG9nX2ZvbGRfY2hhbmdlIGFuZCB0X3N0YXRpc3RpY1xuc3RhdHNfZmlsdGVyZWRfZGYgPC0gc3RhdHNfZGYgJT4lXG4gIGZpbHRlcihjb250cmFzdCA9PSBcXG1hbGVfZmVtYWxlXFwsIFxuICAgICAgICAgYXZnX2V4cHJlc3Npb24gPiA1MCkgJT4lXG4gIHNlbGVjdChsb2dfZm9sZF9jaGFuZ2UsIHRfc3RhdGlzdGljKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# filter to highly expressed \male_female\
+  filter(gene_symbol %in% c("SNCA", "CDKN1A"))
      
+
+
      
+
+
      
+
+
+
      Let’s return to our first filter() and build on to it. This time, let’s keep only some of the columns from the data frame using the select() function. Let’s also save this as a new data frame called stats_filtered_df.
      
+
+
+
+
      # filter to highly expressed "male_female"
 # and select gene_symbol, log_fold_change and t_statistic
 stats_filtered_df <- stats_df %>%
-  filter(contrast == \male_female\, 
+  filter(contrast == "male_female", 
          avg_expression > 50) %>%
   select(log_fold_change, t_statistic)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Let's say we wanted to arrange this dataset so that the genes are arranged by 
-the smallest p values to the largest.
-In order to do this, we would use the function `arrange()` as well as the column
-we would like to sort by (in this case `p_value`).
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgJT4lIFxuICBhcnJhbmdlKHBfdmFsdWUpIFxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-stats_df %>% 
+
+
+
+
      Let’s say we wanted to arrange this dataset so that the genes are arranged by the smallest p values to the largest. In order to do this, we would use the function arrange() as well as the column we would like to sort by (in this case p_value).
      
+
+
+
+
      stats_df %>% 
   arrange(p_value) 
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What if we want to sort from largest to smallest? 
-Like if we want to see the genes with the highest average expression?
-We can use the same function, but instead use the `desc()` function and now we 
-are using `avg_expression` column. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBhcnJhbmdlIGRlc2NlbmRpbmcgYnkgYXZnX2V4cHJlc3Npb25cbnN0YXRzX2RmICU+JVxuICBhcnJhbmdlKGRlc2MoYXZnX2V4cHJlc3Npb24pKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# arrange descending by avg_expression
+
+
      
+
+
      
+
+
+
      What if we want to sort from largest to smallest? Like if we want to see the genes with the highest average expression? We can use the same function, but instead use the desc() function and now we are using avg_expression column.
      
+
+
+
+
      # arrange descending by avg_expression
 stats_df %>%
   arrange(desc(avg_expression))
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What if we would like to create a new column of values?
-For that we use `mutate()` function.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgJT4lIFxuICBtdXRhdGUobG9nMTBfcF92YWx1ZSA9IC1sb2cxMChwX3ZhbHVlKSlcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-stats_df %>% 
+
+
      
+
+
      
+
+
+
      What if we would like to create a new column of values? For that we use mutate() function.
      
+
+
+
+
      stats_df %>% 
   mutate(log10_p_value = -log10(p_value))
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What if we want to obtain summary statistics for a column or columns?
-The `summarize` function allows us to calculate summary statistics for a column. 
-Here we will use summarize to obtain an mean log folder change over all the
-genes, and its standard deviation.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgJT4lIFxuICBzdW1tYXJpemUobWVhbihsb2dfZm9sZF9jaGFuZ2UpLFxuICAgICAgICAgICAgc2QobG9nX2ZvbGRfY2hhbmdlKSlcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-stats_df %>% 
-  summarize(mean(log_fold_change),
-            sd(log_fold_change))
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-What if we'd like to obtain a summary statistics but have them for various 
-groups?
-Conveniently named, there's a function called `group_by()` that seamlessly 
-allows us to do this. 
-Also note that `group_by()` allows us to group by multiple variables at a time 
-if you want to.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfc3VtbWFyeV9kZiA8LSBzdGF0c19kZiAlPiVcbiAgICAgIGdyb3VwX2J5KGNvbnRyYXN0KSAlPiUgXG4gICAgICBzdW1tYXJpemUobWVhbihsb2dfZm9sZF9jaGFuZ2UpLFxuICAgICAgICAgICAgICAgIHNkKGxvZ19mb2xkX2NoYW5nZSkpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-stats_summary_df <- stats_df %>%
-      group_by(contrast) %>% 
-      summarize(mean(log_fold_change),
-                sd(log_fold_change))
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Let's look at a preview of what we made:
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfc3VtbWFyeV9kZlxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-stats_summary_df
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Here we have the mean log fold change expression per each contrast we made. 
-
-## A brief intro to the `apply` family of functions
-
-In base R, the `apply` family of functions can be an alternative methods for 
-performing transformations across a data frame, matrix or other object structures. 
-
-One of this family is (shockingly) the function `apply()`, which operates on 
-matrices.
-
-A matrix is similar to a data frame in that it is a rectangular table of data,
-but it has an additional constraint: 
-rather than each column having a type, ALL data in a matrix has the same type.
-
-The first argument to `apply()` is the data object we want to work on.
-The third argument is the function we will apply to each row or column of the 
-data object.
-The second argument in specifies whether we are applying the function 
-across rows or across columns (1 for rows, 2 for columns).
-
-Remember that `gene_df` is a gene x sample gene expression data frame that has
-columns of two different types, character and numeric.
-Converting it to a matrix will require us to make them all the same type.
-We can coerce it into a matrix using `as.matrix()`, in which case R will
-pick a type that it can convert everything to.
-What does it choose?
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBDb2VyY2UgYGdlbmVfZGZgIGludG8gYSBtYXRyaXhcbmdlbmVfbWF0cml4IDwtIGFzLm1hdHJpeChnZW5lX2RmKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Coerce `gene_df` into a matrix
+
+
      
+
+
      
+
+
+
      What if we want to obtain summary statistics for a column or columns? The summarize function allows us to calculate summary statistics for a column. Here we will use summarize to obtain an mean log folder change over all the genes, and its standard deviation.
      
+
+
+
+
      stats_df %>% 
+  summarize(mean(log_fold_change),
+            sd(log_fold_change))
      
+
+
      
+
+
      
+
+
+
      What if we’d like to obtain a summary statistics but have them for various groups? Conveniently named, there’s a function called group_by() that seamlessly allows us to do this. Also note that group_by() allows us to group by multiple variables at a time if you want to.
      
+
+
+
+
      stats_summary_df <- stats_df %>%
+      group_by(contrast) %>% 
+      summarize(mean(log_fold_change),
+                sd(log_fold_change))
      
+
+
+
+
      Let’s look at a preview of what we made:
      
+
+
+
+
      stats_summary_df
      
+
+
      
+
+
      
+
+
+
      Here we have the mean log fold change expression per each contrast we made.
      
+
      
+
      
+
      A brief intro to the apply family of functions
      
+
      In base R, the apply family of functions can be an alternative methods for performing transformations across a data frame, matrix or other object structures.
      
+
      One of this family is (shockingly) the function apply(), which operates on matrices.
      
+
      A matrix is similar to a data frame in that it is a rectangular table of data, but it has an additional constraint: rather than each column having a type, ALL data in a matrix has the same type.
      
+
      The first argument to apply() is the data object we want to work on. The third argument is the function we will apply to each row or column of the data object. The second argument in specifies whether we are applying the function across rows or across columns (1 for rows, 2 for columns).
      
+
      Remember that gene_df is a gene x sample gene expression data frame that has columns of two different types, character and numeric. Converting it to a matrix will require us to make them all the same type. We can coerce it into a matrix using as.matrix(), in which case R will pick a type that it can convert everything to. What does it choose?
      
+
+
+
+
      # Coerce `gene_df` into a matrix
 gene_matrix <- as.matrix(gene_df)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBFeHBsb3JlIHRoZSBzdHJ1Y3R1cmUgb2YgdGhlIGBnZW5lX21hdHJpeGAgb2JqZWN0XG5zdHIoZ2VuZV9tYXRyaXgpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Explore the structure of the `gene_matrix` object
+
+
+
+
+
+
+
      # Explore the structure of the `gene_matrix` object
 str(gene_matrix)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiIGNociBbMToyMDA1NiwgMTo1OV0gXFxFTlNHMDAwMDAwMDAwMDNcXCBcXEVOU0cwMDAwMDAwMDAwNVxcIFxcRU5TRzAwMDAwMDAwNDE5XFwgLi4uXG4gLSBhdHRyKCosIFxcZGltbmFtZXNcXCk9TGlzdCBvZiAyXG4gIC4uJCA6IE5VTExcbiAgLi4kIDogY2hyIFsxOjU5XSBcXEdlbmVcXCBcXEdTTTEwOTQ4MTRcXCBcXEdTTTEwOTQ4MTVcXCBcXEdTTTEwOTQ4MTZcXCAuLi5cbiJ9 -->
-
      
-
      chr [1:20056, 1:59]    … - attr(*, )=List of 2 ..$ : NULL ..$ : chr [1:59]     …
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-While that worked, it is rare that we want numbers converted to text, so we are
-going to select only the columns with numeric values before converting it to a matrix.
-We can do this most easily by removing the first column, which contains the gene names 
-stored as character values.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBMZXQncyBzYXZlIGEgbmV3IG1hdHJpeCBvYmplY3QgbmFtZXMgYGdlbmVfbnVtX21hdHJpeGAgY29udGFpbmluZyBvbmx5XG4jIHRoZSBudW1lcmljIHZhbHVlc1xuZ2VuZV9udW1fbWF0cml4IDwtIGFzLm1hdHJpeChnZW5lX2RmWywgLTFdKVxuXG4jIEV4cGxvcmUgdGhlIHN0cnVjdHVyZSBvZiB0aGUgYGdlbmVfbnVtX21hdHJpeGAgb2JqZWN0XG5zdHIoZ2VuZV9udW1fbWF0cml4KVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Let's save a new matrix object names `gene_num_matrix` containing only
+
+
+
       chr [1:20056, 1:59] "ENSG00000000003" "ENSG00000000005" "ENSG00000000419" ...
+ - attr(*, "dimnames")=List of 2
+  ..$ : NULL
+  ..$ : chr [1:59] "Gene" "GSM1094814" "GSM1094815" "GSM1094816" ...
      
+
+
+
+
      While that worked, it is rare that we want numbers converted to text, so we are going to select only the columns with numeric values before converting it to a matrix. We can do this most easily by removing the first column, which contains the gene names stored as character values.
      
+
+
+
+
      # Let's save a new matrix object names `gene_num_matrix` containing only
 # the numeric values
 gene_num_matrix <- as.matrix(gene_df[, -1])
 
 # Explore the structure of the `gene_num_matrix` object
 str(gene_num_matrix)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiIG51bSBbMToyMDA1NiwgMTo1OF0gOS41OTUxIC0wLjA0MzYgOC41MjQ2IDEuNjAxMyAwLjYxODkgLi4uXG4gLSBhdHRyKCosIFxcZGltbmFtZXNcXCk9TGlzdCBvZiAyXG4gIC4uJCA6IE5VTExcbiAgLi4kIDogY2hyIFsxOjU4XSBcXEdTTTEwOTQ4MTRcXCBcXEdTTTEwOTQ4MTVcXCBcXEdTTTEwOTQ4MTZcXCBcXEdTTTEwOTQ4MTdcXCAuLi5cbiJ9 -->
-
      
-
      num [1:20056, 1:58] 9.5951 -0.0436 8.5246 1.6013 0.6189 … - attr(*, )=List of 2 ..$ : NULL ..$ : chr [1:58]     …
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Why do we have a `[, -1]` after `gene_df` in the above chunk?
-
-Now that the matrix is all numbers, we can do things like calculate the column
-or row statistics using `apply()`.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBDYWxjdWxhdGUgcm93IG1lYW5zXG5nZW5lX21lYW5zIDwtIGFwcGx5KGdlbmVfbnVtX21hdHJpeCwgMSwgbWVhbikgIyBOb3RpY2Ugd2UgYXJlIHVzaW5nIDEgaGVyZVxuXG4jIEhvdyBsb25nIHdpbGwgYGdlbmVfbWVhbnNgIGJlPyBcbmxlbmd0aChnZW5lX21lYW5zKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Calculate row means
+
+
+
       num [1:20056, 1:58] 9.5951 -0.0436 8.5246 1.6013 0.6189 ...
+ - attr(*, "dimnames")=List of 2
+  ..$ : NULL
+  ..$ : chr [1:58] "GSM1094814" "GSM1094815" "GSM1094816" "GSM1094817" ...
      
+
+
+
+
      Why do we have a [, -1] after gene_df in the above chunk?
      
+
      Now that the matrix is all numbers, we can do things like calculate the column or row statistics using apply().
      
+
+
+
+
      # Calculate row means
 gene_means <- apply(gene_num_matrix, 1, mean) # Notice we are using 1 here
 
 # How long will `gene_means` be? 
 length(gene_means)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIDIwMDU2XG4ifQ== -->
-
      
-
      [1] 20056
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Note that we can obtain the same results if we select just the columns with numeric
-values from the `gene_df` data frame.
-This allows R to do the as.matrix() coercion automatically, and can be a handy shortcut
-if you have a *mostly* numeric data frame. 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBDYWxjdWxhdGUgcm93IG1lYW5zIHVzaW5nIHRoZSBgZ2VuZV9kZmAgb2JqZWN0IGFmdGVyIHJlbW92aW5nIHRoZSBjaGFyYWN0ZXIgY29sdW1uXG4jIGFwcGx5KCkgY29udmVydHMgdGhpcyB0byBhIG1hdHJpeCBpbnRlcm5hbGx5XG5nZW5lX21lYW5zX2Zyb21fZGYgPC0gYXBwbHkoZ2VuZV9kZlssIC0xXSwgMSwgbWVhbikgXG5cbiMgTGV0J3MgY2hlY2sgdGhhdCB0aGUgdHdvIGdlbmUgbWVhbnMgb2JqZWN0cyBhcmUgZXF1YWxcbmFsbC5lcXVhbChnZW5lX21lYW5zLCBnZW5lX21lYW5zX2Zyb21fZGYpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Calculate row means using the `gene_df` object after removing the character column
+
+
+
      [1] 20056
      
+
+
+
+
      Note that we can obtain the same results if we select just the columns with numeric values from the gene_df data frame. This allows R to do the as.matrix() coercion automatically, and can be a handy shortcut if you have a mostly numeric data frame.
      
+
+
+
+
      # Calculate row means using the `gene_df` object after removing the character column
 # apply() converts this to a matrix internally
 gene_means_from_df <- apply(gene_df[, -1], 1, mean) 
 
 # Let's check that the two gene means objects are equal
 all.equal(gene_means, gene_means_from_df)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFRSVUVcbiJ9 -->
-
      
-
      [1] TRUE
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Now let's investigate the same set up, but use 2 to `apply` over the columns of
-our matrix.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBDYWxjdWxhdGUgc2FtcGxlIG1lYW5zXG5zYW1wbGVfbWVhbnMgPC0gYXBwbHkoZ2VuZV9udW1fbWF0cml4LCAyLCBtZWFuKSAjIE5vdGljZSB3ZSB1c2UgMiBoZXJlXG5cbiMgSG93IGxvbmcgd2lsbCBgc2FtcGxlX21lYW5zYCBiZT8gXG5sZW5ndGgoc2FtcGxlX21lYW5zKVxuYGBgXG5gYGAifQ== -->
-
-```r
-```r
-# Calculate sample means
+
+
+
      [1] TRUE
      
+
+
+
+
      Now let’s investigate the same set up, but use 2 to apply over the columns of our matrix.
      
+
+
+
+
      # Calculate sample means
 sample_means <- apply(gene_num_matrix, 2, mean) # Notice we use 2 here
 
 # How long will `sample_means` be? 
 length(sample_means)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIDU4XG4ifQ== -->
-
      
-
      [1] 58
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-We can put the gene names back into the numeric matrix object by
-assigning them as rownames.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBBc3NpZ24gdGhlIGdlbmUgbmFtZXMgZnJvbSBnZW5lX2RmJEdlbmUgdG8gdGhlIGBnZW5lX251bV9tYXRyaXhgIG9iamVjdCB1c2luZ1xuIyB0aGUgYHJvd25hbWVzKClgIGZ1bmN0aW9uXG5yb3duYW1lcyhnZW5lX251bV9tYXRyaXgpIDwtIGdlbmVfZGYkR2VuZVxuXG4jIEV4cGxvcmUgdGhlIGBnZW5lX251bV9tYXRyaXhgIG9iamVjdFxuaGVhZChnZW5lX251bV9tYXRyaXgpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Assign the gene names from gene_df$Gene to the `gene_num_matrix` object using
+
+
+
      [1] 58
      
+
+
+
+
      We can put the gene names back into the numeric matrix object by assigning them as rownames.
      
+
+
+
+
      # Assign the gene names from gene_df$Gene to the `gene_num_matrix` object using
 # the `rownames()` function
 rownames(gene_num_matrix) <- gene_df$Gene
 
 # Explore the `gene_num_matrix` object
 head(gene_num_matrix)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiICAgICAgICAgICAgICAgICBHU00xMDk0ODE0IEdTTTEwOTQ4MTUgR1NNMTA5NDgxNiAgIEdTTTEwOTQ4MTcgIEdTTTEwOTQ4MThcbkVOU0cwMDAwMDAwMDAwMyAgOS41OTUxMDE1MCAgOC40Nzg1MDcwIDEyLjY4MDIxMjkgIDguNjc3NjE0ODM4IDEwLjc1NTUyOTQ2XG5FTlNHMDAwMDAwMDAwMDUgLTAuMDQzNjE4MzggLTAuMTMwNzg4OSAgMC41MzQ1OTMxIC0wLjAwNTgwNTE2NiAtMC4wNTQzMDI1NVxuRU5TRzAwMDAwMDAwNDE5ICA4LjUyNDU4NTcxICA5Ljg0MDU3MjUgMTEuOTkyMzIwMSAgOS42MzkxNjMzMTcgMTAuMDMzNDkzMjdcbkVOU0cwMDAwMDAwMDQ1NyAgMS42MDEzMDU1MiAgMS44ODk1NTU0ICAxLjM3NDczODggIDEuNjM3ODI2MjE0ICAxLjYzNTYyNDkzXG5FTlNHMDAwMDAwMDA0NjAgIDAuNjE4OTEyODUgIDAuNTMyMTcwOCAgMC40ODA1NTk4ICAwLjYxNzk0Nzk3NiAgMC43MDYzNjEzNVxuRU5TRzAwMDAwMDAwOTM4ICAwLjU1NTczMDU4ICAwLjk5NDI4NjIgIDEuODAzMDE3NiAgMS4yMzczMTc0NTcgIDAuODQxNTI4NTJcbiAgICAgICAgICAgICAgICAgR1NNMTA5NDgxOSAgR1NNMTA5NDgyMCBHU00xMDk0ODIxICBHU00xMDk0ODIyIEdTTTEwOTQ4MjNcbkVOU0cwMDAwMDAwMDAwMyAgNi4zNzQ3MDY5MSAgOS4xMDAyODU4NCAgNy4zNTQ2ODYwICA4LjUxODQ3MTkwICA5LjQyMTYxMTNcbkVOU0cwMDAwMDAwMDAwNSAtMC4wNDgzMTE3NCAgMC4wMTQxMTM1OSAtMC4xMTA4Mjc5IC0wLjAyNjI1Nzc2IC0wLjE2OTI2MDRcbkVOU0cwMDAwMDAwMDQxOSAxMi43ODMzNTgyNiAxMC43NTU1Mjk0NiAgOS4xNzExMTEzICA5LjMwMjEwMTc0ICA5LjQ5MTU0MTVcbkVOU0cwMDAwMDAwMDQ1NyAgMS40NjU4NjA3MSAgMS43OTg1MjAzMiAgMS42Mzg5MjU5ICAxLjgwNTg2NzQ4ICAxLjU4MTM5NzlcbkVOU0cwMDAwMDAwMDQ2MCAgMC43NzIyNDU3MiAgMC44OTYwNzEzMiAgMC42NzQwNTU5ICAwLjYzMTU3OTU0ICAwLjc0ODA1NTZcbkVOU0cwMDAwMDAwMDkzOCAgMy4zMjQwNDYwNiAgMC44MTU2Mjg1NiAgMC45NzI4NjE3ICAwLjc3MTI5NzAwICAxLjM1OTY0MDJcbiAgICAgICAgICAgICAgICBHU00xMDk0ODI0ICBHU00xMDk0ODI1IEdTTTEwOTQ4MjYgIEdTTTEwOTQ4MjcgR1NNMTA5NDgyOFxuRU5TRzAwMDAwMDAwMDAzICA1LjAyMzk2MjkgIDcuODk3Mzc0NjAgIDguMTEyNjg3NiAgNy4wMzQ0NDY0MCAgOS42OTg0OTE4XG5FTlNHMDAwMDAwMDAwMDUgLTAuMTM1OTI0NyAtMC4wODYyNDI4NiAtMC4yMDQ0ODM5IC0wLjA5MDM3ODg3IC0wLjE2MDI0MTZcbkVOU0cwMDAwMDAwMDQxOSAxMS44ODM1ODk3IDEwLjg4MDc5NzgyICA5LjkxNzQ5MzAgMTAuNDE3NTM3MDEgMTAuMjY5NTUwM1xuRU5TRzAwMDAwMDAwNDU3ICAxLjU1MjU0MTAgIDEuOTI0ODkyNTQgIDEuODA0NjU5MCAgMS41MDM4MjE1OSAgMS42MTk4MDY5XG5FTlNHMDAwMDAwMDA0NjAgIDAuNzA3MjI3MyAgMC44OTE5NjA2OCAgMC44MjIzNTU5ICAwLjYxOTcwOTgyICAwLjY3NzY1NDlcbkVOU0cwMDAwMDAwMDkzOCAgMC44NzU4NDIxICAwLjYyMTkxNTE1ICAwLjc2NzU5NzEgIDAuOTI3OTEzMzggIDEuMDM1MTA2N1xuICAgICAgICAgICAgICAgICBHU00xMDk0ODI5IEdTTTEwOTQ4MzAgR1NNMTA5NDgzMSBHU00xMDk0ODMyICBHU00xMDk0ODMzXG5FTlNHMDAwMDAwMDAwMDMgMTMuOTg2ODkyMzAgMTAuNTg2ODMzMSAgNy42ODM2MjIzICA4LjM4NjI1ODcgMTEuMTg5MzI3NjNcbkVOU0cwMDAwMDAwMDAwNSAtMC4wNTAzODcwNSAgMC4zMDk2MDMxIC0wLjE1NTEwNjIgLTAuMTkzODk5NCAtMC4wODM2OTUzN1xuRU5TRzAwMDAwMDAwNDE5IDEwLjEyMTA0MDUzICA5LjM2NTM1NzYgMTAuMjE4NDExMCAgOS41OTUxMDE1IDEyLjU4NzEzOTgyXG5FTlNHMDAwMDAwMDA0NTcgIDEuNjc3NDE4MzIgIDEuNTc2MjQ3MSAgMi4wNjYzNDkzICAxLjc1MDQ5MjggIDEuNjczMjE2MzJcbkVOU0cwMDAwMDAwMDQ2MCAgMC43MTc4NjI1MCAgMC40OTkxNjIwICAwLjc5MTI1NTkgIDAuODEwMzAyMyAgMC45NDI0ODY5OFxuRU5TRzAwMDAwMDAwOTM4ICAwLjgyMTUyMTY1ICAwLjY1NTY1NzIgIDAuOTc4MjU5OSAgMC42NTY4MzUzICAwLjczNDU4NzgyXG4gICAgICAgICAgICAgICAgR1NNMTA5NDgzNCBHU00xMDk0ODM1ICBHU00xMDk0ODM2IEdTTTEwOTQ4MzcgR1NNMTA5NDgzOFxuRU5TRzAwMDAwMDAwMDAzICA5Ljc1NjIwMDMgIDkuNjk4NDkxOCAxMC41Njg5MTUxMCAgOS45MzkxMDI1ICA3Ljg3MzgxMzFcbkVOU0cwMDAwMDAwMDAwNSAtMC4wNDM3NjAxIC0wLjExMjA3NTUgLTAuMDgyMDgzMDYgLTAuMjA2NzExMiAtMC4xMjExODkxXG5FTlNHMDAwMDAwMDA0MTkgIDkuOTc5OTY0NiAxMC4wNzk4OTc0ICA5LjU5NTEwMTUwICA5Ljc0MTc5MjcgMTAuMjEwNTgyN1xuRU5TRzAwMDAwMDAwNDU3ICAxLjM3Nzg1OTQgIDEuNTYzMDg4OSAgMS43NDE0NjUzMiAgMS42NTE4MDM2ICAxLjc4MDYxMzNcbkVOU0cwMDAwMDAwMDQ2MCAgMC42MjAxOTI1ICAwLjU1NzAzMDAgIDAuNzAwODQ5ODMgIDAuNzEzNzExOCAgMC43MzU1MTU0XG5FTlNHMDAwMDAwMDA5MzggIDAuNzY3NDEzMiAgMS4yMTY1MjI4ICAwLjYwODU2MTA2ICAwLjYwNDE2NDUgIDEuMDYyNDA2N1xuICAgICAgICAgICAgICAgIEdTTTEwOTQ4MzkgIEdTTTEwOTQ4NDAgR1NNMTA5NDg0MSBHU00xMDk0ODQyIEdTTTEwOTQ4NDNcbkVOU0cwMDAwMDAwMDAwMyAgOC42MzExMzUzICA4LjU4MDc3NTU3ICA5LjE1Nzk1ODUgIDYuMzMxNzAxOSAxMC4xOTM5Mzg3XG5FTlNHMDAwMDAwMDAwMDUgLTAuMTEwOTA3MCAtMC4wMzk2MzU2NCAtMC4xMTQ4MTA2IC0wLjExMzcxNTAgLTAuMTY0NTA0MFxuRU5TRzAwMDAwMDAwNDE5IDEwLjE1MTczNDQgMTEuMTg5MzI3NjMgMTAuNTc3NTEzMiAxMy4zNzYwOTcxIDEyLjIyNzE2OTNcbkVOU0cwMDAwMDAwMDQ1NyAgMS43MjgxNDQ2ICAxLjcwMjQyMjg3ICAxLjY1MDMwOTAgIDEuMjk5MDIwOCAgMS41Njg3ODY2XG5FTlNHMDAwMDAwMDA0NjAgIDAuNTI4NDIxOSAgMC42NzY0MzQ2NiAgMC43MzUzMjc2ICAwLjYyMjM5OTAgIDAuNTY0NjQwNlxuRU5TRzAwMDAwMDAwOTM4ICAwLjY0NjkwODQgIDAuODgxMjA5NDIgIDAuNTIzMDQxNCAgMC45OTA5NTE3ICAwLjg0ODQxNzRcbiAgICAgICAgICAgICAgICAgR1NNMTA5NDg0NCAgR1NNMTA5NDg0NSAgR1NNMTA5NDg0NiBHU00xMDk0ODQ3ICBHU00xMDk0ODQ4XG5FTlNHMDAwMDAwMDAwMDMgMTAuNDQzNjQxNTkgIDkuNjI0MzU3MjIgMTYuMDUwNzU5NDQgIDYuOTMzNDUwOCAgOC41NTE4MDkxMFxuRU5TRzAwMDAwMDAwMDA1IC0wLjAzMTMyNDI3IC0wLjAxNDAwNTM0IC0wLjAzNTI5MTEyICAwLjEyNjg4OTkgLTAuMDM4NTczODJcbkVOU0cwMDAwMDAwMDQxOSAxMC4yODU3MTk0NyAxMS40NzY4MjQyNCAgOS44ODU0MDUyMyAgOC45NjQ2NjgyIDExLjEwOTExMzMwXG5FTlNHMDAwMDAwMDA0NTcgIDEuNjYxNTAxNzUgIDEuNjIzMTI4MjkgIDEuMzczMjA3MjkgIDEuMzQwMjc0MiAgMC44MTcwMzkzMVxuRU5TRzAwMDAwMDAwNDYwICAwLjY2Nzk4OTI2ICAwLjU4MDg5NjU5ICAwLjQ2OTU3NjA3ICAwLjQyMjI0NTUgIDAuMjk1MDA2NTdcbkVOU0cwMDAwMDAwMDkzOCAgMC41MzcyNjQ4NCAgMS4wODk5NzUzNSAgMC45MTg1OTY2NCAgMC44MTcwMzkzICAxLjkyNDg5MjU0XG4gICAgICAgICAgICAgICAgIEdTTTEwOTQ4NDkgR1NNMTA5NDg1MCBHU00xMDk0ODUxICBHU00xMDk0ODUyICBHU00xMDk0ODUzXG5FTlNHMDAwMDAwMDAwMDMgIDkuMjk0OTc3NjAgIDcuNTAyNzA5OCAgNi45NTkzMTE5ICA4LjMzNTg4NTMyICA4LjE2ODI2MTEwXG5FTlNHMDAwMDAwMDAwMDUgLTAuMDkyNjk3NzcgLTAuMTcxMjU0NSAgMC42MzU5NDU1IC0wLjA0OTUxOTE2IC0wLjA1NTc2NjQ0XG5FTlNHMDAwMDAwMDA0MTkgMTEuMTY5NDEyNjAgMTQuNDQzMjM4OSAgOS41MzI5NDQwIDExLjUwODU0NDUwICA5LjQ2MzYxNjc1XG5FTlNHMDAwMDAwMDA0NTcgIDEuNDU1NTc1NjcgIDEuNDUyODc0NCAgMS4xODcyMDI5ICAxLjMxNzQ0OTcxICAxLjIwODYzNTY1XG5FTlNHMDAwMDAwMDA0NjAgIDAuNDM2MTE0ODIgIDAuMzYyMzAwOCAgMC41NTQ0MzQwICAwLjQxNDIzNzA4ICAwLjMzOTA5MTg2XG5FTlNHMDAwMDAwMDA5MzggIDEuMTgwMjU2MzkgIDEuNTY4MDU2MCAgMS42MjU2MzQ1ICAwLjg2NzA0MTQwICAxLjQ4ODUzMjMxXG4gICAgICAgICAgICAgICAgR1NNMTA5NDg1NCBHU00xMDk0ODU1IEdTTTEwOTQ4NTYgR1NNMTA5NDg1NyAgR1NNMTA5NDg1OFxuRU5TRzAwMDAwMDAwMDAzICA5LjgwMjAwNzcgNy45MjU4MDQ1MSAgOC41MTIyNDI2ICA4LjUzMDAyMTcgIDYuNDU3NzQxMjRcbkVOU0cwMDAwMDAwMDAwNSAtMC4xNjA2Njg3IDAuMDIyMjMzOTMgLTAuMTM0MDc2MiAgMC4wMTQzMTI2IC0wLjAyMDQzMTYzXG5FTlNHMDAwMDAwMDA0MTkgMTEuODAxMTI2MiA4LjY2NjA2ODczICA4LjY0ODQwMTMgIDguNzc3NTIxMiAgOS44ODU0MDUyM1xuRU5TRzAwMDAwMDAwNDU3ICAxLjQ1ODIwMjMgMS40MDEyNzEzMyAgMS4yMTExNTE0ICAxLjMzNTY3NzggIDEuNzg5MTE1MzNcbkVOU0cwMDAwMDAwMDQ2MCAgMC44MDg2ODc1IDAuNDQ5MjIzNTQgIDAuNTk5MjM5MCAgMC4zNzU5ODE4ICAwLjM1ODc4NDg5XG5FTlNHMDAwMDAwMDA5MzggIDIuMjA4OTg0NyAwLjc1NzQ2NDcyICAwLjc2Nzc5MTMgIDAuODUxOTI3MSAgMS4wNTE4NDkzOFxuICAgICAgICAgICAgICAgICBHU00xMDk0ODU5ICBHU00xMDk0ODYwIEdTTTEwOTQ4NjEgR1NNMTA5NDg2MiBHU00xMDk0ODYzXG5FTlNHMDAwMDAwMDAwMDMgIDguMDY4MzQ5MDYgIDYuMjk3MDQ5NDYgOS41OTUxMDE1MCAgOC4yMTk4NTcxICA2LjAyMDc5ODhcbkVOU0cwMDAwMDAwMDAwNSAtMC4wODM1NTE0MiAtMC4wNDY1NjcxNiAwLjAzMDMxMjQ5IC0wLjE2NDYxNDYgLTAuMTQwNTI4NFxuRU5TRzAwMDAwMDAwNDE5ICA4LjY3NzYxNDg0IDE0LjU5ODQzNzYzIDguMDMzODI4ODQgMTEuMDg4MTc5MiAxMC4xMTIzMjUxXG5FTlNHMDAwMDAwMDA0NTcgIDEuNDU3ODUwNDUgIDEuMTk4NTU3MTMgMS4zMTIyNDQ1NSAgMS4xNTg5NDcyICAxLjUwMjEzMDdcbkVOU0cwMDAwMDAwMDQ2MCAgMC42MjM1MjE1NiAgMC42NDQ3NzM1NCAwLjMwMzgwNzM1ICAwLjMyMTc1MzEgIDAuMzEzMDg0MlxuRU5TRzAwMDAwMDAwOTM4ICAwLjcwNzU3NDEyICAxLjc4MTgxOTY2IDAuNzg0OTYzNjggIDEuNTU1NjcwMCAgMC42OTUwNDg4XG4gICAgICAgICAgICAgICAgIEdTTTEwOTQ4NjQgIEdTTTEwOTQ4NjUgIEdTTTEwOTQ4NjYgIEdTTTEwOTQ4NjcgIEdTTTEwOTQ4NjhcbkVOU0cwMDAwMDAwMDAwMyAxMC42MDc4MzE3NiAxMC4yMzUzNjYwOSA5LjAzMTk2NDIxMiAgNy42NjYyOTU0MCAgMS4wNjQ5NDUwMlxuRU5TRzAwMDAwMDAwMDA1ICAwLjAyNjAyNjU2IC0wLjA3OTY4NzM0IDAuMDA3MzI2Njc4IC0wLjA5MDMwNzIzIC0wLjA5NTQzOTc3XG5FTlNHMDAwMDAwMDA0MTkgIDkuNjg0MTI2MjEgMTIuNzk5OTM1NTcgOS43OTQ0ODQ2MzkgMTAuMjM1MzY2MDkgMTIuOTcwNzIyMjRcbkVOU0cwMDAwMDAwMDQ1NyAgMS40Nzg3NjQxOCAgMS40NjA4NDE4NSAxLjU4NTM4OTk2MiAgMS41OTExMTYwNCAgMS43NjIzMjUzMVxuRU5TRzAwMDAwMDAwNDYwICAwLjU3NzE3MTc1ICAwLjY0ODg4MjM0IDAuNzc3NDQ2OTExICAwLjY0NTEwOTQwICAwLjExOTg2NDkxXG5FTlNHMDAwMDAwMDA5MzggIDAuNTQ1NjExOTkgIDAuNTk4NjI1NjQgMC40Nzk3NzQzOTcgIDAuMzc3OTA2MjMgIDAuODM4ODA2NDlcbiAgICAgICAgICAgICAgICBHU00xMDk0ODY5IEdTTTEwOTQ4NzAgR1NNMTA5NDg3MVxuRU5TRzAwMDAwMDAwMDAzICAxLjA0MDgzMzIgIDEuNzI2MjA3OSAgMS4wMjkyMjU1XG5FTlNHMDAwMDAwMDAwMDUgLTAuMTAyMzczNCAtMC4xNTM3OTEwIC0wLjExMDA2MDNcbkVOU0cwMDAwMDAwMDQxOSAgOC42MTk2MzAwIDExLjk1ODgxODggMTAuODkwMDg0NlxuRU5TRzAwMDAwMDAwNDU3ICAxLjU3NTU1NDEgIDEuNzQ0NTM2MiAgMS42Mjc1MzA4XG5FTlNHMDAwMDAwMDA0NjAgIDAuMjM3MjU4NiAgMC4zNDU2NDU0ICAwLjE4ODUyODlcbkVOU0cwMDAwMDAwMDkzOCAgMC43MjAwMzY0ICAwLjkxOTk2NjcgIDAuNzI0MDI1NVxuIn0= -->
-
      
-
                   GSM1094814 GSM1094815 GSM1094816   GSM1094817  GSM1094818
      
-
      ENSG00000000003 9.59510150 8.4785070 12.6802129 8.677614838 10.75552946 ENSG00000000005 -0.04361838 -0.1307889 0.5345931 -0.005805166 -0.05430255 ENSG00000000419 8.52458571 9.8405725 11.9923201 9.639163317 10.03349327 ENSG00000000457 1.60130552 1.8895554 1.3747388 1.637826214 1.63562493 ENSG00000000460 0.61891285 0.5321708 0.4805598 0.617947976 0.70636135 ENSG00000000938 0.55573058 0.9942862 1.8030176 1.237317457 0.84152852 GSM1094819 GSM1094820 GSM1094821 GSM1094822 GSM1094823 ENSG00000000003 6.37470691 9.10028584 7.3546860 8.51847190 9.4216113 ENSG00000000005 -0.04831174 0.01411359 -0.1108279 -0.02625776 -0.1692604 ENSG00000000419 12.78335826 10.75552946 9.1711113 9.30210174 9.4915415 ENSG00000000457 1.46586071 1.79852032 1.6389259 1.80586748 1.5813979 ENSG00000000460 0.77224572 0.89607132 0.6740559 0.63157954 0.7480556 ENSG00000000938 3.32404606 0.81562856 0.9728617 0.77129700 1.3596402 GSM1094824 GSM1094825 GSM1094826 GSM1094827 GSM1094828 ENSG00000000003 5.0239629 7.89737460 8.1126876 7.03444640 9.6984918 ENSG00000000005 -0.1359247 -0.08624286 -0.2044839 -0.09037887 -0.1602416 ENSG00000000419 11.8835897 10.88079782 9.9174930 10.41753701 10.2695503 ENSG00000000457 1.5525410 1.92489254 1.8046590 1.50382159 1.6198069 ENSG00000000460 0.7072273 0.89196068 0.8223559 0.61970982 0.6776549 ENSG00000000938 0.8758421 0.62191515 0.7675971 0.92791338 1.0351067 GSM1094829 GSM1094830 GSM1094831 GSM1094832 GSM1094833 ENSG00000000003 13.98689230 10.5868331 7.6836223 8.3862587 11.18932763 ENSG00000000005 -0.05038705 0.3096031 -0.1551062 -0.1938994 -0.08369537 ENSG00000000419 10.12104053 9.3653576 10.2184110 9.5951015 12.58713982 ENSG00000000457 1.67741832 1.5762471 2.0663493 1.7504928 1.67321632 ENSG00000000460 0.71786250 0.4991620 0.7912559 0.8103023 0.94248698 ENSG00000000938 0.82152165 0.6556572 0.9782599 0.6568353 0.73458782 GSM1094834 GSM1094835 GSM1094836 GSM1094837 GSM1094838 ENSG00000000003 9.7562003 9.6984918 10.56891510 9.9391025 7.8738131 ENSG00000000005 -0.0437601 -0.1120755 -0.08208306 -0.2067112 -0.1211891 ENSG00000000419 9.9799646 10.0798974 9.59510150 9.7417927 10.2105827 ENSG00000000457 1.3778594 1.5630889 1.74146532 1.6518036 1.7806133 ENSG00000000460 0.6201925 0.5570300 0.70084983 0.7137118 0.7355154 ENSG00000000938 0.7674132 1.2165228 0.60856106 0.6041645 1.0624067 GSM1094839 GSM1094840 GSM1094841 GSM1094842 GSM1094843 ENSG00000000003 8.6311353 8.58077557 9.1579585 6.3317019 10.1939387 ENSG00000000005 -0.1109070 -0.03963564 -0.1148106 -0.1137150 -0.1645040 ENSG00000000419 10.1517344 11.18932763 10.5775132 13.3760971 12.2271693 ENSG00000000457 1.7281446 1.70242287 1.6503090 1.2990208 1.5687866 ENSG00000000460 0.5284219 0.67643466 0.7353276 0.6223990 0.5646406 ENSG00000000938 0.6469084 0.88120942 0.5230414 0.9909517 0.8484174 GSM1094844 GSM1094845 GSM1094846 GSM1094847 GSM1094848 ENSG00000000003 10.44364159 9.62435722 16.05075944 6.9334508 8.55180910 ENSG00000000005 -0.03132427 -0.01400534 -0.03529112 0.1268899 -0.03857382 ENSG00000000419 10.28571947 11.47682424 9.88540523 8.9646682 11.10911330 ENSG00000000457 1.66150175 1.62312829 1.37320729 1.3402742 0.81703931 ENSG00000000460 0.66798926 0.58089659 0.46957607 0.4222455 0.29500657 ENSG00000000938 0.53726484 1.08997535 0.91859664 0.8170393 1.92489254 GSM1094849 GSM1094850 GSM1094851 GSM1094852 GSM1094853 ENSG00000000003 9.29497760 7.5027098 6.9593119 8.33588532 8.16826110 ENSG00000000005 -0.09269777 -0.1712545 0.6359455 -0.04951916 -0.05576644 ENSG00000000419 11.16941260 14.4432389 9.5329440 11.50854450 9.46361675 ENSG00000000457 1.45557567 1.4528744 1.1872029 1.31744971 1.20863565 ENSG00000000460 0.43611482 0.3623008 0.5544340 0.41423708 0.33909186 ENSG00000000938 1.18025639 1.5680560 1.6256345 0.86704140 1.48853231 GSM1094854 GSM1094855 GSM1094856 GSM1094857 GSM1094858 ENSG00000000003 9.8020077 7.92580451 8.5122426 8.5300217 6.45774124 ENSG00000000005 -0.1606687 0.02223393 -0.1340762 0.0143126 -0.02043163 ENSG00000000419 11.8011262 8.66606873 8.6484013 8.7775212 9.88540523 ENSG00000000457 1.4582023 1.40127133 1.2111514 1.3356778 1.78911533 ENSG00000000460 0.8086875 0.44922354 0.5992390 0.3759818 0.35878489 ENSG00000000938 2.2089847 0.75746472 0.7677913 0.8519271 1.05184938 GSM1094859 GSM1094860 GSM1094861 GSM1094862 GSM1094863 ENSG00000000003 8.06834906 6.29704946 9.59510150 8.2198571 6.0207988 ENSG00000000005 -0.08355142 -0.04656716 0.03031249 -0.1646146 -0.1405284 ENSG00000000419 8.67761484 14.59843763 8.03382884 11.0881792 10.1123251 ENSG00000000457 1.45785045 1.19855713 1.31224455 1.1589472 1.5021307 ENSG00000000460 0.62352156 0.64477354 0.30380735 0.3217531 0.3130842 ENSG00000000938 0.70757412 1.78181966 0.78496368 1.5556700 0.6950488 GSM1094864 GSM1094865 GSM1094866 GSM1094867 GSM1094868 ENSG00000000003 10.60783176 10.23536609 9.031964212 7.66629540 1.06494502 ENSG00000000005 0.02602656 -0.07968734 0.007326678 -0.09030723 -0.09543977 ENSG00000000419 9.68412621 12.79993557 9.794484639 10.23536609 12.97072224 ENSG00000000457 1.47876418 1.46084185 1.585389962 1.59111604 1.76232531 ENSG00000000460 0.57717175 0.64888234 0.777446911 0.64510940 0.11986491 ENSG00000000938 0.54561199 0.59862564 0.479774397 0.37790623 0.83880649 GSM1094869 GSM1094870 GSM1094871 ENSG00000000003 1.0408332 1.7262079 1.0292255 ENSG00000000005 -0.1023734 -0.1537910 -0.1100603 ENSG00000000419 8.6196300 11.9588188 10.8900846 ENSG00000000457 1.5755541 1.7445362 1.6275308 ENSG00000000460 0.2372586 0.3456454 0.1885289 ENSG00000000938 0.7200364 0.9199667 0.7240255
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Row names like this can be very convenient for keeping matrices organized, but
-row names (and column names) can be lost or misordered if you are not careful,
-especially during input and output, so treat them with care.
-
-Although the `apply` functions may not be as easy to use as the tidyverse 
-functions, for some applications, `apply` methods can be better suited.
-In this workshop, we will not delve too deeply into the various other apply 
-functions (`tapply()`, `lapply()`, etc.) but you can read more information about 
-them [here](https://www.guru99.com/r-apply-sapply-tapply.html).
-
-## The dplyr::join functions
-
-Let's say we have a scenario where we have two data frames that we would like to 
-combine. 
-Recall that `stats_df` and `gene_df` are data frames that contain information 
-about some of the same genes.
-The [`dplyr::join` family of functions](https://dplyr.tidyverse.org/reference/join.html) 
-are useful for various scenarios of combining data frames. 
-
-For now, we will focus on `inner_join()`, which will combine data frames by only
-keeping information about matching rows that are in both data frames.
-We need to use the `by` argument to designate what column(s) 
-should be used as a key to match the data frames.
-In this case we want to match the gene information between the two, so we will 
-specify that we want to compare values in the `ensembl_id` column from 
-`stats_df` to the `Gene` column from `gene_df`.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuc3RhdHNfZGYgJT4lIFxuICBpbm5lcl9qb2luKGdlbmVfZGYsIGJ5ID0gYygnZW5zZW1ibF9pZCcgPSAnR2VuZScpKSBcbmBgYFxuYGBgIn0= -->
-
-```r
-```r
-stats_df %>% 
+
+
+
                       GSM1094814 GSM1094815 GSM1094816   GSM1094817  GSM1094818
+ENSG00000000003  9.59510150  8.4785070 12.6802129  8.677614838 10.75552946
+                 GSM1094819  GSM1094820 GSM1094821  GSM1094822 GSM1094823
+ENSG00000000003  6.37470691  9.10028584  7.3546860  8.51847190  9.4216113
+                GSM1094824  GSM1094825 GSM1094826  GSM1094827 GSM1094828
+ENSG00000000003  5.0239629  7.89737460  8.1126876  7.03444640  9.6984918
+                 GSM1094829 GSM1094830 GSM1094831 GSM1094832  GSM1094833
+ENSG00000000003 13.98689230 10.5868331  7.6836223  8.3862587 11.18932763
+                GSM1094834 GSM1094835  GSM1094836 GSM1094837 GSM1094838
+ENSG00000000003  9.7562003  9.6984918 10.56891510  9.9391025  7.8738131
+                GSM1094839  GSM1094840 GSM1094841 GSM1094842 GSM1094843
+ENSG00000000003  8.6311353  8.58077557  9.1579585  6.3317019 10.1939387
+                 GSM1094844  GSM1094845  GSM1094846 GSM1094847  GSM1094848
+ENSG00000000003 10.44364159  9.62435722 16.05075944  6.9334508  8.55180910
+                 GSM1094849 GSM1094850 GSM1094851  GSM1094852  GSM1094853
+ENSG00000000003  9.29497760  7.5027098  6.9593119  8.33588532  8.16826110
+                GSM1094854 GSM1094855 GSM1094856 GSM1094857  GSM1094858
+ENSG00000000003  9.8020077 7.92580451  8.5122426  8.5300217  6.45774124
+                 GSM1094859  GSM1094860 GSM1094861 GSM1094862 GSM1094863
+ENSG00000000003  8.06834906  6.29704946 9.59510150  8.2198571  6.0207988
+                 GSM1094864  GSM1094865  GSM1094866  GSM1094867  GSM1094868
+ENSG00000000003 10.60783176 10.23536609 9.031964212  7.66629540  1.06494502
+                GSM1094869 GSM1094870 GSM1094871
+ENSG00000000003  1.0408332  1.7262079  1.0292255
+ [ reached getOption("max.print") -- omitted 5 rows ]
      
+
+
+
+
      Row names like this can be very convenient for keeping matrices organized, but row names (and column names) can be lost or misordered if you are not careful, especially during input and output, so treat them with care.
      
+
      Although the apply functions may not be as easy to use as the tidyverse functions, for some applications, apply methods can be better suited. In this workshop, we will not delve too deeply into the various other apply functions (tapply(), lapply(), etc.) but you can read more information about them here.
      
+
      
+
      
+
      The dplyr::join functions
      
+
      Let’s say we have a scenario where we have two data frames that we would like to combine. Recall that stats_df and gene_df are data frames that contain information about some of the same genes. The dplyr::join family of functions are useful for various scenarios of combining data frames.
      
+
      For now, we will focus on inner_join(), which will combine data frames by only keeping information about matching rows that are in both data frames. We need to use the by argument to designate what column(s) should be used as a key to match the data frames. In this case we want to match the gene information between the two, so we will specify that we want to compare values in the ensembl_id column from stats_df to the Gene column from gene_df.
      
+
+
+
+
      stats_df %>% 
   inner_join(gene_df, by = c('ensembl_id' = 'Gene')) 
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-## Save data to files
-
-#### Save to TSV files
-
-Let's write some of the data frames we created to a file.
-To do this, we can use the `readr` library of `_write()` functions. 
-The first argument of `write_tsv()` is the data we want to write, and the second 
-argument is a character string that describes the path to the new file we would 
-like to create.
-Remember that we created a `results` directory to put our output in, 
-but if we want to save our data to a directory other than our working directory, 
-we need to specify this. 
-This is what we will use the `file.path()` function for. 
-Let's look in a bit more detail what `file.path()` does, by examining the 
-results of the function in the examples below.
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuIyBXaGljaCBvZiB0aGVzZSBmaWxlIHBhdGhzIGlzIHdoYXQgd2Ugd2FudCB0byB1c2UgdG8gc2F2ZSBvdXIgZGF0YSB0byB0aGVcbiMgcmVzdWx0cyBkaXJlY3Rvcnkgd2UgY3JlYXRlZCBhdCB0aGUgYmVnaW5uaW5nIG9mIHRoaXMgbm90ZWJvb2s/XG5maWxlLnBhdGgoXFxkb2NrZXItaW5zdGFsbFxcLCBcXHN0YXRzX3N1bW1hcnkudHN2XFwpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-# Which of these file paths is what we want to use to save our data to the
+
+
      
+
+
      
+
+
+
      
+
      
+
      Save data to files
      
+
      
+
      Save to TSV files
      
+
      Let’s write some of the data frames we created to a file. To do this, we can use the readr library of _write() functions. The first argument of write_tsv() is the data we want to write, and the second argument is a character string that describes the path to the new file we would like to create. Remember that we created a results directory to put our output in, but if we want to save our data to a directory other than our working directory, we need to specify this. This is what we will use the file.path() function for. Let’s look in a bit more detail what file.path() does, by examining the results of the function in the examples below.
      
+
+
+
+
      # Which of these file paths is what we want to use to save our data to the
 # results directory we created at the beginning of this notebook?
-file.path(\docker-install\, \stats_summary.tsv\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFxcZG9ja2VyLWluc3RhbGwvc3RhdHNfc3VtbWFyeS50c3ZcXFxuZG9ja2VyLWluc3RhbGwvc3RhdHNfc3VtbWFyeS50c3ZcbiJ9 -->
-
      
-
      [1] -install/stats_summary.tsv
      
-docker-install/stats_summary.tsv
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuZmlsZS5wYXRoKFxccmVzdWx0c1xcLCBcXHN0YXRzX3N1bW1hcnkudHN2XFwpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-file.path(\results\, \stats_summary.tsv\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFxccmVzdWx0cy9zdGF0c19zdW1tYXJ5LnRzdlxcXG5yZXN1bHRzL3N0YXRzX3N1bW1hcnkudHN2XG4ifQ== -->
-
      
-
      [1] /stats_summary.tsv
      
-results/stats_summary.tsv
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-source-begin eyJkYXRhIjoiYGBgclxuYGBgclxuZmlsZS5wYXRoKFxcc3RhdHNfc3VtbWFyeS50c3ZcXCwgXFxyZXN1bHRzXFwpXG5gYGBcbmBgYCJ9 -->
-
-```r
-```r
-file.path(\stats_summary.tsv\, \results\)
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiWzFdIFxcc3RhdHNfc3VtbWFyeS50c3YvcmVzdWx0c1xcXG5zdGF0c19zdW1tYXJ5LnRzdi9yZXN1bHRzXG4ifQ== -->
-
      
-
      [1] _summary.tsv/results
      
-stats_summary.tsv/results
      
-
      
-
-
-<!-- rnb-output-end -->
-
-<!-- rnb-chunk-end -->
-
-
-<!-- rnb-text-begin -->
-
-
-Replace `<NEW_FILE_PATH>` below with the `file.path()` statement from above that 
-will successfully save our file to the `results` folder 
-
-
-<!-- rnb-text-end -->
-
-
-<!-- rnb-chunk-begin -->
-
-
-<!-- rnb-source-begin eyJkYXRhIjpbIiMgV3JpdGUgb3VyIGRhdGEgZnJhbWUgdG8gYSBUU1YgZmlsZSIsInJlYWRyOjp3cml0ZV90c3Yoc3RhdHNfc3VtbWFyeV9kZiwgPE5FV19GSUxFX1BBVEg+KSJdfQ== -->
-
-```r
-# Write our data frame to a TSV file
+file.path("docker-install", "stats_summary.tsv")
      
+
+
+
      [1] "docker-install/stats_summary.tsv"
      
+
+
+
      docker-install/stats_summary.tsv
      
+
+
+
      file.path("results", "stats_summary.tsv")
      
+
+
+
      [1] "results/stats_summary.tsv"
      
+
+
+
      results/stats_summary.tsv
      
+
+
+
      file.path("stats_summary.tsv", "results")
      
+
+
+
      [1] "stats_summary.tsv/results"
      
+
+
+
      stats_summary.tsv/results
      
+
+
+
+
      Replace <NEW_FILE_PATH> below with the file.path() statement from above that will successfully save our file to the results folder
      
+
+
+
+
      # Write our data frame to a TSV file
 readr::write_tsv(stats_summary_df, <NEW_FILE_PATH>)
      
 
 
@@ -1674,44 +1071,54 @@ 
      Read an RDS file
      
 
      Session Info
      
 
 
-
-
      ```r
-# Print out the versions and packages we are using in this session
+
+
      # Print out the versions and packages we are using in this session
 sessionInfo()
      
-
      
-<!-- rnb-source-end -->
-
-<!-- rnb-output-begin eyJkYXRhIjoiUiB2ZXJzaW9uIDQuMC4zICgyMDIwLTEwLTEwKVxuUGxhdGZvcm06IHg4Nl82NC1wYy1saW51eC1nbnUgKDY0LWJpdClcblJ1bm5pbmcgdW5kZXI6IFVidW50dSAyMC4wNCBMVFNcblxuTWF0cml4IHByb2R1Y3RzOiBkZWZhdWx0XG5CTEFTL0xBUEFDSzogL3Vzci9saWIveDg2XzY0LWxpbnV4LWdudS9vcGVuYmxhcy1wdGhyZWFkL2xpYm9wZW5ibGFzcC1yMC4zLjguc29cblxubG9jYWxlOlxuIFsxXSBMQ19DVFlQRT1lbl9VUy5VVEYtOCAgICAgICBMQ19OVU1FUklDPUMgICAgICAgICAgICAgIFxuIFszXSBMQ19USU1FPWVuX1VTLlVURi04ICAgICAgICBMQ19DT0xMQVRFPWVuX1VTLlVURi04ICAgIFxuIFs1XSBMQ19NT05FVEFSWT1lbl9VUy5VVEYtOCAgICBMQ19NRVNTQUdFUz1DICAgICAgICAgICAgIFxuIFs3XSBMQ19QQVBFUj1lbl9VUy5VVEYtOCAgICAgICBMQ19OQU1FPUMgICAgICAgICAgICAgICAgIFxuIFs5XSBMQ19BRERSRVNTPUMgICAgICAgICAgICAgICBMQ19URUxFUEhPTkU9QyAgICAgICAgICAgIFxuWzExXSBMQ19NRUFTVVJFTUVOVD1lbl9VUy5VVEYtOCBMQ19JREVOVElGSUNBVElPTj1DICAgICAgIFxuXG5hdHRhY2hlZCBiYXNlIHBhY2thZ2VzOlxuWzFdIHN0YXRzICAgICBncmFwaGljcyAgZ3JEZXZpY2VzIHV0aWxzICAgICBkYXRhc2V0cyAgbWV0aG9kcyAgIGJhc2UgICAgIFxuXG5vdGhlciBhdHRhY2hlZCBwYWNrYWdlczpcbiBbMV0gZm9yY2F0c18wLjUuMCAgIHN0cmluZ3JfMS40LjAgICBkcGx5cl8xLjAuMyAgICAgcHVycnJfMC4zLjQgICAgXG4gWzVdIHJlYWRyXzEuNC4wICAgICB0aWR5cl8xLjEuMiAgICAgdGliYmxlXzMuMC41ICAgIGdncGxvdDJfMy4zLjMgIFxuIFs5XSB0aWR5dmVyc2VfMS4zLjAgb3B0cGFyc2VfMS42LjYgXG5cbmxvYWRlZCB2aWEgYSBuYW1lc3BhY2UgKGFuZCBub3QgYXR0YWNoZWQpOlxuIFsxXSBSY3BwXzEuMC42ICAgICAgICBjZWxscmFuZ2VyXzEuMS4wICBwaWxsYXJfMS40LjcgICAgICBjb21waWxlcl80LjAuMyAgIFxuIFs1XSBkYnBseXJfMi4wLjAgICAgICB0b29sc180LjAuMyAgICAgICBkaWdlc3RfMC42LjI3ICAgICBsdWJyaWRhdGVfMS43LjkuMlxuIFs5XSBqc29ubGl0ZV8xLjcuMiAgICBldmFsdWF0ZV8wLjE0ICAgICBsaWZlY3ljbGVfMC4yLjAgICBndGFibGVfMC4zLjAgICAgIFxuWzEzXSBwa2djb25maWdfMi4wLjMgICBybGFuZ18wLjQuMTAgICAgICByZXByZXhfMC4zLjAgICAgICBjbGlfMi4yLjAgICAgICAgIFxuWzE3XSByc3R1ZGlvYXBpXzAuMTMgICBEQklfMS4xLjEgICAgICAgICB5YW1sXzIuMi4xICAgICAgICBoYXZlbl8yLjMuMSAgICAgIFxuWzIxXSB4ZnVuXzAuMjAgICAgICAgICB3aXRocl8yLjQuMCAgICAgICB4bWwyXzEuMy4yICAgICAgICBodHRyXzEuNC4yICAgICAgIFxuWzI1XSBrbml0cl8xLjMwICAgICAgICBmc18xLjUuMCAgICAgICAgICBobXNfMS4wLjAgICAgICAgICBnZW5lcmljc18wLjEuMCAgIFxuWzI5XSB2Y3Ryc18wLjMuNiAgICAgICBncmlkXzQuMC4zICAgICAgICBnZXRvcHRfMS4yMC4zICAgICB0aWR5c2VsZWN0XzEuMS4wIFxuWzMzXSBnbHVlXzEuNC4yICAgICAgICBSNl8yLjUuMCAgICAgICAgICBmYW5zaV8wLjQuMiAgICAgICByZWFkeGxfMS4zLjEgICAgIFxuWzM3XSBybWFya2Rvd25fMi42ICAgICBtb2RlbHJfMC4xLjggICAgICBtYWdyaXR0cl8yLjAuMSAgICBwc18xLjUuMCAgICAgICAgIFxuWzQxXSBiYWNrcG9ydHNfMS4yLjEgICBzY2FsZXNfMS4xLjEgICAgICBlbGxpcHNpc18wLjMuMSAgICBodG1sdG9vbHNfMC41LjEuMVxuWzQ1XSBydmVzdF8wLjMuNiAgICAgICBhc3NlcnR0aGF0XzAuMi4xICBjb2xvcnNwYWNlXzIuMC0wICBzdHJpbmdpXzEuNS4zICAgIFxuWzQ5XSBtdW5zZWxsXzAuNS4wICAgICBicm9vbV8wLjcuMyAgICAgICBjcmF5b25fMS4zLjQgICAgIFxuIn0= -->
-
      
-
      R version 4.0.3 (2020-10-10) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 20.04 LTS
      
-
      Matrix products: default BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
      
-
      locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
      
-[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
      
-[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
      
-[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
      
-[9] LC_ADDRESS=C LC_TELEPHONE=C
      
-[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
      
-
      attached base packages: [1] stats graphics grDevices utils datasets methods base
      
-
      other attached packages: [1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.3 purrr_0.3.4
      
-[5] readr_1.4.0 tidyr_1.1.2 tibble_3.0.5 ggplot2_3.3.3
      
-[9] tidyverse_1.3.0 optparse_1.6.6
      
-
      loaded via a namespace (and not attached): [1] Rcpp_1.0.6 cellranger_1.1.0 pillar_1.4.7 compiler_4.0.3
      
-[5] dbplyr_2.0.0 tools_4.0.3 digest_0.6.27 lubridate_1.7.9.2 [9] jsonlite_1.7.2 evaluate_0.14 lifecycle_0.2.0 gtable_0.3.0
      
-[13] pkgconfig_2.0.3 rlang_0.4.10 reprex_0.3.0 cli_2.2.0
      
-[17] rstudioapi_0.13 DBI_1.1.1 yaml_2.2.1 haven_2.3.1
      
-[21] xfun_0.20 withr_2.4.0 xml2_1.3.2 httr_1.4.2
      
-[25] knitr_1.30 fs_1.5.0 hms_1.0.0 generics_0.1.0
      
-[29] vctrs_0.3.6 grid_4.0.3 getopt_1.20.3 tidyselect_1.1.0 [33] glue_1.4.2 R6_2.5.0 fansi_0.4.2 readxl_1.3.1
      
-[37] rmarkdown_2.6 modelr_0.1.8 magrittr_2.0.1 ps_1.5.0
      
-[41] backports_1.2.1 scales_1.1.1 ellipsis_0.3.1 htmltools_0.5.1.1 [45] rvest_0.3.6 assertthat_0.2.1 colorspace_2.0-0 stringi_1.5.3
      
-[49] munsell_0.5.0 broom_0.7.3 crayon_1.3.4
      
-```
      
+
+
+
      R version 4.0.3 (2020-10-10)
+Platform: x86_64-pc-linux-gnu (64-bit)
+Running under: Ubuntu 20.04 LTS
+
+Matrix products: default
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
+
+locale:
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
+
+attached base packages:
+[1] stats     graphics  grDevices utils     datasets  methods   base     
+
+other attached packages:
+ [1] forcats_0.5.0   stringr_1.4.0   dplyr_1.0.3     purrr_0.3.4    
+ [5] readr_1.4.0     tidyr_1.1.2     tibble_3.0.5    ggplot2_3.3.3  
+ [9] tidyverse_1.3.0 optparse_1.6.6 
+
+loaded via a namespace (and not attached):
+ [1] Rcpp_1.0.6        cellranger_1.1.0  pillar_1.4.7      compiler_4.0.3   
+ [5] dbplyr_2.0.0      tools_4.0.3       digest_0.6.27     lubridate_1.7.9.2
+ [9] jsonlite_1.7.2    evaluate_0.14     lifecycle_0.2.0   gtable_0.3.0     
+[13] pkgconfig_2.0.3   rlang_0.4.10      reprex_0.3.0      cli_2.2.0        
+[17] rstudioapi_0.13   DBI_1.1.1         yaml_2.2.1        haven_2.3.1      
+[21] xfun_0.20         withr_2.4.0       xml2_1.3.2        httr_1.4.2       
+[25] knitr_1.30        fs_1.5.0          hms_1.0.0         generics_0.1.0   
+[29] vctrs_0.3.6       grid_4.0.3        getopt_1.20.3     tidyselect_1.1.0 
+[33] glue_1.4.2        R6_2.5.0          fansi_0.4.2       readxl_1.3.1     
+[37] rmarkdown_2.6     modelr_0.1.8      magrittr_2.0.1    ps_1.5.0         
+[41] backports_1.2.1   scales_1.1.1      ellipsis_0.3.1    htmltools_0.5.1.1
+[45] rvest_0.3.6       assertthat_0.2.1  colorspace_2.0-0  stringi_1.5.3    
+[49] munsell_0.5.0     broom_0.7.3       crayon_1.3.4     
      
 
 
 
      
 
      
 
-
      LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIHRpZHl2ZXJzZSIKYXV0aG9yOiAiQ0NETCBmb3IgQUxTRiIgCmRhdGU6IDIwMjEKb3V0cHV0OiAgIAogIGh0bWxfbm90ZWJvb2s6IAogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKZWRpdG9yX29wdGlvbnM6IAogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgoKIyMgT2JqZWN0aXZlcwoKVGhpcyBub3RlYm9vayB3aWxsIGRlbW9uc3RyYXRlIGhvdyB0bzoKCi0gVXNlIHRoZSB0aWR5dmVyc2UgcGFja2FnZSB0byByZWFkIGFuZCB3cml0ZSBkYXRhIGZyYW1lcwotIFVzZSBkcGx5ciBwaXBlcyB0byBtYW5pcHVsYXRlIGRhdGEgZnJhbWVzCi0gSW1wbGVtZW50IGFuZCB1c2UgZnVuY3Rpb25zIGluIHRoZSB0aWR5dmVyc2UgcGFja2FnZSB0byB3cmFuZ2xlIGRhdGEgKGkuZS4gZmlsdGVyLCBtdXRhdGUsIGFycmFuZ2UsIGpvaW4pCi0gVXNlIHRoZSBhcHBseSBmdW5jdGlvbiB0byBhcHBseSBmdW5jdGlvbnMgYWNyb3NzIGVudGlyZSBkYXRhIGZyYW1lcyAgCgpXZSdsbCB1c2UgdGhlIHNhbWUgZ2VuZSBleHByZXNzaW9uIGRhdGFzZXQgd2UgdXNlZCBpbiB0aGUgW3ByZXZpb3VzIG5vdGVib29rXSguLzAyLWludHJvX3RvX2dncGxvdDIuUm1kKS4KSXQgaXMgYSBwcmUtcHJvY2Vzc2VkIFthc3Ryb2N5dG9tYSBtaWNyb2FycmF5IGRhdGFzZXRdKGh0dHBzOi8vd3d3LnJlZmluZS5iaW8vZXhwZXJpbWVudHMvR1NFNDQ5NzEvZ2VuZS1leHByZXNzaW9uLWRhdGEtZnJvbS1waWxvY3l0aWMtYXN0cm9jeXRvbWEtdHVtb3VyLXNhbXBsZXMtYW5kLW5vcm1hbC1jZXJlYmVsbHVtLWNvbnRyb2xzKSAKdGhhdCB3ZSBwZXJmb3JtZWQgYSBzZXQgb2YgW2RpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2VzIG9uXSguL3NjcmlwdHMvMDAtc2V0dXAtaW50cm8tdG8tUi5SKS4gIAoKKipNb3JlIHRpZHl2ZXJzZSByZXNvdXJjZXM6KiogCgotIFtSIGZvciBEYXRhIFNjaWVuY2VdKGh0dHBzOi8vcjRkcy5oYWQuY28ubnovKSAgCi0gW3RpZHl2ZXJzZSBkb2N1bWVudGF0aW9uXShodHRwczovL2RwbHlyLnRpZHl2ZXJzZS5vcmcvKSAgCi0gW0NoZWF0c2hlZXQgb2YgdGlkeXZlcnNlIGRhdGEgdHJhbnNmb3JtYXRpb25dKGh0dHBzOi8vZ2l0aHViLmNvbS9yc3R1ZGlvL2NoZWF0c2hlZXRzL3Jhdy9tYXN0ZXIvZGF0YS10cmFuc2Zvcm1hdGlvbi5wZGYpICAKLSBbT25saW5lIHRpZHl2ZXJzZSBib29rIGNoYXB0ZXJdKGh0dHBzOi8vcHJpdmVmbC5naXRodWIuaW8vYWR2cjM4Ym9vay90aWR5dmVyc2UuaHRtbCkgIAoKIyMgU2V0IFVwCgpUaGUgdGlkeXZlcnNlIGlzIGEgY29sbGVjdGlvbiBvZiBwYWNrYWdlcyB0aGF0IGFyZSBoYW5keSBmb3IgZ2VuZXJhbCBkYXRhIAp3cmFuZ2xpbmcsIGFuYWx5c2lzLCBhbmQgdmlzdWFsaXphdGlvbi4gCk90aGVyIHBhY2thZ2VzIHRoYXQgYXJlIHNwZWNpZmljYWxseSBoYW5keSBmb3IgZGlmZmVyZW50IGJpb2xvZ2ljYWwgYW5hbHlzZXMgYXJlIApmb3VuZCBvbiBbQmlvY29uZHVjdG9yXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnLykuCklmIHdlIHdhbnQgdG8gdXNlIGEgcGFja2FnZSdzIGZ1bmN0aW9ucyB3ZSBmaXJzdCBuZWVkIHRvIGluc3RhbGwgdGhlbS4KCk91ciBSU3R1ZGlvIFNlcnZlciBhbHJlYWR5IGhhcyB0aGUgYHRpZHl2ZXJzZWAgZ3JvdXAgb2YgcGFja2FnZXMgaW5zdGFsbGVkIGZvciB5b3UuIApCdXQgaWYgeW91IG5lZWRlZCB0byBpbnN0YWxsIGl0IG9yIG90aGVyIHBhY2thZ2VzIGF2YWlsYWJsZSBvbiBDUkFOLCB5b3UgCmRvIGl0IHVzaW5nIHRoZSBgaW5zdGFsbC5wYWNrYWdlcygpYCBmdW5jdGlvbiBsaWtlIHRoaXM6IApgaW5zdGFsbC5wYWNrYWdlcygidGlkeXZlcnNlIilgLgoKYGBge3IgdGlkeXZlcnNlfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIyMgUmVmZXJlbmNpbmcgYSBsaWJyYXJ5J3MgZnVuY3Rpb24gd2l0aCBgOjpgCgpOb3RlIHRoYXQgaWYgd2UgaGFkIG5vdCBpbXBvcnRlZCB0aGUgdGlkeXZlcnNlIHNldCBvZiBwYWNrYWdlcyB1c2luZyBgbGlicmFyeSgpYCAKbGlrZSBhYm92ZSwgYW5kIHdlIHdhbnRlZCB0byB1c2UgYSB0aWR5dmVyc2UgZnVuY3Rpb24gbGlrZSBgcmVhZF90c3YoKWAsIHdlIAp3b3VsZCBuZWVkIHRvIHRlbGwgUiB3aGF0IHBhY2thZ2UgdG8gZmluZCB0aGlzIGZ1bmN0aW9uIGluLgpUbyBkbyB0aGlzLCB3ZSB3b3VsZCB1c2UgYDo6YCB0byB0ZWxsIFIgdG8gbG9hZCBpbiB0aGlzIGZ1bmN0aW9uIGZyb20gdGhlIApgcmVhZHJgIHBhY2thZ2UgYnkgdXNpbmcgYHJlYWRyOjpyZWFkX3RzdigpYC4KWW91IHdpbGwgc2VlIHRoaXMgYDo6YCBtZXRob2Qgb2YgcmVmZXJlbmNpbmcgbGlicmFyaWVzIHdpdGhpbiBwYWNrYWdlcyAKdGhyb3VnaG91dCB0aGUgY291cnNlLiAKV2UgbGlrZSB0byB1c2UgaXQgaW4gcGFydCB0byByZW1vdmUgYW55IGFtYmlndWl0eSBpbiB3aGljaCB2ZXJzaW9uIG9mIGEgCmZ1bmN0aW9uIHdlIGFyZSB1c2luZzsgaXQgaXMgbm90IHRvbyB1bmNvbW1vbiBmb3IgZGlmZmVyZW50IHBhY2thZ2VzIHRvIHVzZSB0aGUgCnNhbWUgbmFtZSBmb3IgdmVyeSBkaWZmZXJlbnQgZnVuY3Rpb25zIQoKIyMgTWFuYWdpbmcgZGlyZWN0b3JpZXMKCkJlZm9yZSB3ZSBjYW4gaW1wb3J0IHRoZSBkYXRhIHdlIG5lZWQsIHdlIHNob3VsZCBkb3VibGUgY2hlY2sgd2hlcmUgUiBpcyAKbG9va2luZyBmb3IgZmlsZXMsIGFrYSB0aGUgY3VycmVudCAqKndvcmtpbmcgZGlyZWN0b3J5KiouIApXZSBjYW4gZG8gdGhpcyBieSB1c2luZyB0aGUgYGdldHdkKClgIGZ1bmN0aW9uLCB3aGljaCB3aWxsIHRlbGwgdXMgd2hhdCBmb2xkZXIKd2UgYXJlIGluLiAKCmBgYHtyIHdvcmtpbmdkaXIsIGxpdmUgPSBUUlVFfQojIExldCdzIGNoZWNrIHdoYXQgZGlyZWN0b3J5IHdlIGFyZSBpbjoKZ2V0d2QoKQpgYGAKCkZvciBSbWQgZmlsZXMsIHRoZSB3b3JraW5nIGRpcmVjdG9yeSBpcyB3aGVyZXZlciB0aGUgZmlsZSBpcyBsb2NhdGVkLCBidXQgCmNvbW1hbmRzIGV4ZWN1dGVkIGluIHRoZSBjb25zb2xlIG1heSBoYXZlIGEgZGlmZmVyZW50IHdvcmtpbmcgZGlyZWN0b3J5LgoKV2Ugd2lsbCB3YW50IHRvIG1ha2UgYSBkaXJlY3RvcnkgZm9yIG91ciBvdXRwdXQgYW5kIHdlIHdpbGwgY2FsbCB0aGlzIGRpcmVjdG9yeTogCmByZXN1bHRzYC4gCkJ1dCBiZWZvcmUgd2UgY3JlYXRlIHRoZSBkaXJlY3RvcnksIHdlIHNob3VsZCBjaGVjayBpZiBpdCBhbHJlYWR5IGV4aXN0cy4gCldlIHdpbGwgc2hvdyB0d28gd2F5cyB0aGF0IHdlIGNhbiBkbyB0aGlzLiAKCkZpcnN0LCB3ZSBjYW4gdXNlIHRoZSBgZGlyKClgIGZ1bmN0aW9uIHRvIGhhdmUgUiBsaXN0IHRoZSBmaWxlcyBpbiBvdXIgd29ya2luZyAKZGlyZWN0b3J5LiAKCmBgYHtyfQojIExldCdzIGNoZWNrIHdoYXQgZmlsZXMgYXJlIGhlcmUKZGlyKCkKYGBgCgpUaGlzIHNob3dzIHVzIHRoZXJlIGlzIG5vIGZvbGRlciBjYWxsZWQgInJlc3VsdHMiIHlldC4gCgpJZiB3ZSB3YW50IHRvIG1vcmUgcG9pbnRlZGx5IGxvb2sgZm9yICJyZXN1bHRzIiBpbiBvdXIgd29ya2luZyBkaXJlY3Rvcnkgd2UgY2FuIAp1c2UgdGhlIGBkaXIuZXhpc3RzKClgIGZ1bmN0aW9uLgoKYGBge3IgY2hlY2stZGlyLCBsaXZlID0gVFJVRX0KIyBDaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzCmRpci5leGlzdHMoInJlc3VsdHMiKQpgYGAKCklmIHRoZSBhYm92ZSBzYXlzIGBGQUxTRWAgdGhhdCBtZWFucyB3ZSB3aWxsIG5lZWQgdG8gY3JlYXRlIGEgYHJlc3VsdHNgIApkaXJlY3RvcnkgdXNpbmcgdGhlIGZ1bmN0aW9uIGBkaXIuY3JlYXRlKClgLgoKYGBge3IgY3JlYXRlLWRpciwgbGl2ZSA9IFRSVUV9CiMgTWFrZSBhIGRpcmVjdG9yeSB3aXRoaW4gdGhlIHdvcmtpbmcgZGlyZWN0b3J5IGNhbGxlZCAncmVzdWx0cycKZGlyLmNyZWF0ZSgicmVzdWx0cyIpCmBgYAoKQWZ0ZXIgY3JlYXRpbmcgdGhlIHJlc3VsdHMgZGlyZWN0b3J5IGFib3ZlLCBsZXQncyByZS1ydW4gYGRpci5leGlzdHMoKWAgdG8gc2VlIAppZiBub3cgaXQgZXhpc3RzLgoKYGBge3IgY2hlY2stZGlyLWFnYWluLCBsaXZlID0gVFJVRX0KIyBSZS1jaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzCmRpci5leGlzdHMoInJlc3VsdHMiKQpgYGAKCldlIGNhbiB1c2UgdGhlIG91dHB1dCBvZiBgZGlyLmV4aXN0cygpYCB0byBhdXRvbWF0aWNhbGx5IGNyZWF0ZSBvciBob2xkIG9mZiBvbiAKY3JlYXRpbmcgYSBkaXJlY3RvcnkgYnkgcHV0dGluZyB0aGlzIHRvZ2V0aGVyIGluIGFuIGBpZmAgc3RhdGVtZW50IGxpa2UgYmVsb3cuIApBbiBgaWZgIHN0YXRlbWVudCBoYXMgdHdvIG1haW4gcGFydHM6CkZpcnN0LCB0aGUgdGVzdCwgd2hpY2ggaXMgYW4gZXhwcmVzc2lvbiB0aGF0IHdpbGwgcmVzdWx0IGluIGVpdGhlciBgVFJVRWAgb3IgYEZBTFNFYC4KVGhpcyBpcyBwdXQgaW4gcGFyZW50aGVzaXMgaW1tZWRpYXRlbHkgYWZ0ZXIgdGhlIGBpZmAuClRoZSBuZXh0IHBhcnQgaXMgdGhlIGJvZHksIHdoaWNoIGlzIHRoZSBjb21tYW5kcyB0aGF0IHdpbGwgYmUgZXhlY3V0ZWQgKmlmKiB0aGUgCnRlc3QgaXMgYFRSVUVgLgpUaGVzZSBhcmUgcGxhY2VkIHdpdGhpbiBhIHNldCBvZiBicmFjZXMgYHsgfWAuCk5vdGUgdGhhdCB3ZSB1c2VkIGFuIGV4Y2xhbWF0aW9uIHBvaW50IGluIHRoZSB0ZXN0IHRvIHNpZ25pZnkgdGhhdCB3ZSB3YW50IGEgCmRpcmVjdG9yeSB0byBiZSBjcmVhdGVkIG9ubHkgKmlmKiBgZGlyLmV4aXN0cyhyZXN1bHRzKWAgaXMgTk9UIGVxdWFsIHRvIGBUUlVFYC4KCmBgYHtyIGNyZWF0ZS1pZn0KIyBJZiAncmVzdWx0cycgZGlyZWN0b3J5IGRvZXNuJ3QgZXhpc3QuLi4KaWYgKCFkaXIuZXhpc3RzKCJyZXN1bHRzIikpIHsKICAjIC4uLiBjcmVhdGUgYSAncmVzdWx0cycgZGlyZWN0b3J5CiAgZGlyLmNyZWF0ZSgicmVzdWx0cyIpCn0KYGBgCgpUaGUgYGRpci5leGlzdHMoKWAgZnVuY3Rpb24gd2lsbCBub3Qgd29yayBvbiBmaWxlcyB0aGVtc2VsdmVzLgpJbiB0aGF0IGNhc2UsIHRoZXJlIGlzIGFuIGFuYWxvZ291cyBmdW5jdGlvbiBjYWxsZWQgYGZpbGUuZXhpc3RzKClgLgoKVHJ5IHVzaW5nIHRoZSBgZmlsZS5leGlzdHMoKWAgZnVuY3Rpb24gdG8gc2VlIGlmIHRoZSBmaWxlIApgZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdmAgZXhpc3RzIGluIHRoZSBjdXJyZW50IGRpcmVjdG9yeS4KVXNlIHRoZSBjb2RlIGNodW5rIHdlIHNldCB1cCBmb3IgeW91IGJlbG93LiAKTm90ZSB0aGF0IGluIG91ciBub3RlYm9va3MgKGFuZCBzb21ldGltZXMgZWxzZXdoZXJlKSwgd2hlcmV2ZXIgeW91IHNlZSBhIApgPEZJTExfSU5fVEhFX0JMQU5LPmAgbGlrZSBpbiB0aGUgY2h1bmsgYmVsb3csIHRoYXQgaXMgbWVhbnQgZm9yIHlvdSB0byByZXBsYWNlIAooaW5jbHVkaW5nIHRoZSBhbmdsZSBicmFja2V0cykgd2l0aCB0aGUgY29ycmVjdCBwaHJhc2UgYmVmb3JlIHlvdSBydW4gdGhlIGNodW5rIAoob3RoZXJ3aXNlIHlvdSB3aWxsIGdldCBhbiBlcnJvcikuCgpgYGB7ciBmaWxlLWNoZWNrLCBldmFsPUZBTFNFfQojIFJlcGxhY2UgdGhlIDxQVVRfRklMRV9OQU1FX0hFUkU+IHdpdGggdGhlIG5hbWUgb2YgdGhlIGZpbGUgeW91IGFyZSBsb29raW5nIGZvcgojIFJlbWVtYmVyIHRvIHVzZSBxdW90ZXMgdG8gbWFrZSBpdCBhIGNoYXJhY3RlciBzdHJpbmcKZmlsZS5leGlzdHMoPFBVVF9GSUxFX05BTUVfSEVSRT4pCmBgYAoKTm93IHRoYXQgd2UndmUgZGV0ZXJtaW5lZCB0aGF0IGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YCBleGlzdHMsIHdlIGFyZSByZWFkeSAKdG8gcmVhZCBpdCBpbnRvIG91ciBSIGVudmlyb25tZW50LgoKIyMjIyBSZWFkIGEgVFNWIGZpbGUKCkRlY2xhcmUgdGhlIG5hbWUgb2YgdGhlIGRpcmVjdG9yeSB3aGVyZSB3ZSB3aWxsIHJlYWQgaW4gdGhlIGRhdGEuIAoKYGBge3J9CmRhdGFfZGlyIDwtICJkYXRhIgpgYGAKCkFsdGhvdWdoIGJhc2UgUiBoYXMgZnVuY3Rpb25zIHRvIHJlYWQgaW4gZGF0YSBmaWxlcywgdGhlIGZ1bmN0aW9ucyBpbiB0aGUgCmByZWFkcmAgcGFja2FnZSAocGFydCBvZiB0aGUgdGlkeXZlcnNlKSBhcmUgZmFzdGVyIGFuZCBtb3JlIHN0cmFpZ2h0Zm9yd2FyZCAKdG8gdXNlIHNvIHdlIGFyZSBnb2luZyB0byB1c2UgdGhvc2UgaGVyZS4gCkJlY2F1c2UgdGhlIGZpbGUgd2UgYXJlIHJlYWRpbmcgaW4gaXMgYSBUU1YgKHRhYiBzZXBhcmF0ZWQgdmFsdWVzKSBmaWxlIHdlIHdpbGwgCmJlIHVzaW5nIHRoZSBgcmVhZF90c3ZgIGZ1bmN0aW9uLiAKVGhlcmUgYXJlIGFuYWxvZ291cyBmdW5jdGlvbnMgZm9yIENTViAoY29tbWEgc2VwYXJhdGVkIHZhbHVlcykgZmlsZXMgCihgcmVhZF9jc3YoKWApIGFuZCBvdGhlciBmaWxlcyB0eXBlcy4KCiMjIFJlYWQgaW4gdGhlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIHJlc3VsdHMgZmlsZQoKYGBge3IgcmVhZC1yZXN1bHRzfQpzdGF0c19kZiA8LSByZWFkcjo6cmVhZF90c3YoCiAgZmlsZS5wYXRoKGRhdGFfZGlyLAogICAgICAgICAgICAiZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdiIpCiAgKQpgYGAKCkZvbGxvd2luZyB0aGUgdGVtcGxhdGUgb2YgdGhlIHByZXZpb3VzIGNodW5rLCB1c2UgdGhpcyBjaHVuayB0byByZWFkIGluIHRoZSBmaWxlCmBHU0U0NDk3MS50c3ZgIHRoYXQgaXMgaW4gdGhlIGBkYXRhYCBmb2xkZXIgYW5kIHNhdmUgaXQgaW4gdGhlIHZhcmlhYmxlIGBnZW5lX2RmYC4gCgpgYGB7ciByZWFkLWV4cHIsIGxpdmUgPSBUUlVFfQojIFVzZSB0aGlzIGNodW5rIHRvIHJlYWQgaW4gZGF0YSBmcm9tIHRoZSBmaWxlIGBHU0U0NDk3MS50c3ZgCmdlbmVfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KAogIGZpbGUucGF0aChkYXRhX2RpciwKICAgICAgICAgICAgIkdTRTQ0OTcxLnRzdiIpCiAgKQpgYGAKClVzZSB0aGlzIGNodW5rIHRvIGV4cGxvcmUgd2hhdCBgZ2VuZV9kZmAgbG9va3MgbGlrZS4gCgpgYGB7ciBleHBsb3JlfQojIEV4cGxvcmUgYGdlbmVfZGZgCgpgYGAKCldoYXQgaW5mb3JtYXRpb24gaXMgY29udGFpbmVkIGluIGBnZW5lX2RmYD8KCiMjIGRwbHlyIHBpcGVzCgpPbmUgbmlmdHkgZmVhdHVyZSBvZiB0aGUgdGlkeXZlcnNlIGlzIHBpcGVzOiBgJT4lYApUaGVzZSBoYW5keSB0aGluZ3MgYWxsb3dzIHlvdSB0byBmdW5uZWwgdGhlIHJlc3VsdCBvZiBvbmUgZXhwcmVzc2lvbiB0byB0aGUgbmV4dCwKbWFraW5nIHlvdXIgY29kZSBhIGxpdHRsZSBtb3JlIHN0cmVhbWxpbmVkLgoKRm9yIGV4YW1wbGUsIHRoZSBvdXRwdXQgZnJvbSB0aGlzOiAgCgpgYGB7ciBmaWx0ZXJ9CmZpbHRlcihzdGF0c19kZiwgY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIikKYGBgICAKCi4uLmlzIHRoZSBzYW1lIGFzIHRoZSBvdXRwdXQgZnJvbSB0aGlzOiAgCgpgYGB7ciBmaWx0ZXItcGlwZX0Kc3RhdHNfZGYgJT4lIGZpbHRlcihjb250cmFzdCA9PSAibWFsZV9mZW1hbGUiKQpgYGAgIAogIApUaGlzIGNhbiBtYWtlIHlvdXIgY29kZSBjbGVhbmVyIGFuZCBlYXNpZXIgdG8gZm9sbG93IGEgc2VyaWVzIG9mIHJlbGF0ZWQgCmNvbW1hbmRzLiAKTGV0J3MgbG9vayBhdCBhbiBleGFtcGxlIHdpdGggb3VyIHN0YXRzIG9mIG9mIGhvdyB0aGUgc2FtZSAKZnVuY3Rpb25zIGxvb2sgd2l0aCBvciB3aXRob3V0IHBpcGVzOgoKKkV4YW1wbGUgMToqIHdpdGhvdXQgcGlwZXM6IAoKYGBge3Igc3RlcHMtbm9waXBlfQpzdGF0c19hcnJhbmdlZCA8LSBhcnJhbmdlKHN0YXRzX2RmLCB0X3N0YXRpc3RpYykKc3RhdHNfZmlsdGVyZWQgPC0gZmlsdGVyKHN0YXRzX2FycmFuZ2VkLCBhdmdfZXhwcmVzc2lvbiA+IDUwKQpzdGF0c19ub3BpcGUgPC0gc2VsZWN0KHN0YXRzX2ZpbHRlcmVkLCBjb250cmFzdCwgbG9nX2ZvbGRfY2hhbmdlLCBwX3ZhbHVlKQpgYGAKICAKVUdILCB3ZSBoYXZlIHRvIGtlZXAgdHJhY2sgb2YgYWxsIG9mIHRob3NlIGRpZmZlcmVudCBpbnRlcm1lZGlhdGUgZGF0YSBmcmFtZXMgCmFuZCB0eXBlIHRoZWlyIG5hbWVzIHNvIG1hbnkgdGltZXMgaGVyZSEgCldlIGNvdWxkIG1heWJlIHN0cmVhbWxpbmUgdGhpbmdzIGJ5IHVzaW5nIHRoZSBzYW1lIHZhcmlhYmxlIG5hbWUgYXQgZWFjaCBzdGFnZSwgCmJ1dCBldmVuIHRoZW4gdGhlcmUgaXMgYSBsb3Qgb2YgZXh0cmEgdHlwaW5nLCBhbmQgaXQgaXMgZWFzeSB0byBnZXQgY29uZnVzZWQgCmFib3V0IHdoYXQgaGFzIGJlZW4gZG9uZSB3aGVyZS4KSXQncyBhbm5veWluZyBhbmQgbWFrZXMgaXQgaGFyZGVyIGZvciBwZW9wbGUgdG8gcmVhZC4gCiAgCipFeGFtcGxlIDI6KiBTYW1lIHJlc3VsdCBhcyAxIGJ1dCB3aXRoIHBpcGVzIQoKYGBge3Igc3RlcHMtcGlwZSwgbGl2ZSA9IFRSVUV9CiMgRXhhbXBsZSBvZiB0aGUgc2FtZSBtb2RpZmljYXRpb25zIGFzIGFib3ZlIGJ1dCB3aXRoIHBpcGVzIQpzdGF0c19waXBlICA8LSBzdGF0c19kZiAlPiUKICAgICAgICAgICAgICAgYXJyYW5nZSh0X3N0YXRpc3RpYykgJT4lCiAgICAgICAgICAgICAgIGZpbHRlcihhdmdfZXhwcmVzc2lvbiA+IDUwKSAlPiUKICAgICAgICAgICAgICAgc2VsZWN0KGNvbnRyYXN0LCBsb2dfZm9sZF9jaGFuZ2UsIHBfdmFsdWUpCmBgYAoKV2hhdCB0aGUgYCU+JWAgKHBpcGUpIGlzIGRvaW5nIGhlcmUgaXMgZmVlZGluZyB0aGUgcmVzdWx0IG9mIHRoZSBleHByZXNzaW9uIG9uIAppdHMgbGVmdCBpbnRvIHRoZSBmaXJzdCBhcmd1bWVudCBvZiB0aGUgbmV4dCBmdW5jdGlvbiAodG8gaXRzIHJpZ2h0LCBvciBvbiB0aGUgCm5leHQgbGluZSBoZXJlKS4gCldlIGNhbiB0aGVuIHNraXAgdGhhdCBmaXJzdCBhcmd1bWVudCAodGhlIGRhdGEgaW4gdGhlc2UgY2FzZXMpLCBhbmQgbW92ZSByaWdodCAKb24gdG8gdGhlIHBhcnQgd2UgY2FyZSBhYm91dCBhdCB0aGF0IHN0ZXA6IHdoYXQgd2UgYXJlIGFycmFuZ2luZywgZmlsdGVyaW5nLCBvciAKc2VsZWN0aW5nICBpbiB0aGlzIGNhc2UuCgpMZXQncyBkb3VibGUgY2hlY2sgdGhhdCB0aGVzZSBhcmUgdGhlIHNhbWUgYnkgdXNpbmcgdGhlIGZ1bmN0aW9uLCBgYWxsLmVxdWFsKClgLiAKCmBgYHtyIGNoZWNrLXBpcGV9CmFsbC5lcXVhbChzdGF0c19ub3BpcGUsIHN0YXRzX3BpcGUpCmBgYAoKYGFsbC5lcXVhbCgpYCBpcyBsZXR0aW5nIHVzIGtub3cgdGhhdCB0aGVzZSB0d28gb2JqZWN0cyBhcmUgdGhlIHNhbWUuIAoKTm93IHRoYXQgaG9wZWZ1bGx5IHlvdSBhcmUgY29udmluY2VkIHRoYXQgdGhlIHRpZHl2ZXJzZSBjYW4gaGVscCB5b3UgbWFrZSB5b3VyIApjb2RlIG5lYXRlciBhbmQgZWFzaWVyIHRvIHVzZSBhbmQgcmVhZCwgbGV0J3MgZ28gdGhyb3VnaCBzb21lIG9mIHRoZSBwb3B1bGFyIAp0aWR5dmVyc2UgZnVuY3Rpb25zIGFuZCBzbyB3ZSBjYW4gY3JlYXRlIHBpcGVsaW5lcyBsaWtlIHRoaXMuIAoKIyMgQ29tbW9uIHRpZHl2ZXJzZSBmdW5jdGlvbnMKCkxldCdzIHNheSB3ZSB3YW50ZWQgdG8gZmlsdGVyIHRoaXMgZ2VuZSBleHByZXNzaW9uIGRhdGFzZXQgdG8gcGFydGljdWxhciBzYW1wbGUKZ3JvdXBzLgpJbiBvcmRlciB0byBkbyB0aGlzLCB3ZSB3b3VsZCB1c2UgdGhlIGZ1bmN0aW9uIGBmaWx0ZXIoKWAgYXMgd2VsbCBhcyBhIGxvZ2ljIApzdGF0ZW1lbnQgKHVzdWFsbHkgb25lIHRoYXQgcmVmZXJzIHRvIGEgY29sdW1uIG9yIGNvbHVtbnMgaW4gdGhlIGRhdGEgZnJhbWUpLgoKYGBge3IgZmlsdGVyLWdlbmV9CiMgSGVyZSBsZXQncyBmaWx0ZXIgc3RhdHNfZGYgdG8gdGhlIGdlbmVfc3ltYm9sICJTTkNBIgpzdGF0c19kZiAlPiUgCiAgZmlsdGVyKGdlbmVfc3ltYm9sID09ICJTTkNBIikKYGBgCgpXZSBjYW4gdXNlIGBmaWx0ZXIoKWAgc2ltaWxhcmx5IGZvciBudW1lcmljIHN0YXRlbWVudHMuICAKCmBgYHtyIGZpbHRlci1udW1lcmljLCBsaXZlID0gVFJVRX0KIyBIZXJlIGxldCdzIGZpbHRlciB0aGUgZGF0YSB0byByb3dzIHdpdGggYXZlcmFnZSBleHByZXNzaW9uIHZhbHVlcyBhYm92ZSA1MApzdGF0c19kZiAlPiUKICBmaWx0ZXIoYXZnX2V4cHJlc3Npb24gPiA1MCkKYGBgCgpXZSBjYW4gYXBwbHkgbXVsdGlwbGUgZmlsdGVycyBhdCBvbmNlLCB3aGljaCB3aWxsIHJlcXVpcmUgYWxsIG9mIHRoZW0gdG8gYmUgCnNhdGlzZmllZCBmb3IgZXZlcnkgcm93IGluIHRoZSByZXN1bHRzOgoKYGBge3IgZmlsdGVyLTIsIGxpdmUgPSBUUlVFfQojIGZpbHRlciB0byBoaWdobHkgZXhwcmVzc2VkIGdlbmVzIHdpdGggY29udHJhc3QgIm1hbGVfZmVtYWxlIgpzdGF0c19kZiAlPiUKICBmaWx0ZXIoY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIiwgCiAgICAgICAgIGF2Z19leHByZXNzaW9uID4gNTApCmBgYAoKV2hlbiB3ZSBhcmUgZmlsdGVyaW5nLCB0aGUgYCVpbiVgIG9wZXJhdG9yIGNhbiBjb21lIGluIGhhbmR5IGlmIHdlIGhhdmUgbXVsdGlwbGUKaXRlbXMgd2Ugd291bGQgbGlrZSB0byBtYXRjaC4KTGV0J3MgdGFrZSBhIGxvb2sgYXQgd2hhdCB1c2luZyBgJWluJWAgZG9lcy4KCmBgYHtyIGluLWV4YW1wbGUsIGV2YWwgPSBGQUxTRX0KZ2VuZXNfb2ZfaW50ZXJlc3QgPC0gYygiU05DQSIsICJDREtOMUEiKQpzdGF0c19kZiRnZW5lX3N5bWJvbCAlaW4lIGdlbmVzX29mX2ludGVyZXN0CmBgYAoKYCVpbiVgIHJldHVybnMgYSBsb2dpY2FsIHZlY3RvciB0aGF0IG5vdyB3ZSBjYW4gdXNlIGluIGBkcGx5cjo6ZmlsdGVyYC4KCmBgYHtyIGZpbHRlci1pbiwgbGl2ZSA9IFRSVUV9CiMgZmlsdGVyIHRvIGdlbmVzIG9mIGludGVyZXN0CnN0YXRzX2RmICU+JSAKICBmaWx0ZXIoZ2VuZV9zeW1ib2wgJWluJSBjKCJTTkNBIiwgIkNES04xQSIpKQpgYGAKCkxldCdzIHJldHVybiB0byBvdXIgZmlyc3QgYGZpbHRlcigpYCBhbmQgYnVpbGQgb24gdG8gaXQuIApUaGlzIHRpbWUsIGxldCdzIGtlZXAgb25seSBzb21lIG9mIHRoZSBjb2x1bW5zIGZyb20gdGhlIGRhdGEgZnJhbWUgdXNpbmcgdGhlIApgc2VsZWN0KClgIGZ1bmN0aW9uLiAKTGV0J3MgYWxzbyBzYXZlIHRoaXMgYXMgYSBuZXcgZGF0YSBmcmFtZSBjYWxsZWQgYHN0YXRzX2ZpbHRlcmVkX2RmYC4KCmBgYHtyIGZpbHRlci1zZWxlY3QsIGxpdmUgPSBUUlVFfQojIGZpbHRlciB0byBoaWdobHkgZXhwcmVzc2VkICJtYWxlX2ZlbWFsZSIKIyBhbmQgc2VsZWN0IGdlbmVfc3ltYm9sLCBsb2dfZm9sZF9jaGFuZ2UgYW5kIHRfc3RhdGlzdGljCnN0YXRzX2ZpbHRlcmVkX2RmIDwtIHN0YXRzX2RmICU+JQogIGZpbHRlcihjb250cmFzdCA9PSAibWFsZV9mZW1hbGUiLCAKICAgICAgICAgYXZnX2V4cHJlc3Npb24gPiA1MCkgJT4lCiAgc2VsZWN0KGxvZ19mb2xkX2NoYW5nZSwgdF9zdGF0aXN0aWMpCmBgYAoKTGV0J3Mgc2F5IHdlIHdhbnRlZCB0byBhcnJhbmdlIHRoaXMgZGF0YXNldCBzbyB0aGF0IHRoZSBnZW5lcyBhcmUgYXJyYW5nZWQgYnkgCnRoZSBzbWFsbGVzdCBwIHZhbHVlcyB0byB0aGUgbGFyZ2VzdC4KSW4gb3JkZXIgdG8gZG8gdGhpcywgd2Ugd291bGQgdXNlIHRoZSBmdW5jdGlvbiBgYXJyYW5nZSgpYCBhcyB3ZWxsIGFzIHRoZSBjb2x1bW4Kd2Ugd291bGQgbGlrZSB0byBzb3J0IGJ5IChpbiB0aGlzIGNhc2UgYHBfdmFsdWVgKS4KCmBgYHtyIGFycmFuZ2V9CnN0YXRzX2RmICU+JSAKICBhcnJhbmdlKHBfdmFsdWUpIApgYGAKCldoYXQgaWYgd2Ugd2FudCB0byBzb3J0IGZyb20gbGFyZ2VzdCB0byBzbWFsbGVzdD8gCkxpa2UgaWYgd2Ugd2FudCB0byBzZWUgdGhlIGdlbmVzIHdpdGggdGhlIGhpZ2hlc3QgYXZlcmFnZSBleHByZXNzaW9uPwpXZSBjYW4gdXNlIHRoZSBzYW1lIGZ1bmN0aW9uLCBidXQgaW5zdGVhZCB1c2UgdGhlIGBkZXNjKClgIGZ1bmN0aW9uIGFuZCBub3cgd2UgCmFyZSB1c2luZyBgYXZnX2V4cHJlc3Npb25gIGNvbHVtbi4gCgpgYGB7ciBhcnJhbmdlLWRlc2N9CiMgYXJyYW5nZSBkZXNjZW5kaW5nIGJ5IGF2Z19leHByZXNzaW9uCnN0YXRzX2RmICU+JQogIGFycmFuZ2UoZGVzYyhhdmdfZXhwcmVzc2lvbikpCmBgYCAKCldoYXQgaWYgd2Ugd291bGQgbGlrZSB0byBjcmVhdGUgYSBuZXcgY29sdW1uIG9mIHZhbHVlcz8KRm9yIHRoYXQgd2UgdXNlIGBtdXRhdGUoKWAgZnVuY3Rpb24uCgpgYGB7ciBtdXRhdGV9CnN0YXRzX2RmICU+JSAKICBtdXRhdGUobG9nMTBfcF92YWx1ZSA9IC1sb2cxMChwX3ZhbHVlKSkKYGBgCgpXaGF0IGlmIHdlIHdhbnQgdG8gb2J0YWluIHN1bW1hcnkgc3RhdGlzdGljcyBmb3IgYSBjb2x1bW4gb3IgY29sdW1ucz8KVGhlIGBzdW1tYXJpemVgIGZ1bmN0aW9uIGFsbG93cyB1cyB0byBjYWxjdWxhdGUgc3VtbWFyeSBzdGF0aXN0aWNzIGZvciBhIGNvbHVtbi4gCkhlcmUgd2Ugd2lsbCB1c2Ugc3VtbWFyaXplIHRvIG9idGFpbiBhbiBtZWFuIGxvZyBmb2xkZXIgY2hhbmdlIG92ZXIgYWxsIHRoZQpnZW5lcywgYW5kIGl0cyBzdGFuZGFyZCBkZXZpYXRpb24uCgpgYGB7ciBzdW1tYXJpemV9CnN0YXRzX2RmICU+JSAKICBzdW1tYXJpemUobWVhbihsb2dfZm9sZF9jaGFuZ2UpLAogICAgICAgICAgICBzZChsb2dfZm9sZF9jaGFuZ2UpKQpgYGAKCldoYXQgaWYgd2UnZCBsaWtlIHRvIG9idGFpbiBhIHN1bW1hcnkgc3RhdGlzdGljcyBidXQgaGF2ZSB0aGVtIGZvciB2YXJpb3VzIApncm91cHM/CkNvbnZlbmllbnRseSBuYW1lZCwgdGhlcmUncyBhIGZ1bmN0aW9uIGNhbGxlZCBgZ3JvdXBfYnkoKWAgdGhhdCBzZWFtbGVzc2x5IAphbGxvd3MgdXMgdG8gZG8gdGhpcy4gCkFsc28gbm90ZSB0aGF0IGBncm91cF9ieSgpYCBhbGxvd3MgdXMgdG8gZ3JvdXAgYnkgbXVsdGlwbGUgdmFyaWFibGVzIGF0IGEgdGltZSAKaWYgeW91IHdhbnQgdG8uCgpgYGB7ciBzdW1tYXJpemUtZ3JvdXBzLCBsaXZlID0gVFJVRX0Kc3RhdHNfc3VtbWFyeV9kZiA8LSBzdGF0c19kZiAlPiUKICAgICAgZ3JvdXBfYnkoY29udHJhc3QpICU+JSAKICAgICAgc3VtbWFyaXplKG1lYW4obG9nX2ZvbGRfY2hhbmdlKSwKICAgICAgICAgICAgICAgIHNkKGxvZ19mb2xkX2NoYW5nZSkpCmBgYAoKTGV0J3MgbG9vayBhdCBhIHByZXZpZXcgb2Ygd2hhdCB3ZSBtYWRlOgoKYGBge3J9CnN0YXRzX3N1bW1hcnlfZGYKYGBgCgpIZXJlIHdlIGhhdmUgdGhlIG1lYW4gbG9nIGZvbGQgY2hhbmdlIGV4cHJlc3Npb24gcGVyIGVhY2ggY29udHJhc3Qgd2UgbWFkZS4gCgojIyBBIGJyaWVmIGludHJvIHRvIHRoZSBgYXBwbHlgIGZhbWlseSBvZiBmdW5jdGlvbnMKCkluIGJhc2UgUiwgdGhlIGBhcHBseWAgZmFtaWx5IG9mIGZ1bmN0aW9ucyBjYW4gYmUgYW4gYWx0ZXJuYXRpdmUgbWV0aG9kcyBmb3IgCnBlcmZvcm1pbmcgdHJhbnNmb3JtYXRpb25zIGFjcm9zcyBhIGRhdGEgZnJhbWUsIG1hdHJpeCBvciBvdGhlciBvYmplY3Qgc3RydWN0dXJlcy4gCgpPbmUgb2YgdGhpcyBmYW1pbHkgaXMgKHNob2NraW5nbHkpIHRoZSBmdW5jdGlvbiBgYXBwbHkoKWAsIHdoaWNoIG9wZXJhdGVzIG9uIAptYXRyaWNlcy4KCkEgbWF0cml4IGlzIHNpbWlsYXIgdG8gYSBkYXRhIGZyYW1lIGluIHRoYXQgaXQgaXMgYSByZWN0YW5ndWxhciB0YWJsZSBvZiBkYXRhLApidXQgaXQgaGFzIGFuIGFkZGl0aW9uYWwgY29uc3RyYWludDogCnJhdGhlciB0aGFuIGVhY2ggY29sdW1uIGhhdmluZyBhIHR5cGUsIEFMTCBkYXRhIGluIGEgbWF0cml4IGhhcyB0aGUgc2FtZSB0eXBlLgoKVGhlIGZpcnN0IGFyZ3VtZW50IHRvIGBhcHBseSgpYCBpcyB0aGUgZGF0YSBvYmplY3Qgd2Ugd2FudCB0byB3b3JrIG9uLgpUaGUgdGhpcmQgYXJndW1lbnQgaXMgdGhlIGZ1bmN0aW9uIHdlIHdpbGwgYXBwbHkgdG8gZWFjaCByb3cgb3IgY29sdW1uIG9mIHRoZSAKZGF0YSBvYmplY3QuClRoZSBzZWNvbmQgYXJndW1lbnQgaW4gc3BlY2lmaWVzIHdoZXRoZXIgd2UgYXJlIGFwcGx5aW5nIHRoZSBmdW5jdGlvbiAKYWNyb3NzIHJvd3Mgb3IgYWNyb3NzIGNvbHVtbnMgKDEgZm9yIHJvd3MsIDIgZm9yIGNvbHVtbnMpLgoKUmVtZW1iZXIgdGhhdCBgZ2VuZV9kZmAgaXMgYSBnZW5lIHggc2FtcGxlIGdlbmUgZXhwcmVzc2lvbiBkYXRhIGZyYW1lIHRoYXQgaGFzCmNvbHVtbnMgb2YgdHdvIGRpZmZlcmVudCB0eXBlcywgY2hhcmFjdGVyIGFuZCBudW1lcmljLgpDb252ZXJ0aW5nIGl0IHRvIGEgbWF0cml4IHdpbGwgcmVxdWlyZSB1cyB0byBtYWtlIHRoZW0gYWxsIHRoZSBzYW1lIHR5cGUuCldlIGNhbiBjb2VyY2UgaXQgaW50byBhIG1hdHJpeCB1c2luZyBgYXMubWF0cml4KClgLCBpbiB3aGljaCBjYXNlIFIgd2lsbApwaWNrIGEgdHlwZSB0aGF0IGl0IGNhbiBjb252ZXJ0IGV2ZXJ5dGhpbmcgdG8uCldoYXQgZG9lcyBpdCBjaG9vc2U/CgpgYGB7ciBtYXRyaXh9CiMgQ29lcmNlIGBnZW5lX2RmYCBpbnRvIGEgbWF0cml4CmdlbmVfbWF0cml4IDwtIGFzLm1hdHJpeChnZW5lX2RmKQpgYGAKCmBgYHtyIG1hdHJpeC10eXBlLCBsaXZlID0gVFJVRX0KIyBFeHBsb3JlIHRoZSBzdHJ1Y3R1cmUgb2YgdGhlIGBnZW5lX21hdHJpeGAgb2JqZWN0CnN0cihnZW5lX21hdHJpeCkKYGBgCgpXaGlsZSB0aGF0IHdvcmtlZCwgaXQgaXMgcmFyZSB0aGF0IHdlIHdhbnQgbnVtYmVycyBjb252ZXJ0ZWQgdG8gdGV4dCwgc28gd2UgYXJlCmdvaW5nIHRvIHNlbGVjdCBvbmx5IHRoZSBjb2x1bW5zIHdpdGggbnVtZXJpYyB2YWx1ZXMgYmVmb3JlIGNvbnZlcnRpbmcgaXQgdG8gYSBtYXRyaXguCldlIGNhbiBkbyB0aGlzIG1vc3QgZWFzaWx5IGJ5IHJlbW92aW5nIHRoZSBmaXJzdCBjb2x1bW4sIHdoaWNoIGNvbnRhaW5zIHRoZSBnZW5lIG5hbWVzIApzdG9yZWQgYXMgY2hhcmFjdGVyIHZhbHVlcy4KCmBgYHtyIG1hdHJpeC1udW1lcmljLCBsaXZlID0gVFJVRX0KIyBMZXQncyBzYXZlIGEgbmV3IG1hdHJpeCBvYmplY3QgbmFtZXMgYGdlbmVfbnVtX21hdHJpeGAgY29udGFpbmluZyBvbmx5CiMgdGhlIG51bWVyaWMgdmFsdWVzCmdlbmVfbnVtX21hdHJpeCA8LSBhcy5tYXRyaXgoZ2VuZV9kZlssIC0xXSkKCiMgRXhwbG9yZSB0aGUgc3RydWN0dXJlIG9mIHRoZSBgZ2VuZV9udW1fbWF0cml4YCBvYmplY3QKc3RyKGdlbmVfbnVtX21hdHJpeCkKYGBgCgpXaHkgZG8gd2UgaGF2ZSBhIGBbLCAtMV1gIGFmdGVyIGBnZW5lX2RmYCBpbiB0aGUgYWJvdmUgY2h1bms/CgpOb3cgdGhhdCB0aGUgbWF0cml4IGlzIGFsbCBudW1iZXJzLCB3ZSBjYW4gZG8gdGhpbmdzIGxpa2UgY2FsY3VsYXRlIHRoZSBjb2x1bW4Kb3Igcm93IHN0YXRpc3RpY3MgdXNpbmcgYGFwcGx5KClgLgoKYGBge3Igcm93bWVhbnN9CiMgQ2FsY3VsYXRlIHJvdyBtZWFucwpnZW5lX21lYW5zIDwtIGFwcGx5KGdlbmVfbnVtX21hdHJpeCwgMSwgbWVhbikgIyBOb3RpY2Ugd2UgYXJlIHVzaW5nIDEgaGVyZQoKIyBIb3cgbG9uZyB3aWxsIGBnZW5lX21lYW5zYCBiZT8gCmxlbmd0aChnZW5lX21lYW5zKQpgYGAKCk5vdGUgdGhhdCB3ZSBjYW4gb2J0YWluIHRoZSBzYW1lIHJlc3VsdHMgaWYgd2Ugc2VsZWN0IGp1c3QgdGhlIGNvbHVtbnMgd2l0aCBudW1lcmljCnZhbHVlcyBmcm9tIHRoZSBgZ2VuZV9kZmAgZGF0YSBmcmFtZS4KVGhpcyBhbGxvd3MgUiB0byBkbyB0aGUgYXMubWF0cml4KCkgY29lcmNpb24gYXV0b21hdGljYWxseSwgYW5kIGNhbiBiZSBhIGhhbmR5IHNob3J0Y3V0CmlmIHlvdSBoYXZlIGEgKm1vc3RseSogbnVtZXJpYyBkYXRhIGZyYW1lLiAKCmBgYHtyIHJvd21lYW5zLWRhdGFmcmFtZX0KIyBDYWxjdWxhdGUgcm93IG1lYW5zIHVzaW5nIHRoZSBgZ2VuZV9kZmAgb2JqZWN0IGFmdGVyIHJlbW92aW5nIHRoZSBjaGFyYWN0ZXIgY29sdW1uCiMgYXBwbHkoKSBjb252ZXJ0cyB0aGlzIHRvIGEgbWF0cml4IGludGVybmFsbHkKZ2VuZV9tZWFuc19mcm9tX2RmIDwtIGFwcGx5KGdlbmVfZGZbLCAtMV0sIDEsIG1lYW4pIAoKIyBMZXQncyBjaGVjayB0aGF0IHRoZSB0d28gZ2VuZSBtZWFucyBvYmplY3RzIGFyZSBlcXVhbAphbGwuZXF1YWwoZ2VuZV9tZWFucywgZ2VuZV9tZWFuc19mcm9tX2RmKQpgYGAKCk5vdyBsZXQncyBpbnZlc3RpZ2F0ZSB0aGUgc2FtZSBzZXQgdXAsIGJ1dCB1c2UgMiB0byBgYXBwbHlgIG92ZXIgdGhlIGNvbHVtbnMgb2YKb3VyIG1hdHJpeC4KCmBgYHtyIGNvbG1lYW5zfQojIENhbGN1bGF0ZSBzYW1wbGUgbWVhbnMKc2FtcGxlX21lYW5zIDwtIGFwcGx5KGdlbmVfbnVtX21hdHJpeCwgMiwgbWVhbikgIyBOb3RpY2Ugd2UgdXNlIDIgaGVyZQoKIyBIb3cgbG9uZyB3aWxsIGBzYW1wbGVfbWVhbnNgIGJlPyAKbGVuZ3RoKHNhbXBsZV9tZWFucykKYGBgCgpXZSBjYW4gcHV0IHRoZSBnZW5lIG5hbWVzIGJhY2sgaW50byB0aGUgbnVtZXJpYyBtYXRyaXggb2JqZWN0IGJ5CmFzc2lnbmluZyB0aGVtIGFzIHJvd25hbWVzLgoKYGBge3IgbWF0cml4LXJvd25hbWVzLCBsaXZlID0gVFJVRX0KIyBBc3NpZ24gdGhlIGdlbmUgbmFtZXMgZnJvbSBnZW5lX2RmJEdlbmUgdG8gdGhlIGBnZW5lX251bV9tYXRyaXhgIG9iamVjdCB1c2luZwojIHRoZSBgcm93bmFtZXMoKWAgZnVuY3Rpb24Kcm93bmFtZXMoZ2VuZV9udW1fbWF0cml4KSA8LSBnZW5lX2RmJEdlbmUKCiMgRXhwbG9yZSB0aGUgYGdlbmVfbnVtX21hdHJpeGAgb2JqZWN0CmhlYWQoZ2VuZV9udW1fbWF0cml4KQpgYGAKClJvdyBuYW1lcyBsaWtlIHRoaXMgY2FuIGJlIHZlcnkgY29udmVuaWVudCBmb3Iga2VlcGluZyBtYXRyaWNlcyBvcmdhbml6ZWQsIGJ1dApyb3cgbmFtZXMgKGFuZCBjb2x1bW4gbmFtZXMpIGNhbiBiZSBsb3N0IG9yIG1pc29yZGVyZWQgaWYgeW91IGFyZSBub3QgY2FyZWZ1bCwKZXNwZWNpYWxseSBkdXJpbmcgaW5wdXQgYW5kIG91dHB1dCwgc28gdHJlYXQgdGhlbSB3aXRoIGNhcmUuCgpBbHRob3VnaCB0aGUgYGFwcGx5YCBmdW5jdGlvbnMgbWF5IG5vdCBiZSBhcyBlYXN5IHRvIHVzZSBhcyB0aGUgdGlkeXZlcnNlIApmdW5jdGlvbnMsIGZvciBzb21lIGFwcGxpY2F0aW9ucywgYGFwcGx5YCBtZXRob2RzIGNhbiBiZSBiZXR0ZXIgc3VpdGVkLgpJbiB0aGlzIHdvcmtzaG9wLCB3ZSB3aWxsIG5vdCBkZWx2ZSB0b28gZGVlcGx5IGludG8gdGhlIHZhcmlvdXMgb3RoZXIgYXBwbHkgCmZ1bmN0aW9ucyAoYHRhcHBseSgpYCwgYGxhcHBseSgpYCwgZXRjLikgYnV0IHlvdSBjYW4gcmVhZCBtb3JlIGluZm9ybWF0aW9uIGFib3V0IAp0aGVtIFtoZXJlXShodHRwczovL3d3dy5ndXJ1OTkuY29tL3ItYXBwbHktc2FwcGx5LXRhcHBseS5odG1sKS4KCiMjIFRoZSBkcGx5cjo6am9pbiBmdW5jdGlvbnMKCkxldCdzIHNheSB3ZSBoYXZlIGEgc2NlbmFyaW8gd2hlcmUgd2UgaGF2ZSB0d28gZGF0YSBmcmFtZXMgdGhhdCB3ZSB3b3VsZCBsaWtlIHRvIApjb21iaW5lLiAKUmVjYWxsIHRoYXQgYHN0YXRzX2RmYCBhbmQgYGdlbmVfZGZgIGFyZSBkYXRhIGZyYW1lcyB0aGF0IGNvbnRhaW4gaW5mb3JtYXRpb24gCmFib3V0IHNvbWUgb2YgdGhlIHNhbWUgZ2VuZXMuClRoZSBbYGRwbHlyOjpqb2luYCBmYW1pbHkgb2YgZnVuY3Rpb25zXShodHRwczovL2RwbHlyLnRpZHl2ZXJzZS5vcmcvcmVmZXJlbmNlL2pvaW4uaHRtbCkgCmFyZSB1c2VmdWwgZm9yIHZhcmlvdXMgc2NlbmFyaW9zIG9mIGNvbWJpbmluZyBkYXRhIGZyYW1lcy4gCgpGb3Igbm93LCB3ZSB3aWxsIGZvY3VzIG9uIGBpbm5lcl9qb2luKClgLCB3aGljaCB3aWxsIGNvbWJpbmUgZGF0YSBmcmFtZXMgYnkgb25seQprZWVwaW5nIGluZm9ybWF0aW9uIGFib3V0IG1hdGNoaW5nIHJvd3MgdGhhdCBhcmUgaW4gYm90aCBkYXRhIGZyYW1lcy4KV2UgbmVlZCB0byB1c2UgdGhlIGBieWAgYXJndW1lbnQgdG8gZGVzaWduYXRlIHdoYXQgY29sdW1uKHMpIApzaG91bGQgYmUgdXNlZCBhcyBhIGtleSB0byBtYXRjaCB0aGUgZGF0YSBmcmFtZXMuCkluIHRoaXMgY2FzZSB3ZSB3YW50IHRvIG1hdGNoIHRoZSBnZW5lIGluZm9ybWF0aW9uIGJldHdlZW4gdGhlIHR3bywgc28gd2Ugd2lsbCAKc3BlY2lmeSB0aGF0IHdlIHdhbnQgdG8gY29tcGFyZSB2YWx1ZXMgaW4gdGhlIGBlbnNlbWJsX2lkYCBjb2x1bW4gZnJvbSAKYHN0YXRzX2RmYCB0byB0aGUgYEdlbmVgIGNvbHVtbiBmcm9tIGBnZW5lX2RmYC4KCmBgYHtyIGlubmVyLWpvaW59CnN0YXRzX2RmICU+JSAKICBpbm5lcl9qb2luKGdlbmVfZGYsIGJ5ID0gYygnZW5zZW1ibF9pZCcgPSAnR2VuZScpKSAKYGBgCgojIyBTYXZlIGRhdGEgdG8gZmlsZXMKCiMjIyMgU2F2ZSB0byBUU1YgZmlsZXMKCkxldCdzIHdyaXRlIHNvbWUgb2YgdGhlIGRhdGEgZnJhbWVzIHdlIGNyZWF0ZWQgdG8gYSBmaWxlLgpUbyBkbyB0aGlzLCB3ZSBjYW4gdXNlIHRoZSBgcmVhZHJgIGxpYnJhcnkgb2YgYF93cml0ZSgpYCBmdW5jdGlvbnMuIApUaGUgZmlyc3QgYXJndW1lbnQgb2YgYHdyaXRlX3RzdigpYCBpcyB0aGUgZGF0YSB3ZSB3YW50IHRvIHdyaXRlLCBhbmQgdGhlIHNlY29uZCAKYXJndW1lbnQgaXMgYSBjaGFyYWN0ZXIgc3RyaW5nIHRoYXQgZGVzY3JpYmVzIHRoZSBwYXRoIHRvIHRoZSBuZXcgZmlsZSB3ZSB3b3VsZCAKbGlrZSB0byBjcmVhdGUuClJlbWVtYmVyIHRoYXQgd2UgY3JlYXRlZCBhIGByZXN1bHRzYCBkaXJlY3RvcnkgdG8gcHV0IG91ciBvdXRwdXQgaW4sIApidXQgaWYgd2Ugd2FudCB0byBzYXZlIG91ciBkYXRhIHRvIGEgZGlyZWN0b3J5IG90aGVyIHRoYW4gb3VyIHdvcmtpbmcgZGlyZWN0b3J5LCAKd2UgbmVlZCB0byBzcGVjaWZ5IHRoaXMuIApUaGlzIGlzIHdoYXQgd2Ugd2lsbCB1c2UgdGhlIGBmaWxlLnBhdGgoKWAgZnVuY3Rpb24gZm9yLiAKTGV0J3MgbG9vayBpbiBhIGJpdCBtb3JlIGRldGFpbCB3aGF0IGBmaWxlLnBhdGgoKWAgZG9lcywgYnkgZXhhbWluaW5nIHRoZSAKcmVzdWx0cyBvZiB0aGUgZnVuY3Rpb24gaW4gdGhlIGV4YW1wbGVzIGJlbG93LgoKYGBge3IgZmlsZS1wYXRoLXF1aXp9CiMgV2hpY2ggb2YgdGhlc2UgZmlsZSBwYXRocyBpcyB3aGF0IHdlIHdhbnQgdG8gdXNlIHRvIHNhdmUgb3VyIGRhdGEgdG8gdGhlCiMgcmVzdWx0cyBkaXJlY3Rvcnkgd2UgY3JlYXRlZCBhdCB0aGUgYmVnaW5uaW5nIG9mIHRoaXMgbm90ZWJvb2s/CmZpbGUucGF0aCgiZG9ja2VyLWluc3RhbGwiLCAic3RhdHNfc3VtbWFyeS50c3YiKQpmaWxlLnBhdGgoInJlc3VsdHMiLCAic3RhdHNfc3VtbWFyeS50c3YiKQpmaWxlLnBhdGgoInN0YXRzX3N1bW1hcnkudHN2IiwgInJlc3VsdHMiKQpgYGAKClJlcGxhY2UgYDxORVdfRklMRV9QQVRIPmAgYmVsb3cgd2l0aCB0aGUgYGZpbGUucGF0aCgpYCBzdGF0ZW1lbnQgZnJvbSBhYm92ZSB0aGF0IAp3aWxsIHN1Y2Nlc3NmdWxseSBzYXZlIG91ciBmaWxlIHRvIHRoZSBgcmVzdWx0c2AgZm9sZGVyIAoKYGBge3IgZXZhbD1GQUxTRX0KIyBXcml0ZSBvdXIgZGF0YSBmcmFtZSB0byBhIFRTViBmaWxlCnJlYWRyOjp3cml0ZV90c3Yoc3RhdHNfc3VtbWFyeV9kZiwgPE5FV19GSUxFX1BBVEg+KQpgYGAKCkNoZWNrIGluIHlvdXIgYHJlc3VsdHNgIGRpcmVjdG9yeSB0byBzZWUgaWYgeW91ciBuZXcgZmlsZSBoYXMgc3VjY2Vzc2Z1bGx5IHNhdmVkLgoKIyMjIyBTYXZlIHRvIFJEUyBmaWxlcwoKRm9yIHRoaXMgZXhhbXBsZSB3ZSBoYXZlIGJlZW4gd29ya2luZyB3aXRoIGRhdGEgZnJhbWVzLCB3aGljaCBhcmUgY29udmVuaWVudGx5IApyZXByZXNlbnRlZCBhcyBUU1Ygb3IgQ1NWIHRhYmxlcy4gCkhvd2V2ZXIsIGluIG90aGVyIHNpdHVhdGlvbnMgd2UgbWF5IHdhbnQgdG8gc2F2ZSBtb3JlIGNvbXBsaWNhdGVkIG9yIHZlcnkgbGFyZ2UgCmRhdGEgc3RydWN0dXJlcywgUkRTIChSIERhdGEgU2VyaWFsaXplZC9TaW5nbGUpIGZpbGVzIG1heSBiZSBhIGJldHRlciBvcHRpb24gZm9yCnNhdmluZyBvdXIgZGF0YS4KUkRTIGlzIFIncyBzcGVjaWFsIGZpbGUgZm9ybWF0IGZvciBob2xkaW5nIGRhdGEgZXhhY3RseSBhcyB5b3UgaGF2ZSBpdCBpbiB5b3VyIApSIGVudmlyb25tZW50LiAKUkRTIGZpbGVzIGNhbiBhbHNvIGJlIGNvbXByZXNzZWQsIG1lYW5pbmcgdGhleSB3aWxsIHRha2UgdXAgbGVzcyBzcGFjZSBvbiB5b3VyIApjb21wdXRlci4gCkxldCdzIHNhdmUgb3VyIGRhdGEgdG8gYW4gUkRTIGZpbGUgaW4gb3VyIGByZXN1bHRzYCBmb2xkZXIuCllvdSB3aWxsIG5lZWQgdG8gcmVwbGFjZSB0aGUgYC50c3ZgIHdpdGggYC5SRFNgLCBidXQgeW91IGNhbiB1c2Ugd2hhdCB3ZSAKZGV0ZXJtaW5lZCBhcyBvdXIgZmlsZSBwYXRoIGZvciB0aGUgbGFzdCBjaHVuayBhcyB5b3VyIHRlbXBsYXRlLiAKCmBgYHtyIGV2YWw9RkFMU0V9CiMgV3JpdGUgeW91ciBvYmplY3QgdG8gYW4gUkRTIGZpbGUKcmVhZHI6OndyaXRlX3JkcyhzdGF0c19zdW1tYXJ5LCA8UFVUX0NPUlJFQ1RfRklMRV9QQVRIX0hFUkU+KQpgYGAKCiMjIyMgUmVhZCBhbiBSRFMgZmlsZQoKU2luY2Ugbm93IHlvdSBoYXZlIGxlYXJuZWQgdGhlIGByZWFkcmAgZnVuY3Rpb25zOiBgcmVhZF90c3YoKWAsIGB3cml0ZV90c3YoKWAsIAphbmQgbm93LCBgd3JpdGVfcmRzKClgLCB3aGF0IGRvIHlvdSBzdXBwb3NlIHRoZSBmdW5jdGlvbiB5b3Ugd2lsbCBuZWVkIHRvIHJlYWQgCnlvdXIgUkRTIGZpbGUgaXMgY2FsbGVkPyAKVXNlIHRoYXQgZnVuY3Rpb24gaGVyZSB0byByZS1pbXBvcnQgeW91ciBkYXRhIGluIHRoZSBjaHVuayB3ZSBzZXQgdXAgZm9yIHlvdQpiZWxvdy4KCmBgYHtyIGV2YWw9RkFMU0V9CiMgUmVhZCBpbiB5b3VyIFJEUyBmaWxlCnJlaW1wb3J0X2RmIDwtIDxQVVRfRlVOQ1RJT05fTkFNRT4oZmlsZS5wYXRoKCJyZXN1bHRzIiwgInN0YXRzX2NsZWFuLlJEUyIpKQpgYGAKCkFzIGlzIGdvb2QgcHJhY3RpY2UsIHdlIHdpbGwgZW5kIHRoaXMgc2Vzc2lvbiBieSBwcmludGluZyBvdXQgb3VyIHNlc3Npb24gaW5mby4gCgojIyMgU2Vzc2lvbiBJbmZvCgpgYGB7cn0KIyBQcmludCBvdXQgdGhlIHZlcnNpb25zIGFuZCBwYWNrYWdlcyB3ZSBhcmUgdXNpbmcgaW4gdGhpcyBzZXNzaW9uCnNlc3Npb25JbmZvKCkKYGBgCg==
      
+
      LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIHRpZHl2ZXJzZSIKYXV0aG9yOiAiQ0NETCBmb3IgQUxTRiIgCmRhdGU6IDIwMjEKb3V0cHV0OiAgIAogIGh0bWxfbm90ZWJvb2s6IAogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKZWRpdG9yX29wdGlvbnM6IAogIGNodW5rX291dHB1dF90eXBlOiBpbmxpbmUKLS0tCgoKIyMgT2JqZWN0aXZlcwoKVGhpcyBub3RlYm9vayB3aWxsIGRlbW9uc3RyYXRlIGhvdyB0bzoKCi0gVXNlIGZ1bmN0aW9ucyBmcm9tIHRoZSB0aWR5dmVyc2UgdG8gcmVhZCBhbmQgd3JpdGUgZGF0YSBmcmFtZXMKLSBJbXBsZW1lbnQgYW5kIHVzZSB0aWR5dmVyc2UgZnVuY3Rpb25zIHRvIHdyYW5nbGUgZGF0YSAoaS5lLiBmaWx0ZXIsIG11dGF0ZSwgYXJyYW5nZSwgam9pbikKLSBVc2UgYG1hZ3JpdHRyYCBwaXBlcyAoYCU+JWApIHRvIGNvbWJpbmUgbXVsdGlwbGUgb3BlcmF0aW9ucyAKLSBVc2UgdGhlIGBhcHBseSgpYCBmdW5jdGlvbiB0byBhcHBseSBmdW5jdGlvbnMgYWNyb3NzIHJvd3Mgb3IgY29sdW1ucyBvZiBhIG1hdHJpeAoKLS0tCgpXZSdsbCB1c2UgdGhlIHNhbWUgZ2VuZSBleHByZXNzaW9uIGRhdGFzZXQgd2UgdXNlZCBpbiB0aGUgW3ByZXZpb3VzIG5vdGVib29rXSguLzAyLWludHJvX3RvX2dncGxvdDIuUm1kKS4KSXQgaXMgYSBwcmUtcHJvY2Vzc2VkIFthc3Ryb2N5dG9tYSBtaWNyb2FycmF5IGRhdGFzZXRdKGh0dHBzOi8vd3d3LnJlZmluZS5iaW8vZXhwZXJpbWVudHMvR1NFNDQ5NzEvZ2VuZS1leHByZXNzaW9uLWRhdGEtZnJvbS1waWxvY3l0aWMtYXN0cm9jeXRvbWEtdHVtb3VyLXNhbXBsZXMtYW5kLW5vcm1hbC1jZXJlYmVsbHVtLWNvbnRyb2xzKSAKdGhhdCB3ZSBwZXJmb3JtZWQgYSBzZXQgb2YgW2RpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2VzIG9uXSguL3NjcmlwdHMvMDAtc2V0dXAtaW50cm8tdG8tUi5SKS4gIAoKKipNb3JlIHRpZHl2ZXJzZSByZXNvdXJjZXM6KiogCgotIFtSIGZvciBEYXRhIFNjaWVuY2VdKGh0dHBzOi8vcjRkcy5oYWQuY28ubnovKSAgCi0gW3RpZHl2ZXJzZSBkb2N1bWVudGF0aW9uXShodHRwczovL2RwbHlyLnRpZHl2ZXJzZS5vcmcvKSAgCi0gW0NoZWF0c2hlZXQgb2YgdGlkeXZlcnNlIGRhdGEgdHJhbnNmb3JtYXRpb25dKGh0dHBzOi8vZ2l0aHViLmNvbS9yc3R1ZGlvL2NoZWF0c2hlZXRzL3Jhdy9tYXN0ZXIvZGF0YS10cmFuc2Zvcm1hdGlvbi5wZGYpICAKLSBbT25saW5lIHRpZHl2ZXJzZSBib29rIGNoYXB0ZXJdKGh0dHBzOi8vcHJpdmVmbC5naXRodWIuaW8vYWR2cjM4Ym9vay90aWR5dmVyc2UuaHRtbCkgIAoKIyMgU2V0IFVwCgpUaGUgdGlkeXZlcnNlIGlzIGEgY29sbGVjdGlvbiBvZiBwYWNrYWdlcyB0aGF0IGFyZSBoYW5keSBmb3IgZ2VuZXJhbCBkYXRhIAp3cmFuZ2xpbmcsIGFuYWx5c2lzLCBhbmQgdmlzdWFsaXphdGlvbi4gCk90aGVyIHBhY2thZ2VzIHRoYXQgYXJlIHNwZWNpZmljYWxseSBoYW5keSBmb3IgZGlmZmVyZW50IGJpb2xvZ2ljYWwgYW5hbHlzZXMgYXJlIApmb3VuZCBvbiBbQmlvY29uZHVjdG9yXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnLykuCklmIHdlIHdhbnQgdG8gdXNlIGEgcGFja2FnZSdzIGZ1bmN0aW9ucyB3ZSBmaXJzdCBuZWVkIHRvIGluc3RhbGwgdGhlbS4KCk91ciBSU3R1ZGlvIFNlcnZlciBhbHJlYWR5IGhhcyB0aGUgYHRpZHl2ZXJzZWAgZ3JvdXAgb2YgcGFja2FnZXMgaW5zdGFsbGVkIGZvciB5b3UuIApCdXQgaWYgeW91IG5lZWRlZCB0byBpbnN0YWxsIGl0IG9yIG90aGVyIHBhY2thZ2VzIGF2YWlsYWJsZSBvbiBDUkFOLCB5b3UgCmRvIGl0IHVzaW5nIHRoZSBgaW5zdGFsbC5wYWNrYWdlcygpYCBmdW5jdGlvbiBsaWtlIHRoaXM6IApgaW5zdGFsbC5wYWNrYWdlcygidGlkeXZlcnNlIilgLgoKYGBge3IgdGlkeXZlcnNlfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIyMgUmVmZXJlbmNpbmcgYSBsaWJyYXJ5J3MgZnVuY3Rpb24gd2l0aCBgOjpgCgpOb3RlIHRoYXQgaWYgd2UgaGFkIG5vdCBpbXBvcnRlZCB0aGUgdGlkeXZlcnNlIHNldCBvZiBwYWNrYWdlcyB1c2luZyBgbGlicmFyeSgpYCAKbGlrZSBhYm92ZSwgYW5kIHdlIHdhbnRlZCB0byB1c2UgYSB0aWR5dmVyc2UgZnVuY3Rpb24gbGlrZSBgcmVhZF90c3YoKWAsIHdlIAp3b3VsZCBuZWVkIHRvIHRlbGwgUiB3aGF0IHBhY2thZ2UgdG8gZmluZCB0aGlzIGZ1bmN0aW9uIGluLgpUbyBkbyB0aGlzLCB3ZSB3b3VsZCB1c2UgYDo6YCB0byB0ZWxsIFIgdG8gbG9hZCBpbiB0aGlzIGZ1bmN0aW9uIGZyb20gdGhlIApgcmVhZHJgIHBhY2thZ2UgYnkgdXNpbmcgYHJlYWRyOjpyZWFkX3RzdigpYC4KWW91IHdpbGwgc2VlIHRoaXMgYDo6YCBtZXRob2Qgb2YgcmVmZXJlbmNpbmcgbGlicmFyaWVzIHdpdGhpbiBwYWNrYWdlcyAKdGhyb3VnaG91dCB0aGUgY291cnNlLiAKV2UgbGlrZSB0byB1c2UgaXQgaW4gcGFydCB0byByZW1vdmUgYW55IGFtYmlndWl0eSBpbiB3aGljaCB2ZXJzaW9uIG9mIGEgCmZ1bmN0aW9uIHdlIGFyZSB1c2luZzsgaXQgaXMgbm90IHRvbyB1bmNvbW1vbiBmb3IgZGlmZmVyZW50IHBhY2thZ2VzIHRvIHVzZSB0aGUgCnNhbWUgbmFtZSBmb3IgdmVyeSBkaWZmZXJlbnQgZnVuY3Rpb25zIQoKIyMgTWFuYWdpbmcgZGlyZWN0b3JpZXMKCkJlZm9yZSB3ZSBjYW4gaW1wb3J0IHRoZSBkYXRhIHdlIG5lZWQsIHdlIHNob3VsZCBkb3VibGUgY2hlY2sgd2hlcmUgUiBpcyAKbG9va2luZyBmb3IgZmlsZXMsIGFrYSB0aGUgY3VycmVudCAqKndvcmtpbmcgZGlyZWN0b3J5KiouIApXZSBjYW4gZG8gdGhpcyBieSB1c2luZyB0aGUgYGdldHdkKClgIGZ1bmN0aW9uLCB3aGljaCB3aWxsIHRlbGwgdXMgd2hhdCBmb2xkZXIKd2UgYXJlIGluLiAKCmBgYHtyIHdvcmtpbmdkaXIsIGxpdmUgPSBUUlVFfQojIExldCdzIGNoZWNrIHdoYXQgZGlyZWN0b3J5IHdlIGFyZSBpbjoKZ2V0d2QoKQpgYGAKCkZvciBSbWQgZmlsZXMsIHRoZSB3b3JraW5nIGRpcmVjdG9yeSBpcyB3aGVyZXZlciB0aGUgZmlsZSBpcyBsb2NhdGVkLCBidXQgCmNvbW1hbmRzIGV4ZWN1dGVkIGluIHRoZSBjb25zb2xlIG1heSBoYXZlIGEgZGlmZmVyZW50IHdvcmtpbmcgZGlyZWN0b3J5LgoKV2Ugd2lsbCB3YW50IHRvIG1ha2UgYSBkaXJlY3RvcnkgZm9yIG91ciBvdXRwdXQgYW5kIHdlIHdpbGwgY2FsbCB0aGlzIGRpcmVjdG9yeTogCmByZXN1bHRzYC4gCkJ1dCBiZWZvcmUgd2UgY3JlYXRlIHRoZSBkaXJlY3RvcnksIHdlIHNob3VsZCBjaGVjayBpZiBpdCBhbHJlYWR5IGV4aXN0cy4gCldlIHdpbGwgc2hvdyB0d28gd2F5cyB0aGF0IHdlIGNhbiBkbyB0aGlzLiAKCkZpcnN0LCB3ZSBjYW4gdXNlIHRoZSBgZGlyKClgIGZ1bmN0aW9uIHRvIGhhdmUgUiBsaXN0IHRoZSBmaWxlcyBpbiBvdXIgd29ya2luZyAKZGlyZWN0b3J5LiAKCmBgYHtyfQojIExldCdzIGNoZWNrIHdoYXQgZmlsZXMgYXJlIGhlcmUKZGlyKCkKYGBgCgpUaGlzIHNob3dzIHVzIHRoZXJlIGlzIG5vIGZvbGRlciBjYWxsZWQgInJlc3VsdHMiIHlldC4gCgpJZiB3ZSB3YW50IHRvIG1vcmUgcG9pbnRlZGx5IGxvb2sgZm9yICJyZXN1bHRzIiBpbiBvdXIgd29ya2luZyBkaXJlY3Rvcnkgd2UgY2FuIAp1c2UgdGhlIGBkaXIuZXhpc3RzKClgIGZ1bmN0aW9uLgoKYGBge3IgY2hlY2stZGlyLCBsaXZlID0gVFJVRX0KIyBDaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzCmRpci5leGlzdHMoInJlc3VsdHMiKQpgYGAKCklmIHRoZSBhYm92ZSBzYXlzIGBGQUxTRWAgdGhhdCBtZWFucyB3ZSB3aWxsIG5lZWQgdG8gY3JlYXRlIGEgYHJlc3VsdHNgIApkaXJlY3RvcnkgdXNpbmcgdGhlIGZ1bmN0aW9uIGBkaXIuY3JlYXRlKClgLgoKYGBge3IgY3JlYXRlLWRpciwgbGl2ZSA9IFRSVUV9CiMgTWFrZSBhIGRpcmVjdG9yeSB3aXRoaW4gdGhlIHdvcmtpbmcgZGlyZWN0b3J5IGNhbGxlZCAncmVzdWx0cycKZGlyLmNyZWF0ZSgicmVzdWx0cyIpCmBgYAoKQWZ0ZXIgY3JlYXRpbmcgdGhlIHJlc3VsdHMgZGlyZWN0b3J5IGFib3ZlLCBsZXQncyByZS1ydW4gYGRpci5leGlzdHMoKWAgdG8gc2VlIAppZiBub3cgaXQgZXhpc3RzLgoKYGBge3IgY2hlY2stZGlyLWFnYWluLCBsaXZlID0gVFJVRX0KIyBSZS1jaGVjayBpZiB0aGUgcmVzdWx0cyBkaXJlY3RvcnkgZXhpc3RzCmRpci5leGlzdHMoInJlc3VsdHMiKQpgYGAKCldlIGNhbiB1c2UgdGhlIG91dHB1dCBvZiBgZGlyLmV4aXN0cygpYCB0byBhdXRvbWF0aWNhbGx5IGNyZWF0ZSBvciBob2xkIG9mZiBvbiAKY3JlYXRpbmcgYSBkaXJlY3RvcnkgYnkgcHV0dGluZyB0aGlzIHRvZ2V0aGVyIGluIGFuIGBpZmAgc3RhdGVtZW50IGxpa2UgYmVsb3cuIApBbiBgaWZgIHN0YXRlbWVudCBoYXMgdHdvIG1haW4gcGFydHM6CkZpcnN0LCB0aGUgdGVzdCwgd2hpY2ggaXMgYW4gZXhwcmVzc2lvbiB0aGF0IHdpbGwgcmVzdWx0IGluIGVpdGhlciBgVFJVRWAgb3IgYEZBTFNFYC4KVGhpcyBpcyBwdXQgaW4gcGFyZW50aGVzaXMgaW1tZWRpYXRlbHkgYWZ0ZXIgdGhlIGBpZmAuClRoZSBuZXh0IHBhcnQgaXMgdGhlIGJvZHksIHdoaWNoIGlzIHRoZSBjb21tYW5kcyB0aGF0IHdpbGwgYmUgZXhlY3V0ZWQgKmlmKiB0aGUgCnRlc3QgaXMgYFRSVUVgLgpUaGVzZSBhcmUgcGxhY2VkIHdpdGhpbiBhIHNldCBvZiBicmFjZXMgYHsgfWAuCk5vdGUgdGhhdCB3ZSB1c2VkIGFuIGV4Y2xhbWF0aW9uIHBvaW50IGluIHRoZSB0ZXN0IHRvIHNpZ25pZnkgdGhhdCB3ZSB3YW50IGEgCmRpcmVjdG9yeSB0byBiZSBjcmVhdGVkIG9ubHkgKmlmKiBgZGlyLmV4aXN0cyhyZXN1bHRzKWAgaXMgTk9UIGVxdWFsIHRvIGBUUlVFYC4KCmBgYHtyIGNyZWF0ZS1pZn0KIyBJZiAncmVzdWx0cycgZGlyZWN0b3J5IGRvZXNuJ3QgZXhpc3QuLi4KaWYgKCFkaXIuZXhpc3RzKCJyZXN1bHRzIikpIHsKICAjIC4uLiBjcmVhdGUgYSAncmVzdWx0cycgZGlyZWN0b3J5CiAgZGlyLmNyZWF0ZSgicmVzdWx0cyIpCn0KYGBgCgpUaGUgYGRpci5leGlzdHMoKWAgZnVuY3Rpb24gd2lsbCBub3Qgd29yayBvbiBmaWxlcyB0aGVtc2VsdmVzLgpJbiB0aGF0IGNhc2UsIHRoZXJlIGlzIGFuIGFuYWxvZ291cyBmdW5jdGlvbiBjYWxsZWQgYGZpbGUuZXhpc3RzKClgLgoKVHJ5IHVzaW5nIHRoZSBgZmlsZS5leGlzdHMoKWAgZnVuY3Rpb24gdG8gc2VlIGlmIHRoZSBmaWxlIApgZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdmAgZXhpc3RzIGluIHRoZSBjdXJyZW50IGRpcmVjdG9yeS4KVXNlIHRoZSBjb2RlIGNodW5rIHdlIHNldCB1cCBmb3IgeW91IGJlbG93LiAKTm90ZSB0aGF0IGluIG91ciBub3RlYm9va3MgKGFuZCBzb21ldGltZXMgZWxzZXdoZXJlKSwgd2hlcmV2ZXIgeW91IHNlZSBhIApgPEZJTExfSU5fVEhFX0JMQU5LPmAgbGlrZSBpbiB0aGUgY2h1bmsgYmVsb3csIHRoYXQgaXMgbWVhbnQgZm9yIHlvdSB0byByZXBsYWNlIAooaW5jbHVkaW5nIHRoZSBhbmdsZSBicmFja2V0cykgd2l0aCB0aGUgY29ycmVjdCBwaHJhc2UgYmVmb3JlIHlvdSBydW4gdGhlIGNodW5rIAoob3RoZXJ3aXNlIHlvdSB3aWxsIGdldCBhbiBlcnJvcikuCgpgYGB7ciBmaWxlLWNoZWNrLCBldmFsPUZBTFNFfQojIFJlcGxhY2UgdGhlIDxQVVRfRklMRV9OQU1FX0hFUkU+IHdpdGggdGhlIG5hbWUgb2YgdGhlIGZpbGUgeW91IGFyZSBsb29raW5nIGZvcgojIFJlbWVtYmVyIHRvIHVzZSBxdW90ZXMgdG8gbWFrZSBpdCBhIGNoYXJhY3RlciBzdHJpbmcKZmlsZS5leGlzdHMoPFBVVF9GSUxFX05BTUVfSEVSRT4pCmBgYAoKTm93IHRoYXQgd2UndmUgZGV0ZXJtaW5lZCB0aGF0IGBnZW5lX3Jlc3VsdHNfR1NFNDQ5NzEudHN2YCBleGlzdHMsIHdlIGFyZSByZWFkeSAKdG8gcmVhZCBpdCBpbnRvIG91ciBSIGVudmlyb25tZW50LgoKIyMjIyBSZWFkIGEgVFNWIGZpbGUKCkRlY2xhcmUgdGhlIG5hbWUgb2YgdGhlIGRpcmVjdG9yeSB3aGVyZSB3ZSB3aWxsIHJlYWQgaW4gdGhlIGRhdGEuIAoKYGBge3J9CmRhdGFfZGlyIDwtICJkYXRhIgpgYGAKCkFsdGhvdWdoIGJhc2UgUiBoYXMgZnVuY3Rpb25zIHRvIHJlYWQgaW4gZGF0YSBmaWxlcywgdGhlIGZ1bmN0aW9ucyBpbiB0aGUgCmByZWFkcmAgcGFja2FnZSAocGFydCBvZiB0aGUgdGlkeXZlcnNlKSBhcmUgZmFzdGVyIGFuZCBtb3JlIHN0cmFpZ2h0Zm9yd2FyZCAKdG8gdXNlIHNvIHdlIGFyZSBnb2luZyB0byB1c2UgdGhvc2UgaGVyZS4gCkJlY2F1c2UgdGhlIGZpbGUgd2UgYXJlIHJlYWRpbmcgaW4gaXMgYSBUU1YgKHRhYiBzZXBhcmF0ZWQgdmFsdWVzKSBmaWxlIHdlIHdpbGwgCmJlIHVzaW5nIHRoZSBgcmVhZF90c3ZgIGZ1bmN0aW9uLiAKVGhlcmUgYXJlIGFuYWxvZ291cyBmdW5jdGlvbnMgZm9yIENTViAoY29tbWEgc2VwYXJhdGVkIHZhbHVlcykgZmlsZXMgCihgcmVhZF9jc3YoKWApIGFuZCBvdGhlciBmaWxlcyB0eXBlcy4KCiMjIFJlYWQgaW4gdGhlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIHJlc3VsdHMgZmlsZQoKYGBge3IgcmVhZC1yZXN1bHRzfQpzdGF0c19kZiA8LSByZWFkcjo6cmVhZF90c3YoCiAgZmlsZS5wYXRoKGRhdGFfZGlyLAogICAgICAgICAgICAiZ2VuZV9yZXN1bHRzX0dTRTQ0OTcxLnRzdiIpCiAgKQpgYGAKCkZvbGxvd2luZyB0aGUgdGVtcGxhdGUgb2YgdGhlIHByZXZpb3VzIGNodW5rLCB1c2UgdGhpcyBjaHVuayB0byByZWFkIGluIHRoZSBmaWxlCmBHU0U0NDk3MS50c3ZgIHRoYXQgaXMgaW4gdGhlIGBkYXRhYCBmb2xkZXIgYW5kIHNhdmUgaXQgaW4gdGhlIHZhcmlhYmxlIGBnZW5lX2RmYC4gCgpgYGB7ciByZWFkLWV4cHIsIGxpdmUgPSBUUlVFfQojIFVzZSB0aGlzIGNodW5rIHRvIHJlYWQgaW4gZGF0YSBmcm9tIHRoZSBmaWxlIGBHU0U0NDk3MS50c3ZgCmdlbmVfZGYgPC0gcmVhZHI6OnJlYWRfdHN2KAogIGZpbGUucGF0aChkYXRhX2RpciwKICAgICAgICAgICAgIkdTRTQ0OTcxLnRzdiIpCiAgKQpgYGAKClVzZSB0aGlzIGNodW5rIHRvIGV4cGxvcmUgd2hhdCBgZ2VuZV9kZmAgbG9va3MgbGlrZS4gCgpgYGB7ciBleHBsb3JlfQojIEV4cGxvcmUgYGdlbmVfZGZgCgpgYGAKCldoYXQgaW5mb3JtYXRpb24gaXMgY29udGFpbmVkIGluIGBnZW5lX2RmYD8KCiMjIGBtYWdyaXR0cmAgcGlwZXMKCk9uZSBuaWZ0eSBmZWF0dXJlIG9mIHRoZSB0aWR5dmVyc2UgaXMgcGlwZXM6IGAlPiVgClRoZXNlIGhhbmR5IHRoaW5ncywgd2hpY2ggY29tZSBmcm9tIHRoZSBgbWFncml0dHJgIHBhY2thZ2UsIGFsbG93IHlvdSB0byBmdW5uZWwgdGhlIHJlc3VsdCBvZiBvbmUgZXhwcmVzc2lvbiB0byB0aGUgbmV4dCwKbWFraW5nIHlvdXIgY29kZSBhIGxpdHRsZSBtb3JlIHN0cmVhbWxpbmVkLgoKRm9yIGV4YW1wbGUsIHRoZSBvdXRwdXQgZnJvbSB0aGlzOiAgCgpgYGB7ciBmaWx0ZXJ9CmZpbHRlcihzdGF0c19kZiwgY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIikKYGBgICAKCi4uLmlzIHRoZSBzYW1lIGFzIHRoZSBvdXRwdXQgZnJvbSB0aGlzOiAgCgpgYGB7ciBmaWx0ZXItcGlwZX0Kc3RhdHNfZGYgJT4lIGZpbHRlcihjb250cmFzdCA9PSAibWFsZV9mZW1hbGUiKQpgYGAgIAogIApUaGlzIGNhbiBtYWtlIHlvdXIgY29kZSBjbGVhbmVyIGFuZCBlYXNpZXIgdG8gZm9sbG93IGEgc2VyaWVzIG9mIHJlbGF0ZWQgCmNvbW1hbmRzLiAKTGV0J3MgbG9vayBhdCBhbiBleGFtcGxlIHdpdGggb3VyIHN0YXRzIG9mIG9mIGhvdyB0aGUgc2FtZSAKZnVuY3Rpb25zIGxvb2sgd2l0aCBvciB3aXRob3V0IHBpcGVzOgoKKkV4YW1wbGUgMToqIHdpdGhvdXQgcGlwZXM6IAoKYGBge3Igc3RlcHMtbm9waXBlfQpzdGF0c19hcnJhbmdlZCA8LSBhcnJhbmdlKHN0YXRzX2RmLCB0X3N0YXRpc3RpYykKc3RhdHNfZmlsdGVyZWQgPC0gZmlsdGVyKHN0YXRzX2FycmFuZ2VkLCBhdmdfZXhwcmVzc2lvbiA+IDUwKQpzdGF0c19ub3BpcGUgPC0gc2VsZWN0KHN0YXRzX2ZpbHRlcmVkLCBjb250cmFzdCwgbG9nX2ZvbGRfY2hhbmdlLCBwX3ZhbHVlKQpgYGAKICAKVUdILCB3ZSBoYXZlIHRvIGtlZXAgdHJhY2sgb2YgYWxsIG9mIHRob3NlIGRpZmZlcmVudCBpbnRlcm1lZGlhdGUgZGF0YSBmcmFtZXMgCmFuZCB0eXBlIHRoZWlyIG5hbWVzIHNvIG1hbnkgdGltZXMgaGVyZSEgCldlIGNvdWxkIG1heWJlIHN0cmVhbWxpbmUgdGhpbmdzIGJ5IHVzaW5nIHRoZSBzYW1lIHZhcmlhYmxlIG5hbWUgYXQgZWFjaCBzdGFnZSwgCmJ1dCBldmVuIHRoZW4gdGhlcmUgaXMgYSBsb3Qgb2YgZXh0cmEgdHlwaW5nLCBhbmQgaXQgaXMgZWFzeSB0byBnZXQgY29uZnVzZWQgCmFib3V0IHdoYXQgaGFzIGJlZW4gZG9uZSB3aGVyZS4KSXQncyBhbm5veWluZyBhbmQgbWFrZXMgaXQgaGFyZGVyIGZvciBwZW9wbGUgdG8gcmVhZC4gCiAgCipFeGFtcGxlIDI6KiBTYW1lIHJlc3VsdCBhcyAxIGJ1dCB3aXRoIHBpcGVzIQoKYGBge3Igc3RlcHMtcGlwZSwgbGl2ZSA9IFRSVUV9CiMgRXhhbXBsZSBvZiB0aGUgc2FtZSBtb2RpZmljYXRpb25zIGFzIGFib3ZlIGJ1dCB3aXRoIHBpcGVzIQpzdGF0c19waXBlICA8LSBzdGF0c19kZiAlPiUKICAgICAgICAgICAgICAgYXJyYW5nZSh0X3N0YXRpc3RpYykgJT4lCiAgICAgICAgICAgICAgIGZpbHRlcihhdmdfZXhwcmVzc2lvbiA+IDUwKSAlPiUKICAgICAgICAgICAgICAgc2VsZWN0KGNvbnRyYXN0LCBsb2dfZm9sZF9jaGFuZ2UsIHBfdmFsdWUpCmBgYAoKV2hhdCB0aGUgYCU+JWAgKHBpcGUpIGlzIGRvaW5nIGhlcmUgaXMgZmVlZGluZyB0aGUgcmVzdWx0IG9mIHRoZSBleHByZXNzaW9uIG9uIAppdHMgbGVmdCBpbnRvIHRoZSBmaXJzdCBhcmd1bWVudCBvZiB0aGUgbmV4dCBmdW5jdGlvbiAodG8gaXRzIHJpZ2h0LCBvciBvbiB0aGUgCm5leHQgbGluZSBoZXJlKS4gCldlIGNhbiB0aGVuIHNraXAgdGhhdCBmaXJzdCBhcmd1bWVudCAodGhlIGRhdGEgaW4gdGhlc2UgY2FzZXMpLCBhbmQgbW92ZSByaWdodCAKb24gdG8gdGhlIHBhcnQgd2UgY2FyZSBhYm91dCBhdCB0aGF0IHN0ZXA6IHdoYXQgd2UgYXJlIGFycmFuZ2luZywgZmlsdGVyaW5nLCBvciAKc2VsZWN0aW5nICBpbiB0aGlzIGNhc2UuCgpMZXQncyBkb3VibGUgY2hlY2sgdGhhdCB0aGVzZSBhcmUgdGhlIHNhbWUgYnkgdXNpbmcgdGhlIGZ1bmN0aW9uLCBgYWxsLmVxdWFsKClgLiAKCmBgYHtyIGNoZWNrLXBpcGV9CmFsbC5lcXVhbChzdGF0c19ub3BpcGUsIHN0YXRzX3BpcGUpCmBgYAoKYGFsbC5lcXVhbCgpYCBpcyBsZXR0aW5nIHVzIGtub3cgdGhhdCB0aGVzZSB0d28gb2JqZWN0cyBhcmUgdGhlIHNhbWUuIAoKTm93IHRoYXQgaG9wZWZ1bGx5IHlvdSBhcmUgY29udmluY2VkIHRoYXQgdGhlIHRpZHl2ZXJzZSBjYW4gaGVscCB5b3UgbWFrZSB5b3VyIApjb2RlIG5lYXRlciBhbmQgZWFzaWVyIHRvIHVzZSBhbmQgcmVhZCwgbGV0J3MgZ28gdGhyb3VnaCBzb21lIG9mIHRoZSBwb3B1bGFyIAp0aWR5dmVyc2UgZnVuY3Rpb25zIGFuZCBzbyB3ZSBjYW4gY3JlYXRlIHBpcGVsaW5lcyBsaWtlIHRoaXMuIAoKIyMgQ29tbW9uIHRpZHl2ZXJzZSBmdW5jdGlvbnMKCkxldCdzIHNheSB3ZSB3YW50ZWQgdG8gZmlsdGVyIHRoaXMgZ2VuZSBleHByZXNzaW9uIGRhdGFzZXQgdG8gcGFydGljdWxhciBzYW1wbGUKZ3JvdXBzLgpJbiBvcmRlciB0byBkbyB0aGlzLCB3ZSB3b3VsZCB1c2UgdGhlIGZ1bmN0aW9uIGBmaWx0ZXIoKWAgYXMgd2VsbCBhcyBhIGxvZ2ljIApzdGF0ZW1lbnQgKHVzdWFsbHkgb25lIHRoYXQgcmVmZXJzIHRvIGEgY29sdW1uIG9yIGNvbHVtbnMgaW4gdGhlIGRhdGEgZnJhbWUpLgoKYGBge3IgZmlsdGVyLWdlbmV9CiMgSGVyZSBsZXQncyBmaWx0ZXIgc3RhdHNfZGYgdG8gdGhlIGdlbmVfc3ltYm9sICJTTkNBIgpzdGF0c19kZiAlPiUgCiAgZmlsdGVyKGdlbmVfc3ltYm9sID09ICJTTkNBIikKYGBgCgpXZSBjYW4gdXNlIGBmaWx0ZXIoKWAgc2ltaWxhcmx5IGZvciBudW1lcmljIHN0YXRlbWVudHMuICAKCmBgYHtyIGZpbHRlci1udW1lcmljLCBsaXZlID0gVFJVRX0KIyBIZXJlIGxldCdzIGZpbHRlciB0aGUgZGF0YSB0byByb3dzIHdpdGggYXZlcmFnZSBleHByZXNzaW9uIHZhbHVlcyBhYm92ZSA1MApzdGF0c19kZiAlPiUKICBmaWx0ZXIoYXZnX2V4cHJlc3Npb24gPiA1MCkKYGBgCgpXZSBjYW4gYXBwbHkgbXVsdGlwbGUgZmlsdGVycyBhdCBvbmNlLCB3aGljaCB3aWxsIHJlcXVpcmUgYWxsIG9mIHRoZW0gdG8gYmUgCnNhdGlzZmllZCBmb3IgZXZlcnkgcm93IGluIHRoZSByZXN1bHRzOgoKYGBge3IgZmlsdGVyLTIsIGxpdmUgPSBUUlVFfQojIGZpbHRlciB0byBoaWdobHkgZXhwcmVzc2VkIGdlbmVzIHdpdGggY29udHJhc3QgIm1hbGVfZmVtYWxlIgpzdGF0c19kZiAlPiUKICBmaWx0ZXIoY29udHJhc3QgPT0gIm1hbGVfZmVtYWxlIiwgCiAgICAgICAgIGF2Z19leHByZXNzaW9uID4gNTApCmBgYAoKV2hlbiB3ZSBhcmUgZmlsdGVyaW5nLCB0aGUgYCVpbiVgIG9wZXJhdG9yIGNhbiBjb21lIGluIGhhbmR5IGlmIHdlIGhhdmUgbXVsdGlwbGUKaXRlbXMgd2Ugd291bGQgbGlrZSB0byBtYXRjaC4KTGV0J3MgdGFrZSBhIGxvb2sgYXQgd2hhdCB1c2luZyBgJWluJWAgZG9lcy4KCmBgYHtyIGluLWV4YW1wbGUsIGV2YWwgPSBGQUxTRX0KZ2VuZXNfb2ZfaW50ZXJlc3QgPC0gYygiU05DQSIsICJDREtOMUEiKQpzdGF0c19kZiRnZW5lX3N5bWJvbCAlaW4lIGdlbmVzX29mX2ludGVyZXN0CmBgYAoKYCVpbiVgIHJldHVybnMgYSBsb2dpY2FsIHZlY3RvciB0aGF0IG5vdyB3ZSBjYW4gdXNlIGluIGBkcGx5cjo6ZmlsdGVyYC4KCmBgYHtyIGZpbHRlci1pbiwgbGl2ZSA9IFRSVUV9CiMgZmlsdGVyIHRvIGdlbmVzIG9mIGludGVyZXN0CnN0YXRzX2RmICU+JSAKICBmaWx0ZXIoZ2VuZV9zeW1ib2wgJWluJSBjKCJTTkNBIiwgIkNES04xQSIpKQpgYGAKCkxldCdzIHJldHVybiB0byBvdXIgZmlyc3QgYGZpbHRlcigpYCBhbmQgYnVpbGQgb24gdG8gaXQuIApUaGlzIHRpbWUsIGxldCdzIGtlZXAgb25seSBzb21lIG9mIHRoZSBjb2x1bW5zIGZyb20gdGhlIGRhdGEgZnJhbWUgdXNpbmcgdGhlIApgc2VsZWN0KClgIGZ1bmN0aW9uLiAKTGV0J3MgYWxzbyBzYXZlIHRoaXMgYXMgYSBuZXcgZGF0YSBmcmFtZSBjYWxsZWQgYHN0YXRzX2ZpbHRlcmVkX2RmYC4KCmBgYHtyIGZpbHRlci1zZWxlY3QsIGxpdmUgPSBUUlVFfQojIGZpbHRlciB0byBoaWdobHkgZXhwcmVzc2VkICJtYWxlX2ZlbWFsZSIKIyBhbmQgc2VsZWN0IGdlbmVfc3ltYm9sLCBsb2dfZm9sZF9jaGFuZ2UgYW5kIHRfc3RhdGlzdGljCnN0YXRzX2ZpbHRlcmVkX2RmIDwtIHN0YXRzX2RmICU+JQogIGZpbHRlcihjb250cmFzdCA9PSAibWFsZV9mZW1hbGUiLCAKICAgICAgICAgYXZnX2V4cHJlc3Npb24gPiA1MCkgJT4lCiAgc2VsZWN0KGxvZ19mb2xkX2NoYW5nZSwgdF9zdGF0aXN0aWMpCmBgYAoKTGV0J3Mgc2F5IHdlIHdhbnRlZCB0byBhcnJhbmdlIHRoaXMgZGF0YXNldCBzbyB0aGF0IHRoZSBnZW5lcyBhcmUgYXJyYW5nZWQgYnkgCnRoZSBzbWFsbGVzdCBwIHZhbHVlcyB0byB0aGUgbGFyZ2VzdC4KSW4gb3JkZXIgdG8gZG8gdGhpcywgd2Ugd291bGQgdXNlIHRoZSBmdW5jdGlvbiBgYXJyYW5nZSgpYCBhcyB3ZWxsIGFzIHRoZSBjb2x1bW4Kd2Ugd291bGQgbGlrZSB0byBzb3J0IGJ5IChpbiB0aGlzIGNhc2UgYHBfdmFsdWVgKS4KCmBgYHtyIGFycmFuZ2V9CnN0YXRzX2RmICU+JSAKICBhcnJhbmdlKHBfdmFsdWUpIApgYGAKCldoYXQgaWYgd2Ugd2FudCB0byBzb3J0IGZyb20gbGFyZ2VzdCB0byBzbWFsbGVzdD8gCkxpa2UgaWYgd2Ugd2FudCB0byBzZWUgdGhlIGdlbmVzIHdpdGggdGhlIGhpZ2hlc3QgYXZlcmFnZSBleHByZXNzaW9uPwpXZSBjYW4gdXNlIHRoZSBzYW1lIGZ1bmN0aW9uLCBidXQgaW5zdGVhZCB1c2UgdGhlIGBkZXNjKClgIGZ1bmN0aW9uIGFuZCBub3cgd2UgCmFyZSB1c2luZyBgYXZnX2V4cHJlc3Npb25gIGNvbHVtbi4gCgpgYGB7ciBhcnJhbmdlLWRlc2N9CiMgYXJyYW5nZSBkZXNjZW5kaW5nIGJ5IGF2Z19leHByZXNzaW9uCnN0YXRzX2RmICU+JQogIGFycmFuZ2UoZGVzYyhhdmdfZXhwcmVzc2lvbikpCmBgYCAKCldoYXQgaWYgd2Ugd291bGQgbGlrZSB0byBjcmVhdGUgYSBuZXcgY29sdW1uIG9mIHZhbHVlcz8KRm9yIHRoYXQgd2UgdXNlIGBtdXRhdGUoKWAgZnVuY3Rpb24uCgpgYGB7ciBtdXRhdGV9CnN0YXRzX2RmICU+JSAKICBtdXRhdGUobG9nMTBfcF92YWx1ZSA9IC1sb2cxMChwX3ZhbHVlKSkKYGBgCgpXaGF0IGlmIHdlIHdhbnQgdG8gb2J0YWluIHN1bW1hcnkgc3RhdGlzdGljcyBmb3IgYSBjb2x1bW4gb3IgY29sdW1ucz8KVGhlIGBzdW1tYXJpemVgIGZ1bmN0aW9uIGFsbG93cyB1cyB0byBjYWxjdWxhdGUgc3VtbWFyeSBzdGF0aXN0aWNzIGZvciBhIGNvbHVtbi4gCkhlcmUgd2Ugd2lsbCB1c2Ugc3VtbWFyaXplIHRvIG9idGFpbiBhbiBtZWFuIGxvZyBmb2xkZXIgY2hhbmdlIG92ZXIgYWxsIHRoZQpnZW5lcywgYW5kIGl0cyBzdGFuZGFyZCBkZXZpYXRpb24uCgpgYGB7ciBzdW1tYXJpemV9CnN0YXRzX2RmICU+JSAKICBzdW1tYXJpemUobWVhbihsb2dfZm9sZF9jaGFuZ2UpLAogICAgICAgICAgICBzZChsb2dfZm9sZF9jaGFuZ2UpKQpgYGAKCldoYXQgaWYgd2UnZCBsaWtlIHRvIG9idGFpbiBhIHN1bW1hcnkgc3RhdGlzdGljcyBidXQgaGF2ZSB0aGVtIGZvciB2YXJpb3VzIApncm91cHM/CkNvbnZlbmllbnRseSBuYW1lZCwgdGhlcmUncyBhIGZ1bmN0aW9uIGNhbGxlZCBgZ3JvdXBfYnkoKWAgdGhhdCBzZWFtbGVzc2x5IAphbGxvd3MgdXMgdG8gZG8gdGhpcy4gCkFsc28gbm90ZSB0aGF0IGBncm91cF9ieSgpYCBhbGxvd3MgdXMgdG8gZ3JvdXAgYnkgbXVsdGlwbGUgdmFyaWFibGVzIGF0IGEgdGltZSAKaWYgeW91IHdhbnQgdG8uCgpgYGB7ciBzdW1tYXJpemUtZ3JvdXBzLCBsaXZlID0gVFJVRX0Kc3RhdHNfc3VtbWFyeV9kZiA8LSBzdGF0c19kZiAlPiUKICAgICAgZ3JvdXBfYnkoY29udHJhc3QpICU+JSAKICAgICAgc3VtbWFyaXplKG1lYW4obG9nX2ZvbGRfY2hhbmdlKSwKICAgICAgICAgICAgICAgIHNkKGxvZ19mb2xkX2NoYW5nZSkpCmBgYAoKTGV0J3MgbG9vayBhdCBhIHByZXZpZXcgb2Ygd2hhdCB3ZSBtYWRlOgoKYGBge3J9CnN0YXRzX3N1bW1hcnlfZGYKYGBgCgpIZXJlIHdlIGhhdmUgdGhlIG1lYW4gbG9nIGZvbGQgY2hhbmdlIGV4cHJlc3Npb24gcGVyIGVhY2ggY29udHJhc3Qgd2UgbWFkZS4gCgojIyBBIGJyaWVmIGludHJvIHRvIHRoZSBgYXBwbHlgIGZhbWlseSBvZiBmdW5jdGlvbnMKCkluIGJhc2UgUiwgdGhlIGBhcHBseWAgZmFtaWx5IG9mIGZ1bmN0aW9ucyBjYW4gYmUgYW4gYWx0ZXJuYXRpdmUgbWV0aG9kcyBmb3IgCnBlcmZvcm1pbmcgdHJhbnNmb3JtYXRpb25zIGFjcm9zcyBhIGRhdGEgZnJhbWUsIG1hdHJpeCBvciBvdGhlciBvYmplY3Qgc3RydWN0dXJlcy4gCgpPbmUgb2YgdGhpcyBmYW1pbHkgaXMgKHNob2NraW5nbHkpIHRoZSBmdW5jdGlvbiBgYXBwbHkoKWAsIHdoaWNoIG9wZXJhdGVzIG9uIAptYXRyaWNlcy4KCkEgbWF0cml4IGlzIHNpbWlsYXIgdG8gYSBkYXRhIGZyYW1lIGluIHRoYXQgaXQgaXMgYSByZWN0YW5ndWxhciB0YWJsZSBvZiBkYXRhLApidXQgaXQgaGFzIGFuIGFkZGl0aW9uYWwgY29uc3RyYWludDogCnJhdGhlciB0aGFuIGVhY2ggY29sdW1uIGhhdmluZyBhIHR5cGUsIEFMTCBkYXRhIGluIGEgbWF0cml4IGhhcyB0aGUgc2FtZSB0eXBlLgoKVGhlIGZpcnN0IGFyZ3VtZW50IHRvIGBhcHBseSgpYCBpcyB0aGUgZGF0YSBvYmplY3Qgd2Ugd2FudCB0byB3b3JrIG9uLgpUaGUgdGhpcmQgYXJndW1lbnQgaXMgdGhlIGZ1bmN0aW9uIHdlIHdpbGwgYXBwbHkgdG8gZWFjaCByb3cgb3IgY29sdW1uIG9mIHRoZSAKZGF0YSBvYmplY3QuClRoZSBzZWNvbmQgYXJndW1lbnQgaW4gc3BlY2lmaWVzIHdoZXRoZXIgd2UgYXJlIGFwcGx5aW5nIHRoZSBmdW5jdGlvbiAKYWNyb3NzIHJvd3Mgb3IgYWNyb3NzIGNvbHVtbnMgKDEgZm9yIHJvd3MsIDIgZm9yIGNvbHVtbnMpLgoKUmVtZW1iZXIgdGhhdCBgZ2VuZV9kZmAgaXMgYSBnZW5lIHggc2FtcGxlIGdlbmUgZXhwcmVzc2lvbiBkYXRhIGZyYW1lIHRoYXQgaGFzCmNvbHVtbnMgb2YgdHdvIGRpZmZlcmVudCB0eXBlcywgY2hhcmFjdGVyIGFuZCBudW1lcmljLgpDb252ZXJ0aW5nIGl0IHRvIGEgbWF0cml4IHdpbGwgcmVxdWlyZSB1cyB0byBtYWtlIHRoZW0gYWxsIHRoZSBzYW1lIHR5cGUuCldlIGNhbiBjb2VyY2UgaXQgaW50byBhIG1hdHJpeCB1c2luZyBgYXMubWF0cml4KClgLCBpbiB3aGljaCBjYXNlIFIgd2lsbApwaWNrIGEgdHlwZSB0aGF0IGl0IGNhbiBjb252ZXJ0IGV2ZXJ5dGhpbmcgdG8uCldoYXQgZG9lcyBpdCBjaG9vc2U/CgpgYGB7ciBtYXRyaXh9CiMgQ29lcmNlIGBnZW5lX2RmYCBpbnRvIGEgbWF0cml4CmdlbmVfbWF0cml4IDwtIGFzLm1hdHJpeChnZW5lX2RmKQpgYGAKCmBgYHtyIG1hdHJpeC10eXBlLCBsaXZlID0gVFJVRX0KIyBFeHBsb3JlIHRoZSBzdHJ1Y3R1cmUgb2YgdGhlIGBnZW5lX21hdHJpeGAgb2JqZWN0CnN0cihnZW5lX21hdHJpeCkKYGBgCgpXaGlsZSB0aGF0IHdvcmtlZCwgaXQgaXMgcmFyZSB0aGF0IHdlIHdhbnQgbnVtYmVycyBjb252ZXJ0ZWQgdG8gdGV4dCwgc28gd2UgYXJlCmdvaW5nIHRvIHNlbGVjdCBvbmx5IHRoZSBjb2x1bW5zIHdpdGggbnVtZXJpYyB2YWx1ZXMgYmVmb3JlIGNvbnZlcnRpbmcgaXQgdG8gYSBtYXRyaXguCldlIGNhbiBkbyB0aGlzIG1vc3QgZWFzaWx5IGJ5IHJlbW92aW5nIHRoZSBmaXJzdCBjb2x1bW4sIHdoaWNoIGNvbnRhaW5zIHRoZSBnZW5lIG5hbWVzIApzdG9yZWQgYXMgY2hhcmFjdGVyIHZhbHVlcy4KCmBgYHtyIG1hdHJpeC1udW1lcmljLCBsaXZlID0gVFJVRX0KIyBMZXQncyBzYXZlIGEgbmV3IG1hdHJpeCBvYmplY3QgbmFtZXMgYGdlbmVfbnVtX21hdHJpeGAgY29udGFpbmluZyBvbmx5CiMgdGhlIG51bWVyaWMgdmFsdWVzCmdlbmVfbnVtX21hdHJpeCA8LSBhcy5tYXRyaXgoZ2VuZV9kZlssIC0xXSkKCiMgRXhwbG9yZSB0aGUgc3RydWN0dXJlIG9mIHRoZSBgZ2VuZV9udW1fbWF0cml4YCBvYmplY3QKc3RyKGdlbmVfbnVtX21hdHJpeCkKYGBgCgpXaHkgZG8gd2UgaGF2ZSBhIGBbLCAtMV1gIGFmdGVyIGBnZW5lX2RmYCBpbiB0aGUgYWJvdmUgY2h1bms/CgpOb3cgdGhhdCB0aGUgbWF0cml4IGlzIGFsbCBudW1iZXJzLCB3ZSBjYW4gZG8gdGhpbmdzIGxpa2UgY2FsY3VsYXRlIHRoZSBjb2x1bW4Kb3Igcm93IHN0YXRpc3RpY3MgdXNpbmcgYGFwcGx5KClgLgoKYGBge3Igcm93bWVhbnN9CiMgQ2FsY3VsYXRlIHJvdyBtZWFucwpnZW5lX21lYW5zIDwtIGFwcGx5KGdlbmVfbnVtX21hdHJpeCwgMSwgbWVhbikgIyBOb3RpY2Ugd2UgYXJlIHVzaW5nIDEgaGVyZQoKIyBIb3cgbG9uZyB3aWxsIGBnZW5lX21lYW5zYCBiZT8gCmxlbmd0aChnZW5lX21lYW5zKQpgYGAKCk5vdGUgdGhhdCB3ZSBjYW4gb2J0YWluIHRoZSBzYW1lIHJlc3VsdHMgaWYgd2Ugc2VsZWN0IGp1c3QgdGhlIGNvbHVtbnMgd2l0aCBudW1lcmljCnZhbHVlcyBmcm9tIHRoZSBgZ2VuZV9kZmAgZGF0YSBmcmFtZS4KVGhpcyBhbGxvd3MgUiB0byBkbyB0aGUgYXMubWF0cml4KCkgY29lcmNpb24gYXV0b21hdGljYWxseSwgYW5kIGNhbiBiZSBhIGhhbmR5IHNob3J0Y3V0CmlmIHlvdSBoYXZlIGEgKm1vc3RseSogbnVtZXJpYyBkYXRhIGZyYW1lLiAKCmBgYHtyIHJvd21lYW5zLWRhdGFmcmFtZX0KIyBDYWxjdWxhdGUgcm93IG1lYW5zIHVzaW5nIHRoZSBgZ2VuZV9kZmAgb2JqZWN0IGFmdGVyIHJlbW92aW5nIHRoZSBjaGFyYWN0ZXIgY29sdW1uCiMgYXBwbHkoKSBjb252ZXJ0cyB0aGlzIHRvIGEgbWF0cml4IGludGVybmFsbHkKZ2VuZV9tZWFuc19mcm9tX2RmIDwtIGFwcGx5KGdlbmVfZGZbLCAtMV0sIDEsIG1lYW4pIAoKIyBMZXQncyBjaGVjayB0aGF0IHRoZSB0d28gZ2VuZSBtZWFucyBvYmplY3RzIGFyZSBlcXVhbAphbGwuZXF1YWwoZ2VuZV9tZWFucywgZ2VuZV9tZWFuc19mcm9tX2RmKQpgYGAKCk5vdyBsZXQncyBpbnZlc3RpZ2F0ZSB0aGUgc2FtZSBzZXQgdXAsIGJ1dCB1c2UgMiB0byBgYXBwbHlgIG92ZXIgdGhlIGNvbHVtbnMgb2YKb3VyIG1hdHJpeC4KCmBgYHtyIGNvbG1lYW5zfQojIENhbGN1bGF0ZSBzYW1wbGUgbWVhbnMKc2FtcGxlX21lYW5zIDwtIGFwcGx5KGdlbmVfbnVtX21hdHJpeCwgMiwgbWVhbikgIyBOb3RpY2Ugd2UgdXNlIDIgaGVyZQoKIyBIb3cgbG9uZyB3aWxsIGBzYW1wbGVfbWVhbnNgIGJlPyAKbGVuZ3RoKHNhbXBsZV9tZWFucykKYGBgCgpXZSBjYW4gcHV0IHRoZSBnZW5lIG5hbWVzIGJhY2sgaW50byB0aGUgbnVtZXJpYyBtYXRyaXggb2JqZWN0IGJ5CmFzc2lnbmluZyB0aGVtIGFzIHJvd25hbWVzLgoKYGBge3IgbWF0cml4LXJvd25hbWVzLCBsaXZlID0gVFJVRX0KIyBBc3NpZ24gdGhlIGdlbmUgbmFtZXMgZnJvbSBnZW5lX2RmJEdlbmUgdG8gdGhlIGBnZW5lX251bV9tYXRyaXhgIG9iamVjdCB1c2luZwojIHRoZSBgcm93bmFtZXMoKWAgZnVuY3Rpb24Kcm93bmFtZXMoZ2VuZV9udW1fbWF0cml4KSA8LSBnZW5lX2RmJEdlbmUKCiMgRXhwbG9yZSB0aGUgYGdlbmVfbnVtX21hdHJpeGAgb2JqZWN0CmhlYWQoZ2VuZV9udW1fbWF0cml4KQpgYGAKClJvdyBuYW1lcyBsaWtlIHRoaXMgY2FuIGJlIHZlcnkgY29udmVuaWVudCBmb3Iga2VlcGluZyBtYXRyaWNlcyBvcmdhbml6ZWQsIGJ1dApyb3cgbmFtZXMgKGFuZCBjb2x1bW4gbmFtZXMpIGNhbiBiZSBsb3N0IG9yIG1pc29yZGVyZWQgaWYgeW91IGFyZSBub3QgY2FyZWZ1bCwKZXNwZWNpYWxseSBkdXJpbmcgaW5wdXQgYW5kIG91dHB1dCwgc28gdHJlYXQgdGhlbSB3aXRoIGNhcmUuCgpBbHRob3VnaCB0aGUgYGFwcGx5YCBmdW5jdGlvbnMgbWF5IG5vdCBiZSBhcyBlYXN5IHRvIHVzZSBhcyB0aGUgdGlkeXZlcnNlIApmdW5jdGlvbnMsIGZvciBzb21lIGFwcGxpY2F0aW9ucywgYGFwcGx5YCBtZXRob2RzIGNhbiBiZSBiZXR0ZXIgc3VpdGVkLgpJbiB0aGlzIHdvcmtzaG9wLCB3ZSB3aWxsIG5vdCBkZWx2ZSB0b28gZGVlcGx5IGludG8gdGhlIHZhcmlvdXMgb3RoZXIgYXBwbHkgCmZ1bmN0aW9ucyAoYHRhcHBseSgpYCwgYGxhcHBseSgpYCwgZXRjLikgYnV0IHlvdSBjYW4gcmVhZCBtb3JlIGluZm9ybWF0aW9uIGFib3V0IAp0aGVtIFtoZXJlXShodHRwczovL3d3dy5ndXJ1OTkuY29tL3ItYXBwbHktc2FwcGx5LXRhcHBseS5odG1sKS4KCiMjIFRoZSBkcGx5cjo6am9pbiBmdW5jdGlvbnMKCkxldCdzIHNheSB3ZSBoYXZlIGEgc2NlbmFyaW8gd2hlcmUgd2UgaGF2ZSB0d28gZGF0YSBmcmFtZXMgdGhhdCB3ZSB3b3VsZCBsaWtlIHRvIApjb21iaW5lLiAKUmVjYWxsIHRoYXQgYHN0YXRzX2RmYCBhbmQgYGdlbmVfZGZgIGFyZSBkYXRhIGZyYW1lcyB0aGF0IGNvbnRhaW4gaW5mb3JtYXRpb24gCmFib3V0IHNvbWUgb2YgdGhlIHNhbWUgZ2VuZXMuClRoZSBbYGRwbHlyOjpqb2luYCBmYW1pbHkgb2YgZnVuY3Rpb25zXShodHRwczovL2RwbHlyLnRpZHl2ZXJzZS5vcmcvcmVmZXJlbmNlL2pvaW4uaHRtbCkgCmFyZSB1c2VmdWwgZm9yIHZhcmlvdXMgc2NlbmFyaW9zIG9mIGNvbWJpbmluZyBkYXRhIGZyYW1lcy4gCgpGb3Igbm93LCB3ZSB3aWxsIGZvY3VzIG9uIGBpbm5lcl9qb2luKClgLCB3aGljaCB3aWxsIGNvbWJpbmUgZGF0YSBmcmFtZXMgYnkgb25seQprZWVwaW5nIGluZm9ybWF0aW9uIGFib3V0IG1hdGNoaW5nIHJvd3MgdGhhdCBhcmUgaW4gYm90aCBkYXRhIGZyYW1lcy4KV2UgbmVlZCB0byB1c2UgdGhlIGBieWAgYXJndW1lbnQgdG8gZGVzaWduYXRlIHdoYXQgY29sdW1uKHMpIApzaG91bGQgYmUgdXNlZCBhcyBhIGtleSB0byBtYXRjaCB0aGUgZGF0YSBmcmFtZXMuCkluIHRoaXMgY2FzZSB3ZSB3YW50IHRvIG1hdGNoIHRoZSBnZW5lIGluZm9ybWF0aW9uIGJldHdlZW4gdGhlIHR3bywgc28gd2Ugd2lsbCAKc3BlY2lmeSB0aGF0IHdlIHdhbnQgdG8gY29tcGFyZSB2YWx1ZXMgaW4gdGhlIGBlbnNlbWJsX2lkYCBjb2x1bW4gZnJvbSAKYHN0YXRzX2RmYCB0byB0aGUgYEdlbmVgIGNvbHVtbiBmcm9tIGBnZW5lX2RmYC4KCmBgYHtyIGlubmVyLWpvaW59CnN0YXRzX2RmICU+JSAKICBpbm5lcl9qb2luKGdlbmVfZGYsIGJ5ID0gYygnZW5zZW1ibF9pZCcgPSAnR2VuZScpKSAKYGBgCgojIyBTYXZlIGRhdGEgdG8gZmlsZXMKCiMjIyMgU2F2ZSB0byBUU1YgZmlsZXMKCkxldCdzIHdyaXRlIHNvbWUgb2YgdGhlIGRhdGEgZnJhbWVzIHdlIGNyZWF0ZWQgdG8gYSBmaWxlLgpUbyBkbyB0aGlzLCB3ZSBjYW4gdXNlIHRoZSBgcmVhZHJgIGxpYnJhcnkgb2YgYF93cml0ZSgpYCBmdW5jdGlvbnMuIApUaGUgZmlyc3QgYXJndW1lbnQgb2YgYHdyaXRlX3RzdigpYCBpcyB0aGUgZGF0YSB3ZSB3YW50IHRvIHdyaXRlLCBhbmQgdGhlIHNlY29uZCAKYXJndW1lbnQgaXMgYSBjaGFyYWN0ZXIgc3RyaW5nIHRoYXQgZGVzY3JpYmVzIHRoZSBwYXRoIHRvIHRoZSBuZXcgZmlsZSB3ZSB3b3VsZCAKbGlrZSB0byBjcmVhdGUuClJlbWVtYmVyIHRoYXQgd2UgY3JlYXRlZCBhIGByZXN1bHRzYCBkaXJlY3RvcnkgdG8gcHV0IG91ciBvdXRwdXQgaW4sIApidXQgaWYgd2Ugd2FudCB0byBzYXZlIG91ciBkYXRhIHRvIGEgZGlyZWN0b3J5IG90aGVyIHRoYW4gb3VyIHdvcmtpbmcgZGlyZWN0b3J5LCAKd2UgbmVlZCB0byBzcGVjaWZ5IHRoaXMuIApUaGlzIGlzIHdoYXQgd2Ugd2lsbCB1c2UgdGhlIGBmaWxlLnBhdGgoKWAgZnVuY3Rpb24gZm9yLiAKTGV0J3MgbG9vayBpbiBhIGJpdCBtb3JlIGRldGFpbCB3aGF0IGBmaWxlLnBhdGgoKWAgZG9lcywgYnkgZXhhbWluaW5nIHRoZSAKcmVzdWx0cyBvZiB0aGUgZnVuY3Rpb24gaW4gdGhlIGV4YW1wbGVzIGJlbG93LgoKYGBge3IgZmlsZS1wYXRoLXF1aXp9CiMgV2hpY2ggb2YgdGhlc2UgZmlsZSBwYXRocyBpcyB3aGF0IHdlIHdhbnQgdG8gdXNlIHRvIHNhdmUgb3VyIGRhdGEgdG8gdGhlCiMgcmVzdWx0cyBkaXJlY3Rvcnkgd2UgY3JlYXRlZCBhdCB0aGUgYmVnaW5uaW5nIG9mIHRoaXMgbm90ZWJvb2s/CmZpbGUucGF0aCgiZG9ja2VyLWluc3RhbGwiLCAic3RhdHNfc3VtbWFyeS50c3YiKQpmaWxlLnBhdGgoInJlc3VsdHMiLCAic3RhdHNfc3VtbWFyeS50c3YiKQpmaWxlLnBhdGgoInN0YXRzX3N1bW1hcnkudHN2IiwgInJlc3VsdHMiKQpgYGAKClJlcGxhY2UgYDxORVdfRklMRV9QQVRIPmAgYmVsb3cgd2l0aCB0aGUgYGZpbGUucGF0aCgpYCBzdGF0ZW1lbnQgZnJvbSBhYm92ZSB0aGF0IAp3aWxsIHN1Y2Nlc3NmdWxseSBzYXZlIG91ciBmaWxlIHRvIHRoZSBgcmVzdWx0c2AgZm9sZGVyIAoKYGBge3IgZXZhbD1GQUxTRX0KIyBXcml0ZSBvdXIgZGF0YSBmcmFtZSB0byBhIFRTViBmaWxlCnJlYWRyOjp3cml0ZV90c3Yoc3RhdHNfc3VtbWFyeV9kZiwgPE5FV19GSUxFX1BBVEg+KQpgYGAKCkNoZWNrIGluIHlvdXIgYHJlc3VsdHNgIGRpcmVjdG9yeSB0byBzZWUgaWYgeW91ciBuZXcgZmlsZSBoYXMgc3VjY2Vzc2Z1bGx5IHNhdmVkLgoKIyMjIyBTYXZlIHRvIFJEUyBmaWxlcwoKRm9yIHRoaXMgZXhhbXBsZSB3ZSBoYXZlIGJlZW4gd29ya2luZyB3aXRoIGRhdGEgZnJhbWVzLCB3aGljaCBhcmUgY29udmVuaWVudGx5IApyZXByZXNlbnRlZCBhcyBUU1Ygb3IgQ1NWIHRhYmxlcy4gCkhvd2V2ZXIsIGluIG90aGVyIHNpdHVhdGlvbnMgd2UgbWF5IHdhbnQgdG8gc2F2ZSBtb3JlIGNvbXBsaWNhdGVkIG9yIHZlcnkgbGFyZ2UgCmRhdGEgc3RydWN0dXJlcywgUkRTIChSIERhdGEgU2VyaWFsaXplZC9TaW5nbGUpIGZpbGVzIG1heSBiZSBhIGJldHRlciBvcHRpb24gZm9yCnNhdmluZyBvdXIgZGF0YS4KUkRTIGlzIFIncyBzcGVjaWFsIGZpbGUgZm9ybWF0IGZvciBob2xkaW5nIGRhdGEgZXhhY3RseSBhcyB5b3UgaGF2ZSBpdCBpbiB5b3VyIApSIGVudmlyb25tZW50LiAKUkRTIGZpbGVzIGNhbiBhbHNvIGJlIGNvbXByZXNzZWQsIG1lYW5pbmcgdGhleSB3aWxsIHRha2UgdXAgbGVzcyBzcGFjZSBvbiB5b3VyIApjb21wdXRlci4gCkxldCdzIHNhdmUgb3VyIGRhdGEgdG8gYW4gUkRTIGZpbGUgaW4gb3VyIGByZXN1bHRzYCBmb2xkZXIuCllvdSB3aWxsIG5lZWQgdG8gcmVwbGFjZSB0aGUgYC50c3ZgIHdpdGggYC5SRFNgLCBidXQgeW91IGNhbiB1c2Ugd2hhdCB3ZSAKZGV0ZXJtaW5lZCBhcyBvdXIgZmlsZSBwYXRoIGZvciB0aGUgbGFzdCBjaHVuayBhcyB5b3VyIHRlbXBsYXRlLiAKCmBgYHtyIGV2YWw9RkFMU0V9CiMgV3JpdGUgeW91ciBvYmplY3QgdG8gYW4gUkRTIGZpbGUKcmVhZHI6OndyaXRlX3JkcyhzdGF0c19zdW1tYXJ5LCA8UFVUX0NPUlJFQ1RfRklMRV9QQVRIX0hFUkU+KQpgYGAKCiMjIyMgUmVhZCBhbiBSRFMgZmlsZQoKU2luY2Ugbm93IHlvdSBoYXZlIGxlYXJuZWQgdGhlIGByZWFkcmAgZnVuY3Rpb25zOiBgcmVhZF90c3YoKWAsIGB3cml0ZV90c3YoKWAsIAphbmQgbm93LCBgd3JpdGVfcmRzKClgLCB3aGF0IGRvIHlvdSBzdXBwb3NlIHRoZSBmdW5jdGlvbiB5b3Ugd2lsbCBuZWVkIHRvIHJlYWQgCnlvdXIgUkRTIGZpbGUgaXMgY2FsbGVkPyAKVXNlIHRoYXQgZnVuY3Rpb24gaGVyZSB0byByZS1pbXBvcnQgeW91ciBkYXRhIGluIHRoZSBjaHVuayB3ZSBzZXQgdXAgZm9yIHlvdQpiZWxvdy4KCmBgYHtyIGV2YWw9RkFMU0V9CiMgUmVhZCBpbiB5b3VyIFJEUyBmaWxlCnJlaW1wb3J0X2RmIDwtIDxQVVRfRlVOQ1RJT05fTkFNRT4oZmlsZS5wYXRoKCJyZXN1bHRzIiwgInN0YXRzX2NsZWFuLlJEUyIpKQpgYGAKCkFzIGlzIGdvb2QgcHJhY3RpY2UsIHdlIHdpbGwgZW5kIHRoaXMgc2Vzc2lvbiBieSBwcmludGluZyBvdXQgb3VyIHNlc3Npb24gaW5mby4gCgojIyMgU2Vzc2lvbiBJbmZvCgpgYGB7cn0KIyBQcmludCBvdXQgdGhlIHZlcnNpb25zIGFuZCBwYWNrYWdlcyB3ZSBhcmUgdXNpbmcgaW4gdGhpcyBzZXNzaW9uCnNlc3Npb25JbmZvKCkKYGBgCg==
      
 
 
 
diff --git a/machine-learning/01-openpbta_heatmap.nb.html b/machine-learning/01-openpbta_heatmap.nb.html
index ed74a473..222fa3cb 100644
--- a/machine-learning/01-openpbta_heatmap.nb.html
+++ b/machine-learning/01-openpbta_heatmap.nb.html
@@ -611,7 +611,7 @@ 
      Annotation
      
 
 
      
 
 
      
 
diff --git a/machine-learning/02-openpbta_consensus_clustering.nb.html b/machine-learning/02-openpbta_consensus_clustering.nb.html
index 936a8a41..bdeaf8cf 100644
--- a/machine-learning/02-openpbta_consensus_clustering.nb.html
+++ b/machine-learning/02-openpbta_consensus_clustering.nb.html
@@ -907,7 +907,7 @@ 
      PCA
      
 
 
 
      
-
+
 
 
 
diff --git a/machine-learning/04-openpbta_plot_LV.nb.html b/machine-learning/04-openpbta_plot_LV.nb.html
index 2516ed0d..bbbe6235 100644
--- a/machine-learning/04-openpbta_plot_LV.nb.html
+++ b/machine-learning/04-openpbta_plot_LV.nb.html
@@ -508,7 +508,7 @@ 
      PLIER results
      
 
 
      
 
 
      
 
diff --git a/pathway-analysis/01-overrepresentation_analysis-live.Rmd b/pathway-analysis/01-overrepresentation_analysis-live.Rmd
index 8f08a7c0..957cbe79 100644
--- a/pathway-analysis/01-overrepresentation_analysis-live.Rmd
+++ b/pathway-analysis/01-overrepresentation_analysis-live.Rmd
@@ -8,6 +8,17 @@ author: CCDL for ALSF
 date: 2020
 ---
 
+## Objectives
+
+This notebook will demonstrate how to:
+
+- Perform gene identifier conversion with [`AnnotationDBI` annotation packages](https://bioconductor.org/packages/release/bioc/vignettes/AnnotationDbi/inst/doc/IntroToAnnotationPackages.pdf)
+- Access [Molecular Signatures Database gene set collections](https://www.gsea-msigdb.org/gsea/msigdb/collections.jsp) via the `msigdbr` package
+- Prepare gene sets for over-representation analysis, including an appropriate background set
+- Perform over-representation analysis with the `clusterProfiler` package
+
+---
+
 In this notebook, we'll cover a type of pathway or gene set analysis called over-representation analysis (ORA).
 The idea behind ORA is relatively straightforward: given a set of genes, do these genes overlap with a pathway more than we expect by chance?
 The simplicity of only requiring an input gene set (sort of, more on that below) can be attractive.
@@ -219,9 +230,9 @@ vs_low_df <- vs_low_df %>%
 ```
 
 **Now we'll read in our data frame of DGE results from another comparison.**
-To save us some time during instruction, we've already done the gene identifier conversion and filtering to remove `NA` values in [this notebook](https://github.com/AlexsLemonade/training-modules/tree/master/pathway-analysis/setup/03-leukemia_DGE.Rmd). 
+To save us some time during instruction, we've already done the gene identifier conversion and filtering to remove `NA` values in [this notebook](https://github.com/AlexsLemonade/training-modules/tree/master/pathway-analysis/setup/01-leukemia_DGE.Rmd). 
 We took a different series of steps to achieve the same thing, which is often possible in R!
- 
+
 
 ```{r read_in_unsorted}
 vs_unsorted_df <- readr::read_tsv(vs_unsorted_file)
@@ -268,8 +279,7 @@ The authors sorted populations of primary leukemia cells and examined the stem c
 
 We compared the population that the authors identified as having high stem cell capacity to a low stem cell capacity population.
 We also compared the high stem cell capacity cells to a mix of populations (e.g., unsorted cells).
-You can see the code in [here](https://github.com/AlexsLemonade/training-modules/tree/master/pathway-analysis/setup/03-leukemia_DGE.Rmd).
-
+You can see the code in [here](https://github.com/AlexsLemonade/training-modules/tree/master/pathway-analysis/setup/01-leukemia_DGE.Rmd).
 
 We're interested in what pathways are over-represented in genes that specifically distinguish the high capacity population from the low capacity population.
 
diff --git a/pathway-analysis/01-overrepresentation_analysis.nb.html b/pathway-analysis/01-overrepresentation_analysis.nb.html
index a1a2bd94..07850257 100644
--- a/pathway-analysis/01-overrepresentation_analysis.nb.html
+++ b/pathway-analysis/01-overrepresentation_analysis.nb.html
@@ -345,87 +345,125 @@ 
      Set up
      
 
      Libraries
      
 
 
-
+
 
      # Pipes
 library(magrittr)
 # Package we'll use to 
 library(clusterProfiler)
      
 
-
+
+
      
+
+
 
      clusterProfiler v3.18.1  For help: https://guangchuangyu.github.io/software/clusterProfiler
 
 If you use clusterProfiler in published research, please cite:
-Guangchuang Yu, Li-Gen Wang, Yanyan Han, Qing-Yu He. clusterProfiler: an R package for comparing biological themes among gene clusters. OMICS: A Journal of Integrative Biology. 2012, 16(5):284-287.
-
-Attaching package: ‘clusterProfiler’
-
-The following object is masked from ‘package:stats’:
+Guangchuang Yu, Li-Gen Wang, Yanyan Han, Qing-Yu He. clusterProfiler: an R package for comparing biological themes among gene clusters. OMICS: A Journal of Integrative Biology. 2012, 16(5):284-287.
      
+
+
+
      
+Attaching package: 'clusterProfiler'
      
+
+
+
      The following object is masked from 'package:stats':
 
     filter
      
-
-
+
+
 
      # Package that contains MSigDB gene sets in tidy format
 library(msigdbr)
 # Mus musculus annotation package we'll use for gene identifier conversion
 library(org.Mm.eg.db)
      
 
-
-
      Loading required package: AnnotationDbi
-Loading required package: stats4
-Loading required package: BiocGenerics
-Loading required package: parallel
-
-Attaching package: ‘BiocGenerics’
-
-The following objects are masked from ‘package:parallel’:
-
-    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, clusterExport, clusterMap, parApply,
-    parCapply, parLapply, parLapplyLB, parRapply, parSapply, parSapplyLB
-
-The following objects are masked from ‘package:stats’:
-
-    IQR, mad, sd, var, xtabs
-
-The following objects are masked from ‘package:base’:
-
-    anyDuplicated, append, as.data.frame, basename, cbind, colnames, dirname, do.call, duplicated,
-    eval, evalq, Filter, Find, get, grep, grepl, intersect, is.unsorted, lapply, Map, mapply, match,
-    mget, order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank, rbind, Reduce, rownames,
-    sapply, setdiff, sort, table, tapply, union, unique, unsplit, which.max, which.min
-
-Loading required package: Biobase
-Welcome to Bioconductor
-
-    Vignettes contain introductory material; view with 'browseVignettes()'. To cite Bioconductor, see
-    'citation("Biobase")', and for packages 'citation("pkgname")'.
-
-Loading required package: IRanges
-Loading required package: S4Vectors
-
-Attaching package: ‘S4Vectors’
-
-The following object is masked from ‘package:clusterProfiler’:
-
-    rename
-
-The following object is masked from ‘package:base’:
-
-    expand.grid
-
-
-Attaching package: ‘IRanges’
-
-The following object is masked from ‘package:clusterProfiler’:
-
-    slice
-
-
-Attaching package: ‘AnnotationDbi’
-
-The following object is masked from ‘package:clusterProfiler’:
+
+
      Loading required package: AnnotationDbi
      
+
+
+
      Loading required package: stats4
      
+
+
+
      Loading required package: BiocGenerics
      
+
+
+
      Loading required package: parallel
      
+
+
+
      
+Attaching package: 'BiocGenerics'
      
+
+
+
      The following objects are masked from 'package:parallel':
+
+    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
+    clusterExport, clusterMap, parApply, parCapply, parLapply,
+    parLapplyLB, parRapply, parSapply, parSapplyLB
      
+
+
+
      The following objects are masked from 'package:stats':
+
+    IQR, mad, sd, var, xtabs
      
+
+
+
      The following objects are masked from 'package:base':
+
+    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
+    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
+    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
+    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
+    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
+    union, unique, unsplit, which.max, which.min
      
+
+
+
      Loading required package: Biobase
      
+
+
+
      Welcome to Bioconductor
+
+    Vignettes contain introductory material; view with
+    'browseVignettes()'. To cite Bioconductor, see
+    'citation("Biobase")', and for packages 'citation("pkgname")'.
      
+
+
+
      Loading required package: IRanges
      
+
+
+
      Loading required package: S4Vectors
      
+
+
+
      
+Attaching package: 'S4Vectors'
      
+
+
+
      The following object is masked from 'package:clusterProfiler':
+
+    rename
      
+
+
+
      The following object is masked from 'package:base':
+
+    expand.grid
      
+
+
+
      
+Attaching package: 'IRanges'
      
+
+
+
      The following object is masked from 'package:clusterProfiler':
+
+    slice
      
+
+
+
      
+Attaching package: 'AnnotationDbi'
      
+
+
+
      The following object is masked from 'package:clusterProfiler':
 
     select
      
-
+
+
+
      
+
 
 
 
@@ -435,7 +473,7 @@ 
      Directories and files
      
 
      Directories
      
 
 
-
+
 
      # We'll create a directory to specifically hold the ORA results if it doesn't
 # exist yet
 results_dir <- file.path("results", "leukemia")
@@ -451,7 +489,7 @@ 
      Input files
      
 
      For our ORA example, we’re going to use two tables of differential gene expression (DGE) analysis results.
      
 
 
-
+
 
      input_dir <- file.path("data", "leukemia")
 
 # This file contains the DGE results for a cell population with high stem cell
@@ -472,7 +510,7 @@ 
      Output files
      
 
      We’ll save the table of ORA results (e.g., p-values).
      
 
 
-
+
 
      kegg_results_file <- file.path(results_dir, "leukemia_kegg_ora_results.tsv")
      
 
 
@@ -486,22 +524,20 @@ 
      Gene sets
      
 
      Let’s take a look at what organisms the package supports.
      
 
 
-
+
 
      msigdbr_species()
      
 
-
 
      
 
 
      
-
 
 
 
      The results we’re interested in here come from mouse samples, so we can obtain just the gene sets relevant to M. musculus with the species argument to msigdbr().
      
 
 
-
+
 
      mm_msigdb_df <- msigdbr(species = "Mus musculus")
      
 
 
@@ -522,7 +558,7 @@ 
      Gene sets
      
 
      And are a subset of C2: curated gene sets. Specifically, we will use the KEGG (Kyoto Encyclopedia of Genes and Genomes) pathways.
      
 
 
-
+
 
      # Filter the mouse data frame to the KEGG pathways that are included in the
 # curated gene sets
 mm_kegg_df <- mm_msigdb_df %>%
@@ -534,15 +570,51 @@ 
      Gene sets
      
 
      Note: We could specified that we wanted the KEGG gene sets using the category and subcategory arguments of msigdbr(), but we’re going for general steps!
      
 
 
-
+
 
      colnames(mm_kegg_df)
      
 
-
-
       [1] "gs_cat"               "gs_subcat"            "gs_name"              "entrez_gene"         
- [5] "gene_symbol"          "human_entrez_gene"    "human_gene_symbol"    "gs_id"               
- [9] "gs_pmid"              "gs_geoid"             "gs_exact_source"      "gs_url"              
-[13] "gs_description"       "species_name"         "species_common_name"  "ortholog_sources"    
-[17] "num_ortholog_sources"
      
+
+
       [1] "gs_cat"               "gs_subcat"            "gs_name"             
+ [4] "entrez_gene"          "gene_symbol"          "human_entrez_gene"   
+ [7] "human_gene_symbol"    "gs_id"                "gs_pmid"             
+[10] "gs_geoid"             "gs_exact_source"      "gs_url"              
+[13] "gs_description"       "species_name"         "species_common_name" 
+[16] "ortholog_sources"     "num_ortholog_sources"
      
+
+
+
      gs_cat
+
+gs_subcat
+
+gs_name
+
+entrez_gene
+
+gene_symbol
+
+human_entrez_gene
+
+human_gene_symbol
+
+gs_id
+
+gs_pmid
+
+gs_geoid
+
+gs_exact_source
+
+gs_url
+
+gs_description
+
+species_name
+
+species_common_name
+
+ortholog_sources
+
+num_ortholog_sources
      
 
 
 
@@ -552,12 +624,12 @@ 
      Gene sets
      
 
      Read in DGE results and prep
      
 
 
-
+
 
      vs_low_df <- readr::read_tsv(vs_low_file)
      
 
-
+
 
      
-── Column specification ───────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ────────────────────────────────────────────────────────
 cols(
   Gene = col_character(),
   baseMean = col_double(),
@@ -567,22 +639,20 @@ 
      Read in DGE results and prep
      
   pvalue = col_double(),
   padj = col_double()
 )
      
-
+
 
 
 
      Let’s take a peek at the top of the DGE results data frame.
      
 
 
-
+
 
      head(vs_low_df)
      
 
-
 
      
 
 
      
-
 
 
 
      
@@ -593,14 +663,64 @@ 
      Gene identifier conversion
      
 
      We can see what types of IDs are available to us in an annotation package with keytypes().
      
 
 
-
+
 
      keytypes(org.Mm.eg.db)
      
 
-
-
       [1] "ACCNUM"       "ALIAS"        "ENSEMBL"      "ENSEMBLPROT"  "ENSEMBLTRANS" "ENTREZID"     "ENZYME"      
- [8] "EVIDENCE"     "EVIDENCEALL"  "GENENAME"     "GO"           "GOALL"        "IPI"          "MGI"         
-[15] "ONTOLOGY"     "ONTOLOGYALL"  "PATH"         "PFAM"         "PMID"         "PROSITE"      "REFSEQ"      
-[22] "SYMBOL"       "UNIGENE"      "UNIPROT"     
      
+
+
       [1] "ACCNUM"       "ALIAS"        "ENSEMBL"      "ENSEMBLPROT"  "ENSEMBLTRANS"
+ [6] "ENTREZID"     "ENZYME"       "EVIDENCE"     "EVIDENCEALL"  "GENENAME"    
+[11] "GO"           "GOALL"        "IPI"          "MGI"          "ONTOLOGY"    
+[16] "ONTOLOGYALL"  "PATH"         "PFAM"         "PMID"         "PROSITE"     
+[21] "REFSEQ"       "SYMBOL"       "UNIGENE"      "UNIPROT"     
      
+
+
+
      ACCNUM
+
+ALIAS
+
+ENSEMBL
+
+ENSEMBLPROT
+
+ENSEMBLTRANS
+
+ENTREZID
+
+ENZYME
+
+EVIDENCE
+
+EVIDENCEALL
+
+GENENAME
+
+GO
+
+GOALL
+
+IPI
+
+MGI
+
+ONTOLOGY
+
+ONTOLOGYALL
+
+PATH
+
+PFAM
+
+PMID
+
+PROSITE
+
+REFSEQ
+
+SYMBOL
+
+UNIGENE
+
+UNIPROT
      
 
 
 
@@ -608,7 +728,7 @@ 
      Gene identifier conversion
      
 
      The function we will use to map from Ensembl gene IDs to gene symbols is called mapIds().
      
 
 
-
+
 
      # This returns a named vector which we can convert to a data frame, where
 # the keys (Ensembl IDs) are the names
 symbols_vector <- mapIds(org.Mm.eg.db,  # Specify the annotation package
@@ -624,10 +744,10 @@ 
      Gene identifier conversion
      
                          # first one. This is default behavior!
                          multiVals = "first") 
      
 
-
+
 
      'select()' returned 1:many mapping between keys and columns
      
-
-
+
+
 
      # We would like a data frame we can join to the DGE results
 symbols_df <- data.frame(
   ensembl_id = names(symbols_vector),
@@ -641,7 +761,7 @@ 
      Gene identifier conversion
      
 
      Let’s do this first for the comparison to the low stem cell capacity population.
      
 
 
-
+
 
      vs_low_df <- symbols_df %>%
   # An *inner* join will only return rows that are in both data frames
   dplyr::inner_join(vs_low_df,
@@ -658,7 +778,7 @@ 
      Drop NA values
      
 
      Let’s filter to rows that do not have any NA using a function tidyr::drop_na(). This will also drop genes that have an Ensembl gene identifier but no gene symbol!
      
 
 
-
+
 
      # Remove rows that are not complete (e.g., contain NAs) by filtering to only 
 # complete rows
 vs_low_df <- vs_low_df %>%
@@ -669,12 +789,12 @@ 
      Drop NA values
      
 
      Now we’ll read in our data frame of DGE results from another comparison. To save us some time during instruction, we’ve already done the gene identifier conversion and filtering to remove NA values in this notebook. We took a different series of steps to achieve the same thing, which is often possible in R!
      
 
 
-
+
 
      vs_unsorted_df <- readr::read_tsv(vs_unsorted_file)
      
 
-
+
 
      
-── Column specification ───────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ────────────────────────────────────────────────────────
 cols(
   Gene = col_character(),
   baseMean = col_double(),
@@ -685,7 +805,7 @@ 
      Drop NA values
      
   padj = col_double(),
   gene_symbol = col_character()
 )
      
-
+
 
 
 
      
@@ -702,7 +822,7 @@ 
      Over-representation Analysis (ORA)
      
 
      We’ll call genes that are differentially expressed gene_in_interest and genes that are in the gene set in_gene_set.
      
 
 
-
+
 
      gene_table <- data.frame(
   gene_not_interest = c(2613, 15310),
   gene_in_interest = c(28, 29)
@@ -711,19 +831,17 @@ 
      Over-representation Analysis (ORA)
      
 
 gene_table
      
 
-
 
      
 
 
      
-
 
 
 
      We can assess if the 28 overlapping genes mean that the differentially expressed genes are over-represented in the gene set with the hypergeometric distribution. This corresponds to a one-sided Fisher’s exact test.
      
 
 
-
+
 
      fisher.test(gene_table, alternative = "greater")
      
 
 
@@ -751,7 +869,7 @@ 
      High stem cell capacity ORA
      
 
      We’ll start with the high stem cell capacity vs. low stem cell capacity population comparison. Genes with positive log2 fold-changes (LFC) will be more highly expressed in the high stem cell capacity cells based on how we set up the analysis.
      
 
 
-
+
 
      vs_low_genes <- vs_low_df %>%
   # Filter to the positive LFC and filter based on significance too (padj)
   dplyr::filter(log2FoldChange > 0,
@@ -765,7 +883,7 @@ 
      High stem cell capacity ORA
      
 
      Now, we’ll take the same steps for our other results.
      
 
 
-
+
 
      vs_unsorted_genes <- vs_unsorted_df %>%
   dplyr::filter(log2FoldChange > 0,
                 padj < 0.05) %>%
@@ -776,7 +894,7 @@ 
      High stem cell capacity ORA
      
 
      We want genes that are in the first comparison but not in the second! We can use setdiff(), a base R function for set operations, to get the list that we want.
      
 
 
-
+
 
      # What genes are in the first set but *not* in the second set
 genes_for_ora <- setdiff(vs_low_genes, vs_unsorted_genes) 
      
 
@@ -793,7 +911,7 @@ 
      Background set
      
 
      We can use another function for set operations, intersect(), to get our background set of genes that were included in both comparisons.
      
 
 
-
+
 
      # intersect() will return the genes in both sets - we are using the entire data
 # frame here (complete cases), not just the significant genes
 background_set <- intersect(vs_low_df$gene_symbol,
@@ -812,7 +930,7 @@ 
      Run enricher()
      
 
      Now that we have our background set, our genes of interest, and our pathway information, we’re ready to run ORA using the enricher() function.
      
 
 
-
+
 
      kegg_ora_results <- enricher(
   gene = genes_for_ora,  # Genes of interest
   pvalueCutoff = 0.05,  
@@ -839,7 +957,7 @@ 
      Run enricher()
      
 
      The information we’re most likely interested in is in the results slot. Let’s convert this into a data frame that we can write to file.
      
 
 
-
+
 
      kegg_result_df <- data.frame(kegg_ora_results@result)
      
 
 
@@ -850,25 +968,25 @@ 
      Visualizing results
      
 
      We can use a dot plot to visualize our significant enrichment results.
      
 
 
-
+
 
      enrichplot::dotplot(kegg_ora_results)
      
 
-
+
 
      wrong orderBy parameter; set to default `orderBy = "x"`
      
-
-
-
      
+
+
+
      
 
 
 
 
      We can use an UpSet plot to visualize the overlap between the gene sets that were returned as significant.
      
 
 
-
+
 
      enrichplot::upsetplot(kegg_ora_results)
      
 
-
-
      
+
+
      
 
 
 
@@ -876,18 +994,16 @@ 
      Visualizing results
      
 
      We can look at the geneID column of our results to see what genes overlap; it’s a good idea to take a look.
      
 
 
-
+
 
      kegg_result_df %>%
   # Use dplyr::select() - the name of the pathway is in the ID column
   dplyr::select(ID, geneID)
      
 
-
 
      
 
 
      
-
 
 
 
@@ -895,7 +1011,7 @@ 
      Visualizing results
      
 
      Write results to file
      
 
 
-
+
 
      readr::write_tsv(kegg_result_df, file = kegg_results_file)
      
 
 
@@ -907,50 +1023,65 @@ 
      Write results to file
      
 
      Session Info
      
 
 
-
+
 
      sessionInfo()
      
 
-
+
 
      R version 4.0.3 (2020-10-10)
 Platform: x86_64-pc-linux-gnu (64-bit)
-Running under: Ubuntu 18.04.3 LTS
+Running under: Ubuntu 20.04 LTS
 
 Matrix products: default
-BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
-LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
 
 locale:
- [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
- [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
- [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
 
 attached base packages:
-[1] parallel  stats4    stats     graphics  grDevices utils     datasets  methods   base     
+[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
+[8] methods   base     
 
 other attached packages:
-[1] org.Mm.eg.db_3.12.0    AnnotationDbi_1.52.0   IRanges_2.24.1         S4Vectors_0.28.1      
-[5] Biobase_2.50.0         BiocGenerics_0.36.0    msigdbr_7.2.1          clusterProfiler_3.18.1
-[9] magrittr_2.0.1        
+ [1] org.Mm.eg.db_3.12.0    AnnotationDbi_1.52.0   IRanges_2.24.1        
+ [4] S4Vectors_0.28.1       Biobase_2.50.0         BiocGenerics_0.36.0   
+ [7] msigdbr_7.2.1          clusterProfiler_3.18.1 magrittr_2.0.1        
+[10] optparse_1.6.6        
 
 loaded via a namespace (and not attached):
- [1] enrichplot_1.10.2   bit64_4.0.5         RColorBrewer_1.1-2  tools_4.0.3         R6_2.5.0           
- [6] DBI_1.1.1           colorspace_2.0-0    tidyselect_1.1.0    gridExtra_2.3       bit_4.0.4          
-[11] compiler_4.0.3      cli_2.2.0           scatterpie_0.1.5    labeling_0.4.2      shadowtext_0.0.7   
-[16] scales_1.1.1        readr_1.4.0         stringr_1.4.0       digest_0.6.27       ggupset_0.3.0      
-[21] rmarkdown_2.6       DOSE_3.16.0         pkgconfig_2.0.3     htmltools_0.5.1.1   fastmap_1.1.0      
-[26] rlang_0.4.10        rstudioapi_0.13     RSQLite_2.2.3       farver_2.0.3        generics_0.1.0     
-[31] jsonlite_1.7.2      BiocParallel_1.24.1 GOSemSim_2.16.1     dplyr_1.0.3         GO.db_3.12.1       
-[36] Matrix_1.3-2        fansi_0.4.2         Rcpp_1.0.6          munsell_0.5.0       viridis_0.5.1      
-[41] lifecycle_0.2.0     stringi_1.5.3       yaml_2.2.1          ggraph_2.0.4        MASS_7.3-53        
-[46] plyr_1.8.6          qvalue_2.22.0       grid_4.0.3          blob_1.2.1          ggrepel_0.9.1      
-[51] DO.db_2.9           crayon_1.3.4        lattice_0.20-41     graphlayouts_0.7.1  cowplot_1.1.1      
-[56] splines_4.0.3       hms_1.0.0           knitr_1.30          pillar_1.4.7        fgsea_1.16.0       
-[61] igraph_1.2.6        reshape2_1.4.4      fastmatch_1.1-0     glue_1.4.2          evaluate_0.14      
-[66] downloader_0.4      data.table_1.13.6   renv_0.12.5-2       BiocManager_1.30.10 vctrs_0.3.6        
-[71] tweenr_1.0.1        gtable_0.3.0        purrr_0.3.4         polyclip_1.10-0     tidyr_1.1.2        
-[76] assertthat_0.2.1    cachem_1.0.1        ggplot2_3.3.3       xfun_0.20           ggforce_0.3.2      
-[81] tidygraph_1.2.0     viridisLite_0.3.0   tibble_3.0.5        rvcheck_0.1.8       memoise_1.1.0      
-[86] ellipsis_0.3.1     
      
+ [1] enrichplot_1.10.2   bit64_4.0.5         RColorBrewer_1.1-2 
+ [4] tools_4.0.3         R6_2.5.0            DBI_1.1.1          
+ [7] colorspace_2.0-0    tidyselect_1.1.0    gridExtra_2.3      
+[10] bit_4.0.4           compiler_4.0.3      cli_2.2.0          
+[13] scatterpie_0.1.5    labeling_0.4.2      shadowtext_0.0.7   
+[16] scales_1.1.1        readr_1.4.0         stringr_1.4.0      
+[19] digest_0.6.27       ggupset_0.3.0       rmarkdown_2.6      
+[22] DOSE_3.16.0         pkgconfig_2.0.3     htmltools_0.5.1.1  
+[25] fastmap_1.1.0       rlang_0.4.10        rstudioapi_0.13    
+[28] RSQLite_2.2.3       farver_2.0.3        generics_0.1.0     
+[31] jsonlite_1.7.2      BiocParallel_1.24.1 GOSemSim_2.16.1    
+[34] dplyr_1.0.3         GO.db_3.12.1        Matrix_1.3-2       
+[37] fansi_0.4.2         Rcpp_1.0.6          munsell_0.5.0      
+[40] viridis_0.5.1       lifecycle_0.2.0     stringi_1.5.3      
+[43] yaml_2.2.1          ggraph_2.0.4        MASS_7.3-53        
+[46] plyr_1.8.6          qvalue_2.22.0       grid_4.0.3         
+[49] blob_1.2.1          ggrepel_0.9.1       DO.db_2.9          
+[52] crayon_1.3.4        lattice_0.20-41     graphlayouts_0.7.1 
+[55] cowplot_1.1.1       splines_4.0.3       hms_1.0.0          
+[58] ps_1.5.0            knitr_1.30          pillar_1.4.7       
+[61] fgsea_1.16.0        igraph_1.2.6        reshape2_1.4.4     
+[64] fastmatch_1.1-0     glue_1.4.2          evaluate_0.14      
+[67] downloader_0.4      data.table_1.13.6   BiocManager_1.30.10
+[70] vctrs_0.3.6         tweenr_1.0.1        gtable_0.3.0       
+[73] getopt_1.20.3       purrr_0.3.4         polyclip_1.10-0    
+[76] tidyr_1.1.2         assertthat_0.2.1    cachem_1.0.1       
+[79] ggplot2_3.3.3       xfun_0.20           ggforce_0.3.2      
+[82] tidygraph_1.2.0     viridisLite_0.3.0   tibble_3.0.5       
+[85] rvcheck_0.1.8       memoise_1.1.0       ellipsis_0.3.1     
      
 
 
 
diff --git a/pathway-analysis/02-gene_set_enrichment_analysis-live.Rmd b/pathway-analysis/02-gene_set_enrichment_analysis-live.Rmd
index 99403826..7fced4a3 100644
--- a/pathway-analysis/02-gene_set_enrichment_analysis-live.Rmd
+++ b/pathway-analysis/02-gene_set_enrichment_analysis-live.Rmd
@@ -8,20 +8,27 @@ author: CCDL for ALSF
 date: 2020
 ---
 
+## Objectives
+
+This notebook will demonstrate how to:
+
+- Prepare tabular data of gene-level statistics for use with Gene Set Enrichment Analysis (GSEA), including how to remove duplicate gene identifiers
+- Perform GSEA with the `clusterProfiler` package
+- Visualize GSEA results with the `enrichplot` package
+
+---
+
 In this notebook, we will perform Gene Set Enrichment Analysis (GSEA) on the neuroblastoma cell line differential gene expression (DGE) results we generated during the RNA-seq module.
 
 To refresh our memory, we analyzed data from [Harenza *et al.* (2017)](https://doi.org/10.1038/sdata.2017.33) and specifically tested for DGE between with _MYCN_ amplified cell lines and non-amplified cell lines using `DESeq2`.
 We have a table of results that contains our log2 fold changes and adjusted p-values.
 
-*Note: We've taken an additional step to get more accurate log2 fold change estimates. 
-You can see the code for that in `setup/01-prepare_NB_cell_line.Rmd`.*
-
-GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian, Tamayo *et al.* (2005)](https://doi.org/10.1073/pnas.0506580102).
+GSEA is a functional class scoring (FCS) approach to pathway analysis that was first introduced in [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102).
 The rationale behind FCS approaches is that small changes in individual genes that participate in the same biological process or pathway can be significant and of biological interest.
 FCS methods are better suited for identifying these pathways that show coordinated changes than ORA.
 In ORA, we pick a cutoff that _typically_ only captures genes with large individual changes.
 
-There are 3 general steps in FCS methods ([Khatri, Sirota, and Butte. 2012]( https://doi.org/10.1371/journal.pcbi.1002375)):
+There are 3 general steps in FCS methods ([Khatri _et al._ 2012]( https://doi.org/10.1371/journal.pcbi.1002375)):
  
 1. Calculate a gene-level statistic (we'll use log2 fold change from DESeq2 here)
 2. Gene-level statistics are aggregated into a pathway-level statistic
@@ -31,7 +38,8 @@ There are 3 general steps in FCS methods ([Khatri, Sirota, and Butte. 2012]( htt
 
 * For another example using `clusterProfiler` for GSEA, see [_Intro to DGE: Functional Analysis._ from Harvard Chan Bioinformatics Core Training.](https://hbctraining.github.io/DGE_workshop/lessons/09_functional_analysis.html)
 * The way we'll use `clusterProfiler` here uses `fgsea` (Fast Gene Set Enrichment Analysis) under the hood.
-You can read more about fgsea in [Korotkevich, Sukhov, and Sergushichev. (2019)](https://doi.org/10.1101/060012).
+You can read more about fgsea in [Korotkevich _et al._ (2021)](https://doi.org/10.1101/060012).
+* [refine.bio examples Gene set enrichment analysis - RNA-seq](https://alexslemonade.github.io/refinebio-examples/03-rnaseq/pathway-analysis_rnaseq_02_gsea.html) from which this material has been adapted.
 
 ## Set up
 
@@ -42,8 +50,6 @@ You can read more about fgsea in [Korotkevich, Sukhov, and Sergushichev. (2019)]
 library(clusterProfiler)
 # Package that contains the MSigDB gene sets in tidy format
 library(msigdbr)
-# Annotation package that we will use for human gene identifier conversion
-library(org.Hs.eg.db)
 ```
 
 ### Directories and Files
@@ -52,11 +58,11 @@ library(org.Hs.eg.db)
 
 ```{r}
 # Where the DGE results are stored
-input_dir <- file.path("results", "gene-metrics")
+input_dir <- file.path("..", "RNA-seq", "results", "NB-cell")
 
 # We will create a directory to specifically hold our GSEA results if it does 
 # not yet exist
-output_dir <- file.path("results", "gsea")
+output_dir <- file.path("results", "NB-cell")
 if (!dir.exists(output_dir)) {
   dir.create(output_dir, recursive = TRUE)
 }
@@ -67,7 +73,7 @@ if (!dir.exists(output_dir)) {
 ```{r input_files}
 # DGE results
 dge_results_file <- file.path(input_dir,
-                             "nb_cell_line_mycn_amplified_v_nonamplified.tsv")
+                             "NB-cell_DESeq_amplified_v_nonamplified_results.tsv")
 ```
 
 #### Output files
@@ -75,10 +81,30 @@ dge_results_file <- file.path(input_dir,
 ```{r output_files}
 # GSEA pathway-level scores and statistics
 gsea_results_file <- file.path(output_dir,
-                               "nb_cell_line_gsea_results.tsv")
+                               "NB-cell_gsea_results.tsv")
 ```
 
-## Gene sets
+## Gene Set Enrichment Analysis
+
+_Adapted from [refine.bio examples](https://github.com/AlexsLemonade/refinebio-examples/blob/33cdeff66d57f9fe8ee4fcb5156aea4ac2dce07f/03-rnaseq/pathway-analysis_rnaseq_02_gsea.Rmd)_
+
+![](diagrams/subramanian_fig1.jpg)
+
+**Figure 1. [Subramanian _et al._ (2005)](https://doi.org/10.1073/pnas.0506580102).**
+
+GSEA calculates a pathway-level metric, called an enrichment score (sometimes abbreviated as ES), by ranking genes by a gene-level statistic.
+This score reflects whether or not a gene set or pathway is over-represented at the top or bottom of the gene rankings ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)) 
+
+Specifically, all genes are ranked from most positive to most negative based on their statistic and a running sum is calculated:
+Starting with the most highly ranked genes, the running sum increases for each gene in the pathway and decreases for each gene not in the pathway.
+The enrichment score for a pathway is the running sum's maximum deviation from zero.
+GSEA also assesses statistical significance of the scores for each pathway through permutation testing. 
+As a result, each input pathway will have a p-value associated with it that is then corrected for multiple hypothesis testing ([Subramanian _et al._ 2005](https://doi.org/10.1073/pnas.0506580102); [Yu](http://yulab-smu.top/clusterProfiler-book/chapter2.html#gene-set-enrichment-analysis)).
+
+The implementation of GSEA we use in here examples requires a gene list ordered by some statistic and input gene sets. 
+When you use previously computed gene-level statistics with GSEA, it is called GSEA pre-ranked.
+
+### Gene sets
 
 In the previous notebook, we used KEGG pathways for over-representation analysis.
 We identified pathways that were significantly over-represented and found that the significant pathways shared genes.
@@ -110,72 +136,117 @@ We can retrieve only the Hallmark gene sets by specifying `category = "H"` to th
 head(dge_results_df)
 ```
 
-We can take the same steps we did earlier to convert these Ensembl gene IDs to Entrez IDs. 
-We have to change the `column` argument to `mapIds()` (and the downstream steps).
+Since this data frame of DGE results includes gene symbols, we do not need to perform any kind of gene identifier conversion.
+We do, however, need to check for duplicate gene symbols. 
+We can accomplish this with `duplicated()`, which returns a logical vector (e.g., `TRUE` or `FALSE`).
+The function `sum()` will count `TRUE` values as 1s and `FALSE` as 0s, so using it with `duplicated()` will count the number of duplicate values.
+
+```{r any_duplicated, live = TRUE}
+
+```
 
-When `mapIds()` is run with `multiValues = "first"` and it encounters multiple matches, only the first identifier is returned.
-This is the default behavior.
-It will also return a named vector of the IDs you queried for (`column`), where the names are the input identifiers (`keys`), in the same order as the keys which means you can use it directly with `dplyr::mutate()`.
+This will cause a problem when we go to run GSEA.
 
-```{r convert_entrez, live = TRUE}
+### Removing duplicates
 
-  # Create a new column 'entrez_id' that contains the Entrez IDs returned by
-  # mapIds()
+The GSEA approach requires on discriminating between genes that are in a gene set and those that are not. 
+Practically speaking, gene sets are just collections of gene identifiers!
+When the function we use for GSEA pre-ranked gets a list with duplicated gene identifiers, it can produce unexpected results.
+
+Compared to the total number of genes that are in our results, there are not a lot of duplicates but we'll still need to make a decision about how to handle them.
+
+Let's get a vector of the duplicated gene symbols so we can use it to explore our filtering steps.
+
+```{r gene_dups, live = TRUE}
+
+```
 
-  # We need these gene identifiers to perform our pathway analysis, so remove
-  # any genes where we don't have Entrez IDs
+Now we'll look at the values for the the duplicated gene symbols.
 
+```{r show_gene_dups}
+dge_results_df %>%
+  dplyr::filter(gene_symbol %in% duplicated_gene_symbols) %>%
+  dplyr::arrange(gene_symbol)
 ```
 
-### Preranked list
+We can see that the associated values vary for each row.
 
-The `GSEA()` function takes a preranked (sorted) named vector of statistics, where the names in the vector are gene identifiers.
+Let's keep the gene symbols associated with the higher absolute value of the log2 fold change.
+
+Retaining the instance of the gene symbols with the higher absolute value of a gene-level statistic means that we will retain the value that is likely to be more highly- or lowly-ranked or, put another way, the values less likely to be towards the middle of the ranked gene list.
+We should keep this decision in mind when interpreting our results.
+For example, if all the duplicate identifiers happened to be in a particular gene set, we may get an overly optimistic view of how perturbed that gene set is because we preferentially selected instances of the identifier that have a higher absolute value of the statistic used for ranking.
+
+In the next chunk, we are going to filter out the duplicated rows using the `dplyr::distinct()` function after sorting by absolute value of the log2 fold change.
+This will keep the first row with the duplicated value thus keeping the row with the largest absolute value.
+
+```{r filter_dge}
+filtered_dge_df <- dge_results_df %>%
+  # Sort so that the highest absolute values of the log2 fold change are at the
+  # top
+  dplyr::arrange(dplyr::desc(abs(log2FoldChange))) %>%
+  # Filter out the duplicated rows using `dplyr::distinct()`
+  dplyr::distinct(gene_symbol, .keep_all = TRUE)
+```
+
+Let's see what happened to our duplicate identifiers.
+
+```{r show_filtered_dge, live = TRUE}
+# Subset to & arrange by gene symbols that were duplicated in the original 
+# data frame of results
+
+```
+
+Now we're ready to prep our pre-ranked list for GSEA.
+
+### Pre-ranked list
+
+The `GSEA()` function takes a pre-ranked (sorted) named vector of statistics, where the names in the vector are gene identifiers.
 This is step 1 -- gene-level statistics.
 
 ```{r lfc_vector}
-lfc_vector <- dge_results_df$log2FoldChange
-names(lfc_vector) <- dge_results_df$entrez_id
+lfc_vector <- filtered_dge_df %>%
+  # Extract a vector of `log2FoldChange` named by `gene_symbol`
+  dplyr::pull(log2FoldChange, name = gene_symbol)
 lfc_vector <- sort(lfc_vector, decreasing = TRUE)
 ```
 
+Let's look at the top ranked values.
+
 ```{r head_lfc, live = TRUE}
 # Look at first entries of the log2 fold change vector
 
 ```
 
+And the bottom of the list.
+
 ```{r tail_lfc, live = TRUE}
 # Look at the last entries of the log2 fold change vector
 
 ```
 
-## GSEA
-
-![](diagrams/subramanian_fig1.jpg)
-
-**Figure 1. [Subramanian, Tamayo *et al.* (2005)](https://doi.org/10.1073/pnas.0506580102).**
+## Run GSEA
 
-The enrichment score (ES) for a pathway, a pathway-level statistic, is calculated using our gene-level statistics.
-Genes are ranked from most highly positive to most highly negative and weighting them according to their gene-level statistic.
-A running score is calculated by starting with the most highly ranked genes and increasing the score when a gene is in the pathway and decreasing the score when a gene is not in the pathway.
-The ES is the maximum deviation from zero.
-Significance is assessed by generating a null distribution by sampling random gene sets of the same size and an FDR (false discovery rate) value is calculated to account for multiple hypothesis testing.
-([Subramanian, Tamayo *et al.* 2005](https://doi.org/10.1073/pnas.0506580102); [Korotkevich, Sukhov, and Sergushichev. 2019](https://doi.org/10.1101/060012)).
+Now for the analysis!
 
 We can use the `GSEA()` function to perform GSEA with any generic set of gene sets, but there are several functions for using specific, commonly used gene sets (e.g., `gseKEGG()`).
 
 ```{r run_gsea}
 gsea_results <- GSEA(geneList = lfc_vector,  # ordered ranked gene list
-                     nPerm = 1000,  # number of permutations
                      minGSSize = 25,  # minimum gene set size
                      maxGSSize = 500,  # maximum gene set set
                      pvalueCutoff = 0.05,
-                     pAdjustMethod = "BH",  # Benjamini-Hochberg correction
+                     pAdjustMethod = "BH",  # correction for multiple hypothesis testing
                      TERM2GENE = dplyr::select(hs_hallmark_df,
                                                gs_name,
-                                               entrez_gene))
+                                               gene_symbol))
 ```
 
-Let's take a look at the results.
+The warning about ties means that there are multiple genes that have the same log2 fold change value. 
+This percentage is small and unlikely to impact our results.
+A large number of ties might tell us there's something wrong with our DGE results ([Ballereau _et al._ 2018](https://bioinformatics-core-shared-training.github.io/cruk-summer-school-2018/RNASeq2018/html/06_Gene_set_testing.nb.html)).
+
+Let's take a look at the GSEA results.
 
 ```{r view_gsea, live = TRUE, eval = FALSE}
 
@@ -189,21 +260,20 @@ Let's write these results to file.
 gsea_results@result %>% readr::write_tsv(gsea_results_file)
 ```
 
-
 ### Visualizing GSEA results
 
 We can visualize GSEA results for individual pathways or gene sets using `enrichplot::gseaplot()`.
-Let's take a look at 3 different pathways -- one with a highly positive NES, one with a highly negative NES, and one somewhere in the middle -- to get more insight into how ES are calculated.
+Let's take a look at 3 different pathways -- one with a highly positive NES, one with a highly negative NES, and one that was not a significant result -- to get more insight into how ES are calculated.
 
 #### Highly Positive NES
 
-The gene set `HALLMARK_MYC_TARGETS_V2` had high positive log2 fold changes.
+The gene set `HALLMARK_MYC_TARGETS_V1` had high positive log2 fold changes.
 Recall a positive log2 fold change means a it had a higher expression value in _MYCN_ amplified cell lines.
 
-```{r myc_v2}
+```{r myc_v1}
 enrichplot::gseaplot(gsea_results,
-                     geneSetID = "HALLMARK_MYC_TARGETS_V2",
-                     title = "HALLMARK_MYC_TARGETS_V2",
+                     geneSetID = "HALLMARK_MYC_TARGETS_V1",
+                     title = "HALLMARK_MYC_TARGETS_V1",
                      color.line = "#0066FF")
 ```
 
@@ -211,32 +281,29 @@ Notice how the genes that are in the gene set, indicated by the black bars, tend
 
 #### Highly Negative NES
 
-The gene set `HALLMARK_INFLAMMATORY_RESPONSE` had a highly negative NES.
+The gene set `HALLMARK_INTERFERON_ALPHA_RESPONSE` had a highly negative NES.
 
 ```{r inflammatory}
 enrichplot::gseaplot(gsea_results,
-                     geneSetID = "HALLMARK_INFLAMMATORY_RESPONSE",
-                     title = "HALLMARK_INFLAMMATORY_RESPONSE",
+                     geneSetID = "HALLMARK_INTERFERON_ALPHA_RESPONSE",
+                     title = "HALLMARK_INTERFERON_ALPHA_RESPONSE",
                      color.line = "#0066FF")
 ```
 
 This gene set shows the opposite pattern -- genes in the pathway tend to be on the right side of the graph.
 
-#### Somewhere in the middle
+#### A non-significant result
 
-A moderately negative NES is somewhere in the middle in this particular experiment.
-Let's look at `HALLMARK_P53_PATHWAY`.
+The `@results` slot will only show gene sets that pass the `pvalueCutoff` threshold we supplied to `GSEA()`, but we can plot any gene set so long as we know its name.
+Let's look at `HALLMARK_P53_PATHWAY`, which was not in the results we viewed earlier.
+
+```{r p53, live = TRUE}
 
-```{r p53}
-enrichplot::gseaplot(gsea_results,
-                     geneSetID = "HALLMARK_P53_PATHWAY",
-                     title = "HALLMARK_P53_PATHWAY",
-                     color.line = "#0066FF")
 ```
 
 Genes in the pathway are distributed more evenly throughout the ranked list, resulting in a more "middling" score.
 
-*Note: The plots returned by `enrichplot::gseaplot` are ggplots, so we can use `ggplot2::ggsave()` to save them to file.*
+*Note: The plots returned by `enrichplot::gseaplot` are ggplots, so we could use `ggplot2::ggsave()` to save them to file if we wanted to.*
 
 ## Session Info
 
diff --git a/pathway-analysis/02-gene_set_enrichment_analysis.nb.html b/pathway-analysis/02-gene_set_enrichment_analysis.nb.html
index 724644de..a46c6641 100644
--- a/pathway-analysis/02-gene_set_enrichment_analysis.nb.html
+++ b/pathway-analysis/02-gene_set_enrichment_analysis.nb.html
@@ -350,23 +350,29 @@ 
      Set up
      
 
      Libraries
      
 
 
-
+
 
      # Package to run GSEA
 library(clusterProfiler)
      
 
-
+
+
      
+
+
 
      clusterProfiler v3.18.1  For help: https://guangchuangyu.github.io/software/clusterProfiler
 
 If you use clusterProfiler in published research, please cite:
-Guangchuang Yu, Li-Gen Wang, Yanyan Han, Qing-Yu He. clusterProfiler: an R package for comparing biological themes among gene clusters. OMICS: A Journal of Integrative Biology. 2012, 16(5):284-287.
-
-Attaching package: ‘clusterProfiler’
-
-The following object is masked from ‘package:stats’:
+Guangchuang Yu, Li-Gen Wang, Yanyan Han, Qing-Yu He. clusterProfiler: an R package for comparing biological themes among gene clusters. OMICS: A Journal of Integrative Biology. 2012, 16(5):284-287.
      
+
+
+
      
+Attaching package: 'clusterProfiler'
      
+
+
+
      The following object is masked from 'package:stats':
 
     filter
      
-
-
+
+
 
      # Package that contains the MSigDB gene sets in tidy format
 library(msigdbr)
      
 
@@ -379,7 +385,7 @@ 
      Directories and Files
      
 
      Directories
      
 
 
-
+
 
      # Where the DGE results are stored
 input_dir <- file.path("..", "RNA-seq", "results", "NB-cell")
 
@@ -397,7 +403,7 @@ 
      Directories
      
 
      Input files
      
 
 
-
+
 
      # DGE results
 dge_results_file <- file.path(input_dir,
                              "NB-cell_DESeq_amplified_v_nonamplified_results.tsv")
      
@@ -409,7 +415,7 @@ 
      Input files
      
 
      Output files
      
 
 
-
+
 
      # GSEA pathway-level scores and statistics
 gsea_results_file <- file.path(output_dir,
                                "NB-cell_gsea_results.tsv")
      
@@ -438,7 +444,7 @@ 
      Gene sets
      
 
      We can retrieve only the Hallmark gene sets by specifying category = "H" to the msigdbr() function.
      
 
 
-
+
 
      hs_hallmark_df <- msigdbr(species = "Homo sapiens",
                           category = "H")
      
 
@@ -450,13 +456,13 @@ 
      Gene sets
      
 
      Differential gene expression results
      
 
 
-
+
 
      # Read in the DGE results
 dge_results_df <- readr::read_tsv(dge_results_file)
      
 
-
+
 
      
-── Column specification ───────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ────────────────────────────────────────────────────────
 cols(
   gene_id = col_character(),
   baseMean = col_double(),
@@ -466,27 +472,25 @@ 
      Differential gene expression results
      
   padj = col_double(),
   gene_symbol = col_character()
 )
      
-
+
 
 
 
 
-
+
 
      head(dge_results_df)
      
 
-
 
      
 
 
      
-
 
 
 
      Since this data frame of DGE results includes gene symbols, we do not need to perform any kind of gene identifier conversion. We do, however, need to check for duplicate gene symbols. We can accomplish this with duplicated(), which returns a logical vector (e.g., TRUE or FALSE). The function sum() will count TRUE values as 1s and FALSE as 0s, so using it with duplicated() will count the number of duplicate values.
      
 
 
-
+
 
      sum(duplicated(dge_results_df$gene_symbol))
      
 
 
@@ -502,7 +506,7 @@ 
      Removing duplicates
      
 
      Let’s get a vector of the duplicated gene symbols so we can use it to explore our filtering steps.
      
 
 
-
+
 
      duplicated_gene_symbols <- dge_results_df %>%
   dplyr::filter(duplicated(gene_symbol)) %>%
   dplyr::pull(gene_symbol)
      
@@ -512,18 +516,16 @@ 
      Removing duplicates
      
 
      Now we’ll look at the values for the the duplicated gene symbols.
      
 
 
-
+
 
      dge_results_df %>%
   dplyr::filter(gene_symbol %in% duplicated_gene_symbols) %>%
   dplyr::arrange(gene_symbol)
      
 
-
 
      
 
 
      
-
 
 
 
      We can see that the associated values vary for each row.
      
@@ -532,7 +534,7 @@ 
      Removing duplicates
      
 
      In the next chunk, we are going to filter out the duplicated rows using the dplyr::distinct() function after sorting by absolute value of the log2 fold change. This will keep the first row with the duplicated value thus keeping the row with the largest absolute value.
      
 
 
-
+
 
      filtered_dge_df <- dge_results_df %>%
   # Sort so that the highest absolute values of the log2 fold change are at the
   # top
@@ -545,20 +547,18 @@ 
      Removing duplicates
      
 
      Let’s see what happened to our duplicate identifiers.
      
 
 
-
+
 
      # Subset to & arrange by gene symbols that were duplicated in the original 
 # data frame of results
 filtered_dge_df %>%
   dplyr::filter(gene_symbol %in% duplicated_gene_symbols) %>%
   dplyr::arrange(gene_symbol)
      
 
-
 
      
 
 
      
-
 
 
 
      Now we’re ready to prep our pre-ranked list for GSEA.
      
@@ -568,7 +568,7 @@ 
      Pre-ranked list
      
 
      The GSEA() function takes a pre-ranked (sorted) named vector of statistics, where the names in the vector are gene identifiers. This is step 1 – gene-level statistics.
      
 
 
-
+
 
      lfc_vector <- filtered_dge_df %>%
   # Extract a vector of `log2FoldChange` named by `gene_symbol`
   dplyr::pull(log2FoldChange, name = gene_symbol)
@@ -579,26 +579,26 @@ 
      Pre-ranked list
      
 
      Let’s look at the top ranked values.
      
 
 
-
+
 
      # Look at first entries of the log2 fold change vector
 head(lfc_vector)
      
 
-
+
 
        GAGE12D    GABBR1   FABP5P7  KIAA0355      MNX1    GAGE2A 
-23.620487 23.130027 22.682631 22.562940  6.813764  6.801245 
      
+23.620487 23.130046 22.682641 22.562941  6.813764  6.801245 
      
 
 
 
 
      And the bottom of the list.
      
 
 
-
+
 
      # Look at the last entries of the log2 fold change vector
 tail(lfc_vector)
      
 
-
+
 
           MUC15      TFPI2      LENG1       MT1M       DAZ2   CSPG4P4Y 
- -8.017004  -8.768669 -22.754783 -25.318887 -25.459848 -26.095612 
      
+ -8.017004  -8.768669 -22.754783 -25.318887 -25.444288 -26.095612 
      
 
 
 
@@ -610,7 +610,7 @@ 
      Run GSEA
      
 
      We can use the GSEA() function to perform GSEA with any generic set of gene sets, but there are several functions for using specific, commonly used gene sets (e.g., gseKEGG()).
      
 
 
-
+
 
      gsea_results <- GSEA(geneList = lfc_vector,  # ordered ranked gene list
                      minGSSize = 25,  # minimum gene set size
                      maxGSSize = 500,  # maximum gene set set
@@ -620,13 +620,22 @@ 
      Run GSEA
      
                                                gs_name,
                                                gene_symbol))
      
 
-
-
      preparing geneSet collections...
-GSEA analysis...
-There are ties in the preranked stats (0.06% of the list).
-The order of those tied genes will be arbitrary, which may produce unexpected results.leading edge analysis...
-done...
      
-
+
+
      preparing geneSet collections...
      
+
+
+
      GSEA analysis...
      
+
+
+
      Warning in preparePathwaysAndStats(pathways, stats, minSize, maxSize, gseaParam, : There are ties in the preranked stats (0.06% of the list).
+The order of those tied genes will be arbitrary, which may produce unexpected results.
      
+
+
+
      leading edge analysis...
      
+
+
+
      done...
      
+
 
 
 
      The warning about ties means that there are multiple genes that have the same log2 fold change value. This percentage is small and unlikely to impact our results. A large number of ties might tell us there’s something wrong with our DGE results (Ballereau et al. 2018).
      
@@ -644,7 +653,7 @@ 
      Run GSEA
      
 
      Let’s write these results to file.
      
 
 
-
+
 
      gsea_results@result %>% readr::write_tsv(gsea_results_file)
      
 
 
@@ -657,14 +666,14 @@ 
      Highly Positive NES
      
 
      The gene set HALLMARK_MYC_TARGETS_V1 had high positive log2 fold changes. Recall a positive log2 fold change means a it had a higher expression value in MYCN amplified cell lines.
      
 
 
-
+
 
      enrichplot::gseaplot(gsea_results,
                      geneSetID = "HALLMARK_MYC_TARGETS_V1",
                      title = "HALLMARK_MYC_TARGETS_V1",
                      color.line = "#0066FF")
      
 
-
-
      
+
+
      
 
 
 
@@ -675,14 +684,14 @@ 
      Highly Negative NES
      
 
      The gene set HALLMARK_INTERFERON_ALPHA_RESPONSE had a highly negative NES.
      
 
 
-
+
 
      enrichplot::gseaplot(gsea_results,
                      geneSetID = "HALLMARK_INTERFERON_ALPHA_RESPONSE",
                      title = "HALLMARK_INTERFERON_ALPHA_RESPONSE",
                      color.line = "#0066FF")
      
 
-
-
      
+
+
      
 
 
 
@@ -693,14 +702,14 @@ 
      A non-significant result
      
 
      The @results slot will only show gene sets that pass the pvalueCutoff threshold we supplied to GSEA(), but we can plot any gene set so long as we know its name. Let’s look at HALLMARK_P53_PATHWAY, which was not in the results we viewed earlier.
      
 
 
-
+
 
      enrichplot::gseaplot(gsea_results,
                      geneSetID = "HALLMARK_P53_PATHWAY",
                      title = "HALLMARK_P53_PATHWAY",
                      color.line = "#0066FF")
      
 
-
-
      
+
+
      
 
 
 
@@ -713,49 +722,64 @@ 
      A non-significant result
      
 
      Session Info
      
 
 
-
+
 
      sessionInfo()
      
 
-
+
 
      R version 4.0.3 (2020-10-10)
 Platform: x86_64-pc-linux-gnu (64-bit)
-Running under: Ubuntu 18.04.3 LTS
+Running under: Ubuntu 20.04 LTS
 
 Matrix products: default
-BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
-LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
 
 locale:
- [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
- [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
- [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
 
 attached base packages:
-[1] stats     graphics  grDevices datasets  utils     methods   base     
+[1] stats     graphics  grDevices utils     datasets  methods   base     
 
 other attached packages:
-[1] msigdbr_7.2.1          clusterProfiler_3.18.1
+[1] msigdbr_7.2.1          clusterProfiler_3.18.1 optparse_1.6.6        
 
 loaded via a namespace (and not attached):
- [1] enrichplot_1.10.2    bit64_4.0.5          RColorBrewer_1.1-2   tools_4.0.3          R6_2.5.0            
- [6] DBI_1.1.1            BiocGenerics_0.36.0  colorspace_2.0-0     tidyselect_1.1.0     gridExtra_2.3       
-[11] bit_4.0.4            compiler_4.0.3       cli_2.2.0            Biobase_2.50.0       scatterpie_0.1.5    
-[16] labeling_0.4.2       shadowtext_0.0.7     scales_1.1.1         readr_1.4.0          stringr_1.4.0       
-[21] digest_0.6.27        rmarkdown_2.6        DOSE_3.16.0          pkgconfig_2.0.3      htmltools_0.5.1.1   
-[26] fastmap_1.1.0        rlang_0.4.10         rstudioapi_0.13      RSQLite_2.2.3        farver_2.0.3        
-[31] generics_0.1.0       jsonlite_1.7.2       BiocParallel_1.24.1  GOSemSim_2.16.1      dplyr_1.0.3         
-[36] magrittr_2.0.1       GO.db_3.12.1         Matrix_1.3-2         fansi_0.4.2          Rcpp_1.0.6          
-[41] munsell_0.5.0        S4Vectors_0.28.1     viridis_0.5.1        lifecycle_0.2.0      stringi_1.5.3       
-[46] yaml_2.2.1           ggraph_2.0.4         MASS_7.3-53          plyr_1.8.6           qvalue_2.22.0       
-[51] grid_4.0.3           blob_1.2.1           parallel_4.0.3       ggrepel_0.9.1        DO.db_2.9           
-[56] crayon_1.3.4         lattice_0.20-41      graphlayouts_0.7.1   cowplot_1.1.1        splines_4.0.3       
-[61] hms_1.0.0            knitr_1.30           pillar_1.4.7         fgsea_1.16.0         igraph_1.2.6        
-[66] reshape2_1.4.4       stats4_4.0.3         fastmatch_1.1-0      glue_1.4.2           evaluate_0.14       
-[71] downloader_0.4       data.table_1.13.6    renv_0.12.5-2        BiocManager_1.30.10  vctrs_0.3.6         
-[76] tweenr_1.0.1         gtable_0.3.0         purrr_0.3.4          polyclip_1.10-0      tidyr_1.1.2         
-[81] assertthat_0.2.1     cachem_1.0.1         ggplot2_3.3.3        xfun_0.20            ggforce_0.3.2       
-[86] tidygraph_1.2.0      viridisLite_0.3.0    tibble_3.0.5         rvcheck_0.1.8        AnnotationDbi_1.52.0
-[91] memoise_1.1.0        IRanges_2.24.1       ellipsis_0.3.1      
      
+ [1] enrichplot_1.10.2    bit64_4.0.5          RColorBrewer_1.1-2  
+ [4] tools_4.0.3          R6_2.5.0             DBI_1.1.1           
+ [7] BiocGenerics_0.36.0  colorspace_2.0-0     tidyselect_1.1.0    
+[10] gridExtra_2.3        bit_4.0.4            compiler_4.0.3      
+[13] cli_2.2.0            Biobase_2.50.0       scatterpie_0.1.5    
+[16] labeling_0.4.2       shadowtext_0.0.7     scales_1.1.1        
+[19] readr_1.4.0          stringr_1.4.0        digest_0.6.27       
+[22] rmarkdown_2.6        DOSE_3.16.0          pkgconfig_2.0.3     
+[25] htmltools_0.5.1.1    fastmap_1.1.0        rlang_0.4.10        
+[28] rstudioapi_0.13      RSQLite_2.2.3        farver_2.0.3        
+[31] generics_0.1.0       jsonlite_1.7.2       BiocParallel_1.24.1 
+[34] GOSemSim_2.16.1      dplyr_1.0.3          magrittr_2.0.1      
+[37] GO.db_3.12.1         Matrix_1.3-2         fansi_0.4.2         
+[40] Rcpp_1.0.6           munsell_0.5.0        S4Vectors_0.28.1    
+[43] viridis_0.5.1        lifecycle_0.2.0      stringi_1.5.3       
+[46] yaml_2.2.1           ggraph_2.0.4         MASS_7.3-53         
+[49] plyr_1.8.6           qvalue_2.22.0        grid_4.0.3          
+[52] blob_1.2.1           parallel_4.0.3       ggrepel_0.9.1       
+[55] DO.db_2.9            crayon_1.3.4         lattice_0.20-41     
+[58] graphlayouts_0.7.1   cowplot_1.1.1        splines_4.0.3       
+[61] hms_1.0.0            ps_1.5.0             knitr_1.30          
+[64] pillar_1.4.7         fgsea_1.16.0         igraph_1.2.6        
+[67] reshape2_1.4.4       stats4_4.0.3         fastmatch_1.1-0     
+[70] glue_1.4.2           evaluate_0.14        downloader_0.4      
+[73] data.table_1.13.6    BiocManager_1.30.10  vctrs_0.3.6         
+[76] tweenr_1.0.1         gtable_0.3.0         getopt_1.20.3       
+[79] purrr_0.3.4          polyclip_1.10-0      tidyr_1.1.2         
+[82] assertthat_0.2.1     cachem_1.0.1         ggplot2_3.3.3       
+[85] xfun_0.20            ggforce_0.3.2        tidygraph_1.2.0     
+[88] viridisLite_0.3.0    tibble_3.0.5         rvcheck_0.1.8       
+[91] AnnotationDbi_1.52.0 memoise_1.1.0        IRanges_2.24.1      
+[94] ellipsis_0.3.1      
      
 
 
 
diff --git a/pathway-analysis/03-gene_set_variation_analysis-live.Rmd b/pathway-analysis/03-gene_set_variation_analysis-live.Rmd
index fc47b880..13369210 100644
--- a/pathway-analysis/03-gene_set_variation_analysis-live.Rmd
+++ b/pathway-analysis/03-gene_set_variation_analysis-live.Rmd
@@ -8,15 +8,25 @@ author: CCDL for ALSF
 date: 2020
 ---
 
+## Objectives
+
+This notebook will demonstrate how to:
+
+- Identify when Gene Set Variation Analysis (GSVA) is well-suited for an analysis
+- Perform GSVA on transformed RNA-seq data with the `GSVA` package
+- Explore the dependence of GSVA scores on gene set size with random gene sets
+
+---
+
 So far every pathway analysis method we've covered relies on some information about groups of samples in our data.
-For over-representation analysis (ORA), we took the top 100 genes that distinguished neurons from other cell types in a scRNA-seq experiment -- it relied on cell type labels.
+For over-representation analysis (ORA), we created gene sets from two different two group comparisons.
 In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared _MYCN_ amplified cell lines to non-amplified cell lines; we needed that amplification status information.
 
 What if we're less sure about groups in our data or we want to analyze our data in a more unsupervised manner?
 
-In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) ([Hänzelmann, Castelo, and Guinney. 2013](https://doi.org/10.1186/1471-2105-14-7)) that allows us to calculate gene set or pathway scores on a per-sample basis.
+In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) ([Hänzelmann _et al._ 2013](https://doi.org/10.1186/1471-2105-14-7)) that allows us to calculate gene set or pathway scores on a per-sample basis.
 
-We like this quote from the GSVA paper ([Hänzelmann, Castelo, and Guinney. 2013](https://doi.org/10.1186/1471-2105-14-7)) to set the stage:
+We like this quote from the GSVA paper ([Hänzelmann _et al._ 2013](https://doi.org/10.1186/1471-2105-14-7)) to set the stage:
 
 > While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.
 
@@ -45,10 +55,10 @@ library(GSVA)
 ```{r directories}
 # We have some medulloblastoma data from the OpenPBTA project that we've 
 # prepared ahead of time
-input_dir <- file.path("data", "medulloblastoma")
+input_dir <- file.path("data", "open-pbta")
 
-# Create a directory specifically for our GSVA results if it does not yet exist
-output_dir <- file.path("results", "gsva") 
+# Create a directory specifically for the results using this dataset
+output_dir <- file.path("results", "open-pbta") 
 if (!dir.exists(output_dir)) {
   dir.create(output_dir, recursive = TRUE)
 }
@@ -124,14 +134,14 @@ For GSVA, we need a matrix.
 
 ![](diagrams/hanzelmann_fig1.jpg)
 
-**Figure 1 from [Hänzelmann, Castelo, and Guinney. (2013)](https://doi.org/10.1186/1471-2105-14-7).**
+**Figure 1 from [Hänzelmann _et al._ (2013)](https://doi.org/10.1186/1471-2105-14-7).**
 
 You may notice that GSVA has some commonalities with GSEA.
 Rather than ranking genes based on some statistic _we_ selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution.
-This is a way of asking if a gene _i_ is highly or lowly expressed in a sample _j_ in the context of this experiment and ranking accordingly ([Hänzelmann, Castelo, and Guinney. 2013](https://doi.org/10.1186/1471-2105-14-7)).
+This is a way of asking if a gene _i_ is highly or lowly expressed in a sample _j_ in the context of this experiment and ranking accordingly ([Hänzelmann _et al._ 2013](https://doi.org/10.1186/1471-2105-14-7)).
 The pathway-level score calculated is a way of asking how genes _within_ a gene set vary as compared to genes that are _outside_ of that gene set ([Malhotra. 2018](https://towardsdatascience.com/decoding-gene-set-variation-analysis-8193a0cfda3)). 
 (This is sometimes called a competitive test in gene set enrichment literature.)
-The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) ([Hänzelmann, Castelo, and Guinney. 2013](https://doi.org/10.1186/1471-2105-14-7)).
+The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) ([Hänzelmann _et al._ 2013](https://doi.org/10.1186/1471-2105-14-7)).
 
 The output is a gene set by sample matrix of GSVA scores.
 
@@ -174,22 +184,22 @@ all_genes <- rownames(rnaseq_mat)
 set.seed(2020)
 ```
 
-Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We'll use the same minimum and maximum values for our random gene sets and some values in between.
+Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. 
+We'll use the same minimum and maximum values for our random gene sets and some values in between.
 
-```{r}
+```{r gene_set_sizes}
 # Make a list of integers that indicate the random gene set sizes
 gene_set_size <- list(15, 25, 50, 100, 250, 500)
 ```
 
-For each gene set size, we will generate 100 random gene sets
+For each gene set size, we will generate 100 random gene sets.
 
-```{r}
+```{r random_gene_sets}
 # Set number of replicates
-nreps <-  100
+nreps <- 100
 # Generate 100 random gene sets of each size
-random_gene_sets <- 
+random_gene_sets <- rep(gene_set_size, nreps) %>%  # Repeat gene sizes so we run `nreps` times 
   purrr::map(
-    rep(gene_set_size, nreps),  # Repeat gene sizes so we run `nreps` times 
     # Sample the vector of all genes, choosing the number of items specified
     # in the element of gene set size
     ~ base::sample(x = all_genes,
@@ -199,16 +209,19 @@ random_gene_sets <-
 
 The Hallmarks list we used earlier stored the gene set names as the name of the list, so let's add names to our random gene sets that indicate what size they are and so `gsva()` doesn't get upset.
 
-```{r}
+```{r name_random_gene_sets}
 # We will include the size of the gene set in the gene set name 
 # Start by taking the length of each pathway and appending "pathway_" to that
 # number
-lengths_vector <- purrr::map(random_gene_sets, ~ length(.x)) %>%
+lengths_vector <- random_gene_sets %>%
+  # Get the length of each gene set (number of genes)
+  purrr::map(~ length(.x)) %>%
+  # Make it "pathway_"
   purrr::map(~ paste0("pathway_", .x)) %>%
   # Return a vector
   purrr::flatten_chr()
 
-# Add the names in lengths_vector to the list
+# Add the names in lengths_vector to the list - "pathway_"
 random_gene_sets <- random_gene_sets %>% 
   # make.names() appends a "version" if something is not unique
   purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))
@@ -299,7 +312,7 @@ Here's a figure from the OpenPBTA project, where the middle panel is a heatmap o
 gsva_results %>%
   as.data.frame() %>%
   tibble::rownames_to_column("pathway") %>%
-  readr::write_tsv(path = gsva_results_file)
+  readr::write_tsv(file = gsva_results_file)
 ```
 
 ## Session Info
diff --git a/pathway-analysis/03-gene_set_variation_analysis.nb.html b/pathway-analysis/03-gene_set_variation_analysis.nb.html
index 6350426d..c3733022 100644
--- a/pathway-analysis/03-gene_set_variation_analysis.nb.html
+++ b/pathway-analysis/03-gene_set_variation_analysis.nb.html
@@ -347,7 +347,7 @@ 
      Set up
      
 
      Libraries
      
 
 
-
+
 
      # Pipes
 library(magrittr)
 # Gene Set Variation Analysis
@@ -362,7 +362,7 @@ 
      Directories and files
      
 
      Directories
      
 
 
-
+
 
      # We have some medulloblastoma data from the OpenPBTA project that we've 
 # prepared ahead of time
 input_dir <- file.path("data", "open-pbta")
@@ -381,7 +381,7 @@ 
      Input
      
 
      We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).
      
 
 
-
+
 
      rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")
      
 
 
@@ -391,7 +391,7 @@ 
      Input
      
 
      Output
      
 
 
-
+
 
      gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")
      
 
 
@@ -407,7 +407,7 @@ 
      Gene sets
      
 
      The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.
      
 
 
-
+
 
      # R can often read in data from a URL
 hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"
 
@@ -433,39 +433,37 @@ 
      RNA-seq data
      
 
      We’re only working with the medulloblastoma samples in this example.
      
 
 
-
+
 
      rnaseq_df <- readr::read_tsv(rnaseq_file)
      
 
-
+
 
      
-── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────
+── Column specification ────────────────────────────────────────────────────────
 cols(
   .default = col_double(),
   gene_symbol = col_character()
 )
 ℹ Use `spec()` for the full column specifications.
      
-
+
 
 
 
 
-
+
 
      # What does the RNA-seq data frame look like?
 rnaseq_df[1:5, 1:5]
      
 
-
 
      
 
 
      
-
 
 
 
      For GSVA, we need a matrix.
      
 
 
-
+
 
      rnaseq_mat <- rnaseq_df %>%
   tibble::column_to_rownames("gene_symbol") %>%
   as.matrix()
      
@@ -484,7 +482,7 @@ 
      GSVA
      
 
      Perform GSVA
      
 
 
-
+
 
      gsva_results <- gsva(rnaseq_mat, 
                      hallmarks_list,
                      method = "gsva",
@@ -497,132 +495,143 @@ 
      Perform GSVA
      
                      # Compute Gaussian-distributed scores
                      mx.diff = TRUE)
      
 
-
-
      945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
      
-
-
+
+
      Warning in .filterFeatures(expr, method): 945 genes with constant expression
+values throuhgout the samples.
      
+
+
+
      Warning in .filterFeatures(expr, method): Since argument method!="ssgsea", genes
+with constant expression values are discarded.
      
+
+
 
      Estimating GSVA scores for 50 gene sets.
 Estimating ECDFs with Gaussian kernels
 
-  |                                                                                                                          
-  |                                                                                                                    |   0%
-  |                                                                                                                          
-  |==                                                                                                                  |   2%
-  |                                                                                                                          
-  |=====                                                                                                               |   4%
-  |                                                                                                                          
-  |=======                                                                                                             |   6%
-  |                                                                                                                          
-  |=========                                                                                                           |   8%
-  |                                                                                                                          
-  |============                                                                                                        |  10%
-  |                                                                                                                          
-  |==============                                                                                                      |  12%
-  |                                                                                                                          
-  |================                                                                                                    |  14%
-  |                                                                                                                          
-  |===================                                                                                                 |  16%
-  |                                                                                                                          
-  |=====================                                                                                               |  18%
-  |                                                                                                                          
-  |=======================                                                                                             |  20%
-  |                                                                                                                          
-  |==========================                                                                                          |  22%
-  |                                                                                                                          
-  |============================                                                                                        |  24%
-  |                                                                                                                          
-  |==============================                                                                                      |  26%
-  |                                                                                                                          
-  |================================                                                                                    |  28%
-  |                                                                                                                          
-  |===================================                                                                                 |  30%
-  |                                                                                                                          
-  |=====================================                                                                               |  32%
-  |                                                                                                                          
-  |=======================================                                                                             |  34%
-  |                                                                                                                          
-  |==========================================                                                                          |  36%
-  |                                                                                                                          
-  |============================================                                                                        |  38%
-  |                                                                                                                          
-  |==============================================                                                                      |  40%
-  |                                                                                                                          
-  |=================================================                                                                   |  42%
-  |                                                                                                                          
-  |===================================================                                                                 |  44%
-  |                                                                                                                          
-  |=====================================================                                                               |  46%
-  |                                                                                                                          
-  |========================================================                                                            |  48%
-  |                                                                                                                          
-  |==========================================================                                                          |  50%
-  |                                                                                                                          
-  |============================================================                                                        |  52%
-  |                                                                                                                          
-  |===============================================================                                                     |  54%
-  |                                                                                                                          
-  |=================================================================                                                   |  56%
-  |                                                                                                                          
-  |===================================================================                                                 |  58%
-  |                                                                                                                          
-  |======================================================================                                              |  60%
-  |                                                                                                                          
-  |========================================================================                                            |  62%
-  |                                                                                                                          
-  |==========================================================================                                          |  64%
-  |                                                                                                                          
-  |=============================================================================                                       |  66%
-  |                                                                                                                          
-  |===============================================================================                                     |  68%
-  |                                                                                                                          
-  |=================================================================================                                   |  70%
-  |                                                                                                                          
-  |====================================================================================                                |  72%
-  |                                                                                                                          
-  |======================================================================================                              |  74%
-  |                                                                                                                          
-  |========================================================================================                            |  76%
-  |                                                                                                                          
-  |==========================================================================================                          |  78%
-  |                                                                                                                          
-  |=============================================================================================                       |  80%
-  |                                                                                                                          
-  |===============================================================================================                     |  82%
-  |                                                                                                                          
-  |=================================================================================================                   |  84%
-  |                                                                                                                          
-  |====================================================================================================                |  86%
-  |                                                                                                                          
-  |======================================================================================================              |  88%
-  |                                                                                                                          
-  |========================================================================================================            |  90%
-  |                                                                                                                          
-  |===========================================================================================================         |  92%
-  |                                                                                                                          
-  |=============================================================================================================       |  94%
-  |                                                                                                                          
-  |===============================================================================================================     |  96%
-  |                                                                                                                          
-  |==================================================================================================================  |  98%
-  |                                                                                                                          
-  |====================================================================================================================| 100%
      
+  |                                                                            
+  |                                                                      |   0%
+  |                                                                            
+  |=                                                                     |   2%
+  |                                                                            
+  |===                                                                   |   4%
+  |                                                                            
+  |====                                                                  |   6%
+  |                                                                            
+  |======                                                                |   8%
+  |                                                                            
+  |=======                                                               |  10%
+  |                                                                            
+  |========                                                              |  12%
+  |                                                                            
+  |==========                                                            |  14%
+  |                                                                            
+  |===========                                                           |  16%
+  |                                                                            
+  |=============                                                         |  18%
+  |                                                                            
+  |==============                                                        |  20%
+  |                                                                            
+  |===============                                                       |  22%
+  |                                                                            
+  |=================                                                     |  24%
+  |                                                                            
+  |==================                                                    |  26%
+  |                                                                            
+  |====================                                                  |  28%
+  |                                                                            
+  |=====================                                                 |  30%
+  |                                                                            
+  |======================                                                |  32%
+  |                                                                            
+  |========================                                              |  34%
+  |                                                                            
+  |=========================                                             |  36%
+  |                                                                            
+  |===========================                                           |  38%
+  |                                                                            
+  |============================                                          |  40%
+  |                                                                            
+  |=============================                                         |  42%
+  |                                                                            
+  |===============================                                       |  44%
+  |                                                                            
+  |================================                                      |  46%
+  |                                                                            
+  |==================================                                    |  48%
+  |                                                                            
+  |===================================                                   |  50%
+  |                                                                            
+  |====================================                                  |  52%
+  |                                                                            
+  |======================================                                |  54%
+  |                                                                            
+  |=======================================                               |  56%
+  |                                                                            
+  |=========================================                             |  58%
+  |                                                                            
+  |==========================================                            |  60%
+  |                                                                            
+  |===========================================                           |  62%
+  |                                                                            
+  |=============================================                         |  64%
+  |                                                                            
+  |==============================================                        |  66%
+  |                                                                            
+  |================================================                      |  68%
+  |                                                                            
+  |=================================================                     |  70%
+  |                                                                            
+  |==================================================                    |  72%
+  |                                                                            
+  |====================================================                  |  74%
+  |                                                                            
+  |=====================================================                 |  76%
+  |                                                                            
+  |=======================================================               |  78%
+  |                                                                            
+  |========================================================              |  80%
+  |                                                                            
+  |=========================================================             |  82%
+  |                                                                            
+  |===========================================================           |  84%
+  |                                                                            
+  |============================================================          |  86%
+  |                                                                            
+  |==============================================================        |  88%
+  |                                                                            
+  |===============================================================       |  90%
+  |                                                                            
+  |================================================================      |  92%
+  |                                                                            
+  |==================================================================    |  94%
+  |                                                                            
+  |===================================================================   |  96%
+  |                                                                            
+  |===================================================================== |  98%
+  |                                                                            
+  |======================================================================| 100%
      
 
 
 
 
      Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.
      
 
 
-
+
 
      # Let's explore what the output of gsva() looks like
 gsva_results[1:5, 1:5]
      
 
-
-
                                          BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
-HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
-HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
-HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
-HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
-HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437
      
+
+
                                          BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT
+HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193
+HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759
+HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933
+HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076
+HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594
+                                    BS_1QXEC43H BS_1TWCV047
+HALLMARK_TNFA_SIGNALING_VIA_NFKB     0.32300630  -0.1468081
+HALLMARK_HYPOXIA                     0.36247083  -0.2971559
+HALLMARK_CHOLESTEROL_HOMEOSTASIS     0.32418657  -0.4561386
+HALLMARK_MITOTIC_SPINDLE            -0.07023068  -0.1861483
+HALLMARK_WNT_BETA_CATENIN_SIGNALING  0.41372523  -0.1152437
      
 
 
 
@@ -633,7 +642,7 @@ 
      A note on gene set size
      
 
      We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.
      
 
 
-
+
 
      # Use all the gene symbols in the dataset as the pool of possible genes
 all_genes <- rownames(rnaseq_mat)
 
@@ -645,7 +654,7 @@ 
      A note on gene set size
      
 
      Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.
      
 
 
-
+
 
      # Make a list of integers that indicate the random gene set sizes
 gene_set_size <- list(15, 25, 50, 100, 250, 500)
      
 
@@ -654,7 +663,7 @@ 
      A note on gene set size
      
 
      For each gene set size, we will generate 100 random gene sets.
      
 
 
-
+
 
      # Set number of replicates
 nreps <- 100
 # Generate 100 random gene sets of each size
@@ -671,7 +680,7 @@ 
      A note on gene set size
      
 
      The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.
      
 
 
-
+
 
      # We will include the size of the gene set in the gene set name 
 # Start by taking the length of each pathway and appending "pathway_" to that
 # number
@@ -693,7 +702,7 @@ 
      A note on gene set size
      
 
      Run GSVA on our dataset with the same parameters as before, but now with random gene sets.
      
 
 
-
+
 
      random_gsva_results <- gsva(rnaseq_mat, 
                             random_gene_sets,
                             method = "gsva",
@@ -707,414 +716,343 @@ 
      A note on gene set size
      
                             # Compute Gaussian-distributed scores
                             mx.diff = TRUE)                            
      
 
-
-
      945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
      
-
-
+
+
      Warning in .filterFeatures(expr, method): 945 genes with constant expression
+values throuhgout the samples.
      
+
+
+
      Warning in .filterFeatures(expr, method): Since argument method!="ssgsea", genes
+with constant expression values are discarded.
      
+
+
 
      Estimating GSVA scores for 579 gene sets.
 Estimating ECDFs with Gaussian kernels
 
-  |                                                                                                                          
-  |                                                                                                                    |   0%
-  |                                                                                                                          
-  |=                                                                                                                   |   1%
-  |                                                                                                                          
-  |==                                                                                                                  |   1%
-  |                                                                                                                          
-  |==                                                                                                                  |   2%
-  |                                                                                                                          
-  |===                                                                                                                 |   2%
-  |                                                                                                                          
-  |===                                                                                                                 |   3%
-  |                                                                                                                          
-  |====                                                                                                                |   3%
-  |                                                                                                                          
-  |====                                                                                                                |   4%
-  |                                                                                                                          
-  |=====                                                                                                               |   4%
-  |                                                                                                                          
-  |=====                                                                                                               |   5%
-  |                                                                                                                          
-  |======                                                                                                              |   5%
-  |                                                                                                                          
-  |======                                                                                                              |   6%
-  |                                                                                                                          
-  |=======                                                                                                             |   6%
-  |                                                                                                                          
-  |========                                                                                                            |   7%
-  |                                                                                                                          
-  |=========                                                                                                           |   7%
-  |                                                                                                                          
-  |=========                                                                                                           |   8%
-  |                                                                                                                          
-  |==========                                                                                                          |   8%
-  |                                                                                                                          
-  |==========                                                                                                          |   9%
-  |                                                                                                                          
-  |===========                                                                                                         |   9%
-  |                                                                                                                          
-  |===========                                                                                                         |  10%
-  |                                                                                                                          
-  |============                                                                                                        |  10%
-  |                                                                                                                          
-  |============                                                                                                        |  11%
-  |                                                                                                                          
-  |=============                                                                                                       |  11%
-  |                                                                                                                          
-  |=============                                                                                                       |  12%
-  |                                                                                                                          
-  |==============                                                                                                      |  12%
-  |                                                                                                                          
-  |===============                                                                                                     |  13%
-  |                                                                                                                          
-  |================                                                                                                    |  13%
-  |                                                                                                                          
-  |================                                                                                                    |  14%
-  |                                                                                                                          
-  |=================                                                                                                   |  14%
-  |                                                                                                                          
-  |=================                                                                                                   |  15%
-  |                                                                                                                          
-  |==================                                                                                                  |  15%
-  |                                                                                                                          
-  |==================                                                                                                  |  16%
-  |                                                                                                                          
-  |===================                                                                                                 |  16%
-  |                                                                                                                          
-  |===================                                                                                                 |  17%
-  |                                                                                                                          
-  |====================                                                                                                |  17%
-  |                                                                                                                          
-  |====================                                                                                                |  18%
-  |                                                                                                                          
-  |=====================                                                                                               |  18%
-  |                                                                                                                          
-  |======================                                                                                              |  19%
-  |                                                                                                                          
-  |=======================                                                                                             |  20%
-  |                                                                                                                          
-  |========================                                                                                            |  20%
-  |                                                                                                                          
-  |========================                                                                                            |  21%
-  |                                                                                                                          
-  |=========================                                                                                           |  21%
-  |                                                                                                                          
-  |=========================                                                                                           |  22%
-  |                                                                                                                          
-  |==========================                                                                                          |  22%
-  |                                                                                                                          
-  |==========================                                                                                          |  23%
-  |                                                                                                                          
-  |===========================                                                                                         |  23%
-  |                                                                                                                          
-  |===========================                                                                                         |  24%
-  |                                                                                                                          
-  |============================                                                                                        |  24%
-  |                                                                                                                          
-  |============================                                                                                        |  25%
-  |                                                                                                                          
-  |=============================                                                                                       |  25%
-  |                                                                                                                          
-  |==============================                                                                                      |  26%
-  |                                                                                                                          
-  |===============================                                                                                     |  26%
-  |                                                                                                                          
-  |===============================                                                                                     |  27%
-  |                                                                                                                          
-  |================================                                                                                    |  27%
-  |                                                                                                                          
-  |================================                                                                                    |  28%
-  |                                                                                                                          
-  |=================================                                                                                   |  28%
-  |                                                                                                                          
-  |=================================                                                                                   |  29%
-  |                                                                                                                          
-  |==================================                                                                                  |  29%
-  |                                                                                                                          
-  |==================================                                                                                  |  30%
-  |                                                                                                                          
-  |===================================                                                                                 |  30%
-  |                                                                                                                          
-  |===================================                                                                                 |  31%
-  |                                                                                                                          
-  |====================================                                                                                |  31%
-  |                                                                                                                          
-  |=====================================                                                                               |  32%
-  |                                                                                                                          
-  |======================================                                                                              |  32%
-  |                                                                                                                          
-  |======================================                                                                              |  33%
-  |                                                                                                                          
-  |=======================================                                                                             |  33%
-  |                                                                                                                          
-  |=======================================                                                                             |  34%
-  |                                                                                                                          
-  |========================================                                                                            |  34%
-  |                                                                                                                          
-  |========================================                                                                            |  35%
-  |                                                                                                                          
-  |=========================================                                                                           |  35%
-  |                                                                                                                          
-  |=========================================                                                                           |  36%
-  |                                                                                                                          
-  |==========================================                                                                          |  36%
-  |                                                                                                                          
-  |==========================================                                                                          |  37%
-  |                                                                                                                          
-  |===========================================                                                                         |  37%
-  |                                                                                                                          
-  |============================================                                                                        |  38%
-  |                                                                                                                          
-  |=============================================                                                                       |  39%
-  |                                                                                                                          
-  |==============================================                                                                      |  39%
-  |                                                                                                                          
-  |==============================================                                                                      |  40%
-  |                                                                                                                          
-  |===============================================                                                                     |  40%
-  |                                                                                                                          
-  |===============================================                                                                     |  41%
-  |                                                                                                                          
-  |================================================                                                                    |  41%
-  |                                                                                                                          
-  |================================================                                                                    |  42%
-  |                                                                                                                          
-  |=================================================                                                                   |  42%
-  |                                                                                                                          
-  |=================================================                                                                   |  43%
-  |                                                                                                                          
-  |==================================================                                                                  |  43%
-  |                                                                                                                          
-  |==================================================                                                                  |  44%
-  |                                                                                                                          
-  |===================================================                                                                 |  44%
-  |                                                                                                                          
-  |====================================================                                                                |  45%
-  |                                                                                                                          
-  |=====================================================                                                               |  45%
-  |                                                                                                                          
-  |=====================================================                                                               |  46%
-  |                                                                                                                          
-  |======================================================                                                              |  46%
-  |                                                                                                                          
-  |======================================================                                                              |  47%
-  |                                                                                                                          
-  |=======================================================                                                             |  47%
-  |                                                                                                                          
-  |=======================================================                                                             |  48%
-  |                                                                                                                          
-  |========================================================                                                            |  48%
-  |                                                                                                                          
-  |========================================================                                                            |  49%
-  |                                                                                                                          
-  |=========================================================                                                           |  49%
-  |                                                                                                                          
-  |=========================================================                                                           |  50%
-  |                                                                                                                          
-  |==========================================================                                                          |  50%
-  |                                                                                                                          
-  |===========================================================                                                         |  50%
-  |                                                                                                                          
-  |===========================================================                                                         |  51%
-  |                                                                                                                          
-  |============================================================                                                        |  51%
-  |                                                                                                                          
-  |============================================================                                                        |  52%
-  |                                                                                                                          
-  |=============================================================                                                       |  52%
-  |                                                                                                                          
-  |=============================================================                                                       |  53%
-  |                                                                                                                          
-  |==============================================================                                                      |  53%
-  |                                                                                                                          
-  |==============================================================                                                      |  54%
-  |                                                                                                                          
-  |===============================================================                                                     |  54%
-  |                                                                                                                          
-  |===============================================================                                                     |  55%
-  |                                                                                                                          
-  |================================================================                                                    |  55%
-  |                                                                                                                          
-  |=================================================================                                                   |  56%
-  |                                                                                                                          
-  |==================================================================                                                  |  56%
-  |                                                                                                                          
-  |==================================================================                                                  |  57%
-  |                                                                                                                          
-  |===================================================================                                                 |  57%
-  |                                                                                                                          
-  |===================================================================                                                 |  58%
-  |                                                                                                                          
-  |====================================================================                                                |  58%
-  |                                                                                                                          
-  |====================================================================                                                |  59%
-  |                                                                                                                          
-  |=====================================================================                                               |  59%
-  |                                                                                                                          
-  |=====================================================================                                               |  60%
-  |                                                                                                                          
-  |======================================================================                                              |  60%
-  |                                                                                                                          
-  |======================================================================                                              |  61%
-  |                                                                                                                          
-  |=======================================================================                                             |  61%
-  |                                                                                                                          
-  |========================================================================                                            |  62%
-  |                                                                                                                          
-  |=========================================================================                                           |  63%
-  |                                                                                                                          
-  |==========================================================================                                          |  63%
-  |                                                                                                                          
-  |==========================================================================                                          |  64%
-  |                                                                                                                          
-  |===========================================================================                                         |  64%
-  |                                                                                                                          
-  |===========================================================================                                         |  65%
-  |                                                                                                                          
-  |============================================================================                                        |  65%
-  |                                                                                                                          
-  |============================================================================                                        |  66%
-  |                                                                                                                          
-  |=============================================================================                                       |  66%
-  |                                                                                                                          
-  |=============================================================================                                       |  67%
-  |                                                                                                                          
-  |==============================================================================                                      |  67%
-  |                                                                                                                          
-  |==============================================================================                                      |  68%
-  |                                                                                                                          
-  |===============================================================================                                     |  68%
-  |                                                                                                                          
-  |================================================================================                                    |  69%
-  |                                                                                                                          
-  |=================================================================================                                   |  69%
-  |                                                                                                                          
-  |=================================================================================                                   |  70%
-  |                                                                                                                          
-  |==================================================================================                                  |  70%
-  |                                                                                                                          
-  |==================================================================================                                  |  71%
-  |                                                                                                                          
-  |===================================================================================                                 |  71%
-  |                                                                                                                          
-  |===================================================================================                                 |  72%
-  |                                                                                                                          
-  |====================================================================================                                |  72%
-  |                                                                                                                          
-  |====================================================================================                                |  73%
-  |                                                                                                                          
-  |=====================================================================================                               |  73%
-  |                                                                                                                          
-  |=====================================================================================                               |  74%
-  |                                                                                                                          
-  |======================================================================================                              |  74%
-  |                                                                                                                          
-  |=======================================================================================                             |  75%
-  |                                                                                                                          
-  |========================================================================================                            |  75%
-  |                                                                                                                          
-  |========================================================================================                            |  76%
-  |                                                                                                                          
-  |=========================================================================================                           |  76%
-  |                                                                                                                          
-  |=========================================================================================                           |  77%
-  |                                                                                                                          
-  |==========================================================================================                          |  77%
-  |                                                                                                                          
-  |==========================================================================================                          |  78%
-  |                                                                                                                          
-  |===========================================================================================                         |  78%
-  |                                                                                                                          
-  |===========================================================================================                         |  79%
-  |                                                                                                                          
-  |============================================================================================                        |  79%
-  |                                                                                                                          
-  |============================================================================================                        |  80%
-  |                                                                                                                          
-  |=============================================================================================                       |  80%
-  |                                                                                                                          
-  |==============================================================================================                      |  81%
-  |                                                                                                                          
-  |===============================================================================================                     |  82%
-  |                                                                                                                          
-  |================================================================================================                    |  82%
-  |                                                                                                                          
-  |================================================================================================                    |  83%
-  |                                                                                                                          
-  |=================================================================================================                   |  83%
-  |                                                                                                                          
-  |=================================================================================================                   |  84%
-  |                                                                                                                          
-  |==================================================================================================                  |  84%
-  |                                                                                                                          
-  |==================================================================================================                  |  85%
-  |                                                                                                                          
-  |===================================================================================================                 |  85%
-  |                                                                                                                          
-  |===================================================================================================                 |  86%
-  |                                                                                                                          
-  |====================================================================================================                |  86%
-  |                                                                                                                          
-  |====================================================================================================                |  87%
-  |                                                                                                                          
-  |=====================================================================================================               |  87%
-  |                                                                                                                          
-  |======================================================================================================              |  88%
-  |                                                                                                                          
-  |=======================================================================================================             |  88%
-  |                                                                                                                          
-  |=======================================================================================================             |  89%
-  |                                                                                                                          
-  |========================================================================================================            |  89%
-  |                                                                                                                          
-  |========================================================================================================            |  90%
-  |                                                                                                                          
-  |=========================================================================================================           |  90%
-  |                                                                                                                          
-  |=========================================================================================================           |  91%
-  |                                                                                                                          
-  |==========================================================================================================          |  91%
-  |                                                                                                                          
-  |==========================================================================================================          |  92%
-  |                                                                                                                          
-  |===========================================================================================================         |  92%
-  |                                                                                                                          
-  |===========================================================================================================         |  93%
-  |                                                                                                                          
-  |============================================================================================================        |  93%
-  |                                                                                                                          
-  |=============================================================================================================       |  94%
-  |                                                                                                                          
-  |==============================================================================================================      |  94%
-  |                                                                                                                          
-  |==============================================================================================================      |  95%
-  |                                                                                                                          
-  |===============================================================================================================     |  95%
-  |                                                                                                                          
-  |===============================================================================================================     |  96%
-  |                                                                                                                          
-  |================================================================================================================    |  96%
-  |                                                                                                                          
-  |================================================================================================================    |  97%
-  |                                                                                                                          
-  |=================================================================================================================   |  97%
-  |                                                                                                                          
-  |=================================================================================================================   |  98%
-  |                                                                                                                          
-  |==================================================================================================================  |  98%
-  |                                                                                                                          
-  |==================================================================================================================  |  99%
-  |                                                                                                                          
-  |=================================================================================================================== |  99%
-  |                                                                                                                          
-  |====================================================================================================================| 100%
      
+  |                                                                            
+  |                                                                      |   0%
+  |                                                                            
+  |                                                                      |   1%
+  |                                                                            
+  |=                                                                     |   1%
+  |                                                                            
+  |=                                                                     |   2%
+  |                                                                            
+  |==                                                                    |   2%
+  |                                                                            
+  |==                                                                    |   3%
+  |                                                                            
+  |===                                                                   |   4%
+  |                                                                            
+  |===                                                                   |   5%
+  |                                                                            
+  |====                                                                  |   5%
+  |                                                                            
+  |====                                                                  |   6%
+  |                                                                            
+  |=====                                                                 |   7%
+  |                                                                            
+  |=====                                                                 |   8%
+  |                                                                            
+  |======                                                                |   8%
+  |                                                                            
+  |======                                                                |   9%
+  |                                                                            
+  |=======                                                               |   9%
+  |                                                                            
+  |=======                                                               |  10%
+  |                                                                            
+  |=======                                                               |  11%
+  |                                                                            
+  |========                                                              |  11%
+  |                                                                            
+  |========                                                              |  12%
+  |                                                                            
+  |=========                                                             |  12%
+  |                                                                            
+  |=========                                                             |  13%
+  |                                                                            
+  |==========                                                            |  14%
+  |                                                                            
+  |==========                                                            |  15%
+  |                                                                            
+  |===========                                                           |  15%
+  |                                                                            
+  |===========                                                           |  16%
+  |                                                                            
+  |============                                                          |  17%
+  |                                                                            
+  |============                                                          |  18%
+  |                                                                            
+  |=============                                                         |  18%
+  |                                                                            
+  |=============                                                         |  19%
+  |                                                                            
+  |==============                                                        |  19%
+  |                                                                            
+  |==============                                                        |  20%
+  |                                                                            
+  |==============                                                        |  21%
+  |                                                                            
+  |===============                                                       |  21%
+  |                                                                            
+  |===============                                                       |  22%
+  |                                                                            
+  |================                                                      |  22%
+  |                                                                            
+  |================                                                      |  23%
+  |                                                                            
+  |=================                                                     |  24%
+  |                                                                            
+  |=================                                                     |  25%
+  |                                                                            
+  |==================                                                    |  25%
+  |                                                                            
+  |==================                                                    |  26%
+  |                                                                            
+  |===================                                                   |  27%
+  |                                                                            
+  |===================                                                   |  28%
+  |                                                                            
+  |====================                                                  |  28%
+  |                                                                            
+  |====================                                                  |  29%
+  |                                                                            
+  |=====================                                                 |  29%
+  |                                                                            
+  |=====================                                                 |  30%
+  |                                                                            
+  |=====================                                                 |  31%
+  |                                                                            
+  |======================                                                |  31%
+  |                                                                            
+  |======================                                                |  32%
+  |                                                                            
+  |=======================                                               |  32%
+  |                                                                            
+  |=======================                                               |  33%
+  |                                                                            
+  |=======================                                               |  34%
+  |                                                                            
+  |========================                                              |  34%
+  |                                                                            
+  |========================                                              |  35%
+  |                                                                            
+  |=========================                                             |  35%
+  |                                                                            
+  |=========================                                             |  36%
+  |                                                                            
+  |==========================                                            |  36%
+  |                                                                            
+  |==========================                                            |  37%
+  |                                                                            
+  |==========================                                            |  38%
+  |                                                                            
+  |===========================                                           |  38%
+  |                                                                            
+  |===========================                                           |  39%
+  |                                                                            
+  |============================                                          |  39%
+  |                                                                            
+  |============================                                          |  40%
+  |                                                                            
+  |============================                                          |  41%
+  |                                                                            
+  |=============================                                         |  41%
+  |                                                                            
+  |=============================                                         |  42%
+  |                                                                            
+  |==============================                                        |  42%
+  |                                                                            
+  |==============================                                        |  43%
+  |                                                                            
+  |==============================                                        |  44%
+  |                                                                            
+  |===============================                                       |  44%
+  |                                                                            
+  |===============================                                       |  45%
+  |                                                                            
+  |================================                                      |  45%
+  |                                                                            
+  |================================                                      |  46%
+  |                                                                            
+  |=================================                                     |  46%
+  |                                                                            
+  |=================================                                     |  47%
+  |                                                                            
+  |=================================                                     |  48%
+  |                                                                            
+  |==================================                                    |  48%
+  |                                                                            
+  |==================================                                    |  49%
+  |                                                                            
+  |===================================                                   |  49%
+  |                                                                            
+  |===================================                                   |  50%
+  |                                                                            
+  |===================================                                   |  51%
+  |                                                                            
+  |====================================                                  |  51%
+  |                                                                            
+  |====================================                                  |  52%
+  |                                                                            
+  |=====================================                                 |  52%
+  |                                                                            
+  |=====================================                                 |  53%
+  |                                                                            
+  |=====================================                                 |  54%
+  |                                                                            
+  |======================================                                |  54%
+  |                                                                            
+  |======================================                                |  55%
+  |                                                                            
+  |=======================================                               |  55%
+  |                                                                            
+  |=======================================                               |  56%
+  |                                                                            
+  |========================================                              |  56%
+  |                                                                            
+  |========================================                              |  57%
+  |                                                                            
+  |========================================                              |  58%
+  |                                                                            
+  |=========================================                             |  58%
+  |                                                                            
+  |=========================================                             |  59%
+  |                                                                            
+  |==========================================                            |  59%
+  |                                                                            
+  |==========================================                            |  60%
+  |                                                                            
+  |==========================================                            |  61%
+  |                                                                            
+  |===========================================                           |  61%
+  |                                                                            
+  |===========================================                           |  62%
+  |                                                                            
+  |============================================                          |  62%
+  |                                                                            
+  |============================================                          |  63%
+  |                                                                            
+  |============================================                          |  64%
+  |                                                                            
+  |=============================================                         |  64%
+  |                                                                            
+  |=============================================                         |  65%
+  |                                                                            
+  |==============================================                        |  65%
+  |                                                                            
+  |==============================================                        |  66%
+  |                                                                            
+  |===============================================                       |  66%
+  |                                                                            
+  |===============================================                       |  67%
+  |                                                                            
+  |===============================================                       |  68%
+  |                                                                            
+  |================================================                      |  68%
+  |                                                                            
+  |================================================                      |  69%
+  |                                                                            
+  |=================================================                     |  69%
+  |                                                                            
+  |=================================================                     |  70%
+  |                                                                            
+  |=================================================                     |  71%
+  |                                                                            
+  |==================================================                    |  71%
+  |                                                                            
+  |==================================================                    |  72%
+  |                                                                            
+  |===================================================                   |  72%
+  |                                                                            
+  |===================================================                   |  73%
+  |                                                                            
+  |====================================================                  |  74%
+  |                                                                            
+  |====================================================                  |  75%
+  |                                                                            
+  |=====================================================                 |  75%
+  |                                                                            
+  |=====================================================                 |  76%
+  |                                                                            
+  |======================================================                |  77%
+  |                                                                            
+  |======================================================                |  78%
+  |                                                                            
+  |=======================================================               |  78%
+  |                                                                            
+  |=======================================================               |  79%
+  |                                                                            
+  |========================================================              |  79%
+  |                                                                            
+  |========================================================              |  80%
+  |                                                                            
+  |========================================================              |  81%
+  |                                                                            
+  |=========================================================             |  81%
+  |                                                                            
+  |=========================================================             |  82%
+  |                                                                            
+  |==========================================================            |  82%
+  |                                                                            
+  |==========================================================            |  83%
+  |                                                                            
+  |===========================================================           |  84%
+  |                                                                            
+  |===========================================================           |  85%
+  |                                                                            
+  |============================================================          |  85%
+  |                                                                            
+  |============================================================          |  86%
+  |                                                                            
+  |=============================================================         |  87%
+  |                                                                            
+  |=============================================================         |  88%
+  |                                                                            
+  |==============================================================        |  88%
+  |                                                                            
+  |==============================================================        |  89%
+  |                                                                            
+  |===============================================================       |  89%
+  |                                                                            
+  |===============================================================       |  90%
+  |                                                                            
+  |===============================================================       |  91%
+  |                                                                            
+  |================================================================      |  91%
+  |                                                                            
+  |================================================================      |  92%
+  |                                                                            
+  |=================================================================     |  92%
+  |                                                                            
+  |=================================================================     |  93%
+  |                                                                            
+  |==================================================================    |  94%
+  |                                                                            
+  |==================================================================    |  95%
+  |                                                                            
+  |===================================================================   |  95%
+  |                                                                            
+  |===================================================================   |  96%
+  |                                                                            
+  |====================================================================  |  97%
+  |                                                                            
+  |====================================================================  |  98%
+  |                                                                            
+  |===================================================================== |  98%
+  |                                                                            
+  |===================================================================== |  99%
+  |                                                                            
+  |======================================================================|  99%
+  |                                                                            
+  |======================================================================| 100%
      
 
 
 
 
      Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.
      
 
 
-
+
 
      # The random results are a matrix
 random_long_df <- random_gsva_results %>%
   data.frame() %>%
@@ -1130,15 +1068,14 @@ 
      A note on gene set size
      
   dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
   # We want to plot smallest no. genes -> largest no. genes
   dplyr::mutate(gene_set_size = factor(gene_set_size, 
-                                       levels = c(15, 25, 50, 100, 250, 500)))
-
      
+                                       levels = c(15, 25, 50, 100, 250, 500)))
      
 
 
 
 
      Let’s make a violin plot so we can look at the distribution of scores by gene set size.
      
 
 
-
+
 
      # Violin plot comparing GSVA scores of different random gene set sizes
 random_long_df %>%
   ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
@@ -1161,8 +1098,8 @@ 
      A note on gene set size
      
                 y = "GSVA score") +
   ggplot2::theme_bw()
      
 
-
-
      
+
+
      
 
 
 
@@ -1179,7 +1116,7 @@ 
      How can you use these scores?
      
 
      Write results to file
      
 
 
-
+
 
      gsva_results %>%
   as.data.frame() %>%
   tibble::rownames_to_column("pathway") %>%
@@ -1193,51 +1130,75 @@ 
      Write results to file
      
 
      Session Info
      
 
 
-
+
 
      sessionInfo()
      
 
-
+
 
      R version 4.0.3 (2020-10-10)
 Platform: x86_64-pc-linux-gnu (64-bit)
-Running under: Ubuntu 18.04.3 LTS
+Running under: Ubuntu 20.04 LTS
 
 Matrix products: default
-BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
-LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
+BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so
 
 locale:
- [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
- [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
-[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
+ [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
+ [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
+ [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
+ [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
+ [9] LC_ADDRESS=C               LC_TELEPHONE=C            
+[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
 
 attached base packages:
-[1] stats     graphics  grDevices datasets  utils     methods   base     
+[1] stats     graphics  grDevices utils     datasets  methods   base     
 
 other attached packages:
-[1] GSVA_1.38.2    magrittr_2.0.1
+[1] GSVA_1.38.2    magrittr_2.0.1 optparse_1.6.6
 
 loaded via a namespace (and not attached):
- [1] MatrixGenerics_1.2.0        Biobase_2.50.0              httr_1.4.2                  tidyr_1.1.2                
- [5] bit64_4.0.5                 jsonlite_1.7.2              assertthat_0.2.1            BiocManager_1.30.10        
- [9] stats4_4.0.3                blob_1.2.1                  renv_0.12.5-2               GenomeInfoDbData_1.2.4     
-[13] pillar_1.4.7                RSQLite_2.2.3               lattice_0.20-41             glue_1.4.2                 
-[17] limma_3.46.0                digest_0.6.27               GenomicRanges_1.42.0        XVector_0.30.0             
-[21] colorspace_2.0-0            Matrix_1.3-2                GSEABase_1.52.1             XML_3.99-0.5               
-[25] pkgconfig_2.0.3             zlibbioc_1.36.0             purrr_0.3.4                 xtable_1.8-4               
-[29] mvtnorm_1.1-1               scales_1.1.1                BiocParallel_1.24.1         emmeans_1.5.3              
-[33] tibble_3.0.5                annotate_1.68.0             farver_2.0.3                generics_0.1.0             
-[37] IRanges_2.24.1              ggplot2_3.3.3               ellipsis_0.3.1              SummarizedExperiment_1.20.0
-[41] BiocGenerics_0.36.0         cli_2.2.0                   crayon_1.3.4                memoise_1.1.0              
-[45] estimability_1.3            fansi_0.4.2                 nlme_3.1-151                graph_1.68.0               
-[49] tools_4.0.3                 hms_1.0.0                   lifecycle_0.2.0             matrixStats_0.57.0         
-[53] stringr_1.4.0               S4Vectors_0.28.1            fftw_1.0-6                  munsell_0.5.0              
-[57] DelayedArray_0.16.2         AnnotationDbi_1.52.0        compiler_4.0.3              GenomeInfoDb_1.26.2        
-[61] rlang_0.4.10                grid_4.0.3                  RCurl_1.98-1.2              rstudioapi_0.13            
-[65] labeling_0.4.2              bitops_1.0-6                qusage_2.24.0               gtable_0.3.0               
-[69] DBI_1.1.1                   R6_2.5.0                    knitr_1.30                  dplyr_1.0.3                
-[73] bit_4.0.4                   readr_1.4.0                 stringi_1.5.3               parallel_4.0.3             
-[77] Rcpp_1.0.6                  vctrs_0.3.6                 tidyselect_1.1.0            xfun_0.20                  
-[81] coda_0.19-4                
      
+ [1] MatrixGenerics_1.2.0        Biobase_2.50.0             
+ [3] httr_1.4.2                  tidyr_1.1.2                
+ [5] bit64_4.0.5                 jsonlite_1.7.2             
+ [7] assertthat_0.2.1            stats4_4.0.3               
+ [9] blob_1.2.1                  GenomeInfoDbData_1.2.4     
+[11] yaml_2.2.1                  pillar_1.4.7               
+[13] RSQLite_2.2.3               lattice_0.20-41            
+[15] glue_1.4.2                  limma_3.46.0               
+[17] digest_0.6.27               GenomicRanges_1.42.0       
+[19] XVector_0.30.0              colorspace_2.0-0           
+[21] htmltools_0.5.1.1           Matrix_1.3-2               
+[23] GSEABase_1.52.1             XML_3.99-0.5               
+[25] pkgconfig_2.0.3             zlibbioc_1.36.0            
+[27] purrr_0.3.4                 xtable_1.8-4               
+[29] mvtnorm_1.1-1               scales_1.1.1               
+[31] getopt_1.20.3               BiocParallel_1.24.1        
+[33] emmeans_1.5.3               tibble_3.0.5               
+[35] annotate_1.68.0             farver_2.0.3               
+[37] ggplot2_3.3.3               generics_0.1.0             
+[39] IRanges_2.24.1              ellipsis_0.3.1             
+[41] SummarizedExperiment_1.20.0 BiocGenerics_0.36.0        
+[43] cli_2.2.0                   crayon_1.3.4               
+[45] memoise_1.1.0               estimability_1.3           
+[47] evaluate_0.14               ps_1.5.0                   
+[49] fansi_0.4.2                 nlme_3.1-151               
+[51] graph_1.68.0                tools_4.0.3                
+[53] hms_1.0.0                   lifecycle_0.2.0            
+[55] matrixStats_0.57.0          stringr_1.4.0              
+[57] S4Vectors_0.28.1            fftw_1.0-6                 
+[59] munsell_0.5.0               DelayedArray_0.16.2        
+[61] AnnotationDbi_1.52.0        compiler_4.0.3             
+[63] GenomeInfoDb_1.26.2         rlang_0.4.10               
+[65] grid_4.0.3                  RCurl_1.98-1.2             
+[67] rstudioapi_0.13             labeling_0.4.2             
+[69] bitops_1.0-6                rmarkdown_2.6              
+[71] qusage_2.24.0               gtable_0.3.0               
+[73] DBI_1.1.1                   R6_2.5.0                   
+[75] knitr_1.30                  dplyr_1.0.3                
+[77] bit_4.0.4                   readr_1.4.0                
+[79] stringi_1.5.3               parallel_4.0.3             
+[81] Rcpp_1.0.6                  vctrs_0.3.6                
+[83] tidyselect_1.1.0            xfun_0.20                  
+[85] coda_0.19-4                
      
 
 
 
diff --git a/scRNA-seq/02-normalizing_scRNA-seq.nb.html b/scRNA-seq/02-normalizing_scRNA-seq.nb.html
index 8557a52f..6f490d01 100644
--- a/scRNA-seq/02-normalizing_scRNA-seq.nb.html
+++ b/scRNA-seq/02-normalizing_scRNA-seq.nb.html
@@ -762,7 +762,7 @@ 
      Identify marker genes