New Team Member Onboarding Kit

[patricksimonian]

This onboarding kit should be made in mind for junior dev personas.

Somethings to keep in mind

• DevHub overview/readmes

• Algolia Prod Account

• describing project structure

• describing project structure pit falls and journey map

• tech stack and resources to learn more

• unit-testing

• e2e testing (and its pitfalls)

• styled-components and its journey map

• how devhub works

• how the ci/cd pipeline works (high-level explanation on the usage of github actions and jenkins)

[patricksimonian]

@sheaphillips I am compiling an onboarding list for Billy because I will be away on his first day. Is there anything here that I am missing?

[DerekSiemens]

Here's a react tutorial I did during the first couple days to help get a feel for the basics, https://reactjs.org/tutorial/tutorial.html , its a learn by doing approach but also links out to a step by step guide for learning the concepts from the ground up

[patricksimonian]

Github Flow

by @DerekSgov
What is GitHub Flow and How Does the DevHub Use it?

GitHub flow is a lightweight, branch-based workflow that supports teams and projects where deployments are made regularly.

When developing in GitHub flow, branches are created off of master. These branches could contain new features, bug fixes, content curation etc, and the branch names should reflect what type of work is being done and the purpose of that branch.

When working on your branch, commit early and often, with clear and concise commit messages as this will allow others on your team to understand your work more effectively and allow for better de-bugging if an issue arrises. Also try to make sure your code follows best practices and is well documented.

At some point during the development process, you will need to open a pull request for your branch. This can be either when you are almost done your work on the branch, stuck on an issue and need some advice, or really any time during the development process. This allows team members to easily see what you’re working on, give feedback and review your pull request to put it into production.

After it has been approved, you can then promote it through the various stages of the DevHubs Jenkins pipeline, with the final one putting the changes into production. After that has been done and the app is running smoothly, go back to your PR and merge it there, deleting the branch after.

What if Another Team Member Merges New Code While you’re still working?

To handle this issue, the “rebase” command is used. Rebase, similar to “merge” also integrates changes from a branch to another, but unlike merge it rewrites the commit history in order to produce a straight, linear succession of commits.

When visualized, it looks something like this

A rebase looks like this (https://www.atlassian.com/git/tutorials/rewriting-history/git-rebase)

To implement a successful rebase, first you would make sure your master branch is up to date, then on your feature branch do the following command ‘git rebase master’, this would rebase the current production code with your changes. You will then have to make sure there are no conflicts, after that a final step of force pushing as you have previously done to create your pull request is required.

(More detailed info on how to use here https://git-scm.com/docs/git-rebase)

[patricksimonian]

Devhub Overview

As described in the readme, the Devhub is front end application that pulls data from a variety of sources: Github, Eventbrite, Local Filesystem, and converts them into objects that are then presen ted as either standalone HTML pages, or a collection of HTML pages within a 'Topic'

A few writeups on the devhub can be found here

• about devhub
• README.md

[patricksimonian]

Project Structure Pit Falls and Journey Map

You will notice the directory structure could be cleaned up and this is something we are totally aware of.

For a journey map what I am hoping to see is:

• the complete isolation of md documents that are used as gatsby data sources into a single directory, currently we have multiple directories serving this purpose
• as components are refactored, their file/folder structure is refactored as well to adopt a new standard yet to be defined (my guess is we will use the implicit import method where all component directories will contain index.js files possibly)

[patricksimonian]

Testing

Unit Testing

Unit testing is done with Jest. Jest is pre-built into gatsby applications.

Enzyme

Originally, our jest unit tests were interfacing with Enzyme, a react testing utility library but there were incompatibilities with Enzyme and React 16 features. We are no longer writing tests using enzyme. The plan is to deprecate its usage and remove the library altogether.

@testing-library/react

We are using testing library in place of Jest. This testing library has several benefits as described by the author of the repo. The tests now behave more like how a user would interact with a component which really helps to nail down what spec we are testing for.

more info here https://github.com/testing-library/react-testing-library

E2E Testing

What is it?

End to end testing is a way of writing test cases that simulate user behaviour from start to finish. Tests are run against a running version of the front end in a controlled browser environment. Interactions are all done through the ui and then assertions are made.

Library

The library we are using is called Cypress JS. This library is shiny but promising, the developer experience is incredible for writing functional tests.

Pitfalls

We have yet found a way to correctly integrate it into our pipeline. For some reason, the tests work fine when run locally but fail when run in platforms like Github Actions.

update Jan 6th 2019
E2E tests are now apart of our github action workflows. PR #1210

[patricksimonian]

Tech Stack

These are the following technologies and key libraries that the Devhub is using. Please note that there may be libraries that are being deprecated.

• Gatsby JS https://www.gatsbyjs.org/
• React (Gatsby leverages React) https://reactjs.org/
• GraphQL (Gatsby leverages GraphQL, gatsby has great tutorials on graphql) https://graphql.org/
• Styled Components with Emotion JS https://emotion.sh/docs/introduction
• Jest for unit testing https://jestjs.io/
• Cypress for E2E testing https://cypress.io

[patricksimonian]

Styled Components

What are they?

Styled Components is a way of creating React Components that have CSS styles embedded at them. More than that, they provide a way to create isolated, programmable, and composable styles.

There are a couple of styled component libraries currently dominating NPM at the moment and they are styled-components and emotion. We are using emotion. Emotion has improved performance and utility over styled components. When and if one of these libraries get's deprecated in favor of the 'winning' one, the api for both libraries will seamlessly integrate as they both use string template functions to create styles.

What about CSS Modules?

In the early days of Devhub, we were utilizing a styling technique called CSS Modules. You may notice some component files with a stylesheet named something like component.module.css.
This was a great way to create isolated styles but had a few drawbacks.

• Stylesheets were still in vanilla CSS (so no logic or themes could be embedded).
• More files being created per component make organization clunky.

Journey Map

Since we are in the inbetween stages of CSS Modules and Styled Components. All future styles should utilize styled components. If you are working on a component that uses CSS Modules, please convert it to emotion.js

End Goal

1. Convert all CSS modules to styled components
2. Reduce the repetition in the amount of styled component we are using and abstract them down to common ui components (we will find that the CSS module conversion will introduce a ton of repetition)
3. Integrate styled-system as a predictable way of creating layouts, spacing and font sizing.

[patricksimonian]

Project Structure

The root application structure looks somethinng like this.

.
├── app-web
├── .jenkins
├── .pipeline
├── docs
├── functional-tests
├── matomo
├── openshift
└── pipeline

app-web

This is where the Gatsby Project lives and where you will most likely be spending your time working in

What else?

The app-web directory is essentially and standard Gatsby Project. Gatsby Projects include some conventions from React projects.

Journey Map

You will node this directory needs some love! It is a bit messy, and we are looking for a way to clean it up!

.jenkins

This is the bcdk jenkins configuration and pipeline (including all jobs for other projects without our openshift namespace)

.pipeline

This is the bcdk pipeline for the devhub gatsby project

docs

Misc docs in regards to the project

functional tests

This is the BDD stack which is currently not in use and may be cleaned up.

matomo

This is all the infra code required to support the matomo instance in our tools namespace

openshift

This is all the infra code required to support the devhub, the .pipeline folder references this code

pipeline

This is the gradle runner for the BDD stack which, again, may be removed shortly since we are not using BDD stack

[patricksimonian]

How the Devhub Works

Introduction

The Devhub has a couple of primary functions. These are:

1. To get content from a variety of sources and display them in meaningful ways on the site. This content can be from markdown files, links to HTML sites, Eventbrite and more

2. To index these sources for searchability as well as provide a means to search external sources such as Rocket.Chat, Github

This document will discuss in detail how data is sourced; how it is annotated in different ways with metadata to make meaningful connections between data points; how the data is indexed; and lastly, how external search sources are made available.

How Data is Sourced

When you goto the Devhub, you'll see something like this.

Clicking on that card will render something like this.

This is clearly some type of documentation. How does the Devhub render that, how does it get there?

Data is primarily sourced by way of Gatsby Source Plugin. We leverage a combination of 3rd party (gatsby-source-eventbrite), hosted authored (gatsby-plugin-matomo,gatsby-source-github-raw), and locally authored plugins (gatsby-source-github-all).

Most of the content that is sourced for the Devhub is managed via a set of Registry Files found at app-web/registry. These files declare arguments for a particular set of source plugins and also establish the initial metadata for each data point that will eventually be created.

Making a card

To make the card, most data sources come with some metadata that was either explicitly written with the data (such as frontmatter in a markdown file), assigned in the devhub registry files, or implied by some conventions.

The metadata that makes up a card is a title, description, link, resourceType, and a few more. More information on what it takes to make a card can be found here

Making the page

Essentially most content comes in as Markdown files, we use a transformer plugin called gatsby-transformer-remark which converts it into useable HTML. This is then embedded into a set of React Template files found at app-web/templates based on certain circumstances.

How Data is Annotated to make connections

Each data point that represents a card can be thought of as an individual Node. All these Nodes come from unlike data structures. The Devhub, based on certain conventions, makes manual connections between these nodes to associate certain relationships.

Nodes can be connected to other Special Nodes called Topics. This creates a parent-child relationship. This relationship is many to many. A child node can belong to many topic nodes.
The metadata that describes this relationship can be found within a nodes data point node.fields.topics.

We utilize several Gatsby Node API's to accomplish making connections. They can all be found within the app-web/gatsby folder.

Making connections between unlike nodes is complex. We have tried to achieve this by providing a standard interface between sourced nodes within the node.fields property (as described in the Devhub Card Spec document).

How Data is Indexed

We are leveraging Elasticlunr as our front end search engine. There is a gatsby plugin that helps initialize and program the index for this plugin. This plus a custom React Hook, useSearch, is how we are able to search the index. The configuration for producing the index can be found within gatsby-config.js.

External Search Sources

When searching through the local index, additional functions can be called to leverage the ApolloClient React component. This ApolloClient will pass a graphql query to a Federated Search called Search Gate. The results are passed back to the component and then are subsequently rendered. This is how Rocket.Chat search results are displayed when authenticated and searching.

[patricksimonian]

CI/CD Pipeline

We are currently leveraging two different pipelines to accomplish CI and CD.

Continuous Integration is done with Github Actions.

Although not continuous, our deployment to Openshift is accomplished with Jenkins.

Github Actions and Journey Map

Github actions have been an excellent way for us to offload some heavy lifting to check unit and integration tests. In addition, the PR clearly displays whether tests passed or not which is a bonus.

To modify the Github Action, you can navigate to the .github folder inside of the repo (on Github) and go through the GUI.

Currently we are blocked on running E2E tests with Github Actions. For some reason, we are getting some issues with utilizing the Cypress docker image. This appears to be a known issue and hopefully gets resolved shortly.

Jenkins Pipeline and Road Map.

Jenkins is responsible for building and deploying our images to each namespace in the Devhub Project Set. It also does extra things like provision a keycloak client during dev deploys.

There are some things left to be desired with Jenkins which we will hope to work on:

• automatic pr merge after clean up stage
• setting up sonarqube for static code analysis

[w8896699]

https://www.gatsbyjs.org/tutorial/part-one/
I found that this Gatsby official document is really helpful.

[patricksimonian]

Algolia Production Account

Our team account is registered under my email. If you require access to the production Algolia account please contact me :)