Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default Review CLI interface / Add walkthrough in Custom React docs page #697

Merged
merged 5 commits into from
Mar 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 79 additions & 7 deletions docs/web/docs/guides/tutorials/custom_react.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ mephisto:
task_tags: "test,simple,button"
```

The only change we'll make is to the `task_name` (as you should on any new tasks!), but we will update the other fields as we go. It's important to note the `task_source` and `extra_source_dir` arguments though, as this is where the `Blueprint` will be looking a compiled react app bundle, and for extra static resources for the page.
The only change we'll make is to the `task_name` (as you should on any new tasks!), but we will update the other fields as we go. It's important to note the `task_source` and `extra_source_dir` arguments though, as this is where the `StaticReactBlueprint` class will be looking for the compiled React app's Javascript bundle as well as a folder for extra static resources for the page, respectively.

### 1.2 Launching the task
From the current directory, you should be able to execute the run script and get a job working. We're using a different `task_name` to prevent polluting our later task with data that won't share the same format. It is a good practice to do this with initial iterations, and to change the `task_name` any time you change input or output arguments.
Expand Down Expand Up @@ -201,7 +201,7 @@ onClick={() => onSubmit({ rating: "bad" })}
```
Based on how `onSubmit` is wired to Mephisto's `handleSubmit` function, anything (json-serializable) in the object passed to `onSubmit` will be what is saved for the task. For this tutorial we'll want to put something else useful into this object. To this end, let's add an input box that allows workers to submit an edit to correct the sentence in some way.

We can add some state to `SimpleFrontend` with one line. This provides us with an `editedText` value initialized to `taskData.text`, which we'll set to track the worker edits, and a setter to alter that value.
We can add some state to `SimpleFrontend` with the [`useState` React Hook](https://reactjs.org/docs/hooks-state.html). This provides us with an `editedText` value initialized to `taskData.text`, which we'll set to track the worker edits, and a setter to alter that value.
```jsx
// webapp/src/components/core_components.jsx
function SimpleFrontend({ taskData, isOnboarding, onSubmit, onError }) {
Expand Down Expand Up @@ -330,14 +330,86 @@ output_string = f"Provided rating: {outputs['rating']}\n{edit_text_string}"

While creating review scripts is powerful, it's not always the easiest way to review data. Annotations that are best in full context, like videos for instance, would likely benefit from being able to view the data directly.

### 4.2 Creating a viewer
### 4.2 Using a web-based review flow

For tasks that are best reviewed through a full UI, Mephisto offers a way to create web-based review flows.

TODO - Provide an explanation for how to use the web/UI based viewer for this specific task.
To view the results of the task we just ran through the web-based workflow, run the `mephisto review` CLI tool:

# Building from task blocks with @Annotated
```
mephisto review --db custom-react-tutorial-iterating --stdout --all
```

This will launch a local server where you will be able to browse, filter, and drill into the data collected for your task.

![](/custom_react_review_all.png)

You can also drill into a specific task to explore the details further.

![](/custom_react_review_single.png)

### 4.3 Customizing the web-based review flow

By default, we ship with a batteries-included review experience, however you can easily create your own interface with custom "renderers" for visualizing your collected data.

Let's walk through creating our own custom renderer for our data.

First, we'll create our review application by using the `create-react-app` npm package, with a mephisto review template. Continuing from within our project folder at `tmp/static_tutorial/` let's run:

```bash
npx create-react-app@latest custom-review --template mephisto-review
```
Once the template is done installing we'll create a new file at `custom-review/src/custom/MyDataItem.js` and fill it in as such:

```jsx
import React from "react";
import { Card } from "@blueprintjs/core";

export default function MyDataItem({item}) {
const rating = item.data.data.outputs.final_data.rating;
const duration = Math.round(item.data.data.times.task_end - item.data.data.times.task_start);
return <Card>
<h1>{ rating === "good" ? "👍" : "👎"}</h1>
<p>{duration} seconds</p>
</Card>
}

```

Note that we're leveraging the `<Card />` component from the rich suite of components [provided by BlueprintJS](https://blueprintjs.com/docs/#core/components/card) to create our renderer. By default, the review template imports the `@blueprintjs/core` library so you can immediately start using any of the UI components provided there.

Now that we've created our own ItemRenderer, we'll use it by updating `custom-review/src/index.js` to first import the renderer:

```jsx
import MyDataItem from "./custom/MyDataItem"
```

Then we'll pass it to the `<CollectionView />` component to modify the way in which our data renders in the grid view:

```jsx
<CollectionView
collectionRenderer={GridCollection}
itemRenderer={MyDataItem}
pagination={true}
resultsPerPage={9}
/>
```

To see our new renderer in action, let's build our app and invoke the mephisto review CLI with it.

```bash
cd custom-review/
npm run build
mephisto review build/ --db custom-react-tutorial-iterating --stdout --all
```

*Note:* Notice that the `mephisto review` command here is similar to the one run in the previous section, except this time we pass in the relative path to the build folder as an argument.

![](/custom_react_review_renderer.png)


# Building from task blocks with `@annotated` [BETA]

Mephisto at the core is built for complete flexibility over the kinds of tasks you can create. Building everything from scratch though can be a lot, so we've created the annotation toolkit and Annotated libraries. These have components that may be useful for your annotation tasks, as well as examples of developed flows you can use or extend for your tasks.
Mephisto at the core is built for complete flexibility over the kinds of tasks you can create. Building everything from scratch though can be a lot, so we've created the annotation toolkit and `@annotated` suite of libraries. These have components that may be useful for your annotation tasks, as well as examples of developed flows you can use or extend for your tasks.

TODO - add links to the storybook and component documentation.
The full suite of tools is currently in beta and can be found in the [Storybook here](https://annotation-toolkit-storybook.vercel.app/).
Binary file added docs/web/static/custom_react_review_all.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/web/static/custom_react_review_renderer.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/web/static/custom_react_review_single.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 6 additions & 1 deletion mephisto/client/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
# LICENSE file in the root directory of this source tree.

import click # type: ignore
import os
from click_default_group import DefaultGroup # type: ignore

from omegaconf import MISSING
Expand Down Expand Up @@ -57,7 +58,11 @@ def config(identifier, value):


@cli.command("review")
@click.argument("review_app_dir", type=click.Path(exists=True))
@click.argument(
"review_app_dir",
type=click.Path(exists=True),
default=os.path.join(os.path.dirname(__file__), "review/default-ui"),
)
@click.option("-p", "--port", type=(int), default=5000)
@click.option("-o", "--output", type=(str), default="")
@click.option("-a", "--assets", "assets_dir", type=(str), default=None)
Expand Down
16 changes: 16 additions & 0 deletions mephisto/client/review/default-ui/asset-manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"files": {
"main.css": "/static/css/main.41f44bb3.css",
"main.js": "/static/js/main.7cc18a59.js",
"static/media/icons-20.eot": "/static/media/icons-20.cde033c5d3f24283f757.eot",
"static/media/icons-20.woff": "/static/media/icons-20.1ef633d3a28d0986f63e.woff",
"static/media/icons-20.ttf": "/static/media/icons-20.57b3e708b232fdcb64f9.ttf",
"static/media/icons-16.eot": "/static/media/icons-16.2368f88a078780d80145.eot",
"static/media/icons-16.woff": "/static/media/icons-16.1645f50fb7f7c109f64e.woff",
"static/media/icons-16.ttf": "/static/media/icons-16.13933033991f62d6bb64.ttf",
"index.html": "/index.html",
"main.41f44bb3.css.map": "/static/css/main.41f44bb3.css.map",
"main.7cc18a59.js.map": "/static/js/main.7cc18a59.js.map"
},
"entrypoints": ["static/css/main.41f44bb3.css", "static/js/main.7cc18a59.js"]
}
Binary file added mephisto/client/review/default-ui/favicon.ico
Binary file not shown.
21 changes: 21 additions & 0 deletions mephisto/client/review/default-ui/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="/favicon.ico" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="Web site created using create-react-app"
/>
<link rel="apple-touch-icon" href="/logo192.png" />
<title>Mephisto Review</title>
<script defer="defer" src="/static/js/main.7cc18a59.js"></script>
<link href="/static/css/main.41f44bb3.css" rel="stylesheet" />
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>
Binary file added mephisto/client/review/default-ui/logo192.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added mephisto/client/review/default-ui/logo512.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions mephisto/client/review/default-ui/robots.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# https://www.robotstxt.org/robotstxt.html
User-agent: *
Disallow:
Loading