Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iterate UI for AI job modal #7368

Merged
merged 33 commits into from
Oct 23, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
5640d14
WIP: renew ai job modal
knollengewaechs Sep 29, 2023
cd6d3d9
trying out radio buttons but probably discarding
knollengewaechs Sep 30, 2023
eeaf452
style radio button using card
knollengewaechs Sep 30, 2023
282c56f
update store with new action
knollengewaechs Oct 2, 2023
9e5faa2
Merge branch 'master' into ai-job-modal
knollengewaechs Oct 2, 2023
11b9029
not pretty but working: first version of full modal
knollengewaechs Oct 2, 2023
39a822a
improve code structure
knollengewaechs Oct 5, 2023
9444b4d
use type for ai job modal state
knollengewaechs Oct 6, 2023
157c19b
WIP: reintroduce description
knollengewaechs Oct 6, 2023
ad6e369
style disclaimer
knollengewaechs Oct 6, 2023
3b0b9ef
WIP: improve spacing
knollengewaechs Oct 7, 2023
6fd003d
remove dev remnants and style modal some more
knollengewaechs Oct 9, 2023
29ad42e
refactor code
knollengewaechs Oct 9, 2023
7d3abc5
improve styling and design
knollengewaechs Oct 9, 2023
eae3b0d
trying to fix materialize volume modal
knollengewaechs Oct 9, 2023
93f6407
Merge branch 'master' into ai-job-modal
knollengewaechs Oct 9, 2023
1ca1565
fix materialize volume annotation
knollengewaechs Oct 9, 2023
254d395
add changelog and rename
knollengewaechs Oct 9, 2023
19369ef
improve start modal state, rename classes
knollengewaechs Oct 10, 2023
313060a
Fix triggering of volume annotation materialization.
daniel-wer Oct 10, 2023
268e3de
address code review
knollengewaechs Oct 16, 2023
ca8295c
improve select title
knollengewaechs Oct 16, 2023
7a94d35
diable AI button without color layer and propose whole-layer-bb
knollengewaechs Oct 17, 2023
013f50f
merge master
knollengewaechs Oct 17, 2023
6d70123
format code and remove comment
knollengewaechs Oct 18, 2023
2f92e49
downsample images
knollengewaechs Oct 18, 2023
be1d657
chnage docs
knollengewaechs Oct 18, 2023
19e12fd
improve wording in docs and always propose whole layer bounding box
knollengewaechs Oct 19, 2023
83fc117
Merge branch 'master' into ai-job-modal
knollengewaechs Oct 19, 2023
985fec0
merge master
knollengewaechs Oct 23, 2023
b886458
lint
knollengewaechs Oct 23, 2023
334d9ba
remove wrapper
knollengewaechs Oct 23, 2023
4725215
Merge branch 'master' into ai-job-modal
knollengewaechs Oct 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released

### Changed
- Updated backend code to Scala 2.13, with upgraded Dependencies for optimized performance. [#7327](https://github.com/scalableminds/webknossos/pull/7327)
- Remote datasets with a datasource-properties.json can now also be imported without the need for OME metadata. [#7372](https://github.com/scalableminds/webknossos/pull/7372)
- Occurrences of isosurface were renamed to ad-hoc mesh. This also applies to configuration files. [#7350](https://github.com/scalableminds/webknossos/pull/7350)
- Improved user interface to start automatic AI analysis. [#7368](https://github.com/scalableminds/webknossos/pull/7368)

### Fixed
Expand Down
117 changes: 67 additions & 50 deletions frontend/javascripts/oxalis/view/action-bar/starting_job_modals.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
import React from "react";
import type { APIJob, APIDataLayer } from "types/api_flow_types";
import { Modal, Select, Button, Form, Input, Slider, Row, Space, Radio, Card, Tooltip } from "antd";
import {
Modal,
Select,
Button,
Form,
Input,
Slider,
Row,
Space,
Radio,
Card,
Tooltip,
Alert,
} from "antd";
import {
startNucleiInferralJob,
startMaterializingVolumeAnnotationJob,
Expand All @@ -20,16 +33,21 @@ import {
import { getUserBoundingBoxesFromState } from "oxalis/model/accessors/tracing_accessor";
import Toast from "libs/toast";
import type { OxalisState, UserBoundingBox, HybridTracing } from "oxalis/store";
import { Unicode, type Vector3 } from "oxalis/constants";
import { ControlModeEnum, Unicode, type Vector3 } from "oxalis/constants";
import { Model, Store } from "oxalis/singletons";
import { clamp, computeArrayFromBoundingBox, rgbToHex } from "libs/utils";
import {
clamp,
computeArrayFromBoundingBox,
computeBoundingBoxFromBoundingBoxObject,
rgbToHex,
} from "libs/utils";
import { getBaseSegmentationName } from "oxalis/view/right-border-tabs/segments_tab/segments_view_helper";
import { V3 } from "libs/mjs";
import { ResolutionInfo } from "oxalis/model/helpers/resolution_info";
import { isBoundingBoxExportable } from "./download_modal_view";
import features from "features";
import { setAIJobModalStateAction } from "oxalis/model/actions/ui_actions";
import { ExclamationCircleOutlined, InfoCircleOutlined } from "@ant-design/icons";
import { InfoCircleOutlined } from "@ant-design/icons";

const { ThinSpace } = Unicode;

Expand Down Expand Up @@ -149,7 +167,7 @@ function LayerSelectionFormItem({
const layerType = chooseSegmentationLayer ? "segmentation" : "color";
return (
<Form.Item
label={layerType}
label="Layer"
name="layerName"
rules={[
{
Expand Down Expand Up @@ -241,9 +259,14 @@ function BoundingBoxSelectionFormItem({
value: selectedBoundingBoxId,
}: BoundingBoxSelectionProps): JSX.Element {
const dataset = useSelector((state: OxalisState) => state.dataset);
const isInDatasetViewMode = useSelector(
(state: OxalisState) => state.temporaryConfiguration.controlMode === ControlModeEnum.VIEW,
);
const colorLayer = getColorLayers(dataset)[0];
const mag1 = colorLayer.resolutions[0];

const howToCreateBoundingBoxText = isInDatasetViewMode
? "To process only a part of the dataset, please create an annotation and create a bounding box in it using the bounding box tool in the toolbar at the top."
: "You can create a new bounding box for the desired volume with the bounding box tool in the toolbar at the top. The created bounding boxes will be listed below.";
return (
<div style={isBoundingBoxConfigurable ? {} : { display: "none" }}>
<Form.Item
Expand All @@ -252,10 +275,7 @@ function BoundingBoxSelectionFormItem({
<Space>
Bounding Box
<Tooltip
title="Please select the bounding box for which the segmentation should be computed. Note that
large bounding boxes can take very long. You can create a new bounding box for the desired
volume with the bounding box tool in the toolbar at the top. The created bounding boxes will
be listed below."
title={`Please select the bounding box which should be processed. Note that large bounding boxes can take very long. ${howToCreateBoundingBoxText}`}
>
<InfoCircleOutlined />
</Tooltip>
Expand All @@ -269,23 +289,26 @@ function BoundingBoxSelectionFormItem({
message: "Please select the bounding box for which the inferral should be computed.",
},
{
required: true,
validator: (_rule, value) => {
if (!isBoundingBoxConfigurable) return Promise.resolve();

const selectedBoundingBox = userBoundingBoxes.find((bbox) => bbox.id === value);
let rejectionReason = "";
if (selectedBoundingBox) {
const { isExportable, alerts: _ } = isBoundingBoxExportable(
selectedBoundingBox.boundingBox,
mag1,
);
if (isExportable) return Promise.resolve();
rejectionReason = `The volume of the selected bounding box is too large. The AI neuron segmentation trial is only supported for up to ${
features().exportTiffMaxVolumeMVx
} Megavoxels. Additionally, no bounding box edge should be longer than ${
features().exportTiffMaxEdgeLengthVx
}vx.`;
}
return Promise.reject();
// In case no bounding box was selected, the rejectionReason will be "", because the previous rule already checks that.
return Promise.reject(rejectionReason);
},
message: `The volume of the selected bounding box is too large. The AI neuron segmentation trial is only supported for up to ${
features().exportTiffMaxVolumeMVx
} Megavoxels. Additionally, no bounding box edge should be longer than ${
features().exportTiffMaxEdgeLengthVx
}vx.`,
},
]}
hidden={!isBoundingBoxConfigurable}
Expand Down Expand Up @@ -382,7 +405,7 @@ export function StartAIJobModal({ aIJobModalState }: StartAIJobModalProps) {
title={
<>
<i className="fas fa-magic" />
Automated analysis with WEBKNOSSOS
AI Analysis
</>
}
onCancel={() => Store.dispatch(setAIJobModalStateAction("invisible"))}
Expand All @@ -398,7 +421,7 @@ export function StartAIJobModal({ aIJobModalState }: StartAIJobModalProps) {
>
<Card bordered={false}>
<Space direction="vertical" size="small">
<Row>Neuron segmentation</Row>
<Row className="ai-job-title">Neuron segmentation</Row>
<Row>
<img
src={`/assets/images/${jobNameToImagePath.neuron_inferral}`}
Expand All @@ -418,11 +441,11 @@ export function StartAIJobModal({ aIJobModalState }: StartAIJobModalProps) {
>
<Card bordered={false}>
<Space direction="vertical" size="small">
<Row>Nuclei detection</Row>
<Row className="ai-job-title">Nuclei detection</Row>
<Row>
<img
src={`/assets/images/${jobNameToImagePath.nuclei_inferral}`}
alt={"Nuclei inferral example"}
alt={"Nuclei detection example"}
style={centerImageStyle}
/>
</Row>
Expand All @@ -439,7 +462,7 @@ export function StartAIJobModal({ aIJobModalState }: StartAIJobModalProps) {
>
<Card bordered={false}>
<Space direction="vertical" size="small">
<Row>Mitochondria detection</Row>
<Row className="ai-job-title">Mitochondria detection</Row>
<Row>
<img
src={`/assets/images/${jobNameToImagePath.mitochondria_inferral}`}
Expand All @@ -453,7 +476,7 @@ export function StartAIJobModal({ aIJobModalState }: StartAIJobModalProps) {
</Tooltip>
</Space>
{aIJobModalState === "neuron_inferral" ? <NeuronSegmentationForm /> : null}
{aIJobModalState === "nuclei_inferral" ? <NucleiSegmentationForm /> : null}
{aIJobModalState === "nuclei_inferral" ? <NucleiDetectionForm /> : null}
</Space>
</Modal>
) : null;
Expand All @@ -464,14 +487,25 @@ function StartJobForm(props: StartJobFormProps) {
const chooseSegmentationLayer = props.chooseSegmentationLayer || false;
const { handleClose, jobName, jobApiCall, fixedSelectedLayer, title, description } = props;
const [form] = Form.useForm();
const userBoundingBoxes = useSelector((state: OxalisState) =>
const rawUserBoundingBoxes = useSelector((state: OxalisState) =>
getUserBoundingBoxesFromState(state),
);
const dataset = useSelector((state: OxalisState) => state.dataset);
const tracing = useSelector((state: OxalisState) => state.tracing);
const activeUser = useSelector((state: OxalisState) => state.activeUser);
const layers = chooseSegmentationLayer ? getSegmentationLayers(dataset) : getColorLayers(dataset);
const allLayers = getDataLayers(dataset);
const defaultBBForLayers: UserBoundingBox[] = layers.map((layer, index) => {
return {
id: -1 * index,
name: `Full ${layer.name} layer`,
boundingBox: computeBoundingBoxFromBoundingBoxObject(layer.boundingBox),
color: [255, 255, 255],
isVisible: true,
};
});
const userBoundingBoxes =
rawUserBoundingBoxes?.length === 0 ? defaultBBForLayers : rawUserBoundingBoxes;

const startJob = async ({
layerName,
Expand Down Expand Up @@ -595,12 +629,12 @@ function StartJobForm(props: StartJobFormProps) {
);
}

export function NucleiSegmentationForm() {
export function NucleiDetectionForm() {
const dataset = useSelector((state: OxalisState) => state.dataset);
return (
<StartJobForm
handleClose={() => Store.dispatch(setAIJobModalStateAction("invisible"))}
buttonLabel="Start AI neuron segmentation"
buttonLabel="Start AI nuclei detection"
jobName={"nuclei_inferral"}
title="AI Nuclei Segmentation"
suggestedDatasetSuffix="with_nuclei"
Expand Down Expand Up @@ -664,29 +698,12 @@ export function NeuronSegmentationForm() {
will create a copy of this dataset containing the new neuron segmentation.
</Row>
<Row style={{ display: "grid", marginBottom: 16 }}>
<div
style={{
gridColumnStart: 1,
gridColumnEnd: 2,
gridRowStart: 1,
gridRowEnd: 3,
margin: "auto",
}}
>
<ExclamationCircleOutlined style={{ color: "#ffda33", fontSize: 22 }} />
</div>
<div
style={{
gridColumnStart: 2,
gridColumnEnd: 3,
gridRowStart: 1,
gridRowEnd: 3,
marginLeft: 4,
}}
>
Please note that this feature is experimental and currently only works with electron
microscopy data.
</div>
<Alert
message="Please note that this feature is experimental and currently only works with electron
microscopy data."
type="warning"
showIcon
/>
</Row>
</Space>
</>
Expand Down Expand Up @@ -760,7 +777,7 @@ export function MaterializeVolumeAnnotationModal({
open
width={700}
footer={null}
title="Start Materializing this Volume Annotation"
title="Volume Annotation Materialization"
>
<StartJobForm
handleClose={handleClose}
Expand Down
19 changes: 12 additions & 7 deletions frontend/javascripts/oxalis/view/action_bar_view.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { Alert, Popover, Tooltip } from "antd";
import { Alert, Popover } from "antd";
import { connect, useDispatch, useSelector } from "react-redux";
import * as React from "react";
import type { APIDataset, APIUser } from "types/api_flow_types";
Expand Down Expand Up @@ -30,6 +30,7 @@ import {
getVisibleSegmentationLayer,
getMappingInfoForSupportedLayer,
getUnifiedAdditionalCoordinates,
getColorLayers,
} from "oxalis/model/accessors/dataset_accessor";
import { AsyncButton } from "components/async_clickables";
import { setAdditionalCoordinatesAction } from "oxalis/model/actions/flycam_actions";
Expand Down Expand Up @@ -198,16 +199,17 @@ class ActionBarView extends React.PureComponent<Props, State> {
location.href = `${location.origin}/annotations/${annotation.typ}/${annotation.id}${location.hash}`;
};

renderStartAIJobButton(): React.ReactNode {
renderStartAIJobButton(disabled: boolean): React.ReactNode {
const tooltipText = disabled ? "The dataset needs to have a color layer to start AI processing jobs." : "Start a processing job using AI" // TODO maybe hint how to create one?
knollengewaechs marked this conversation as resolved.
Show resolved Hide resolved
return (
<ButtonComponent
key="ai-job-button"
onClick={() => Store.dispatch(setAIJobModalStateAction("neuron_inferral"))}
style={{ marginLeft: 12 }}
style={{ marginLeft: 12, pointerEvents: "auto" }}
disabled={disabled}
title={tooltipText}
>
<Tooltip title="Start a processing job using AI">
<i className="fas fa-magic" /> AI Analysis
</Tooltip>
</ButtonComponent>
);
}
Expand All @@ -233,6 +235,7 @@ class ActionBarView extends React.PureComponent<Props, State> {

render() {
const {
dataset,
is2d,
isReadOnly,
showVersionRestore,
Expand Down Expand Up @@ -264,6 +267,8 @@ class ActionBarView extends React.PureComponent<Props, State> {
onDeleteLayout: this.handleLayoutDeleted,
});

const datasetHasColorLayer = getColorLayers(dataset).length > 0;

return (
<React.Fragment>
<div className="action-bar">
Expand All @@ -276,7 +281,7 @@ class ActionBarView extends React.PureComponent<Props, State> {
<DatasetPositionView />
<AdditionalCoordinatesInputView />
{isArbitrarySupported && !is2d ? <ViewModesView /> : null}
{isAIAnalysisEnabled() ? this.renderStartAIJobButton() : null}
{isAIAnalysisEnabled() ? this.renderStartAIJobButton(!datasetHasColorLayer) : null}
{!isReadOnly && constants.MODES_PLANE.indexOf(viewMode) > -1 ? <ToolbarView /> : null}
{isViewMode ? this.renderStartTracingButton() : null}
</div>
Expand All @@ -289,7 +294,7 @@ class ActionBarView extends React.PureComponent<Props, State> {
})
}
/>
<StartAIJobModal aIJobModalState={this.props.aiJobModalState} />
<StartAIJobModal aIJobModalState={this.props.aiJobModalState}/>
</React.Fragment>
);
}
Expand Down
5 changes: 5 additions & 0 deletions frontend/stylesheets/main.less
Original file line number Diff line number Diff line change
Expand Up @@ -663,3 +663,8 @@ i.without-icon-margin,
}
}
}

.ai-job-title {
text-align: center;
display: inherit;
}