Vis-Builder Integration with Data Explorer Proposal

[ananzh]

Overview

The Data Explorer project has taken a step forward in streamlining the data exploration experience in OpenSearch Dashboards (OSD) by addressing security concerns and offering a unified platform. As a strategic move, it's essential to integrate the vis-builder plugin with Data Explorer to provide a comprehensive data analysis framework for users.

Vis-builder has been instrumental in visual data exploration, offering capabilities like chart creation and advanced visualization settings. Its incorporation into Data Explorer would enhance the platform's overall data analytics and visualization potential.

There are two stages of the work:

Stage 1: Incorporate the vis-builder plugin as a view inside the Data Explorer.

Goals

• Ensure a seamless user experience with only minor UI adjustments.

Note: This proposal focuses primarily on Stage 1.

Stage 2: Enhancements (currently under design & discussion)

Goals

• Navigation Improvements between various views within Data Explorer.

Note: Not the scope of current proposal.

Objective

To seamlessly incorporate the vis-builder plugin as a view inside the Data Explorer to offer a unified data exploration experience, thereby enriching the data analysis capabilities of OSD.

Tasks

[Task 1] Reconstruct and Reroute VisBuilder Plugin

• Remove URL tracking
  • Should removed the URL tracking mechanism that utilized the createOsdUrlTracker function, which involved setting up a base URL, managing the app's state with URL changes, and stopping URL tracking. All of these are handled in Data Explorer.
• Simplify app mounting
  • Don’t need to load the application bundle, nor does it set up history, store, or other application configurations. Instead, it should directly navigate to DataExplorer with a specific path.
• Update dependencies
  • Add a new dependency dataExplorer , which means the VisBuilderPlugin is now directly interfacing with the DataExplorerPlugin.
• Register as a view in Data Explorer
  • Add a new registration mechanism, dataExplorer.registerView, that allows it to register itself as a view in the data-explorer application. This provides a direct linkage between the two plugins, allowing the VisBuilderPlugin to benefit from the services and utilities provided by the parent DataExplorerPlugin.
• App Registration Changes
  • The main app registration logic remains largely the same, with the major difference being the removal of the URL tracking logic and the new navigation mechanism to the data-explorer.
• Embeddable and Visualization Registration
  • These two registration should remain unchanged.

[Task 2] State Management Migration

There are issues to migrate states. First, simplify metadata slice which should only ffocus on the editor's state.

• Both slices have originatingApp which seems to be used for the same purpose.
  • In order to avoid redundancy: Use the originatingApp from the DataExplorer's slice since it's the main application, and remove it from the VisBuilder's slice.
• Keep one scopedHistory shared between both VisBuilder and DataExplorer. This scopedHistory should be associated with the DataExplorer, and VisBuilder can access it as needed.

Second migrate simplified metadata slice and other 3 slices, there are two proposals.

Proposal 1: Allowing Data Explorer to Register Multiple Slices for a Single View

Objective

Retain the modularity of individual slices for each component of the Vis Builder while simplifying integration with the Data Explorer.

Steps

• Modify the registerSlice Method: Allow the registerSlice function to handle an array of slices.

export const registerSlices = (slices: Slice[]) => {
  slices.forEach(slice => {
    if (dynamicReducers[slice.name]) {
      throw new Error(`Slice ${slice.name} already registered`);
    }
    dynamicReducers[slice.name] = slice.reducer;
  });
};

• Update the View Registration: Instead of registering a single slice, views will now register an array of slices.

views.forEach((view) => {
  if (!view.ui) return;

  const { slices } = view.ui;
  registerSlices(slices);
});

• Modify the ViewDefinition Interface: Replace the singular slice property with a plural slices property.

export interface ViewDefinition<T = any> {
  // ... other properties ...

  readonly ui?: {
    defaults: DefaultViewState | (() => DefaultViewState) | (() => Promise<DefaultViewState>);
    slices: Slice<T>[];  // This is now an array
  };

  // ... other properties ...
}

• Ensure Uniqueness of Slice Names and Action Names: As the slices grow, there's a need to ensure that slice names and action names remain unique to avoid conflicts. This can be enforced through naming conventions and linting tools. For instance, prefixing actions and slices with their respective feature or module name, like visBuilderMetadata which could make sure differentiation from current metadata slice.
• Dispatch actions to specific slices: With potentially multiple slices having similar action names, ensure that actions are dispatched to the intended slices. This can be achieved using the unique naming conventions proposed above. So I don’t expect much changes in current visBuilder.
• Understand and check extraReducers: The extraReducers in each individual slice (like styleSlice and visualizationSlice) listen for external actions and handle them by modifying their respective slice of the state. The action setActiveVisualization is defined in the vis builder and is only specific to the vis builder. Both slices using the action (setActiveVisualization) seem to handle it in a way that they only update their own state. Thus, dispatching the action won't lead to unintended state updates elsewhere in Data Explorer, as long as the action payload structure remains consistent.
• RenderState is created from combined state, not combined reducers. Since each reducer is created by createSlice with initial state, I expect no differences in usage.

export interface VisBuilderState {
  vbEditor: EditorState;
  vbStyle: StyleState;
  vbUi: UIStateState;
  vbVisualization: VisualizationState;
}
export type RenderState = Omit<VisBuilderState, 'vbEditor'>;

• Handle side effects: In the Data Explorer store, we are dynamically adding slices (reducers) for each view. In the visBuilder, there are two side effects: handlerEditorState and handlerParentAggs. These side effects are listeners that perform specific actions when certain changes in the state occur. The current Data Explorer does not have these side effects integrated, which leads us to the main challenge. However, we see there is a TODO in Data Explore store config Add Side effects here to apply after changes to the store are made. None for now., which indicates an initial design considerations on how integrate view’s side effect. Based on this, here are steps

  • Define Side Effects in ViewDefinition.ui: Extend the ui object of each view to potentially include an array of side effects. This will allow each view to specify its set of side effects.

   export interface ViewDefinition<T = any> {
         …
         
         readonly ui?: {
          defaults: DefaultViewState | (() => DefaultViewState) | (() => Promise<DefaultViewState>);
          slice: Slice<T>[];
          sideEffects?: Array<(store: Store, viewName: string, currentState: RootState, previousState?: RootState, services?: DataExplorerServices) => void>;
        };
         …
      }
  

  • Modify handleChange to execute side Effects in getPreloadedStore: handleChange should iterate through each registered view and execute its side effects if they exist.

   export const getPreloadedStore = async (services: DataExplorerServices) => {
        // For each view preload the data and register the slice
        const views = services.viewRegistry.all();
        const viewSideEffectsMap: Record<string, Function[]> = {};
      
        views.forEach((view) => {
          if (!view.ui) return;
      
          const { slices, sideEffects } = view.ui;
          registerSlices(slices);
      
          // Save side effects if they exist
          if (sideEffects) {
            viewSideEffectsMap[view.id] = sideEffects;
          }
        });
      
        const preloadedState = await loadReduxState(services);
        const store = configurePreloadedStore(preloadedState);
      
        let previousState = store.getState();
      
        // Listen to changes
        const handleChange = () => {
          const state = store.getState();
          persistReduxState(state, services);
      
          if (isEqual(state, previousState)) return;
      
          // Execute view-specific side effects.
          Object.entries(viewSideEffectsMap).forEach(([viewId, effects]) => {
            effects.forEach((effect) => {
              try {
                effect(state, previousState, services);
              } catch (error) {
                console.log(`Error executing side effect for view ${viewId}:`, error);
              }
            });
          });
      
          previousState = state;
        };
      
        // the store subscriber will automatically detect changes and call handleChange function
        const unsubscribe = store.subscribe(handleChange);
      
        // This is necessary because browser navigation updates URL state that isnt reflected in the redux state
        services.scopedHistory.listen(async (location, action) => {
          const urlState = await loadReduxState(services);
          const currentState = store.getState();
      
          // If the url state is different from the current state, then we need to update the store
          // the state should have a view property if it was loaded from the url
          if (action === 'POP' && urlState.metadata?.view && !isEqual(urlState, currentState)) {
            store.dispatch(hydrate(urlState as RootState));
          }
        });
      
        const onUnsubscribe = () => {
          dynamicReducers = {
            ...commonReducers,
          };
      
          unsubscribe();
        };
      
        return { store, unsubscribe: onUnsubscribe };
      };
  

  • Modify side effects functions based on new definition.
  • Register Side Effects for visBuilder: When defining the visBuilder UI, ensure to add the modified handlerEditorState and handlerParentAggs functions to the sideEffects array.

• Some cleaning work: Clean out redux store from VisBuilder and utilize the one from Data Explorer.

Pros and Cons

• Pros:
  • Retains modularity. Each modular/component (not view) maintains its individual state slice.
  • Easier integration with the current Data Explorer without needing to modify its architecture significantly, especially dispatch.
  • Flexibility in managing individual component states.
• Cons:
  • Potential conflicts in action naming. Requires strict naming conventions.
  • Slices must be entirely isolated. Have to make sure no shared state, no shared logic and order of action execution does not matter.

Proposal 2: Combining VisBuilder Slices and register one slice to Data Explorer

This proposal defines a combined VisBuilderState which encompasses all the individual sub-states. Data Explorer current setting is one slice/view. A view, for example Discover, needs to specify a slice like below and data explorer register a slice via registerSlice.

ui: {
        defaults: async () => {
          this.initializeServices?.();
          const services = getServices();
          return await getPreloadedState(services);
        },
        slice: discoverSlice,
   }

export const registerSlice = (slice: Slice) => {
  if (dynamicReducers[slice.name]) {
    throw new Error(`Slice ${slice.name} already registered`);
  }
  dynamicReducers[slice.name] = slice.reducer;
};    

Different from Discover, VisBuilder has multiple slices. Then for consistency and manage purpose, it might be ideal to keep the one slice for one view.

Steps

• Combine slices: We would need to manually creating one with createSlice. We can’t use combineReducers here because this method returns a reducer function, not a slice object. Therefore this will bring incompatibility with registerSlice in Data Explorer. It would be convenient if we could directly utilize registerSlice and don’t have to make modifications in the upstream Data Explorer.

const visBuilderSlice = createSlice({
  name: 'visBuilder',
  initialState: initialState,
  reducers: {
    updateStyle: (state, action: PayloadAction<StyleState>) => {
      state.style = styleReducer(state.style, action);
    },
    updateVisualization: (state, action: PayloadAction<VisualizationState>) => {
      state.visualization = visualizationReducer(state.visualization, action);
    },
    updateMetadata: (state, action: PayloadAction<MetadataState>) => {
      state.metadata = metadataReducer(state.metadata, action);
    },
    updateUI: (state, action: PayloadAction<UIStateState>) => {
      state.ui = uiStateReducer(state.ui, action);
    },
  },
});

• Combine preloaded state: create combined getPreloadedState .

export const getPreloadedState = async (
  services: VisBuilderServices
): Promise<DefaultViewState<VisBuilderState>> => {
  const styleState = await getPreloadedStyleState(services);
  const visualizationState = await getPreloadedVisualizationState(services);
  const metadataState = await getPreloadedMetadataState(services);
  const uiState = await getPreloadedUIState(services);

  const preloadedState: DefaultViewState<VisBuilderState> = {
    state: {
      ...{
        style: styleState,
        visualization: visualizationState,
        metadata: metadataState,
        ui: uiState,
      },
    },
  };

  return preloadedState;
};

• Dispatch Actions: As mentioned above, dispatch needs to get updated. First take the current action with specific slice, for example setStyleState(newStyleValue) specific to the styleSlice. Then wrap that action with the setStyleState action creator to make it specific to the visBuilderSlice. Then dispatch the resulting action.

• Understand and check extraReducers: Since we have combined the slices, it's crucial that extraReducers is moved in the combined slice (VisBuilderSlice) which should know which subsection of the state to modify.

  • Style Slice: The setActiveVisualization action updates the entire style state with the style provided in the action's payload. This means whenever the setActiveVisualization action is dispatched, the style slice updates its state to match the style specified in that action.

  extraReducers(builder) {
      builder.addCase(setActiveVisualization, (state, action) => {
         return action.payload.style;
      });
   }
  

  • Visualization Slice: The setActiveVisualization action modifies the activeVisualization property of the state. It only updates the name and aggConfigParams from the action's payload.

  extraReducers(builder) {
      builder.addCase(setActiveVisualization, (state, action) => {
         state.activeVisualization = {
           name: action.payload.name,
           aggConfigParams: action.payload.aggConfigParams,
        };
     });
  }
  

  • As long as the combined VisBuilderSlice correctly handles the setActiveVisualization action by updating both the style and visualization parts of the state, and the component dispatches the action with the expected payload structure, we shouldn't need to modify the component that calls setActiveVisualization.

  export const visBuilderSlice = createSlice({
    name: 'visBuilder',
    initialState,
    reducers: {
      updateStyle: (state, action: PayloadAction<StyleState>) => {
        state.style = styleReducer(state.style, action);
      },
      updateVisualization: (state, action: PayloadAction<any>) => {
        state.visualization = visualizationReducer(state.visualization, action);
      },
      updateMetadata: (state, action: PayloadAction<any>) => {
        state.metadata = metadataReducer(state.metadata, action);
      },
      updateUI: (state, action: PayloadAction<UIStateState>) => {
        state.ui = uiStateReducer(state.ui, action);
      },
    },
    extraReducers: (builder) => {
      builder.addCase(setActiveVisualization, (state, action) => {
        // handling the visualization slice
        state.visualization.activeVisualization = {
          name: action.payload.name,
          aggConfigParams: action.payload.aggConfigParams,
        };
  
        // handling the style slice
         state.style = action.payload.style;
      });
    },
  });
  

• Handle side effects: Similar to Proposal 1.

Pros and Cons

• Pros
  • Will create a centralized slice and don’t need any updates in Data Explorer.
  • Compared with the first proposal, there is a loser requirement on the isolate of the slices. If states can be completed isolated in shared action or side effects, then this option is more suitable.
• Cons
  • More work to implement, such as modifying dispatch, like dispatch(updateStyle(setStyleState(newStyleValue))) if we want to dispatch a style action by setStyleState. Because with this slice setting, a style action first goes to the visBuilderSlice reducer, and specifically to the updateStyle reducer: updateStyle: (state, action: PayloadAction<StyleState>) => { state.style = styleReducer(state.style, action); }

[Task 3] Context Provider Migration/Creation

• Categorize context provider to global (can be used in both Panel and Canvas) and local
• Create global context provider if needed in VisBuilder for index pattern only.
• Retain local context provider, such as DragDropProvider in its current state due to its specificity to panels/canvases.

[Task 4] UI modifications

The main part of this task is to divide src/plugins/vis_builder/public/application/app.tsx into a canvas area (VisBuilderCanvas) and a side panel (VisBuilderPanel). The canvas is where users interact with visualizations, and the side panel only provides field selector. Below is the brief re-construction:

• VisBuilderPanel should constructed from cleaned LeftNav .

  • Remove DataSourceSelect as DataExplorer will render comp to select data source.
  • FieldSelector is now VisBuilderPanel .

  export const VisBuilderPanel = () => {
    return (
      <section className="vbSidenav left">
        <FieldSelector />
      </section>
    );
  };
  

  • ConfigPanel should be put in VisBuilderCanvas.

• VisBuilderCanvas is the main canvas of VisBuilder. It should have the TopNav at the top, ConfigPanel on the left, Workspace in the middle, and RightNav on the right.

export const VisBuilderCanvas = () => {
  return (
    <I18nProvider>
      <DragDropProvider>
        <EuiPage className="vbLayout">
          <TopNav />
          <EuiResizableContainer className="vbLayout__resizeContainer">
            {(EuiResizablePanel, EuiResizableButton) => (
              <>
                <EuiResizablePanel
                  className="vbLayout__configPanelResize"
                  paddingSize="none"
                  initialSize={20}
                  minSize="250px"
                  mode="main"
                >
                  <ConfigPanel />
                </EuiResizablePanel>
                <EuiResizableButton className="vbLayout__resizeButton" />
                <EuiResizablePanel
                  className="vbLayout__workspaceResize"
                  paddingSize="none"
                  initialSize={60}
                  minSize="300px"
                  mode="main"
                >
                  <Workspace />
                </EuiResizablePanel>
                <EuiResizableButton className="vbLayout__resizeButton" />
                <EuiResizablePanel
                  className="vbLayout__rightNavResize"
                  paddingSize="none"
                  initialSize={20}
                  minSize="250px"
                  mode="main"
                >
                  <RightNav />
                </EuiResizablePanel>
              </>
            )}
          </EuiResizableContainer>
        </EuiPage>
      </DragDropProvider>
    </I18nProvider>
  );
};

[Task 5] Comprehensive test for VisBuilder

• Audit all vis-builder functional tests.
• Ensure all functionalities, especially known ones, are covered. This will validate post-migration operations.
• Develop tests for any identified missing functionalities.

Tentative Timeline

Based on 2-3 devs

Week 1-2

• [Vis-Builder] Create vis_builder_integration branch and set up the basic plugin structure. #5492
  • Allow register vis-builder as a view within Data Explorer
  • Complete re-route logic to ensure Data Explorer render out vis-builder
  • Temporarily implement an EUI spinner or no-image display for panel and Canvas during rendering.
• [Vis-Builder] Migrate state management to Data Explorer #5493
• [Vis-Builder] Research and document functional tests. Document missing functionalities. #5502
  • Review and summarize all functional tests for vis-builder
  • Ensure all functionalities are covered, especially the known ones. These tests will validate functionalities post-migration.
  • Incorporate tests for any missing functionalities.

Week 3-4

• [Vis-Builder] Migrate/Create context provider to Data Explorer #5500
  • Research on current context provider.
  • Retain the DragDropProvider in its current state as it isn’t a context provider for the entire panel/canvas.
• [Vis-Builder] UI Modifications #5501
  • Integrate TopNav, right navigation, and Workspace into the Canvas.
  • Migrate the left navigation to Panel.
  • Validate that all functionalities remain intact.

Week 5-6

• [Vis-Builder] Allow navigation between different views
• [Vis-Builder] Tests

Additional Considerations

• User Feedback: Consider establishing feedback loops to gain insights into how users perceive the new integration and any potential areas of improvement.
• Performance: Given that two major plugins are being integrated, performance benchmarks should be set to ensure that the user experience remains fluid.
• Documentation: Update user and developer documentation to reflect changes and assist in smoother adoption.
• Fallback Mechanism: Given the significance of the changes, having a fallback or rollback mechanism can be crucial in case any critical issues arise post-integration.

[ashwin-pc]

Great proposal @ananzh! Definitely more inclined to option one. Here are a few suggestions to that.

1. Prefixing the slice with the view Id should suffice for name conflicts. If we want to make using the state easier, we can add helper methods that make it easier to work with prefixed slice names.
2. In case you don't have it on your radar, you also need to ensure that legacy urls for VisBuilder still work after this migration. You can follow the pattern used during the discover migration. E.g. saved object urls, edit and create urls etc.

Thanks for the detailed description of the migration!

[ananzh]

Prefixing the slice

To address the challenge of managing states in a centralized system with different applications like VisBuilder and DataExplorer, while avoiding naming conflicts and ensuring easy access to the states, we propose the following steps:

Prefix State Keys with view.id

Prefixing state keys with the view.id to avoid naming conflicts. Then we can maintain the original state names within each view (like style, visualization, and ui in vis builder) but prepend the view's identifier when integrating them into the RootState. The only exception is metadata, because it is duplicate as the one in RootState.

State naming in Data Explorer

The register slice id in Data Explorer store (src/plugins/data_explorer/public/utils/state_management/preload.ts):

const id = slice.name == view.id? slice.name: `${view.id}-${slice.name}`;

Therefore in plugin.ts, each view needs to register a proper ID. If there are multiple slices in one view, the slice id would be ${view.id}-${slice.name}. For example, in the following vis_builder, the view.id is vis_builder

export const PLUGIN_ID = 'vis-builder';

dataExplorer.registerView<any>({
      id: PLUGIN_ID,
      title: PLUGIN_NAME,
      defaultPath: '#/',
      appExtentions: {
        savedObject: {
          docTypes: [VISBUILDER_SAVED_OBJECT],
          toListItem: (obj) => ({
            id: obj.id,
            label: obj.title,
          }),
        },
      },

…

There are four slices: ui, style, editor, visualization. The registered slice names in Data Explorer are vis-builder-ui, vis-builder-style, vis-builder-editor and vis-builder-visualization.

For view with single state, then the easiest naming is to have state name equal to PLUGIN_ID. For example, in Discover, both registered view ID and state is discover. Then in Data Explorer, the registered slice name is discover. Discover does not need further state access.

Dynamic State Access

To access these namespaced states easily, we can create a selector function that dynamically constructs the state key based on the view's ID. For example, within VisBuilder, we use a generic selector that automatically appends vis-builder- to the state keys. Here are some reference helper functions.

• access one specific state object directly

// Function to access specific state with prefix
export const getViewState = (stateKey: 'editor' | 'ui' | 'visualization' | 'style') => 
  (state: VisBuilderRootState) => state[`${PLUGIN_ID}-${stateKey}`]

To use

// Usage in VisBuilder
const styleState = useTypedSelector(getViewState('style'));

• access one specific state from the entire state object

type ValidStateKeys = 'editor' | 'ui' | 'visualization' | 'style';
type ValidStateTypes = StyleState | UiState | VisualizationState | EditorState;

const getDynamicState = <T extends ValidStateTypes>(rootState: VisBuilderRootState, stateKey: ValidStateKeys): T => {
    const dynamicKey = `${PLUGIN_ID}-${stateKey}`;
    return rootState[dynamicKey] as T;
};

To use:

const rootState = useSelector(state => state);
const editorState = getDynamicState<EditorState>(rootState, 'editor');

// TypeScript will ensure that editorState is of type EditorState
if (editorState && editorState.status === 'loaded') {
    ...
}

Dispatch Mechanism in Centralized State Management

In the centralized state management system, where different slices of state are prefixed with their respective view IDs (like vis-builder-editor for the editor slice), we actually don't need to modify the actions:

• Action Type Prefixing: Redux Toolkit automatically namespaces action types with the slice name, making them unique. For example, the setStatus action from the editor slice has a type like "editor/setState".

• Reducer Handling: When setStatus is dispatched, Redux invokes the reducer associated with "editor/setStatus". The namespacing of slices in the global state, such as vis-builder-editor, does not require any additional handling when dispatching actions.

• No Need for Explicit State Specification: There is no need to manually specify which part of the state should be affected by an action. The linkage between the action and its target slice (the editor slice will point to vis-builder-editor reference in the global root state) is managed by the setup and registration of reducers and slices.

• Simple Dispatching: Actions are dispatched using their creators as usual. Redux handles routing these actions to the appropriate slice reducer based on their types.

[ashwin-pc]

How will this work with auto complete and typescript types and suggestions? today selecting using the selector is typesafe

[ananzh]

@ashwin-pc the above is more general discussion on how to migrate/work with a centralized state management store. In vis-builder, we will add one helper function

export const getViewState = (stateKey: 'editor' | 'ui' | 'visualization' | 'style') => 
  (state: VisBuilderRootState) => state[`${PLUGIN_ID}-${stateKey}`];

To use it:

const styleState = useTypedSelector(getViewState('style'));

The stateKey is limited to specific strings ('editor', 'ui', 'visualization', 'style'), which correspond to the keys in VisBuilderRootState. This provides the following benefits:

Type Safety: Ensures that only valid keys for the VisBuilder's state are used, reducing the risk of runtime errors due to incorrect state keys.

Autocomplete: When using this selector function, developers will receive autocomplete suggestions for the stateKey parameter, which can speed up development and reduce errors.

Error Detection: If a developer tries to use an invalid key, TypeScript will flag it as an error, catching potential issues early in the development process.

The types defined in Data Explorer is more general to fit more views. But VisBuilder and other views could maintain type safety across the entire application.

[ananzh]

@ashwin-pc I updated the previous comment. For access one specific state object directly and access one specific state from the entire state object, which one you like? Currently I am using the first one. Restrict the type to view specific slices. No need to load the entire state. What do you think?

[ashwin-pc]

Oh I like it if the helper exists. I just don't want to have to remember how to get to the state value without auto complete 😂