[Discover-Next] Meta Issue for Bugs, Updates, Fast-Follows

[kavilla]

Issues list

• [Discover-Next] support dataframes, not just a single dataframe #7032
• [Discover-Next][Docs] Query editor and toggle #7036
• [Discover-Next][UI] Query editor darkmode highlight bad contrast #7037
• [Discover-Next][UI] Query editor can be collapsed #7038
• [Discover-Next][UX] Query editor keyboard actions #7039
• [Discover-Next][UI] Query editor run button #7040
• [Discover-Next][UI] Total hits left of the date histogram #7041
• [Discover-Next][UI] Date histogram can be collapsed #7042
• [Discover-Next][UI] data source selector hidden based on languages #7046
• Add more tests
• stats display in table
  • is this log patterns?

Query Assist

• Clean up reference of queryEditorHeaderRef [COMMENT][COMMENT]
• Pass dataSource to TopNavMenu once dataSource work is completed [COMMENT]

Async queries (not in review into main yet)

• Loading state, ensuring the loading is giving a proper indication for long standing queries

Pending items post second bug bash

• [Discover-Next][UI] Validate total hits value #7043
• Async queries, when navigating the request, the request is still polling
  • Should we cancel or should we let it go and not fire again on landing
• Better error handling
• Extend OSD field type mapper function registration points
• Remove temp logic for [Discover-Next] Adding datasources support for dataframes #7106 if not needed
• Highlight fields if PPL where filter
• Time field width gets rendered before data populates causing time column to get cropped and displayed poorly
  • Validate this is still happening after all bug fixs
  • If it is still happening ensure we don't render until we got the data
• UI shifts when hovering over quick filter buttons
  • Validate this is still happening after all bug fixs
  • This could be fixed with the timefield render issue
  • If not this could be setting a min width in the sidebar to prevent pushing from canvas
• when toggling on and then off, we force the query to be '' and language to kuery for DQL
  • SOURCE
  • This one done to make sure users get back to a default state if the previous query was not natively supported like PPL or SQL.
  • We should get UX involved to ensure that's ok or at least maybe we can pull from the Advanced Setting search:queryLanguage
• clean up styling
  • clean up styles location of query editor and query bar [COMMENT]
    • for 2.15 to reduce the number of changes (and potential impact) added the styles for the editor in the same location as the query bar. We should clean this up to follow standards.
  • better naming related to function [COMMENT]
  • We dont support IE, clean up wording to be more accurate [COMMENT]
  • Verify if we support Edge still [COMMENT]
  • Create issue for stylelint for classNames [COMMENT]
• move redundant components from the query bar and query editor to the a shared location [EXAMPLE]
• code clean up within query editor and query bar
  • if more params added to settings in UI service then use object [COMMENT]
  • avoid single statement if block with no braces [COMMENT]
  • use item || undefined instead of a ternary operator that checks the item first [COMMENT]
  • do we need query editor rect anymore once we add suggestions? if so should we keep it undefined?[COMMENT]
  • make issue with TODO for entire repo, follow up with specific issues within the query editor [COMMENT]
• re-add fetching index patterns for suggestions [COMMENT]
  • clean up ported over fetch index patterns file [COMMENT]
  • better fallback for lazy components [COMMENT]
• follow up on TODO: MQL inline comments and convert to issue [COMMENT]
• improve readability [COMMENT]

Archived: 2.15 experimental release (post first bug bash)

Blockers for 2.15 experimental release

If to target in 2.15 experiment the following list is required and fixing major bugs discovered during a bug bash. Here's what we will need (NOTE: subject to change).

Our current criteria is default state (disabled by config value and/or toggled off in advanced settings) is completely intact within Discover, with Data Explorer, with in Observability, with in dashboards and visualizations with OpenSearch having Flint & ML, only Flint, or only ML.

If the config is enabled and toggled on in advanced settings any major issues below should be fixed:

feature flag: @kavilla

• get state machine of different experiences for testing purposes
• PR to get the enhancements out of repo
• Advanced Setting toggle completely reverts back to the query bar (old language selector with option for dql, lucene) not just the input
  • enabled should be exactly the old way
  • verify no lingering issues from when it was on
• restore the redirect for default way [link]
• don't show new query bar outside discover

datasource: @paulstn

• datasource type indicating if flint or not (ideally it should be more generic like), or maybe we just utilize something like type on the data source like (polling or not default)
  • this is blocker for @joshuali925 if the domain supports query assist or not
    • the implication here is that we want in context query assistant suggestion related to the datasource, so we need to check if the data source has ml commons or not.
      • we could check if an existing API is on the data source when switching the source
      • consider fast follows and base query assist on your default (local cluster)
      • make it an advanced setting (that we can override when hosting) that lets people enable query assist or not
  • this could be a blocker for sean's work as he will need to key off this value to do async or not
    • an option could be to just allow any language for any data source and we just show the appropriate error messages
• this can be moved to fast follows based on previous point about appropriate error messages, but data sources should poplate the language switcher with languages available for that data source. it should probably be related to the data source and data source connection work but it could be added here: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/feature/discover-next/src/plugins/data/public/ui/types.ts#L45 as unsupportedDataSourceConnections i'm fine with the initial implmenetation using the hardcode data source types but something we need to track when we remove those hard coded values for registration points

sql (async): @sean li

• loading spinner when querying
  • need to verify if it's ok just to showing loading for now and fast follow as different loading status's like "loading", "not-loading", "processing"
• SQL (async) shouldn't be an option, use datasource information to pass to SQL search strategy to do async work or not
• Don't create temporary index pattern for dataframe if the data source is a specific index, [HERE IS THE CODE IT'S HAPPENING] will solve this issue:
  
• ensure search service public and server folder are aligned in implementation since we only modified public folder, related to the dataframeScache and stuff

Archived: Query Assist: @joshuali925 (Use comment below)

query assist: @joshuali925

• verification and validation next steps with query assist AND flint enabled to avoid issues in that state
• any bugs within this workflow or endpoint
• based on paul's work, another suggestion could be another toggle in advanced settings to let the admin decide if to disable or enable in general as a workaround until GA
• user preference for toggling time filter?

not blocker, but fast follow

sql (async): @sean li:

• clean up dataframe and dataframes cache, we likely are doing redudant work. consolidate
• ppl (async) support

If to target prod-ready, the following list is required:

feature flag: @kavilla

• query enhancements plugin and repo

[sejli]

Async query timeouts

In the case that a polling job gets stuck/starts to hang for too long, there should be a mechanism in place to abort it. Advanced settings can have some configurable timeout that will stop the ongoing query. To do that, two things need to happen:

1. The polling itself must stop. Upon timeout, the polling logic should return true and quit.
2. DELETE /_plugins/_async_query/{queryId} should be run to cancel the job in the backend if it's still ongoing.

[joshuali925]

query assist:

• verification and validation next steps with query assist AND flint enabled to avoid issues in that state
  • opened PR #6972
• any bugs within this workflow or endpoint
  • needs more testing
• based on paul's work, another suggestion could be another toggle in advanced settings to let the admin decide if to disable or enable in general as a workaround until GA
  • this is not needed since item 1 is doable
• user preference for toggling time filter?
  • this is not needed per UX, because current discover already mentions about expanding time range if query didn't match results. it's a generic UX problem
• add curly lint rules [Discover-next] Add query editor extensions #7034 (comment) (needs separate issue)