Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] Load <LogStream> entries via async searches #86899

Merged
merged 42 commits into from
Feb 2, 2021
Merged

[Logs UI] Load <LogStream> entries via async searches #86899

Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
eb8ad64
Move log entry types out of http directory
weltenwort Dec 23, 2020
d38126c
Create basic (dysfunctional) log entries search strategy
weltenwort Dec 23, 2020
6f0df96
Add log entries query params
weltenwort Dec 23, 2020
61cd20b
Expand the log entries query
weltenwort Jan 7, 2021
a51b624
remove the center cursor correctly extract the others
weltenwort Jan 11, 2021
1ce7b3d
Fix cursor property on single log entry test
weltenwort Jan 12, 2021
6527f4c
Wire up field fetching and message reconstruction
weltenwort Jan 12, 2021
d13086a
Remove unused graphql types
weltenwort Jan 13, 2021
2802b93
Fix LogEntry imports after moving it
weltenwort Jan 13, 2021
4994d05
Include index in old log entry API response
weltenwort Jan 13, 2021
4c86d39
Fix import in api integration test
weltenwort Jan 13, 2021
981d511
Align function names
weltenwort Jan 14, 2021
668f687
Comment out unfinished tests to avoid type errors
weltenwort Jan 14, 2021
880485e
WIP
weltenwort Jan 14, 2021
162ad28
Factor out handling of search responses
weltenwort Jan 14, 2021
4d53acd
Document the response handling function
weltenwort Jan 15, 2021
fb08749
WIP
weltenwort Jan 15, 2021
24d4d2b
Remove erroneous terminate_after
weltenwort Jan 15, 2021
caacaa5
Continue to make the search strategy more complete
weltenwort Jan 15, 2021
98a07ac
Factor out data search response flattening
weltenwort Jan 15, 2021
7137a53
Temporarily load "log entries after" in parallel
weltenwort Jan 15, 2021
070f450
Factor out hook to track data search response state
weltenwort Jan 18, 2021
0d2bc5c
Replace the tracked promise with the new async search
weltenwort Jan 18, 2021
0a0ed37
Simplify useDataSearchRequest
weltenwort Jan 19, 2021
379bb69
Make data search transformations more composable
weltenwort Jan 20, 2021
5fce211
Remove old loading mechanism from useLogStream
weltenwort Jan 21, 2021
25252d5
Remove unneeded code
weltenwort Jan 26, 2021
3929eba
Fix the useDataSearch unit test
weltenwort Jan 26, 2021
631cc4a
Update the documenting storybook
weltenwort Jan 26, 2021
532daff
Fix JSON types import
weltenwort Jan 26, 2021
d5f34d9
Remove debug output
weltenwort Jan 26, 2021
46f30b9
Re-enable logEntriesSearchStrategy unit tests
weltenwort Jan 26, 2021
2342f1f
Remove unused import
weltenwort Jan 26, 2021
73d1c20
Try to always derive a top and bottom cursor
weltenwort Jan 27, 2021
812c6cb
Update the `<LogStream />` storybook
weltenwort Jan 27, 2021
acc0840
Remove more unused code
weltenwort Jan 27, 2021
9ad32f8
Merge branch 'master' into logs-ui-async-log-entries-search
weltenwort Jan 28, 2021
42c8b5e
Fix incorrect imports after merge
weltenwort Jan 28, 2021
290371d
Merge branch 'master' into logs-ui-async-log-entries-search
weltenwort Feb 1, 2021
05669db
Fix types after merge
weltenwort Feb 1, 2021
a257eab
Merge branch 'master' into logs-ui-async-log-entries-search
weltenwort Feb 2, 2021
6f2144f
Deduplicate common arguments
weltenwort Feb 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
337 changes: 0 additions & 337 deletions x-pack/plugins/infra/common/graphql/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,6 @@ export interface InfraSource {
configuration: InfraSourceConfiguration;
/** The status of the source */
status: InfraSourceStatus;
/** A consecutive span of log entries surrounding a point in time */
logEntriesAround: InfraLogEntryInterval;
/** A consecutive span of log entries within an interval */
logEntriesBetween: InfraLogEntryInterval;
/** Sequences of log entries matching sets of highlighting queries within an interval */
logEntryHighlights: InfraLogEntryInterval[];

/** A snapshot of nodes */
snapshot?: InfraSnapshotResponse | null;
Expand Down Expand Up @@ -129,80 +123,6 @@ export interface InfraIndexField {
/** Whether the field should be displayed based on event.module and a ECS allowed list */
displayable: boolean;
}
/** A consecutive sequence of log entries */
export interface InfraLogEntryInterval {
/** The key corresponding to the start of the interval covered by the entries */
start?: InfraTimeKey | null;
/** The key corresponding to the end of the interval covered by the entries */
end?: InfraTimeKey | null;
/** Whether there are more log entries available before the start */
hasMoreBefore: boolean;
/** Whether there are more log entries available after the end */
hasMoreAfter: boolean;
/** The query the log entries were filtered by */
filterQuery?: string | null;
/** The query the log entries were highlighted with */
highlightQuery?: string | null;
/** A list of the log entries */
entries: InfraLogEntry[];
}
/** A representation of the log entry's position in the event stream */
export interface InfraTimeKey {
/** The timestamp of the event that the log entry corresponds to */
time: number;
/** The tiebreaker that disambiguates events with the same timestamp */
tiebreaker: number;
}
/** A log entry */
export interface InfraLogEntry {
/** A unique representation of the log entry's position in the event stream */
key: InfraTimeKey;
/** The log entry's id */
gid: string;
/** The source id */
source: string;
/** The columns used for rendering the log entry */
columns: InfraLogEntryColumn[];
}
/** A special built-in column that contains the log entry's timestamp */
export interface InfraLogEntryTimestampColumn {
/** The id of the corresponding column configuration */
columnId: string;
/** The timestamp */
timestamp: number;
}
/** A special built-in column that contains the log entry's constructed message */
export interface InfraLogEntryMessageColumn {
/** The id of the corresponding column configuration */
columnId: string;
/** A list of the formatted log entry segments */
message: InfraLogMessageSegment[];
}
/** A segment of the log entry message that was derived from a field */
export interface InfraLogMessageFieldSegment {
/** The field the segment was derived from */
field: string;
/** The segment's message */
value: string;
/** A list of highlighted substrings of the value */
highlights: string[];
}
/** A segment of the log entry message that was derived from a string literal */
export interface InfraLogMessageConstantSegment {
/** The segment's message */
constant: string;
}
/** A column that contains the value of a field of the log entry */
export interface InfraLogEntryFieldColumn {
/** The id of the corresponding column configuration */
columnId: string;
/** The field name of the column */
field: string;
/** The value of the field in the log entry */
value: string;
/** A list of highlighted substrings of the value */
highlights: string[];
}

export interface InfraSnapshotResponse {
/** Nodes of type host, container or pod grouped by 0, 1 or 2 terms */
Expand Down Expand Up @@ -276,21 +196,6 @@ export interface DeleteSourceResult {
// InputTypes
// ====================================================

export interface InfraTimeKeyInput {
time: number;

tiebreaker: number;
}
/** A highlighting definition */
export interface InfraLogEntryHighlightInput {
/** The query to highlight by */
query: string;
/** The number of highlighted documents to include beyond the beginning of the interval */
countBefore: number;
/** The number of highlighted documents to include beyond the end of the interval */
countAfter: number;
}

export interface InfraTimerangeInput {
/** The interval string to use for last bucket. The format is '{value}{unit}'. For example '5m' would return the metrics for the last 5 minutes of the timespan. */
interval: string;
Expand Down Expand Up @@ -381,34 +286,6 @@ export interface SourceQueryArgs {
/** The id of the source */
id: string;
}
export interface LogEntriesAroundInfraSourceArgs {
/** The sort key that corresponds to the point in time */
key: InfraTimeKeyInput;
/** The maximum number of preceding to return */
countBefore?: number | null;
/** The maximum number of following to return */
countAfter?: number | null;
/** The query to filter the log entries by */
filterQuery?: string | null;
}
export interface LogEntriesBetweenInfraSourceArgs {
/** The sort key that corresponds to the start of the interval */
startKey: InfraTimeKeyInput;
/** The sort key that corresponds to the end of the interval */
endKey: InfraTimeKeyInput;
/** The query to filter the log entries by */
filterQuery?: string | null;
}
export interface LogEntryHighlightsInfraSourceArgs {
/** The sort key that corresponds to the start of the interval */
startKey: InfraTimeKeyInput;
/** The sort key that corresponds to the end of the interval */
endKey: InfraTimeKeyInput;
/** The query to filter the log entries by */
filterQuery?: string | null;
/** The highlighting to apply to the log entries */
highlights: InfraLogEntryHighlightInput[];
}
export interface SnapshotInfraSourceArgs {
timerange: InfraTimerangeInput;

Expand Down Expand Up @@ -565,15 +442,6 @@ export type InfraSourceLogColumn =
| InfraSourceMessageLogColumn
| InfraSourceFieldLogColumn;

/** A column of a log entry */
export type InfraLogEntryColumn =
| InfraLogEntryTimestampColumn
| InfraLogEntryMessageColumn
| InfraLogEntryFieldColumn;

/** A segment of the log entry message */
export type InfraLogMessageSegment = InfraLogMessageFieldSegment | InfraLogMessageConstantSegment;

// ====================================================
// END: Typescript template
// ====================================================
Expand All @@ -582,46 +450,6 @@ export type InfraLogMessageSegment = InfraLogMessageFieldSegment | InfraLogMessa
// Documents
// ====================================================

export namespace LogEntryHighlightsQuery {
export type Variables = {
sourceId?: string | null;
startKey: InfraTimeKeyInput;
endKey: InfraTimeKeyInput;
filterQuery?: string | null;
highlights: InfraLogEntryHighlightInput[];
};

export type Query = {
__typename?: 'Query';

source: Source;
};

export type Source = {
__typename?: 'InfraSource';

id: string;

logEntryHighlights: LogEntryHighlights[];
};

export type LogEntryHighlights = {
__typename?: 'InfraLogEntryInterval';

start?: Start | null;

end?: End | null;

entries: Entries[];
};

export type Start = InfraTimeKeyFields.Fragment;

export type End = InfraTimeKeyFields.Fragment;

export type Entries = InfraLogEntryHighlightFields.Fragment;
}

export namespace MetricsQuery {
export type Variables = {
sourceId: string;
Expand Down Expand Up @@ -820,50 +648,6 @@ export namespace WaffleNodesQuery {
};
}

export namespace LogEntries {
export type Variables = {
sourceId?: string | null;
timeKey: InfraTimeKeyInput;
countBefore?: number | null;
countAfter?: number | null;
filterQuery?: string | null;
};

export type Query = {
__typename?: 'Query';

source: Source;
};

export type Source = {
__typename?: 'InfraSource';

id: string;

logEntriesAround: LogEntriesAround;
};

export type LogEntriesAround = {
__typename?: 'InfraLogEntryInterval';

start?: Start | null;

end?: End | null;

hasMoreBefore: boolean;

hasMoreAfter: boolean;

entries: Entries[];
};

export type Start = InfraTimeKeyFields.Fragment;

export type End = InfraTimeKeyFields.Fragment;

export type Entries = InfraLogEntryFields.Fragment;
}

export namespace SourceConfigurationFields {
export type Fragment = {
__typename?: 'InfraSourceConfiguration';
Expand Down Expand Up @@ -994,124 +778,3 @@ export namespace InfraSourceFields {
origin: string;
};
}

export namespace InfraLogEntryFields {
export type Fragment = {
__typename?: 'InfraLogEntry';

gid: string;

key: Key;

columns: Columns[];
};

export type Key = {
__typename?: 'InfraTimeKey';

time: number;

tiebreaker: number;
};

export type Columns =
| InfraLogEntryTimestampColumnInlineFragment
| InfraLogEntryMessageColumnInlineFragment
| InfraLogEntryFieldColumnInlineFragment;

export type InfraLogEntryTimestampColumnInlineFragment = {
__typename?: 'InfraLogEntryTimestampColumn';

columnId: string;

timestamp: number;
};

export type InfraLogEntryMessageColumnInlineFragment = {
__typename?: 'InfraLogEntryMessageColumn';

columnId: string;

message: Message[];
};

export type Message =
| InfraLogMessageFieldSegmentInlineFragment
| InfraLogMessageConstantSegmentInlineFragment;

export type InfraLogMessageFieldSegmentInlineFragment = {
__typename?: 'InfraLogMessageFieldSegment';

field: string;

value: string;
};

export type InfraLogMessageConstantSegmentInlineFragment = {
__typename?: 'InfraLogMessageConstantSegment';

constant: string;
};

export type InfraLogEntryFieldColumnInlineFragment = {
__typename?: 'InfraLogEntryFieldColumn';

columnId: string;

field: string;

value: string;
};
}

export namespace InfraLogEntryHighlightFields {
export type Fragment = {
__typename?: 'InfraLogEntry';

gid: string;

key: Key;

columns: Columns[];
};

export type Key = {
__typename?: 'InfraTimeKey';

time: number;

tiebreaker: number;
};

export type Columns =
| InfraLogEntryMessageColumnInlineFragment
| InfraLogEntryFieldColumnInlineFragment;

export type InfraLogEntryMessageColumnInlineFragment = {
__typename?: 'InfraLogEntryMessageColumn';

columnId: string;

message: Message[];
};

export type Message = InfraLogMessageFieldSegmentInlineFragment;

export type InfraLogMessageFieldSegmentInlineFragment = {
__typename?: 'InfraLogMessageFieldSegment';

field: string;

highlights: string[];
};

export type InfraLogEntryFieldColumnInlineFragment = {
__typename?: 'InfraLogEntryFieldColumn';

columnId: string;

field: string;

highlights: string[];
};
}
1 change: 1 addition & 0 deletions x-pack/plugins/infra/common/http_api/log_sources/log_source_configuration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ export const logSourceColumnConfigurationRT = rt.union([
logSourceMessageColumnConfigurationRT,
logSourceFieldColumnConfigurationRT,
]);
export type LogSourceColumnConfiguration = rt.TypeOf<typeof logSourceColumnConfigurationRT>;

export const logSourceConfigurationPropertiesRT = rt.strict({
name: rt.string,
Expand Down
Loading