Add Fleet action results system data stream #71667

jaymode · 2021-04-14T02:55:09Z

This commit adds support for system data streams and also the first use
of a system data stream with the fleet action results data stream. A
system data stream is one that is used to store system data that users
should not interact with directly. Elasticsearch will manage these data
streams. REST API access is available for external system data streams
so that other stack components can store system data within a system
datastream. System data streams will not use the system index read and
write threadpools.

This commit adds support for system data streams and also the first use of a system data stream with the fleet action results data stream.

jaymode · 2021-04-14T14:56:14Z

@elasticmachine run elasticsearch-ci/1

aleksmaus · 2021-04-14T22:06:28Z

Tested this change with the latest fleet/agent/osquerybeat end-to-end all works.

The data gets stored:

The mapping looks correct:

The .fleet-actions-results data stream probably should be hidden, otherwise kibana shows error trying to render the ilm policy details.

If we hide it though, how would the user can change ilm policy? Do the user need to be able to change it?

x-pack/plugin/core/src/main/resources/fleet-actions-results.json

jaymode · 2021-04-15T03:00:08Z

The ILM policy did not exist when you tested. I just pushed a change that will create the ILM policy but I do wonder if the UI needs to be updated for system data streams; maybe some options don't make sense? For the ILM policy, I'd expect the user to be able to edit it to change what they want to store in the datastream and how long they want to keep data in a certain data tier even.

aleksmaus · 2021-04-15T12:04:27Z

picked up the latest changes, seeing these in the log

java.lang.IllegalArgumentException: unable to load policy [.fleet-actions-results-ilm-policy] from [/fleet-actions-results-ilm-policy.json]
        at org.elasticsearch.xpack.core.ilm.LifecyclePolicyUtils.loadPolicy(LifecyclePolicyUtils.java:45) ~[?:?]
        at org.elasticsearch.xpack.core.template.LifecyclePolicyConfig.load(LifecyclePolicyConfig.java:43) ~[?:?]
        at org.elasticsearch.xpack.core.template.IndexTemplateRegistry.lambda$addIndexLifecyclePoliciesIfMissing$8(IndexTemplateRegistry.java:396) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
        at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.xpack.core.template.IndexTemplateRegistry.addIndexLifecyclePoliciesIfMissing(IndexTemplateRegistry.java:397) ~[?:?]
        at org.elasticsearch.xpack.core.template.IndexTemplateRegistry.clusterChanged(IndexTemplateRegistry.java:176) ~[?:?]
        at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:509) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:499) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:467) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:407) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:151) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:669) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [2:3] [lifecycle_policy] unknown field [policy]
        at org.elasticsearch.common.xcontent.ObjectParser.lambda$errorOnUnknown$2(ObjectParser.java:93) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:293) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:166) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:158) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.core.ilm.LifecyclePolicy.parse(LifecyclePolicy.java:101) ~[?:?]
        at org.elasticsearch.xpack.core.ilm.LifecyclePolicyUtils.loadPolicy(LifecyclePolicyUtils.java:42) ~[?:?]
        ... 22 more

"Index lifecycle policy None"

jaymode · 2021-04-15T15:43:36Z

seeing these in the log

That's what I get for not writing a test; sorry. Should be good to go now with a test.

aleksmaus · 2021-04-15T16:16:59Z

Retested, no error this time

the ilm policy with index:

the ilm policy details:

assuming the

Index lifecycle policy
    None

is correct since it's datastream. please confirm.

aleksmaus · 2021-04-15T16:22:23Z

saw this error:

[2021-04-15T12:15:52,087][ERROR][o.e.x.i.IndexLifecycleRunner] [runTask-0] policy [.fleet-actions-results-ilm-policy] for index [.ds-.fleet-actions-results-2021.04.15-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: data stream [.fleet-actions-results] cannot be rolled over, because it is a replicated data stream
        at org.elasticsearch.cluster.metadata.DataStream.rollover(DataStream.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.resolveDataStreamRolloverNames(MetadataRolloverService.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.admin.indices.rollover.MetadataRolloverService.resolveRolloverNames(MetadataRolloverService.java:124) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.lambda$masterOperation$1(TransportRolloverAction.java:101) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:100) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:164) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:163) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:217) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:389) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:355) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:332) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$5.handleResponse(TransportService.java:618) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1158) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1236) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1216) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.finishHim(TransportBroadcastByNodeAction.java:455) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler$1.onResponse(TransportBroadcastByNodeAction.java:419) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler$1.onResponse(TransportBroadcastByNodeAction.java:413) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$3.innerOnResponse(ActionListener.java:414) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler$2.onResponse(TransportBroadcastByNodeAction.java:495) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler$2.onResponse(TransportBroadcastByNodeAction.java:489) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:445) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:91) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:40) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:489) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:397) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:232) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:286) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:95) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:116) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:77) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:379) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:95) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:116) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:77) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:22) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:379) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:95) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:116) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:379) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:95) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:116) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:77) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:24) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:379) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:101) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:165) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:95) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:87) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:133) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:116) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:53) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:106) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:385) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:362) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:302) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:687) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:662) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:556) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:336) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:725) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:599) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:290) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:725) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:367) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:286) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:723) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:289) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:723) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$5(RBACEngine.java:328) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:352) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:325) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:300) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:265) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:229) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:127) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:221) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:133) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:121) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:231) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:112) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:336) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$8(AuthenticationService.java:414) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:425) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:333) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:184) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:103) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:293) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:793) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-04-15T12:15:59,816][INFO ][o.e.c.m.MetadataMappingService] [runTask-0] [.ds-ilm-history-5-2021.04.15-000001/XIPdjQukS-eNHHks-aJ0Nw] update_mapping [_doc]
[2021-04-15T12:16:09,368][WARN ][o.e.x.d.a.GetDataStreamsTransportAction] [runTask-0] couldn't find any matching template for data stream [.fleet-actions-results]. has it been restored (and possibly renamed)from a snapshot?

elasticmachine · 2021-04-19T14:26:48Z

Pinging @elastic/es-core-features (Team:Core/Features)

williamrandolph

Looks good! I left a few small comments.

client/rest-high-level/src/main/java/org/elasticsearch/client/indices/DataStream.java

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateDataStreamService.java

server/src/main/java/org/elasticsearch/indices/SystemDataStreamDescriptor.java

martijnvg

Implementation looks good to me. I left a few small comments.

server/src/main/java/org/elasticsearch/indices/SystemDataStreamDescriptor.java

...a-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/SystemDataStreamIT.java

martijnvg · 2021-04-20T07:16:38Z

...a-streams/src/internalClusterTest/java/org/elasticsearch/datastreams/SystemDataStreamIT.java

+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+
+public class SystemDataStreamIT extends ESIntegTestCase {


Could this test be turned into a qa module inside ../plugin/data-streams/qa?
That feels more robust to me. (the qa module would then both be an esplugin and java-rest-test)

Would you be ok with this refactor being a follow up?

williamrandolph

Still looking good. One little suggestion on a variable name.

server/src/main/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java

This commit adds support for system data streams and also the first use of a system data stream with the fleet action results data stream. A system data stream is one that is used to store system data that users should not interact with directly. Elasticsearch will manage these data streams. REST API access is available for external system data streams so that other stack components can store system data within a system data stream. System data streams will not use the system index read and write threadpools. Backport of #71667

This change reenables BWC tests after elastic#71667 was backported in elastic#71966.

This change reenables BWC tests after #71667 was backported in #71966.

jaymode added 3 commits April 13, 2021 20:52

Add Fleet action results system data stream

6815f50

This commit adds support for system data streams and also the first use of a system data stream with the fleet action results data stream.

spotless

d4b58f1

more spotless

8cee729

jaymode added 3 commits April 14, 2021 10:02

add checks to other apis

6f242cb

test fixes

dd4e918

Merge branch 'master' into system_data_streams

625ae1a

aleksmaus reviewed Apr 14, 2021

View reviewed changes

x-pack/plugin/core/src/main/resources/fleet-actions-results.json Outdated Show resolved Hide resolved

add ILM policy installation

694e314

bump value

45a3aeb

jaymode added 2 commits April 15, 2021 09:42

fix ilm policy

00e2a13

Merge branch 'master' into system_data_streams

0c62feb

jaymode requested a review from williamrandolph April 15, 2021 15:43

fix bug in getting datastream action and creating

da805a3

aleksmaus mentioned this pull request Apr 15, 2021

Remove fleet datastream bootstrap elastic/integrations#915

Merged

3 tasks

jaymode added 7 commits April 15, 2021 14:03

code docs

2b1b78e

better validation for overlapping names

1b1a720

fix test

72c7b2e

Merge branch 'master' into system_data_streams

2d85e3e

javadocs on SystemDataStreamDescriptor

116ca63

create system datastream unit test

5ead66d

rollover fixes

281d04e

ruflin mentioned this pull request Apr 19, 2021

Manage fleet server indices as system indices in the Elastic Stack elastic/fleet-server#135

Closed

jaymode added the :Data Management/Data streams Data streams and their lifecycles label Apr 19, 2021

jaymode added v7.13.0 v8.0.0 labels Apr 19, 2021

jaymode marked this pull request as ready for review April 19, 2021 14:26

elasticmachine added the Team:Data Management Meta label for data/management team label Apr 19, 2021

williamrandolph approved these changes Apr 20, 2021

View reviewed changes

jaymode added 4 commits April 19, 2021 20:41

more test and feedback

d5fda7d

Merge branch 'master' into system_data_streams

8e68e90

disable bwc

587ed7b

fix delete back to way it was

1411259

martijnvg reviewed Apr 20, 2021

View reviewed changes

jaymode added 2 commits April 20, 2021 09:56

available processors

28577de

feedback

c4a85d3

jaymode mentioned this pull request Apr 20, 2021

Add Fleet action results system data stream #71966

Merged

clean up fix

9a78abc

williamrandolph reviewed Apr 20, 2021

View reviewed changes

server/src/main/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java Outdated Show resolved Hide resolved

variable naming

9c47e00

jaymode merged commit a7dbb31 into elastic:master Apr 20, 2021

jaymode added the backport pending label Apr 20, 2021

jaymode added a commit to jaymode/elasticsearch that referenced this pull request Apr 20, 2021

Reenable BWC tests after elastic#71667

a5aa83e

This change reenables BWC tests after elastic#71667 was backported in elastic#71966.

jaymode mentioned this pull request Apr 20, 2021

Reenable BWC tests after #71667 #71985

Merged

jaymode added a commit that referenced this pull request Apr 21, 2021

Reenable BWC tests after #71667 (#71985)

c2b4bb1

This change reenables BWC tests after #71667 was backported in #71966.

jaymode removed the backport pending label Apr 21, 2021

jaymode deleted the system_data_streams branch April 21, 2021 16:16

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

This was referenced Jul 22, 2024

Get snapshot API returns duplicate information for .fleet-actions-results system data stream #111146

Open

Missing .fleet-actions-results system data stream in the fleet feature state reported by Get snapshot API #111148

Open

stefnestor mentioned this pull request Nov 2, 2024

.fleet-action-results cannot be accessed by cloud #116139

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fleet action results system data stream #71667

Add Fleet action results system data stream #71667

jaymode commented Apr 14, 2021 •

edited

Loading

jaymode commented Apr 14, 2021

aleksmaus commented Apr 14, 2021

jaymode commented Apr 15, 2021

aleksmaus commented Apr 15, 2021 •

edited

Loading

jaymode commented Apr 15, 2021

aleksmaus commented Apr 15, 2021

aleksmaus commented Apr 15, 2021

elasticmachine commented Apr 19, 2021

williamrandolph left a comment

martijnvg left a comment

martijnvg Apr 20, 2021

jaymode Apr 20, 2021

martijnvg Apr 20, 2021

williamrandolph left a comment

Add Fleet action results system data stream #71667

Add Fleet action results system data stream #71667

Conversation

jaymode commented Apr 14, 2021 • edited Loading

jaymode commented Apr 14, 2021

aleksmaus commented Apr 14, 2021

jaymode commented Apr 15, 2021

aleksmaus commented Apr 15, 2021 • edited Loading

jaymode commented Apr 15, 2021

aleksmaus commented Apr 15, 2021

aleksmaus commented Apr 15, 2021

elasticmachine commented Apr 19, 2021

williamrandolph left a comment

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Apr 20, 2021

Choose a reason for hiding this comment

jaymode Apr 20, 2021

Choose a reason for hiding this comment

martijnvg Apr 20, 2021

Choose a reason for hiding this comment

williamrandolph left a comment

Choose a reason for hiding this comment

jaymode commented Apr 14, 2021 •

edited

Loading

aleksmaus commented Apr 15, 2021 •

edited

Loading