REST high-level client: add synced flush API (2) #30650

sohaibiftikhar · 2018-05-16T12:38:34Z

Relates to #27205
Continuation of #29189

sohaibiftikhar · 2018-05-16T12:41:42Z

@nik9000 @javanna I moved the stuff to a separate class. I am not sure about the package I put it in but you can suggest the correct one and I will move it to that.

elasticmachine · 2018-05-16T13:39:37Z

Pinging @elastic/es-core-infra

imotov · 2018-05-16T13:40:11Z

@elasticmachine test this please

imotov · 2018-05-16T15:45:19Z

@elasticmachine retest this please

nik9000 · 2018-05-16T19:10:14Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+public class SyncedFlushResponse extends ActionResponse implements ToXContentFragment {
+
+    // Field name declaration
+    public static final String SHARDS = "_shards";


I'd drop the comment above and call just call it SHARD_FIELD.

nik9000 · 2018-05-16T19:10:25Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+        this.indexResults = Collections.unmodifiableMap(indexResults);
+    }
+
+    public int totalShards() {


It'd probably be nice to have javadoc on these.

nik9000 · 2018-05-16T19:13:53Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+        }
+    }
+
+    private static int getIntValueFromField(XContentParser parser, String fieldName) throws IOException {


I think ObjectParser or ConstructingObjectParser would probably be better here because they'll handle things like out order response.

nik9000 · 2018-05-16T19:14:20Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+    // Only used as a container for XContent
+    public static final class ShardFailure implements ToXContentFragment {
+
+        // Field name declaration


Same comment about field declarations.

nik9000 · 2018-05-16T19:15:48Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+    }
+
+    // Only used as a container for XContent
+    public static final class ShardFailure implements ToXContentFragment {


I don't think the comment above is accurate. It looks like you return it from the API so it deserves some description like "Description of a shard that failed to synced flush".

nik9000 · 2018-05-16T19:17:49Z

...l/src/test/java/org/elasticsearch/client/response/indices/flush/AbstractSyncedFlushTest.java

+import org.elasticsearch.indices.flush.SyncedFlushService;
+import org.elasticsearch.test.ESTestCase;
+
+public abstract class AbstractSyncedFlushTest extends ESTestCase {


I don't think it is worth making an abstract superclass with just a single subclass.

I do like the TestPlan object though.

Yeah, the idea before was that both Flush and Syncedflush were using it. I kind of used the same structure. Now, this thing moved to a new package. I'll merge it to a single class.

javanna · 2018-05-17T11:00:02Z

client/rest-high-level/src/main/java/org/elasticsearch/client/RequestConverters.java

@@ -209,6 +210,14 @@ static Request flush(FlushRequest flushRequest) {
        return request;
    }

+    static Request syncedFlush(SyncedFlushRequest syncedFlushRequest) {


can we call it flushSynced too?

imotov · 2018-05-17T16:07:21Z

@elasticmachine retest this please

imotov · 2018-05-17T18:10:15Z

add to whitelist

sohaibiftikhar · 2018-05-22T08:04:17Z

@nik9000 @javanna Could you have a look again, please?

sohaibiftikhar · 2018-05-22T14:22:55Z

docs/java-rest/high-level/indices/flush_synced.asciidoc

+<7> Failed shards for index mentioned in 4.
+<8> One of the failed shard ids of the failed index mentioned in 4.
+<9> Reason for failure of copies of the shard mentioned in 8.
+<10> Optional. Routing information (like id, state, version etc.) for the failed shard copies.


Note to self: Need to change asciidoc. This field is now a Map<String, Object>.

sohaibiftikhar · 2018-05-24T14:25:01Z

CI looks good now.

javanna · 2018-05-28T13:15:52Z

@nik9000 can you have another look at this one? Or shall I?

nik9000

@sohaibiftikhar, I left a few really minor things. I think this is ready to go except for moving a couple of files around and fixing some indentation.

nik9000 · 2018-05-29T13:17:22Z

client/rest-high-level/src/test/java/org/elasticsearch/client/IndicesClientIT.java

@@ -563,6 +565,39 @@ public void testFlush() throws IOException {
        }
    }

+    public void testSyncedFlush() throws IOException {
+        {
+                String index = "index";


Extra indentation.

nik9000 · 2018-05-29T13:17:31Z

client/rest-high-level/src/test/java/org/elasticsearch/client/RequestConvertersTests.java

+        String[] indices = randomBoolean() ? null : randomIndicesNames(0, 5);
+        SyncedFlushRequest syncedFlushRequest;
+        if (randomBoolean()) {
+                syncedFlushRequest = new SyncedFlushRequest(indices);


Extra indentation here too.

nik9000 · 2018-05-29T13:18:42Z

...level/src/test/java/org/elasticsearch/client/documentation/IndicesClientDocumentationIT.java

+    public void testSyncedFlushIndex() throws Exception {
+        RestHighLevelClient client = highLevelClient();
+
+        {


Can you remove the extra block? I don't think it buys anything.

I mean, we still need to create the index, but i don't think we need to do it inside a new block.

Yes but I think the reason here was to follow how it was done before (for eg: in testSyncedFlush). It kind of states that this is a pre-step. Before you test flushing an index it actually needs to exist. If you still think it is unnecessary I will remove it.

I see what you are talking about. Weird. I'm fine with it then. I mean, I don't like it, but I don't really have to like it. It makes sense to make it look like the test above even if the test above looks funny to me!

nik9000 · 2018-05-29T13:24:28Z

...level/src/main/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponse.java

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.client.response.indices.flush;


How would you feel about moving this into the org.elasticsearch.client package? I think we're not going to have too many of these so we may as well stick them into a single package. Less packages to dig through feels like a good thing to me.

nik9000 · 2018-05-29T13:25:59Z

.../src/test/java/org/elasticsearch/client/response/indices/flush/SyncedFlushResponseTests.java

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.elasticsearch.client.response.indices.flush;


Could you also move this to org.elasticsearch.client?

-- Also Fixed indentation

Adds the synced flush API to the high level REST client. Relates to #27205.

nik9000 · 2018-05-30T14:49:13Z

Backported!

* master: Add Verify Repository High Level REST API (#30934) [CI] Mute SamlAuthenticatorTests testIncorrectSigningKeyIsRejected [DOCS] Fixes kibana security file location SQL: Remove log4j and joda from JDBC dependencies (#30938) Revert accidentally pushed changes in NoriAnalysisTests Fix composite agg serialization error Change ScriptException status to 400 (bad request) (#30861) Fix synced flush docs REST high-level client: add synced flush API (2) (#30650) Fix missing option serialization after backport Cross Cluster Search: do not use dedicated masters as gateways (#30926) Fix AliasMetaData parsing (#30866) Fsync state file before exposing it (#30929)

* 6.x: [DOCS] Fixes kibana security file location Fix synced flush docs REST high-level client: add synced flush API (2) (#30650) stable filemode for zip distributions (#30854) Fix index_prefixes cross-type compatibility check (#30956) Add deprecation notice for missing option Add missing_bucket option in the composite agg (#29465) Amend skip version for snapshot REST test Relates #18543 Reintroduces SSL tests (#30947) Fsync state file before exposing it (#30929) Rename index_prefix to index_prefixes (#30932)

* Remove AllocatedPersistentTask.getState() (#30858) This commit removes the method AllocatedPersistentTask.getState() that exposes the internal state of an AllocatedPersistentTask and replaces it with a new isCompleted() method. Related to #29608. * Improve allocation-disabling instructions (#30248) Clarify the “one minute” in the instructions to disable the shard allocation when doing maintenance to say that it is configurable. * Replace several try-finally statements (#30880) This change replaces some existing try-finally statements that close resources in their finally block with the slightly shorter and safer try-with-resources pattern. * Move list tasks under Tasks namespace (#30906) Our API spec define the tasks API as e.g. tasks.list, meaning that they belong to their own namespace. This commit moves them from the cluster namespace to their own namespace. Relates to #29546 * Deprecate accepting malformed requests in stored script API (#28939) The stored scripts API today accepts malformed requests instead of throwing an exception. This PR deprecates accepting malformed put stored script requests (requests not using the official script format). Relates to #27612 * Remove log traces in AzureStorageServiceImpl and fix test (#30924) This commit removes some log traces in AzureStorageServiceImpl and also fixes the AzureStorageServiceTests so that is uses the real implementation to create Azure clients. * Fix IndexTemplateMetaData parsing from xContent (#30917) We failed to register "aliases" and "version" into the list of keywords in the IndexTemplateMetaData; then fail to parse the following index template. ``` { "aliases": {"log": {}}, "index_patterns": ["pattern-1"] } ``` This commit registers that missing keywords. * [DOCS] Reset edit links (#30909) * Limit the scope of BouncyCastle dependency (#30358) Limits the scope of the runtime dependency on BouncyCastle so that it can be eventually removed. * Splits functionality related to reading and generating certificates and keys in two utility classes so that reading certificates and keys doesn't require BouncyCastle. * Implements a class for parsing PEM Encoded key material (which also adds support for reading PKCS8 encoded encrypted private keys). * Removes BouncyCastle dependency for all of our test suites(except for the tests that explicitly test certificate generation) by using pre-generated keys/certificates/keystores. * Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928) This snapshot includes LUCENE-8328 which is needed to stabilize CCR builds. * Moved keyword tokenizer to analysis-common module (#30642) Relates to #23658 * [test] packaging test logging for suse distros * Fix location of AbstractHttpServerTransport (#30888) Currently AbstractHttpServerTransport is in a netty4 module. This is the incorrect location. This commit moves it out of netty4 module. Additionally, it moves unit tests that test AbstractHttpServerTransport logic to server. * [test] packaging: use shell when running commands (#30852) When subprocesses are started with ProcessBuilder, they're forked by the java process directly rather than from a shell, which can be surprising for our use case here in the packaging tests which is similar to scripting. This commit changes the tests to run their subprocess commands in a shell, using the bash -c <script> syntax for commands on linux and using the powershell.exe -Command <script> syntax for commands on windows. This syntax on windows is essentially what the tests were already doing. * [DOCS] Adds missing TLS settings for auditing (#30822) * stable filemode for zip distributions (#30854) Applies default file and directory permissions to zip distributions similar to how they're set for the tar distributions. Previously zip distributions would retain permissions they had on the build host's working tree, which could vary depending on its umask For #30799 * Minor clean-up in InternalRange. (#30886) * Make sure all instance variables are final. * Make generateKey a private static method, instead of protected. * Rename formatter -> format for consistency. * Serialize bucket keys as strings as opposed to optional strings. * Pull the stream serialization logic for buckets into the Bucket class. * [DOCS] Remove reference to platinum Docker image (#30916) * Use dedicated ML APIs in tests (#30941) ML has dedicated APIs for datafeeds and jobs yet base test classes and some tests were relying on the cluster state for this state. This commit removes this usage in favor of using the dedicated endpoints. * Update the version checks around range bucket keys, now that the change was backported. * [DOCS] Fix watcher file location * Rename methods in PersistentTasksService (#30837) This commit renames methods in the PersistentTasksService, to make obvious that the methods send requests in order to change the state of persistent tasks. Relates to #29608. * Rename index_prefix to index_prefixes (#30932) This commit also adds index_prefixes tests to TextFieldMapperTests to ensure that cloning and wire-serialization work correctly * Add missing_bucket option in the composite agg (#29465) This change adds a new option to the composite aggregation named `missing_bucket`. This option can be set by source and dictates whether documents without a value for the source should be ignored. When set to true, documents without a value for a field emits an explicit `null` value which is then added in the composite bucket. The `missing` option that allows to set an explicit value (instead of `null`) is deprecated in this change and will be removed in a follow up (only in 7.x). This commit also changes how the big arrays are allocated, instead of reserving the provided `size` for all sources they are created with a small intial size and they grow depending on the number of buckets created by the aggregation: Closes #29380 * Fsync state file before exposing it (#30929) With multiple data paths, we write the state files for index metadata to all data paths. We only properly fsync on the first location, though. For other locations, we possibly expose the file before its contents is properly fsynced. This can lead to situations where, after a crash, and where the first data path is not available anymore, ES will see a partially-written state file, preventing the node to start up. * Fix AliasMetaData parsing (#30866) AliasMetaData should be parsed more leniently so that the high-level REST client can support forward compatibility on it. This commit addresses this issue that was found as part of #28799 and adds dedicated XContent tests as well. * Cross Cluster Search: do not use dedicated masters as gateways (#30926) When we are connecting to a remote cluster we should never select dedicated master nodes as gateway nodes, or we will end up loading them with requests that should rather go to other type of nodes e.g. data nodes or coord_only nodes. This commit adds the selection based on the node role, to the existing selection based on version and potential node attributes. Closes #30687 * Fix missing option serialization after backport Relates #29465 * REST high-level client: add synced flush API (2) (#30650) Adds the synced flush API to the high level REST client. Relates to #27205. * Fix synced flush docs They had some copy and paste errors that failed the docs build. * Change ScriptException status to 400 (bad request) (#30861) Currently failures to compile a script usually lead to a ScriptException, which inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does not contain another root cause. Instead, this should be a 400 Bad Request error. This PR changes this more generally for script compilation errors by changing ScriptException to return 400 (bad request) as status code. Closes #12315 * Fix composite agg serialization error Fix serialization after backport Relates #29465 * Revert accidentally pushed changes in NoriAnalysisTests * SQL: Remove log4j and joda from JDBC dependencies (#30938) More cleanup of JDBC driver project Relates to #29856 * [DOCS] Fixes kibana security file location * [CI] Mute SamlAuthenticatorTests testIncorrectSigningKeyIsRejected Tracked by #30970 * Add Verify Repository High Level REST API (#30934) This commit adds Verify Repository, the associated docs and tests for the high level REST API client. A few small changes to the Verify Repository Response went into the commit as well. Relates #27205 * Add “took” timing info to response for _msearch/template API (#30961) Add “took” timing info to response for _msearch/template API Closes #30957 * Mute FlushIT tests We have identified the source causing these tests failed. This commit mutes them again until we have a proper fix. Relates #29392 * [CI] Mute HttpSecretsIntegrationTests#testWebhookAction test Tracked by #30094 * [Test] Prefer ArrayList over Vector (#30965) Replaces some occurances of Vector class with ArrayList in tests of the rank-eval module. * Fix license on AcitveDirectorySIDUtil (#30972) This code is from an Apache 2.0 licensed codebase and when we imported it into our codebase it carried the Apache 2.0 license as well. However, during the migration of the X-Pack codebase from the internal private repository to the elastic/elasticsearch repository, the migration tool mistakently changed the license on this source file from the Apache 2.0 license to the Elastic license. This commit addresses this mistake by reapplying the Apache 2.0 license. * [CI] Mute Ml rolling upgrade tests Tracked by #30982 * Make AllocatedPersistentTask.isCompleted() protected (#30949) This commit changes the isCompleted() method to be protected so that classes that extends AllocatedPersistentTask can use it. Related to #30858 * [CI] Mute Ml rolling upgrade test for mixed cluster too It can fail in either the mixed cluster or the upgraded cluster, so it needs to be muted in both. Tracked by #30982 * [Docs] Fix typo in Min Aggregation reference (#30899) * Refactor Sniffer and make it testable (#29638) This commit reworks the Sniffer component to simplify it and make it possible to test it. In particular, it no longer takes out the host that failed when sniffing on failure, but rather relies on whatever the cluster returns. This is the result of some valid comments from #27985. Taking out one single host is too naive, hard to test and debug. A new Scheduler abstraction is introduced to abstract the tasks scheduling away and make it possible to plug in any test implementation and take out timing aspects when testing. Concurrency aspects have also been improved, synchronized methods are no longer required. At the same time, we were able to take #27697 and #25701 into account and fix them, especially now that we can more easily add tests. Last but not least, unit tests are added for the Sniffer component, long overdue. Closes #27697 Closes #25701 * Deprecates indexing and querying a context completion field without context (#30712) This change deprecates completion queries and documents without context that target a context enabled completion field. Querying without context degrades the search performance considerably (even when the number of indexed contexts is low). This commit targets master but the deprecation will take place in 6.x and the functionality will be removed in 7 in a follow up. Closes #29222 * Core: Remove RequestBuilder from Action (#30966) This commit removes the RequestBuilder generic type from Action. It was needed to be used by the newRequest method, which in turn was used by client.prepareExecute. Both of these methods are now removed, along with the existing users of prepareExecute constructing the appropriate builder directly. * Ensure intended key is selected in SamlAuthenticatorTests (#30993) * Ensure that a purposefully wrong key is used Uses a specific keypair for tests that require a purposefully wrong keypair instead of selecting one randomly from the same pull from which the correct one is selected. Entropy is low because of the small space and the same key can be randomly selected as both the correct one and the wrong one, causing the tests to fail. The purposefully wrong key is also used in testSigningKeyIsReloadedForEachRequest and needs to be cleaned up afterwards so the rest of the tests don't use that for signing. Resolves #30970 * [DOCS] Update readme for testing x-pack code snippets (#30696) * Remove version read/write logic in Verify Response (#30879) Since master will always communicate with a >=6.4 node, the logic for checking if the node is 6.4 and conditionally reading and writing based on that can be removed from master. This logic will stay in 6.x as it is the bridge to the cleaner response in master. This also unmutes the failing test due to this bwc change. Closes #30807 * HLRest: Allow caller to set per request options (#30490) This modifies the high level rest client to allow calling code to customize per request options for the bulk API. You do the actual customization by passing a `RequestOptions` object to the API call which is set on the `Request` that is generated by the high level client. It also makes the `RequestOptions` a thing in the low level rest client. For now that just means you use it to customize the headers and the `httpAsyncResponseConsumerFactory` and we'll add node selectors and per request timeouts in a follow up. I only implemented this on the bulk API because it is the first one in the list alphabetically and I wanted to keep the change small enough to review. I'll convert the remaining APIs in a followup. * [DOCS] Clarify not all PKCS12 usable as truststores (#30750) Although elasticsearch-certutil generates PKCS#12 files which are usable as both keystore and truststore this is uncommon in practice. Settle these expectations for the users following our security guides. * Transport client: Don't validate node in handshake (#30737) This is related to #30141. Right now in the transport client we open a temporary node connection and take the node information. This node information is used to open a permanent connection that is used for the client. However, we continue to use the configured transport address. If the configured transport address is a load balancer, you might connect to a different node for the permanent connection. This causes the handshake validation to fail. This commit removes the handshake validation for the transport client when it simple node sample mode. * Remove unused query methods from MappedFieldType. (#30987) * Remove MappedFieldType#nullValueQuery, as it is now unused. * Remove MappedFieldType#queryStringTermQuery, as it is never overridden. * Reuse expiration date of trial licenses (#30950) * Retain the expiryDate for trial licenses While updating the license signature to the new license spec retain the trial license expiration date to that of the existing license. Resolves #30882 * Watcher: Give test a little more time Changes watcher's integration tests to wait 30 seconds when starting watcher rather than 10 seconds because this build failed when starting took 12 seconds: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+periodic/222/console

sohaibiftikhar mentioned this pull request May 16, 2018

REST high-level client: add synced flush API #29189

Closed

imotov added >enhancement v7.0.0 :Core/Java High Level REST Client v6.4.0 labels May 16, 2018

sohaibiftikhar force-pushed the synced_flush_v2 branch from 1672969 to ecf93b2 Compare May 16, 2018 15:36

nik9000 reviewed May 16, 2018

View reviewed changes

javanna reviewed May 17, 2018

View reviewed changes

sohaibiftikhar commented May 22, 2018

View reviewed changes

javanna mentioned this pull request May 28, 2018

Java high-level REST client completeness #27205

Closed

80 tasks

nik9000 requested changes May 29, 2018

View reviewed changes

sohaibiftikhar added 7 commits May 29, 2018 23:39

Synced Flush High Level API

a11702a

removed AbstractSyncedFlushTests

b3d24af

renamed fields and added javadoc

5d64a4d

changed to constructingobjectparser

4329999

rename to flushSynced

45028a3

fixed ascii doc for shard routing

eac85e8

moved SyncedFlushResponse to org.elasticsearch.client

90d23ad

-- Also Fixed indentation

sohaibiftikhar force-pushed the synced_flush_v2 branch from 63e44a9 to 90d23ad Compare May 29, 2018 21:58

nik9000 approved these changes May 30, 2018

View reviewed changes

nik9000 merged commit 2c27c58 into elastic:master May 30, 2018

nik9000 pushed a commit that referenced this pull request May 30, 2018

REST high-level client: add synced flush API (2) (#30650)

da71143

Adds the synced flush API to the high level REST client. Relates to #27205.

sohaibiftikhar deleted the synced_flush_v2 branch June 1, 2018 09:13

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REST high-level client: add synced flush API (2) #30650

REST high-level client: add synced flush API (2) #30650

sohaibiftikhar commented May 16, 2018 •

edited

Loading

sohaibiftikhar commented May 16, 2018

elasticmachine commented May 16, 2018

imotov commented May 16, 2018

imotov commented May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

nik9000 May 16, 2018

sohaibiftikhar May 16, 2018

javanna May 17, 2018

imotov commented May 17, 2018

imotov commented May 17, 2018

sohaibiftikhar commented May 22, 2018

sohaibiftikhar May 22, 2018

sohaibiftikhar commented May 24, 2018

javanna commented May 28, 2018

nik9000 left a comment

nik9000 May 29, 2018

nik9000 May 29, 2018

nik9000 May 29, 2018

nik9000 May 29, 2018

sohaibiftikhar May 29, 2018 •

edited

Loading

nik9000 May 29, 2018

nik9000 May 29, 2018

nik9000 May 29, 2018

nik9000 commented May 30, 2018

REST high-level client: add synced flush API (2) #30650

REST high-level client: add synced flush API (2) #30650

Conversation

sohaibiftikhar commented May 16, 2018 • edited Loading

sohaibiftikhar commented May 16, 2018

elasticmachine commented May 16, 2018

imotov commented May 16, 2018

imotov commented May 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imotov commented May 17, 2018

imotov commented May 17, 2018

sohaibiftikhar commented May 22, 2018

Choose a reason for hiding this comment

sohaibiftikhar commented May 24, 2018

javanna commented May 28, 2018

nik9000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sohaibiftikhar May 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented May 30, 2018

sohaibiftikhar commented May 16, 2018 •

edited

Loading

sohaibiftikhar May 29, 2018 •

edited

Loading