Adding a parameter to catalogs, Pipeline and Recurrence information, …

…and object refactoring (#1893)

* Adding a parameter to catalogs, Pipeline and Recurrence information, and object refactoring

* Catalogs.java: added a parameter, called basic, to indicate that you
want basic information rather than nested information about a table
(i.e., list tables method) for performance increase

* Adding Pipeline and Recurrence information to jobs
* Added methods to get such information

* Refactoring and comment changes

* Code gen for ADLS

* Refactoring and comment changes

* [CODE GEN] - BREAKING CHANGE: Adl object refactor

* Code gen based on this PR:
Azure/azure-rest-api-specs#1429

* Modifying the gulpfile to align with the directory revamp of the
Swagger repo

* Modifying the test files to work with the object-refactoring changes
in the PR

* Adding a few more tests to align with the well-covered tests for .NET
SDK

* Removing "import com.microsoft.azure.SubResource"

* [BUGFIX] - Fixing up and commenting out tests

* For JobOperationsTests, I found a server side bug while testing this
call in the .NET SDK, so I was not surprised to see that it fails here
as well.

* For CatalogOperationsTests, I commented out "BUCKETS" from the script
because apparently that keyword is deprecated.  Furthermore, I commented
out the "canCreateUpdateDeleteSecretsAndCredentials" test because the
scripts that are being used are deprecated.  According to the error
message, I must use Azure Powershell cmdlets for such opeartions.  I am
not surprised at this because, to my knowledge, it's been a while since
this test case was run.  I will look into how I can test this in the
near future.

* Other than that, I made sure that JobOperationsTests as well as
AccountOperationsTests (for both ADLA and ADLS) passed.

* [CODE REFACTOR] - Removing an import to pass style check

* Removing the unused import java.io.IOException across files to pass
Maven style check

* [CODE REFACTOR] - Generating UUID from SdkContext

* This is so that Travis passes!

* [FILE REFACTOR] - Updating Session Records

* [BUGFIX] - Making sure variables are caught in recording

* Three variables in DataLakeAnalyticsMangementTestBase.java weren't
recorded

* [COMMENT] - The moving of the declaration of variables

* [COMMENT REFACTORING] - Minor refactoring

* [CODE REFACTOR] - Adding a small sleep for better test results

* [CODE REFACTOR] - Making the sleep longer

* Apparently the cache is restored every 5 minutes

azure-mgmt-datalake-analytics/src/main/java/com/microsoft/azure/management/datalake/analytics/DataLakeAnalyticsJobManagementClient.java

@@ -102,6 +102,18 @@ public interface DataLakeAnalyticsJobManagementClient {
      */
     DataLakeAnalyticsJobManagementClient withGenerateClientRequestId(boolean generateClientRequestId);
 
+    /**
+     * Gets the Pipelines object to access its operations.
+     * @return the Pipelines object.
+     */
+    Pipelines pipelines();
+
+    /**
+     * Gets the Recurrences object to access its operations.
+     * @return the Recurrences object.
+     */
+    Recurrences recurrences();
+
     /**
      * Gets the Jobs object to access its operations.
      * @return the Jobs object.

azure-mgmt-datalake-analytics/src/main/java/com/microsoft/azure/management/datalake/analytics/Pipelines.java

@@ -0,0 +1,247 @@
+/**
+ * Copyright (c) Microsoft Corporation. All rights reserved.
+ * Licensed under the MIT License. See License.txt in the project root for
+ * license information.
+ *
+ * Code generated by Microsoft (R) AutoRest Code Generator.
+ */
+
+package com.microsoft.azure.management.datalake.analytics;
+
+import com.microsoft.azure.CloudException;
+import com.microsoft.azure.ListOperationCallback;
+import com.microsoft.azure.management.datalake.analytics.models.JobPipelineInformation;
+import com.microsoft.azure.Page;
+import com.microsoft.azure.PagedList;
+import com.microsoft.rest.ServiceCallback;
+import com.microsoft.rest.ServiceFuture;
+import com.microsoft.rest.ServiceResponse;
+import java.util.List;
+import java.util.UUID;
+import org.joda.time.DateTime;
+import rx.Observable;
+
+/**
+ * An instance of this class provides access to all the operations defined
+ * in Pipelines.
+ */
+public interface Pipelines {
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @throws CloudException thrown if the request is rejected by server
+     * @throws RuntimeException all other wrapped checked exceptions if the request fails to be sent
+     * @return the PagedList<JobPipelineInformation> object if successful.
+     */
+    PagedList<JobPipelineInformation> list(final String accountName);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param serviceCallback the async ServiceCallback to handle successful and failed responses.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the {@link ServiceFuture} object
+     */
+    ServiceFuture<List<JobPipelineInformation>> listAsync(final String accountName, final ListOperationCallback<JobPipelineInformation> serviceCallback);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<Page<JobPipelineInformation>> listAsync(final String accountName);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<ServiceResponse<Page<JobPipelineInformation>>> listWithServiceResponseAsync(final String accountName);
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param startDateTime The start date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @throws CloudException thrown if the request is rejected by server
+     * @throws RuntimeException all other wrapped checked exceptions if the request fails to be sent
+     * @return the PagedList&lt;JobPipelineInformation&gt; object if successful.
+     */
+    PagedList<JobPipelineInformation> list(final String accountName, final DateTime startDateTime, final DateTime endDateTime);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param startDateTime The start date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param serviceCallback the async ServiceCallback to handle successful and failed responses.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the {@link ServiceFuture} object
+     */
+    ServiceFuture<List<JobPipelineInformation>> listAsync(final String accountName, final DateTime startDateTime, final DateTime endDateTime, final ListOperationCallback<JobPipelineInformation> serviceCallback);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param startDateTime The start date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<Page<JobPipelineInformation>> listAsync(final String accountName, final DateTime startDateTime, final DateTime endDateTime);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param startDateTime The start date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the list of pipelines. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<ServiceResponse<Page<JobPipelineInformation>>> listWithServiceResponseAsync(final String accountName, final DateTime startDateTime, final DateTime endDateTime);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @throws CloudException thrown if the request is rejected by server
+     * @throws RuntimeException all other wrapped checked exceptions if the request fails to be sent
+     * @return the JobPipelineInformation object if successful.
+     */
+    JobPipelineInformation get(String accountName, UUID pipelineIdentity);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @param serviceCallback the async ServiceCallback to handle successful and failed responses.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the {@link ServiceFuture} object
+     */
+    ServiceFuture<JobPipelineInformation> getAsync(String accountName, UUID pipelineIdentity, final ServiceCallback<JobPipelineInformation> serviceCallback);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the JobPipelineInformation object
+     */
+    Observable<JobPipelineInformation> getAsync(String accountName, UUID pipelineIdentity);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the JobPipelineInformation object
+     */
+    Observable<ServiceResponse<JobPipelineInformation>> getWithServiceResponseAsync(String accountName, UUID pipelineIdentity);
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @param startDateTime The start date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @throws CloudException thrown if the request is rejected by server
+     * @throws RuntimeException all other wrapped checked exceptions if the request fails to be sent
+     * @return the JobPipelineInformation object if successful.
+     */
+    JobPipelineInformation get(String accountName, UUID pipelineIdentity, DateTime startDateTime, DateTime endDateTime);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @param startDateTime The start date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param serviceCallback the async ServiceCallback to handle successful and failed responses.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the {@link ServiceFuture} object
+     */
+    ServiceFuture<JobPipelineInformation> getAsync(String accountName, UUID pipelineIdentity, DateTime startDateTime, DateTime endDateTime, final ServiceCallback<JobPipelineInformation> serviceCallback);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @param startDateTime The start date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the JobPipelineInformation object
+     */
+    Observable<JobPipelineInformation> getAsync(String accountName, UUID pipelineIdentity, DateTime startDateTime, DateTime endDateTime);
+
+    /**
+     * Gets the Pipeline information for the specified pipeline ID.
+     *
+     * @param accountName The Azure Data Lake Analytics account to execute job operations on.
+     * @param pipelineIdentity Pipeline ID.
+     * @param startDateTime The start date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @param endDateTime The end date for when to get the pipeline and aggregate its data. The startDateTime and endDateTime can be no more than 30 days apart.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the JobPipelineInformation object
+     */
+    Observable<ServiceResponse<JobPipelineInformation>> getWithServiceResponseAsync(String accountName, UUID pipelineIdentity, DateTime startDateTime, DateTime endDateTime);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param nextPageLink The NextLink from the previous successful call to List operation.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @throws CloudException thrown if the request is rejected by server
+     * @throws RuntimeException all other wrapped checked exceptions if the request fails to be sent
+     * @return the PagedList&lt;JobPipelineInformation&gt; object if successful.
+     */
+    PagedList<JobPipelineInformation> listNext(final String nextPageLink);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param nextPageLink The NextLink from the previous successful call to List operation.
+     * @param serviceFuture the ServiceFuture object tracking the Retrofit calls
+     * @param serviceCallback the async ServiceCallback to handle successful and failed responses.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the {@link ServiceFuture} object
+     */
+    ServiceFuture<List<JobPipelineInformation>> listNextAsync(final String nextPageLink, final ServiceFuture<List<JobPipelineInformation>> serviceFuture, final ListOperationCallback<JobPipelineInformation> serviceCallback);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param nextPageLink The NextLink from the previous successful call to List operation.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<Page<JobPipelineInformation>> listNextAsync(final String nextPageLink);
+
+    /**
+     * Lists all pipelines.
+     *
+     * @param nextPageLink The NextLink from the previous successful call to List operation.
+     * @throws IllegalArgumentException thrown if parameters fail the validation
+     * @return the observable to the PagedList&lt;JobPipelineInformation&gt; object
+     */
+    Observable<ServiceResponse<Page<JobPipelineInformation>>> listNextWithServiceResponseAsync(final String nextPageLink);
+
+}