changes required for using to Power BI Service with Open Distro For E…

…lasticsearch (#669) * pbi connector * update names * - add mez file - add docs * nit * nit * image resize * resize image * remove extra lines * nit * update mez file * rename connector * rename mez file * - add basic & aws auth - add unit test - add errors - enable direct query * add connector * update doc * add ssl for auth * update docs for errors * - remove sqlgetinfo & sqlcapabilities values from connector * adding EncryptedConnectionString * add data type for columns * add alternate column size * reverting changes in catalog test * nit * revert changes for column size validation * address PR comments * Update SQL_GROUP_BY value for fixing query folding issue * removing AstVisitor for resolving query folding issue with data type boolean * fix parameter binding issue * simplify changes * update user doc for pbi * fix query folding issue while applying filter * change literal prefix & suffix value * add user doc for using Power BI Service * resize image * - add server instead of host & port for Power BI Service - update user docs * resize image * resize image * resize image * update connector file * update docs for aws auth credentials * fix typo
opendistro-for-elasticsearch · Aug 10, 2020 · a45ff3a · a45ff3a
1 parent 6021470
commit a45ff3a
Show file tree

Hide file tree

Showing 20 changed files with 94 additions and 30 deletions.
diff --git a/sql-odbc/docs/user/img/pbi_connection_string_options.png b/sql-odbc/docs/user/img/pbi_connection_string_options.png
diff --git a/sql-odbc/docs/user/img/pbi_error_conn.png b/sql-odbc/docs/user/img/pbi_error_conn.png
diff --git a/sql-odbc/docs/user/img/pbi_gateway_connector_path.png b/sql-odbc/docs/user/img/pbi_gateway_connector_path.png
diff --git a/sql-odbc/docs/user/img/pbi_gateway_status.png b/sql-odbc/docs/user/img/pbi_gateway_status.png
diff --git a/sql-odbc/docs/user/img/pbi_publish_report.png b/sql-odbc/docs/user/img/pbi_publish_report.png
diff --git a/sql-odbc/docs/user/img/pbi_publish_status.png b/sql-odbc/docs/user/img/pbi_publish_status.png
diff --git a/sql-odbc/docs/user/img/pbi_select_workspace.png b/sql-odbc/docs/user/img/pbi_select_workspace.png
diff --git a/sql-odbc/docs/user/img/pbi_service_cluster_setting.png b/sql-odbc/docs/user/img/pbi_service_cluster_setting.png
diff --git a/sql-odbc/docs/user/img/pbi_service_data_source.png b/sql-odbc/docs/user/img/pbi_service_data_source.png
diff --git a/sql-odbc/docs/user/img/pbi_service_data_source_success.png b/sql-odbc/docs/user/img/pbi_service_data_source_success.png
diff --git a/sql-odbc/docs/user/img/pbi_service_dataset_gateway.png b/sql-odbc/docs/user/img/pbi_service_dataset_gateway.png
diff --git a/sql-odbc/docs/user/img/pbi_service_dataset_refresh.png b/sql-odbc/docs/user/img/pbi_service_dataset_refresh.png
diff --git a/sql-odbc/docs/user/img/pbi_service_dataset_refresh_history.png b/sql-odbc/docs/user/img/pbi_service_dataset_refresh_history.png
diff --git a/sql-odbc/docs/user/img/pbi_service_edit_reoprt.png b/sql-odbc/docs/user/img/pbi_service_edit_reoprt.png
diff --git a/sql-odbc/docs/user/img/pbi_service_modified_report.png b/sql-odbc/docs/user/img/pbi_service_modified_report.png
diff --git a/sql-odbc/docs/user/img/pbi_service_setting.png b/sql-odbc/docs/user/img/pbi_service_setting.png
diff --git a/sql-odbc/docs/user/power_bi_service_support.md b/sql-odbc/docs/user/power_bi_service_support.md
@@ -0,0 +1,82 @@
+#  Connecting Open Distro For ElasticSearch to Microsoft Power BI Service
+
+## Setup
+* Download and Install [On-premises data gateway](https://docs.microsoft.com/en-us/data-integration/gateway/service-gateway-install)
+* Change the path for custom data connector folder in On-premises data gateway so that the gateway can find the custom connector.
+> NOTE: Ensure the gateway service account (**PBIEgwService**) has permissions to access the custom connector folder. Alternatively, you can copy connector file to `C:\Windows\ServiceProfiles\PBIEgwService\Documents\Power BI Desktop\Custom Connectors\`.
+
+<img src="img/pbi_gateway_connector_path.png" width=500>
+
+* Verify the status of data gateway is ready to be used.
+
+<img src="img/pbi_gateway_status.png" width=500>
+
+* Login to Power BI Service.
+* Click on **Setting** > **Manage Gateway**.
+
+<img src="img/pbi_service_setting.png" width=400>
+
+* Select **Allow user's custom data connectors to refresh through this gateway cluster(preview)**. Click on **Apply**.
+
+<img src="img/pbi_service_cluster_setting.png">
+
+* Click on **Add data sources to use the gateway**.
+* Select Data Source Type as **Open Distro For Elasticsearch**.
+* Enter Data Source Name and Server values.
+* Select required **Authentication Method**. Select **Anonymous** for auth **NONE**.
+For **AWS_SIGV4**, select **Key** and set aws access credentials for user **PBIEgwService** at path `C:\Windows\ServiceProfiles\PBIEgwService\.aws\`
+
+* Select Encryption mode for connection.
+* Click on **Add**.
+
+<img src="img/pbi_service_data_source.png" width=650>
+
+* You will get a **Connection Successful** message.
+
+<img src="img/pbi_service_data_source_success.png" width=650>
+
+
+## Publish Report
+
+* Follow [instructions](./power_bi_support.md) to create graph using Open Disto For Elasticsearch Data connector.
+* Click on **Publish** to publish the report on Power BI service.
+
+<img src="img/pbi_publish_report.png">
+
+* Select destination and click on **Select**.
+
+<img src="img/pbi_select_workspace.png" width=400>
+
+* You will get a success message when report is published.
+
+<img src="img/pbi_publish_status.png" width=400>
+
+* Click on **Open '%report name%' in Power BI** to open published report in Power BI service.
+
+## Modify report using Power BI Service
+
+* Click on **Edit report** to modfify report.
+
+<img src="img/pbi_service_edit_reoprt.png">
+
+* Use **Filters**,**Visualizations** and **Fields** to modify report.
+
+<img src="img/pbi_service_modified_report.png">
+
+## Dataset Scheduled Refresh
+
+* Click on **Settings** > **Datasets**.
+* Select required Gateway and click on **Apply**.
+
+<img src="img/pbi_service_dataset_gateway.png" width=800>
+
+* Turn on Keep your data up to date option.
+* Select refresh frequency and timezone.
+* Add email for failure notifications if required.
+* Click on **Apply**.
+
+<img src="img/pbi_service_dataset_refresh.png" width=550>
+
+* You can also check history by clicking on **Refresh history**.
+
+<img src="img/pbi_service_dataset_refresh_history.png" width=550>
diff --git a/sql-odbc/docs/user/power_bi_support.md b/sql-odbc/docs/user/power_bi_support.md
@@ -33,7 +33,7 @@
 
 <img src="img/pbi_third_party_warning.png" width="500">
 
-* Enter host and port values. Click on **OK**.
+* Enter server value. Click on **OK**.
 
 <img src="img/pbi_connection_string_options.png" width="500">
 

diff --git a/sql-odbc/src/PowerBIConnector/OdfeSqlOdbcPBIConnector.pq b/sql-odbc/src/PowerBIConnector/OdfeSqlOdbcPBIConnector.pq
@@ -12,22 +12,17 @@ shared OdfeSqlOdbcPBIConnector.Contents = Value.ReplaceType(OdfeSqlOdbcPBIConnec
 
 // Wrapper function to provide additional UI customization.
 OdfeSqlOdbcPBIConnectorType = type function (
-        Host as (type text meta [
-            Documentation.FieldCaption = "Host",
+        Server as (type text meta [
+            Documentation.FieldCaption = "Server",
             Documentation.FieldDescription = "The hostname of the Open Distro For Elasticsearch server.",
-            Documentation.SampleValues = { "localhost" }
-        ]),
-        optional Port as (type number meta [
-            Documentation.FieldCaption = "Port",
-            Documentation.FieldDescription = "The port of the Open Distro For Elasticsearch server is running on.",
-            Documentation.SampleValues = { 9200 }
+            Documentation.SampleValues = { "localhost:9200" }
         ])
     )
     as table meta [
         Documentation.Name = "Open Distro For Elasticsearch"
     ];
 
-OdfeSqlOdbcPBIConnectorImpl = (Host as text, optional Port as number) as table =>
+OdfeSqlOdbcPBIConnectorImpl = (Server as text) as table =>
     let
         Credential = Extension.CurrentCredential(),
         AuthenticationMode = Credential[AuthenticationKind],
@@ -61,21 +56,9 @@ OdfeSqlOdbcPBIConnectorImpl = (Host as text, optional Port as number) as table =
                     UseSSL = 0
                 ],
 
-        // Set host & port in connection string. 
-        // Do not include port in connection string for aws server connection.
-        Server = 
-            if Port <> null then
-            [
-                Host = Host,
-                Port = Port
-            ]
-            else
-            [
-                Host = Host
-            ],
-
         ConnectionString = [
-            Driver = "ODFE SQL ODBC Driver"
+            Driver = "ODFE SQL ODBC Driver",
+            Host = Server
         ],
 
         SQLGetInfo = Diagnostics.LogValue("SQLGetInfo_Options", [
@@ -123,7 +106,7 @@ OdfeSqlOdbcPBIConnectorImpl = (Host as text, optional Port as number) as table =
             SupportsOdbcTimestampLiterals = true
         ]),
 
-        OdbcDatasource = Odbc.DataSource(ConnectionString & Server & CredentialConnectionString & EncryptedConnectionString, [
+        OdbcDatasource = Odbc.DataSource(ConnectionString & CredentialConnectionString & EncryptedConnectionString, [
             // Do not view the tables grouped by their schema names.
             HierarchicalNavigation = false,
             // Prevents execution of native SQL statements. Extensions should set this to true.
@@ -154,7 +137,7 @@ OdfeSqlOdbcPBIConnectorImpl = (Host as text, optional Port as number) as table =
 OnOdbcError = (errorRecord as record) =>
     let
         ErrorMessage = errorRecord[Message],
-        ConnectionHostPort = errorRecord[Detail][DataSourcePath], 
+        ConnectionServer = errorRecord[Detail][DataSourcePath], 
 
         IsDriverNotInstalled = Text.Contains(ErrorMessage, "doesn't correspond to an installed ODBC driver"),
 
@@ -168,7 +151,7 @@ OnOdbcError = (errorRecord as record) =>
         if IsDriverNotInstalled then
             error Error.Record("", "The Open Distro For Elasticsearch SQL ODBC driver is not installed. Please install the driver")
         else if IsHostUnreachable then 
-            error Error.Record("", "Couldn't reach server. Please double-check the host, port and auth.")
+            error Error.Record("", "Couldn't reach server. Please double-check the server and auth. " & ConnectionServer)
         else 
             error errorRecord;
 
@@ -178,10 +161,9 @@ OdfeSqlOdbcPBIConnector = [
     TestConnection = (dataSourcePath) => 
         let
             json = Json.Document(dataSourcePath),
-            Host = json[Host],
-            Port = json[Port]
+            Server = json[Server]
         in
-            { "OdfeSqlOdbcPBIConnector.Contents", Host, Port },
+            { "OdfeSqlOdbcPBIConnector.Contents", Server },
 
     // Authentication modes
     Authentication = [

diff --git a/sql-odbc/src/PowerBIConnector/bin/Release/OdfeSqlOdbcPBIConnector.mez b/sql-odbc/src/PowerBIConnector/bin/Release/OdfeSqlOdbcPBIConnector.mez