Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: Connection to Data Warehouse SQL Endpoint in Microsoft Fabric Fails with "Read: EOF" Error #230

Open
jaume-pladevall-itk opened this issue Dec 4, 2024 · 13 comments

Comments

@jaume-pladevall-itk
Copy link

We implemented a connection to a Data Warehouse SQL endpoint in Microsoft Fabric, but it suddenly stopped working and began returning the error message “Read: EOF” without any changes to our code.
We use the following code to connect to the database:

import (
	"database/sql"
	"fmt"

	"github.com/microsoft/go-mssqldb/azuread"
)

...

connString := fmt.Sprintf("sqlserver://%s?database=%s&fedauth=ActiveDirectoryAzCli&connection+timeout=30", url, database)

db, err := sql.Open(azuread.DriverName, connString)
if err != nil {
    fmt.Println("Error creating connection pool: ", err.Error())
    return
}
defer db.Close()

err = db.Ping()
if err != nil {
    log.Println("Ping failed:", err)    
}

...

The code above returns the error "Ping failed: Read: EOF" where url has the following format: "long_mix_of_numbers_and_chars.datawarehouse.fabric.microsoft.com" and database is the name of the table we want to access inside the Data Warehouse.

We are using Go 1.22.0 and go-mssqldb version v1.7.2.

Upon debugging the go-mssqldb library, it appears that a first connection is established, as it enters the loginAckStruct case (line 1366 of the file tds.go). Then, it enters the condition sess.routedServer != "" (line 1393 of the file tds.go) and initiates another connection, which is the one that returns the "Read: EOF" error.

We also tried connecting to the SQL endpoint using SQL Server Management Studio (SSMS) and directly accessing the Data Warehouse from Microsoft Fabric. Both methods work successfully.

Has anyone else experienced the same issue? If so, could you share your insights or solutions? Additionally, if anyone has any clues as to why this might be happening or how to resolve it, your help would be greatly appreciated. Thank you!

@shueybubbles
Copy link
Collaborator

I think it's due to #226

@shueybubbles
Copy link
Collaborator

@jaume-pladevall-itk could you try building with the latest commit in main (#228 ) and see if it fixes your issue? I'd love a 3rd party validation before I push a 1.8 tag.

@jaume-pladevall-itk
Copy link
Author

@shueybubbles Now with the latest commit in main (#228) it works again. Thank you very much!

@flarco
Copy link

flarco commented Dec 18, 2024

Seems it's still happening when using fedauth=ActiveDirectoryAzCli. Please see slingdata-io/sling-cli#468 (comment)

@shueybubbles
Copy link
Collaborator

@flarco did your sling dev build upgrade go-mssqldb to v1.8?

@flarco
Copy link

flarco commented Dec 18, 2024

Yes, see here: https://github.com/slingdata-io/sling-cli/compare/v1.3.4 (commit eb7ff40)
A dev build is released right after each commit.
FYI @johnoscott can you run with --debug and share the stack trace here?

@shueybubbles
Copy link
Collaborator

FWIW if I use the latest go-sqlcmd based on go-mssqldb I can access a Fabric DW that's in our "daily" environment. I can use both -G and --authentication-method

sqlcmd - S <funkystringofcharacters>datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryAzCli
1> select @@version
2> go


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Microsoft Azure SQL Data Warehouse (RTM) - 12.0.2000.8
        Dec 12 2024 16:54:01
        Copyright (C) 2024 Microsoft Corporation

@johnoscott
Copy link

Here is the output with --debug

❯ ./sling --version                    
Version: 1.3.4.dev (2024-12-17)

 ./sling conns test FABRIC_WH2 --debug
2024-12-19 07:53:22 DBG closed "sqlserver" connection (conn-sqlserver-1L4)
fatal:
--- proc.go:271 main ---
--- sling_cli.go:474 main ---
--- sling_cli.go:510 cliInit ---
--- cli.go:286 CliProcess ---
--- sling_conns.go:180 processConns ---
--- connection_local.go:64 Test ---
~ could not test FABRIC_WH2
--- connection_discover.go:24 Test ---
~ could not connect to FABRIC_WH2
--- database_sqlserver.go:175 Connect ---
~ could not connect to database
--- database.go:663 Connect ---
Read: EOF

and my ~/.sling/.env file

  FABRIC_WH2: # DOESNT WORK !!!!
    type: sqlserver # also tried azure-synapse-analytics
    host: REDACTED.datawarehouse.fabric.microsoft.com
    user: REDACTED
    password: "REDACTED"
    database: warehouse2
    schema: src_customer
    fedauth: ActiveDirectoryServicePrincipal

@shueybubbles shueybubbles reopened this Dec 18, 2024
@shueybubbles
Copy link
Collaborator

shueybubbles commented Dec 18, 2024

@johnoscott is it possible you can attach a logger to the driver with the log level set to 0x8fff?
The driver logs will have output like this:

Dialing with protocol tcp
Returning connection from protocol tcp
Creating prelogin packet with connection id '75639738-CA26-4942-806A-17FAEA55CC09' and activity id '680A2E5D-56F5-BB4E-9713-02813DE08F15'
Starting federated authentication using ADAL
aid:680A2E5D-56F5-BB4E-9713-02813DE08F15 cid:75639738-CA26-4942-806A-17FAEA55CC09 - got token tokenFedAuthInfo

At minimum you'd be able to verify the driver is correctly adding a connection id and activity id to the prelogin packet, and you could give those guids to Microsoft support if you need them to help identify why the connection is being closed.

One way you could get this same information is use the newest release from github.com/microsoft/go-sqlcmd and pass it on the command line:

sqlcmd -S xxx.datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryDefault -Q "select @@version" --driver-logging-level 33279

@johnoscott
Copy link

johnoscott commented Dec 19, 2024

I am using macOS and I am expecting that authenticating with method ActiveDirectoryServicePrincipal will work (even though interactive authentication (which i am not using) is known not to work )

sqlcmd '-?'
Microsoft (R) SQL Server Command Line Tool
Version 18.4.0001.1 Linux

sqlcmd -S REDACTED.datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryServicePrincipal -d fabric -U REDACTED -P "REDACTED" -Q "select @@version" --driver-logging-level 33279

Sqlcmd: '-authentication-method': Unknown Option. Enter '-?' for help.

So I tried Windows 11

sqlcmd '-?'
sqlcmd: Install/Create/Query SQL Server, Azure SQL, and Tools
Version: v1.8.0

sqlcmd -S REDACTED.datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryServicePrincipal -d fabric -U REDACTED -P "REDACTED" -Q "select @@version" --driver-logging-level 33279
>> 
DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Starting federated authentication using ADAL
DRIVER:got token tokenLoginAck
DRIVER:got token tokenEnvChange
DRIVER:got token tokenDone
DRIVER:got DONE or DONEPROC status=0
DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Starting federated authentication using ADAL
DRIVER:got token tokenFedAuthInfo
DRIVER:BeginRead failed EOF
Read: EOF
Read: EOF

@johnoscott
Copy link

OK that macOS version was using sqlcmd from a package of mssql-tools, which explains the different version number. When i install sqlcmd directly i get ...

❯ sqlcmd '-?'
sqlcmd: Install/Create/Query SQL Server, Azure SQL, and Tools

Version: 1.8.0

sqlcmd -S REDACTED.datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryServicePrincipal -d fabric -U REDACTED -P "REDACTED" -Q "select @@version" --driver-logging-level 33279

DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Starting federated authentication using ADAL
DRIVER:got token tokenLoginAck
DRIVER:got token tokenEnvChange
DRIVER:got token tokenDone
DRIVER:got DONE or DONEPROC status=0
DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Starting federated authentication using ADAL
DRIVER:got token tokenFedAuthInfo
DRIVER:BeginRead failed EOF
Read: EOF
Read: EOF

@shueybubbles
Copy link
Collaborator

@johnoscott sorry I should have been more explicit - go-sqlcmd v1.8.2 still hasn't been marked as "latest" but it's the version that has the activity id and connection id fix.

@johnoscott
Copy link

OK it appears to be working with upcoming v1.8.2

sqlcmd '-?'
sqlcmd: Install/Create/Query SQL Server, Azure SQL, and Tools

Version: local-build 1.8.2

sqlcmd -S REDACTED.datawarehouse.fabric.microsoft.com --authentication-method ActiveDirectoryServicePrincipal -d fabric -U REDACTED -P "REDACTED" -Q "select @@version" --driver-logging-level 33279

DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Creating prelogin packet with connection id 'FD8ED07E-5F0C-3B40-888C-BDBA8232105A' and activity id '5206C7D1-1467-DD49-BE25-065036BA6BD2'
DRIVER:Starting federated authentication using ADAL
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:FD8ED07E-5F0C-3B40-888C-BDBA8232105A - got token tokenLoginAck
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:FD8ED07E-5F0C-3B40-888C-BDBA8232105A - got token tokenEnvChange
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:FD8ED07E-5F0C-3B40-888C-BDBA8232105A - got token tokenDone
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:FD8ED07E-5F0C-3B40-888C-BDBA8232105A - got DONE or DONEPROC status=0
DRIVER:Dialing with protocol tcp
DRIVER:Returning connection from protocol tcp
DRIVER:Creating prelogin packet with connection id '0067E946-CD28-8D42-BB2B-377560D94EE6' and activity id '5206C7D1-1467-DD49-BE25-065036BA6BD2'
DRIVER:Starting federated authentication using ADAL
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenFedAuthInfo
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenEnvChange
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenInfo
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got INFO 5701 Changed database context to 'warehouse2'.
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - Changed database context to 'warehouse2'.
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenEnvChange
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenEnvChange
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenInfo
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got INFO 5703 Changed language setting to us_english.
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - Changed language setting to us_english.
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenLoginAck
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenEnvChange
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenFeatureExtAck
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenDone
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got DONE or DONEPROC status=0
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - select @@version
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenColMetadata
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenRow
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got token tokenDone
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - got DONE or DONEPROC status=16
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - (Rows affected: 1)
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - Columns() token type:[]mssql.columnStruct
                                                                                                                                                                                                                                                                                                            
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - Next() token type:[]interface {}
Microsoft Azure SQL Data Warehouse (RTM) - 12.0.2000.8 
	Nov 11 2024 10:20:51 
	Copyright (C) 2024 Microsoft Corporation
                                                                                                                                                                                   
DRIVER:aid:5206C7D1-1467-DD49-BE25-065036BA6BD2 cid:0067E946-CD28-8D42-BB2B-377560D94EE6 - Next() token type:mssql.doneStruct

(1 row affected)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants