Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Terminal is unable to create the iceberg table, when Kerberos authentication is enabled #1996

Closed
2 tasks done
Tracked by #1930
FWLamb opened this issue Sep 19, 2023 · 5 comments · Fixed by #2014
Closed
2 tasks done
Tracked by #1930

[Bug]: Terminal is unable to create the iceberg table, when Kerberos authentication is enabled #1996

FWLamb opened this issue Sep 19, 2023 · 5 comments · Fixed by #2014
Labels
type:bug Something isn't working

Comments

@FWLamb
Copy link
Contributor

FWLamb commented Sep 19, 2023 •
edited
Loading

What happened?

When I want to create an iceberg table in Terminal, it shows that the file cannot be deleted. In fact, this is the first time a table has been created, and the backend log shows that Kerberos authentication has failed.
e6db387f760cd82991153d7501922e9
6ef0d316ccbdc0761905a4dbaa5b06d

catalog config:
cf1596e1a4425eeba86d38c7b854a00

Affects Versions

0.5.0

What engines are you seeing the problem on?

No response

How to reproduce

No response

Relevant log output

2023-09-19 09:46:31,581 WARN [JettyServerThreadPool-44] [org.apache.hadoop.ipc.Client] [] - Exception encountered while connecting to the server 
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818) ~[hadoop-common-3.2.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1452) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy66.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:644) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy67.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1612) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:949) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:946) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:956) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:108) ~[iceberg-core-1.3.0.jar:?]
	at org.apache.iceberg.io.ResolvingFileIO.deleteFile(ResolvingFileIO.java:83) ~[iceberg-core-1.3.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.lambda$createTable$9(IcebergRestCatalogService.java:305) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.lambda$handleNamespace$17(IcebergRestCatalogService.java:447) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.handleCatalog(IcebergRestCatalogService.java:436) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.handleNamespace(IcebergRestCatalogService.java:443) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.createTable(IcebergRestCatalogService.java:265) ~[amoro-ams-server-0.5.0.jar:?]
	at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:98) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-10$lambda-9(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-10$lambda-9(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89) ~[javalin-4.6.1.jar:4.6.1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
	at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58) ~[javalin-4.6.1.jar:4.6.1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[javalin-4.6.1.jar:4.6.1]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]
2023-09-19 09:46:31,581 WARN [JettyServerThreadPool-44] [com.netease.arctic.server.IcebergRestCatalogService] [] - InternalServer Error
org.apache.iceberg.exceptions.RuntimeIOException: Failed to delete file: hdfs://hhht-ywglxt-bfdsj-srv04:8020/user/amoro/iceberg_catalog/db/user/metadata/v00001-473f0e65-1a89-4d3f-a229-6d2bca8b9d70.metadata.json
	at org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:110) ~[iceberg-core-1.3.0.jar:?]
	at org.apache.iceberg.io.ResolvingFileIO.deleteFile(ResolvingFileIO.java:83) ~[iceberg-core-1.3.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.lambda$createTable$9(IcebergRestCatalogService.java:305) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.lambda$handleNamespace$17(IcebergRestCatalogService.java:447) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.handleCatalog(IcebergRestCatalogService.java:436) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.handleNamespace(IcebergRestCatalogService.java:443) ~[amoro-ams-server-0.5.0.jar:?]
	at com.netease.arctic.server.IcebergRestCatalogService.createTable(IcebergRestCatalogService.java:265) ~[amoro-ams-server-0.5.0.jar:?]
	at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:98) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-10$lambda-9(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-10$lambda-9(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:112) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) ~[?:1.8.0_141]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_141]
	at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.1.jar:4.6.1]
	at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89) ~[javalin-4.6.1.jar:4.6.1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
	at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58) ~[javalin-4.6.1.jar:4.6.1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[javalin-4.6.1.jar:4.6.1]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.46.v20220331.jar:9.4.46.v20220331]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]
Caused by: java.io.IOException: DestHost:destPort hhht-ywglxt-bfdsj-srv04:8020 , LocalHost:localPort HHHT-YWGLXT-BFDSJ-SRV06/10.2.68.62:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:836) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:811) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1508) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy66.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:644) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy67.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1612) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:949) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:946) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:956) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:108) ~[iceberg-core-1.3.0.jar:?]
	... 67 more
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:778) ~[hadoop-common-3.2.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:732) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:835) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1452) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy66.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:644) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy67.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1612) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:949) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:946) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:956) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:108) ~[iceberg-core-1.3.0.jar:?]
	... 67 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818) ~[hadoop-common-3.2.2.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1452) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy66.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:644) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[hadoop-common-3.2.2.jar:?]
	at com.sun.proxy.$Proxy67.delete(Unknown Source) ~[?:?]
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1612) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:949) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:946) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.2.2.jar:?]
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:956) ~[hadoop-hdfs-client-3.2.2.jar:?]
	at org.apache.iceberg.hadoop.HadoopFileIO.deleteFile(HadoopFileIO.java:108) ~[iceberg-core-1.3.0.jar:?]
	... 67 more

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct
@FWLamb FWLamb added the type:bug Something isn't working label Sep 19, 2023
@FWLamb
Copy link
Contributor Author

FWLamb commented Sep 19, 2023

When using an external catalog with hive metastore, data can be written to hdfs normally, and Kerberos is also enabled.
image

@baiyangtx
Copy link
Contributor

The problem is that the IcebergRestCatalogService directly uses the CatalogUtil.loadFileIO method of Iceberg to create a FileIO object, which works well on S3 but requires process-level authentication in a Kerberos environment.

The problem can be easily fixed by wrapping the object returned by IcebergTableUtil.newIcebergFileIo method with ArcticFileIOs.buildAdaptIcebergFileIO. Would you like to fix this problem?

@FWLamb
Copy link
Contributor Author

FWLamb commented Sep 20, 2023

The problem is that the IcebergRestCatalogService directly uses the CatalogUtil.loadFileIO method of Iceberg to create a FileIO object, which works well on S3 but requires process-level authentication in a Kerberos environment.

The problem can be easily fixed by wrapping the object returned by IcebergTableUtil.newIcebergFileIo method with ArcticFileIOs.buildAdaptIcebergFileIO. Would you like to fix this problem?

I will try it.

@FWLamb
Copy link
Contributor Author

FWLamb commented Sep 20, 2023 •
edited
Loading

Hello, how do I verify the correctness of the code after I have modified it? @baiyangtx

@FWLamb FWLamb mentioned this issue Sep 21, 2023
Merged
3 tasks
@baiyangtx
Copy link
Contributor

Hello, how do I verify the correctness of the code after I have modified it? @baiyangtx

After the fix, the Files page should open normally when the Internal Catalog with Kerberos is registered on the page. Then it should be possible to successfully create a table through the terminal.

@wangtaohz wangtaohz mentioned this issue Sep 25, 2023
Closed
56 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[AMORO-1996] Fix iceberg format on hdfs not support kerberos FWLamb/amoro
2 participants
@baiyangtx @FWLamb