Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug report] Can not operate STRUCT column of iceberg table #1313

Closed
TEOTEO520 opened this issue Jan 3, 2024 · 3 comments · Fixed by #1721
Closed

[Bug report] Can not operate STRUCT column of iceberg table #1313

TEOTEO520 opened this issue Jan 3, 2024 · 3 comments · Fixed by #1721
Assignees
Labels
bug Something isn't working

Comments

@TEOTEO520
Copy link
Contributor

Describe what's wrong

can not operate STRUCT column of iceberg table :
example: create iceberg
{
"name":"gravitino_table_create_iceberg_test_02",
"columns":[
{
"name":"id",
"type":"integer",
"nullable":true,
"comment":"Id of the user"
},
{
"name":"name",
"type":"string",
"nullable":true,
"comment":"Name of the user"
},
{
"name":"struct_type",
"type":{
"type": "struct",
"fields":[
{
"name":"street",
"type":"string",
"nullable":true,
"comment":"11"
},
{
"name":"state",
"type":"string",
"nullable":true,
"comment":"11"
}
]
},
"nullable":true,
"comment":"Name of the user"
}
],
"comment":"Create a new Table",
"properties":{
"format":"ORC"
}
}

Error message and/or stacktrace

{
"code": 1002,
"type": "RuntimeException",
"message": "Failed to operate object [gravitino_table_create_iceberg_test_02] operation [CREATE] under [test_iceberg_bdp], reason [ClassCastException]",
"stack": [
"java.lang.ClassCastException: com.datastrato.gravitino.rel.types.Types$StructType cannot be cast to com.datastrato.gravitino.rel.types.Type$PrimitiveType",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ToIcebergTypeVisitor.visit(ToIcebergTypeVisitor.java:56)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ToIcebergTypeVisitor.visit(ToIcebergTypeVisitor.java:35)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.converter.ConvertUtil.toIcebergSchema(ConvertUtil.java:23)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergTable.toCreateTableRequest(IcebergTable.java:47)",
"\tat com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.createTable(IcebergCatalogOperations.java:526)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$null$30(CatalogOperationDispatcher.java:431)",
"\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithTableOps$1(CatalogManager.java:96)",
"\tat com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:69)",
"\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithTableOps(CatalogManager.java:91)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$createTable$31(CatalogOperationDispatcher.java:429)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.doWithCatalog(CatalogOperationDispatcher.java:698)",
"\tat com.datastrato.gravitino.catalog.CatalogOperationDispatcher.createTable(CatalogOperationDispatcher.java:426)",
"\tat com.datastrato.gravitino.server.web.rest.TableOperations.createTable(TableOperations.java:88)",
"\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
"\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
"\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
"\tat java.lang.reflect.Method.invoke(Method.java:498)",
"\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)",
"\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)",
"\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)",
"\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)",
"\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
"\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
"\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
"\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
"\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)",
"\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)",
"\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
"\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)",
"\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
"\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)",
"\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)",
"\tat com.datastrato.gravitino.server.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:57)",
"\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
"\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
"\tat com.datastrato.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:107)",
"\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)",
"\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)",
"\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)",
"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)",
"\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
"\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)",
"\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
"\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)",
"\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
"\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)",
"\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
"\tat org.eclipse.jetty.server.Server.handle(Server.java:516)",
"\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)",
"\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)",
"\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)",
"\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)",
"\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
"\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
"\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)",
"\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)",
"\tat java.lang.Thread.run(Thread.java:748)"
]
}

How to reproduce

create table like the expamle

Additional context

ADD JUDGEMENT:

class:ToIcebergTypeVisitor

public static T visit(Type type, ToIcebergTypeVisitor visitor) {
if (type instanceof Types.MapType) {
Types.MapType map = (Types.MapType) type;
return visitor.map(map, visit(map.keyType(), visitor), visit(map.valueType(), visitor));
} else if (type instanceof Types.ListType) {
Types.ListType list = (Types.ListType) type;
return visitor.array(list, visit(list.elementType(), visitor));
} else {
return visitor.atomic((Type.PrimitiveType) type);
}
}

@TEOTEO520 TEOTEO520 changed the title [Bug report] [Bug report] Can not operate STRUCT column of iceberg table Jan 3, 2024
@qqqttt123
Copy link
Contributor

cc @mchades

@mchades
Copy link
Contributor

mchades commented Jan 3, 2024

As you described in Additional context, this issue is related to the class ToIcebergTypeVisitor, where it needs to include a check for Types.StructType. Are you interested in fixing this issue? If so, I will assign this issue to you.

@TEOTEO520
Copy link
Contributor Author

OK,it may cost me one week to finish it

@jerryshao jerryshao added this to the Gravitino 0.4.0 milestone Jan 16, 2024
@jerryshao jerryshao added the bug Something isn't working label Jan 25, 2024
FANNG1 pushed a commit that referenced this issue Feb 5, 2024
…berg (#1721)

### What changes were proposed in this pull request?

add  struct column support for iceberg 

### Why are the changes needed?

Fix: #1313
Fix: #1471 

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

UT added

---------

Co-authored-by: teo <[email protected]>
github-actions bot pushed a commit that referenced this issue Feb 5, 2024
…berg (#1721)

### What changes were proposed in this pull request?

add  struct column support for iceberg 

### Why are the changes needed?

Fix: #1313
Fix: #1471 

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

UT added

---------

Co-authored-by: teo <[email protected]>
yuqi1129 pushed a commit that referenced this issue Feb 6, 2024
### What changes were proposed in this pull request?

add  struct column support for iceberg 

### Why are the changes needed?

Fix: #1313
Fix: #1471 

### Does this PR introduce _any_ user-facing change?

no

### How was this patch tested?

UT added

Co-authored-by: Tianhang Li <[email protected]>
Co-authored-by: teo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants