Support array of objects in YAML property files #216

javahippie · 2019-12-22T10:42:37Z

Description

I am using Smallrye Config via Quarkus 1.1.0.

Using arrays in application.yaml currently works fine for single values like this:

de:
  javahippie:
    mpadmin:
      instances:
        - "https://bing.com"
        - "https://www.google.com"

I can access the property without any additional configuration as a list of strings. I was curious if it was possible to write my own converter for a list of objects, as I need more data in my application than just the urls, and tried this:

de:
  javahippie:
    mpadmin:
      instances:
        -
          name: "Bing"
          uri: "https://bing.com"
        -
          name: "Google"
          uri: "https://www.google.com"

This is valid YAML and something that works in Spring Boot, when using a YAML as a property file. However, when I try to run Quarkus dev mode, it throws an exception:

11:30:37,337 ERROR [io.qua.dev.DevModeMain] Failed to start Quarkus: java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649)
	at java.lang.StringBuilder.append(StringBuilder.java:202)
	at io.smallrye.config.source.yaml.YamlConfigSource.escape(YamlConfigSource.java:146)
	at io.smallrye.config.source.yaml.YamlConfigSource.appendEscaped(YamlConfigSource.java:151)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:118)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)
	at io.smallrye.config.source.yaml.YamlConfigSource.populateFromEntryNode(YamlConfigSource.java:119)

11:30:37,343 INFO  [io.qua.dev.DevModeMain] Attempting to start hot replacement endpoint to recover from previous Quarkus startup failure
11:30:37,347 ERROR [io.qua.dev.DevModeMain] Failed to start quarkus: java.lang.IllegalStateException: No configuration is available for this class loader
	at io.smallrye.config.SmallRyeConfigProviderResolver.getConfig(SmallRyeConfigProviderResolver.java:89)
	at io.smallrye.config.SmallRyeConfigProviderResolver.getConfig(SmallRyeConfigProviderResolver.java:74)
	at org.eclipse.microprofile.config.ConfigProvider.getConfig(ConfigProvider.java:93)
	at io.quarkus.runtime.configuration.ConfigInstantiator.handleObject(ConfigInstantiator.java:45)
	at io.quarkus.vertx.http.runtime.VertxHttpRecorder.startServerAfterFailedStart(VertxHttpRecorder.java:114)
	at io.quarkus.vertx.http.deployment.devmode.VertxHotReplacementSetup.handleFailedInitialStart(VertxHotReplacementSetup.java:30)
	at io.quarkus.dev.RuntimeUpdatesProcessor.startupFailed(RuntimeUpdatesProcessor.java:451)
	at io.quarkus.dev.DevModeMain.doStart(DevModeMain.java:192)
	at io.quarkus.dev.DevModeMain.start(DevModeMain.java:96)

Would it be possible to implement a feature like this with a MicroProfile Config source? If no custom converter is provided, this could be interpreted as a List<Map<String, String>> as default?

The text was updated successfully, but these errors were encountered:

dmlloyd · 2019-12-22T13:11:13Z

It certainly isn't expected for it to fail in this manner. Since MP Config deals in strings, the value is meant to to be converted to a list of key-value pairs. We don't have any map converters yet (just because there hasn't been a use case yet).

radcortez · 2020-03-09T15:46:16Z

This should fix the Stackoverflow issue:
#252

Regarding support for Arrays of Objects or Maps is another story. I recommend that we open a separate issue for that.

#216 - Fixed stackoverflow issue.

javahippie · 2020-03-10T07:40:26Z

Thank you very much!

radcortez · 2020-03-10T09:14:54Z

Actually, I'll leave this one open and create a new issue for the Stackoverflow problem.

rolandjohann · 2020-03-27T10:35:13Z

@dmlloyd how can we register custom converters at quarkus? I currently have an use case where I need to decode a collection of objects via quarkus.

EDIT: Eventually I was puzzled by the distributed docs and looked at the wrong place. The answer is there: https://quarkus.io/guides/config#custom-configuration

radcortez · 2020-03-27T10:44:54Z

Hi @rolandjohann, you can register converters in Quarkus per the definition of MP Config. Please check here on how to do it:
https://quarkus.io/guides/config#custom-configuration-converters

rolandjohann · 2020-03-27T11:04:42Z

@radcortez thanks for the quick answer. Remembered that section previously. The converter turns out to be pretty useless as the string representation contains key value pairs but the delimiter is missing. When using different levels of nesting it gets more confusing. Is it possible to implement confiuration schemas like that posted by op?

de:
  javahippie:
    mpadmin:
      instances:
        -
          name: "Bing"
          uri: "https://bing.com"
        -
          name: "Google"
          uri: "https://www.google.com"

radcortez · 2020-03-27T11:07:53Z

Yes, that is something that we need to figure out. I'll try to have a look on how we can implement this.

Also, we would love help if you want to contribute that piece :)

rolandjohann · 2020-03-27T11:10:28Z

Sure, but I need some guidance here as the internals of smallrye are pretty unknown to me

radcortez · 2020-03-27T11:42:47Z

Thanks! Try to have a look into the project and I'll help as much as I can.

Probably, you should start by looking into:
https://github.com/smallrye/smallrye-config/blob/58f66dadcf217b91d22d6393fe0cbd9f63984f83/sources/yaml/src/main/java/io/smallrye/config/source/yaml/YamlConfigSource.java

dmlloyd · 2020-03-27T12:08:16Z

@radcortez thanks for the quick answer. Remembered that section previously. The converter turns out to be pretty useless as the string representation contains key value pairs but the delimiter is missing.

What do you mean by "missing"?

fabmars · 2020-06-07T10:04:53Z

He means that:

    admin:
      users:
        -
          email: "[email protected]"
          username: "joe"
          password: "123456"
          roles:
            - "Moderator"
            - "Admin"
        -
          email: "[email protected]"
          username: "jack"
          password: "654321"
          roles:
            - "Moderator"

When mapped to List<String>, gives [[email protected]=joepassword=123456roles=Moderator\, Admin, [email protected]=jackpassword=654321roles=Moderator]
So, 2 values in the list of strings, no separator between each value and the next key.

Worse, as you can see in my example, I used a nested list, and if we now map that config tree to List<User> with a custom converter, it's called 3 times (not two) with the following values.

"email=[email protected]=joepassword=123456roles=Moderator"
"Admin"
"email=[email protected]=jackpassword=654321roles=Moderator"

This wasn't unexpected though.
That was just for the sake of giving an example to @rolandjohann 's statement and answer the question above.

dmlloyd · 2020-06-08T13:43:07Z

That is definitely a bug... the code is supposed to insert the right number of escapes based on the nesting level, but there must be a logic error that occurs when going from list to map.

fabmars · 2020-06-08T15:55:30Z

OK I'm definitely willing to take a look when I'm done with my current project...say within a week.

radcortez · 2020-06-08T16:16:29Z

Cool. Looking forward for your PR. Thanks! :)

radcortez · 2020-06-18T15:07:23Z

I have been looking into this and I think we should change a little bit on how we do things.

When we have a simple case, where the yaml end nodes can be translated 1 to 1 to property keys everything is ok. The issue is when we mix Maps and Lists that don't translate directly to a single unique key.

In the case, the value was going into a single key with several escapes that required the consumer to parse. Well, what if we just keep it in the original format which is easier to convert? :) Also we could add additional indexed keys that allow to retrieve values from lists.

Please have a look in here #335 and let me know any feedback. @fabmars @javahippie @rolandjohann

radcortez added this to the 1.7.0 milestone Mar 3, 2020

radcortez added a commit to radcortez/smallrye-config that referenced this issue Mar 9, 2020

smallrye#216 - Fixed stackoverflow issue.

58f66da

radcortez added a commit that referenced this issue Mar 9, 2020

Merge pull request #252 from radcortez/#216

e702372

#216 - Fixed stackoverflow issue.

radcortez modified the milestones: 1.7.0, 2.0.0 (MicroProfile Config 2.0) Mar 10, 2020

dmlloyd mentioned this issue Mar 31, 2020

Make our Converter Interface Converter<S,T> smallrye/smallrye-converters#5

Open

radcortez mentioned this issue Jul 9, 2020

Devmode, Regression in 1.6.0: Cannot override property from profile via system property quarkusio/quarkus#10570

Closed

radcortez modified the milestones: 2.0.0 (MicroProfile Config 2.0), 1.8.2 Jul 15, 2020

radcortez closed this as completed Jul 15, 2020

radcortez mentioned this issue Sep 11, 2020

Parse yaml to List<POJO> #391

Closed

radcortez mentioned this issue Nov 12, 2020

Quarkus YAML configuration keys are implicitly escaped quarkusio/quarkus#11744

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support array of objects in YAML property files #216

Support array of objects in YAML property files #216

javahippie commented Dec 22, 2019

dmlloyd commented Dec 22, 2019 •

edited

Loading

radcortez commented Mar 9, 2020

javahippie commented Mar 10, 2020

radcortez commented Mar 10, 2020

rolandjohann commented Mar 27, 2020 •

edited

Loading

radcortez commented Mar 27, 2020

rolandjohann commented Mar 27, 2020

radcortez commented Mar 27, 2020

rolandjohann commented Mar 27, 2020

radcortez commented Mar 27, 2020

dmlloyd commented Mar 27, 2020

fabmars commented Jun 7, 2020 •

edited

Loading

dmlloyd commented Jun 8, 2020

fabmars commented Jun 8, 2020

radcortez commented Jun 8, 2020

radcortez commented Jun 18, 2020

Support array of objects in YAML property files #216

Support array of objects in YAML property files #216

Comments

javahippie commented Dec 22, 2019

Description

dmlloyd commented Dec 22, 2019 • edited Loading

radcortez commented Mar 9, 2020

javahippie commented Mar 10, 2020

radcortez commented Mar 10, 2020

rolandjohann commented Mar 27, 2020 • edited Loading

radcortez commented Mar 27, 2020

rolandjohann commented Mar 27, 2020

radcortez commented Mar 27, 2020

rolandjohann commented Mar 27, 2020

radcortez commented Mar 27, 2020

dmlloyd commented Mar 27, 2020

fabmars commented Jun 7, 2020 • edited Loading

dmlloyd commented Jun 8, 2020

fabmars commented Jun 8, 2020

radcortez commented Jun 8, 2020

radcortez commented Jun 18, 2020

dmlloyd commented Dec 22, 2019 •

edited

Loading

rolandjohann commented Mar 27, 2020 •

edited

Loading

fabmars commented Jun 7, 2020 •

edited

Loading