Enhance summary output to show success percentages as well as failures

[jkodroff]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Today, the summary output from running tfgen looks like this:

General metrics:
	1241 total resources containing 13068 total inputs.
	511 total functions.
	0 entities are missing docs entirely because they could not be found in the upstream provider.

Description metrics:
	0 entity descriptions contained an <elided> reference and were dropped, including examples.
	0 entity descriptions contained an <elided> reference and were dropped, but examples were preserved.

Example conversion metrics:
	95 HCL examples failed to convert in all languages
	59 HCL examples were converted in at least one language but failed to convert to TypeScript
	59 HCL examples were converted in at least one language but failed to convert to Python
	59 HCL examples were converted in at least one language but failed to convert to Go
	59 HCL examples were converted in at least one language but failed to convert to C#
	73 entity document sections contained unexpected HCL code snippets. Examples will be converted, but may not display correctly in the registry, e.g. lacking tabs.

Argument metrics:
	9594 argument descriptions were parsed from the upstream docs
	196 top-level input property descriptions came from an upstream attribute (as opposed to an argument). Nested arguments are not included in this count.
	0 arguments contained an <elided> reference and had their descriptions dropped.
	0 nested arguments contained an <elided> reference and had their descriptions dropped.
	542 of 13068 resource inputs (4.15%) are missing descriptions in the schema

Attribute metrics:
	0 attributes contained an <elided> reference and had their descriptions dropped.

It would be better if each line showed "X/Y (Z%)". By giving the total number and the success rate, we'll have a clearer picture of how any changes in either the bridge or the provider code affect conversion, drop rates, etc.

This might also be a good opportunity to unify the summary output code with the JSON file that gives very detailed conversion info. FYI, our build processes either used to or still do upload that JSON file to an S3 bucket for the purpose of keeping metrics on historical conversion rates. It's not clear whether Pulumi is still actively using those metrics. There's duplication of functionality between the summary code I enhanced and the JSON output code.

The ideal solution could be some sort of event-centric logging that would allow us to capture each relevant event, stream those events to wherever we want (stdout, file), and also aggregate/transform them however we want in Go code (e.g. being able to do logging of the events, but also easily aggregate the totals, which we could slice and dice by module, resource, etc.). This solution could be overkill, but it would give us maximum flexibility for observability, which was a huge challenge in tfgen when I worked on it. My hunch is that this investment would pay off in the (possibly very) long run.

Affected area/feature

[t0yv0]

Sounds good. There's already code present to emit better summary activated by COVERAGE_OUTPUT_DIR=somewhere perhaps we could make that show.

FYI, our build processes either used to or still do upload that JSON file to an S3 bucket for the purpose of keeping metrics on historical conversion rates.

Yes I found this:

    - name: Upload coverage data to S3
      run: >-
        summaryName="${PROVIDER}_summary_$(date +"%Y-%m-%d_%H-%M-%S").json"

        s3FullURI="s3://${{ secrets.S3_COVERAGE_BUCKET_NAME }}/summaries/${summaryName}"

        aws s3 cp "${{ env.COVERAGE_OUTPUT_DIR }}/summary.json" "${s3FullURI}" --acl bucket-owner-full-control

Digging up instructions.

[jkodroff]

@t0yv0 If you know of an embeddable event-driven logging library in Go that can do the stuff I described, I'm curious to see what's out there. When I worked on tfgen, I did a survey of all the popular logging tools but I didn't find anything that did what I wanted. I basically wanted something like https://vector.dev/, which can do transforms on structured logs, but Vector runs as an external service whereas we would want something that can be embedded. If Vector has an automation API equivalent, that could be worth a look.

[t0yv0]

There is https://go.dev/blog/slog now but I don't know that this issue needs it, whatever's in place is fine we can just print better summaries.