Metrics readiness

[Amzani]

Related to #1

• Time to first API Design
• System errors
• Validation errors
• Time to fix a validation error
• Number of created AsyncAPI files
• AsyncAPI 3.0x Adoption

[Amzani]

/progress 10

Checking the metrics we collect

[Amzani]

/progress 15

We have Number of created AsyncAPI files and AsyncAPI 3.0x Adoption

In NewRelic

AsyncAPI 3.0x Adoption

SELECT count(1) FROM Metric WHERE metricName = 'asyncapi_adoption.action.finished' AND action = 'validate' AND  asyncapi_version = '3.0.0' SINCE last week

Number of created AsyncAPI files

SELECT count(1) FROM Metric WHERE metricName = 'asyncapi_adoption.action.finished' AND action = 'new' SINCE last week

[Amzani]

/progress 20

SELECT count(1) doesn't work as expected.

SELECT sum(asyncapi_adoption.action.finished) as 'Asyncapi V3 adoption'
FROM Metric 
WHERE action = 'validate' 
AND  asyncapi_version = '3.0.0'

SELECT sum(asyncapi_adoption.action.finished) as 'Created AsyncAPI files'
FROM Metric 
WHERE action = 'new:file'

Dashboard: https://onenr.io/0Vwga07k8jJ

[Amzani]

/progress 25

For Time to first API Design we expect to add two metadata in newRelic Metric table :

• user -> UUID v4
• source -> SHA1(FileName)

e.g

SELECT filter(latest(endTimestamp), WHERE action = 'generate:fromTemplate' AND source is not null AND success = true)/1000 - filter(latest(endTimestamp), WHERE action = 'new:file' AND source is not null AND success = true)/1000 AS 'TS'
FROM Metric FACET source, user

Then compute averages, median...

[Amzani]

/progress 100

System errors

SELECT sum(asyncapi_adoption.action.finished) as 'System errors'
FROM Metric
WHERE success = false

Validation errors

SELECT sum(asyncapi_adoption.action.finished) as 'Validation Errors'
FROM Metric
WHERE action = 'validate'
AND validation_result='invalid'

Time to fix a validation error

SELECT filter(latest(endTimestamp), WHERE action = 'validate' AND validation_result ='valid' AND source is not null AND success = true)/1000 - filter(latest(endTimestamp), WHERE action = 'validate' AND validation_result ='invalid' AND source is not null AND success = true)/1000 AS 'TS'
FROM Metric FACET source, user

Then compute averages, median...

[peter-rr]

SELECT sum(asyncapi_adoption.action.finished) as 'Created AsyncAPI files'
FROM Metric 
WHERE action = 'new:file'

@Amzani
I've realised about these improvements about Number of created AsyncAPI files:

• We should also consider WHERE action = 'new' since that command generates a new file too.
• We need to include metrics collection in asyncapi new command. So far, we've only covered the following commands: validate, convert, optimize, generate:fromTemplate and bundle.