Package validations can be made much quicker with a "queue-back"

[joelverhagen]

Downstream validators can queue back to orchestrator when they are done. This can save minutes. From testing on DEV, it can save on average roughly 2 minutes and 30 seconds per initial package validation.

1. Apply Validation DB migration.
2. Deploy orchestrators that handle the new message type.
   • Deployment for week of August 5, 2019 #7394
3. Deploy symbol validator and symbol ingester.
4. Turn on the feature flag.
5. E2E tests and manual testing of symbol validation.
6. Turn off the feature flag.
7. Deploy revalidate certificate job.
8. Deploy package validators.
9. Deploy scan and sign submitter.
10. Enable feature flag monitoring

Results

Over the past 7 days (4 days with queue-back, 3 days without queue-back), this is the improvement of validation times at a variety of percentiles. In short, we shaved off 2 minutes of validation time!

Snapshot on 9/24

jobName	p50	p90	p95	p99
Validation.SymbolsOrchestrator	43.53355	134.0543	165.6078	190.0541
Validation.Orchestrator	129.1221	158.1307	184.3262	216.7317

This is the P95 validation time, per 6 hours over the past 7 days.

Snapshot on 10/4

jobName	p50	p90	p95	p99
Validation.Orchestrator	128.6116	147.2313	214.5644	405.1642
Validation.SymbolsOrchestrator	82.77271	131.3943	120.9216	63.7498

Queries

let enabledData = todatetime('2019-09-19T23:11:46.558');
let summary = materialize (
    customMetrics
    | where timestamp > ago(7d)
    | where name == "Orchestrator.TotalValidationDurationSeconds"
    | extend instanceName = tostring(customDimensions.InstanceName)
    | extend jobName = iff(instanceName startswith "Validation.Orchestrator", "Validation.Orchestrator", "Validation.SymbolsOrchestrator")
    | extend hasQueueBack = timestamp > enabledData
    | summarize avg(value), percentiles(value, 50, 90, 95, 99) by jobName, hasQueueBack
);
summary
| where hasQueueBack == false
| join (
   summary
   | where hasQueueBack == true
) on jobName
| project jobName,
          p50 = percentile_value_50 - percentile_value_501,
          p90 = percentile_value_90 - percentile_value_901,
          p95 = percentile_value_95 - percentile_value_951,
          p99 = percentile_value_99 - percentile_value_991;

let enabledData = todatetime('2019-09-19T23:11:46.558');
let data = materialize(
    customMetrics
    | where timestamp > ago(7d)
    | where name == "Orchestrator.TotalValidationDurationSeconds"
    | extend instanceName = tostring(customDimensions.InstanceName)
    | extend jobName = iff(instanceName startswith "Validation.Orchestrator", "Validation.Orchestrator", "Validation.SymbolsOrchestrator")
    | extend hasQueueBack = timestamp > enabledData
    | summarize value = percentile(value, 95) by bin(timestamp, 6h), jobName, hasQueueBack
);
data
| where jobName == "Validation.Orchestrator"
| project timestamp, Validation_Orchestrator = value
| join (
    data
    | where jobName == "Validation.SymbolsOrchestrator"
    | project timestamp, Validation_SymbolsOrchestrator = value
) on timestamp
| project-away timestamp1
| extend zero = 0
| render timechart