Fix package pattern in json schema configuration registry

[WangWeiLin-MV]

Fix microsoft/vcpkg/issues/39259 for package pattern with an optional trailing *.

Change

• Add definitions/reserved-name for reuse
  • Vcpkg reserved name by ^(core|default)$
  • Win32 filesystem reserved name by a loose pattern ^(con|prn|aux|nul|com[0-9]|lpt[0-9])(\\.[^.]+)*$, which include COM0 and LPT0. See documents
• Add definitions/package-pattern to fix optional trailing *
  • Pattern of ^([a-z0-9]+-)*([a-z0-9]+[*]?|[*])$ to match wildcard or ‘boost*’ or ‘boost-*’
• Replace definitions/port-name with definitions/identifier
  • Pattern of port-name matches 'dot-separated identifier'. No relevant documents found, remove this definition.
  • Pattern of identifier matches 'lowercase with digits and dashes'
• Append # to $schema
  • New version 7.4.0 doesn't handle JSON schema correctly PowerShell/PowerShell#20743 (comment)
• Add more unit tests
• Add platform independent json schema e2e test step in pr.yaml

Test

Schema vcpkg.schema.json validated all ports vcpkg.json with json-schema-tests-dir/vcpkg-ports-json.test.ps1.

Schema vcpkg-configuration.schema.json tested pass with json-schema-tests-dir/json-schema-bvt.test.ps1

[BillyONeal]

Format failures fixed in #1433

[WangWeiLin-MV]

Schema vcpkg-configuration.schema.json tested cases pass in PowerShell 7.4.2

param(
    [string]$VcpkgSrcDir = (Join-Path $PSScriptRoot '../..' | Resolve-Path),
    [string]$TempWorkDir = (New-Item -ItemType Directory Temp:/$(New-Guid)).FullName
)
$ErrorActionPreference = 'Stop'
# . $PSScriptRoot/../end-to-end-tests-prelude.ps1

$VcpkgJsonSchema = @{
    Artifact      = 'artifact.schema.json'
    Configuration = 'vcpkg-configuration.schema.json'
    Definitions   = 'vcpkg-schema-definitions.schema.json'
    Port          = 'vcpkg.schema.json'
}

# remove `$id` in schema for error 'Test-Json: Cannot parse the JSON schema.'
$VcpkgJsonSchema.Values
| ForEach-Object {
    Copy-Item -Path (Join-path $VcpkgSrcDir 'docs' $_) -Destination $TempWorkDir
    Join-Path $TempWorkDir $_
}
| ForEach-Object {
    (Get-Content -Raw -Path $_) -replace '(?s)\n  "\$id".+?\n', "`n"
    | Set-Content -NoNewline -Path $_
}
| Out-Null

[scriptblock]$_Gen__ = {
    param(
        [Parameter(Mandatory)][bool]$Expected,
        [Parameter(Mandatory)][string[]]$PackageArray,
        [string]$Baseline = '0' * 40
    )
    return @(
        $Expected,
        @{
            registries = @(
                [pscustomobject]@{
                    kind       = 'git'
                    repository = ''
                    baseline   = $Baseline
                    packages   = $PackageArray
                }
            )
        }
    )
}

# See src/vcpkg-test/registries.cpp "check valid package patterns"
@{
    $VcpkgJsonSchema.Configuration =
    @{
        'packages: ["a"]'                 = & $_Gen__ $true  @('a')
        'packages: empty'                 = & $_Gen__ $false [string[]]@('')
        'packages: blank'                 = & $_Gen__ $false @(' ')
        'packages: baseline 39'           = & $_Gen__ $false @('a')       ('0' * 39)
        'packages: hashtag ["*"]'         = & $_Gen__ $true  @('*')
        'packages: hashtag ["a*"]'        = & $_Gen__ $true  @('a*')
        'packages: hashtag ["*a"]'        = & $_Gen__ $false @('*a')
        'packages: hashtag ["b-*"]'       = & $_Gen__ $true  @('b-*')
        'packages: hashtag ["c-d-*"]'     = & $_Gen__ $true  @('c-d-*')
        'packages: hashtag dup ["a**"]'   = & $_Gen__ $false @('a**')
        'packages: hashtag dup ["b-**"]'  = & $_Gen__ $false @('b-**')
        'packages: hashtag dup ["c--*"]'  = & $_Gen__ $false @('c--*')
        'packages: hashtag dup ["d-*-*"]' = & $_Gen__ $false @('d-*-*')
        'packages: hashtag mid ["a*b"]'   = & $_Gen__ $false @('a*b')
        'packages: mix array ["a*","b"]'  = & $_Gen__ $true  @('a*', 'b')
        'packages: symbols ["a+"]'        = & $_Gen__ $false @('a+')
        'packages: symbols ["a?"]'        = & $_Gen__ $false @('a?')
    }
    $VcpkgJsonSchema.Port          =
    @{
        # test identifiers
        'port-name: "co"'                 = $true, @{name = 'co' }
        'port-name: "rapidjson"'          = $true, @{name = 'rapidjson' }
        'port-name: "boost-tuple"'        = $true, @{name = 'boost-tuple' }
        'port-name: "vcpkg-boost-helper"' = $true, @{name = 'vcpkg-boost-helper' }
        'port-name: "lpt"'                = $true, @{name = 'lpt' }
        'port-name: "com"'                = $true, @{name = 'com' }
        # reject invalid characters
        'port-name: ""'                   = $false, @{name = '' }
        'port-name: " "'                  = $false, @{name = ' ' }
        'port-name: "boost_tuple"'        = $false, @{name = 'boost_tuple' }
        'port-name: "boost.'              = $false, @{name = 'boost.' }
        'port-name: "boost.tuple"'        = $false, @{name = 'boost.tuple' }
        'port-name: "boost@1"'            = $false, @{name = 'boost@1' }
        'port-name: "boost#1"'            = $false, @{name = 'boost#1' }
        'port-name: "boost:x64-windows"'  = $false, @{name = 'boost:x64-windows' }
        # accept legacy
        'port-name: "all_modules"'        = $false, @{name = 'all_modules' } # removed in json-schema
        # reject reserved keywords
        'port-name: "prn"'                = $false, @{name = 'prn' }
        'port-name: "aux"'                = $false, @{name = 'aux' }
        'port-name: "nul"'                = $false, @{name = 'nul' }
        'port-name: "con"'                = $false, @{name = 'con' }
        'port-name: "core"'               = $false, @{name = 'core' }
        'port-name: "default"'            = $false, @{name = 'default' }
        'port-name: "lpt0"'               = $false, @{name = 'lpt0' }
        'port-name: "lpt9"'               = $false, @{name = 'lpt9' }
        'port-name: "com0"'               = $false, @{name = 'com0' }
        'port-name: "com9"'               = $false, @{name = 'com9' }
        # reject incomplete segments
        'port-name: "-a"'                 = $false, @{name = '-a' }
        'port-name: "a-"'                 = $false, @{name = 'a-' }
        'port-name: "a--"'                = $false, @{name = 'a--' }
        'port-name: "---"'                = $false, @{name = '---' }
    }
}.GetEnumerator()
| ForEach-Object {
    @{
        SchemaName = $_.Key
        JsonCases  = $_.Value.GetEnumerator() | ForEach-Object { @{Title = $_.Key; Expected = $_.Value[0]; Json = ConvertTo-Json -InputObject $_.Value[1] -Depth 5 -Compress } }
    }
}
| ForEach-Object {
    $_SchemaName = $_.SchemaName
    $_SchemaPath = Join-Path $TempWorkDir $_SchemaName
    $_.JsonCases | ForEach-Object {
        $_Title = $_.Title
        $_Expected = $_.Expected

        $_Actual = Test-Json -ea:0 -Json $_.Json -SchemaFile $_SchemaPath
        $_Result = $_.Expected -eq $_Actual ? 'Pass':'Fail'
        if ($_Result -eq 'Fail') {
            throw "$_SchemaName validate fail with $_Title, expected $_Expected"
        }
        [pscustomobject]@{
            SchemaName = $_SchemaName
            Title      = $_Title
            Expected   = $_Expected
            Actual     = $_Actual
            Result     = $_Result
        }
    }
}
| Sort-Object SchemaName, Title
| Format-Table

Remove-Item -Recurse -Force $TempWorkDir

Schema vcpkg.schema.json validated ports vcpkg.json pass in PowerShell 7.4.2

param(
    [string]$VcpkgSrcDir = (Join-Path $PSScriptRoot '../..' | Resolve-Path),
    [string]$VcpkgRegistryDir = $env:VCPKG_ROOT,
    [string]$TempWorkDir = (New-Item -ItemType Directory Temp:/$(New-Guid)).FullName
)
$ErrorActionPreference = 'Stop'
# . $PSScriptRoot/../end-to-end-tests-prelude.ps1

$VcpkgJsonSchema = @{
    Artifact      = 'artifact.schema.json'
    Configuration = 'vcpkg-configuration.schema.json'
    Definitions   = 'vcpkg-schema-definitions.schema.json'
    Port          = 'vcpkg.schema.json'
}

# remove `$id` in schema for error 'Test-Json: Cannot parse the JSON schema.'
$VcpkgJsonSchema.Values
| ForEach-Object {
    Copy-Item -Path (Join-path $VcpkgSrcDir 'docs' $_) -Destination $TempWorkDir
    Join-Path $TempWorkDir $_
}
| ForEach-Object {
    (Get-Content -Raw -Path $_) -replace '(?s)\n  "\$id".+?\n', "`n"
    | Set-Content -NoNewline -Path $_
}
| Out-Null

$VcpkgPortSchemaPath = Join-Path $TempWorkDir $VcpkgJsonSchema.Port

Get-ChildItem -Directory -Path (Join-Path $VcpkgRegistryDir 'ports')
| ForEach-Object -Parallel {
    $PortName = $_.Name
    $PortDir = $_.FullName
    $PortJsonPath = Join-Path $PortDir 'vcpkg.json'
    $Schema = $using:VcpkgPortSchemaPath

    $_Actual = Test-Json -ea:0 -LiteralPath $PortJsonPath -SchemaFile $Schema
    [pscustomobject]@{
        PortName = $PortName
        Actual   = $_Actual
    }
}
| ForEach-Object { Write-Host $_; $_ }
| Where-Object Actual -EQ $false
| ForEach-Object { Write-Error $_; $_ }

Remove-Item -Recurse -Force $TempWorkDir

[BillyONeal]

Add definitions/package-pattern to fix optional trailing *

• Pattern of ^[a-z0-9]+(-[a-z0-9]+)*(-?\*)*$ to match ‘boost*’ and ‘boost-*’

Is this actually accepted by the product code?

[BillyONeal]

• Pattern of port-name matches 'dot-separated identifier'. No relevant documents found, remove this definition.

This was done to give better intellisense behavior / descriptive text. But I think it's fine to remove it.

[WangWeiLin-MV]

match ‘boost*’ and ‘boost-*’

Is this actually accepted by the product code?

There are some places to parse the pattern:

• src/vcpkg/configuration.cpp#L1016 is_package_pattern
• src/vcpkg/registries.cpp#L1474 package_pattern_match

And there are some test cases covering it:

• src/vcpkg-test/registries.cpp#L155 CHECK(is_package_pattern("boost-*"));
• src/vcpkg-test/registries.cpp#L188 rs.push_back(make_registry(4, {"boost-*"}));

[BillyONeal]

This 'request changes' is only for the nonstandard tests being in end-to-end-tests-dir, all other comments in here are extreme nitpicks you can take or leave.

[BillyONeal]

Oh, and thanks for the fixes!

[BillyONeal]

Thanks for the fix!