HLSL: Type mismatch between vertex shader output and hull shader input

[dustin-lunarg]

When creating a Vulkan graphics pipeline with a hull shader, vkCreateGraphicsPipelines is generating a validation error that indicates there is a type mismatch between the vertex and hull shader stages.

When the following struct is used as the vertex shader output/hull shader input:

struct HS_Input
{
    float4  m_Position        : SV_POSITION;
    float4  m_TexCoord0       : TEXCOORD0;
    float4  m_TexCoord1       : TEXCOORD1;
};

The SPIR-V emitted for the vertex shader has the output struct split into separate values, where the user defined values each have their own location:

OpDecorate %_entryPointOutput_m_TexCoord0 Location 0
OpDecorate %_entryPointOutput_m_TexCoord1 Location 1

The SPIR-V emitted for the hull shader has the input struct split into a single value for the position builtin and a struct containing the user defined values, where the struct containing the user defined values is assigned location 0. In the following example the input struct was originally named I, which became I_0 for the struct containing the user defined values after the split:

OpName %HS_Input_0 "HS_Input_0"
OpMemberName %HS_Input_0 0 "m_TexCoord0"
OpMemberName %HS_Input_0 1 "m_TexCoord1"
OpDecorate %I_0 Location 0

This produces the following validation error:

VALIDATION ERROR 0x1 Type mismatch on location 0.0: 'ptr to output vec4 of float32' vs 'ptr to input arr[3] of struct of (vec4 of float32, vec4 of float32)'

If I am deciphering this correctly, it says that the vec4 at output location 0 (%_entryPointOutput_m_TexCoord0) does not match the struct at input location 0 (%I_0).

Modifying the SPIR-V for the hull shader to split the input struct into separate vec4 values eliminates the validation error:

OpDecorate %I_m_TexCoord0 Location 0
OpDecorate %I_m_TexCoord1 Location 1

[johnkslang]

Yes, the I/O structures should be flattened. I will look into why that is not happening.

[johnkslang]

@dustin-lunarg Curious, if I try using

struct HS_Input
{
    float4  m_Position        : SV_POSITION;
    float4  m_TexCoord0       : TEXCOORD0;
    float4  m_TexCoord1       : TEXCOORD1;
};

In a hull shader with fxc, I get errors on the TEXCOORD semantics.

Do your shaders work with fxc?

[dustin-lunarg]

@johnkslang The hull shaders producing the validation error are working with fxc. The following simplified example should compile with fxc /T hs_5_0 /E HS_Main:

struct HS_Output
{
    float4  m_Position        : POSITION;
    float4  m_TexCoord0       : TEXCOORD0;
    float4  m_TexCoord1       : TEXCOORD1;
};

struct HS_PCFOutput
{
    float m_TessFactor[3]    : SV_TessFactor;
    float m_InsideTessFactor : SV_InsideTessFactor;
};

struct HS_Input
{
    float4  m_Position        : POSITION;
    float4  m_TexCoord0       : TEXCOORD0;
    float4  m_TexCoord1       : TEXCOORD1;
};

HS_PCFOutput HS_TrianglePCF(InputPatch<HS_Input, 3> I)
{
    HS_PCFOutput O = (HS_PCFOutput)0;
    O.m_TessFactor[0] = 1.0f;
    O.m_TessFactor[1] = 1.0f;
    O.m_TessFactor[2] = 1.0f;
    O.m_InsideTessFactor = 1.0f;
    return O;
}

[domain("tri")]
[partitioning("fractional_odd")]
[outputtopology("triangle_cw")]
[patchconstantfunc("HS_TrianglePCF")]
[outputcontrolpoints(3)]
HS_Output HS_Main(InputPatch<HS_Input, 3> I, uint uCPID : SV_OutputControlPointID)
{
    HS_Output output = (HS_Output)0;

    output.m_Position = I[uCPID].m_Position;
    output.m_TexCoord0 = I[uCPID].m_TexCoord0;
    output.m_TexCoord1 = I[uCPID].m_TexCoord1;

    return output;
}

In this case, I have some tessellation shaders that work with the Direct3D version of an application, but have issues when used with the Vulkan version. There is the validation error reported here and a rendering issue reported in #1181. Modifying the SPIR-V to eliminate the validation error reported here does not seem to affect the issue reported in #1181.

[johnkslang]

So, InputPatch<HS_Input, 3> is classified as

• not being patch
• being arrayed I/O
• but not containing any built-ins

Currently, non-patch, arrayed I/O that does not contain a built-in is not flattened or split.

[johnkslang]

Perhaps the problem is that the top level of hierarchy should not be flattened, because it is an array, but the next level down should be flattened.

@loopdawg ? I think you dealt with InputPatch; any input here? Thanks.

[johnkslang]

Wait a minute, @dustin-lunarg , between me saying I couldn't reproduce and you posting a more complete example, you changed SV_POSITION -> POSITION.

SV_POSITION is a built-in, while POSITION is not.

Is your interface mixing these?

[dustin-lunarg]

@johnkslang Sorry, I meant to use SV_POSITION in both examples. I pasted the wrong code for the second example.

Both vertex shader output and hull shader input are using SV_POSITION with the shaders that produce this validation error.

With SV_POSITION, the hull shader input I is split into I_m_Position for the built-in and an I_0 struct for the user-defined values. It is a comparison between the I_0 struct input type and the vertex shader _entryPointOutput_m_TexCoord0 output type that seems to be the source of the validation error.

[johnkslang]

The fundamental problem is that arrayed I/O containing a built-in (e.g., an array of struct input to hull) is "split", while a simple struct (e.g., an output struct from vertex) is "flattened".