[Tracking] Certification tests

[AlCalzone]

✅ Category: Automatic

• ✅ Association Group Data
  • ✅ AGD_AssociationGroupData_Rev01
• ✅Command Class Support Requirements
  • ✅ CSR_AssocV3OrMCAV4IsSupported_Rev01
  • ✅ CSR_LifelineAssociation_Rev02
• ✅ Common Device Requirements
  • ✅ CDR_MultiChannelCCInNIF_Rev01
  • ✅ CDR_ReceiveEncapCommands_Rev02
  • ✅ CDR_ZWavePlusInfoCCInNIF_Rev01
  • ✅ CDR_ZWPv2InfoReport_Rev01
  • ✅ CDR_ZWPv2MandatoryCCs_Rev01
  • ✅ CDR_ZWPv2NonOptionalCCs_Rev01
  • ✅ CDR_ZWPv2OptionalBit_Rev01
• ✅ Controller Requirements
  • ✅ CR_NoMultiChannelCCInNIF_Rev01
• ✅ Device Type Requirements
  • ✅ DT_DeviceClasses_Rev01
  • ✅ DT_GenericCtrlControlledActuatorCCs_Rev01
  • ✅ DT_MandatorySupportedCCs_Rev01
• ✅ Management Command Classes
  • ✅ CCM_VersionCmdClassCCVersions_Rev01
• ✅ Role Type Requirements
  • ✅ RT_ListeningFlag_Rev02
• ✅ S2 Certification
  • ✅ S2C_02070_DuplicateDetection_Rev01
  • ✅ S2C_02566_MsgEncapCorrectResponse_Rev01
  • ✅ S2C_02567_MsgEncapSequenceResAndExt_Rev01
  • ✅ S2C_02568_MsgEncapExtensionObject_Rev01
  • ✅ S2C_02570_MsgEncapEncyptedExtensions_Rev01
  • ✅ S2C_02571_MsgEncapSPANExtension_Rev01
  • ✅ S2C_02572_MsgEncapMPANExtension_Rev01
  • ✅ S2C_02575_MsgEncapMGRPExtension_Rev01
  • ✅ S2C_02576_MsgEncapMOSExtension_Rev01
  • ✅ S2C_02578_SPANManagement_Rev01
  • ✅ S2C_02580_S2NonceGet_Rev01

Category: Interactive

• Command Class Control Requirements
  • ✅ CCR_AssociationCC_Rev02
  • ✅ CCR_BarrierOperatorCC_Rev02
  • ✅ CCR_BasicCC_Rev02
  • ✅ CCR_BatteryCC_Rev02
  • ✅ CCR_BinarySwitchCC_Rev02
  • ✅ CCR_CentralSceneCC_Rev03
  • ❌ CCR_ColorSwitchCC_Rev02 - RGB colors are rounded incorrectly certification-backlog#51 ¹
  • ❌ CCR_ConfigurationCC_Rev02 - Fix service call for setting a config value certification-backlog#12
  • ✅ CCR_DeviceResetLocallyCC_Rev02
  • ✅ CCR_DiscardLowerSecurityCmds_Rev02
  • ❌ CCR_DoorLockCC_Rev02 - Implement missing functionality for Door Locks certification-backlog#21
  • ❌ CCR_EntryControlCC_Rev03 - No support for Entry Control CC (such as an alarm keypad) certification-backlog#14
  • ✅ CCR_ImplementedCCVersions_Rev02
  • ✅ CCR_IndicatorCC_Rev02
  • ✅ CCR_InterviewCompletion_Rev02
  • ❌ CCR_MeterCC_Rev02 - Z-Wave JS: meter_reset service call issues the reset command multiple times, does not refresh values home-assistant/core#116222
  • ✅ CCR_MultiChannelCC_Rev03
  • ❌ CCR_MultilevelSensorCC_Rev02 - Multilevel Sensor CC: Wrong unit shown when a report with an unknown scale is used certification-backlog#26
  • ✅ CCR_MultilevelSwitchCC_Rev02
  • ✅ CCR_NotificationCC_Rev04 - ²
  • ❌ CCR_SoundSwitchCC_Rev02 - Z-Wave JS: Certification requirements for Sound Switch CC not fulfilled certification-backlog#16
  • ✅ CCR_ThermostatModeCC_Rev02 - Note: Needs service call, document this!
  • ❌ CCR_ThermostatSetbackCC_Rev02 - Add support for Thermostat Setback CC certification-backlog#15
  • ❌ CCR_ThermostatSetpointCC_Rev03 - Z-Wave JS: Unable to perform certification tests for Thermostat Setpoint CC home-assistant/core#92810
  • ❌ CCR_UserCodeCC_Rev04 - no user code support in HA
  • ✅ CCR_VersionCC_Rev02
  • ✅ CCR_WakeUpCC_Rev02
  • ❌ CCR_WindowCoveringCC_Rev02 - Window Covering CC support status? certification-backlog#30
• ✅Command Class Support Requirements
  • ✅ CSR_CommandClassIsSupported_Rev02
  • ✅ CSR_LifelineMandatoryReports_Rev02
• ✅ Common Device Requirements
  • ✅ CDR_ZWPv2IndicatorCCRequirements_Rev01 - Note: Document that this shows up in the notification sidebar
• ✅ Controller Requirements
  • ✅ CR_NodeCommissionInterviewOnRequest_Rev01
  • ✅ CR_NoInterviewOnConfigReport_Rev01
• ✅ Device and Role Type
  • ✅ DR_DeviceAndRoleTypeSelection_Rev01
• Device Data
  • ✅ DD_ManufacturerSpecificCCData_Rev01
  • ⁉️ DD_VersionCCData_Rev01 - Passes! Form needs to be updated to actual versions prior to certification
  • ✅ DD_ZWavePlusInfoCCData_Rev01
• ✅ Device Type Requirements
  • ✅ DT_MandatoryControlledCCs_Rev02
• Product Information
  • ⁉️ PI_CertificationData_Rev01 - The requested info does not exist on the certification page
• ✅ Role Type Requirements
  • ✅ RT_PowerSupply_Rev02
• ✅ S2 Certification
  • ✅ S2C_02010_S2NonceGetAndReport_Rev01
  • ✅ S2C_02579_MPANManagement_Rev01

✅ Category: Scripts

• ✅ AssociationCmdClassV3_Rev04_Orig
• ✅ AssociationGrpInfoCmdClassV3_Rev02_Orig
• ✅ BasicMappingNotAllowed_Rev03_Orig
• ✅ IndicatorCmdClassV4_Rev05_Orig
• ✅ MultiChannelAssociationCmdClassV4_Rev06_Orig
• ✅ MultiCommandCmdClassV1_Rev04_Orig
• ✅ VersionCmdClassV3_Rev09_Orig
• ✅ ZWavePlusInfoCmdClassV2_Rev09_Orig

Category: Inclusion

• Command Class Control Requirements
  • ❌ CCR_MultiChannelAssociationCC_Rev02 - Association CC: Must not be exposed/interviewed when Multi Channel Association CC is supported #5874 Association CC: Mandatory node interview includes a SET command for the lifeline, even if already set #5875
• ✅ Command Class Support Requirements
  • ✅ CSR_LifelineReportToMultipleNodes_Rev01
  • ✅ CSR_LifelineReportTriggeredByZWave_Rev01
  • ✅ CSR_ManufacturerS0Replies_Rev01
• ✅ Common Device Requirements
  • ✅ CDR_ControlMark_Rev01
  • ✅ CDR_DeviceResetLocallyReport_Rev01
  • ✅ CDR_EncapOrder_Rev02
  • ✅ CDR_MulticastNoResponse_Rev02
  • ✅ CDR_NIFRequirements_Rev01
  • ✅ CDR_ResetWithoutDelivery_Rev01
  • ✅ CDR_ResponseEncap_Rev02
  • ✅ CDR_WhenNodeReset_Rev01
• ✅ Controller Requirements
  • ✅ CR_InteroperabilityRequirements_Rev01
• ✅ Device Type Requirements
  • ✅ DT_SecurityClasses_Rev01
• ✅ Network-Protocol Command Classes
  • ✅ CCN_PowerlevelCmdClassTestNode_Rev01
• Role Type Requirements
  • ✅ RT_CSCIsFirstNodeInNetwork_Rev02
  • ✅ RT_CSCLifelineConfigIfSIS_Rev02
  • ✅ RT_CSCMustAssumeSISRole_Rev02
  • ✅ RT_CSCMustSupportS0AndS2_Rev01 - Note: 2nd inclusion must use "Legacy Secure"
  • ✅ RT_CSCWakeUpConfigIfSIS_Rev02
  • ✅ RT_CSCWakeUpConfigPENIfSIS_Rev01
  • ✅ RT_SISReplaceFailingNode_Rev02 - Not supported by HA
  • ❌ RT_SUCAndSISFunctionality_Rev01 - No PIN entry shown for remote inclusion: Bootstrapping node on behalf of inclusion controller does nothing - possibly HA/Server issue #5888
  • ✅ RT_WakeUpNoMoreToReportingNodes_Rev01
• S2 Certification
  • ✅ S2C_02008_SecureInclusion_Rev01
  • ✅ S2C_02014_ManageFullSPANTable_Rev01
  • ❌ S2C_02487_InclusionControllerCC_Rev01 - Bootstrapping node on behalf of inclusion controller does nothing - possibly HA/Server issue #5888
  • ✅ S2C_02561_PublicKeyReport_Rev01
  • ✅ S2C_02562_KEXReport_Rev01
  • ✅ S2C_02563_WrongKEXReport_Rev02
  • ✅ S2C_02564_KEXGet_Rev01
  • ✅ S2C_02573_MsgEncapMPANExtMoreToFollow_Rev01
  • ✅ S2C_02574_MsgEncapMPANExtGroupID_Rev01
  • ✅ S2C_02577_CompatibilityConsiderations_Rev01
  • ✅ S2C_02581_S2NonceReport_Rev01
  • ✅ S2C_02582_WrongPublicKeyReport_Rev01
  • ✅ S2C_02583_NetworkKeyGet_Rev01
  • ✅ S2C_02587_WrongTransferEnd_Rev01
  • ✅ S2C_02588_S2CmdsSupportedGet_Rev01
  • ✅ S2C_02590_S2CmdsSupportedReport_Rev01
  • ✅ S2C_02593_KEXSet_Rev01
  • ✅ S2C_02594_WrongKEXSet_Rev01
  • ✅ S2C_02596_KeyExchange_Rev01
  • ⁉️ S2C_02599_FailingKeyExchange_Rev01 - Step 2.3: Granting no keys is not possible, because the user is prevented from doing so after choosing "secure if possible". Other steps pass
  • ✅ S2C_02600_BootstrappingInterruptPoints_Rev01
  • ✅ S2C_02601a_BootstrTimeoutsIncludingNode_Rev01
  • ✅ S2C_02601b_BootstrTimeoutsJoiningNode_Rev01
• Security Requirements
  • ✅ S0_ControllerCanIncludeS0Devices_Rev01
  • ✅ S0_ControllerPerformsS0Bootstrap_Rev01
  • ✅ S0_DelayNetworkKeyVerify_Rev01
  • ✅ S0_DelayNonceGetInclusionByDUT_Rev01
  • ✅ S0_DelayNonceReportInclusionByDUT_Rev01
  • ✅ S0_DelaySchemeReport_Rev01
  • ✅ S0_DelaySchemeReportAfterSchemeInherit_Rev01
  • ✅ S0_InvalidMACInNetworkKeyVerify_Rev01
  • ✅ S0_InvalidNetworkKeyVerify_Rev01
  • ✅ S0_InvalidNonceInNetworkKeyVerify_Rev01
  • ✅ S0_SecurityS0IsImplemented_Rev01
  • ✅ S0_SendInvalidSchemeReport_Rev01
  • ✅ S2_AbortAfterEmptyKexReport_Rev01
  • ✅ S2_BootstrTimeoutsIncludingNode_Rev01
  • ✅ S2_BootstrTimeoutsJoiningNode_Rev01
  • ✅ S2_CommandClasses_Rev01
  • ✅ S2_ConfirmForAuthenticated_Rev01
  • ✅ S2_ControllerMustAcceptExtendedCCs_Rev01
  • ✅ S2_ControllerPerformsBootstraping_Rev01
  • ✅ S2_GrantedS2Classes_Rev01
  • ✅ S2_S0IncByNonSecureIncController_Rev01
  • ✅ S2_SeveralSeparateECDHKeyPairs_Rev02
  • ❌ S2_SISInclusionCCSide_Rev02 - Bootstrapping node on behalf of inclusion controller does nothing - possibly HA/Server issue #5888
  • ✅ S2_SISMustHaveS2ClassInputAndDisplay_Rev01
  • ✅ S2_SISMustPerformS2Bootstrap_Rev01
  • ✅ S2_SISMustSupportAnyS2ClassCombination_Rev01
  • ✅ S2_StopParsingAtCommandClassMark_Rev01
  • ✅ S2_SupportedCCRequirement_Rev02
  • ✅ S2_WarningHighestKeyNotGranted_Rev01
  • ⁉️ S2_WarningS0BootstrapInS2Network_Rev01 - The 2nd test checks an impossible scenario: To use S0 over no security, the user has to manually opt in (except for certain device types). So the user is aware that the device is using S0 only.
• ✅ Smart Start Requirements ^{3 4}
  • ✅ SSR_CanPerformSmartStartInclusion_Rev01
  • ✅ SSR_MustPerformS2BootstrapTest3_Rev02
  • ✅ SSR_NodeProvListAddRemoveAndStatus_Rev01 ⁵
  • ✅ SSR_NodeProvListAdvancedJoining_Rev01
  • ✅ SSR_NodeProvListBootstrappingMode_Rev01
  • ✅ SSR_NodeProvListInclusionSetting_Rev01
  • ✅ SSR_PendingNodeProvisioningListEntry_Rev02
  • ✅ SSR_RetryAfterFailedInclusion_Rev02
  • ✅ SSR_S2AuthenticatedInclusionSupport_Rev01
  • ✅ SSR_SmartStartControlNotAsInclCtrl_Rev01
• ✅ Transport-Encap Command Classes
  • ✅ CCT_TransportServiceCmdClassV2_Rev02

Category: Manual

• Command Class Control Requirements
  • ✅ CCR_DocuRequirement_Rev01
  • ✅ CCR_FirmwareUpdateMetaDataCC_Rev02
  • ❌ CCR_FullyOrPartiallyControlledCmdClasses_Rev01 - Need to decide which functionality to hide due to missing HA support, also document controlled CCs
  • ✅ CCR_NoControlDefinition_Rev01
  • ✅ CCR_Security0CC_Rev02
  • ✅ CCR_Security2CC_Rev02
  • ✅ CCR_SecurityClassLearning_Rev01 - No support for learn mode, DUT "learn" security classes during device inclusion
  • ✅ CCR_SupervisionCC_Rev02
• ✅ Command Class Support Requirements
  • ✅ CSR_ControlViaAssociationSecurityLearning_Rev01 - DUT does not send any controlling command via any association group, no support for learn mode, DUT "learn" security classes during device inclusion
  • ✅ CSR_FirmwareUpdateCCConfig_Rev01 - No support for learn mode, DUT firmware cannot be OTA updated
  • ✅ CSR_SupervisionEncapViaAssociation_Rev01 - No support for learn mode, DUT "learn" security classes during device inclusion
• ✅ Common Device Requirements
  • ✅ CDR_ApplicationStatusCCSupport_Rev01
  • ✅ CDR_DetermineResetCompleted_Rev01
  • ✅ CDR_DynamicCapabilities_Rev01
  • ✅ CDR_MulticastNoGet_Rev01
  • ✅ CDR_PollingRequirements_Rev01 - Z-Wave JS does not poll automatically
  • ✅ CDR_ResponseTimeouts_Rev01
  • ✅ CDR_UnsolicitedFrames_Rev01 - Z-Wave JS does not send unsolicited frames
  • ✅ CDR_UseOfMultiCommandEncap_Rev01 - Z-Wave JS does not send Multi Command frames, it only decodes them
• ✅ Controller Requirements
  • ✅ CR_MinimumControlFunctionality_Rev02
  • ✅ CR_NetworkManagementFunctionality_Rev01
• Long Range Requirements
  • ❌ LR_ControllerBootstrappingModeTLV_Rev01 - Driver passes, HA has no support yet
  • ❌ LR_ControllerCanIncludeNodesIntoLRNetwork_Rev01 - Driver passes, HA has no support yet
  • ❌ LR_ControllerGrantsKeysRequiringAuthentication_Rev01 - Driver passes, HA has no support yet
  • LR_ControllerUsesDifferentKeysForClassicAndLR_Rev01
• ✅ Management Command Classes
  • ✅ CCM_AssociationCmdClass_Rev01 - Note: Only Device Reset Locally is sent via Associations
  • ✅ CCM_AssociationGrpInfoCmdClass_Rev01 - Note: Only Device Reset Locally is sent via Associations
• Product Information
  • ✅ PI_CertifiedZWaveStack_Rev01
  • PI_MarketAndFrequency_Rev02
  • ✅ PI_ZWavePlusProductType_Rev01
• Required Documentation
  • RD_AdditionalDocuRequirements_Rev01
  • RD_AssociationCC_Rev01
  • RD_FactoryDefaultReset_Rev01
  • RD_IndicatorCC_Rev01
  • RD_LongRange_Rev01
  • RD_MultipleManufacturers_Rev01
  • RD_SecurityS2CC_Rev01
  • RD_SmartStart_Rev02
  • RD_SmartStartControllers_Rev01
  • RD_SupervisionCC_Rev01
  • RD_ZWPv2ControllerDocu_Rev02
  • RD_ZWPv2RequiredTerminology_Rev01
• ✅ Role Type Requirements
  • ✅ RT_SupportedSmartStartFunctionality_Rev01
• ✅ Security Requirements
  • ✅ S2_GrantingSecurityClasses_Rev01
  • ✅ S2_SecurityClassAdjustment_Rev01
  • ✅ S2_SISMaySupportCSA_Rev01
  • ✅ S2_UpgradeExistingDevice_Rev01
• ✅ Smart Start Requirements
  • ✅ SSR_ControllerHasQRScanning_Rev01
  • ✅ SSR_ControllerHasGen1QRScanning_Rev01
• Transport-Encap Command Classes
  • ⁉️ CCT_CRC16EncapsulationCmdClassV1_Rev01 ⁶

Footnotes

1. For viewing the current color, the device needs to be turned on ↩

2. Some events appear at the top, some under Diagnostics at the bottom. Updating the notification list is done by updating the Z-Wave integration and/or addon
   HA currently requires manually enabling some notifications: https://github.com/orgs/zwave-js/projects/1?pane=issue&itemId=63127528 ↩

3. To add a device using Gen1 QR code, press SUBMIT in the dialog despite any warnings. Add mode will be started anyways. ↩

4. To add a DSK to the provisioning list, use https://zwave-js.github.io/qr/ and copy the QR code string or scan the image. ↩

5. Is the node reported as included in the network? - Yes, after closing and re-opening the provisioning list. ↩

6. ⁉️ CRC-16_Command_Class_advertised_in_NIF - Should be covered by automatic NIF tests, learn mode not supported
   ⁉️ CRC-16_Command_Class_not_advertised_in_Security_Commands_Support - Learn mode not supported
   ✅ CRC-16_Command_Class_MUST_NOT_be_encapsulated_by_any_other_command_class - If more than one encapsulation command class is used for controlled command classes the test houses must receive sufficient information about how such encapsulated messages can be triggered:

   • Supervision: Control any device that supports Supervision
   • Security S0/S2: Control any securely included device
   • Multi Channel: Control an endpoint on any supporting device
     ✅ CRC16_Command_Class_must_answer_as_asked
     ✅ Receive_and_interpret_all_command_classes_in_the_NIF_also_with_CRC-16_encapsulation
     ✅ Ensure_destination_supports_CRC-16_Encapsulation_Command_Class - Z-Wave JS does not use CRC-16 to control on its own.
   ↩