merge master (#2062)

* docs: fix simple typo, completition -> completion

There is a small typo in iothub_service_client/src/iothub_messaging_ll.c, iothub_service_client/tests/iothub_msging_ll_ut/iothub_messaging_ll_ut.c.

Should read `completion` rather than `completition`.

* Addressing the Y2k38 issue for SAS tokens (#1807)

* Updating c-utility

* umock changes to allow uint64_t for SAS expiry time.

* Updating SAS internal API usage and UTs.

* Updating c-utility.

* Fixing UTs. Addressing PR feedback.

* Updating submodules.

* modify string buffer size (#1825)

* Update submodules (#1831)

* version.txt update (#1834)

* Update LTS support table (#1835)

* Update certs when using IoT Hub Public Preview with ECC (#1832)

* [SDL] NULL pointer check (#1836)

* [SDL] NULL pointer check

* update submodules

* Update submodules (#1840)

* Nuget Security Analysis Update NuGet.Config (#1841)

* Update NuGet.Config

* remove multiple feeds

* Fuzzers for AMQP and MQTT protocols (#1837)

* iot hub client fuzz harness

* Added protocol files to fuzz

* Updated MQTT fuzz directory

* fix linux build break

* added linux openssl fuzz interface

* Revert "added linux openssl fuzz interface"

This reverts commit c8692fd.

* added linux openssl fuzz interface

* fix build break

* fix linux build break

* fidex AFL issue on PUBACK

* fix linux build break

* fix test harness memory leak

* move test bin into individual folders

* Added docs and send link list fix

* updated documentation

* fuzzer doc updates

* remove queue on xio close

* update fuzz docs

* fuzzer for AMQP

* finish amqp fuzzer

* remove test files

* allow fuzz packets lod from file

* fix twin destroy msg_ctx

* removed win32 include

* removed win32 include
fix linux build

* fix linux build break

* updae fuzzer tracing

* remove AMQP fuzz fixes

* Fixed white spaces

* Fix MAC OS build break

* fixed MACOSX build

* Fixed linux build break

* fixed cmake and whitespace

* uamqp submodule update

* Fixed fuzzer cmake

* Update default build to VS2017 (removes support for VS2015) (#1846)

* Update default build to VS2017

* update build vs2017 path

* cmake default to "Visual Studio 15 2017"

* removed VS2015 from readme's

* move build to cmake version 3.17.5 (#1847)

* Update submodules (#1848)

* LONGHAUL: Close and reopen the service handle if the C2D send message fails. (#1850)

* Close and reopen the service handle if the C2D send message fails.

* added debug trace

* add trace info to longhaul tests (#1851)

* add trace info to longhaul tests

* update c-utility (#1852)

* Update to latest parson

* Update documentation for correct curl installation with wolfSSL and HTTP (#1855)

* update http set option to convert http return result

* add mock return values

* Update ECC certificate info in docs (#1856)

* [longhaul test] Filter out device IDs from event hub notifications (#1858)

* longhaul debug trace

* added more trace logs

* Filter AMQP message based on the device ID

* remove longhaul log noise

* Update umqtt submodule

* If no argument given to certGen.sh the usage is not printed (#1862)

* If no argument given to certGen.sh the usage is not printed

* bug #1859/#1860/#1861

* fixed running under WSL

* Do not call tput without $TERM (#1864)

Attempting to use this script non-interactively (e.g. during AzDO pipelines) results in $TERM being undefined, causing tput to fail which immediately terminates the script execution.

* add device creation delay before starting the longhaul tests (#1867)

* [LONGHAUL] Ignore twin updates if the twin version is 1 (#1868)

* ignore version 1 of twin updates

* Change Edge input message MQTT topic subscription from QoS 0 to 1

* Fix longhaul device_twin_desired first message (#1874)

* added traces to longhaul invoke_device_method() (#1878)

* [longhaul] fix device method retry (#1879)

* added traces to longhaul invoke_device_method()

* fix device method retry

* Update return value to MU_FAILURE where appropriate (#1877)

* AMQP protocol stack fuzz (#1875)

* iot hub client fuzz harness

* Added protocol files to fuzz

* Updated MQTT fuzz directory

* fix linux build break

* added linux openssl fuzz interface

* fidex AFL issue on PUBACK

* fix linux build break

* fix test harness memory leak

* Added docs and send link list fix

* updated documentation

* fuzzer doc updates

* remove queue on xio close

* update fuzz docs

* fuzzer for AMQP

* finish amqp fuzzer

* remove test files

* allow fuzz packets load from file

* fix twin destroy msg_ctx

* fixed twin guid crash

* bug:8781049 cancel the pending messages before destorying the device instance

* update fuzzer tracing

* fix the bug 8781049 regression

* ampq fuzz, handle smaler fuzz packet size

* fixed amqp fuzz overflow memcpy

* Bug 9202574: [FuzzAMQP] heap-buffer-overflow internal_decoder_decode_bytes /home/ericwol/azure-iot-sdk-c/uamqp/src/amqpvalue.c:5941

* Bug 9202574: [FuzzAMQP] heap-buffer-overflow internal_decoder_decode_bytes /home/ericwol/azure-iot-sdk-c/uamqp/src/amqpvalue.c:5941

* update uamqp submodule

* update cutil submodule

* fix amqp unit tests

* backout bug fix 8781049

* Added module to longhaul tests to cleanup orphaned hub device resources (#1885)

* Added module to longhaul tests to cleanup orphaned hub device resources

* added longhaul_resource_cleanup.h

* fixed consts

* pr feedback

* update uamqp submodule (#1886)

* Update umqtt submodule

* Update devbox_setup.md (#1888)


* Update doc/devbox_setup.md

Co-authored-by: Dane Walton <[email protected]>

* add sample certs to c2d sample

* Submodule udpate

* Fix order CMakelists.txt adds dependencies for unit tests

By adding umock before ctest and testrunner it makes umock to look
into its own submodules. If --recursive is not used on git clone or
git submodule update, then the following error occurs:

```cmake
-- Detecting CXX compile features - done
-- IoT Client SDK Version = 1.6.0
CMake Warning (dev) at deps/umock-c/CMakeLists.txt:13 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'use_cppunittest'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error at deps/umock-c/CMakeLists.txt:63 (add_subdirectory):
  The source directory

    /home/ewertons/code/azure-iot-sdk-c-2/deps/umock-c/deps/testrunner

  does not contain a CMakeLists.txt file.

CMake Error at deps/umock-c/CMakeLists.txt:66 (add_subdirectory):
  The source directory

    /home/ewertons/code/azure-iot-sdk-c-2/deps/umock-c/deps/ctest

  does not contain a CMakeLists.txt file.

-- Looking for include file stdint.h
```

Changing the order on C SDK CMakelists.txt causes ctest and testrunner
to be include from the C SDK deps, preventing umock of trying to adds
its own.

* update c-utility submodule (#1908)

* update c-utility submodule

* update dependent submodules

* [longhaul tests] Allow IoT hub event reconnect due to network loss (#1909)

* update c-utility

* update dependent submodules

* [longhaul] Allow IoT hub event reconnect due to network loss

* fixed described_filter_value double deletes

* Added AMQP packet documentation (#1911)

* Added longhaul ListenForEventAsync traces (#1912)

* Added longhaul ListenForEventAsync traces

Co-authored-by: Dane Walton <[email protected]>

* Fixing global buffer overflow when copying global string due to incorrect string length value. (#1913)

* Fixed longhaul memory leaks (#1915)

* [LONGHAUL] Use connection state change to reconnect the AMQP service connection (#1917)

* [LONGHAUL] Use connection state change to reconnect the AMQP service connection

* PR feedback

* [longhaul] Fixed memory leaks and thread locks (#1922)

* [longhaul] Fixed memory leaks and thread locks

* PR feedback

* Submodules update and add cancellation option to cbs auth

* Fix amqp transport cbs tests

* Fix amqp transport cbs tests (x2)

* Updating TLS requirements reference. (#1926)

* fileupload network interface support (#1884)

* fileupload network interface support

* file upload through network interface

* changed curl interface to networkInterface and updated c-utility submodule

* documentation update

* setting option-curl-interface

* updating c-utility

Co-authored-by: Aggrawal, Raj Kumar <[email protected]>
Co-authored-by: John Spaith <[email protected]>

* Updating c-utility submodule (#1928)

ESP tests failed with unrelated "out of space" error for multiple runs.

* [longhaul] retry IoTHubDeviceTwin_UpdateTwin on network failure (#1932)

* use checkout sources to clean directory (#1930)

* Allow jenkins agent to cleanup cmake folder (#1934)

* allow cmake folder to be deleted

* Removing APT package support. (#1933)

* Refactor device method subscription code (#1897)

Cleanup device method code to enable future PnP logic to be more cleanly added.

* Update readme.md (#1937)

Removing a confusing statement from the release section and deprecating the release schedule picture since it was completely outdated and not necessarily we'll have 2 LTSs releases every year.

* Update readme.md (#1938)

* Cleanup initial readme and dev box setup (#1936)

* Added framework for running longhaul tests under valgrind (#1940)

* longhaul valgrind test

* enable valgrind on longhaul_amqp_c2d

* update cutility

* include global_valgrind_suppression

* global_valgrind_suppression

* fix memory leak

* Add locking to longhaul get_statistics

* fix build break

* fix build break

* fix build break

* fixed iothub_client_statistics_add_c2d_info locks

* fixed iothub_client_statistics_add_c2d_info locks

* fixed on_c2d_message_sent

* fixed on_c2d_message_sent

* fixed on_c2d_message_sent

* fixed iothub_client_statistics_add_telemetry_info locking

* fixed longhaul_run_telemetry_tests locks

* update suppression file

* added lock to iothub_client_statistics_add_device_twin_reported_info

* added locks to iothub_client_statistics_add_device_twin_reported_info

* added locks to iothub_client_statistics_add_device_twin_reported_info

* added locks to iothub_client_statistics_add_device_twin_reported_info

* update suppression file

* remove resubscribing to events on telemetry_messages

* remove longhaul_stop_listening_for_telemetry_messages()

* add back longhaul_stop_listening_for_telemetry_messages

* test

* test

* added lock to IOTHUB_VALIDATION_INFO

* add lock to IOTHUB_VALIDATION_INFO

* update locks

* master merge

* retry IoTHubDeviceTwin_UpdateTwin on network failure

* update to valgrind cutil

* update c-utility

* update cutility

* update cutility

* clean up cmake directory

* show valgrind version

* update valgrind

* upgrade valgrind

* apt update

* show valgrind version

* dump os version

* show os version

* disable longhaul tests running under valgrind

* remove short timeout

* update c-utility

* update c-utility to master

* update c-utility submodules

* handle failures in SASL states (#1942)

* handle failures in SASL states

* check for each enum values

* change ut to c99 and turn off cppunittest by default (#1941)

* [longhaul] filter twin desired messages that was lost from network disconnect (#1945)

* [longhaul] filter twin desired messages that was from network disconnect

* fix linux build break

* Update readme for PnP samples (#1946)

* Update readme for samples

* Change terminology

* update the uamqp linking (#1948)

* fix Static Analyzer issues (#1949)

* fix Static Analyzer issues

* fix unit test

* Update submodules (#1958)

* Sample indexing test

* reconnect AMQP if CONNECTION_STATE_CLOSE_RCVD state is received (#1960)

* Add more details to longhaul test failures (#1969)

* iothub_client/samples: Fix typo demonstarates -> demonstrates (#1972)

* enable tracing in longhaul tests (#1974)

* This fixes an error message when we get the DPS operation status that does not have a rety app property

* Update submodules

* Updating CA roots (#1971)

* Adding DigiCert G2 root to samples.

* Adding new CA roots, changing to generic sample certificate store.

* Adding CA roots to test tools.

* Adding Apple M1 instructions for sample

* Fix unit tests for change in prov_transport_amqp_common

* Updating documentation for ECC server certificate chain support. (#1979)

Updating documentation for ECC server certificate chain support.

* Update readme.md

* Initial commit for async C2D message ack

* Fix first set of tests for C2D async ack

* Exposed SendMessageDisposition functions in client and module/LL and convenience APIs (part 1)

The main public API changes are done.
Still pending are test updates for gate builds to pass, test additions,
add documentation for new functions.

* Exposed SendMessageDisposition functions in client and module/LL and convenience APIs (part 2)

Fixed existing tests.

* Add API documentation for C2D disposition functions

* Add tests for new *SendDisposition functions

* Update mqtt_client_send_message_response calls

* Provisioning client with AMQP-WS uses an incorrect port when proxy is configured (#2002)

* filter device_twin_reported event if there was a network disconnect (#2001)

* filter device_twin_reported event if there was a network disconnect

* update travel time..now that we have retry logic

* [ACR/1] C2D Async ACK

* Fail to upload long Blob using mbedtls+compact(Issue #1995)

* take in account pull request remarks

* Update iothub_client_common_longhaul.c (#2008)

Set device method travel time back to 300

* [ACR/2] C2D Async ACK

* update rasp reqs (#2013)

* provisioning_client: explicitly states function prototypes (#2012)

* provisioning_client: explicitly states function prototypes

When building with `-Wstrict-prototypes` flag enabled, functions with no
explicit prototype are reported (i.e. `()` instead of `(void)`).

Replacing the empty parenthesis with the correct form of declaring a
function without arguments fixes the issue.

Signed-off-by: Francesco Giancane <[email protected]>

* tests: provisioning_hsm.c update functions' signature

Update signatures of function marked with empty argument list instead
of explicitly states that they have no argument at all (i.e. `()` versus
`(void)` ).

This should make the code more compliant with language specifications
and also suppresses warnings in more pedantic compilers.

Signed-off-by: Francesco Giancane <[email protected]>

* tests: symm_key: update function signatures

explicitly states with `(void)` when functions do not accept arguments.

Signed-off-by: Francesco Giancane <[email protected]>

* tests: tpm_msr: update functions signatures

explicitly states with `(void)` when functions do not accept arguments.

Signed-off-by: Francesco Giancane <[email protected]>

* tests: x509: update functions signature

Explicitly states with `(void)` that functions do not accept arguments.

Signed-off-by: Francesco Giancane <[email protected]>

Co-authored-by: Eric Wolz <[email protected]>

* increase blob upload size (#2014)

* Rewrite MQTT PUBLISH topic parser for C2D / IoT Edge parsing (#1756)

* Add strict-prototypes to gcc build (#2015)

* [ACR/1] Add support for sending async responses for C2D messages

* Add notice about upcoming service TLS changes to the readme (#2025)

* Add notice about upcoming service TLS changes to the readme

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* Update readme.md

Co-authored-by: Eric Wolz <[email protected]>

* [ACR/2] Add support for sending async responses for C2D messages

* [ACR/3] Add support for sending async responses for C2D messages

* Update c-utility

* Submodule update

* Update Cert Readme for Edge devices (#2040)

* Bump version to 1.8.0 and update LTS documentation for release. (#2042)

* Bump version to 1.8.0 and update LTS documentation for release.

* Add Documenation button that links to device developer docs to pnp device sample readmes (#2039)

Co-authored-by: Eric Wolz <[email protected]>

* Test windows hosted agent (#2017)

* Update longhaul.vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update longhaul.vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update build.cmd

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update build.cmd

* Update .vsts-ci.yml

* Update build.cmd

* Update .vsts-ci.yml

* Update longhaul.vsts-ci.yml

* Update iothubtest.c

* Update .vsts-ci.yml

* Update build.cmd

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* Update .vsts-ci.yml

* added amqp debug

* amqp

* update amqp

* add amqp logging

* update amqp branch

* Added CPP wrapper to unit tests

* update cmake min version to 3.5

* fix iothubtransport_mqtt_common_ut  cmake

* fix unit test

* fix cmake script

* fix cmake if statement

* restore iothubmessage_ut static

* run cppunit test only on win32 builds

* move to latest uamqp

* remove debug build cmd

* Update submodules

* Ericwol windows hosted agent (#2052)

* update dependednt submodules for hosted agents

* disable raspberrypi pipeline (#2056)

* Always turn on renegotiation on x509 auth with upload to blob (#2057)

Co-authored-by: Tim Gates <[email protected]>
Co-authored-by: Cristian Pop <[email protected]>
Co-authored-by: Dane Walton <[email protected]>
Co-authored-by: Fanzhe Lyu <[email protected]>
Co-authored-by: Mollie Munoz <[email protected]>
Co-authored-by: Ewerton Scaboro da Silva <[email protected]>
Co-authored-by: Dane Walton <[email protected]>
Co-authored-by: Stewart Adam <[email protected]>
Co-authored-by: Kevin Cadieux <[email protected]>
Co-authored-by: rajaggrawal <[email protected]>
Co-authored-by: Aggrawal, Raj Kumar <[email protected]>
Co-authored-by: John Spaith <[email protected]>
Co-authored-by: Wellington Duraes <[email protected]>
Co-authored-by: Dominic Betts <[email protected]>
Co-authored-by: Thomas Bracht Laumann Jespersen <[email protected]>
Co-authored-by: Hatem Maalla <[email protected]>
Co-authored-by: Francesco Giancane <[email protected]>
Co-authored-by: timtay-microsoft <[email protected]>
Co-authored-by: hihigupt <[email protected]>
Co-authored-by: Jimaco Brannian <[email protected]>
Co-authored-by: Mollie Munoz <[email protected]>

iothub_client/src/iothub_client_core.c

@@ -2682,6 +2682,7 @@ IOTHUB_CLIENT_RESULT IoTHubClientCore_SendMessageDisposition(IOTHUB_CLIENT_CORE_
 
     return result;
 }
+
 IOTHUB_CLIENT_RESULT IoTHubClientCore_EnablePolicyConfiguration(IOTHUB_CLIENT_CORE_HANDLE iotHubClientHandle, POLICY_CONFIGURATION_TYPE policyType, bool enablePolicyConfiguration)
 {
     IOTHUB_CLIENT_RESULT result;

iothub_client/src/iothub_client_ll_uploadtoblob.c

@@ -707,11 +707,15 @@ IOTHUB_CLIENT_RESULT IoTHubClient_LL_UploadMultipleBlocksToBlob_Impl(IOTHUB_CLIE
             }
             else
             {
+                // If client is using x509 auth, renegotiation must be turned on in order to work properly.
+                bool renegotiation_is_on = true;
+
                 /*transmit the x509certificate and x509privatekey*/
                 /*Codes_SRS_IOTHUBCLIENT_LL_02_106: [ - x509certificate and x509privatekey saved options shall be passed on the HTTPAPIEX_SetOption ]*/
                 if ((upload_data->cred_type == IOTHUB_CREDENTIAL_TYPE_X509 || upload_data->cred_type == IOTHUB_CREDENTIAL_TYPE_X509_ECC) &&
                     ((HTTPAPIEX_SetOption(iotHubHttpApiExHandle, OPTION_X509_CERT, upload_data->credentials.x509_credentials.x509certificate) != HTTPAPIEX_OK) ||
-                    (HTTPAPIEX_SetOption(iotHubHttpApiExHandle, OPTION_X509_PRIVATE_KEY, upload_data->credentials.x509_credentials.x509privatekey) != HTTPAPIEX_OK))
+                    (HTTPAPIEX_SetOption(iotHubHttpApiExHandle, OPTION_X509_PRIVATE_KEY, upload_data->credentials.x509_credentials.x509privatekey) != HTTPAPIEX_OK) ||
+                    (HTTPAPIEX_SetOption(iotHubHttpApiExHandle, OPTION_SET_TLS_RENEGOTIATION, &renegotiation_is_on) != HTTPAPIEX_OK))
                     )
                 {
                     LogError("unable to HTTPAPIEX_SetOption for x509 certificate");

iothub_client/src/iothubtransport_mqtt_common.c

@@ -102,6 +102,7 @@ static const char* REQUEST_ID_PROPERTY = "?$rid=";
 static const char* DIAGNOSTIC_CONTEXT_CREATION_TIME_UTC_PROPERTY = "creationtimeutc";
 
 static const char* DISTRIBUTED_TRACING_PROPERTY = "tracestate";
+
 static const char DT_MODEL_ID_TOKEN[] = "model-id";
 static const char DEFAULT_IOTHUB_PRODUCT_IDENTIFIER[] = CLIENT_DEVICE_TYPE_PREFIX "/" IOTHUB_SDK_VERSION;
 

iothub_client/tests/iothubclient_ll_u2b_ut/iothub_client_ll_u2b_ut.c

@@ -779,6 +779,7 @@ static void setup_upload_blocks_mocks(IOTHUB_CREDENTIAL_TYPE cred_type, bool pro
     {
         STRICT_EXPECTED_CALL(HTTPAPIEX_SetOption(IGNORED_PTR_ARG, OPTION_X509_CERT, IGNORED_PTR_ARG));
         STRICT_EXPECTED_CALL(HTTPAPIEX_SetOption(IGNORED_PTR_ARG, OPTION_X509_PRIVATE_KEY, IGNORED_PTR_ARG));
+        STRICT_EXPECTED_CALL(HTTPAPIEX_SetOption(IGNORED_PTR_ARG, OPTION_SET_TLS_RENEGOTIATION, IGNORED_PTR_ARG));
     }
     if (trusted_cert)
     {

iothub_client/tests/iothubtransport_mqtt_common_ut/iothubtransport_mqtt_common_ut.c

@@ -1086,6 +1086,7 @@ static void setup_message_recv_with_properties_mocks(
     bool auto_decode, 
     bool msgCbResult)
 {
+
     setup_message_receive_initial_calls(TEST_MQTT_MSG_TOPIC_W_1_PROP, false);
     STRICT_EXPECTED_CALL(STRING_c_str(IGNORED_PTR_ARG)).CallCannotFail().SetReturn(TEST_MQTT_MESSAGE_TOPIC);
     EXPECTED_CALL(STRING_TOKENIZER_create_from_char(TEST_MQTT_MSG_TOPIC_W_1_PROP));