Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iothub_registrymanager_sample fails; curl_easy_perform() failed: Out of memory #219

Closed
jcwren opened this issue Aug 24, 2017 · 21 comments
Closed
Assignees
Labels

Comments

@jcwren
Copy link

jcwren commented Aug 24, 2017

  • OS and version used: Linux debian9 4.9.0-3-amd64 Connection intermittant #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux
  • SDK version used: 1.1.21
  • Libraries: libssl-dev 1.1.0f-3, libcurl4-openssl-dev 7.52.1-5

Description of the issue:

Trying to run iothub_registrymanager_sample, and it is failing with curl_easy_perform() reporting out of memory.

Code sample exhibiting the issue:

See samples/iothub_registrymanager_sample/iothub_registrymanager_sample.c

Additional notes

I don't believe this is causing the problem, as other sample code in the project is running, but when during the linking phase of any binary, I see the following:

/usr/bin/ld: warning: libssl.so.1.0.2, needed by /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may conflict with libssl.so.1.1
/usr/bin/ld: warning: libcrypto.so.1.0.2, needed by /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libcurl.so, may conflict with libcrypto.so.1.1

Even the most recent versions of libcurl say only libssl < 1.1 is supported. However, many other packages in the system depend on 1.1, so there's no way to downgrade everything to 1.0.2. I have not found any way to tell cmake to use the libssl.so.1.0.2 and libcrypto.so.1.0.2 that libcurl4-openssl-dev installs. Conducting an earlier experiment by linking libssl.so to libssl.so.1.0.2 resulted in several functions not being found (libcrypto was also redirected at the same time).

Console log of the issue:

$ iothub_service_client/samples/iothub_registrymanager_sample/iothub_registrymanager_sample
Calling IoTHubServiceClientAuth_CreateFromConnectionString with the connection string
iotHubServiceClientHandle has been created successfully
Creating RegistryManager...
RegistryManager has been created successfully
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_CreateDevice failed with unknown error
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_CreateDevice failed with unknown error
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_GetDevice failed with unknown error
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_DeleteDevice failed with unknown error
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_GetDeviceList failed with unknown error
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Thu Aug 24 12:32:35 2017 File:/home/jcw/azure_linux/azure-iot-sdk-c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:960 HTTPAPIEX_SAS_ExecuteRequest failed
IoTHubRegistryManager_GetStatistics failed with unknown error
Calling IoTHubRegistryManager_Destroy...
Calling IoTHubServiceClientAuth_Destroy...
$
@jcwren
Copy link
Author

jcwren commented Sep 7, 2017

Replacing libcurl-openssl-dev with libcurl-gnutls-dev allows the example to run successfully.

@jebrando
Copy link
Contributor

Thanks for the heads up, we are looking into this and will try to repro the issue.

@vbisbest
Copy link

vbisbest commented Oct 3, 2017

Same issue running the python client sample.

Error: Time:Tue Oct 3 16:10:32 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Tue Oct 3 16:10:32 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Tue Oct 3 16:10:32 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_client/src/iothubtransporthttp.c Func:DoMessages Line:2073 unable to HTTPAPIEX_SAS_ExecuteRequest
^CIoTHubClient sample stopped
Error: Time:Tue Oct 3 16:10:32 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

@vbisbest
Copy link

vbisbest commented Oct 3, 2017

@jcwren Can you explain how? "Replacing libcurl-openssl-dev with libcurl-gnutls-dev allows the example to run successfully." Thanks.

@vbisbest
Copy link

vbisbest commented Oct 3, 2017

And looks like this was addressed in iot-edge.
Azure/iot-edge-v1#28

@jcwren
Copy link
Author

jcwren commented Oct 3, 2017

I'm not sure I can explain how. I uninstalled libcurl-openssl-dev, installed libcurl-gnutls-dev, recompiled the project, and it didn't generate the out of memory error. Switched it back, it failed. If that's not sufficient, can you give me a little more detail on what you're looking for, please?

@jakuboz
Copy link

jakuboz commented Oct 11, 2017

This comment applies to Raspbian Stretch and Blob upload but error is the same.
While using libcurl4-openssl-dev:

Starting the IoTHub client sample upload to blob...
Info: IoT Hub SDK for C, version 1.1.24
Error: Time:Wed Oct 11 09:26:42 2017 File:/home/pi/azure-iot-sdk/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:552 curl_easy_perform() failed: Out of memory

Error: Time:Wed Oct 11 09:26:42 2017 File:/home/pi/azure-iot-sdk/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:554 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Wed Oct 11 09:26:42 2017 File:/home/pi/azure-iot-sdk/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Wed Oct 11 09:26:42 2017 File:/home/pi/azure-iot-sdk/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:395 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Wed Oct 11 09:26:42 2017 File:/home/pi/azure-iot-sdk/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_Impl Line:857 error in IoTHubClient_LL_UploadToBlob_step1
hello world failed to upload

Replacing openssl with libcurl4-gnutls-dev helps, but generates different error although file is successfully uploaded:

Starting the IoTHub client sample upload to blob...
Info: IoT Hub SDK for C, version 1.1.24
Error: Time:Wed Oct 11 08:47:06 2017 File:/home/pi/azure-iot-sdk/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_SetOption Line:794 failure in curl_easy_setopt - CURLOPT_SSL_CTX_FUNCTION
Error: Time:Wed Oct 11 08:47:06 2017 File:/home/pi/azure-iot-sdk/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:400 HTTPAPI_SetOption failed when called for option TrustedCerts
Error: Time:Wed Oct 11 08:47:08 2017 File:/home/pi/azure-iot-sdk/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_SetOption Line:794 failure in curl_easy_setopt - CURLOPT_SSL_CTX_FUNCTION
Error: Time:Wed Oct 11 08:47:08 2017 File:/home/pi/azure-iot-sdk/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:400 HTTPAPI_SetOption failed when called for option TrustedCerts
hello world has been created

As far as I looked into curl documentation this is due fact that this option (CURLOPT_SSL_CTX_FUNCTION) works only with SSL.
Is there any way it could be corrected?

@ewertons
Copy link
Contributor

ewertons commented Nov 9, 2017

Hi @jcwren, from your comment posted on Sept 6th are you unblocked now?
Is this issue still happening for you? If you plan on verifying it again, please use the latest version of the Azure IoT C SDK.

@jakuboz
Copy link

jakuboz commented Nov 10, 2017

On my raspbian linux and using SDK 1.1.26 problem still persists, with the same solution as in my previous comment

@drhalftone
Copy link

I'm having some problem on Mac OS X:

Info: IoT Hub SDK for C, version 1.1.27
Error: Time:Sat Nov 18 10:41:09 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:556 curl_easy_perform() failed: Out of memory

Error: Time:Sat Nov 18 10:41:09 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:558 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Sat Nov 18 10:41:09 2017 File:/Users/dllau/azure-iot-sdk-c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Sat Nov 18 10:41:09 2017 File:/Users/dllau/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_step1and2 Line:396 unable to HTTPAPIEX_SAS_ExecuteRequest
Error: Time:Sat Nov 18 10:41:09 2017 File:/Users/dllau/azure-iot-sdk-c/iothub_client/src/iothub_client_ll_uploadtoblob.c Func:IoTHubClient_LL_UploadToBlob_Impl Line:873 error in IoTHubClient_LL_UploadToBlob_step1

@jongio
Copy link
Member

jongio commented Dec 4, 2017

get_device also doesn't work:

from iothub_service_client import IoTHubRegistryManager, IoTHubRegistryManagerAuthMethod
IOTHUB_CONNECTION_STRING = "HostName=.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey="
iothub_registry_manager = IoTHubRegistryManager(IOTHUB_CONNECTION_STRING)
iothub_device = iothub_registry_manager.get_device("device1")

Produces this error:

pi@jongpi6-lite:~/azure-iot-sdk-python/device/tests $ python iothub_device_get_device.py
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:556 curl_easy_perform() failed: Out of memory

Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/adapters/httpapi_curl.c Func:HTTPAPI_ExecuteRequest Line:558 (result = HTTPAPI_OPEN_REQUEST_FAILED)
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/c-utility/src/httpapiex.c Func:HTTPAPIEX_ExecuteRequest Line:475 unable to recover sending to a working state
Error: Time:Mon Dec  4 01:23:44 2017 File:/home/pi/azure-iot-sdk-python/c/iothub_service_client/src/iothub_registrymanager.c Func:sendHttpRequestCRUD Line:982 HTTPAPIEX_SAS_ExecuteRequest failed
Traceback (most recent call last):
  File "iothub_device_get_device.py", line 8, in <module>
    iothub_device = iothub_registry_manager.get_device("device1")
iothub_service_client.IoTHubRegistryManagerError: IoTHubRegistryManager.get_device, IoTHubRegistryManagerResult.HTTPAPI_ERROR

@jongio
Copy link
Member

jongio commented Dec 4, 2017

This is a valid workaround that will remove openssl in favor of gnutls.

cd build_all/linux

./setup.sh

sudo apt remove libcurl4-openssl-dev
sudo apt install libcurl4-gnutls-dev

./build.sh

@ewertons
Copy link
Contributor

Hi @jcwren ,
this issue is related to the root cause described in #308

Could you try upgrading the curl libraries and verify if the issue you are experiencing is resolved?

@drhalftone
Copy link

drhalftone commented Jan 12, 2018 via email

@ewertons
Copy link
Contributor

This is interesting.
The Trusted Certs are being set I believe, since there are no defines preventing that on the sample.
It comes all the way down here, and since we didn't see any failures printed on your logs I would believe they are good.
https://github.com/Azure/azure-c-shared-utility/blob/5db432502576db9f953764346298405c6a6d51b8/adapters/httpapi_curl.c#L795

Just to confirm, your machine time is in correct right? And is it on the EAST coast timezone?

I would perhaps ask to get the CURL verbose logs enabled so we can get any more info.
Due to a bug we have, could you change the code below to 1 and rebuild/re-run the sample please? You should see a lot more info from curl printed on the console. Just share it with us so we can dig more into it.
https://github.com/Azure/azure-c-shared-utility/blob/5db432502576db9f953764346298405c6a6d51b8/adapters/httpapi_curl.c#L132

@ewertons
Copy link
Contributor

Hi @jcwren and @drhalftone ,
we are closing this issue. Once you have a chance to try using the solution posted or if you have any other problems regarding this issue please feel free to reopen it if you need to follow up.
Thanks,
Azure IoT Team

@drhalftone
Copy link

drhalftone commented Jan 29, 2018 via email

@ewertons
Copy link
Contributor

The SDK should not be producing any error messages if everything is going well.

To make sure we have a fresh and clean log, would you mind getting the full output you get (with verbose logging if possible) and paste it here for us to analyze?
Thanks @drhalftone

@ewertons ewertons reopened this Jan 29, 2018
@drhalftone
Copy link

drhalftone commented Jan 29, 2018 via email

@jspaith
Copy link
Contributor

jspaith commented Feb 26, 2018

Closing as it's been a month since last communication. As always, please reactive this issue or open a new one if there's further followups needed. Thanks.

@jspaith jspaith closed this as completed Feb 26, 2018
@ewertons
Copy link
Contributor

Hi @jcwren ,
we will close this issue since it is mitigated for most of the users (including you with the OS you are using).

@drhalftone, we will continue tracking your specific issue through the other ticker we have opened (#308). If you have reasons to keep this one opened as well, please feel free to reopen it.

Thanks,
Azure IoT Team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants