feat(http): implement http client based on libcurl

[empiredan]

#1582

Http client is implemented based on libcurl. Both GET and POST methods
are supported.

[neverchanje]

I recommend this library for more modern http https://github.com/yhirose/cpp-httplib

[empiredan]

I recommend this library for more modern http https://github.com/yhirose/cpp-httplib

Thanks for your recommendation ! It seems that this lib provides a set of very elegant and easy-to-use APIs for http client. In addition, it also supports building http server. Later on I'll look deep into this lib.

[acelyc111]

I recommend this library for more modern http https://github.com/yhirose/cpp-httplib

+1, it's not necessary to invent the wheel again :)

[empiredan]

Originally, referencing libcurl into pegasus shell directly would lead to hidden symbol XXX in YYY is referenced by DSO ...:

This error could be eliminated by introducing http client. However, compilation failed for macOS:

Undefined symbols for architecture x86_64:
  "_nghttp2_is_fatal", referenced from:
      _on_frame_recv in libcurl.a(libcurl_la-http2.o)
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
      _http2_recv in libcurl.a(libcurl_la-http2.o)
      _http2_send in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_pack_settings_payload", referenced from:
      _Curl_http2_request_upgrade in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_priority_spec_init", referenced from:
      _h2_pri_spec in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_del", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_new", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_before_frame_send_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_begin_headers_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_data_chunk_recv_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_frame_not_send_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_frame_recv_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_frame_send_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_header_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_invalid_frame_recv_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_on_stream_close_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_callbacks_set_send_callback", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_client_new", referenced from:
      _Curl_http2_init in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_del", referenced from:
      _http2_disconnect in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_get_remote_settings", referenced from:
      _on_frame_recv in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_get_stream_user_data", referenced from:
      _on_frame_recv in libcurl.a(libcurl_la-http2.o)
      _on_invalid_frame_recv in libcurl.a(libcurl_la-http2.o)
      _on_data_chunk_recv in libcurl.a(libcurl_la-http2.o)
      _before_frame_send in libcurl.a(libcurl_la-http2.o)
      _on_frame_send in libcurl.a(libcurl_la-http2.o)
      _on_frame_not_send in libcurl.a(libcurl_la-http2.o)
      _on_stream_close in libcurl.a(libcurl_la-http2.o)
      ...
  "_nghttp2_session_mem_recv", referenced from:
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
      _http2_recv in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_resume_data", referenced from:
      _http2_send in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_send", referenced from:
      _http2_send in libcurl.a(libcurl_la-http2.o)
      _h2_session_send in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_set_stream_user_data", referenced from:
      _Curl_http_done in libcurl.a(libcurl_la-http.o)
      _on_frame_recv in libcurl.a(libcurl_la-http2.o)
      _on_stream_close in libcurl.a(libcurl_la-http2.o)
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_upgrade", referenced from:
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_want_read", referenced from:
      _http2_getsock in libcurl.a(libcurl_la-http2.o)
      _http2_perform_getsock in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_session_want_write", referenced from:
      _http2_getsock in libcurl.a(libcurl_la-http2.o)
      _http2_perform_getsock in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_strerror", referenced from:
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
      _http2_recv in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_submit_priority", referenced from:
      _h2_session_send in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_submit_request", referenced from:
      _http2_send in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_submit_rst_stream", referenced from:
      _on_frame_recv in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_submit_settings", referenced from:
      _Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
  "_nghttp2_version", referenced from:
      _Curl_http2_ver in libcurl.a(libcurl_la-http2.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/http/test/dsn_http_test] Error 1
make[1]: *** [src/http/test/CMakeFiles/dsn_http_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Then after libcurl was bumped to 8.3.0 from 7.47.0, compilation failed for ubuntu:

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: /data/sa_cluster/src/empiredan-pegasus-new/thirdparty/output/lib/libcurl.a(libcurl_la-content_encoding.o): undefined reference to symbol 'ZSTD_isError'
//lib/libzstd.so.1.3.7: error adding symbols: DSO missing from command line

compilation also failed for macOS:

Undefined symbols for architecture x86_64:
  "_BrotliDecoderCreateInstance", referenced from:
      _brotli_init_writer in libcurl.a(libcurl_la-content_encoding.o)
  "_BrotliDecoderDecompressStream", referenced from:
      _brotli_unencode_write in libcurl.a(libcurl_la-content_encoding.o)
  "_BrotliDecoderDestroyInstance", referenced from:
      _brotli_unencode_write in libcurl.a(libcurl_la-content_encoding.o)
      _brotli_close_writer in libcurl.a(libcurl_la-content_encoding.o)
  "_BrotliDecoderGetErrorCode", referenced from:
      _brotli_unencode_write in libcurl.a(libcurl_la-content_encoding.o)
  "_idn2_check_version", referenced from:
      _Curl_idn_decode in libcurl.a(libcurl_la-idn.o)
      _Curl_idnconvert_hostname in libcurl.a(libcurl_la-idn.o)
  "_idn2_free", referenced from:
      _Curl_idn_decode in libcurl.a(libcurl_la-idn.o)
      _Curl_idn_encode in libcurl.a(libcurl_la-idn.o)
      _Curl_free_idnconverted_hostname in libcurl.a(libcurl_la-idn.o)
      _Curl_idnconvert_hostname in libcurl.a(libcurl_la-idn.o)
  "_idn2_lookup_ul", referenced from:
      _Curl_idn_decode in libcurl.a(libcurl_la-idn.o)
      _Curl_idnconvert_hostname in libcurl.a(libcurl_la-idn.o)
  "_idn2_to_unicode_8z8z", referenced from:
      _Curl_idn_encode in libcurl.a(libcurl_la-idn.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/http/test/dsn_http_test] Error 1
make[1]: *** [src/http/test/CMakeFiles/dsn_http_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Therefore, libcurl is built with --without-zstd, and following options especially for macOS:

--without-nghttp2
--without-libidn2
--without-brotli

[acelyc111]

Is the output for debuging? If you want to check what the actual value is when test failed, you can use the macro like this:

ASSERT_*(x,x) << actual_description;

[empiredan]

The reason why this description is printed is that its postfix, i.e. msg field, as is described above, differ in various systems. I think the different message could be shown for each various system.