Add logging to libcudf

[vuule]

Description

Adds a global spdlog logger.
Logging should be done through CUDF_LOG_* macros declared in detail namespace.
Logging level and output file path can be set via environment variables.
Logging can also be configured directly through the singleton object using spdlog APIs (should only be used outside of libcudf).

Added sample (useful) logging to getenv_or.
Fixed a bug discovered through the log :) Environment variable was being read in each hostdevice_vector constructor call.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (branch-23.04@e4557cb). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head f0ae3e2 differs from pull request most recent head 855fcc0. Consider uploading reports for the commit 855fcc0 to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-23.04   #12637   +/-   ##
===============================================
  Coverage                ?   43.53%           
===============================================
  Files                   ?      158           
  Lines                   ?    24868           
  Branches                ?        0           
===============================================
  Hits                    ?    10827           
  Misses                  ?    14041           
  Partials                ?        0           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[bdice]

FYI, we need to fix some issues with the way rmm is packaging/using spdlog. I have a PR here that moves in that direction, but more changes may be needed. Please don’t merge this PR until I can confirm that rmm is fixed up. See: rapidsai/rmm#1177

[jrhemstad]

You may want to make the design a bit more flexible such that it doesn't assume one is always logging to a file.

@jlowe or @revans2 can correct me if I'm wrong, but iirc Spark wants to be able to log to something like cout/cerr. In the RMM logging adaptor we enabled this by adding a constructor that takes an arbitrary std::ostream: https://github.com/rapidsai/rmm/blob/58ed5c9c373e8859bb009b32cd1f7dd0522e1a7d/include/rmm/mr/device/logging_resource_adaptor.hpp#L78-L98

[jlowe]

Spark wants to be able to log to something like cout/cerr

We'd like the ability to glue the cudf logging into the existing Spark/Java logging mechanisms, e.g.: log4j. Being able to specify a custom spdlog sink, ideally at runtime, would give us the flexibility to do that and more.

[vuule]

The singleton spdlog object is accessible via cudf::logger() (unless I broke something when moving definitions to .cpp). It can be used to fully configure logging, including a custom sink.

[vuule]

One caveat: the API that changes the sink is not thread-safe. @jlowe is this an issue?
Here's a sample of how logger can be modified at runtime (will add samples to PR through tests):

std::ostringstream oss;
// Divert the log to the string stream (not thread-safe!)
cudf::logger().sinks() = {std::make_shared<spdlog::sinks::ostream_sink_mt>(oss)};
// Only include the log message (no timestamp, etc.)
cudf::logger().set_formatter(
  std::unique_ptr<spdlog::formatter>(new spdlog::pattern_formatter("%v")));
// Only log warnings and worse
cudf::logger().set_level(spdlog::level::warn);

[bdice]

This PR appears to rely on rmm's spdlog/fmt transitively. I think we want to update our conda recipes and/or CPM logic to explicitly depend on spdlog/fmt? What should our approach be for downstream packages like cudf @robertmaynard ? I'd think the conda packages should be explicit dependencies but I'm not sure how to handle the CMake/CPM side.

[bdice]

The fmt/spdlog conda packaging looks good. The fmt/spdlog CMake packaging looks good to me also, but I would like a second eye from @robertmaynard before merging.

outdated

[robertmaynard]

The fmt/spdlog conda packaging looks good. The fmt/spdlog CMake packaging looks good to me also, but I would like a second eye from @robertmaynard before merging.

Looks good to me

[harrism]

We should see if the numbers in rapidsai/rmm#1222 also accurately capture what happens to compile times here.

Has anything happened with this? I'm curious how urgent it is to change RMM's spdlog dependency to use the pre-compiled library.

[vuule]

We should see if the numbers in rapidsai/rmm#1222 also accurately capture what happens to compile times here.

Has anything happened with this? I'm curious how urgent it is to change RMM's spdlog dependency to use the pre-compiled library.

I tried using the build metrics to evaluate the impact of spdlog dependency. There's a measurable impact on .so size, but compilation time seems hard to compare because of caching. The time difference does not look alarming given the difference between cache hit rates.
no logging

with logging

[vyasr]

Couple of minor things but looks basically ready to go.

[vyasr]

I don't think you would run into any issues. RMM only sets this for compiled components i.e. its tests and benchmarks. The main rmm code is header-only so there are no compiled components where SPDLOG_ACTIVE_LEVEL would be set to conflict with whatever libcudf sets.

[vuule]

/merge