Intro

This is yet another implementation of an intrusive reference counting smart pointer, highly configurable reference counted base class and adapters.

The code requires C++17 or above compiler.

It is known to work with:
Xcode 11 or above
Microsoft Visual Studio 2019 or above
Clang 8 or above
GCC 7.4.0 or above

It can be used either as a classical header-only library or as C++ module (experimental).

Documentation and formal tests are work in progress.

• Why bother?
  • Named conversions from raw pointers
  • No ADL
  • Decent support for output parameters
  • Support for operator->*
  • Atomic access
  • Trivial ABI
  • Correct implementation of a "reference counted base" class
  • Support for weak pointers
• Integration
  • CMake via FetchContent
  • Building and installing on your system
    • Basic use
    • CMake package
    • Via pkg-config
  • Copying to your sources
• Usage
  • Using provided base classes
  • Supporting weak pointers
  • Using with Apple CoreFoundation types
  • Using with Microsoft COM interfaces
  • Using with Python objects
  • Using with non-reference counted types
  • Atomic operations
• Constexpr functionality
• Module support
• Reference

Why bother?

There are multiple other intrusive smart pointers available including one from Boost and nowadays there is even a proposal to add one to the standard C++ library, so why create another one? Unfortunately, as far as I can tell, all existing implementations, and that includes the standard library proposal at the time of this writing, suffer from numerous deficiencies that make them hard or annoying to use in real life code. The most serious problems addressed here are as follows

Named conversions from raw pointers

All other libraries offer a conversion in the form smart_ptr(T * p); In my opinion, this is an extremely bad idea. When looking at a call like smart_ptr(foo()) can you quickly tell whether this adds a reference count or "attaches" the smart pointer to a raw one? That's right, you cannot! The answer depends on the smart pointer implementation or even on specific traits used. This makes the behavior invisible and hard to predict at the call site. It guarantees that someone, somewhere will make a wrong assumption. In my experience, almost all reference counting bugs happen on the boundary between code that uses smart and raw pointers where such conversions are abundant. Just like any form of dangerous cast this one has to be explicit in calling code. As an aside, ObjectiveC ARC did it right with their explicit and visible __bridge casts between raw and smart pointers. Note that having a boolean argument (like what Boost and many other implementations do) in constructor isn't a solution. Can you quickly tell what smart_ptr(p, true) does? Is it "true, add reference" or "true, copy it"?

This library uses named functions to perform conversion. You see exactly what is being done at the call site.

No ADL

Many libraries use ADL to find "add reference" and "release reference" functions for the underlying type. That is, they have expressions like add_ref(p) in their implementation, and expect a function named add_ref that accepts pointer to the underlying type is supposed to be found via ADL. This solution is great in many cases but it breaks when working with some C types like Apple's CTypeRef. This one is actually a typedef to void * so if you have an add_ref that accepts it, you have just made every unrelated void * reference counted (with very bad results if you accidentally put a wrong object into a smart pointer). A better way to define how reference counting is done is to pass a traits class to the smart pointer. (The standard library proposal gets this one right).

This library uses traits.

Decent support for output parameters

Often times you need to pass smart pointer as an output parameter to a C function that takes T ** Many other smart pointers either

• ignore this scenario, requiring you to introduce extra raw pointer and unsafe code, or
• overload operator& which is a horrendously bad idea (it breaks lots of generic code which assumes that &foo gives an address of foo, not something else) The right solution is to have a proxy class convertible to T **. The standard library proposal addresses this problem via generic out_ptr that can work with any smart pointer. If done right, this might be the best solution but the relevant code is not yet widely available anywhere.

This library currently uses an inner proxy class and a get_output_param() method.

Support for operator->*

This might seem to be a minor thing but is really annoying in generic code. For some reason no smart pointers bother to provide operator->* so that pointers to members could be accessed via the same syntax as for raw pointers. In non-generic code you can always work around it via (*p).*whatever but in generic code this is not an option.

Atomic access

Sometimes you need to operate on smart pointers atomically. To the best of my knowledge no library currently provides this functionality.

This library provides a specialization of std::atomic<intrusive_shared_ptr<...>> extending to it the normal std::atomic semantics.

Trivial ABI

When built with CLang compiler intrusive_shared_ptr is marked with [[clang::trivial_abi]] attribute. A good description of what this attribute does and why it is important for performance can be found here. Another take on the performance issue as a comment on standard library proposal can be found here. This page contains details on why this is a good idea and why concerns about order of destruction do not really matter here.

Correct implementation of a "reference counted base" class

This is not directly a problem with smart pointers but with the base classes often provided together with them to implement an intrusively counted class. Very often they contain subtle bugs (see 'A note on implementing reference counted objects' for more details). It is also tricky to create a base class that can work well for different requirements without compromising efficiency.

Support for weak pointers

Continuing on the base class theme, when doing intrusive reference counting, supporting (or not) weak pointers is the responsibility of the counted class. Supporting weak pointers also usually involves tradeoffs in terms of performance or memory consumption. This library base class allows user to enable a decent implementation of weak pointers via policy based design.

Integration

CMake via FetchContent

include(FetchContent)
...
#Uncomment the next line to enable use of C++ module
#set(ISPTR_PROVIDE_MODULE ON)
#Uncomment the next line to enable Python pointers in C++ module 
#set(ISPTR_ENABLE_PYTHON ON)

FetchContent_Declare(isptr
    GIT_REPOSITORY  https://github.com/gershnik/intrusive_shared_ptr.git
    GIT_TAG         v1.5  #use the tag, branch or sha you need
    GIT_SHALLOW     TRUE
)
...
FetchContent_MakeAvailable(isptr)
...
target_link_libraries(mytarget
PRIVATE
  #To use header files:
  isptr::isptr
  #To use C++ module use the following instead (note the *m*):
  #isptrm::isptrm
)

ℹ️ What is FetchContent?

Building and installing on your system

You can also build and install this library on your system using CMake.

1. Download or clone this repository into SOME_PATH
2. On command line:

cd SOME_PATH
cmake -S . -B build 
# If you want to enable use of C++ modules use the following
# cmake -S . -B build -DISPTR_PROVIDE_MODULE=ON
cmake --build build

#Optional
#cmake --build build --target run-test

#install to /usr/local
sudo cmake --install build
#or for a different prefix
#cmake --install build --prefix /usr

Once the library has been installed it can be used int the following ways:

Basic use

To use the header files set the include directory to <prefix>/include where <prefix> is the install prefix from above.

To use C++ module (if enabled during the build) include <prefix>/lib/isptr/isptr.cppm in your build. To have the module expose Python smart pointers make sure you -DISPTR_ENABLE_PYTHON=1 for module file compilation.

CMake package

#Uncomment the next line to enable use of C++ module
#set(ISPTR_PROVIDE_MODULE ON)
#Uncomment the next line to enable Python pointers in C++ module 
#set(ISPTR_ENABLE_PYTHON ON)

find_package(isptr)

target_link_libraries(mytarget
PRIVATE
  #To use header files:
  isptr::isptr
  #To use C++ module use the following instead (note the *m*):
  #isptrm::isptrm
)

Via pkg-config

Add the output of pkg-config --cflags isptr to your compiler flags.

Note that the default installation prefix /usr/local might not be in the list of places your pkg-config looks into. If so you might need to do:

export PKG_CONFIG_PATH=/usr/local/share/pkgconfig

before running pkg-config

Copying to your sources

You can also simply download this repository from Releases page (named intrusive_shared_ptr-X.Y.tar.gz) and unpack it somewhere in your source tree.

To use header files add the inc sub-directory to your include path. To use the module add modules/isptr.cppm to your build. To have the module expose Python smart pointers make sure you -DISPTR_ENABLE_PYTHON=1 for module file compilation.

Usage

All the types in this library are declared in namespace isptr. For brevity the namespace is omitted below. Add isptr:: prefix to all the type or use using declaration in your own code.

The header <intrusive_shared_ptr/intrusive_shared_ptr.h>/module isptr provides a template

template<class T, class Traits>
class intrusive_shared_ptr<T, Traits>;

Where T is the type of the pointee and Traits a class that should provide 2 static functions that look like this

static void add_ref(SomeType * ptr) noexcept
{  
    //increment reference count. ptr is guaranteed to be non-nullptr
}
static void sub_ref(SomeType * ptr) noexcept
{ 
    //decrement reference count. ptr is guaranteed to be non-nullptr
}

SomeType * should be a pointer type to which T * is convertible to. It is possible to make add_ref and sub_ref templates, if desired, though this is usually not necessary.

To create intrusive_shared_ptr from a raw T * there are 2 functions:

//pass the smart pointer in without changing the reference count
template<class T, class Traits>
intrusive_shared_ptr<T, Traits> intrusive_shared_ptr<T, Traits>::noref(T * p) noexcept;

//adopt the pointer and bump the reference count
template<class T, class Traits>
intrusive_shared_ptr<T, Traits> intrusive_shared_ptr<T, Traits>::ref(T * p) noexcept

It is possible to use intrusive_shared_ptr directly but the name is long and ugly so a better approach is to wrap in a typedef and wrapper functions like this

struct my_type
{};

struct my_intrusive_traits
{
    static void add_ref(my_type * ptr) noexcept; //implement
    static void sub_ref(my_type * ptr) noexcept; //implement
};

template<class T>
using my_ptr = intrusive_shared_ptr<T, my_intrusive_traits>;

template<class T> 
my_ptr<T> my_retain_func(T * ptr) {
    return my_ptr<T>::ref(ptr);
}
template<class T> 
my_ptr<T> my_attach_func(T * ptr) {
    return my_ptr<T>::noref(ptr);
}

The library provides such wrappers for some common scenarios. If you fully control the definition of my_type then it is possible to simplify things even further with header refcnt_ptr.h. It adapts intrusive_shared_ptr to traits exposed as inner type refcnt_ptr_traits. You can use it like this:

#include <intrusive_shared_ptr/refcnt_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

struct my_type
{
  struct refcnt_ptr_traits
  {
      static void add_ref(my_type * ptr) noexcept; //implement
      static void sub_ref(my_type * ptr) noexcept; //implement
  };
};

//now you can use refcnt_ptr<my_type> for the pointer type and refcnt_attach and refcnt_retain free functions e.g.

//create from raw pointer (created with count 1)
foo raw = new my_type();
refcnt_ptr<my_type> p1 = refcnt_attach(raw);

//create directly
auto p1 = make_refcnt<my_type>();

//assign from raw pointer bumping reference count
refcnt_ptr<my_type> p2;
p2 = refcnt_retain(raw);

Using provided base classes

To implement my_type above the library provides a base class you can inherit from which will do the right thing.

#include <intrusive_shared_ptr/ref_counted.h>
#include <intrusive_shared_ptr/refcnt_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

class foo : ref_counted<foo>
{
    friend ref_counted;
public:    
    void method();
private:
    ~foo() noexcept = default; //prevent manual deletion
};

//you can use auto to declare p1, p2 and p3. The full type is spelled out for
//demonstration purposes only

//attach from raw pointer (created with count 1)
refcnt_ptr<foo> p1 = refcnt_attach(new foo()); 

//create directly
refcnt_ptr<foo> p2 = make_refcnt<foo>();

//assign from raw pointer bumping reference count
foo * raw = ...
refcnt_ptr<foo> p3 = refcnt_retain(raw);

The type of the reference count is int by default. If you need to you can customize it.

class tiny : ref_counted<tiny, ref_counted_flags::none, char> //use char as count type
{
  friend ref_counted;

  char c;
};

static_assert(sizeof(tiny) == 2);

More details can be found in this document

Supporting weak pointers

If you want to support weak pointers you need to tell ref_counted about it. Since weak pointers include overhead even if you never create one by default they are disabled.

#include <intrusive_shared_ptr/ref_counted.h>
#include <intrusive_shared_ptr/refcnt_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

class foo : weak_ref_counted<foo> //alias for ref_counted<foo, ref_counted_flags::provide_weak_references>
{
    void method();
};

refcnt_ptr<foo> p1 = refcnt_attach(new foo());
foo::weak_ptr w1 = p1->get_weak_ptr();
refcnt_ptr<foo> p2 = w1->lock();

Note that you cannot customize the type of reference count if you support weak pointers - it will always be intptr_t. More details can be found in this document

Using with Apple CoreFoundation types

#include <intrusive_shared_ptr/apple_cf_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

//Use auto in real code. Type is spelled out for clarity
cf_ptr<CStringRef> str = cf_attach(CFStringCreateWithCString(nullptr, 
                                                             "Hello", 
                                                             kCFStringEncodingUTF8));
std::cout << CFStringGetLength(str.get());

CFArrayRef raw = ...;
//Use auto in real code.
cf_ptr<CFArrayRef> array = cf_retain(raw);

Using with Microsoft COM interfaces

#include <intrusive_shared_ptr/com_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

com_shared_ptr<IStream> pStream;
CreateStreamOnHGlobal(nullptr, true, pStream.get_output_param());
pStream->Write(....);

Using with Python objects

#include <intrusive_shared_ptr/python_ptr.h>
//Or, if using modules:
//import isptr;

using namespace isptr;

auto str = py_attach(PyUnicode_FromString("Hello"));
std::cout << PyUnicode_GetLength(str.get());

Note that to use Python smart pointers with C++ module you need ensure -DISPTR_ENABLE_PYTHON=1 is used for module file compilation. When using CMake this is accomplished by set(ISPTR_ENABLE_PYTHON ON) in CMake code or via -DISPTR_ENABLE_PYTHON=ON during configuration.

Using with non-reference counted types

On occasion when you have a code that uses intrusive reference counting a lot you might need to handle a type which you cannot modify and which is not by itself reference counted. In such situation you can use an adapter (if you prefer derivation) or wrapper (if you prefer containment) that makes it such

Adapter:

#include <intrusive_shared_ptr/ref_counted.h>
//Or, if using modules:
//import isptr;

using counted_map = ref_counted_adapter<std::map<string, int>>;

auto ptr = make_refcnt<counted_map>();
(*ptr)["abc"] = 7;
std::cout << ptr->size();

using weakly_counted_map = weak_ref_counted_adapter<std::map<string, int>>;

auto ptr1 = make_refcnt<weakly_counted_map>();
(*ptr1)["abc"] = 7;
std::cout << ptr1->size();
foo::weak_ptr w1 = p1->get_weak_ptr();
refcnt_ptr<weakly_counted_map> p2 = w1->lock();

Wrapper:

#include <intrusive_shared_ptr/ref_counted.h>
//Or, if using modules:
//import isptr;

using counted_map = ref_counted_wrapper<std::map<string, int>>;

auto ptr = make_refcnt<counted_map>();
ptr->wrapped()["abc"] = 7;
std::cout << ptr->wrapped().size();

using weakly_counted_map = weak_ref_counted_wrapper<std::map<string, int>>;

auto ptr1 = make_refcnt<weakly_counted_map>();
ptr1->wrapped()["abc"] = 7;
std::cout << ptr1->wrapped().size();
foo::weak_ptr w1 = p1->get_weak_ptr();
refcnt_ptr<weakly_counted_map> p2 = w1->lock();

Atomic operations

The library provides a partial specialization

template <class T, class Traits>
std::atomic<intrusive_shared_ptr<T, Traits>>;

which exposes normal std::atomic functionality. For example:

using my_ptr = intrusive_shared_ptr<my_type, my_intrusive_traits>;

using my_atomic_ptr = std::atomic<my_ptr>;

my_ptr ptr = ...;
my_atomic_ptr aptr = ptr;

ptr = aptr.load();
//or
ptr = aptr;

aptr.store(ptr);
//or
aptr = ptr;

my_ptr ptr1 = aptr.exchange(ptr);

//etc.

Constexpr functionality

When built with C++20 compiler intrusive_shared_ptr is fully constexpr capable. You can do things like

using my_ptr = intrusive_shared_ptr<my_type, my_intrusive_traits>;

constexpr my_ptr foo;

Due to non-default destructors this functionality is not available on C++17

Module support

Since version 1.5 this library support being used as a C++ module. This mode is currently experimental. Please report bugs if you encounter any issues.

In order to use C++ modules you need a compiler that supports them. Currently CLang >= 16 and MSVC toolset >= 14.34 are definitely known to work. Other compilers/versions may or may not work.

If using CMake follow the requirements at cmake-cxxmodules. In order to enable module support for this library you need to set ISPTR_PROVIDE_MODULE CMake variable to ON before referencing it.

The library consists of a single module file at modules/isptr.cppm. This file is auto-generated from all the library headers.

One notable difference between headers and module use concerns Python pointers. With header files you can simply control whether to use them by including or not including <intrusive_shared_ptr/python_ptr.h>.

With module, which is compiled separately, you need to tell the module file whether to enable Python pointers (and use <Python.h> header) ahead of time.

If you compile module yourself you can control whether Python pointers are enabled by setting -DISPTR_ENABLE_PYTHON=1 for its compilation and make sure the include path contains <Python.h>.

If you use CMake then you need to set CMake option ISPTR_ENABLE_PYTHON to ON either from command line or in CMake code before referencing this library. With this variable set to ON the CMake script will

• Perform find_package (Python3 COMPONENTS Development REQUIRED) if Python development component hasn't been already found.
• Add Python3_INCLUDE_DIRS to module include path and define ISPTR_ENABLE_PYTHON=1
• Add Python3_LIBRARIES to library dependencies.

You can control which Python installation to use by controlling find_package (Python3) (or calling it yourself ahead of time) as described in FindPython3

Reference

• intrusive_shared_ptr.h
• refcnt_ptr.h
• ref_counted.h