Stopping server with active connections causes crash

[mdavidsaver]

While attempting to fix shared_ptr related problems with ServerContext (so that it is possible to actually stop a server and make it drop references to its ChannelProviders) I find that pvaSrv has some problems related to handling CA contexts which manifest when the PVA server is stopped while clients are still connected. I suspect that there are some incorrect assumptions here about ordering during shutdown.

'stopPVAServer' is a new iocsh command I'm adding.

epics> stopPVAServer
2017-05-31T15:55:06.438 Server context still has 1 transport(s) active and closing...
caContext::stop referenceCount != 0 value 1

or

epics> stopPVAServer
2017-05-31T16:24:11.391 Server context still has 1 transport(s) active and closing...
CA.Client.Exception...............................................
    Warning: "Thread is already attached to a client context"
    Context: "Segmentation fault

I don't plan to spend any time trying to fix this in pvaSrv.

[mdavidsaver]

In caContext.cpp the function ca_attach_context() is called after the epicsExit() has run ca_client_exit_handler() which deletes the thread local variable used to hold CA context pointers.

2017-08-09T19:49:36.120 Server context still has 1 transport(s) active and closing...

Thread 1 "softIocPVA" received signal SIGSEGV, Segmentation fault.
0x00007ffff654bf3e in epicsThreadPrivateGet (id=0x0) at ../../../src/libCom/osi/os/posix/osdThread.c:888
888         return pthread_getspecific(*key);
(gdb) bt
#0  0x00007ffff654bf3e in epicsThreadPrivateGet (id=0x0) at ../../../src/libCom/osi/os/posix/osdThread.c:888
#1  0x00007ffff679bd07 in ca_attach_context (pCtx=0x7fffa8002690) at ../access.cpp:761
#2  0x00007ffff7b81011 in caContext::checkContext (this=0x7fffa80025f0) at ../../src/dbPv/3.15/caContext.cpp:107
#3  0x00007ffff7b9bbb6 in CaMonitorPvt::stop (this=0x7fffa8002550) at ../../src/dbPv/3.15/caMonitor.cpp:261
#4  0x00007ffff7b9bd47 in CaMonitor::stop (this=0x7fffa8002530) at ../../src/dbPv/3.15/caMonitor.cpp:296
#5  0x00007ffff7bb2180 in epics::pvaSrv::DbPvMonitor::stop (this=0x7fffa8001aa0) at ../../src/dbPv/3.15/dbPvMonitor.cpp:204
#6  0x00007ffff7bb1dde in epics::pvaSrv::DbPvMonitor::destroy (this=0x7fffa8001aa0) at ../../src/dbPv/3.15/dbPvMonitor.cpp:165
#7  0x00007ffff7408960 in epics::pvAccess::ServerMonitorRequesterImpl::destroy (this=0x7fffa80018d0)
    at ../../src/server/responseHandlers.cpp:2049
#8  0x00007ffff7429fe3 in epics::pvAccess::ServerChannelImpl::destroy (this=0x7fffa8001000) at ../../src/server/serverChannelImpl.cpp:96
#9  0x00007ffff73b125a in epics::pvAccess::detail::BlockingServerTCPTransportCodec::destroyAllChannels (this=0x7fffa40008c0)
    at ../../src/remote/codec.cpp:1561
#10 0x00007ffff73b131f in epics::pvAccess::detail::BlockingServerTCPTransportCodec::internalClose (this=0x7fffa40008c0, force=true)
    at ../../src/remote/codec.cpp:1569
#11 0x00007ffff73af3aa in epics::pvAccess::detail::BlockingTCPTransportCodec::close (this=0x7fffa40008c0)
    at ../../src/remote/codec.cpp:1025
#12 0x00007ffff7424e46 in epics::pvAccess::ServerContextImpl::destroyAllTransports (this=0x55555583bed0)
    at ../../src/server/serverContext.cpp:371
#13 0x00007ffff7424baf in epics::pvAccess::ServerContextImpl::shutdown (this=0x55555583bed0) at ../../src/server/serverContext.cpp:341
#14 0x00007ffff7425b68 in epics::pvAccess::(anonymous namespace)::shutdown_dtor::operator() (this=0x55555585d940, self=0x55555583bed0)
    at ../../src/server/serverContext.cpp:571
#15 0x00007ffff7426a52 in std::_Sp_counted_deleter<epics::pvAccess::ServerContextImpl*, epics::pvAccess::(anonymous namespace)::shutdown_dtor, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55555585d930) at /usr/include/c++/6/bits/shared_ptr_base.h:464
#16 0x00007ffff7b81f38 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55555585d930)
    at /usr/include/c++/6/bits/shared_ptr_base.h:150
#17 0x00007ffff7b8195b in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffff78d33b8 <the_server+8>, 
    __in_chrg=<optimized out>) at /usr/include/c++/6/bits/shared_ptr_base.h:662
#18 0x00007ffff76d0242 in std::__shared_ptr<epics::pvAccess::ServerContext, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (
    this=0x7ffff78d33b0 <the_server>, __in_chrg=<optimized out>) at /usr/include/c++/6/bits/shared_ptr_base.h:928
#19 0x00007ffff76d025e in std::shared_ptr<epics::pvAccess::ServerContext>::~shared_ptr (this=0x7ffff78d33b0 <the_server>, 
    __in_chrg=<optimized out>) at /usr/include/c++/6/bits/shared_ptr.h:93
#20 0x00007ffff5903c7f in __cxa_finalize (d=0x7ffff78d32c8) at cxa_finalize.c:56
#21 0x00007ffff76cf5a3 in __do_global_dtors_aux ()
   from /home/mdavidsaver/work/epics/pv/pvAccess/lib/linux-x86_64-debug/libpvAccessIOC.so.6.0.0
#22 0x00007fffffffdd70 in ?? ()
#23 0x00007ffff7de8e7a in _dl_fini () at dl-fini.c:235
Backtrace stopped: frame did not save the PC