Resolution:
When creating custom resource synchronization/mutex between the submission thread(main thread involves in the submit method) and the collection thread( handleReponse and handleError method), the following synchronization fashion in Windows or POSIX will cause a deadlock when a service cancellation happens:

Submission thread: EnterCriticalSection/pthread_mutex_lock&gtSubmit&gtLeaveCriticalSection/pthread_mutex_unlock

Collection thread: handleError() { EnterCriticalSection/pthread_mutex_lock … LeaveCriticalSection/pthread_mutex_unlock }

The origin of the deadlock is: unlike the normal response and error handling, when the service cancellation happens( while the driver is still submitting), the collection thread locks another internal resource before calling the handleerror method which attempts to lock the user-defined mutex. Since the user-define mutex has been locked by the main thread before submit, the handleerror is then waiting for the mutex being unlocked while the submit method is attempting to obtain the internal resource so that it can submit, hence the deadlock happens

The following two ways can be taken to avoid the deadlock:

   1. Conduct the synchronization before or after the Submit method in submission thread, e.g:

EnterCriticalSection/pthread_mutex_lock&gtLeaveCriticalSection/pthread_mutex_unlock&gtSubmit

Or:

Submit&gtEnterCriticalSection/pthread_mutex_lock&gtLeaveCriticalSection/pthread_mutex_unlock

This way the Submit method is not involved within the synchronization body, so the deadlock will not happen.

2. Identify the exception type of the LiveClusterException after a submit try, if it's an exception resulting from a service cancellation, just break the submission loop, since there is no meaning to continue with the submission at this point. This way will also not incur deadlock.

Notes on custom resource synchronization with CPPDriver