CUDA Error Handling including timeouts
========================================


Handling / Reset ?
-------------------

* http://stackoverflow.com/questions/14038589/what-is-the-canonical-way-to-check-for-errors-using-the-cuda-runtime-api
* http://stackoverflow.com/questions/19632401/how-to-work-around-gpu-watchdog-timer-limitation-on-cuda-code-in-os-x
* http://stackoverflow.com/questions/9602312/gpu-card-resets-after-2-seconds


Compute and Graphics
---------------------

Using GPU for both, forces use of timeouts.

* https://devtalk.nvidia.com/default/topic/483643/cuda-the-launch-timed-out-and-was-terminated/
* https://devtalk.nvidia.com/search/more/sitecommentsearch/Launch%20timeout/


CUDA Driver API Errors
--------------------------


* http://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TYPES.html#group__CUDA__TYPES


`CUDA_ERROR_LAUNCH_FAILED = 700`
    An exception occurred on the device while executing a kernel. Common causes
    include dereferencing an invalid device pointer and accessing out of bounds
    shared memory. The context cannot be used, so it must be destroyed (and a new
    one should be created). All existing device memory allocations from this
    context are invalid and must be reconstructed if the program is to continue
    using CUDA.


`CUDA_ERROR_LAUNCH_TIMEOUT = 702`
    This indicates that the device kernel took too long to execute. This can only
    occur if timeouts are enabled - see the device attribute
    CU_DEVICE_ATTRIBUTE_KERNEL_EXEC_TIMEOUT for more information. 
    The context cannot be used (and must be destroyed similar to CUDA_ERROR_LAUNCH_FAILED). All
    existing device memory allocations from this context are invalid and must be
    reconstructed if the program is to continue using CUDA.



`CU_DEVICE_ATTRIBUTE_KERNEL_EXEC_TIMEOUT = 17`
    Specifies whether there is a run time limit on kernels


deviceQuery
--------------

::

    delta:w blyth$ cuda-samples-bin-deviceQuery | grep limit 
      Run time limit on kernels:                     Yes