GPU streams and synchronization
update_halo! runs on streams of its own: it does not wait for work enqueued before the call, which must therefore be synchronized before, and it returns once the halo update is complete (see Asynchronous halo updates and overlap with computation).
CUDA.jl (from v6.3) and AMDGPU.jl moreover record the stream that last used an array and make a use on another stream wait for it (implicit synchronization). For update_halo!, this means:
It does not replace the synchronization before the call: it orders the kernels that
update_halo!uses for dimension 1 after preceding work on the array, but not the memory copies it uses for dimensions 2 and 3.It prevents overlapping the computation of an array with the update of its halo. To overlap them, disable it for the arrays concerned and order the work explicitly:
CUDA.enable_synchronization!(A, false) # AMDGPU.enable_synchronization!(A, false) with AMDGPU.jl ≥ 2.11
# ... computation of A on another stream, update_halo!(A), synchronization of that stream ...
CUDA.enable_synchronization!(A, true)@hide_communication and @overlap of ParallelStencil.jl do this automatically (from ParallelStencil 0.17.2 on).
Behaviour per version
A kernel uses the array A on stream s1; while it runs, A or another array is used on stream s2. Host waits: the call returns only once s1 is done. GPU waits: the call returns, but the work on s2 starts only once s1 is done. Concurrent: no wait.
Use on s2 | CUDA.jl 5.4–6.2 | CUDA.jl 6.3–6.4 | CUDA.jl ≥ 6.5 | AMDGPU.jl ≥ 2.8 |
|---|---|---|---|---|
Kernel launch using A | concurrent | host waits | GPU waits | host waits |
pointer(A) (e.g. for a memory copy) | concurrent | host waits | host waits | host waits |
pointer(A), if s1 is the task-local stream | concurrent | concurrent | concurrent | concurrent |
| Kernel launch using another array | concurrent | concurrent | concurrent | concurrent |
Any of the above after enable_synchronization!(A, false) | concurrent | concurrent | concurrent | concurrent |
This table is checked weekly on Nvidia GH200 and AMD MI250X GPUs with an old and the latest version of each package (see Continuous integration).