Skip to content

GPU streams and synchronization ​

update_halo! runs on streams of its own: it does not wait for work enqueued before the call, which must therefore be synchronized before, and it returns once the halo update is complete (see Asynchronous halo updates and overlap with computation).

CUDA.jl (from v6.3) and AMDGPU.jl moreover record the stream that last used an array and make a use on another stream wait for it (implicit synchronization). For update_halo!, this means:

  • It does not replace the synchronization before the call: it orders the kernels that update_halo! uses for dimension 1 after preceding work on the array, but not the memory copies it uses for dimensions 2 and 3.

  • It prevents overlapping the computation of an array with the update of its halo. To overlap them, disable it for the arrays concerned and order the work explicitly:

julia
CUDA.enable_synchronization!(A, false)   # AMDGPU.enable_synchronization!(A, false) with AMDGPU.jl ≥ 2.11
# ... computation of A on another stream, update_halo!(A), synchronization of that stream ...
CUDA.enable_synchronization!(A, true)

@hide_communication and @overlap of ParallelStencil.jl do this automatically (from ParallelStencil 0.17.2 on).

Behaviour per version ​

A kernel uses the array A on stream s1; while it runs, A or another array is used on stream s2. Host waits: the call returns only once s1 is done. GPU waits: the call returns, but the work on s2 starts only once s1 is done. Concurrent: no wait.

Use on s2CUDA.jl 5.4–6.2CUDA.jl 6.3–6.4CUDA.jl ≥ 6.5AMDGPU.jl ≥ 2.8
Kernel launch using Aconcurrenthost waitsGPU waitshost waits
pointer(A) (e.g. for a memory copy)concurrenthost waitshost waitshost waits
pointer(A), if s1 is the task-local streamconcurrentconcurrentconcurrentconcurrent
Kernel launch using another arrayconcurrentconcurrentconcurrentconcurrent
Any of the above after enable_synchronization!(A, false)concurrentconcurrentconcurrentconcurrent

This table is checked weekly on Nvidia GH200 and AMD MI250X GPUs with an old and the latest version of each package (see Continuous integration).