Usage
Multi-GPU with three functions
Only three functions are required to perform halo updates close to hardware limit:
The 50-lines Multi-GPU example illustrates how these functions enable the creation of massively parallel applications.
Global coordinates, sizes, indices and extents
Three functions are provided to query Cartesian coordinates with respect to the global computational grid if required:
Moreover, the following three functions allow to query the size of the global grid:
Further tools: global indices (ix_g, iy_g, iz_g), local and global extents of arrays, e.g. for parallel VTK output (extents, extents_g, metagrid) and timing (tic, toc).
GPU support and device selection
Support for Nvidia or AMD GPUs is activated by importing CUDA.jl or AMDGPU.jl before ImplicitGlobalGrid:
using ImplicitGlobalGridusing CUDA
using ImplicitGlobalGridusing AMDGPU
using ImplicitGlobalGridBy default, init_global_grid selects the GPU corresponding to the node-local MPI rank (see select_device); every rank must then see all GPUs of its node. If the job launcher already binds one GPU per rank (e.g. Slurm --gpus-per-task=1), set select_device=false. To run on CPUs only while a GPU package is loaded, set device_type="none".
CUDA-/ROCm-aware MPI support
If the system supports CUDA-aware MPI (for Nvidia GPUs) or ROCm-aware MPI (for AMD GPUs), it can be activated for ImplicitGlobalGrid by setting the following environment variable (at latest before the call to init_global_grid):
export IGG_CUDAAWARE_MPI=1export IGG_ROCMAWARE_MPI=1It can also be activated per dimension, see Environment variables.
GPU-aware MPI with Cray MPICH
With Cray MPICH (HPE Cray EX systems), ImplicitGlobalGrid allocates the device buffers it uses for CUDA-/ROCm-aware MPI outside the CUDA.jl/AMDGPU.jl memory pools (with cuMemAlloc/hipMalloc), because the GPU transport layer of Cray MPICH does not accept pool memory (on Nvidia GPUs, the legacy CUDA IPC API it uses rejects stream-ordered memory). JULIA_CUDA_MEMORY_POOL=none is therefore not needed for ImplicitGlobalGrid (only if you pass your own arrays to MPI). ROCm-aware MPI requires MPICH_GPU_SUPPORT_ENABLED=1 and, currently, HSA_ENABLE_IPC_MODE_LEGACY=1: the GPU IPC path of the transport layer (between processes on the same node) fails with the default IPC mode of the ROCm runtime (observed with ROCm 7.2.3 and Cray MPICH 9.1.0). Alternatively, MPICH_GPU_IPC_ENABLED=0 disables this path, but the exchange between processes on the same node is then much slower (see issue #143). Your own arrays are never passed to MPI by ImplicitGlobalGrid, so no change is needed in your code.
Asynchronous halo updates and overlap with computation
For GPU arrays, update_halo! performs the halo update on streams of its own (non-blocking, with high priority), independently of the task-local stream on which CUDA.jl and AMDGPU.jl enqueue work by default. This is by design: it makes it straightforward to overlap the halo update with computation. The contract is:
update_halo!does, by default, not synchronize with work enqueued before the call, neither on the task-local stream nor on any other stream (automatic synchronization of the task-local stream can be activated in the call toinit_global_gridwith the keywordauto_task_sync=true; the synchronization then covers only the task-local stream).Work that must not run at the same time as the halo update, typically the computation of the arrays passed to
update_halo!or ideally the computation of only the boundary regions of these arrays, must therefore be synchronized before the call by default.update_halo!returns only once the halo update is complete: no synchronization is needed after the call.
@cuda blocks=nb threads=nt compute!(A, B) # Enqueued on the task-local stream.
CUDA.synchronize() # Complete it before the halo update.
update_halo!(A)@roc gridsize=nb groupsize=nt compute!(A, B) # Enqueued on the task-local stream.
AMDGPU.synchronize() # Complete it before the halo update.
update_halo!(A)compute!(backend)(A, B; ndrange=size(A))
KernelAbstractions.synchronize(backend)
update_halo!(A)To overlap the halo update with computation, compute the boundary regions of the arrays first and synchronize, then launch the computation of the inner points on another stream without synchronizing, update the halos and finally synchronize that stream:
@cuda blocks=nb_b threads=nt stream=s_boundary compute_boundaries!(A, B)
CUDA.synchronize(s_boundary)
@cuda blocks=nb_i threads=nt stream=s_inner compute_inner_points!(A, B)
update_halo!(A) # Runs while the inner points are computed.
CUDA.synchronize(s_inner)With CUDA.jl ≥ 6.3 and AMDGPU.jl, this only overlaps if the implicit synchronization of A is disabled (see GPU streams and synchronization). The macro @hide_communication of ParallelStencil.jl implements this pattern.
Limitations
Synchronizing all work issued before the call is the responsibility of the caller, including work issued on other streams or from other tasks.
update_halo!is not designed to be called concurrently from multiple tasks.With CUDA.jl ≥ 6.3 and AMDGPU.jl, arrays keep track of the stream that last used them; see GPU streams and synchronization for what this means for
update_halo!.
Multi-threaded CPU halo copies
Halo copies of CPU arrays can be multi-threaded with Polyester.jl by importing it before ImplicitGlobalGrid, setting IGG_USE_POLYESTER=1 and starting Julia with multiple threads (e.g. julia -t 4):
using Polyester
using ImplicitGlobalGridCellArrays
update_halo! also accepts CellArrays of CellArrays.jl (with blocklength 0 or 1).
Multiple grids
init_global_grid creates and activates one grid. Additional grids are created with create_global_grid and selected with activate_global_grid (or passed to update_halo! with the keyword active_global_grid); get_global_grid returns the active grid:
init_global_grid() # Initialize the package without creating a grid.
gg1 = create_global_grid(nx, ny, nz)
gg2 = create_global_grid(2nx, 2ny, 2nz)
activate_global_grid(gg1)
update_halo!(A1) # Uses gg1.
update_halo!(A2; active_global_grid=gg2) # Uses gg2.In-situ visualization / monitoring
ImplicitGlobalGrid provides a function to gather an array from each process into a one large array on a single process, assembled according to the global grid:
This enables straightforward in-situ visualization or monitoring of Multi-GPU/CPU applications as shown in this example.
Seamless interoperability with MPI.jl
ImplicitGlobalGrid is seamlessly interoperable with MPI.jl. The Cartesian MPI communicator it uses is created by default when calling init_global_grid and can then be obtained as follows (variable comm_cart):
me, dims, nprocs, coords, comm_cart = init_global_grid(nx, ny, nz);Moreover, the automatic initialization and finalization of MPI can be deactivated in order to replace them with direct calls to MPI.jl:
init_global_grid(nx, ny, nz; init_MPI=false);finalize_global_grid(;finalize_MPI=false)Besides, init_global_grid makes every argument it passes to an MPI.jl function customizable via its keyword arguments.
Environment variables
The following environment variables are read when a grid is created (init_global_grid or create_global_grid); a value > 0 activates the feature:
| Variable | Description |
|---|---|
IGG_CUDAAWARE_MPI | Use CUDA-aware MPI in all dimensions. |
IGG_CUDAAWARE_MPI_DIMX, IGG_CUDAAWARE_MPI_DIMY, IGG_CUDAAWARE_MPI_DIMZ | Use CUDA-aware MPI in the given dimension (only read if IGG_CUDAAWARE_MPI is not activated). |
IGG_ROCMAWARE_MPI | Use ROCm-aware MPI in all dimensions. |
IGG_ROCMAWARE_MPI_DIMX, IGG_ROCMAWARE_MPI_DIMY, IGG_ROCMAWARE_MPI_DIMZ | Use ROCm-aware MPI in the given dimension (only read if IGG_ROCMAWARE_MPI is not activated). |
IGG_USE_POLYESTER | Use Polyester for the CPU halo copies in all dimensions (requires importing Polyester). |
IGG_USE_POLYESTER_DIMX, IGG_USE_POLYESTER_DIMY, IGG_USE_POLYESTER_DIMZ | Override IGG_USE_POLYESTER in the given dimension (only read if IGG_USE_POLYESTER is activated). |