Skip to content

Usage ​

Multi-GPU with three functions ​

Only three functions are required to perform halo updates close to hardware limit:

The 50-lines Multi-GPU example illustrates how these functions enable the creation of massively parallel applications.

Global coordinates, sizes, indices and extents ​

Three functions are provided to query Cartesian coordinates with respect to the global computational grid if required:

Moreover, the following three functions allow to query the size of the global grid:

Further tools: global indices (ix_g, iy_g, iz_g), local and global extents of arrays, e.g. for parallel VTK output (extents, extents_g, metagrid) and timing (tic, toc).

GPU support and device selection ​

Support for Nvidia or AMD GPUs is activated by importing CUDA.jl or AMDGPU.jl before ImplicitGlobalGrid:

julia
using ImplicitGlobalGrid
julia
using CUDA
using ImplicitGlobalGrid
julia
using AMDGPU
using ImplicitGlobalGrid

By default, init_global_grid selects the GPU corresponding to the node-local MPI rank (see select_device); every rank must then see all GPUs of its node. If the job launcher already binds one GPU per rank (e.g. Slurm --gpus-per-task=1), set select_device=false. To run on CPUs only while a GPU package is loaded, set device_type="none".

CUDA-/ROCm-aware MPI support ​

If the system supports CUDA-aware MPI (for Nvidia GPUs) or ROCm-aware MPI (for AMD GPUs), it can be activated for ImplicitGlobalGrid by setting the following environment variable (at latest before the call to init_global_grid):

bash
export IGG_CUDAAWARE_MPI=1
bash
export IGG_ROCMAWARE_MPI=1

It can also be activated per dimension, see Environment variables.

GPU-aware MPI with Cray MPICH

With Cray MPICH (HPE Cray EX systems), ImplicitGlobalGrid allocates the device buffers it uses for CUDA-/ROCm-aware MPI outside the CUDA.jl/AMDGPU.jl memory pools (with cuMemAlloc/hipMalloc), because the GPU transport layer of Cray MPICH does not accept pool memory (on Nvidia GPUs, the legacy CUDA IPC API it uses rejects stream-ordered memory). JULIA_CUDA_MEMORY_POOL=none is therefore not needed for ImplicitGlobalGrid (only if you pass your own arrays to MPI). ROCm-aware MPI requires MPICH_GPU_SUPPORT_ENABLED=1 and, currently, HSA_ENABLE_IPC_MODE_LEGACY=1: the GPU IPC path of the transport layer (between processes on the same node) fails with the default IPC mode of the ROCm runtime (observed with ROCm 7.2.3 and Cray MPICH 9.1.0). Alternatively, MPICH_GPU_IPC_ENABLED=0 disables this path, but the exchange between processes on the same node is then much slower (see issue #143). Your own arrays are never passed to MPI by ImplicitGlobalGrid, so no change is needed in your code.

Asynchronous halo updates and overlap with computation ​

For GPU arrays, update_halo! performs the halo update on streams of its own (non-blocking, with high priority), independently of the task-local stream on which CUDA.jl and AMDGPU.jl enqueue work by default. This is by design: it makes it straightforward to overlap the halo update with computation. The contract is:

  • update_halo! does, by default, not synchronize with work enqueued before the call, neither on the task-local stream nor on any other stream (automatic synchronization of the task-local stream can be activated in the call to init_global_grid with the keyword auto_task_sync=true; the synchronization then covers only the task-local stream).

  • Work that must not run at the same time as the halo update, typically the computation of the arrays passed to update_halo! or ideally the computation of only the boundary regions of these arrays, must therefore be synchronized before the call by default.

  • update_halo! returns only once the halo update is complete: no synchronization is needed after the call.

julia
@cuda blocks=nb threads=nt compute!(A, B)    # Enqueued on the task-local stream.
CUDA.synchronize()                            # Complete it before the halo update.
update_halo!(A)
julia
@roc gridsize=nb groupsize=nt compute!(A, B) # Enqueued on the task-local stream.
AMDGPU.synchronize()                          # Complete it before the halo update.
update_halo!(A)
julia
compute!(backend)(A, B; ndrange=size(A))
KernelAbstractions.synchronize(backend)
update_halo!(A)

To overlap the halo update with computation, compute the boundary regions of the arrays first and synchronize, then launch the computation of the inner points on another stream without synchronizing, update the halos and finally synchronize that stream:

julia
@cuda blocks=nb_b threads=nt stream=s_boundary compute_boundaries!(A, B)
CUDA.synchronize(s_boundary)
@cuda blocks=nb_i threads=nt stream=s_inner compute_inner_points!(A, B)
update_halo!(A)                               # Runs while the inner points are computed.
CUDA.synchronize(s_inner)

With CUDA.jl ≥ 6.3 and AMDGPU.jl, this only overlaps if the implicit synchronization of A is disabled (see GPU streams and synchronization). The macro @hide_communication of ParallelStencil.jl implements this pattern.

Limitations

  • Synchronizing all work issued before the call is the responsibility of the caller, including work issued on other streams or from other tasks. update_halo! is not designed to be called concurrently from multiple tasks.

  • With CUDA.jl ≥ 6.3 and AMDGPU.jl, arrays keep track of the stream that last used them; see GPU streams and synchronization for what this means for update_halo!.

Multi-threaded CPU halo copies ​

Halo copies of CPU arrays can be multi-threaded with Polyester.jl by importing it before ImplicitGlobalGrid, setting IGG_USE_POLYESTER=1 and starting Julia with multiple threads (e.g. julia -t 4):

julia
using Polyester
using ImplicitGlobalGrid

CellArrays ​

update_halo! also accepts CellArrays of CellArrays.jl (with blocklength 0 or 1).

Multiple grids ​

init_global_grid creates and activates one grid. Additional grids are created with create_global_grid and selected with activate_global_grid (or passed to update_halo! with the keyword active_global_grid); get_global_grid returns the active grid:

julia
init_global_grid()                          # Initialize the package without creating a grid.
gg1 = create_global_grid(nx, ny, nz)
gg2 = create_global_grid(2nx, 2ny, 2nz)
activate_global_grid(gg1)
update_halo!(A1)                            # Uses gg1.
update_halo!(A2; active_global_grid=gg2)    # Uses gg2.

In-situ visualization / monitoring ​

ImplicitGlobalGrid provides a function to gather an array from each process into a one large array on a single process, assembled according to the global grid:

This enables straightforward in-situ visualization or monitoring of Multi-GPU/CPU applications as shown in this example.

Seamless interoperability with MPI.jl ​

ImplicitGlobalGrid is seamlessly interoperable with MPI.jl. The Cartesian MPI communicator it uses is created by default when calling init_global_grid and can then be obtained as follows (variable comm_cart):

julia
me, dims, nprocs, coords, comm_cart = init_global_grid(nx, ny, nz);

Moreover, the automatic initialization and finalization of MPI can be deactivated in order to replace them with direct calls to MPI.jl:

julia
init_global_grid(nx, ny, nz; init_MPI=false);
julia
finalize_global_grid(;finalize_MPI=false)

Besides, init_global_grid makes every argument it passes to an MPI.jl function customizable via its keyword arguments.

Environment variables ​

The following environment variables are read when a grid is created (init_global_grid or create_global_grid); a value > 0 activates the feature:

VariableDescription
IGG_CUDAAWARE_MPIUse CUDA-aware MPI in all dimensions.
IGG_CUDAAWARE_MPI_DIMX, IGG_CUDAAWARE_MPI_DIMY, IGG_CUDAAWARE_MPI_DIMZUse CUDA-aware MPI in the given dimension (only read if IGG_CUDAAWARE_MPI is not activated).
IGG_ROCMAWARE_MPIUse ROCm-aware MPI in all dimensions.
IGG_ROCMAWARE_MPI_DIMX, IGG_ROCMAWARE_MPI_DIMY, IGG_ROCMAWARE_MPI_DIMZUse ROCm-aware MPI in the given dimension (only read if IGG_ROCMAWARE_MPI is not activated).
IGG_USE_POLYESTERUse Polyester for the CPU halo copies in all dimensions (requires importing Polyester).
IGG_USE_POLYESTER_DIMX, IGG_USE_POLYESTER_DIMY, IGG_USE_POLYESTER_DIMZOverride IGG_USE_POLYESTER in the given dimension (only read if IGG_USE_POLYESTER is activated).