ImplicitGlobalGrid.jl

Appearance

API reference ​

This is the official API reference of ImplicitGlobalGrid. Note that it can also be queried interactively from the Julia REPL using the help mode:

julia
julia> using ImplicitGlobalGrid
julia>?
help?> ImplicitGlobalGrid

Module ​

ImplicitGlobalGrid.ImplicitGlobalGrid Module

Module ImplicitGlobalGrid

Renders the distributed parallelization of stencil-based GPU and CPU applications on a regular staggered grid almost trivial and enables close to ideal weak scaling of real-world applications on thousands of GPUs.

General overview and examples

https://github.com/eth-cscs/ImplicitGlobalGrid.jl

Functions

To see a description of a function type ?<functionname>.

Activation of device support

The support for a device type (CUDA or AMDGPU) is activated by importing the corresponding module (CUDA or AMDGPU) before importing ImplicitGlobalGrid (the corresponding extension will be loaded).

Performance note

If the system supports CUDA-aware MPI (for Nvidia GPUs) or ROCm-aware MPI (for AMD GPUs), it may be activated for ImplicitGlobalGrid by setting one of the following environment variables (at latest before the call to init_global_grid):

shell
shell> export IGG_CUDAAWARE_MPI=1
shell
shell> export IGG_ROCMAWARE_MPI=1
source

Initialization and finalization ​

ImplicitGlobalGrid.init_global_grid Function
julia
init_global_grid(nx, ny, nz)
init_global_grid()
me, dims, nprocs, coords, comm_cart = init_global_grid(nx, ny, nz; <keyword arguments>)
me, dims, nprocs, coords, comm_cart = init_global_grid(<keyword arguments>)

Initialize the package and create a Cartesian grid of MPI processes (and also MPI itself by default) defining implicitly a global grid.

For use cases needing multiple grids, the first grid can be created with init_global_grid and additional grids with create_global_grid. Alternatively, the dispatch without (nx|ny|nz) can be used to do a package initialization and to define create_global_grid keyword defaults without creating a grid; all required grids can then be created by the create_global_grid function. Grids are activated with activate_global_grid.

Arguments

  • {nx|ny|nz}::Integer: the number of elements of the local grid in dimension {x|y|z}.

  • {dimx|dimy|dimz}::Integer=0: the desired number of processes in dimension {x|y|z}. By default, (value 0) the process topology is created as compact as possible with the given constraints. This is handled by the MPI implementation which is installed on your system. For more information, refer to the specifications of MPI_Dims_create in the corresponding documentation.

  • {periodx|periody|periodz}::Bool|Integer=false: whether the grid is periodic (true) or not (false) in dimension {x|y|z}. The argument is also accepted as an integer as traditionally in the MPI standard (1 for true and 0 for false).

  • origin::Tuple|AbstractFloat: the origin of the global grid. By default, it is set to (0.0, 0.0, 0.0) for 3D, (0.0, 0.0) for 2D and 0.0 for 1D.

  • origin_on_vertex::Bool=false: whether the origin is on the cell vertex; else, it is on the cell center (default). The default implies that the space step dx is computed in the user code as dx=lx/(nx-1), where lx is the length of the global grid in dimension x. Setting the origin on the vertex implies that the space step is computed as dx=lx/nx, instead. The analog applies for the dimensions y and z.

  • {centerx|centery|centerz}::Bool=false: whether to center the grid on the origin (true) or not (false) in dimension {x|y|z}. By default, the grid is extends from origin in the positive direction of the corresponding dimension.

  • quiet::Bool=false: whether to suppress printing information like the size of the global grid (true) or not (false).

  • save_kwarg_defaults::Bool=false: whether to update the default values of the keyword arguments of create_global_grid with the ones passed to init_global_grid. If false and no positional arguments are given (nx, ny, nz) and keyword arguments are passed with values different from the current defaults, an error is thrown. The parameters select_device and device_type are not affected by this argument, they can only and always be set on initialization.

Advanced keyword arguments

  • overlaps::Tuple{Int,Int,Int}=(2,2,2): the number of elements adjacent local grids overlap in dimension x, y and z. By default (value (2,2,2)), an array A of size (nx, ny, nz) on process 1 (A_1) overlaps the corresponding array A on process 2 (A_2) by 2 indices if the two processes are adjacent. E.g., if overlaps[1]=2 and process 2 is the right neighbor of process 1 in dimension x, then A_1[end-1:end,:,:] overlaps A_2[1:2,:,:]. That means, after every call update_halo!(A), we have all(A_1[end-1:end,:,:] .== A_2[1:2,:,:]) (A_1[end,:,:] is the halo of process 1 and A_2[1,:,:] is the halo of process 2). The analog applies for the dimensions y and z.

  • halowidths::Tuple{Int,Int,Int}=max.(1,overlaps.÷2): the default width of an array's halo in dimension x, y and z (must be greater or equal to 1). The default can be overwritten per array in the function update_halo!.

  • disp::Integer=1: the displacement argument to MPI.Cart_shift in order to determine the neighbors.

  • reorder::Integer=1: the reorder argument to MPI.Cart_create in order to create the Cartesian process topology.

  • comm::MPI.Comm=MPI.COMM_WORLD: the input communicator argument to MPI.Cart_create in order to create the Cartesian process topology.

  • init_MPI::Bool=true: whether to initialize MPI (true) or not (false).

  • device_type::String="auto": the type of the device to be used if available: "CUDA", "AMDGPU", "none" or "auto". Set device_type="none" if you want to use only CPUs on a system having also GPUs. If device_type is "auto" (default), it is automatically determined, depending on which of the modules used for programming the devices (CUDA.jl or AMDGPU.jl) was imported before ImplicitGlobalGrid; if both were imported, an error will be given if device_type is set as "auto". This can only be set up on initialization.

  • select_device::Bool=true: whether to automatically select the device (GPU) (true) or not (false) if CUDA or AMDGPU was imported and device_type is not "none". If true, it selects the device corresponding to the node-local MPI rank. This method of device selection suits both single and multi-device compute nodes and is recommended in general. It is also the default method of device selection of the function select_device. This can only be set up on initialization.

For more information, refer to the documentation of MPI.jl / MPI.

Return values

  • me: the MPI rank of the process.

  • dims: the number of processes in each dimension.

  • nprocs: the number of processes.

  • coords: the Cartesian coordinates of the process.

  • comm_cart: the MPI communicator of the created Cartesian process topology.

Typical use cases

julia
init_global_grid(nx, ny, nz)                  # Basic call (no optional in and output arguments).
me, = init_global_grid(nx, ny, nz)            # Capture 'me' (note the ','!).
me, dims = init_global_grid(nx, ny, nz)       # Capture 'me' and 'dims'.
init_global_grid(nx, ny, nz; dimx=2, dimy=2)  # Fix the number of processes in the dimensions x and y of the Cartesian grid of MPI processes to 2 (the number of processes can vary only in the dimension z).
init_global_grid(nx, ny, nz; periodz=1)       # Make the boundaries in dimension z periodic.

See also: finalize_global_grid, select_device

source
ImplicitGlobalGrid.finalize_global_grid Function
julia
finalize_global_grid()
finalize_global_grid(;finalize_MPI=true)

Finalize the global grid (and also MPI by default).

Arguments

Advanced keyword arguments

  • finalize_MPI::Bool=true: whether to finalize MPI (true) or not (false).

See also: init_global_grid

source

Halo update ​

ImplicitGlobalGrid.update_halo! Function
julia
update_halo!(A)
update_halo!(A...)
update_halo!(A..., active_global_grid=gg)

Advanced

update_halo!(A, B, (A=C, halowidths=..., (A=D, halowidths=...), ...)

Update the halo of the given GPU/CPU-array(s). If multiple global grids have been defined, the currently active global grid will be used. Optionally a global grid can be passed by argument, overriding the currently active grid.

Typical use cases:

julia
update_halo!(A)                                # Update the halo of the array A.
update_halo!(A, B, C)                          # Update the halos of the arrays A, B and C.
update_halo!(A, B, (A=C, halowidths=(2,2,2)))  # Update the halos of the arrays A, B, C, defining non default halowidth for C.

Performance note

Group subsequent calls to update_halo! in a single call for better performance (enables additional pipelining).

Performance note

If the system supports CUDA-aware MPI (for Nvidia GPUs) or ROCm-aware MPI (for AMD GPUs), it may be activated for ImplicitGlobalGrid by setting one of the following environment variables (at latest before the call to init_global_grid):

shell
shell> export IGG_CUDAAWARE_MPI=1
shell
shell> export IGG_ROCMAWARE_MPI=1
source

Multiple grids ​

ImplicitGlobalGrid.create_global_grid Function
julia
gg = create_global_grid(nx, ny, nz; <keyword arguments>)

Construct and return a Cartesian grid of MPI processes, defining implicitly a global grid. This grid can then be activated by activate_global_grid.

Arguments

  • {nx|ny|nz}::Integer: the number of elements of the local grid in dimension {x|y|z}.

  • {dimx|dimy|dimz}::Integer=0: the desired number of processes in dimension {x|y|z}. By default, (value 0) the process topology is created as compact as possible with the given constraints. This is handled by the MPI implementation which is installed on your system. For more information, refer to the specifications of MPI_Dims_create in the corresponding documentation.

  • {periodx|periody|periodz}::Union{Bool,Integer}=0: whether the grid is periodic (true or 1) or not (false or 0) in dimension {x|y|z}.

  • origin::Tuple|AbstractFloat: the origin of the global grid. By default, it is set to (0.0, 0.0, 0.0) for 3D, (0.0, 0.0) for 2D and 0.0 for 1D.

  • origin_on_vertex::Bool=false: whether the origin is on the cell vertex; else, it is on the cell center (default). The default implies that the space step dx is computed in the user code as dx=lx/(nx-1), where lx is the length of the global grid in dimension x. Setting the origin on the vertex implies that the space step is computed as dx=lx/nx, instead. The analog applies for the dimensions y and z.

  • {centerx|centery|centerz}::Bool=false: whether to center the grid on the origin (true) or not (false) in dimension {x|y|z}. By default, the grid is extends from origin in the positive direction of the corresponding dimension.

  • quiet::Bool=false: whether to suppress printing information like the size of the global grid (true) or not (false).

Advanced keyword arguments

  • overlaps::Tuple{Int,Int,Int}=(2,2,2): the number of elements adjacent local grids overlap in dimension x, y and z. By default (value (2,2,2)), an array A of size (nx, ny, nz) on process 1 (A_1) overlaps the corresponding array A on process 2 (A_2) by 2 indices if the two processes are adjacent. E.g., if overlaps[1]=2 and process 2 is the right neighbor of process 1 in dimension x, then A_1[end-1:end,:,:] overlaps A_2[1:2,:,:]. That means, after every call update_halo!(A), we have all(A_1[end-1:end,:,:] .== A_2[1:2,:,:]) (A_1[end,:,:] is the halo of process 1 and A_2[1,:,:] is the halo of process 2). The analog applies for the dimensions y and z.

  • halowidths::Tuple{Int,Int,Int}=max.(1,overlaps.÷2): the default width of an array's halo in dimension x, y and z (must be greater or equal to 1). The default can be overwritten per array in the function update_halo!.

  • disp::Integer=1: the displacement argument to MPI.Cart_shift in order to determine the neighbors.

  • reorder::Integer=1: the reorder argument to MPI.Cart_create in order to create the Cartesian process topology.

  • comm::MPI.Comm=MPI.COMM_WORLD: the input communicator argument to MPI.Cart_create in order to create the Cartesian process topology.

For more information, refer to the documentation of MPI.jl / MPI. select_device and device_type cannot be changed during grid creation; they must be set during initialization.

Return values

  • gg: the newly defined global grid

Usage example

julia
init_global_grid()                  # Basic call no dimensions are provided thus no grid is defined.
gg = create_global_grid(nx, ny, nz) # A global grid is created
activate_global_grid(gg)            # The new global grid is activated
update_halo!(array)                 # The halo exchange is made following the active global grid

See also: finalize_global_grid, init_global_grid, activate_global_grid

source
ImplicitGlobalGrid.activate_global_grid Function
julia
activate_global_grid(new_gg)

Replaces the current active grid with new_gg. Only one grid can be active at a time. This function returns nothing; use get_global_grid if you need to query the currently active grid.

Argument

  • new_gg::GlobalGrid: the global grid to be set active. It must be a GlobalGrid returned by a call to create_global_grid or a grid created by the one-time call to init_global_grid and retrieved by get_global_grid.

Usage example

Given two local domains of different size and/or ghost cell properties: we have array A1 and B1 on grid gg1, and array A2, B2 on grid gg2.

julia
activate_global_grid(gg1)       # Activate the first grid
update_halo!(A1, B1)            # Update the halo regions of arrays A1 and B1 on grid gg1
activate_global_grid(gg2)       # Activate the second grid
update_halo!(A2, B2)            # Update the halo regions of arrays A2 and B2 on grid gg2

See also: init_global_grid, create_global_grid

source
ImplicitGlobalGrid.get_global_grid Function
julia
get_global_grid() :: GlobalGrid

Returns a deep copy of the currently active global grid. If no grid is active a GLOBAL_GRID_NULL will be returned, with negative nprocs.

source

Device selection ​

ImplicitGlobalGrid.select_device Function
julia
select_device()
select_device(gg::GlobalGrid)

Select the device (GPU) corresponding to the node-local MPI rank and return its ID.

device indexing

  • CUDA.jl device indexing is 0-based.

  • AMDGPU.jl device indexing is 1-based.

  • The returned ID is therefore 0-based for CUDA and 1-based for AMDGPU.

See also: init_global_grid

source

Global grid size, coordinates and indices ​

ImplicitGlobalGrid.nx_g Function
julia
nx_g()

Return the size of the global grid in dimension x.

source
julia
nx_g(A)
nx_g(nx_A)

Return the size of array A in the global grid in dimension x. Alternatively to passing A, the size of the array in dimension x can be provided.

source
ImplicitGlobalGrid.ny_g Function
julia
ny_g()

Return the size of the global grid in dimension y.

source
julia
ny_g(A)
ny_g(ny_A)

Return the size of array A in the global grid in dimension y. Alternatively to passing A, the size of the array in dimension y can be provided.

source
ImplicitGlobalGrid.nz_g Function
julia
nz_g()

Return the size of the global grid in dimension z.

source
julia
nz_g(A)

Return the size of array A in the global grid in dimension z. Alternatively to passing A, the size of the array in dimension z can be provided.

source
ImplicitGlobalGrid.x_g Function
julia
x_g(ix, dx, A)
x_g(ix, dx, A; wrap_periodic)

Return the global x-coordinate for the index ix in the local array A (dx is the space step between the elements).

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the coordinate at the periodic boundaries (default: true).

Examples

julia
julia> using ImplicitGlobalGrid

julia> lx=4; nx=3; ny=3; nz=3;

julia> init_global_grid(nx, ny, nz);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> dx = lx/(nx_g()-1)
2.0

julia> A  = zeros(nx,ny,nz);

julia> Vx = zeros(nx+1,ny,nz);

julia> [x_g(ix, dx, A) for ix=1:size(A, 1)]
3-element Vector{Float64}:
 0.0
 2.0
 4.0

julia> [x_g(ix, dx, Vx) for ix=1:size(Vx, 1)]
4-element Vector{Float64}:
 -1.0
  1.0
  3.0
  5.0

julia> finalize_global_grid()
source
ImplicitGlobalGrid.y_g Function
julia
y_g(iy, dy, A)
y_g(iy, dy, A; wrap_periodic)

Return the global y-coordinate for the index iy in the local array A (dy is the space step between the elements).

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the coordinate at the periodic boundaries (default: true).

Examples

julia
julia> using ImplicitGlobalGrid

julia> ly=4; nx=3; ny=3; nz=3;

julia> init_global_grid(nx, ny, nz);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> dy = ly/(ny_g()-1)
2.0

julia> A  = zeros(nx,ny,nz);

julia> Vy = zeros(nx,ny+1,nz);

julia> [y_g(iy, dy, A) for iy=1:size(A, 2)]
3-element Vector{Float64}:
 0.0
 2.0
 4.0

julia> [y_g(iy, dy, Vy) for iy=1:size(Vy, 2)]
4-element Vector{Float64}:
 -1.0
  1.0
  3.0
  5.0

julia> finalize_global_grid()
source
ImplicitGlobalGrid.z_g Function
julia
z_g(iz, dz, A)
z_g(iz, dz, A; wrap_periodic)

Return the global z-coordinate for the index iz in the local array A (dz is the space step between the elements).

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the coordinate at the periodic boundaries (default: true).

Examples

julia
julia> using ImplicitGlobalGrid

julia> lz=4; nx=3; ny=3; nz=3;

julia> init_global_grid(nx, ny, nz);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> dz = lz/(nz_g()-1)
2.0

julia> A  = zeros(nx,ny,nz);

julia> Vz = zeros(nx,ny,nz+1);

julia> [z_g(iz, dz, A) for iz=1:size(A, 3)]
3-element Vector{Float64}:
 0.0
 2.0
 4.0

julia> [z_g(iz, dz, Vz) for iz=1:size(Vz, 3)]
4-element Vector{Float64}:
 -1.0
  1.0
  3.0
  5.0

julia> finalize_global_grid()
source
ImplicitGlobalGrid.ix_g Function
julia
ix_g(ix, A)
ix_g(ix, A; wrap_periodic)

Return the global x-index for the local index ix in the local array A.

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the index at the periodic boundaries (default: true). If wrap_periodic=false, the global index can be negative or bigger than the size of the global grid.

Examples

julia
julia> using ImplicitGlobalGrid

julia> nx=5; ny=3; nz=3;

julia> init_global_grid(nx, ny, nz, periodx=1);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> A  = zeros(nx,ny,nz);

julia> Vx = zeros(nx+1,ny,nz);

julia> [ix_g(ix, A) for ix=1:size(A, 1)]
5-element Vector{Int64}:
 3
 1
 2
 3
 1

julia> [ix_g(ix, Vx) for ix=1:size(Vx, 1)]
6-element Vector{Int64}:
 3
 1
 2
 3
 1
 2

julia> finalize_global_grid()
source
ImplicitGlobalGrid.iy_g Function
julia
iy_g(iy, A)
iy_g(iy, A; wrap_periodic)

Return the global y-index for the local index iy in the local array A.

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the index at the periodic boundaries (default: true). If wrap_periodic=false, the global index can be negative or bigger than the size of the global grid.

Examples

julia
julia> using ImplicitGlobalGrid

julia> nx=3; ny=5; nz=3;

julia> init_global_grid(nx, ny, nz, periody=1);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> A  = zeros(nx,ny,nz);

julia> Vy = zeros(nx,ny+1,nz);

julia> [iy_g(iy, A) for iy=1:size(A, 2)]
5-element Vector{Int64}:
 3
 1
 2
 3
 1

julia> [iy_g(iy, Vy) for iy=1:size(Vy, 2)]
6-element Vector{Int64}:
 3
 1
 2
 3
 1
 2

julia> finalize_global_grid()
source
ImplicitGlobalGrid.iz_g Function
julia
iz_g(iz, A)
iz_g(iz, A; wrap_periodic)

Return the global z-index for the local index iz in the local array A.

Keyword arguments

  • wrap_periodic::Bool=true: whether to wrap the index at the periodic boundaries (default: true). If wrap_periodic=false, the global index can be negative or bigger than the size of the global grid.

Examples

julia
julia> using ImplicitGlobalGrid

julia> nx=3; ny=3; nz=5;

julia> init_global_grid(nx, ny, nz, periodz=1);
Global grid: 3x3x3 (nprocs: 1, dims: 1x1x1; device support: none)

julia> A  = zeros(nx,ny,nz);

julia> Vz = zeros(nx,ny,nz+1);

julia> [iz_g(iz, A) for iz=1:size(A, 3)]
5-element Vector{Int64}:
 3
 1
 2
 3
 1

julia> [iz_g(iz, A; wrap_periodic=false) for iz=1:size(A, 3)]
5-element Vector{Int64}:
 0
 1
 2
 3
 4

julia> [iz_g(iz, Vz) for iz=1:size(Vz, 3)]
6-element Vector{Int64}:
 3
 1
 2
 3
 1
 2

julia> [iz_g(iz, Vz; wrap_periodic=false) for iz=1:size(Vz, 3)]
6-element Vector{Int64}:
 0
 1
 2
 3
 4
 5

julia> finalize_global_grid()
source

Extents and metadata grids ​

ImplicitGlobalGrid.extents Function
julia
extents(A; fix_global_boundaries, coords)
extents(nxyz; fix_global_boundaries, coords)
extents(; fix_global_boundaries, coords)
extents(A, overlaps; fix_global_boundaries, coords)
extents(nxyz, overlaps; fix_global_boundaries, coords)
extents(overlaps; fix_global_boundaries, coords)

Return the local extents in each dimension of the array A or of an array of length nxyz or the local extents of the base grid with the length nxyz (return type: NTuple of ranges with N = ndims(A) or N = ndims(nxyz)). If neither A nor nxyz is provided, the extents of the base grid are returned as a 3-dimensional tuple of ranges; for 2D grids, it can be preferable to call extents((nx, ny); ...) in order to obtain a 2-dimensional tuple; for 1D grids, the analogue is true.

Arguments

  • overlaps::Integer: the overlap of the "extent" with the neighboring processes' extents in each dimension; the overlaps chosen cannot be bigger than the actual overlaps on the global grid of A or of the base grid, respectively. To obtain the extents as required by VTK, set overlaps=1. The default is the actual full overlaps on the global grid (i.e., the extents are simply the full ranges of the array or of the base grid, respectively).

Keyword arguments

  • fix_global_boundaries::Bool=true: by default, the extents are fixed at the global boundaries to include them on all sides (attention, the extents are not of equal size for all processes in this case). If fix_global_boundaries=false, the extents are not fixed at the global boundaries and the size of the extents is equal for all processes.

  • coords::Tuple{Int,Int,Int}: the coordinates of the process for which the local extents is requested. The default is the coordinates of the current process.

Examples

julia
julia> using ImplicitGlobalGrid

julia> nx=5; ny=5; nz=7;

julia> init_global_grid(nx, ny, nz; periodz=1);
Global grid: 5x5x5 (nprocs: 1, dims: 1x1x1; device support: none)

julia> extents()
(1:5, 1:5, 2:6)

julia> extents(; fix_global_boundaries=false)
(1:5, 1:5, 1:7)

julia> extents(0)
(1:5, 1:5, 2:6)

julia> extents(0; fix_global_boundaries=false)
(2:4, 2:4, 2:6)

julia> extents(1) # The extents as required by VTK
(1:5, 1:5, 2:6)

julia> Vx = zeros(nx+1,ny,nz);

julia> extents(Vx, 1) # The extents as required by VTK
(1:6, 1:5, 2:6)

julia> Vx_IO = view(Vx, extents(Vx, 1)...);

julia> summary(Vx_IO)
"6×5×5 view(::Array{Float64, 3}, 1:6, 1:5, 2:6) with eltype Float64"

julia> finalize_global_grid()
source
ImplicitGlobalGrid.extents_g Function
julia
extents_g(A; dxyz, fix_global_boundaries, coords)
extents_g(nxyz; dxyz, fix_global_boundaries, coords)
extents_g(; dxyz, fix_global_boundaries, coords)
extents_g(A, overlaps; dxyz, fix_global_boundaries, coords)
extents_g(nxyz, overlaps; dxyz, fix_global_boundaries, coords)
extents_g(overlaps; dxyz, fix_global_boundaries, coords)

Return the global extents in each dimension of the array A or of an array of length nxyz or the global extents of the base grid with the length nxyz (return type: NTuple of ranges with N = ndims(A) or N = ndims(nxyz)); if dxyz is set, global Cartesian coordinates extents are returned, else global index extents are returned. If neither A nor nxyz is provided, the global extents of the base grid are returned as a 3-dimensional tuple of ranges; for 2D grids, it can be preferable to call extents_g((nx, ny); ...) in order to obtain a 2-dimensional tuple; for 1D grids, the analogue is true.

Arguments

  • overlaps::Integer: the overlap of the "extent" with the neighboring processes' extents in each dimension; the overlaps chosen cannot be bigger than the actual overlaps on the global grid of A or of the base grid, respectively. To obtain the extents as required by VTK, set overlaps=1. The default is the actual full overlaps on the global grid.

Keyword arguments

  • dxyz::NTuple{N, AbstractFloat}: the space step between the elements in each dimension if global Cartesian coordinates are desired.

  • fix_global_boundaries::Bool=true: by default, the extents are fixed at the global boundaries to include them on all sides (attention, the extents are not of equal size for all processes in this case). If fix_global_boundaries=false, the extents are not fixed at the global boundaries and the size of the extents is equal for all processes.

  • coords::Tuple{Int,Int,Int}: the coordinates of the process for which the global extents is requested. The default is the coordinates of the current process.

Examples

julia
julia> using ImplicitGlobalGrid

julia> lx=8; ly=8; lz=8; nx=5; ny=5; nz=7;

julia> init_global_grid(nx, ny, nz; periodz=1);
Global grid: 5x5x5 (nprocs: 1, dims: 1x1x1; device support: none)

julia> extents_g()
(1:5, 1:5, 1:5)

julia> extents_g(; fix_global_boundaries=false)
(1:5, 1:5, 0:6)

julia> extents_g(0)
(1:5, 1:5, 1:5)

julia> extents_g(0; fix_global_boundaries=false)
(2:4, 2:4, 1:5)

julia> extents_g(1) # The extents as required by VTK
(1:5, 1:5, 1:5)

julia> dx, dy, dz = lx/(nx_g()-1), ly/(ny_g()-1), lz/(nz_g()-1)
(2.0, 2.0, 2.0)

julia> extents_g(; dxyz=(dx, dy, dz))
(0.0:2.0:8.0, 0.0:2.0:8.0, 0.0:2.0:8.0)

julia> extents_g(; dxyz=(dx, dy, dz), fix_global_boundaries=false)
(0.0:2.0:8.0, 0.0:2.0:8.0, -2.0:2.0:10.0)

julia> extents_g(0; dxyz=(dx, dy, dz), fix_global_boundaries=false)
(2.0:2.0:6.0, 2.0:2.0:6.0, 0.0:2.0:8.0)

julia> extents_g(1; dxyz=(dx, dy, dz)) # The Cartesian coordinates corresponding to the extents required by VTK
(0.0:2.0:8.0, 0.0:2.0:8.0, 0.0:2.0:8.0)

julia> finalize_global_grid()
source
ImplicitGlobalGrid.metagrid Function
julia
metagrid(f::Function, args...; kwargs...)
metagrid(dims, f::Function, args...; kwargs...)

Return a metadata grid containing the results of the function f applied to each process coordinates in the global grid; if dims is provided, the function is applied to each process coordinates in an imaginary grid with the dimensions dims (the other grid parameters like periodicity and overlaps are inherited from the actual global grid). The return type is an array with elements of the same type as the return type of f.

Arguments

  • f::Function: the function to apply to each process coordinates; the function must accept the keyword argument coords (a tuple of the coordinates of the process). Furthermore, for imaginary grids, the function can optionally also accept the keyword arguments dims (the dimensions of the imaginary grid).

  • args...: the arguments to pass to the function f.

  • kwargs...: the keyword arguments to pass to the function f; the keyword argument coords (and optionally dims for imaginary grids) is automatically added.

  • dims::Tuple{Int,Int,Int}: the dimensions of the imaginary grid to use instead of the global grid.

Examples

julia
julia> using ImplicitGlobalGrid

julia> lx=8; ly=8; lz=8; nx=5; ny=5; nz=7;

julia> init_global_grid(nx, ny, nz; periodz=1);
Global grid: 5x5x5 (nprocs: 1, dims: 1x1x1; device support: none)

julia> dx, dy, dz = lx/(nx_g()-1), ly/(ny_g()-1), lz/(nz_g()-1)
(2.0, 2.0, 2.0)

julia> metagrid((; coords)->coords)
1×1×1 Array{Tuple{Int64, Int64, Int64}, 3}:
[:, :, 1] =
 (0, 0, 0)

julia> metagrid((2, 2, 2), (; coords)->coords)
2×2×2 Array{Tuple{Int64, Int64, Int64}, 3}:
[:, :, 1] =
 (0, 0, 0)  (0, 1, 0)
 (1, 0, 0)  (1, 1, 0)

[:, :, 2] =
 (0, 0, 1)  (0, 1, 1)
 (1, 0, 1)  (1, 1, 1)

julia> metagrid(extents_g, 0)
1×1×1 Array{Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, 3}:
[:, :, 1] =
 (1:5, 1:5, 1:5)

julia> metagrid(extents_g, 0; fix_global_boundaries=false)
1×1×1 Array{Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, 3}:
[:, :, 1] =
 (2:4, 2:4, 1:5)

julia> metagrid((2, 2, 2), extents_g, 0)
2×2×2 Array{Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, 3}:
[:, :, 1] =
 (1:4, 1:4, 1:5)  (1:4, 5:8, 1:5)
 (5:8, 1:4, 1:5)  (5:8, 5:8, 1:5)

[:, :, 2] =
 (1:4, 1:4, 6:10)  (1:4, 5:8, 6:10)
 (5:8, 1:4, 6:10)  (5:8, 5:8, 6:10)

julia> metagrid(extents_g, 1) # The extents grid as required by VTK
1×1×1 Array{Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, 3}:
[:, :, 1] =
 (1:5, 1:5, 1:5)

julia> metagrid(extents_g, 1; dxyz=(dx, dy, dz)) # The Cartesian coordinates corresponding to the extents required by VTK
1×1×1 Array{Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, 3}:
[:, :, 1] =
 (0.0:2.0:8.0, 0.0:2.0:8.0, 0.0:2.0:8.0)


julia> metagrid((2, 2, 2), extents_g, 1) # The extents grid as required by VTK for an imaginary grid with dimensions (2, 2, 2) and periodic in z (inherited from the global grid)
2×2×2 Array{Tuple{UnitRange{Int64}, UnitRange{Int64}, UnitRange{Int64}}, 3}:
[:, :, 1] =
 (1:4, 1:4, 1:5)  (1:4, 4:8, 1:5)
 (4:8, 1:4, 1:5)  (4:8, 4:8, 1:5)

[:, :, 2] =
 (1:4, 1:4, 5:10)  (1:4, 4:8, 5:10)
 (4:8, 1:4, 5:10)  (4:8, 4:8, 5:10)

julia> metagrid((2, 2, 2), extents_g, 1; dxyz=(dx, dy, dz)) # The Cartesian coordinates corresponding to the extents required by VTK for an imaginary grid with dimensions (2, 2, 2) and periodic in z (inherited from the global grid)
2×2×2 Array{Tuple{StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, 3}:
[:, :, 1] =
 (0.0:2.0:6.0, 0.0:2.0:6.0, 0.0:2.0:8.0)   (0.0:2.0:6.0, 6.0:2.0:14.0, 0.0:2.0:8.0)
 (6.0:2.0:14.0, 0.0:2.0:6.0, 0.0:2.0:8.0)  (6.0:2.0:14.0, 6.0:2.0:14.0, 0.0:2.0:8.0)

[:, :, 2] =
 (0.0:2.0:6.0, 0.0:2.0:6.0, 8.0:2.0:18.0)   (0.0:2.0:6.0, 6.0:2.0:14.0, 8.0:2.0:18.0)
 (6.0:2.0:14.0, 0.0:2.0:6.0, 8.0:2.0:18.0)  (6.0:2.0:14.0, 6.0:2.0:14.0, 8.0:2.0:18.0)

julia> finalize_global_grid()
source

Gathering ​

ImplicitGlobalGrid.gather! Function
julia
gather!(A, A_global)
gather!(A, A_global; root=0)

Advanced

gather!(A, A_global, comm; root=0)

Gather an array A from each member of the Cartesian grid of MPI processes into one large array A_global on the root process (default: 0). The size of the global array size(A_global) must be equal to the product of size(A) and dims, where dims is the number of processes in each dimension of the Cartesian grid, defined in init_global_grid.

Advanced

If the argument comm is given, then this communicator is used for the gather operation and dims extracted from it.

Memory requirements

The memory for the global array only needs to be allocated on the root process; the argument A_global can be nothing on the other processes.

source

Timing ​

ImplicitGlobalGrid.tic Function
julia
tic()

Start chronometer once all processes have reached this point.

Warning

The chronometer may currently add an overhead of multiple 10th of miliseconds at the first usage.

See also: toc

source
ImplicitGlobalGrid.toc Function
julia
toc()

Return the elapsed time from chronometer (since the last call to tic) when all processes have reached this point.

Warning

Every call to toc has to be executed after a call to tic. Calling toc before tic leads to undefined behaviour (likely an MPI error).

The chronometer may currently add an overhead of multiple 10th of miliseconds at the first usage.

See also: tic

source