Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased

A changelog is not maintained for this version.

This version introduces fully-parallel distributed memory data structures for all the steps required in a finite element simulation (geometry handling, fe space setup, linear system assembly) except for the linear solver kernel, which is just a sparse LU solver applied to the global system gathered on a master task (and thus obviously not scalable). These distributed data structures mirror their counterparts in the Gridap.jl software architecture and implement most of their abstract interfaces. This version of GridapDistributed.jl relies on PartitionedArrays.jl (https://github.com/fverdugo/PartitionedArrays.jl) as distributed linear algebra backend (global distributed sparse matrices and vectors).

More details can also be found in https://github.com/gridap/GridapDistributed.jl/issues/39

## [0.1.0] - 2021-09-29

A changelog is not maintained for this version.

This version although functional, is fully deprecated.
24 changes: 18 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,23 @@ Parallel distributed-memory version of `Gridap.jl`. 🚧 work in progress 🚧

## Purpose

This package is currently **experimental, under development**. The final purpose is to provide programming paradigm-neutral, parallel finite element data structures for distributed computing environments. This feature implies that communication among tasks are not tailored for a particular programming model, and thus can be leveraged with, e.g., MPI or the master-worker programming model built-in in Julia. Whenever one sticks to MPI as the underlying communication layer, `GridapDistributed.jl` leverages the suite of tools available in the [PETSc software](https://petsc.org/release/) package for the assembly and solution of distributed discrete systems of equations. Also for MPI, the user may also leverage the [`p4est` software library](https://p4est.github.io/) for generating a uniformly refined forest of quadtrees/octrees of the computational domain (click [here](https://github.com/gridap/GridapDistributed.jl/blob/master/test/MPIPETScTests/MPIPETScUniformlyRefinedForestOfOctreesDiscreteModelsTests.jl) for an example).
This package is currently **experimental, under development**.

`GridapDistributed.jl` provides fully-parallel distributed memory data structures for the Finite Element (FE) numerical solution of Partial Differential Equations (PDEs) on parallel computers, from multi-core CPU desktop computers, to HPC clusters and supercomputers. These distributed data structures are designed to mirror as far as possible their counterparts in the [`Gridap.jl`](https://github.com/gridap/Gridap.jl) software package, while implementing/leveraging most of their abstract interfaces. As a result, sequential Julia scripts written in the high level API of `Gridap.jl` can be used almost verbatim up to minor adjustments in a parallel context using `GridapDistributed.jl`. This is indeed one of the main advantages of `GridapDistributed.jl` and a major design goal that we pursue.

At present, `GridapDistributed.jl` provides scalable parallel data structures for grid handling, finite element spaces setup, and distributed linear system assembly. For the latter part, i.e., global distributed sparse matrices and vectors, `GridapDistributed.jl` relies on [`PartitionedArrays.jl`](https://github.com/fverdugo/PartitionedArrays.jl) as distributed linear algebra backend. This implies, among others, that all `GridapDistributed.jl` driver programs can be either run in sequential execution mode--very useful for developing/debugging parallel programs--, see `test/sequential/` folder for examples, or in message-passing (MPI) execution mode--when you want to deploy the code in the actual parallel computer and perform a fast simulation., see `test/mpi/` folder for examples.

## Caveats

At present, we have the following caveats:
1. Grid handling currently available within `GridapDistributed.jl` is restricted to Cartesian-like meshes of arbitrary-dimensional, topologically n-cube domains.
2. The linear solver kernel within `GridapDistributed.jl`, leveraged, e.g., via the backslash operator `\`, is just a sparse LU solver applied to the global system gathered on a master task (and thus obviously not scalable). It is in our future plans to provide highly scalable linear and nonlinear solvers tailored for the FE discretization of PDEs.

Complementarily, one may leverage two satellite packages of `GridapDistributed.jl` to address 1. and 2., namely [`GridapP4est.jl`](https://github.com/gridap/GridapP4est.jl), for peta-scale handling of meshes which can be decomposed as forest of quadtrees/octrees of the computational domain., and [`GridapPETSc.jl`](https://github.com/gridap/GridapPETSc.jl), which offers to the full set of scalable linear and non-linear solvers in the [PETSc](https://petsc.org/release/) numerical software package. We refer to the readme of these two packages for further details.

## Build

Before using `GridapDistributed.jl` package, we have to build `MPI.jl`, `GridapDistributedPETScWrappers.jl`, and
`P4est_wrapper.jl`. We refer to the main `README.md` of the latter two packages (available [here](https://github.com/gridap/GridapDistributedPETScWrappers.jl) and [here](https://github.com/gridap/p4est_wrapper.jl), resp.) for configuration instructions.
Before using `GridapDistributed.jl` package, one needs to build the [`MPI.jl`](https://github.com/JuliaParallel/MPI.jl) package. We refer to the main documentation of this package for configuration instructions.

## MPI-parallel Julia script execution instructions

Expand All @@ -34,13 +45,14 @@ Alternatively, for convenience, one can assign the path of `mpirun` to an enviro
$ export MPIRUN=$(julia --project=. -e "using MPI;println(MPI.mpiexec_path)")
```

As an example, the MPI-parallel `GridapDistributed.jl` driver `MPIPETScCommunicatorsTests.jl`, located in the `test` directory, can be executed as:
As an example, assuming that we are located on the root directory of `GridapDistributed.jl`,
an hypothetic MPI-parallel `GridapDistributed.jl` driver named `driver.jl` can be executed on 4 MPI tasks as:

```
$MPIRUN -np 2 julia --project=. -J ../Gridap.jl/compile/Gridapv0.14.1.so test/MPIPETScTests/MPIPETScCommunicatorsTests.jl
$MPIRUN -np 4 julia --project=. -J sys-image.so driver.jl
```

where `-J ../Gridap.jl/compile/Gridapv0.14.1.so` is optional, but highly recommended in order to reduce JIT compilation times. More details about how to generate this file can be found [here](https://github.com/gridap/GridapDistributed.jl/blob/master/compile/README.md).
where `-J sys-image.so` is optional, but highly recommended in order to reduce JIT compilation times. Here, `sys-image.so` is assumed to be a Julia system image pre-generated for the driver at hand using the [`PackageCompiler.jl`](https://julialang.github.io/PackageCompiler.jl/dev/index.html) package. See the `test/TestApp/compile` folder for example scripts with system image generation along with a test application with source available at `test/TestApp/`. These scripts are triggered from `.github/workflows/ci.yml` file on Github CI actions.


Two big warnings when executing MPI-parallel drivers:
Expand Down
64 changes: 0 additions & 64 deletions compile/README.md

This file was deleted.

159 changes: 0 additions & 159 deletions compile/create_gridap_image.jl

This file was deleted.

Loading