`partition_mesh` is slow at scale

[inducer]

Observed by @MTCam during a DAT.

cc @majosm

[majosm]

I think the main source of the slowness is this:

meshmode/meshmode/mesh/processing.py

Lines 556 to 558 in 261474e

	return {
	part_id: _get_mesh_part(mesh, part_id_to_elements, part_id)
	for part_id in return_parts}

It does some of the global processing nranks times (or nranks_per_node times in our case), when potentially it could do a single pass.

[inducer]

Would that be solved by just exposing _get_mesh_part (and maybe deprecating partition_mesh)? I looked inside _get_mesh_part, and I didn't see too much redundancy across parts. But maybe I missed it?

[majosm]

I believe it would end up being the same amount of work whether mirgecom calls _get_mesh_part directly or through partition_mesh. We don't load the global mesh onto every rank (due to memory considerations), so ultimately the root rank on the node will still need to iterate over the node's parts and generate them. _get_mesh_part itself would need to be refactored. I vaguely remember _compute_global_elem_to_part_elem being expensive; I'll do some profiling and have a look.

[inducer]

We don't load the global mesh onto every rank (due to memory consideration)

Why not? If it fits onto one rank, it should fit onto all of them. 🤔

[MTCam]

We don't load the global mesh onto every rank (due to memory consideration)

Why not? If it fits onto one rank, it should fit onto all of them. 🤔

nranks/node = ndevices/node at runtime. Matt fixed it up so that we have only 1 serial mesh in memory on each node to solve our running out of memory at partition time for large problems.

[majosm]

We don't load the global mesh onto every rank (due to memory consideration)

Why not? If it fits onto one rank, it should fit onto all of them. 🤔

Maybe I'm misunderstanding, but if we have m ranks per node, and we load it onto every rank, we'd have m copies on the node. Compared to loading it onto node rank 0 and having 1 copy.

[inducer]

OIC, if there's multiple ranks per node, sure. But then there's still an easy way out: do one load one per node.

[MTCam]

OIC, if there's multiple ranks per node, sure. But then there's still an easy way out: do one load one per node.

That's what we currently do. One rank (node rank 0) loads the full serial mesh, runs the partitioner (Metis or that-which-will-not-be-named), and distributes it to the rest of the node ranks.

[inducer]

I don't think that's what I meant; I thought one could load the global mesh on one rank of each node and make that responsible for partitioning for the other ranks on the node.

[majosm]

Well, to me at least it sounds like @inducer and @MTCam are saying exactly the same thing. 🙃

[inducer]

Oh. It is. 🤦 Sorry about being confused.

[inducer]

Then that leaves to figure out what's excessively slow in that setting.