large imbalance from zoltan/parmetis on hex + mixed mesh

[cwsmith]

mixed.tar.gz

Partitioning the attached mesh and model with global zoltan/parmetis produces a large entity imbalance. The pumi RIB partitioner produces reasonable imbalances.

Note, this mesh has two geometric model faces that are matched.

$ mpirun -np 16 ./test/split gs_bumDNS.smd z2/ z16/ 8
mesh z2/ loaded in 0.807597 seconds
number of tet 0 hex 30745 prism 75580 pyramid 0
mesh entity counts: v 69660 e 245640 f 282305 r 106325
planned RIB factor 8 in 0.108743 seconds
mesh expanded from 2 to 16 parts in 0.415002 seconds
mesh migrated from 2 to 16 in 4.512924 seconds
PARMA_STATUS  disconnected <max avg> 1 0.062
PARMA_STATUS  neighbors <max avg> 5 3.750
PARMA_STATUS  smallest side of max neighbor part 12
PARMA_STATUS  num parts with max neighbors 4
PARMA_STATUS  empty parts 0
PARMA_STATUS  small neighbor counts 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:0 9:0 10:0
PARMA_STATUS  weighted vtx <tot max min avg> 89112.0 10116.0 3116.0 5569.500
PARMA_STATUS  weighted edge <tot max min avg> 290875.0 30890.0 11378.0 18179.688
PARMA_STATUS  weighted face <tot max min avg> 308104.0 31066.0 12989.0 19256.500
PARMA_STATUS  weighted rgn <tot max min avg> 106325.0 10291.0 4726.0 6645.312
PARMA_STATUS  owned bdry vtx <tot max min avg> 5424 813 0 339.000
PARMA_STATUS  shared bdry vtx <tot max min avg> 10944 1065 332 684.000
PARMA_STATUS  model bdry vtx <tot max min avg> 31553 3496 1104 1972.062
PARMA_STATUS  sharedSidesToElements <max min avg> 0.141 0.055 0.087
PARMA_STATUS  entity imbalance <v e f r>: 1.82 1.70 1.61 1.55
MDS: reordering before writing smb files
mesh z16/ written in 2.934006 seconds

imbalance blows up using zoltan!

$ mpirun -np 16 ./test/zsplit gs_bumDNS.smd z2/ z16/ 8
mesh z2/ loaded in 0.358378 seconds
number of tet 0 hex 30745 prism 75580 pyramid 0
mesh entity counts: v 69660 e 245640 f 282305 r 106325
planned Zoltan split factor 8 to target imbalance 1.050000 in 0.466251 seconds
mesh expanded from 2 to 16 parts in 0.442006 seconds
mesh migrated from 2 to 16 in 4.656009 seconds
PARMA_STATUS  disconnected <max avg> 0 0.000
PARMA_STATUS  neighbors <max avg> 6 4.250
PARMA_STATUS  smallest side of max neighbor part 12
PARMA_STATUS  num parts with max neighbors 1
PARMA_STATUS  empty parts 0
PARMA_STATUS  small neighbor counts 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:0 9:0 10:0
PARMA_STATUS  weighted vtx <tot max min avg> 87668.0 24900.0 2980.0 5479.250
PARMA_STATUS  weighted edge <tot max min avg> 288257.0 72585.0 10898.0 18016.062
PARMA_STATUS  weighted face <tot max min avg> 306930.0 70136.0 12451.0 19183.125
PARMA_STATUS  weighted rgn <tot max min avg> 106325.0 22450.0 4532.0 6645.312
PARMA_STATUS  owned bdry vtx <tot max min avg> 3962 603 0 247.625
PARMA_STATUS  shared bdry vtx <tot max min avg> 8038 906 251 502.375
PARMA_STATUS  model bdry vtx <tot max min avg> 31048 8839 1000 1940.500
PARMA_STATUS  sharedSidesToElements <max min avg> 0.110 0.034 0.074
PARMA_STATUS  entity imbalance <v e f r>: 4.54 4.03 3.66 3.38
MDS: reordering before writing smb files
mesh z16/ written in 3.105811 seconds

[KennethEJansen]

As a further update on this issue we can report the following for two meshes:

In the original mesh there are 152M elements with 19% wedges and other 81% hexes. At 32k the max vertex load imbalance is 15 and the max region load imbalance is 14. The path to this case is /projects/UnsAdaptCFD_tesp/balin/NSF_DataDrivenTurbMod/DNS/Preliminary/Chef/32k-4k-Chef-SymmRANS/266672.out on Theta.
Here are the params
18 preAdaptBalanceMethod none
19 prePhastaBalanceMethod parma-gap
20 partitionMethod rib
21 LocalPtn 0

Obviously we cannot run this case with such an imbalance so Riccardo altered the me Simmetrix mesh generator to minimize the number of wedges, reducing them to only 0.05% of the mesh. At 32k parts, the max vertex load imbalance is 1.6 while the max region load imbalance is 1.2. The path to this case is /projects/UnsAdaptCFD_tesp/balin/NSF_DataDrivenTurbMod/DNS/Preliminary/Chef/32k-4k-Chef-Quad/268220.out on Theta.
here are the params.
18 preAdaptBalanceMethod none
19 prePhastaBalanceMethod parma-gap
20 partitionMethod rib
21 LocalPtn 0

I know we should expect some imbalance with mixed meshes but, for all intensive purposes, the second is monotopology so I don't think this issue can be blamed on mixed meshes, rather it seems to be an issue with hexes.

[KennethEJansen]

I asked Riccardo to try zrib. Here is is his report:

partitionMethod=zrib and prePhastaBalanceMethod=zrib does worse than partitionMethod=rib and prePhastaBalanceMethod=graph.
Former gives:
max vertex load imbalance of partitioned mesh = 1.797214
ratio of sum of all vertices to sum of owned vertices = 1.258720
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.233574

Latter gives:
max vertex load imbalance of partitioned mesh = 1.610510
ratio of sum of all vertices to sum of owned vertices = 1.244223
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.209429

Do you think we should report this to the Zoltan team?

Is there a way within Chef to tell it to ignore matching (i.e., not to worry about preserving matching information) to determine if it is this feature that is causing the imbalance?

Obviously we could regenerate the mesh without matching (I think so at least, not sure since it is generated by extrusion) but this would be a lot of work and I am hoping there is a flag that can be turned on instead.

[cwsmith]

Our Zoltan load balancer interface will create graph edges between matched faces:

core/zoltan/apfZoltan.h

Lines 100 to 101 in eccd2c3

	Also note that this Balancer will create a Zoltan edge
	between two elements that share matched faces.

The Zoltan and PUMI RIB balancers don't transform coordinates to account for matching though.

Using the ptnParma tool you can balance with Zoltan RCB and experiment with other methods:

core/test/ptnParma.cc

Lines 56 to 68 in eccd2c3

	std::map<std::string, int> ztnMethod;
	ztnMethod["rib"] = apf::RIB;
	ztnMethod["rcb"] = apf::RCB;
	ztnMethod["hg"] = apf::HYPERGRAPH;
	ztnMethod["pmetis"] = apf::PARMETIS;
	std::map<std::string, int> ztnApproach;
	ztnApproach["ptn"] = apf::PARTITION;
	ztnApproach["reptn"] = apf::REPARTITION;
	ztnApproach["refine"] = apf::REFINE;
	ztnApproach["kway"] = apf::PART_KWAY;
	ztnApproach["geomkway"] = apf::PART_GEOM_KWAY;
	ztnApproach["refkway"] = apf::REFINE_KWAY;

Note, this tool runs parma after the splitter and may be very slow given the large imbalances involved:

mpirun -n 16 ./test/ptnParma gs_bumDNS.dmg bumDNS-Smaller.smb zrcb_16/ 16 rcb ptn 0

Applying the attached patch with git am disableZtnMatch.patch disables
disableZtnMatch.patch.gz
Zoltan matching support during graph construction.

The matched and unmatched 1->16 part Zoltan (calling parmetis) partitions are identical on the test mesh uploaded to this issue. That eliminates the possibility that matching is the cause of the balancing issues here.

$ diff zgraph.log zgraph_noMatch.log
1c1
< mesh bumDNS-Smaller.smb loaded in 0.671875 seconds
---
> mesh bumDNS-Smaller.smb loaded in 0.639380 seconds
4,6c4,6
< planned Zoltan split factor 16 to target imbalance 1.050000 in 0.847459 seconds
< mesh expanded from 1 to 16 parts in 0.623988 seconds
< mesh migrated from 1 to 16 in 6.783949 seconds
---
> planned Zoltan split factor 16 to target imbalance 1.050000 in 0.869006 seconds
> mesh expanded from 1 to 16 parts in 0.595982 seconds
> mesh migrated from 1 to 16 in 6.863939 seconds
23c23
< mesh zgraph_16/ written in 1.594624 seconds
---
> mesh zgraph_nomatch_16/ written in 1.732649 seconds

[KennethEJansen]

Would it make sense for us to ship this case to Karen Devine? My gut feeling is that, given that Sandia is the land of cubit and other hex dominant meshing tools, their tools would be expected to kick but on a hex dominant mesh,not come unbolted.

[KennethEJansen]

I have also asked Riccardo to try to push to higher part counts with Simmetrix before converting to mds. I think it will be interesting to see if Simmetrix partitioner (which is probably vanilla parMetis) manages to achieve a better balance that SCOREC/core.

I am also wondering if there is any chance that the choice of model is playing a role. The mesh was generated from a model created using simTranslate (e.g., it is a geomsim model derived from parasolid but this was done before meshing). I can't think of why this would cause problems but I am just throwing out everything I know in hopes that you who know more of the internals have all the available facts. I mean the model could not be simpler -- 5 faces aligned with cartesian x,y, and z and the 6th face has a small deformation/bumb but it is topologically a cube model (with one curved surface).

[KennethEJansen]

Riccardo reports that we have been able to balance all-hex meshes with single and double periodicity in the past. These were with meshes created using a parasolid model and never translated. Here are the numbers he observed there:
First off, mesh stats
number of tet 0 hex 331200 prism 0 pyramid 0
mesh entity counts: v 336000 e 1003200 f 998400 r 331200

Then, going from 1-32 with partitionMethod=graph and prePhastaBalanceMethod=parma-gap,
max vertex load imbalance of partitioned mesh = 1.239238
ratio of sum of all vertices to sum of owned vertices = 1.184869
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.001063

Repeating with partitionMethod=rib and prePhastaBalanceMethod=parma-gap
max vertex load imbalance of partitioned mesh = 1.158000
ratio of sum of all vertices to sum of owned vertices = 1.155938
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.000000
The numbers reported here don't seem to make much sense. With the same topology, how can there be no region imbalance but vertex imbalance? Maybe I am missing something here.

Repeating with partitionMethod=rib and prePhastaBalanceMethos=none
max vertex load imbalance of partitioned mesh = 1.158000
ratio of sum of all vertices to sum of owned vertices = 1.155938
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.000000

Repeating with partitionMethod=zrib and prePhastaBalanceMethod=zrib
max vertex load imbalance of partitioned mesh = 1.156476
ratio of sum of all vertices to sum of owned vertices = 1.155107
max region (3D) or face (2D) load imbalance of partitioned mesh = 1.000000

not fantastic numbers but this is manageable.. much better than the 1.79 from the case with 0.05% wedges which itself is way better than 1500% (or factor of 15) imbalance when there were 19% wedges. This reminds me that for FUN3D you are trying to balance vertices instead of meshes. Can you remind how to make that switch? I wonder if perhaps directly targeting vertices my work better in these cases (of course we still want the elements to be uniquely partition and the vertices on the part boundaries replicated...just changing what we target as the primary balance).

[KennethEJansen]

Riccardo, Cameron and I met and discussed this. He agreed that it would be good for you to try to push up the partitioning in phParAdapt or perhaps in a Simmetrix-based partitioning code (they have examples but I don’t know if it is worth the time to build them rather than just run phParAdapt in partition only mode) to as high of a part count as you can and compare to what you are seeing in chef regarding imbalances. I think on Theta it should be easy/quick to get up to around 16k since 128 nodes are usually a pretty quick turnaround time. Please try the following jobs: 1) your first mesh with wedges outside of the BL, 2) your hex dominant mesh 3) your channel mesh 4) if possible, the mesh at this path kjansen@portal1: /projects/tools/Models/BoeingBump/CpFromRiccardo $ ls -alt total 256264 -rw-r--r-- 1 kjansen users 25580857 Aug 23 21:11 bumDNS-Smaller.sms Obviously this 4th case is a smaller version of 1) and, if it illustrates the same pathology on cooley at proportional elements/per part (and thus much smaller partitions) this will be easier for the SCOREC/core team to debug the situation with. Best regards,

…

---------------------------------------------------------------- Kenneth E. Jansen, Professor Ann and H.J. Smead Department of Aerospace Engineering Sciences ECAE 161 OFFICE (303) 492-4359 429 UCB FAX (303) 492-4990 University of Colorado at Boulder jansenke@colorado.edu Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace ----------------------------------------------------------------

On Sep 8, 2018, at 6:06 AM, Cameron Smith ***@***.***> wrote: Our Zoltan load balancer interface will create graph edges between matched faces: https://github.com/SCOREC/core/blob/eccd2c3c87011c9af338dd2209291404f2b09df0/zoltan/apfZoltan.h#L100-L101 The Zoltan and PUMI RIB balancers don't transform coordinates to account for matching though. Using the ptnParma tool you can balance with Zoltan RCB and experiment with other methods: https://github.com/SCOREC/core/blob/eccd2c3c87011c9af338dd2209291404f2b09df0/test/ptnParma.cc#L56-L68 Note, this tool runs parma after the splitter and may be very slow given the large imbalances involved: mpirun -n 16 ./test/ptnParma gs_bumDNS.dmg bumDNS-Smaller.smb zrcb_16/ 16 rcb ptn 0 Applying the attached patch with git am disableZtnMatch.patch disables disableZtnMatch.patch.gz Zoltan matching support during graph construction. The matched and unmatched 1->16 part Zoltan (calling parmetis) partitions are identical on the test mesh uploaded to this issue. That eliminates the possibility that matching is the cause of the balancing issues here. $ diff zgraph.log zgraph_noMatch.log 1c1 < mesh bumDNS-Smaller.smb loaded in 0.671875 seconds --- > mesh bumDNS-Smaller.smb loaded in 0.639380 seconds 4,6c4,6 < planned Zoltan split factor 16 to target imbalance 1.050000 in 0.847459 seconds < mesh expanded from 1 to 16 parts in 0.623988 seconds < mesh migrated from 1 to 16 in 6.783949 seconds --- > planned Zoltan split factor 16 to target imbalance 1.050000 in 0.869006 seconds > mesh expanded from 1 to 16 parts in 0.595982 seconds > mesh migrated from 1 to 16 in 6.863939 seconds 23c23 < mesh zgraph_16/ written in 1.594624 seconds --- > mesh zgraph_nomatch_16/ written in 1.732649 seconds — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.