Neighbor based analysing tool#642
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Thank you @Tagebh, the functionality in your comment will be very useful to have in orix! When adding your own commits, can you use the develop branch as the target branch here in the orix repo? |
Tagebh
changed the title
Tagebh
changed the title
|
Hey @hakonanes I changed the PR so it commits to the developer branch, and I added the code to orix/crystal_map/_neighbors.py. I also made the PR ready for review |
|
Great, thank you! Just to confirm, is it fine with you if I push to your branch? I plan to push stuff like tests, documentation, and perhaps some restructuring of your file. Alternatively, I can make a PR to the develop branch in your fork, so you can go through my suggested changes before being visible here. |
|
@hakonanes and @Tagebh , do either of you care if I try tackling this actually? Same comment as @hakonanes, making pushes directly to the original branch. @hakonanes, if you've already started, ignore this comment, there are plenty of other things for me to do in ORIX. Also, general question: how do we feel about this pattern of keeping all the neighborhood functions in a |
|
@argerlt, as for me you are more than welcome to tackle this problem if you want to. I don't really know if @hakonanes has started or not, so it's probably best to wait for him to answer as well. I feel like that it's natural to add new neighborhood functions into |
|
@Tagebh, sorry this took so long, thank you for your patience. Some starting notes:
to something like this:
|
for more information, see https://pre-commit.ci
|
^ If you click on pre-commit.ci's commit above, you can see what rules it enforced. |
| from skimage.util._map_array import map_array | ||
|
|
||
| from orix.crystal_map import CrystalMap | ||
| from orix.quaternion import Orientation |
There was a problem hiding this comment.
| from orix.quaternion import Orientation | |
| import orix.quaternion as oqu |
There was a problem hiding this comment.
Not an official change yet, but the current practice is to import modules as described here to avoid confusion
|
|
||
| from orix.crystal_map import CrystalMap | ||
| from orix.quaternion import Orientation | ||
|
|
There was a problem hiding this comment.
I believe the following function will replace '_raveled_offsets_and_distances`
| def _raveled_offsets(array_shape: tuple, | |
| kernel: np.ndarray): | |
| """Compute neighboring pixel offsets in raveled coordinate space. | |
| This is a simlification of the "_raveled_offsets_and_distances" | |
| function in scikit-image.morphology._util, version 0.26.0. | |
| Parameters | |
| ---------- | |
| array_shape | |
| the shape of the array the kernel is being applied to. | |
| equivalent to `array.shape`. | |
| kernel | |
| the 2D array representing the neighborhood. this is normally | |
| a 2D Von Neumann neighborhood such as [[0,1,0],[1,0,1],[0,1,0]], | |
| but could also be a larger array of weighted values. all | |
| non-zero entries will produce an offset value. | |
| Returns | |
| ------- | |
| raveled_offsets | |
| an array of offsets for each non-zero entry in the kernel. | |
| """ | |
| center = tuple(s // 2 for s in kernel.shape) | |
| offsets = np.stack( | |
| [(idx - c) for idx, c in zip(np.nonzero(kernel), center)], axis=-1 | |
| ) | |
| ravel_factors = array_shape[1:] + (1,) | |
| raveled_offsets = (offsets * ravel_factors).sum(axis=1) | |
| return np.sort(raveled_offsets) |
| from orix.quaternion import Orientation | ||
|
|
||
|
|
||
| def Neighbors(data, foot, mask=None): |
There was a problem hiding this comment.
Few comments on this function:
-
Avoid capitalizing functions
-
In general, to improve usability and reduce maintenance down-the-road, we want to require as few inputs and return as few outputs as necessary in a function. I believe (though feel free to disagree, the minimum you need for this function is:
_find_neighbors(mask, kernel)--> [neighbors, indices]
Everything else you output can be quickly calculated if and when you need if from just the nxm raveled neighbors list, I would also go a step farther and make the indices an optional output since they can be calculated in one line from the mask.
| def Neighbors(data, foot, mask=None): | ||
| """Performs the general preperations to do neighbor-based calculations on CrystalMaps in a vectorized form | ||
|
|
||
| Args: | ||
| data: 2d array. Can be a CrystalMap object | ||
| mask: 2D array with same shape as CrystalMap indicating what points are valid or not eg. Non-indexed points | ||
| foot: The kernel that defines which neighbors to look at. Can be both binary and not | ||
|
|
||
| Returns: (maybe should be returned as an object instead?) | ||
| indices: Array where the index for each valid point is repeated as many times as it has valid neighbors | ||
| neighbor_indices: Array with the corosponding valid neighbors | ||
| foot_values: contains how each neighbor in neighbor_indices is weighted defined by a given footprine/kernel | ||
| num_neigbors_valid: the number of neighbors each valid point has | ||
| nodes_valid: The index for valid points with at least one valid neighbor | ||
| nodes_no_valid_neighbors: The index for valid points with no valid neighbors | ||
| nodes_not_valid: The index for non-valid points | ||
|
|
||
| Raises: | ||
| Have to add ValueErrors | ||
| """ | ||
| if mask is None: | ||
| if isinstance(data, CrystalMap): | ||
| mask = data.is_indexed.reshape(data.shape) | ||
| else: | ||
| mask = np.full(data.shape, True) # All pointd are valid | ||
| nodes = np.flatnonzero(mask) | ||
|
|
||
| pad = np.max(foot.shape) // 2 | ||
| padded_mask = np.pad(mask, pad, mode="constant", constant_values=False) | ||
| padded_nodes = np.flatnonzero(padded_mask) | ||
|
|
||
| # The offset are given in 1D for the neighbors defined in 2D | ||
| neighbor_offsets, dist = _raveled_offsets_and_distances( | ||
| padded_mask.shape, footprint=foot | ||
| ) | ||
|
|
||
| padded_neighbors = padded_nodes[:, np.newaxis] + neighbor_offsets | ||
| neighbors = map_array( | ||
| padded_neighbors, padded_nodes, nodes | ||
| ) # finds the indeces (in a non-padded format) for the neighbors belonging to valid points | ||
| neighbors_mask = padded_mask.reshape(-1)[ | ||
| padded_neighbors | ||
| ] # sets which of those neigbors are valid | ||
|
|
||
| num_neighbors = np.sum( | ||
| neighbors_mask, axis=1 | ||
| ) # the number of neighbors each valid point has | ||
| indices = np.repeat( | ||
| nodes, num_neighbors | ||
| ) # Array where the index for each valid point is repeated as many times as it has valid neighbors | ||
| neighbor_indices = neighbors[ | ||
| neighbors_mask | ||
| ] # Array with the corosponding valid neighbors | ||
|
|
||
| # Support for non-binary footprints | ||
| foot_offsets, dist = _raveled_offsets_and_distances(foot.shape, footprint=foot) | ||
| foot_nodes = foot_offsets + foot.size // 2 | ||
| foot_values = foot.reshape(-1)[ | ||
| np.repeat([foot_nodes], mask.sum(), axis=0)[neighbors_mask] | ||
| ] # contains how each neighbor in neighbor_indices is weighted defined by a given footprine | ||
|
|
||
| valid_neighbor_mask = num_neighbors != 0 | ||
| nodes_valid = nodes[ | ||
| valid_neighbor_mask | ||
| ] # The index for valid points with at least one valid neighbor | ||
| nodes_no_valid_neighbors = nodes[ | ||
| ~valid_neighbor_mask | ||
| ] # The index for valid points with no valid neighbors | ||
| nodes_invalid = np.flatnonzero( | ||
| ~mask | ||
| ) # The index for non-valid points (same as the mask input, not really neccesary) | ||
|
|
||
| num_neighbors_valid = num_neighbors[valid_neighbor_mask] | ||
|
|
||
| return ( | ||
| indices, | ||
| neighbor_indices, | ||
| foot_values, | ||
| num_neighbors_valid, | ||
| nodes_valid, | ||
| nodes_no_valid_neighbors, | ||
| nodes_invalid, | ||
| ) |
There was a problem hiding this comment.
Here is my suggestion for an alternate function, which will require editing the remaining functions slightly, but I think will be far more useful down the road
| def Neighbors(data, foot, mask=None): | |
| """Performs the general preperations to do neighbor-based calculations on CrystalMaps in a vectorized form | |
| Args: | |
| data: 2d array. Can be a CrystalMap object | |
| mask: 2D array with same shape as CrystalMap indicating what points are valid or not eg. Non-indexed points | |
| foot: The kernel that defines which neighbors to look at. Can be both binary and not | |
| Returns: (maybe should be returned as an object instead?) | |
| indices: Array where the index for each valid point is repeated as many times as it has valid neighbors | |
| neighbor_indices: Array with the corosponding valid neighbors | |
| foot_values: contains how each neighbor in neighbor_indices is weighted defined by a given footprine/kernel | |
| num_neigbors_valid: the number of neighbors each valid point has | |
| nodes_valid: The index for valid points with at least one valid neighbor | |
| nodes_no_valid_neighbors: The index for valid points with no valid neighbors | |
| nodes_not_valid: The index for non-valid points | |
| Raises: | |
| Have to add ValueErrors | |
| """ | |
| if mask is None: | |
| if isinstance(data, CrystalMap): | |
| mask = data.is_indexed.reshape(data.shape) | |
| else: | |
| mask = np.full(data.shape, True) # All pointd are valid | |
| nodes = np.flatnonzero(mask) | |
| pad = np.max(foot.shape) // 2 | |
| padded_mask = np.pad(mask, pad, mode="constant", constant_values=False) | |
| padded_nodes = np.flatnonzero(padded_mask) | |
| # The offset are given in 1D for the neighbors defined in 2D | |
| neighbor_offsets, dist = _raveled_offsets_and_distances( | |
| padded_mask.shape, footprint=foot | |
| ) | |
| padded_neighbors = padded_nodes[:, np.newaxis] + neighbor_offsets | |
| neighbors = map_array( | |
| padded_neighbors, padded_nodes, nodes | |
| ) # finds the indeces (in a non-padded format) for the neighbors belonging to valid points | |
| neighbors_mask = padded_mask.reshape(-1)[ | |
| padded_neighbors | |
| ] # sets which of those neigbors are valid | |
| num_neighbors = np.sum( | |
| neighbors_mask, axis=1 | |
| ) # the number of neighbors each valid point has | |
| indices = np.repeat( | |
| nodes, num_neighbors | |
| ) # Array where the index for each valid point is repeated as many times as it has valid neighbors | |
| neighbor_indices = neighbors[ | |
| neighbors_mask | |
| ] # Array with the corosponding valid neighbors | |
| # Support for non-binary footprints | |
| foot_offsets, dist = _raveled_offsets_and_distances(foot.shape, footprint=foot) | |
| foot_nodes = foot_offsets + foot.size // 2 | |
| foot_values = foot.reshape(-1)[ | |
| np.repeat([foot_nodes], mask.sum(), axis=0)[neighbors_mask] | |
| ] # contains how each neighbor in neighbor_indices is weighted defined by a given footprine | |
| valid_neighbor_mask = num_neighbors != 0 | |
| nodes_valid = nodes[ | |
| valid_neighbor_mask | |
| ] # The index for valid points with at least one valid neighbor | |
| nodes_no_valid_neighbors = nodes[ | |
| ~valid_neighbor_mask | |
| ] # The index for valid points with no valid neighbors | |
| nodes_invalid = np.flatnonzero( | |
| ~mask | |
| ) # The index for non-valid points (same as the mask input, not really neccesary) | |
| num_neighbors_valid = num_neighbors[valid_neighbor_mask] | |
| return ( | |
| indices, | |
| neighbor_indices, | |
| foot_values, | |
| num_neighbors_valid, | |
| nodes_valid, | |
| nodes_no_valid_neighbors, | |
| nodes_invalid, | |
| ) | |
| def _find_neighbors(mask: np.ndarray | CrystalMap, | |
| kernel: int | np.ndarray = 8, | |
| return_indices: bool=False, | |
| ): | |
| """Given a 2D boolean array, return the neighbors for each pixel. | |
| Parameters | |
| ---------- | |
| mask | |
| kernel | |
| return_indices | |
| Returns | |
| ------- | |
| neighbors | |
| indices | |
| """ | |
| # Convert mask to a 2D aray of booleans | |
| if isinstance(mask, CrystalMap): | |
| mask = mask.is_indexed.reshape(mask.shape) | |
| mask = np.atleast_2d(mask).astype(bool) | |
| # Convert kernel to a 2D array of floats | |
| if isinstance(kernel, int): | |
| von_neuman_dict = { | |
| 4:np.array([[0,1,0],[1,0,1],[0,1,0]]), | |
| 8:np.array([[1,1,1],[1,0,1],[1,1,1]]), | |
| } | |
| try: | |
| kernel = von_neuman_dict[kernel] | |
| except KeyError: | |
| raise ValueError( | |
| "'kernel` must be either 4, 8, or a 2D numpy array") | |
| kernel = np.atleast_2d(kernel).astype(float) | |
| if not np.all([mask.ndim ==2,kernel.ndim==2]): | |
| raise ValueError("_find_neighbors only supports 2D arrays") | |
| # calculate the padding, offsets, and indices's of the queried points. | |
| # note: padding must be at least one in all directions. | |
| pad = [(max(x//2,1),max(x-(x//2)-1,1)) for x in kernel.shape] | |
| offsets = _raveled_offsets(padded_mask.shape, kernel) | |
| nodes = np.flatnonzero(mask) | |
| # calculate neighbors for the padded array. | |
| padded_mask = np.pad(mask, pad, mode='constant', constant_values=False) | |
| padded_nodes = np.flatnonzero(padded_mask) | |
| neighbors = padded_nodes[:, np.newaxis] + offsets | |
| # replace the padded indices with the correct unpadded ones, and set | |
| # out-of-bounds or invalid neighbor indices to -1 | |
| # NOTE: this section is roughly equivalent to the following, faster | |
| # Cython code: | |
| # from skimage.util._map_array import map_array | |
| # neighbors = map_array(neighbors, padded_nodes, nodes+1) -1 | |
| # | |
| # However, _map_array is a private function in skimage 0.26, so the | |
| # following vectorized method is used instead. If this slowdown becomes | |
| # problematic in the future, we should consider writing our own cython | |
| # code. | |
| is_neighbor = np.isin(neighbors,padded_nodes) | |
| depad_dict = dict(zip(padded_nodes, nodes)) | |
| depad_func = np.vectorize(lambda x: depad_dict.get(x,-1)) | |
| neighbors[is_neighbor] = depad_func(neighbors[is_neighbor]) | |
| neighbors[~is_neighbor] = -1 | |
| if return_indices: | |
| return neighbors, indices | |
| return neigbors |
| ) | ||
|
|
||
|
|
||
| def neighbor_misorientation( |
There was a problem hiding this comment.
I'm not sure this needs to be a function, but also the quick one-line version of this is:
(ori1*~ori2).reduce().angle
(reads as "find the difference between ori1 and ori2, reduce them with regard to their symmetries, and return just the angle")
it also works for big arrays, ie:
oris = xmap.orientations
misos = oris[nodes,np.newaxis]*~oris[neighbors]
| return mis_ori | ||
|
|
||
|
|
||
| def KAM_calc( |
There was a problem hiding this comment.
This function should take xmap, and kernel as required inputs and mask and phases as optional (you can skip phase and I can do it if you want, but for multiphase EBSD's, there should be options for all, some, or none of the phases to be included). The docstrings should also be reformatted and the name changed (I'd recommend _kernel_averaged_misorientation_map or '_calc_kam`)
.... also, not an issue now, but down the road, should we include a grain_map input? Most software calculates KAM maps intragrainularly.
| return kam_map_im | ||
|
|
||
|
|
||
| def NSN_calc( |
There was a problem hiding this comment.
Same advice as with KAM_calc.
|
@Tagebh, I added some comments and suggested changes as a review, feel free to ask for clarification or suggest alternatives. Also, once these changes are made, you/we will need to write unit tests and examples. |
3 participants
Description of the change
I am currently working on hybrid indexing as part of my master at NTNU, and I also wrote a project thesis on hybrid indexing in the autumn of last year, where I used the hybrid indexing tutorial made på Håkon as a starting point. As part of this work I have made a Kernel Average Misorientation (KAM) tool based on the code from @argerlt in issue #531.
Progress of the PR
I have structured the code so it has the potential to be used for many things involving neighbors (not only KAM).
The Neighbors() function performs the general preperations to do neighbor-based calculations on a xmap or any 2d array really, and can be used on it's own to do whatever you want with neighbors.
The neighbor_misorientation() and KAM_calc() then uses Neighbors() to make a KAM map.
I have also included a slight variation of KAM which I am calling Number of Same Neighbors, which also uses the results from Neighbors() and neighbor_misorientation() but does a slightly different calculation. A user defines what degree of misorientation that counts as a "different" orientation, and then it counts how many neighbors has the same orientation as a central point, where each point in the dataset acts as a central point once.
Here is the functions as .py file
_neighbors.py
The reulting KAM and NDN (Number of different neigbors (NDN) = total neighbors - NSN) map using a 3x3 binary kernel (8 neighbors with equal weighting) could look something like this (using one of the nickel datasets already in kikuchipy):
I have also made a small jupyter notebook that I used to test the functions
test_neighbors.ipynb
Here is an example from that notebook where the amount of equal colored neighbors is counted in a small dummy dataset
The output with a 3x3 binary footprint
The output when the footprint is 3x3 and not binary (0.5 in the corners)
I do not have any experience with git hub, so I do not really know how to impliment it, but I hope this seems usefull and that someone wants to pick it up.
Kind regards
Tage
For reviewers
__init__.py.section in
CHANGELOG.rst.__credits__inorix/__init__.pyand in.zenodo.json.