Forked branch-0.16 updated from Release branch-0.16 #33
Merged
aschaffer merged 88 commits intoaschaffer:branch-0.16from Oct 6, 2020
Merged
Forked branch-0.16 updated from Release branch-0.16 #33aschaffer merged 88 commits intoaschaffer:branch-0.16from
aschaffer merged 88 commits intoaschaffer:branch-0.16from
Conversation
…ead of passing pointers/references
…s, removed unused graph_new.pxd (which was identical to graph.pxd, which then got renamed to graph_primtypes.pxd)
[REVIEW] upgrade thrust to latest commit
…t the moment since it's not calling the Louvain C++ API which has not been merged yet.
…, added documentation, added deleted copy and assignment operators, added more legacy graph types for future expansion during the transition.
[REVIEW] Sub-communicator initialization for 2D partitioning support
[REVIEW] MNMG Louvain Python updates, Cython cleanup
* update RAFT version * update change log
* remove RMM_LIBRARY and CXX11 ABI handling * changelog * formatting fix
…es and switch statements to factory function (#1152) * Minor update to comment to describe array sizes. * Changed graph container to use smart pointers, added arg for instantiating legacy types and switch statements for it to factory function. * Added PR 1152 to CHANGELOG.md * Removing unnecessary .get() call on unique_ptr instance * Using make_unique() instead of new * Updated to call drop() correctly after cudf API update. Co-authored-by: Rick Ratzel <rratzel@nvidia.com>
* added simple data gen * style * changelog * added 32 or 64 option for id type * style
* added simple data gen * style * changelog * added 32 or 64 option for id type * style * concat_within_workers * parquet test concat_within_workers * changelog * disable mg test on ci * style ...
* nx to graph code and test * updated pagerank * updates * changelog * updatees to return dictionary rather than a dataframe * added edge scores to dictionary * BC and Edge BC using Nx * next set of updates * ECG * k-truss and a new conversion from cugraph.Graph to NetworkX.Graph * clustering algos * k_truss function * fixed issues * bug fixing * WCC and SCC, upadted code based on DF having different names * core number * k_core * updateed docs * hits * Jaccard and Overlap * flake8 * reset * PR 1147 * update log * updated and fixed * reset * new functions * updated Jaccard * added k_truss * k_truss - fix typo * code review fixes * fix * removed MG testing for more that 1 GPU * updated docs * redid core number test * updated docs * updating docs * flake error fix * skipping Nx test * changed to Graph vs DiGraph * skip MG test Co-authored-by: BradReesWork <BradReesWork@users.noreply.github.com>
* update louvain API * try a more helpful error message to diagnose CI issue
* move Nx test to test_nx_convert.py * better testing * update * changelog Co-authored-by: BradReesWork <BradReesWork@users.noreply.github.com>
…, BFS, & SSSP (#838) * draft pattern accelerator API for the pagerank pattern * implement pagerank using the pattern accelerator API * implement katz centrality using the pattern accelerator API * add handle to the pattern accelerator API * fix minor issues in pagerank & katz_centrality * add a pattern to support BFS * draft implementation of BFS using pattern accelerator APIs * move non-public APIs to the detail namespace * minor tweak to bfs * initial draft of sssp using pattern accelerator * merge e_op and e_pred_op and add reduce_op for bfs & sssp patterns * tweaking patterns for BFS & SSSP for better accelerator implementation * raise abstraction level for vertex queue * direction optimized to direction optimizing in BFS * update comments, class & function names, and several additional API changes to enable optimization * add FIXME comments to remove opg as a template parameter from graph analytics functions * rename frontier to better reflect that it is a froniter on adjacency matrix rows * updated pattern accelerator API for better expressiblity and to enable more performance optimizations in accelerator API implementations * remove template parameter bool opg from graph analytics * remove unnecessary code * .cuh to .cu and explicit instantiation for templated graph analytics functions * "split patterns.hpp to three files" * "add aliases vertex_type, edge_type, weight_type to Graph classes (to support e.g. GraphType::vertex_type)" * "add invalid_vertex_id and invalid_edge_id" * "add traits.hpp (initially supporting is_csr and is_csc)" * "fix typos" * add bfs.cu to CMakeLists.txt * misc. fixes * fix several compile errors * add graph_device_view class * add is_opg to graph classes * add a frontier queue class for pattern accelerators * fix bfs compile errors with pattern accelrator API * initail commit of two level pattern accelrators * fix frontier queue compile errors * few tweaks * initial commit of reduce_op.cuh * improve AdjMatrixRowFrontier implementation * first full implementation of two level patterns for BFS & SSSP * first full ipmlementation of BFS using a pattern accelerator * update copyright year and add min to reduce_op * add sssp to CMakeLists.txt * spilt two_levels_patterns.cuh to one file per pattern * thrust::raw_pointer_cast to data().get() following cuDF's convention * move pattern accelerator related files to the patterns directory * add edge_utils.cuh * add transform_reduce_e pattern accelerator implmentation * add utility functions * update bfs with pattern accelerator implementation * update sssp with pattern accelerator implementation * update graph_device_view * update queue implementation * update expand_and_transform_if_e pattern implementation * placeholder * fix merge error * move implemented patterns out from one_level_patterns.cuh * fix a conceptual bug (row in graph adjacency matrix is always source, CSC is a column major representation) * temporary commit to switch a branch * minor fixes on include statements * add experimental BFS to the test suites * use the real raft handle than the temporary placeholder * add experimental BFS test * several bug fixes * run clang-format * GraphType graph => GraphType const& graph_device_view in pattern accelerator * now BFS passes C++ tests * add depth_limit to the reference BFS implementation * run clang-format * remove dead code * fix to work with new RAFT based error handling mechanism * minor code restructuring * apply cutoff * add SSSP test * cosmetic updates to BFS test * SSSP bug fixes * SSSP code restructuring * update template bfs & sssp functions to take pointers for distances and predecessors instead of general iterators * now SSSP passes C++ tests * add fixme comments * temporary commit to change branch * bug fix in graph_device_view.cuh * compile error fix in bfs_test.cpp * add declarations for PageRank and Katz in algorithms.hpp * bug fix in is_column_major * fix namings * implement patter accelerator APIs for PageRank and Katz Centrality * remove unused file * bug fix * modify reference BFS & SSSP to take pointers instead of iterators * compute adj_matrix_out_weights_sums if not provided (nullptr) in PageRank * rename ..._v..._e to ..._v..._nbr * add utilities for atomic_add (better move this to RAFT) * update experimental SSSP test * bug fix in copy_v_transform_reduce_in|out_nbr * reorder pattern accelerator API input parameters * tweak pattern accelerator API * add PageRank tests with the pattern accelerator API * tweak katz centrality with the pattern accelerator * minor tweak for PageRank test code * add katz centrality test * remove experimental:: from graph classes * style fix (use T{} instead of static_cast<T> for constant values * minor style fix * count_if_adj_matrix_row to any_of_adj_matrix_row (adj_matrix_row values are replicated in p_row processes assuming 2D partitioning and p = p_row * p_col, so count_if can be confusing) * AdjMatrixRowFrontier -> VertexFrontier * break update_frontier_v_push_if_out_nbr to two functions (the second part is replaced with copy_to_adj_matrix_row) * add pure_function flag to reduce_op (if this is defined and set to true, reduction_op can be executed in any GPU in OPG) * add documentation for experimental bfs, sssp, pagerank, and katz_centrality using the pattern accelerator API * add documentation * rename opg to multi-GPU * change get_number_of_edges return type from vertex_type to edge_type * fix compile errors * move the pattern accelerator API out from the detail namespace (this will make migration to RAFT easier) * thrust::cuda::par.on(handle.get_stream()) to rmm::exec_policy(handle.get_stream()).on(handle.get_stream()) * fix typo in comments * escape code from exp_graph.hpp to graph_device_view.cuh * partially update tests to work with the new graph class (more updates are necessary) * temp commit to change branch * update tests to use the new graph class * update algorithm public interface to work with the new graph class * update any_of_adj_matrix_row to support MG * temporary commit for branch change * update to work with the new graph class * fix compile errors * clang-format * replace graph_device_view.cuh with vertex_partition_device.cuh & matrix_partition_device.cuh * undo changes in include/graph.hpp, this file is no longer relevant to this PR * additionally undo changes in include/graph.hpp, this file is no longer relevant to this PR * remove unnecessary comments * remove unnecessary template parameters * add copy_to_adj_matrix_col * replace for loops with thrust algorithms and few minor cosmetic fixes * break unnecessary loop carried dependendy * bug fix * bug fix (previously used plus_thrust_tuple where plus_edge_op_result should be used) * fix erreneous comments * clang-format * fixed a bug (copy_v_transform_reduce_nbr worked with only raw pointer VertexValueOutputIterator type) * update change log * clang-format * remove cuda.cuh (this is replaced by raft) * clang-format * clang-format * update raft tag * remove unecessary code * update sG interface of graph_view.hpp to mirror MG interface * replace comm_p_row_key & comm_p_col_key with key_naming_t().row_name() and col_name() * fixed confusing variable names
…parameter displs (#1158) * accomodate the change of raft allgahterv's input parameter displs type from int[] to size_t[] * update RAFT tag
* add shuffle for 2D partitioning * Update CHANGELOG.md * Update CHANGELOG.md * shuffle updates as per review * updated changelog * add transposed Co-authored-by: Brad Rees <34135411+BradReesWork@users.noreply.github.com> Co-authored-by: BradReesWork <BradReesWork@users.noreply.github.com>
* disabled MG tests on single GPU * changelog * style
…to fail (#1166) * fix misspelling of function calls in asserts causing debug build to fail * update changelog
* updated Katz API * added entry for edge betweennees centrality * Added SSSP, BFS, and subgraph as Nx compatible Co-authored-by: BradReesWork <BradReesWork@users.noreply.github.com>
* saving progress * added missing dependencies for #1159 * init files and test for _df entry point * temporary debug print and alternate way of appending * should work now * removed debug print * disabled MG tests on single GPU * changelog * style * skiped tests using #1168 model * style * restoring flag as per review comment * Update symmetrize.py * Update symmetrize.py
…Centrality, BFS, and SSSP implementations (C++ part) (#1151) * draft pattern accelerator API for the pagerank pattern * implement pagerank using the pattern accelerator API * implement katz centrality using the pattern accelerator API * add handle to the pattern accelerator API * fix minor issues in pagerank & katz_centrality * add a pattern to support BFS * draft implementation of BFS using pattern accelerator APIs * move non-public APIs to the detail namespace * minor tweak to bfs * initial draft of sssp using pattern accelerator * merge e_op and e_pred_op and add reduce_op for bfs & sssp patterns * tweaking patterns for BFS & SSSP for better accelerator implementation * raise abstraction level for vertex queue * direction optimized to direction optimizing in BFS * update comments, class & function names, and several additional API changes to enable optimization * add FIXME comments to remove opg as a template parameter from graph analytics functions * rename frontier to better reflect that it is a froniter on adjacency matrix rows * updated pattern accelerator API for better expressiblity and to enable more performance optimizations in accelerator API implementations * remove template parameter bool opg from graph analytics * remove unnecessary code * .cuh to .cu and explicit instantiation for templated graph analytics functions * "split patterns.hpp to three files" * "add aliases vertex_type, edge_type, weight_type to Graph classes (to support e.g. GraphType::vertex_type)" * "add invalid_vertex_id and invalid_edge_id" * "add traits.hpp (initially supporting is_csr and is_csc)" * "fix typos" * add bfs.cu to CMakeLists.txt * misc. fixes * fix several compile errors * add graph_device_view class * add is_opg to graph classes * add a frontier queue class for pattern accelerators * fix bfs compile errors with pattern accelrator API * initail commit of two level pattern accelrators * fix frontier queue compile errors * few tweaks * initial commit of reduce_op.cuh * improve AdjMatrixRowFrontier implementation * first full implementation of two level patterns for BFS & SSSP * first full ipmlementation of BFS using a pattern accelerator * update copyright year and add min to reduce_op * add sssp to CMakeLists.txt * spilt two_levels_patterns.cuh to one file per pattern * thrust::raw_pointer_cast to data().get() following cuDF's convention * move pattern accelerator related files to the patterns directory * add edge_utils.cuh * add transform_reduce_e pattern accelerator implmentation * add utility functions * update bfs with pattern accelerator implementation * update sssp with pattern accelerator implementation * update graph_device_view * update queue implementation * update expand_and_transform_if_e pattern implementation * placeholder * fix merge error * move implemented patterns out from one_level_patterns.cuh * fix a conceptual bug (row in graph adjacency matrix is always source, CSC is a column major representation) * temporary commit to switch a branch * minor fixes on include statements * add experimental BFS to the test suites * use the real raft handle than the temporary placeholder * add experimental BFS test * several bug fixes * run clang-format * GraphType graph => GraphType const& graph_device_view in pattern accelerator * now BFS passes C++ tests * add depth_limit to the reference BFS implementation * run clang-format * remove dead code * fix to work with new RAFT based error handling mechanism * minor code restructuring * apply cutoff * add SSSP test * cosmetic updates to BFS test * SSSP bug fixes * SSSP code restructuring * update template bfs & sssp functions to take pointers for distances and predecessors instead of general iterators * now SSSP passes C++ tests * add fixme comments * temporary commit to change branch * bug fix in graph_device_view.cuh * compile error fix in bfs_test.cpp * add declarations for PageRank and Katz in algorithms.hpp * bug fix in is_column_major * fix namings * implement patter accelerator APIs for PageRank and Katz Centrality * remove unused file * bug fix * modify reference BFS & SSSP to take pointers instead of iterators * compute adj_matrix_out_weights_sums if not provided (nullptr) in PageRank * rename ..._v..._e to ..._v..._nbr * add utilities for atomic_add (better move this to RAFT) * update experimental SSSP test * bug fix in copy_v_transform_reduce_in|out_nbr * reorder pattern accelerator API input parameters * tweak pattern accelerator API * add PageRank tests with the pattern accelerator API * tweak katz centrality with the pattern accelerator * minor tweak for PageRank test code * add katz centrality test * remove experimental:: from graph classes * style fix (use T{} instead of static_cast<T> for constant values * minor style fix * count_if_adj_matrix_row to any_of_adj_matrix_row (adj_matrix_row values are replicated in p_row processes assuming 2D partitioning and p = p_row * p_col, so count_if can be confusing) * AdjMatrixRowFrontier -> VertexFrontier * break update_frontier_v_push_if_out_nbr to two functions (the second part is replaced with copy_to_adj_matrix_row) * add pure_function flag to reduce_op (if this is defined and set to true, reduction_op can be executed in any GPU in OPG) * add documentation for experimental bfs, sssp, pagerank, and katz_centrality using the pattern accelerator API * add documentation * rename opg to multi-GPU * change get_number_of_edges return type from vertex_type to edge_type * fix compile errors * move the pattern accelerator API out from the detail namespace (this will make migration to RAFT easier) * thrust::cuda::par.on(handle.get_stream()) to rmm::exec_policy(handle.get_stream()).on(handle.get_stream()) * fix typo in comments * escape code from exp_graph.hpp to graph_device_view.cuh * partially update tests to work with the new graph class (more updates are necessary) * temp commit to change branch * update tests to use the new graph class * update algorithm public interface to work with the new graph class * update any_of_adj_matrix_row to support MG * temporary commit for branch change * update to work with the new graph class * fix compile errors * clang-format * replace graph_device_view.cuh with vertex_partition_device.cuh & matrix_partition_device.cuh * undo changes in include/graph.hpp, this file is no longer relevant to this PR * additionally undo changes in include/graph.hpp, this file is no longer relevant to this PR * remove unnecessary comments * remove unnecessary template parameters * add copy_to_adj_matrix_col * replace for loops with thrust algorithms and few minor cosmetic fixes * break unnecessary loop carried dependendy * bug fix * bug fix (previously used plus_thrust_tuple where plus_edge_op_result should be used) * fix erreneous comments * clang-format * fixed a bug (copy_v_transform_reduce_nbr worked with only raw pointer VertexValueOutputIterator type) * update change log * clang-format * remove cuda.cuh (this is replaced by raft) * clang-format * clang-format * update raft tag * multi-gpu extension for count_if_v & count_if_e & reduce_v * transform_reduce_v * mark FIXME * multi-gpu extension for transform_reduce_v_with_adj_matrix_row * remove unecessary code * update sG interface of graph_view.hpp to mirror MG interface * replace comm_p_row_key & comm_p_col_key with key_naming_t().row_name() and col_name() * delete unused code * update comment about hyper-graph partitioning based matrix partitioning * add missing #include <raft/handle.hpp> * move block_reduce_edge_op_result form transform_reduce_e.cuh to edge_op_result.cuh * fix misnomer * add warp_reduce_edge_op_result * add vectorized (warp, block) functions * update to use collective_utils.cuh * add additional functions to graph_view * fixed confusing variable names * reorder functions to be consistent with elsewhere * fix a compiler error * accomodate the change of raft allgahterv's input parameter displs type from int[] to size_t[] * update change log * update change log * update RAFT tag * temporary commit of copy_to_adj_matrix_row.cuh and collective_utils.cuh to checkout another branch * fix a bug in matrix partitioning ranges * temporary commit * fix errors in previous merge conflicts * extend copy_to_adj_matrix_row.cuh for MNMG * rename collective_utils.cuh to comm_utils.cuh * rename copy_v_transform_reduce_nbr.cuh to copy_v_transform_reduce_in_out_nbr.cuh * merge copy_to_adj_matrix_row.cuh & copy_to_adj_matrix_col.cuh * extend copy_v_transform_reduce_(in|out)_nbr for MNMG * extend Bucket for MNMG * add get_vertex_partition_size * add more explicit instantiation cases for BFS, SSSP, PageRank, KatzCentrality * extend update_frontier_v_push_if_out_nbr.cuh for MNMG * code refinement * refactor copy_v_transform_reduce_in|out_nbr * bug fix (thanks Rick) * clang-format * bug fix * bug fixes * bug fix * add comments explaining major/minor * update change log * bug fix (calling raft communication collective with host memory pointer)
* update ci/local/README.md * Update CHANGELOG.md
* Minor update to comment to describe array sizes. * Changed graph container to use smart pointers, added arg for instantiating legacy types and switch statements for it to factory function. * Added PR 1152 to CHANGELOG.md * Removing unnecessary .get() call on unique_ptr instance * Using make_unique() instead of new * Updated to call drop() correctly after cudf API update. * Added args to support calling get_vertex_identifiers(). * Style fixes, removed commented out code meant for a future change. * Updated comment with description of new 'identifiers' arg. * Safety commit, still WIP, does not compile - updates for 2D graph support and upcoming 2D shuffle support * safety commit, does not pass tests: updated enough to be able to run the MG Louvain test. * Updated call_louvain() to use the new graph_t types. Still WIP, needs louvain updates to compile. * WIP: updates for incorporating new 2D shuffle data, still does not pass test. * Adding updates from iroy30 for calling shuffle from louvain.py * Updated to extract and pass the partition_t info and call the graph_t ctor. Now having a problem finding the right subcommunicator. * Updates to set up subcomms - having a problem with something needed by subcomms not being initialized: "address not mapped to object at address (nil)" * Added p2p flag to comms initialize() to enable initialization of UCX endpoints needed for MG test. * safety commit: committing with debug prints to allow other team members to debug in parallel. * safety commit: more updates to address problems instantiating graph_t (using num edges for partition instead of global for edgelist) and for debugging (print statments). * Changing how row and col rank are obtained, added debug prints for edge lists info * Fixes to partition_t get_matrix_partition_major/minor methods based on feedback. * Update shuffle.py * Integrating changes from iroy30 to produce "option 1" shuffle output by default, with an option to enable "option 2", temporarily enabled graph expensive checks for debugging. * Addressed review feedback: made var names consistent, fixed weights=None bug in cython code, added copyright to shuffle.py, changed how ranks are retrieved from the raft handle. * Removed debug prints. * Added PR 1163 to CHANGELOG.md * Removed extra newlines accidentally added to clean up diff in the PR, updated comment in cython code. * Added specific newlines back so file does not differ unnecessarily. * Disabled graph_t expensive check that was left enabled for debugging. Co-authored-by: Rick Ratzel <rratzel@nvidia.com> Co-authored-by: Iroy30 <41401566+Iroy30@users.noreply.github.com>
* added cpp packages to dev env * changelog
* Update CMakeLists.txt * Update CHANGELOG.md
* FIX Fix notebook error handlinig * DOC Changelog update
…types (#1178) * Minor update to comment to describe array sizes. * Changed graph container to use smart pointers, added arg for instantiating legacy types and switch statements for it to factory function. * Added PR 1152 to CHANGELOG.md * Removing unnecessary .get() call on unique_ptr instance * Using make_unique() instead of new * Updated to call drop() correctly after cudf API update. * Added args to support calling get_vertex_identifiers(). * Style fixes, removed commented out code meant for a future change. * Updated comment with description of new 'identifiers' arg. * Safety commit, still WIP, does not compile - updates for 2D graph support and upcoming 2D shuffle support * safety commit, does not pass tests: updated enough to be able to run the MG Louvain test. * Updated call_louvain() to use the new graph_t types. Still WIP, needs louvain updates to compile. * WIP: updates for incorporating new 2D shuffle data, still does not pass test. * Adding updates from iroy30 for calling shuffle from louvain.py * Updated to extract and pass the partition_t info and call the graph_t ctor. Now having a problem finding the right subcommunicator. * Updates to set up subcomms - having a problem with something needed by subcomms not being initialized: "address not mapped to object at address (nil)" * Added p2p flag to comms initialize() to enable initialization of UCX endpoints needed for MG test. * some proposed cleanup * safety commit: committing with debug prints to allow other team members to debug in parallel. * new technique for factory * safety commit: more updates to address problems instantiating graph_t (using num edges for partition instead of global for edgelist) and for debugging (print statments). * Changing how row and col rank are obtained, added debug prints for edge lists info * Fixes to partition_t get_matrix_partition_major/minor methods based on feedback. * Update shuffle.py * Integrating changes from iroy30 to produce "option 1" shuffle output by default, with an option to enable "option 2", temporarily enabled graph expensive checks for debugging. * Addressed review feedback: made var names consistent, fixed weights=None bug in cython code, added copyright to shuffle.py, changed how ranks are retrieved from the raft handle. * Removed debug prints. * Added PR 1163 to CHANGELOG.md * Removed extra newlines accidentally added to clean up diff in the PR, updated comment in cython code. * Added specific newlines back so file does not differ unnecessarily. * Disabled graph_t expensive check that was left enabled for debugging. * Added code path in call_louvain to support legacy graph types, to be removed when migration to graph_t types is complete. * Updates based on feedback from PR 1163: code cleanup/removed unused union members, consolidated legacy enum types, updated comments, initial support added for 64-bit vertex types (untested) * plumbed bool set based on running renumbering to set sorted_by_degree flag in graph container. * Added PR 1178 to CHANGELOG.md, C++ style fixes. * Addressed PR review feedback: added support for proper edge_t in cython wrapper and removed unnecessary vertex_t/edge_t int64,int32 combinations. Co-authored-by: Rick Ratzel <rratzel@nvidia.com> Co-authored-by: Chuck Hastings <charlesh@nvidia.com> Co-authored-by: Iroy30 <41401566+Iroy30@users.noreply.github.com>
* instlling raft headers under cugraph
* cuhornet tag * Update CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aschaffer/branch-0.16 <- rapidsai/branch-0.16