Skip to content

Add (de)serialization support for data structures in lib/pcg.#988

Closed
tarunprabhu wants to merge 16 commits intoflexflow:repo-refactorfrom
tarunprabhu:repo-refactor
Closed

Add (de)serialization support for data structures in lib/pcg.#988
tarunprabhu wants to merge 16 commits intoflexflow:repo-refactorfrom
tarunprabhu:repo-refactor

Conversation

@tarunprabhu
Copy link

@tarunprabhu tarunprabhu commented Aug 18, 2023

BROKEN: This will probably not build. I am adding this PR so you have at least something to look at sooner rather than later.

  • Have types use the visitable interface.

  • Add stubs for functions/methods that have a declaration but not definition. These should fail with a "not implemented" message.

  • Some code clean up. Remove unnecessary constructors. Delete code that has been commented out. Remove files that don't contain any non-header files that do not contain any executable code.

Description of changes:

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

Before merging:

  • Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

This change is Reviewable

@tarunprabhu tarunprabhu requested a review from lockshaw August 21, 2023 22:22
@tarunprabhu tarunprabhu marked this pull request as draft August 21, 2023 22:22
@lockshaw lockshaw linked an issue Aug 29, 2023 that may be closed by this pull request
  - Have types use the visitable interface.

  - Add stubs for functions/methods that have a declaration but not definition.
    These should fail with a "not implemented" message.

  - Some code clean up. Remove unnecessary constructors. Delete code that has
    been commented out. Remove files that don't contain any non-header files
    that do not contain any executable code.
lockshaw and others added 15 commits September 19, 2023 14:42
…there are still missing functions. The lowering to v1 for the CompGraphOperatorAttrs is missing a lot.
…ors. Add stubs with NOT_IMPLEMENTED calls for functions and methods whose definitions are missing.
…tures. Some other to_v1 methods have been provided but will error at runtime due to hitting a NOT_IMPLEMENTED macro. Disable some fields and types because of a bug where std::vector is reported as being not serializable to json. The sole test case builds, but fails because of hitting a a NOT_IMPLEMENTED assertion.
…sabled attrs since the std::vector bug has been fixed. Make the Tensor and ParallelTensor more consistent. The test builds, but doesn't actually test anything.
…m_v1 still remain for the graph as a whole. Add a test for json in lib/utils. Fix some of the parallel tensor classes in op-attrs to be a bit more consistent with the serial tensor, but that may need some changing as well.
…or some

incomplete code.

There is currently a bug in deserializing FlexFlow::req<T> that prevents the
code from building.
…sts.

Add an explicit call to join_strings in the FlexFlow namespace in the fmt.
@tarunprabhu tarunprabhu marked this pull request as ready for review November 27, 2023 17:57
@tarunprabhu tarunprabhu changed the title WIP: Add (de)serialization support for data structures in lib/pcg. Add (de)serialization support for data structures in lib/pcg. Nov 27, 2023
@tarunprabhu
Copy link
Author

Code for serialization, deserialization to/from v1 and json have been implemented for most (but not all) objects. Tests have been added for most of these as well. Deserialization tests are currently missing due to a bug in req (if I can build a minimum reproducer, I will file an issue and link the issue here).

@lockshaw
Copy link
Collaborator

Superseded by #1394.

@lockshaw lockshaw closed this May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ModelCompilationResult serialization and deserialization code

3 participants