-
Notifications
You must be signed in to change notification settings - Fork 3.8k
[microTVM]Add test for MLPerfTiny models #13978
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.
Generated by tvm-bot |
f820fe6 to
6e90b2d
Compare
|
|
||
| #include "api/submitter_implemented.h" | ||
|
|
||
| #include <assert.h> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, it is used in TVMBackendAllocWorkspace
| project = tvm.micro.project.generate_project_from_mlf( | ||
| template_project_path, workspace_dir / "project", model_tar_path, project_options | ||
| ) | ||
| project.build() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is any validation of either the prediction or the running time results done for either the host-driven or mlperftiny tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nevermind, I just reread your summary, 😁, but maybe add docstring with that info...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, we are testing accuracy here. I have added that to the doc string
| @pytest.mark.skip_boards( | ||
| ["mps2_an521", "mps3_an547", "stm32f746g_disco", "nucleo_f746zg", "nrf5340dk_nrf5340_cpuapp"] | ||
| ) | ||
| def test_mlperftiny_models(platform, board, workspace_dir, serial_number, model_name, project_type): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add docstring for test (maybe note what it does and doesn't test)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
mehrdadh
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @alanmacd! I addressed your comments
|
|
||
| #include "api/submitter_implemented.h" | ||
|
|
||
| #include <assert.h> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, it is used in TVMBackendAllocWorkspace
| @pytest.mark.skip_boards( | ||
| ["mps2_an521", "mps3_an547", "stm32f746g_disco", "nucleo_f746zg", "nrf5340dk_nrf5340_cpuapp"] | ||
| ) | ||
| def test_mlperftiny_models(platform, board, workspace_dir, serial_number, model_name, project_type): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| project = tvm.micro.project.generate_project_from_mlf( | ||
| template_project_path, workspace_dir / "project", model_tar_path, project_options | ||
| ) | ||
| project.build() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, we are testing accuracy here. I have added that to the doc string
alanmacd
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
mkatanbaf
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mehrdadh, I left a few comments.
9012819 to
4ec52ba
Compare
* Add test for mlperftiny models for host_driven and EEMBC runner versions * refactor mlperftiny models in different links * fix link * address comments and fix error
This PR adds test for MLPerfTiny models for both
mlperftinyproject type andhost_driven. In host drive, a single input data is used for testing but inmlperftinycase we only build the project.