Skip to content

Conversation

@mehrdadh
Copy link
Member

This PR adds test for MLPerfTiny models for both mlperftiny project type and host_driven. In host drive, a single input data is used for testing but in mlperftiny case we only build the project.

@tvm-bot
Copy link
Collaborator

tvm-bot commented Feb 14, 2023

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

Generated by tvm-bot

@mehrdadh mehrdadh requested a review from guberti February 14, 2023 00:35
@mehrdadh mehrdadh force-pushed the micro/add_mlperftiny_models branch from f820fe6 to 6e90b2d Compare February 14, 2023 01:01

#include "api/submitter_implemented.h"

#include <assert.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it is used in TVMBackendAllocWorkspace

project = tvm.micro.project.generate_project_from_mlf(
template_project_path, workspace_dir / "project", model_tar_path, project_options
)
project.build()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is any validation of either the prediction or the running time results done for either the host-driven or mlperftiny tests?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nevermind, I just reread your summary, 😁, but maybe add docstring with that info...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, we are testing accuracy here. I have added that to the doc string

@pytest.mark.skip_boards(
["mps2_an521", "mps3_an547", "stm32f746g_disco", "nucleo_f746zg", "nrf5340dk_nrf5340_cpuapp"]
)
def test_mlperftiny_models(platform, board, workspace_dir, serial_number, model_name, project_type):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add docstring for test (maybe note what it does and doesn't test)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Member Author

@mehrdadh mehrdadh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alanmacd! I addressed your comments


#include "api/submitter_implemented.h"

#include <assert.h>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it is used in TVMBackendAllocWorkspace

@pytest.mark.skip_boards(
["mps2_an521", "mps3_an547", "stm32f746g_disco", "nucleo_f746zg", "nrf5340dk_nrf5340_cpuapp"]
)
def test_mlperftiny_models(platform, board, workspace_dir, serial_number, model_name, project_type):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

project = tvm.micro.project.generate_project_from_mlf(
template_project_path, workspace_dir / "project", model_tar_path, project_options
)
project.build()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, we are testing accuracy here. I have added that to the doc string

Copy link
Contributor

@alanmacd alanmacd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@mkatanbaf mkatanbaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mehrdadh, I left a few comments.

@mehrdadh mehrdadh force-pushed the micro/add_mlperftiny_models branch from 9012819 to 4ec52ba Compare February 15, 2023 00:35
@mehrdadh mehrdadh requested a review from mkatanbaf February 15, 2023 21:26
@masahi masahi merged commit a3b51f1 into apache:main Feb 22, 2023
@mehrdadh mehrdadh deleted the micro/add_mlperftiny_models branch February 22, 2023 23:54
yongwww pushed a commit to yongwww/tvm that referenced this pull request Feb 27, 2023
* Add test for mlperftiny models for host_driven and EEMBC runner versions

* refactor mlperftiny models in different links

* fix link

* address comments and fix error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants