Resolve the discrepancy of latency report between LLMs and non-LLMs

[guangy10]

🐛 Describe the bug

As shown on the dashboard, the avg_inference_latency (ms) is skipped for LLM, and report only generate_time (ms) instead.

Upon checking the iOS run for example, a LLM job will run three tests on-device to report different metrics:

1. test_load_llama_3_2_1b_llama3_fb16_pte_iOS_17_2_1_iPhone15_4
2. test_forward_llama_3_2_1b_llama3_fb16_pte_iOS_17_2_1_iPhone15_4
3. test_generate_llama_3_2_1b_llama3_fb16_pte_tokenizer_model_iOS_17_2_1_iPhone15_4
   While a non-LLM job will only run the first two tests (test_load_ and test_forward_ ) instead.

See detailed jobs here:

• LLM: https://github.com/pytorch/executorch/actions/runs/13403521306/job/37441009799
• non-LLM: https://github.com/pytorch/executorch/actions/runs/13403521306/job/37441008720

Three things to get clarification in this task:

1. Because test_forward_* is reported to both LLM and non-LLM, why isn't reported to the dash?
2. Let's annotate each metrics in the DB so users will know what exactly is measured by each.
   3. Confirm if Android is measuring and reporting exact same metrics Report avg_inference_latency from Android LLM benchmark app #8578

Versions

trunk

cc @huydhn @kirklandsign @shoumikhin @mergennachin @byjlw

[guangy10]

Android benchmark app doesn’t save inference latency for LLM https://github.com/pytorch/executorch/blob/main/extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/LlmBenchmarkActivity.java#L95-L112 while non-LLM does

@kirklandsign Can we add raw inference latency for LLM? It will be useful in the case to detect whether the slowness is from core runtime or the tokenizer itself.

[guangy10]

For iOS app, upon checking test_forward_ is measuring the raw latency around forward(), for both LLM and non-LLM:

executorch/extension/benchmark/apple/Benchmark/Tests/GenericTests.mm

Lines 48 to 94 in 00c1443

	return @{
	@"load" : ^(XCTestCase *testCase){
	[testCase
	measureWithMetrics:@[ [XCTClockMetric new], [XCTMemoryMetric new] ]
	block:^{
	XCTAssertEqual(
	Module(modelPath.UTF8String).load_forward(),
	Error::Ok);
	}];
	},
	@"forward" : ^(XCTestCase *testCase) {
	auto __block module = std::make_unique<Module>(modelPath.UTF8String);
	const auto method_meta = module->method_meta("forward");
	ASSERT_OK_OR_RETURN(method_meta);
	const auto num_inputs = method_meta->num_inputs();
	XCTAssertGreaterThan(num_inputs, 0);
	std::vector<TensorPtr> tensors;
	tensors.reserve(num_inputs);
	for (auto index = 0; index < num_inputs; ++index) {
	const auto input_tag = method_meta->input_tag(index);
	ASSERT_OK_OR_RETURN(input_tag);
	switch (*input_tag) {
	case Tag::Tensor: {
	const auto tensor_meta = method_meta->input_tensor_meta(index);
	ASSERT_OK_OR_RETURN(tensor_meta);
	const auto sizes = tensor_meta->sizes();
	tensors.emplace_back(
	rand({sizes.begin(), sizes.end()}, tensor_meta->scalar_type()));
	XCTAssertEqual(module->set_input(tensors.back(), index), Error::Ok);
	} break;
	default:
	XCTFail("Unsupported tag %i at input %d", *input_tag, index);
	}
	}
	XCTMeasureOptions *options = [[XCTMeasureOptions alloc] init];
	options.iterationCount = 20;
	[testCase measureWithMetrics:@[ [XCTClockMetric new], [XCTMemoryMetric new] ]
	options:options
	block:^{
	XCTAssertEqual(module->forward().error(), Error::Ok);
	}];

. This metric is mapped to the avg_inference_latency on the dash.

test_generate_ is LLM specific, measuring the latency around generate() (forward + tokenization):

executorch/extension/benchmark/apple/Benchmark/Tests/LLaMA/LLaMATests.mm

Lines 76 to 97 in 00c1443

	@"generate" : ^(XCTestCase *testCase){
	auto __block runner = std::make_unique<example::Runner>(
	modelPath.UTF8String, tokenizerPath.UTF8String);
	const auto status = runner->load();
	if (status != Error::Ok) {
	XCTFail("Load failed with error %i", status);
	return;
	}
	TokensPerSecondMetric *tokensPerSecondMetric = [TokensPerSecondMetric new];
	[testCase measureWithMetrics:@[ tokensPerSecondMetric, [XCTMemoryMetric new] ]
	block:^{
	tokensPerSecondMetric.tokenCount = 0;
	const auto status = runner->generate(
	"Once upon a time",
	50,
	[=](const std::string &token) {
	tokensPerSecondMetric.tokenCount++;
	},
	nullptr,
	false);
	XCTAssertEqual(status, Error::Ok);
	}];

. This metric is mapped to the generate_time on the dash.

What each test is measuring is pretty clear, @huydhn I guess the remaining bit is why isn't the avg_inference_latency reported to the dash?