[TEST][FLAKY] test_auo_scheduler_layout_rewrite

[tqchen]

=================================== FAILURES ===================================
__________ test_correctness_layout_rewrite_rewrite_for_preTransformed __________

    @tvm.testing.requires_llvm
    def test_correctness_layout_rewrite_rewrite_for_preTransformed():
        N = 128
        target = tvm.target.Target("llvm")
        task = auto_scheduler.SearchTask(func=matmul_auto_scheduler_test, args=(N, N, N), target=target)
        dag = task.compute_dag
    
        with tempfile.NamedTemporaryFile() as fp:
            log_file = fp.name
    
            search_policy = auto_scheduler.SketchPolicy(task)
    
            measure_ctx = auto_scheduler.LocalRPCMeasureContext()
            tuning_options = auto_scheduler.TuningOptions(
                num_measure_trials=2,
                runner=measure_ctx.runner,
                verbose=2,
                measure_callbacks=[auto_scheduler.RecordToFile(log_file)],
            )
            task.tune(tuning_options, search_policy=search_policy)
            inp, _ = auto_scheduler.load_best_record(log_file, task.workload_key, target)
            s, bufs = dag.apply_steps_from_state(
>               inp.state, layout_rewrite=auto_scheduler.LayoutRewriteOption.REWRITE_FOR_PRE_TRANSFORMED
            )
E           AttributeError: 'NoneType' object has no attribute 'state'

tests/python/unittest/test_auto_scheduler_layout_rewrite.py:89: AttributeError
=============================== warnings summary ===============================

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/625/pipeline

[tqchen]

cc @comaniac @merrymercy

[comaniac]

This happens when no valid schedules can be applied (load_best_record returns None in this case). Maybe in the i386 machine it's easy to get timeout and results in this error. We could consider either of the following:

A. Only perform this test on x86.
B. Improve this test by mocking a valid record.

@merrymercy which one do you prefer or do you have other suggestions?

[masahi]

I remember we had this problem before @jcf94

[merrymercy]

We can use this tuning option to make sure we always get a valid schedule. It will try at most 100 schedules but stops immediately when we get a valid one.

tvm/tests/python/relay/test_auto_scheduler_tuning.py

Lines 41 to 44 in 51dc332

	tune_option = auto_scheduler.TuningOptions(
	num_measure_trials=100,
	num_measures_per_round=2,
	early_stopping=1,

[comaniac]

I'm afraid that this would make the CI time longer especially on slower machines such as i386.

[jcf94]

How about to set a smaller N?

[comaniac]

Combining the above ideas, I'll file a PR to

1. Reduce the workload size (i.e., N) from 128x128 to 16x16.
2. Enlarge trial number to 100 with early_stopping=1.