Properly skip infrequent polling test on time-skipping server #655

Job	Run time
build-lint-test (3.9, ubuntu-latest)	5m 20s
build-lint-test (3.9, macos-intel)	8m 34s
build-lint-test (3.9, macos-arm)	7m 17s
build-lint-test (3.9, windows-latest)	5m 49s
build-lint-test (3.12, ubuntu-latest)	5m 23s
build-lint-test (3.12, macos-intel)	5m 57s
build-lint-test (3.12, macos-arm)	5m 23s
build-lint-test (3.12, windows-latest)	6m 22s
	50m 5s

Provide feedback