[openapi] Adopt enum-driven visibility

[witemple-msft]

This is a tracking issue for the OpenAPI library to adopt enum-driven visibility analysis APIs and implement any enhancements to the core typespec libraries that are required to support OpenAPI's use case.

See #4825

[markcowl]

est: 8

[wanlwanl]

Wait for: #5416

[witemple-msft]

Hey @wanlwanl, I ended up taking this as part of the related issue for the HTTP core. There won't be anything left to do here after #5416 is merged, as OpenAPI doesn't do much with visibility outside of what the HTTP core does for it.