There was a gentoo request to build a pycairo variant without X support, even if cairo exposes it. The use case being that users are allowed to install cairo with or without X, and to only have one pycairo build that works with both.
This could be solved by forcing this at build time via something like -Ddisable-features=xcb_surface,xlib_surface for the various HAS macros: https://pycairo.readthedocs.io/en/latest/reference/constants.html#cairo-has, so we pretend as if cairo doesn't expose those features.
Another approach would be to detect those symbols at runtime, via dlopen()/dlsym() etc. I always thought that's not worth the effort, but maybe worth a try for the X things for starters.