feat: Add traffic streams

[sicarul]

This PR adds the Traffic streams to collect how much traffic did we have on each Repository.

It'll only collect data for the repositories the token has write access to, and will log an info message for those that it couldn't collect.

[sicarul]

It seems the test failed:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tap_github/tests/test_tap.py::test_get_a_repository_in_repo_list_mode[False] - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
FAILED tap_github/tests/test_tap.py::test_get_a_repository_in_repo_list_mode[True] - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
FAILED tap_github/tests/test_tap.py::test_last_state_message_is_valid - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
============= 3 failed, 7 passed, 714 warnings in 84.81s (0:01:24) =============
Error: Process completed with exit code 1.

I think it's because of a permissions issue on the user running the test.

[edgarrmondragon]

It seems the test failed:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tap_github/tests/test_tap.py::test_get_a_repository_in_repo_list_mode[False] - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
FAILED tap_github/tests/test_tap.py::test_get_a_repository_in_repo_list_mode[True] - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
FAILED tap_github/tests/test_tap.py::test_last_state_message_is_valid - singer_sdk.exceptions.FatalAPIError: 403 Client Error: b'{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/metrics/traffic#get-repository-clones"}' (Reason: Forbidden) for path: /repos/MeltanoLabs/tap-github/traffic/clones
============= 3 failed, 7 passed, 714 warnings in 84.81s (0:01:24) =============
Error: Process completed with exit code 1.

I think it's because of a permissions issue on the user running the test.

@sicarul Yeah, can you add note to the readme documenting which permissions are required to pull the traffic streams?

[sicarul]

Added @edgarrmondragon !

[sicarul]

@edgarrmondragon do we need anything else to move this forward?

[edgarrmondragon]

@edgarrmondragon do we need anything else to move this forward?

@sicarul Tests are still failing so I worry that this may cause similar failures to users of the tap if their tokens don't support pulling from the traffic endpoints.

Wdyt about adding a tap setting (e.g. sync_traffic_streams) that by default makes the tap omit these streams?

cc @kgpayne if you have other ideas

[sicarul]

@edgarrmondragon do we need anything else to move this forward?

@sicarul Tests are still failing so I worry that this may cause similar failures to users of the tap if their tokens don't support pulling from the traffic endpoints.

Wdyt about adding a tap setting (e.g. sync_traffic_streams) that by default makes the tap omit these streams?

cc @kgpayne if you have other ideas

Maybe i could also handle this error instead?
“ Resource not accessible by integration”

However, i do not believe any user would try to run this entire tap without selecting streams, some are monstruous.

[edgarrmondragon]

However, i do not believe any user would try to run this entire tap without selecting streams, some are monstruous.

@sicarul You're right, but the SDK tries to be helpful to folks just getting started with singer.io and so all streams are selected by default in the default catalog.

However, we could change that in your streams. I'll propose some changes.

(I also started a discussion in the SDK: meltano/sdk#1652)

[edgarrmondragon]

@sicarul this LGTM! Just a minor comment on the readme and this is good to merge for me 🙂

[edgarrmondragon]

@sicarul Can you update your branch from main?

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[edgarrmondragon]

Thanks @sicarul!