Skip to content

Conversation

@savitojs
Copy link
Contributor

@savitojs savitojs commented Mar 28, 2020

Added support for headers to avoid HTTP error 403 due to robots.txt
One will continue to get 403 if requires web auth forbidden in real.

Following examples which don't work (403 error due to robots.txt) and my CI jobs to fail. Started to work after adding headers support.

Fixes #31

Signed-off-by: Savitoj Singh savsingh@redhat.com

Signed-off-by: Savitoj Singh <savsingh@redhat.com>
Copy link
Member

@digitronik digitronik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for PR 👍
LGTM... one optional question.

@digitronik digitronik added the bug Something isn't working label Mar 28, 2020
@digitronik digitronik self-assigned this Mar 28, 2020
Signed-off-by: Savitoj Singh <savsingh@redhat.com>
@digitronik digitronik merged commit ec9883e into pythonpune:master Mar 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error 403 with some sites where robots.txt don't allow

2 participants