Skip to content

Send along a User-Agent header with urlopen#126

Open
wtoorop wants to merge 1 commit intoicann:mainfrom
NLnetLabs:bugfix/urlopen-with-User-Agent
Open

Send along a User-Agent header with urlopen#126
wtoorop wants to merge 1 commit intoicann:mainfrom
NLnetLabs:bugfix/urlopen-with-User-Agent

Conversation

@wtoorop
Copy link
Copy Markdown

@wtoorop wtoorop commented Feb 15, 2024

Because www.rfc-editor.org returns 403: Forbidden otherwise:

Fetching errata from source of truth https://www.rfc-editor.org/errata.json... 

   ERROR: returned with error: HTTP Error 403: Forbidden.



   ERROR: got unexpected parsing response type <class 'NoneType'>.


Creating output for dns-rfcs.txt...
Scanning for 144 RFC documents in 'raw-originals/':
Downloading rfc1034... 

   ERROR: can't download text file for rfc1034: HTTP Error 403: Forbidden.

Downloading rfc1035... 

   ERROR: can't download text file for rfc1035: HTTP Error 403: Forbidden.

etc. At least with me.

According to this stackoverflow issue because the server denies bot access.

Because www.rfc-editor.org returns 403: Forbidden otherwise
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant