Latest pywb changes#4
Merged
omgoo merged 16 commits intomirrorweb:masterfrom Apr 28, 2021
Merged
Conversation
update to latest wombat (3.1.4)
* Pass collection name to ACL checker to load ACL lists for automatic collections * Typo: file suffix must be `.aclj`
- add unit test to verify unknown output formats are handled if output fields param is in request
* FrontendApp: forward HTTP status of CDX backend to allow clients to handle errors more easily * WarcServer: keep the HTTP status lines short - append the exception message only if the status isn't a string (WbException and inherited classes already have nice status string) - avoid overlong status lines, eg. HTTP/1.1 404 Not Found No Captures found for: https://very-long.url/...
* FrontendApp: forward HTTP status of CDX backend to allow clients to handle errors more easily * Handle CDXExceptions properly, returning the exception status code - make that CDXException is raised early so that it can be handled in the IndexHandler
…630) `page` or `pageSize` are no valid integers
The 'dedup_index_url' configuration option should be inside the 'recorder' section.
- do not apply any filters (param filter, from, to, closest) if counting pages (param showNumPages=true)
The field is unfortunately misnamed compressedendoffset in XML but OWB actually uses this for the compressed length 'S' CDX field. Without this field when WARC files are accessed over HTTP pywb will make open byte range requests which results in a lot more data being read from disk than necessary.
This advertises the Python support that is already in place.
* post append improvements: - parse json primitives for post query - for text/plain, attempt to parse as json, then as binary - standardize post append indexing - include '__wb_method' in urlkey - add 'requestBody' and 'method' to cdxj - support unique dupe params for json-to-query conversion * test fixes: - update tests for test_inputreq, - update post-test.cdxj and post-test.cdx * ci: fixes - tox: run full test suite! - disable appveyor * inputrequest buffering fix: - never truncate reading POST request, must read entire POST data to avoid hung request in live mode - truncate final query string to 4096
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.