From 7622e447b9063ab29c8f51296a5a9bfc5b72b091 Mon Sep 17 00:00:00 2001 From: nuest Date: Thu, 19 Oct 2017 15:59:34 +0200 Subject: [PATCH 1/5] add direct file substitution draft; editorial changes to substitution docs --- docs/compendium/substitute.md | 110 +++++++++++++++++++++++----------- 1 file changed, 75 insertions(+), 35 deletions(-) diff --git a/docs/compendium/substitute.md b/docs/compendium/substitute.md index b4ffc20..8370b53 100644 --- a/docs/compendium/substitute.md +++ b/docs/compendium/substitute.md @@ -1,18 +1,27 @@ # Substitute two compendia Substitution is the combination of an base ERC and an overlay ERC. -A user can choose files from the overlay ERC that will replace files of the base ERC or will be uniquely added. +A user can choose files from the overlay ERC that will replace files of the base ERC, or add new files to the substitution. ## Create substitution -`Create substitution` will produce a new ERC with metadata, saved to MongoDB. -A substitution will be created with an HTTP `POST` request using `multipart/form-data` and content-type `JSON`. Required are the IDs of the base and overlay ERC and at least one pair of substitution files, consisting of a base and an overlayfile. +Creating a substitution produces a new ERC with its own metadata and files. +A substitution is created with an HTTP `POST` request using `multipart/form-data` and content-type `JSON`. +Required fields are the identifiers of base ERC and overlay ERC and at least one pair of substitution files, consisting of one base file and one overlay file. + +### Created metadata and files + +The substitution creation includes an update of the compendium metadata in the file **`erc.yml`**, which implements the actual substitution. +After creation, subsequent [jobs](http://o2r.info/o2r-web-api/job/) consider these settings. + +The **files** of a substituted ERC comprise all base ERC files and the overlay files. +In case of a file naming conflict, the overlay file is preprended with `overlay_`. ### Request `POST /api/v1/substitution` -input of request-body for substitution +**Example** request body: ```json { @@ -27,13 +36,46 @@ input of request-body for substitution } ``` +**Example** request body with embedded file upload: + +```json +{ + "base": "G92NL", + "overlay": "9fCTR", + "substitutionFiles": [ + { + "base": "climate-timeseries.csv", + "overlay": { + "filetype": "text/plain", + "filename": "substitution.txt", + "base64": "T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK" + } + } + ] +} +``` + +The above example was created as follows: + +```bash +$ echo "Opening Reproducible Research" >> substitution.txt +$ base64 -w 0 substitution.txt +T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK +$ echo T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK | base64 -d +Opening Reproducible Research +$ mimetype substitution.txt +substitution.txt: text/plain +``` + ### Request body properties - `base` - id of the base ERC - `overlay` - id of the overlay ERC -- `substitutionFiles` - array of file substitutions specified by `base` and `overlay` - - `base` - filename of the file from the base ERC - - `overlay` - filename of the overlay ERC that will be exchanged for the original file +- `substitutionFiles` - array of file substitutions specified each by one `base` and one `overlay` property + - `base` - filename of a file from the base ERC + - `overlay` - either a reference to a file in the overlay ERC, or embedded file upload + - _overlay file_: filename of a file in the overlay ERC, which is exchanged for the file from the base ERC + - _file upload_: JSON object with metadata and content of the file in the properties `filetype`, `filename` and `base64`, the latter being a [Base64](https://en.wikipedia.org/wiki/Base64) encoding of the file. !!! note "Required user level" @@ -75,26 +117,17 @@ input of request-body for substitution {"error":"overlay ERC not found"} ``` -## Run substitution - -`Run substitution` will run the analysis of a substitution in a docker container. -This is executed by a [job](http://o2r.info/o2r-web-api/job/). - -## View substituted Compendium +## View substituted compendium ### Request -`curl https://.../api/v1/compendium/$ID` - -`GET /api/v1/compendium/:id` - -This request will be handled as a GET-request of an usual compendium. ( [Click for more information.](http://o2r.info/o2r-web-api/compendium/view/#view-single-compendium) ) +This request is the same as for a regular compendium, see [view compendium](view.md#view-single-compendium). ### Response -A substituted ERC will be saved as a usual ERC, but with additional metadata specifying this as a substituted ERC and giving information about the substitution. +A substituted ERC response contains additional metadata (a) marking it as a substituted ERC with the property `metadata.substituted`, and (b) giving information about the substitution in the property `substitution`. -Example 01 - in case there are no conflicts between filenames of any basefile and overlayfile : +**Example** (no naming conflicts): ```json 200 OK @@ -113,6 +146,11 @@ Example 01 - in case there are no conflicts between filenames of any basefile an "base": "climate-timeseries.csv", "overlay": "mytimeseries_data.csv", "filename": "climate-timeseries.csv" + }, + { + "base": "analysis-settings.xml", + "overlay": "better-settings.txt", + "filename": "analysis-settings.xml" } ] }, @@ -122,7 +160,7 @@ Example 01 - in case there are no conflicts between filenames of any basefile an } ``` -Example 02 - in case the overlayfile has the same filename as one of the existing basefiles : +**Example** in case the overlay file has the same filename as one of the existing base files : ```json 200 OK @@ -159,9 +197,9 @@ Example 02 - in case the overlayfile has the same filename as one of the existin - `substitutionFiles` - array of file substitutions specified by `base` and `overlay` - `base` - filename of the file from the base ERC - `overlay` - filename of the file from the overlay ERC - - `filename` - as seen in the examples above, `filename` will be created if there is a conflict with any basefilename and an overlayfilename. In this case the overlayfilename will get an additional "**overlay_**" prepended (see Example 02). *(optional add)* + - `filename` - if there is a conflict with any base filename and an overlay filename, the overlay file is actually stored with `overlay_` prepended to the file name, which is stored in this property. -## List substituted Compendia +## List substituted compendia ### Request @@ -171,7 +209,7 @@ Example 02 - in case the overlayfile has the same filename as one of the existin ### Response -Result will be a list of compendia ids that have been substituted +Result will be a list of compendia ids which are created by a substitution process. ```json 200 OK @@ -186,13 +224,15 @@ Result will be a list of compendia ids that have been substituted } ``` -### Filter results with following parameters: +### Filter + +Results can be filtered with the parameters `base` and `overlay`. `curl https://.../api/v1/substitution?base=$BASE_ID&overlay=$OVERLAY_ID` `GET /api/v1/substitution?base=base_id&overlay=overlay_id` -- Filter by `base`: +**Example** for filtering by `base`: `curl https://.../api/v1/substitution?base=jfL3w` @@ -211,13 +251,13 @@ Result will be a list of compendia ids that have been substituted out of a choos } ``` -- Filter by `overlay`: +**Example** for filtering by `overlay`: `curl https://.../api/v1/substitution?overlay=as4Kj` `GET /api/v1/substitution?overlay=as4Kj` -Result will be a list of compendia ids that have been substituted out of a choosen overlay ERC +Result will be a list of compendia ids that have been substituted out of a chosen overlay ERC. ```json 200 OK @@ -231,13 +271,13 @@ Result will be a list of compendia ids that have been substituted out of a choos } ``` -- Filter by `base` and `overlay`: +**Example** for filtering with both `base` and `overlay`: `curl https://.../api/v1/substitution?base=lO3Td&overlay=as4Kj` `GET /api/v1/substitution?base=lO3Td&overlay=as4Kj` -Result will be a list of compendia ids that have been substituted out of a choosen base and overlay ERC +The result is a list of compendia ids that have been substituted out of a chosen base and overlay ERC. ```json 200 OK @@ -250,6 +290,11 @@ Result will be a list of compendia ids that have been substituted out of a choos } ``` +### URL parameters + +- `:base` - id of the base ERC of substitutions in the result +- `:overlay` - id of the base ERC of substitutions in the result + ### Error responses ```json @@ -275,8 +320,3 @@ Result will be a list of compendia ids that have been substituted out of a choos {"error":"overlay ERC not found"} ``` - -### URL parameters for substituted compendium lists - -- `:base` - id of the base ERC that the results should be related to -- `:overlay` - id of the base ERC that the results should be related to From 370ee72f984bee075dd82c40864a7c81d4a382dd Mon Sep 17 00:00:00 2001 From: nuest Date: Fri, 20 Oct 2017 15:38:35 +0200 Subject: [PATCH 2/5] use the same file type for direct substitution example --- docs/compendium/substitute.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/compendium/substitute.md b/docs/compendium/substitute.md index 8370b53..7f9615e 100644 --- a/docs/compendium/substitute.md +++ b/docs/compendium/substitute.md @@ -46,9 +46,9 @@ In case of a file naming conflict, the overlay file is preprended with `overlay_ { "base": "climate-timeseries.csv", "overlay": { - "filetype": "text/plain", - "filename": "substitution.txt", - "base64": "T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK" + "filetype": "text/csv", + "filename": "my-data.csv", + "base64": "aWQsZGF0ZSx0ZW1wZXJhdHVyZVxuMSwyMDE3LTAxLTAxLDE3XG4yLDIwMTctMDEtMDIsNDIK" } } ] @@ -58,13 +58,13 @@ In case of a file naming conflict, the overlay file is preprended with `overlay_ The above example was created as follows: ```bash -$ echo "Opening Reproducible Research" >> substitution.txt -$ base64 -w 0 substitution.txt -T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK -$ echo T3BlbmluZyBSZXByb2R1Y2libGUgUmVzZWFyY2gK | base64 -d -Opening Reproducible Research -$ mimetype substitution.txt -substitution.txt: text/plain +$ echo "id,date,temperature\n1,2017-01-01,17\n2,2017-01-02,42" >> my-data.csv +$ base64 -w 0 my-data.csv +aWQsZGF0ZSx0ZW1wZXJhdHVyZVxuMSwyMDE3LTAxLTAxLDE3XG4yLDIwMTctMDEtMDIsNDIK +$ echo aWQsZGF0ZSx0ZW1wZXJhdHVyZVxuMSwyMDE3LTAxLTAxLDE3XG4yLDIwMTctMDEtMDIsNDIK | base64 -d +id,date,temperature\n1,2017-01-01,17\n2,2017-01-02,42 +$ mimetype my-data.csv +my-data.csv: text/csv ``` ### Request body properties From c4f870b3bc2abc3b2866856afade77467a03c09c Mon Sep 17 00:00:00 2001 From: nuest Date: Sun, 22 Oct 2017 16:01:22 +0200 Subject: [PATCH 3/5] use o2r blue for small screens --- docs/o2r.css | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/o2r.css b/docs/o2r.css index 3cbc51b..6380792 100644 --- a/docs/o2r.css +++ b/docs/o2r.css @@ -6,10 +6,23 @@ background-color: #434553; } +.wy-nav-top { + background-color: #004286; +} + +.wy-menu-vertical a:active{ + background-color:#004286; +} + .wy-menu a { color: #ffffff; } .wy-menu-vertical a:hover { background:#004286; +} + +h3, h4, h5, h6 { + color: #004286; + font-weight: 300 } \ No newline at end of file From e054844f29d182410375304d00f886b951bdb815 Mon Sep 17 00:00:00 2001 From: nuest Date: Mon, 23 Oct 2017 12:28:00 +0200 Subject: [PATCH 4/5] remove lonely header --- docs/compendium/metadata.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/compendium/metadata.md b/docs/compendium/metadata.md index be8cd64..e773e44 100644 --- a/docs/compendium/metadata.md +++ b/docs/compendium/metadata.md @@ -203,8 +203,6 @@ The `spatial` key has a `union` bounding box, that wraps all extracted bounding } ``` -### GET - ## Update metadata The following endpoint can be used to update the `o2r` metadata elements. From 8681e2fc5bb86fe6acffc1bfcde3ab5b02ae4e7b Mon Sep 17 00:00:00 2001 From: nuest Date: Fri, 1 Dec 2017 10:04:03 +0100 Subject: [PATCH 5/5] Use "compendium" (manual fix after merge) --- docs/compendium/substitute.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/compendium/substitute.md b/docs/compendium/substitute.md index 3702476..9ed3d53 100644 --- a/docs/compendium/substitute.md +++ b/docs/compendium/substitute.md @@ -1,25 +1,26 @@ # Substitution -Substitution is the combination of an base ERC and an overlay ERC. -A user can choose files from the overlay ERC that will replace files of the base ERC, or add new files to the substitution. +Substitution is the combination of an base compendium and an overlay compendium. +A user can choose files from the overlay compendium that will replace files of the base compendium, or add new files to the substitution. ## Create substitution -Creating a substitution produces a new ERC with its own metadata and files. +Creating a substitution produces a new compendium with its own metadata and files. A substitution is created with an HTTP `POST` request using `multipart/form-data` and content-type `JSON`. -Required fields are the identifiers of base ERC and overlay ERC and at least one pair of substitution files, consisting of one base file and one overlay file. +Required fields are the identifiers of base compendium and overlay compendium and at least one pair of _substitution files_, consisting of one base file and one overlay file. !!! Note A substitution process removes potentially existing packaging information, i.e. if the base compendium was a BagIt bag, the substitution will only contain the payload directory contents (`/data` directory). The overlay file is stripped of all paths and is copied directly into the substitution's root directory. + The original path is preserved in the substitution metadata. ### Created metadata and files The substitution creation includes an update of the compendium metadata in the file **`erc.yml`**, which implements the actual substitution. After creation, subsequent [jobs](http://o2r.info/o2r-web-api/job/) consider these settings. -The **files** of a substituted ERC comprise all base ERC files and the overlay files. +The **files** of a substituted compendium comprise all base compendium files and the overlay files. In case of a file naming conflict, the overlay file is preprended with `overlay_`. ### Request @@ -132,7 +133,7 @@ This request is the same as for a regular compendium, see [view compendium](view A substituted compendium is be saved as a usual compendium, but with additional metadata specifying this as a substituted compendium and giving information about the substitution. -Example 01 - in case there are no conflicts between filenames of any base file and overlay file : +**Example** without naming conflicts: ```json 200 OK @@ -165,7 +166,7 @@ Example 01 - in case there are no conflicts between filenames of any base file a } ``` -Example 02 - in case the overlay file has the same filename as one of the existing base files and is in a sub-directory in the overlay compendium: +**Example** with naming conflicts and overlay from sub-directory: ```json 200 OK @@ -193,6 +194,8 @@ Example 02 - in case the overlay file has the same filename as one of the existi } ``` +Note that the base file must not be the file causing the naming conflict. + ### Response additional metadata - `substituted` - is `true`