Download / Parse / Load bioRxiv and medRxiv articles

[jankrepl]

🚀 Feature

• Check licencing
• Investigate whether it is possible to bulk download papers from Arxiv
• Implement a parser for these papers

Motivation

More articles

Pitch

More articles is good

Alternatives

Maybe investigate similar sources like bioRxiv etc.

[pafonta]

The following applies for both bioRxiv and medRxiv.

Licence

CHECKED

[...] bulk TDM, which authors explicitly consent to during submission to bioRxiv [...]. The TDM repository is not intended as a source for further redistribution of articles posted on bioRxiv, or their derivatives, nor does it grant others permission to re-host content posted on bioRxiv. For most articles submitted to bioRxiv, authors retain copyright and reuse rights. If you build indexing services or tools based on the full text of articles, you must therefore link back to the text hosted at bioRxiv rather than re-host content. For reuse/redistribution of individual articles or their derivatives, please consult the licensing terms applied by the authors, which are provided in the metadata. In most cases, this will require you to contact the copyright holder in advance to obtain permission. (source) (same statements for medRxiv here)

Download

DONE

Full-text access via Amazon S3 is available via a requester pays bucket. [...] The bioRxiv AWS bucket is located in the US East (N. Virginia) [...]. (source) (same statements for medRxiv here)

➡️ Bulk download is charged:
Using Requester Pays buckets for storage transfers and usage
AWS S3 pricing

Here is how to bulk download articles:

1. Create an AWS account.

2. Get an IAM role which will receive the bills.

3. Install AWS CLI v2:
   Installing, updating, and uninstalling the AWS CLI version 2

4. Configure the CLI and assume the given IAM role:
   TL;DR Switching to an IAM role (AWS CLI) (See Example scenario: Switch to a production role)
   Configuration basics
   Configuration and credential file settings
   Using an IAM role in the AWS CLI

5. Check everything works fine and summary statistics with:
   aws s3 ls s3://biorxiv-src-monthly --request-payer requester --summarize --human-readable
   (for medRxiv, replace biorxiv with medrxiv)
   List S3 objects and common prefixes under a prefix or all S3 buckets

6. Bulk download articles:
   aws s3 cp s3://biorxiv-src-monthly biorxiv/ --request-payer requester --recursive
   (for medRxiv, replace biorxiv with medrxiv)
   Copy a local file or S3 object to another location locally or in S3

Parsing

DONE. See #493.

The bioRxiv and medRxiv papers are in the JATS v1.2d1 20170631 format.

details

Oldest and latest articles were checked:

bioRxiv

Back_Content/Batch_01/0064b7c2-6c08-1014-9382-821844e0e0e5.meca
Back_Content/Batch_50/ffc4f3bf-6bf9-1014-98b9-a7cd140a0219.meca
Current_Content/April_2019/Batch_#119/1066f54b-6c05-1014-a9e8-b81319115117.meca
Current_Content/October_2021/fff4d64a-6e98-1014-9d81-a5fa3102a66e.meca

medRxiv

Back_Content/medRxiv_Batch_01/001aeac1-6c08-1014-87c2-a31129fa52cb.meca
Back_Content/medRxiv_Batch_15/ffdb9418-6c55-1014-960b-cb862e49d6fe.meca
Current_Content/April_2021/005a462d-6c79-1014-8f66-9f56e093282b.meca
Current_Content/October_2021/ffaa6b98-6cff-1014-95c9-b6cfabaaddaf.meca

Our PMCXMLParser handles JATS v1.2 20190208. This is a more recent JATS specification.

details

Our PMCXMLParser:

Search/src/bluesearch/database/article.py

Lines 128 to 402 in 3d5a64a

	class PMCXMLParser(ArticleParser):
	"""Parser for PubMed Central XML files using the JATS Journal Publishing DTD.
	Parameters
	----------
	path
	The path to the XML file from PubMed.
	"""
	def __init__(self, path: str | Path) -> None:
	super().__init__()
	self.content = ElementTree.parse(str(path))
	self.ids = self.get_ids()
	@property
	def title(self) -> str:
	"""Get the article title.
	Returns
	-------
	str
	The article title.
	"""
	titles = self.content.find(".//title-group/article-title")
	return self._element_to_str(titles)
	@property
	def authors(self) -> Generator[str, None, None]:
	"""Get all author names.
	Yields
	------
	str
	Every author, in the format "Given_Name(s) Surname".
	"""
	authors = self.content.findall(
	".//contrib-group/contrib[@contrib-type='author']"
	)
	for author in authors:
	given_names = self._element_to_str(author.find("name/given-names"))
	surname = self._element_to_str(author.find("name/surname"))
	if given_names == "" or surname == "":
	# In rare cases, an author may not have a given name or a surname,
	# e.g. it could be an organization. We decide to skip those.
	continue
	author_str = given_names + " " + surname
	yield author_str.strip()
	@property
	def abstract(self) -> Generator[str, None, None]:
	"""Get a sequence of paragraphs in the article abstract.
	Yields
	------
	str
	The paragraphs of the article abstract.
	"""
	abstract_pars = self.content.findall(".//abstract//p")
	for paragraph in abstract_pars:
	yield self._element_to_str(paragraph)
	@property
	def paragraphs(self) -> Generator[tuple[str, str], None, None]:
	"""Get all paragraphs and titles of sections they are part of.
	Paragraphs can be parts of text body, or figure or table captions.
	Yields
	------
	section : str
	The section title.
	text : str
	The paragraph content.
	"""
	# Abstract are removed because already parsed in abstract property.
	abstract_p = self.content.findall(".//abstract/p")
	# Caption are removed because already parsed later in the current method
	# with the section name "Caption".
	caption_p = self.content.findall(".//caption/p")
	# Acknowledgements are removed because not useful.
	acknowledg_p = self.content.findall(".//ack/p")
	exclude_list = abstract_p + caption_p + acknowledg_p
	# Paragraphs of text body
	section_dirs = self.get_paragraphs_sections_mapping()
	for paragraph in self.content.findall(".//p"):
	if paragraph not in exclude_list:
	text = self._element_to_str(paragraph)
	section_title = ""
	if paragraph in section_dirs:
	section_title = section_dirs[paragraph]
	yield section_title, text
	# Figure captions
	figs = self.content.findall(".//fig")
	for fig in figs:
	fig_captions = fig.findall("caption")
	if fig_captions is None:
	continue
	caption = " ".join(self._element_to_str(c) for c in list(fig_captions))
	yield "Figure Caption", caption
	# Table captions
	tables = self.content.findall(".//table-wrap")
	for table in tables:
	caption_elements = table.findall(".//caption/p") or table.findall(
	".//caption/title"
	)
	if caption_elements is None:
	continue
	caption = " ".join(self._element_to_str(c) for c in caption_elements)
	yield "Table Caption", caption
	@property
	def pubmed_id(self) -> Optional[str]:
	"""Get Pubmed ID.
	Returns
	-------
	str or None
	Pubmed ID if specified, otherwise None.
	"""
	return self.ids.get("pmid")
	@property
	def pmc_id(self) -> Optional[str]:
	"""Get PMC ID.
	Returns
	-------
	str or None
	PMC ID if specified, otherwise None.
	"""
	return self.ids.get("pmc")
	@property
	def doi(self) -> Optional[str]:
	"""Get DOI.
	Returns
	-------
	str or None
	DOI if specified, otherwise None.
	"""
	return self.ids.get("doi")
	def get_ids(self) -> dict[str, str]:
	"""Get all specified IDs of the paper.
	Returns
	-------
	ids : dict
	Dictionary whose keys are ids type and value are ids values.
	"""
	ids = {}
	article_ids = self.content.findall("./front/article-meta/article-id")
	for article_id in article_ids:
	if "pub-id-type" not in article_id.attrib.keys():
	continue
	ids[article_id.attrib["pub-id-type"]] = article_id.text
	return ids
	def get_paragraphs_sections_mapping(self) -> dict[Element, str]:
	"""Construct mapping between all paragraphs and their section name.
	Returns
	-------
	mapping : dict
	Dictionary whose keys are paragraphs and value are section title.
	"""
	mapping = {}
	for sec in self.content.findall(".//sec"):
	section_title = ""
	section_paragraphs = []
	for element in sec:
	if element.tag == "title":
	section_title = self._element_to_str(element)
	elif element.tag == "p":
	section_paragraphs.append(element)
	for paragraph in section_paragraphs:
	mapping[paragraph] = section_title
	return mapping
	def _inner_text(self, element: Element) -> str:
	"""Convert all inner text and sub-elements to one string.
	In short, we collect all the inner text while also converting all
	sub-elements that we encounter to strings using ``self._element_to_str``.
	All escaped HTML in the raw text is unescaped.
	For example, if schematically the element is given by
	element = "<p>I <bold>like</bold> python &amp; ice cream.<p>"
	then ``_inner_text(element)`` would give
	"I like python & ice cream."
	provided that "<bold>like</bold>" is resolved to "like" by the
	``self._element_to_str`` method.
	Parameters
	----------
	element
	The input XML element.
	Returns
	-------
	str
	The inner text and sub-elements converted to one single string.
	"""
	text_parts = [html.unescape(element.text or "")]
	for sub_element in element:
	# recursively parse the sub-element
	text_parts.append(self._element_to_str(sub_element))
	# don't forget the text after the sub-element
	text_parts.append(html.unescape(sub_element.tail or ""))
	return "".join(text_parts).strip()
	def _element_to_str(self, element: Element | None) -> str:
	"""Convert an element and all its contents to a string.
	Parameters
	----------
	element
	The input XML element.
	Returns
	-------
	str
	A parsed string representation of the input XML element.
	"""
	if element is None:
	return ""
	if element.tag in {
	"bold",
	"italic",
	"monospace",
	"p",
	"sc",
	"styled-content",
	"underline",
	"xref",
	}:
	# Mostly styling tags for which getting the inner text is enough.
	# Currently this is the same as the default handling. Writing it out
	# explicitly here to decouple from the default handling, which may
	# change in the future.
	return self._inner_text(element)
	elif element.tag == "sub":
	return f"_{self._inner_text(element)}"
	elif element.tag == "sup":
	return f"^{self._inner_text(element)}"
	elif element.tag == "inline-formula":
	return "FORMULA"
	elif element.tag == "disp-formula":
	return "\nFORMULA-BLOCK"
	elif element.tag in {"ext-link", "uri"}:
	return "URL"
	elif element.tag == "email":
	return "EMAIL"
	else:
	# Default handling for all other element tags
	return self._inner_text(element)

tested on:

Search/tests/data/sample_file.xml

Line 2 in 3d5a64a

<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "JATS-journalpublishing1.dtd">

➡️ It seems we can use PMCXMLParser to parse bioRxiv and medRxiv papers. Indeed, searching manually into the specification for JATS v1.2d1, it seems we don't use a modification made between JATS v1.2d1 and JATS v1.2.

⚠️ However, we use the pub-id-type pmc which is not in the specification of pub-id-type for JATS v1.2d1 nor for the latest JATS v1.3. Nevertheless, the pub-id-type pmc is used by PMC, as one can see in our paper XML. Furthermore, none of the parsed and loaded PMC articles have a pmc_id which is NULL:

SQL query

SELECT count(article_id) FROM articles WHERE pmc_id IS NULL;

✅ This shouldn't be an issue, though. Indeed, paper coming directly from bioRxiv or medRxiv shouldn't already have a PMC identifier. Then, looking for this not standard pub-id-type pmc at parsing shouldn't be an issue.

Loading

DONE. See #498.

Each article has its own XML file.

However, it is cumbersome to use bbs_database parse with the directory structure of bioRxiv and medRxiv. Indeed, the XML files to parse are under several subdirectories (4 to 5). But currently, bbs_database parse does not handle subdirectories nor non XML files.

details

Example of directory structure for bioRxiv:

Current_Content/
  April_2019/
    Batch_#119/
      1066f54b-6c05-1014-a9e8-b81319115117/
        content/
          286872.xml

Logic of bbs_database parse when input_pathis a directory:

Search/src/bluesearch/entrypoint/database/parse.py

Lines 104 to 105 in 3d5a64a

	elif input_path.is_dir():
	inputs = sorted(input_path.glob("*"))

[pafonta]

Regarding arXiv (arXiv/q-bio):

There is no XML version of the full text of these articles. The source files are in TeX/LaTeX or in PDF. (source)

➡️ We might want to report the download / parsing / loading of the papers from arXiv/q-bio.

Indeed, processing these papers requires new components and sets new challenges.

[FrancescoCasalegno]

Regarding arXiv (arXiv/q-bio):

There is no XML version of the full text of these articles. The source files are in TeX/LaTeX or in PDF. (source)

Just for the sake of completeness, this seems to be true in most cases. But in principle, there may also be some articles on arXiv that have other source files in other formats than TeX, including HTML (source 1) (source 2).

That being said, so far I have only seen arXiv articles with PDF available, some also with TeX, but none with HTML.

[pafonta]

Proposed action for loading

Have bbs_database parse:

1. recursively look for files to parse in the input_path,
2. look only for the file format (e.g. XML, JSON) associated with the input_type.

(details: section 'loading' in #459 (comment))

Please confirm you have read this proposal and made your comments below if needed:

• @FrancescoCasalegno
• @jankrepl
• @EmilieDel
• @Stannislav

[FrancescoCasalegno]

Hi @pafonta,

Yes I think your proposal of recursive parsing makes perfect sense. As for

file format (e.g. XML, JSON) associated with the input_type

I think we should just be a bit careful to cover all possible cases. E.g. for JATS XML, are we sure that .xml is the only possible extension? Didn't we see some .nxml or something like that in the PMC-OA papers?
(Maybe @EmilieDel knows this better.)

---

Maybe we could have an optional parameter (whose the defaults could be the extensions we expect to be associated to the input_type) --extension for bbs_database parse so that we could pass something like --extension=.xml,.nxml if we know the extension of the files at runtime.
This is just an idea though, feel free to move forward with the implementation :)

[FrancescoCasalegno]

Summary of Discussion

Option 1 – bbs_dataset parse called once per file.
Drawbacks: It would imply significant overhead, see #443 (comment)

Option 2 – pipe results of a find into bbs_dataset parse.
Drawbacks: It would mean passing a string containing 1M files as an argument (it may break). Moreover this would mean we have to change the logic of the functionalities of bbs_database.

Option 3 – Support an argument --match-filename (or better name) to identify which files inside a given directory will be matched.
Drawbacks: It may not be clear what comes out of the regex. (Optional): We can have a --dry-run option to print out which files would be matched.

Conclusion

Let's go with Option 3 ! 💪

[pafonta]

All elements have been done (see here). Closing.