Chapter05

[19bsr16054]

• Collaborator: Karthikeyan V

What is semanticClimate?

semanticClimate is a project which aims to convert climate related documents by the IPCC to a semantic form, understood by both humans and computers.

Why are such conversions required?

The goal is to extract useful information from the IPCC documents so that relevant information could be made accessible for everyone to read and understand. This is achieved by creating dictionaries of different types for different purposes.

What is chapter 05 about?

• Title: Demand, services and social aspects of mitigation

Summary of the Chapter

• Summary of the Chapter

Environment and Origin

• Windows 10
• git 2.38.1.windows.1
• python version 3.8.15
• docanalysis version 0.2.0
• py4ami version 0.0.45
• IDE used - Anaconda v.3.0

Tools used: py4ami and docanalysis

How we set up the software:

1. Install Anaconda3 from the Official website
2. Open Anaconda Navigator thorough the Start menu
3. Go to the Environments tab and click on 'New'
4. Name the venv and choose python version 3.8.**
5. Run pip install docanalysis this will install both py4ami and docanalysis
6. Install git from the Official website or for a GUI experience use GitHub Desktop

How to create dictionaries?

We create 3 different dictionaries which are an abbreviation dictionary, a manual dictionary, and a keyword dictionary.

• Create a .html file from the specific chapter PDF by using the py4ami program. This can be achieved by following the steps below:

1. Open the created venv
2. In the terminal using cd navigate to the desired location
3. Run git clone https://github.com/petermr/semanticClimate this clones the repo to the desired location (Ideally a location that doesn't require Admin privileges) or clone using GitHub Desktop.
4. Using cd again navigate into semanticClimate and to your Chapter Directory in this case cd E:\git\semanticClimate\ipcc\ar6\wg3\Chapter05
5. Download the chapter from the IPCC website rename it to fulltext.pdf and place it inside the Chapter folder in this case E:\git\semanticClimate\ipcc\ar6\wg3\Chapter05\fulltext.pdf
6. From the Chapter directory run the following command
   python -m py4ami.ami_pdf --inpath fulltext.pdf --outdir E:\git\semanticClimate\ipcc\ar6\wg3\Chapter05 --maxpage 190
   • --maxpage is used to indicate the number of pages that need to be converted.
   • --outdir provides the path for the extra files generated (fulltext.flow_.html, fulltext_.html, TOC, Executive Summary, References and FAQs) while the final fulltext.html is output to the directory the command is run from regardless of this argument.

PS: The Log message provided by py4ami for py4ami --help is incorrect and has not been fixed yet, so do not use the example commands/arguments provided. The PDF argument doesn't have clear documentation so I have not been able to use as it's supposed to be.

• The abbreviation dictionary is created by docanalysis using the spacy method using the Google Colab Notebook

  • The docanalysis documentation could not be replicated in the venv without running into issues wrt to the C-Tree structure. Would prefer an argument to convert documents without using C-Tree structure.

  As an alternative the Google Colab Notebook created by one of our previous members which can run docanalysis without any issues and is well documented.

  1. Before switching to the Google Colab Notebook, update the repository of the changes you made using git push
  2. cd to the root of the repository E:\git\sematicClimate in this case and run
     git commit
git push -a
  3. The Google Colab will guide us with the further steps

• The keywords dictionary is created using the keyword extraction program which uses the gensim method.

  • Also part of the Google Colab Notebook

• The manual dictionary is manually created by the chapter champions from reading the chapter and picking out words or bi-grams that are less frequently used or are difficult to understand in the context of the report.

Chapter Annotation:

The complete chapter .html file is annotated with the abbreviations from the abbreviations dictionary to make it easier to read. It is done using the following command:
(https://colab.research.google.com/github/petermr/semanticClimate/blob/main/outreach/climate_knowledge_hunt_hackathon/Hackathon_Notebook/Chapter_Analysis_Notebook.ipynb)

py4ami HTML --annotate --dict {insert abbreviation dictionary path here} --inpath {insert the html path here} --outpath {insert output directory path here} --color YELLOW

where,

--dict – the path of the dictionary to be annotated with the dictionary name as well.

--inpath – the path of the full text html with the name of the html file as well.

--outpath – the path of the output, with the name of the output html file.

--color - the colour in which the annotated words should be highlighted in the annotated html.

• Or can be done using the [Google Colab Notebook]

[19bsr16054]

-- Added a new method for the kw extraction file, to just extract the text.txt file. (Raw text file which acts as a base for the existing kw functions)
-- Exploring kw extraction solutions and filtering with yake using text.txt