Graph plotter

requirements-evaluation.txt

@@ -2,3 +2,6 @@ openpyxl==3.0.7
 pandas==1.2.3
 pandarallel
 numpy~=1.20.2
+plotly
+kaleido
+pyyaml

src/python/evaluation/plots/README.md

@@ -0,0 +1,85 @@
+# Hyperstyle evaluation: plots
+This module allows you to visualize the data obtained with the [inspectors](../inspectors) module
+
+## [diffs_plotter.py](diffs_plotter.py)
+This script allows you to visualize a dataset obtained with [diffs_between_df.py](../inspectors/diffs_between_df.py). 
+
+The script can build the following charts: 
+* number of unique issues by category ([Example](#number-of-unique-issues-by-category))
+* number of issues by category ([Example](#number-of-issues-by-category))
+* number of unique penalty issues by category ([Example](#number-of-unique-penalty-issues-by-category))
+* number of penalty issues by category ([Example](#number-of-penalty-issues-by-category))
+* median penalty influence by category ([Example](#median-influence-on-penalty-by-category)) 
+* distribution of penalty influence by category ([Example](#distribution-of-influence-on-penalty-by-category))
+
+### Usage
+Run the [diffs_plotter.py](diffs_plotter.py) with the arguments from command line.
+
+**Required arguments**:
+1. `diffs_file_path` — path to a file with serialized diffs that were founded by [diffs_between_df.py](../inspectors/diffs_between_df.py).
+2. `save_dir` — directory where the plotted charts will be saved.
+3. `config_path` — path to the yaml file containing information about the charts to be plotted. A description of the config and its example is provided in [this section](#config).
+
+
+**Optional arguments**:
+
+Argument | Description
+--- | ---
+**‑‑file‑extension** | Allows you to select the extension of output files. Available extensions: `.png`, `.jpg`, `.jpeg`, `.webp`, `.svg`, `.pdf`, `.eps`, `.json`. Default is `.svg`.
+
+### Config
+The configuration file is a dictionary in yaml format, where each chart you want to build has its parameters.
+
+**Possible values of the charts**: 
+* `unique_issues_by_category` to plot the number of unique issues by category
+* `issues_by_category` to plot the number of issues by category
+* `unique_penalty_issues_by_category` to plot the number of unique penalty issues by category
+* `penalty_issues_by_category` to plot the number of penalty issues by category
+* `median_penalty_influence_by_category` to plot the median penalty influence by category
+* `penalty_influence_distribution` to plot the distribution of penalty influence by category
+
+**Possible parameters**:
+Parametr | Description
+---|---
+**x_axis_name** | Name of the x-axis. The default value depends on the type of chart.
+**y_axis_name** | Name of the y-axis. The default value depends on the type of chart.
+**limit** | A value that allows you to filter the data before displaying them. </br></br> For charts `unique_issues_by_category`, `issues_by_category`, `unique_penalty_issues_by_category` and `penalty_issues_by_category` only those categories will be shown where the number of issues is greater than or equal to the limit. </br></br> For chart `penalty_issues_by_category` only those categories will be shown where the number of median value is greater than or equal to the limit. </br></br> For chart `penalty_influence_distribution` only those categories will be shown where the number of values is greater than or equal to the limit. </br></br> The default value depends on the type of chart.
+**margin** | Defines the outer margin on all four sides of the chart. The available values are specified in the Enum class `MARGIN` from [plots const file](./common/plotly_consts.py). If not specified, the default value provided by Plotly is used.
+**sort_order** | Defines the sorting order of the chart. The available values are specified in the Enum class `SORT_ORDER` from [plots const file](./common/plotly_consts.py). If not specified, the default value provided by Plotly is used.
+**color** | Defines the color of the chart. The available values are specified in the Enum class `COLOR` from [plots const file](./common/plotly_consts.py). If not specified, the default value provided by Plotly is used.
+
+#### Example of config
+```yaml
+unique_issues_by_category:
+  margin: "ZERO"
+  limit: 10
+  sort_order: "total descending"
+  color: "RED"
+unique_penalty_issues_by_category:
+  limit: 30
+  sort_order: "category ascending"
+median_penalty_influence_by_category:
+penalty_influence_distribution:
+```
+
+The result will be four graphs (`unique_issues_by_category`, `unique_penalty_issues_by_category`, `median_penalty_influence_by_category`, `penalty_influence_distribution`) with the corresponding parameters.
+
+### Examples
+
+#### Number of unique issues by category
+<img src="./examples/unique_issues_by_category.png" width="500">
+
+#### Number of issues by category
+<img src="./examples/issues_by_category.png" width="500">
+
+#### Number of unique penalty issues by category
+<img src="./examples/unique_penalty_issues_by_category.png" width="500">
+
+#### Number of penalty issues by category
+<img src="./examples/penalty_issues_by_category.png" width="500">
+
+#### Median influence on penalty by category
+<img src="./examples/median_penalty_influence_by_category.png" width="500">
+
+#### Distribution of influence on penalty by category
+<img src="./examples/penalty_influence_distribution.png" width="500">

src/python/evaluation/plots/common/plotly_consts.py

@@ -0,0 +1,26 @@
+from enum import Enum
+
+
+class MARGIN(Enum):
+    ZERO = {'l': 0, 'r': 0, 'b': 0, 't': 0}
+
+
+class SORT_ORDER(Enum):  # noqa: N801
+    CATEGORY_ASCENDING = 'category ascending'
+    CATEGORY_DESCENDING = 'category descending'
+    TOTAL_ASCENDING = 'total ascending'
+    TOTAL_DESCENDING = 'total descending'
+
+
+class COLOR(Enum):
+    # Colors from px.colors.DEFAULT_PLOTLY_COLORS
+    BLUE = "rgb(31, 119, 180)"
+    ORANGE = "rgb(255, 127, 14)"
+    GREEN = "rgb(44, 160, 44)"
+    RED = "rgb(214, 39, 40)"
+    PURPLE = "rgb(148, 103, 189)"
+    BROWN = "rgb(140, 86, 75)"
+    PINK = "rgb(227, 119, 194)"
+    GRAY = "rgb(127, 127, 127)"
+    YELLOW = "rgb(188, 189, 34)"
+    CYAN = "rgb(23, 190, 207)"

src/python/evaluation/plots/common/plotters.py

@@ -0,0 +1,131 @@
+from statistics import median
+from typing import Any, Callable, Dict, Optional
+
+import pandas as pd
+import plotly.graph_objects as go
+from src.python.evaluation.inspectors.common.statistics import IssuesStatistics, PenaltyInfluenceStatistics
+from src.python.evaluation.plots.common import plotly_consts
+from src.python.evaluation.plots.common.utils import create_bar_plot, create_box_plot
+from src.python.review.inspectors.issue import IssueType
+
+
+def _get_dataframe_from_dict(
+    data_dict: Dict[Any, Any],
+    key_name: str,
+    value_name: str,
+    key_mapper: Callable = lambda x: x,
+    value_mapper: Callable = lambda y: y,
+):
+    """
+    Converts 'data_dict' to a dataframe consisting of two columns: 'key_name', 'value_name'.
+    'key_name' contains all keys of 'data_dict', 'value_name' contains all corresponding
+    values of 'data_dict'. With the functions 'key_mapper' and 'value_mapper' you can
+    additionally convert keys and values respectively.
+    """
+    converted_dict = {
+        key_name: list(map(key_mapper, data_dict.keys())),
+        value_name: list(map(value_mapper, data_dict.values())),
+    }
+
+    return pd.DataFrame.from_dict(converted_dict)
+
+
+def _extract_stats_from_issues_statistics(
+    statistics: IssuesStatistics, limit: int, only_unique: bool,
+) -> Dict[IssueType, int]:
+    categorized_statistics = statistics.get_short_categorized_statistics()
+
+    # If you want to get only unique issues, you should use position 0 of the tuple, otherwise 1.
+    position = int(not only_unique)
+
+    return {
+        issue_type: stat[position] for issue_type, stat in categorized_statistics.items() if stat[position] >= limit
+    }
+
+
+def get_unique_issues_by_category(
+    statistics: IssuesStatistics,
+    x_axis_name: str = 'Categories',
+    y_axis_name: str = 'Number of unique issues',
+    limit: int = 0,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> go.Figure:
+    filtered_stats = _extract_stats_from_issues_statistics(statistics, limit, only_unique=True)
+
+    df = _get_dataframe_from_dict(
+        filtered_stats,
+        key_name=x_axis_name,
+        value_name=y_axis_name,
+        key_mapper=lambda issue_type: issue_type.name,
+    )
+
+    return create_bar_plot(df, x_axis_name, y_axis_name, margin, sort_order, color)
+
+
+def get_issues_by_category(
+    statistics: IssuesStatistics,
+    x_axis_name: str = 'Categories',
+    y_axis_name: str = 'Number of issues',
+    limit: int = 0,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> go.Figure:
+    filtered_stats = _extract_stats_from_issues_statistics(statistics, limit, only_unique=False)
+
+    df = _get_dataframe_from_dict(
+        filtered_stats,
+        key_name=x_axis_name,
+        value_name=y_axis_name,
+        key_mapper=lambda issue_type: issue_type.name,
+    )
+
+    return create_bar_plot(df, x_axis_name, y_axis_name, margin, sort_order, color)
+
+
+def get_median_penalty_influence_by_category(
+    statistics: PenaltyInfluenceStatistics,
+    x_axis_name: str = 'Categories',
+    y_axis_name: str = 'Penalty influence (%)',
+    limit: int = 0,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> go.Figure:
+    stat = statistics.stat
+    filtered_stats = {issue_type: influence for issue_type, influence in stat.items() if median(influence) >= limit}
+
+    df = _get_dataframe_from_dict(
+        filtered_stats,
+        key_name=x_axis_name,
+        value_name=y_axis_name,
+        key_mapper=lambda issue_type: issue_type.name,
+        value_mapper=lambda influence: median(influence),
+    )
+
+    return create_bar_plot(df, x_axis_name, y_axis_name, margin, sort_order, color)
+
+
+def get_penalty_influence_distribution(
+    statistics: PenaltyInfluenceStatistics,
+    x_axis_name: str = 'Categories',
+    y_axis_name: str = 'Penalty influence (%)',
+    limit: int = 0,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+):
+    stat = statistics.stat
+    filtered_stats = {issue_type: influence for issue_type, influence in stat.items() if len(influence) >= limit}
+
+    df = _get_dataframe_from_dict(
+        filtered_stats,
+        key_name=x_axis_name,
+        value_name=y_axis_name,
+        key_mapper=lambda issue_type: issue_type.name,
+    )
+    df = df.explode(y_axis_name)
+
+    return create_box_plot(df, x_axis_name, y_axis_name, margin, sort_order, color)

src/python/evaluation/plots/common/utils.py

@@ -0,0 +1,76 @@
+import os
+from pathlib import Path
+from typing import List, Optional
+
+import pandas as pd
+import plotly.express as px
+import plotly.graph_objects as go
+from src.python.evaluation.plots.common import plotly_consts
+from src.python.review.common.file_system import Extension
+
+
+def get_supported_extensions() -> List[str]:
+    extensions = Extension.get_image_extensions()
+    extensions.append(Extension.JSON)
+    return [extension.value for extension in extensions]
+
+
+def create_bar_plot(
+    df: pd.DataFrame,
+    x_axis: str,
+    y_axis: str,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> go.Figure:
+    fig = px.bar(df, x=x_axis, y=y_axis, text=y_axis)
+    update_figure(fig, margin, sort_order, color)
+    return fig
+
+
+def create_box_plot(
+    df: pd.DataFrame,
+    x_axis: str,
+    y_axis: str,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> go.Figure:
+    fig = px.box(df, x=x_axis, y=y_axis)
+    update_figure(fig, margin, sort_order, color)
+    return fig
+
+
+def update_figure(
+    fig: go.Figure,
+    margin: Optional[plotly_consts.MARGIN] = None,
+    sort_order: Optional[plotly_consts.SORT_ORDER] = None,
+    color: Optional[plotly_consts.COLOR] = None,
+) -> None:
+    new_layout = {}
+
+    if margin is not None:
+        new_layout["margin"] = margin.value
+
+    if sort_order is not None:
+        new_layout["xaxis"] = {"categoryorder": sort_order.value}
+
+    fig.update_layout(**new_layout)
+
+    new_trace = {}
+
+    if color is not None:
+        new_trace["marker"] = {"color": color.value}
+
+    fig.update_traces(**new_trace)
+
+
+def save_plot(
+    fig: go.Figure,
+    dir_path: Path,
+    plot_name: str = "result_plot",
+    extension: Extension = Extension.SVG,
+) -> None:
+    os.makedirs(dir_path, exist_ok=True)
+    file = dir_path / f"{plot_name}{extension.value}"
+    fig.write_image(str(file))