From 790eb90259a613c0147961453d5c43c190bac796 Mon Sep 17 00:00:00 2001 From: Dean <31391056+deanpress@users.noreply.github.com> Date: Sat, 21 Dec 2024 23:57:16 +0400 Subject: [PATCH 1/4] Update README.md There wasn't a readme, so I gave o1 the code and have it explain the purpose of this project to me, giving it the assumption it has to do with determining rewards for contributors, decided by a mix of AI and human evaluation, as part of a funding program (figured because of the "deepfunding" org name). Took the best parts of the various outputs it gave me and put them in a readme along with some personal interpretations. --- README.md | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 118 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 82b0ad4..74bf252 100644 --- a/README.md +++ b/README.md @@ -1 +1,118 @@ -# Scoring mechanism for deep funding +# DeepFunding: Optimal Distribution Weighting + +## Introduction + +This code is designed to help fairly distribute rewards among contributors based on multiple input evaluations and user preferences. By combining different assessment methods through optimization, we ensure that rewards reflect the collective input (given by AI and humans) and defined criteria. + +It could be used to determine fair rewards for contributors by: + +1. **Defining Distributions:** + - **AI Assessment:** Evaluations from AI on each contributor's impact. + - **Community Feedback:** Humans also evaluate each contributor's impact. + - **Egalitarian Approach:** Equal distribution to ensure fairness. + +2. **Gathering Comparisons:** + - Other humans compare pairs of contributors for an achievement, indicating who deserves more recognition and by how much. + +3. **Optimization Process:** + - Combines the distributions to minimize discrepancies, balancing expert opinions, community feedback, and fairness. + +4. **Final Reward Distribution:** + - Applies optimized weights to allocate specific rewards to each contributor. + + +## Project Structure + +The project consists of two main Python scripts: + +- **`scoring.py`**: Contains functions to calculate the cost of a proposed distribution and find the optimal weights to combine multiple distributions. +- **`example.py`**: Provides an example scenario using characters from *The Lord of the Rings*. + +## How It Works + +### `scoring.py` + +- **Cost Function (`cost_function`)**: + + This function calculates the sum of squared differences between the log-probabilities of item pairs as provided by user comparisons. It measures how well a proposed distribution aligns with the provided preferences. + +- **Find Optimal Weights (`find_optimal_weights`)**: + + This function uses the `scipy.optimize.minimize` method to determine the best weights for combining multiple input distributions. The goal is to minimize the cost function while ensuring that the weights sum to 1 and each weight is between 0 and 1. + +### `example.py` + +- **Character List**: + + A predefined list of 20 characters from *The Lord of the Rings* used to simulate the distribution of credit for defeating Sauron. + +- **Distributions**: + + Three different probability distributions are provided: + + 1. **GPT Distribution**: A hypothetical distribution based on a language model's assessment. + 2. **User Opinion**: A manually defined distribution reflecting personal judgment. + 3. **Egalitarian Split**: An equal distribution of credit among all characters. + +- **User Comparisons**: + + The script interactively gathers user preferences by asking the user to compare pairs of characters and specify how much more credit one deserves over the other. + +- **Optimization and Results**: + + After collecting user comparisons, the script calculates the optimal weights for combining the distributions to best match the user's preferences. It then displays the costs of each individual distribution, the optimal weights, the resulting combined distribution, and the cost of this ideal distribution. + +### Prerequisites + +- **Python**: Ensure you have Python 3.6 or higher installed on your system. +- **Python Packages**: The project relies on the following Python libraries: + - `numpy` + - `scipy` + + +### Installation + +1. **Clone the Repository**: + + ```bash + git clone https://github.com/deepfunding/scoring.git + cd optimal-distribution-weighting + ``` + +2. **Install Dependencies**: + + ```bash + pip install numpy scipy + ``` + +### Running the Example + +1. **Execute the Example Script**: + + ```bash + python example.py + ``` + +2. **Provide User Comparisons**: + + The script will prompt you to compare pairs of characters based on who deserves more credit for helping defeat Sauron. For each comparison: + + - **Choose Between Two Characters**: Type '1' or '2' to indicate your preference. + - **Specify the Magnitude**: Input a number representing how many times more credit one character deserves over the other. + + **Example Interaction**: + + ``` + Who deserves more credit for helping defeat Sauron? Gandalf or Aragorn? + Type '1' for Gandalf or '2' for Aragorn: 2 + How many times more credit does the second deserve? Give a number (e.g., 3): 3 + ``` + +3. **View Results**: + + After providing your comparisons, the script will display: + + - **Cost of Each Pure Distribution**: Indicates how well each individual distribution aligns with your preferences. + - **Optimal Weights**: The best combination of weights to minimize the cost. + - **Lowest-Cost Distribution**: The final adjusted distribution based on optimal weighting. + - **Cost of Ideal Distribution**: The minimized cost achieved by the optimal combination. From 0e801cb7fe5a7bace9b316a72820f02fb32b71f9 Mon Sep 17 00:00:00 2001 From: Dean <31391056+deanpress@users.noreply.github.com> Date: Sat, 21 Dec 2024 23:58:23 +0400 Subject: [PATCH 2/4] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 74bf252..41bcfaa 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# DeepFunding: Optimal Distribution Weighting +# Scoring Mechanism for Deep Funding ## Introduction From 0291cb3bd53b0bb9dbcaf3420acbe05f07c5addd Mon Sep 17 00:00:00 2001 From: Dean <31391056+deanpress@users.noreply.github.com> Date: Sun, 22 Dec 2024 00:20:17 +0400 Subject: [PATCH 3/4] Update README.md clarify the deep funding problem --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 41bcfaa..f0e4ce7 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ ## Introduction -This code is designed to help fairly distribute rewards among contributors based on multiple input evaluations and user preferences. By combining different assessment methods through optimization, we ensure that rewards reflect the collective input (given by AI and humans) and defined criteria. +This project tackles the [deep funding problem](https://github.com/opensource-observer/insights/blob/main/community/dependency_graph_funding/readme.md): how to fairly allocate resources (funding, recognition, or both) within a large, interdependent open-source ecosystem. By blending scoring mechanisms that incorporate human and AI feedback with a dependency graph of critical repositories, we arrive at an optimized way to distribute funding. -It could be used to determine fair rewards for contributors by: +The solution can be used to determine fair rewards for contributors by: 1. **Defining Distributions:** - **AI Assessment:** Evaluations from AI on each contributor's impact. From 8ad8be860f08a40e37bbc0d0851cc8a7e2900e18 Mon Sep 17 00:00:00 2001 From: Dean <31391056+deanpress@users.noreply.github.com> Date: Sun, 22 Dec 2024 00:20:43 +0400 Subject: [PATCH 4/4] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f0e4ce7..1bd0502 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ## Introduction -This project tackles the [deep funding problem](https://github.com/opensource-observer/insights/blob/main/community/dependency_graph_funding/readme.md): how to fairly allocate resources (funding, recognition, or both) within a large, interdependent open-source ecosystem. By blending scoring mechanisms that incorporate human and AI feedback with a dependency graph of critical repositories, we arrive at an optimized way to distribute funding. +This project tackles the [deep funding problem](https://github.com/opensource-observer/insights/blob/main/community/dependency_graph_funding/readme.md): how to fairly allocate resources (funding, recognition, or both) within a large, interdependent open-source ecosystem. By blending scoring mechanisms that incorporate human and AI feedback, we arrive at an optimized way to distribute funding. The solution can be used to determine fair rewards for contributors by: