Skip to content

Release 1.2.9#109

Merged
dylanuys merged 14 commits intomainfrom
testnet
Nov 25, 2024
Merged

Release 1.2.9#109
dylanuys merged 14 commits intomainfrom
testnet

Conversation

@dylanuys
Copy link
Contributor

@dylanuys dylanuys commented Nov 24, 2024

TLDR

  • Two new diffusion models for validator challenges
  • Better device management, now exposing --neuron.device through validator.env's DEVICE variable
  • Deprecating unused prompt generation model due to the biases it displayed. We have replaced this with our ImageAnnotationGenerator pipeline.

New Model Details

We are introducing two additional popular open source image models for validator challenges, as part of our continuous effort to meaningfully expand the distribution of synthetic images that miners classify. Our goal is to encourage better miner generalizability to images produced by different deepfake methods.

For validators: No new dependencies/install requirements were added - models will automatically be downloaded to your local huggingface cache.

For miners: We foresee a difficulty increase in subnet challenges as miners adapt to the new models' image distributions.

The two models are:

These were selected by

  1. Manually reviewing state-of-the-art models (tech review blogs, AI news, Hugging Face models) and open source models finetuned on proprietary image generators (e.g. Midjourney) and/or popular generated image genres in-the-wild (e.g. anime)
  2. Applying our subnet API benchmarking procedure to synthetic mirrors generated by candidate models.
    • Notably, our subnet scored worse on openjourney-v4 and animagine-xl-3.1 (68.76% and 83.8% accuracy) than any of our existing validator challenge models, indicating that miners have yet to optimally generalize to those model output distributions (supporting the addition of these models).
    • We sourced the real images for mirroring through a scraping method outlined in the technical blog post (link above).
    • API benchmarking for new model selection was conducted end-to-end within 24-HR spans to increase the likelihood that accuracy measures reflected a stable subnet state (as opposed to accuracy variations caused by significant miner changes on mainnet).

dylanuys and others added 8 commits November 19, 2024 17:17
* adding rich arg, adding coldkeys and hotokeys

* moving rich to payload from headers

* bump version

---------

Co-authored-by: benliang99 <caliangben@gmail.com>
Adding two finetuned image models to expand validator challenges
Updated transformers version to fix tokenizer initialization error
* Made gpu id specification consistent across synthetic image generation models

* Changed gpu_id to device

* Docstring grammar

* add neuron.device to SyntheticImageGenerator init

* Fixed variable names

* adding device to start_validator.sh

* deprecating old/biased random prompt generation

* properly clear gpu of moderation pipeline

* simplifying usage of self.device

* fixing moderation pipeline device

* explicitly defining model/tokenizer for moderation pipeline to avoid accelerate auto device management

* deprecating random prompt generation

---------

Co-authored-by: benliang99 <caliangben@gmail.com>
bump version
@dylanuys dylanuys changed the base branch from main to yolov8_wip November 24, 2024 18:15
@dylanuys dylanuys changed the base branch from yolov8_wip to main November 24, 2024 18:15
@aliang322 aliang322 self-requested a review November 25, 2024 05:53
Copy link
Contributor

@aliang322 aliang322 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Excited for the new models and clarified flows.

@dylanuys dylanuys merged commit 3c35da7 into main Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants