Skip to content

Adding two finetuned image models to expand validator challenges#105

Merged
aliang322 merged 2 commits intotestnetfrom
expand-models/image-finetuned
Nov 21, 2024
Merged

Adding two finetuned image models to expand validator challenges#105
aliang322 merged 2 commits intotestnetfrom
expand-models/image-finetuned

Conversation

@aliang322
Copy link
Contributor

@aliang322 aliang322 commented Nov 21, 2024

These changes introduce two popular open source image models for validator challenges, as part of our continuous effort to meaningfully expand the distribution of synthetic images that miners classify. Our goal is to encourage better miner generalizability to images produced by different deepfake methods.

For validators: No new dependencies/install requirements were added.

For miners: We foresee a difficulty increase in subnet challenges as miners adapt to the new models' image distributions.

The two models are:

These were selected by

  1. Manually reviewing state-of-the-art models (tech review blogs, AI news, Hugging Face models) and open source models finetuned on proprietary image generators (e.g. Midjourney) and/or popular generated image genres in-the-wild (e.g. anime)
  2. Applying our subnet API benchmarking procedure to synthetic mirrors generated by candidate models.
    • Notably, our subnet scored worse on openjourney-v4 and animagine-xl-3.1 (68.76% and 83.8% accuracy) than any of our existing validator challenge models, indicating that miners have yet to optimally generalize to those model output distributions (supporting the addition of these models).
    • We sourced the real images for mirroring through a scraping method outlined in the technical blog post (link above).
    • API benchmarking for new model selection was conducted end-to-end within 24-HR spans to increase the likelihood that accuracy measures reflected a stable subnet state (as opposed to accuracy variations caused by significant miner changes on mainnet).

@aliang322 aliang322 requested a review from benliang99 November 21, 2024 20:51
@dylanuys dylanuys requested review from dylanuys and removed request for benliang99 and dylanuys November 21, 2024 20:54
@aliang322 aliang322 merged commit ec35faf into testnet Nov 21, 2024
@aliang322 aliang322 deleted the expand-models/image-finetuned branch January 10, 2025 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants