Conversation
* V3 * removing v2 ci pipeline * removing outdated .gitmodules * keeping the noise that sample size of 50 and slight decay of .5 in EMA provides to avoid having any one model completely dominate the subnet. * release 2.2.6 datasets, models, and lora support (#188) * deprecate stable-diffusion-inpainting * .env templates * V3/RGB (#191) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * V3 frame extraction (#192) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix * frame extraction --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * setup.sh * removing wandb log call from generator * V3/2.2.9 (#189) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * fixing prompt indexing * properly handling new prompt dictionary key (task type) * V3/2.2.11 (#190) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * prompt sanitation + i2v model * more retries for prompt sanitation * fixing truthy tuple assertion * Update min_compute.yml * fixing setup script name in docs * correct script name * updated requirements.txt with bittensor-cli * removing wandb.off * import cleanup * miner substrate thread restart + vali autoupdate test * temporary v3 branch set to test autoudpate * autoupdate update * lower frequency of audoupdate check * autoudpate test * check autoupdate at setp 0 * typo * autoupdate test * dont set weights immediately at startup in case of many restarts * Pyproject toml (#193) * pyproject setup * executable setup.sh * autoupdate test * resetting version after autoupdate tests * Add Hugging Face model access instructions to validator docs; improve logging and fix LLM device mapping for multi-GPU - Added section to Validating.md with instructions for gaining access to required Hugging Face models (FLUX.1-dev, DeepFloyd IF). - Added logging of generation arguments in generation_pipeline.py. - Fix LLM loading for multi-GPU in prompt_generator.py: use device_map and remove .to(self.device) for quantized models. Quantized LLMs must use device_map for correct device placement; calling .to(self.device) causes device mismatch errors. Parse GPU ID from device string for device_map assignment. * fixing image_samples check for i2i * hf_xet requirement * wandb autorestart * Fix: raise error if image is None for i2i/i2v tasks and ensure image is converted from array * fixing wandb autorestart * error log * Update setup.sh to install Node.js 20.x LTS from NodeSource for pm2 compatibility; add doc note for existing validators' Hugging Face access * external port for proxy cuz tensordock rugged us (#196) * incentive doc * Typo * proxy updates * v2 parity encoding (#197) * final autoupdate test * reset version --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai>
* fixing wandb cache clean paths (#202) * max frames configuration * fn header update * slight increase to timeout
* adding extra metadata to testnet requests for miners * adding label and mediatype to testnet metadata
* log augmentation params * braindead typo
* fix hotkey check in sync_metagraph * bump version
* healthcheck wip * remove old miner health task vars, change miner healtchechk endpoint name * health count logging * fixing query based on health logic * updating healthcheck interval to 10 * removing unecessary lock * use DEFAULT_TIMEOUT * adding blacklisting for bad responses * fixing detect_image fallback * a couple comments * update functionality for generator and proxy
* adding basic health dict to miner tracker * move all request processing logic to epistula module * reflect request processing updates in eval engine * healthy/unhealthy miner uid functions * proxy using simpler miner health from tracker state * new fn signature for score_challenge
* scraper wip * fixing queries with max date set in tbs, also adding placeholder for reverse image search which i cant get to work rn due to captchas * taking first sentence of prompt as initial version of search queries * - Adding specific media scraping interval config - Adding retry logic and error handling to scraping callback * Fixing enum value for output path in scraper * add selenium to requirements * Fixing str treated as enum * cleaning up * increasing media update intervals * [testnet] Safer Miner Prediction History (#214) * centralizing logic for safely getting valid predictions and associated labels * cleaning up
* adding bm-aura-imagegen dataset * log mask as npy artifact * format
* binary image endpoint * adding video endpoint without preprocessing (#221) * fix frames list (should be np array)
* segmentation v1 * remove debug logging * legacy detector backward compatibility * initial reward split for segmentation release * typo fix and increasing window size --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai>
* LLM-based query generation + moving scraping to generator class * Release 3.0.11 (#223) * V3 (#187) * V3 * removing v2 ci pipeline * removing outdated .gitmodules * keeping the noise that sample size of 50 and slight decay of .5 in EMA provides to avoid having any one model completely dominate the subnet. * release 2.2.6 datasets, models, and lora support (#188) * deprecate stable-diffusion-inpainting * .env templates * V3/RGB (#191) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * V3 frame extraction (#192) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix * frame extraction --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * setup.sh * removing wandb log call from generator * V3/2.2.9 (#189) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * fixing prompt indexing * properly handling new prompt dictionary key (task type) * V3/2.2.11 (#190) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * prompt sanitation + i2v model * more retries for prompt sanitation * fixing truthy tuple assertion * Update min_compute.yml * fixing setup script name in docs * correct script name * updated requirements.txt with bittensor-cli * removing wandb.off * import cleanup * miner substrate thread restart + vali autoupdate test * temporary v3 branch set to test autoudpate * autoupdate update * lower frequency of audoupdate check * autoudpate test * check autoupdate at setp 0 * typo * autoupdate test * dont set weights immediately at startup in case of many restarts * Pyproject toml (#193) * pyproject setup * executable setup.sh * autoupdate test * resetting version after autoupdate tests * Add Hugging Face model access instructions to validator docs; improve logging and fix LLM device mapping for multi-GPU - Added section to Validating.md with instructions for gaining access to required Hugging Face models (FLUX.1-dev, DeepFloyd IF). - Added logging of generation arguments in generation_pipeline.py. - Fix LLM loading for multi-GPU in prompt_generator.py: use device_map and remove .to(self.device) for quantized models. Quantized LLMs must use device_map for correct device placement; calling .to(self.device) causes device mismatch errors. Parse GPU ID from device string for device_map assignment. * fixing image_samples check for i2i * hf_xet requirement * wandb autorestart * Fix: raise error if image is None for i2i/i2v tasks and ensure image is converted from array * fixing wandb autorestart * error log * Update setup.sh to install Node.js 20.x LTS from NodeSource for pm2 compatibility; add doc note for existing validators' Hugging Face access * external port for proxy cuz tensordock rugged us (#196) * incentive doc * Typo * proxy updates * v2 parity encoding (#197) * final autoupdate test * reset version --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai> * autoupdate set to main * testing autoupdate on testnet * autoupdate enabled by default * autoudpate testnet * pointing autoupdate at main by default * removing extra state load command * setting back to 360 epoch length * burn for initial v3 release rampup * debug log typo * fixed merge to testnet * Max Frames and Timeout (#203) * fixing wandb cache clean paths (#202) * max frames configuration * fn header update * slight increase to timeout * adding extra metadata to testnet requests for miners (#201) * remove max size arg * Testnet Metadata (#204) * adding extra metadata to testnet requests for miners * adding label and mediatype to testnet metadata * Log Augmentation Parameters (#205) * log augmentation params * braindead typo * bump verison * [testnet] Release 3.0.5 (#207) * fix hotkey check in sync_metagraph * bump version * [testnet] Miner healthchecks (#209) * healthcheck wip * remove old miner health task vars, change miner healtchechk endpoint name * health count logging * fixing query based on health logic * updating healthcheck interval to 10 * removing unecessary lock * use DEFAULT_TIMEOUT * adding blacklisting for bad responses * fixing detect_image fallback * a couple comments * update functionality for generator and proxy * [testnet] Revised Miner Healthcheck (#210) * adding basic health dict to miner tracker * move all request processing logic to epistula module * reflect request processing updates in eval engine * healthy/unhealthy miner uid functions * proxy using simpler miner health from tracker state * new fn signature for score_challenge * [testnet] Image Scraping (#213) * scraper wip * fixing queries with max date set in tbs, also adding placeholder for reverse image search which i cant get to work rn due to captchas * taking first sentence of prompt as initial version of search queries * - Adding specific media scraping interval config - Adding retry logic and error handling to scraping callback * Fixing enum value for output path in scraper * add selenium to requirements * Fixing str treated as enum * cleaning up * increasing media update intervals * [testnet] Safer Miner Prediction History (#214) * centralizing logic for safely getting valid predictions and associated labels * cleaning up * Aura Dataset and Mask W&B Logging (#217) * adding bm-aura-imagegen dataset * log mask as npy artifact * format * increasing media cache refresh default * disabling scrape_new_media_on_interval * warnings for missing keys in miner history state loading * lowering media scraping default (not currenlty used) * remvoning unused healthcheck endpoint in miner * [testnet] Binary Image Proxy Endpoint (#220) * binary image endpoint * adding video endpoint without preprocessing (#221) * fix frames list (should be np array) * adding wan2.1-t2v-1.3B (#222) * removing scraping from this release * version bump * whitespace * putting peft requirement back * ftfy requirement * refactor(models): make Wan2.1 VAE loading lazy - Convert VAE loading to use lazy tuple pattern (fn, args) - Update load_vae to use kwargs for consistency * Remove duplicate function * fix torch_dtype typo for wan * extending window to 200 * imports --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai> * Release/3.0.11 (#224) * Release 3.0.11 (#225) * updating config to allow toggling generation/scraping on and off * updated scraper to get native resolution images * override aug level * native resolution wip * Release 3.1.0 (#227) * V3 (#187) * V3 * removing v2 ci pipeline * removing outdated .gitmodules * keeping the noise that sample size of 50 and slight decay of .5 in EMA provides to avoid having any one model completely dominate the subnet. * release 2.2.6 datasets, models, and lora support (#188) * deprecate stable-diffusion-inpainting * .env templates * V3/RGB (#191) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * V3 frame extraction (#192) * bgr images --> rgb images * proper BGR -> RGB conversion * eradicate all usage of bgr in image challenge flow * extract frames as rgb * skip extraneous rgb conversion * fix deeperforesnics consistency * v2 frame sampling parity + eidon mp4 fix * missing import * handling improper reporting of fps in wembs * correct content-type on miner side * max_fpx setting * improved video metadata extraction * cleaning up ffprobe options * fixing first frame rotation edge case * i2i fix * frame extraction --------- Co-authored-by: Dylan Uys <dylan@bitmind.ai> * setup.sh * removing wandb log call from generator * V3/2.2.9 (#189) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * fixing prompt indexing * properly handling new prompt dictionary key (task type) * V3/2.2.11 (#190) * mugshot dataset * black * i2v support and fixed prompt motion enhancement * gen pipeline updates for i2v * prompt sanitation + i2v model * more retries for prompt sanitation * fixing truthy tuple assertion * Update min_compute.yml * fixing setup script name in docs * correct script name * updated requirements.txt with bittensor-cli * removing wandb.off * import cleanup * miner substrate thread restart + vali autoupdate test * temporary v3 branch set to test autoudpate * autoupdate update * lower frequency of audoupdate check * autoudpate test * check autoupdate at setp 0 * typo * autoupdate test * dont set weights immediately at startup in case of many restarts * Pyproject toml (#193) * pyproject setup * executable setup.sh * autoupdate test * resetting version after autoupdate tests * Add Hugging Face model access instructions to validator docs; improve logging and fix LLM device mapping for multi-GPU - Added section to Validating.md with instructions for gaining access to required Hugging Face models (FLUX.1-dev, DeepFloyd IF). - Added logging of generation arguments in generation_pipeline.py. - Fix LLM loading for multi-GPU in prompt_generator.py: use device_map and remove .to(self.device) for quantized models. Quantized LLMs must use device_map for correct device placement; calling .to(self.device) causes device mismatch errors. Parse GPU ID from device string for device_map assignment. * fixing image_samples check for i2i * hf_xet requirement * wandb autorestart * Fix: raise error if image is None for i2i/i2v tasks and ensure image is converted from array * fixing wandb autorestart * error log * Update setup.sh to install Node.js 20.x LTS from NodeSource for pm2 compatibility; add doc note for existing validators' Hugging Face access * external port for proxy cuz tensordock rugged us (#196) * incentive doc * Typo * proxy updates * v2 parity encoding (#197) * final autoupdate test * reset version --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai> * autoupdate set to main * testing autoupdate on testnet * autoupdate enabled by default * autoudpate testnet * pointing autoupdate at main by default * removing extra state load command * setting back to 360 epoch length * burn for initial v3 release rampup * debug log typo * fixed merge to testnet * Max Frames and Timeout (#203) * fixing wandb cache clean paths (#202) * max frames configuration * fn header update * slight increase to timeout * adding extra metadata to testnet requests for miners (#201) * remove max size arg * Testnet Metadata (#204) * adding extra metadata to testnet requests for miners * adding label and mediatype to testnet metadata * Log Augmentation Parameters (#205) * log augmentation params * braindead typo * bump verison * [testnet] Release 3.0.5 (#207) * fix hotkey check in sync_metagraph * bump version * [testnet] Miner healthchecks (#209) * healthcheck wip * remove old miner health task vars, change miner healtchechk endpoint name * health count logging * fixing query based on health logic * updating healthcheck interval to 10 * removing unecessary lock * use DEFAULT_TIMEOUT * adding blacklisting for bad responses * fixing detect_image fallback * a couple comments * update functionality for generator and proxy * [testnet] Revised Miner Healthcheck (#210) * adding basic health dict to miner tracker * move all request processing logic to epistula module * reflect request processing updates in eval engine * healthy/unhealthy miner uid functions * proxy using simpler miner health from tracker state * new fn signature for score_challenge * [testnet] Image Scraping (#213) * scraper wip * fixing queries with max date set in tbs, also adding placeholder for reverse image search which i cant get to work rn due to captchas * taking first sentence of prompt as initial version of search queries * - Adding specific media scraping interval config - Adding retry logic and error handling to scraping callback * Fixing enum value for output path in scraper * add selenium to requirements * Fixing str treated as enum * cleaning up * increasing media update intervals * [testnet] Safer Miner Prediction History (#214) * centralizing logic for safely getting valid predictions and associated labels * cleaning up * Aura Dataset and Mask W&B Logging (#217) * adding bm-aura-imagegen dataset * log mask as npy artifact * format * increasing media cache refresh default * disabling scrape_new_media_on_interval * warnings for missing keys in miner history state loading * lowering media scraping default (not currenlty used) * remvoning unused healthcheck endpoint in miner * [testnet] Binary Image Proxy Endpoint (#220) * binary image endpoint * adding video endpoint without preprocessing (#221) * fix frames list (should be np array) * adding wan2.1-t2v-1.3B (#222) * [testnet] Segmentation Challenges (#226) * segmentation v1 * remove debug logging * legacy detector backward compatibility * initial reward split for segmentation release * typo fix and increasing window size --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai> * removing scraper for current release * whitespace * fix bug in miner history resetting when a detector changes to a detector without dereg * limit uids * cleanup * proxy updates --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai> * Patch/autoupdate minor version (#229) * stamina for scraping * cleanup --------- Co-authored-by: Benjamin S Liang <caliangben@gmail.com> Co-authored-by: Dylan Uys <dylan@bitmind.ai>
* wandb logging fix for masks * exclude mask in detector challenge headers * return empty lists when no predictions received
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview: Alignment with Real World Data
Key Changes
PromptGeneratorclass.neurons/validator/generator.py