SkyGAN

This repository contains the code for our paper SkyGAN: Realistic Cloud Imagery for Image-based Lighting.

SkyGAN generates cloudy sky images from a user-chosen sun position that are readily usable as an environment map in any rendering system. We leverage an existing clear sky model to produce the input to our neural network which enhances the sky with clouds, haze and horizons learned from real photographs.

SkyGAN runs on Intel GPUs (or other SYCL-compatible accelerators). See the StyleGAN3 to SYCL porting details in this repository.

Project structure

Important files/directories:

• src/stylegan3: SkyGAN – modified StyleGAN3 code with HDR support, implemented encoding of a clear sky generated by the Prague sky model, added reconstruction loss on the clear sky image, azimuth marginalization on data loading, modified visualizer etc.
  • environment.yml: the Conda environment for general use (use that by default)
  • environment_cluster.yml: exported Conda environment we trained in
  • environment_wsl.yml: exported Conda environment we used for running inference (generating preview images)
  • environment_intel.yml: environment for running on Intel GPUs
  • *_FID.py: modified files used for FID computation - replace the matching files if you want to run the evaluation
  • training/FID_prepare.ipynb: a Jupyter notebook used for processing training outputs, filtering and sorting the snapshots to progressively increase the data points density, and creating evaluation jobs
  • training/FID_plot.ipynb: a Jupyter notebook visualising the training progress
• train_cluster.sh: a PBS job launch script used to start training runs (training parameters were changed before each run; the example parameters in the file were used in the "Ours" run: resuming from a previous training which reached 8.7 Mimgs)
• generate_preview_samples.sh: a script we used for generating preview images included in the supplementary material
• ArPragueSkyModelGroundXYZ, sky_image_generator.*: the code and fitted data of the XYZ version of the Prague sky model used for generating a clear sky image passed to our network

Downloads

The data-processing scripts for converting captured RAW images into StyleGAN-compatible format are available in a separate repository.

The SkyGAN dataset is available for non-commercial use under the CC-BY-NC-SA 4.0 license. Contact us to discuss other potential arrangements.

SkyGAN dataset © 2019-2023 by Tobias Rittig, Štěpán Hojdar, Jaroslav Křivánek, Ronan Cailleau, Charles University, Faculty of Mathematics and Physics and Chaos Czech a.s. is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/4.0/

To get the dataset, place the following two files (auto_processed_20230405_1727.csv and download.sh) into the same folder, then review and run the download script. You can choose to download only a part of the dataset, e.g. for training SkyGAN, you only need the "stereographic up EXR" images (~83 GB) and save more than 1.1TB of bandwidth and space by not downloading the latlongs and JPEGs.

Trained network snapshots producing the results presented in the paper (each ~550MB):

• 👉 O ("Ours"), a version trained with all our modifications enabled. This includes the encoded clear sky images, the reconstruction loss (B_C) and the training on azimuth-marginalised images (B_A). 👈
• B (Baseline), a baseline which is trained from scratch without our encoder and clear sky reconstruction loss. This corresponds to a standard StyleGAN3-T with support for HDR values.
• B_A ("Aug"), the same network architecture as B, but trained on azimuth-marginalised images.
• B_C ("Clear"), the Baseline network but including the autoencoder task for the clear sky.

Usage

Installation

1. Prepare the Conda environment using our environment_intel.yml (or other environment we provided if you want to use the exact same package versions we did)

conda env create -f src/stylegan3/environment_intel.yml
source src/stylegan3/prepare_env.inc.sh  # you will need to re-run this after restart or after running a new shell

• Make sure your conda is up to date with conda update -n base conda.
• To avoid long waiting at "Solving Environment step", switching to the libmamba solver is advisable.

2. Make sure gen_images.py (and visualizer.py?) from our ported StyleGAN3 code run correctly (optional but useful for debugging)

• clone our stylegan3 repository, change into its directory, checkout the active branch ('debug_profiling' as of writing this), then verify everything works by running the following:

python gen_images.py --outdir=out --trunc=1 --seeds=2,1 --network=https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl

• Note that the first run will cause compilation of PyTorch plugins/kernels, which may take a couple of minutes.
• If everything works, you should see a cat image in the "out" directory.

3. Build the sky_image_generator module (makes clear sky images, which are encoded in SkyGAN – used in both training and inference)

g++-10 -O3 -Wall -shared -std=c++11 -fPIC -fopenmp $(python3 -m pybind11 --includes) -I"$CONDA_PREFIX"/include/opencv4/ -I"$CONDA_PREFIX"/lib/python3.9/site-packages/numpy/core/include/ -L"$CONDA_PREFIX"/lib/ sky_image_generator.cpp -o sky_image_generator$(python3-config --extension-suffix) -lopencv_core -lopencv_imgcodecs
cd src/stylegan3

then link the compiled module and its data to the current directory

ln -s ../../sky_image_generator$(python3-config --extension-suffix)
ln -s ../../ArPragueSkyModelGroundXYZ/

Inference

Generate a few images using a pre-trained network:

OPENCV_IO_ENABLE_OPENEXR=1 python gen_images.py --network /home/user/Downloads/k00133t_Ours_FID14.6@28.9M_network-snapshot-002343.pkl --normalize-azimuth=True --seeds='elevations+1000' --outdir=out --azimuth=180 --elevations=10,70

• Replace /home/user/Downloads/k00133t_Ours_FID14.6@28.9M_network-snapshot-002343.pkl with a path to a pre-trained network weights.
• The generated skies with clouds are named fake_seed*.png|exr; the clear_rec_* outputs are the clear sky images as reconstructed by the network.
• To generate images for a custom seed(s), change the 'elevations+1000' (--seeds value) to a number, list of numbers or a range. Choose only one sun elevation per script run (and a different outdir) to avoid overwriting previously generated images.

Run the interactive visualizer, then select a training snapshot (.pkl) using the Browse... button on top of the window (tip: checking Force FP32 makes it run faster, at least in WSL2 on Windows 10 with one 1080Ti)

./start_visualiser.sh --browse-dir /home/user/Downloads/

• Replace /home/user/Downloads/ with a path to a directory where you store your pickles.

Training

• TODO test with dummy singleimage?

Citation

If you find this work, the dataset or processing scripts useful in your research, please cite our paper:

@article{https://doi.org/10.1111/cgf.14990,
author = {Mirbauer, Martin and Rittig, Tobias and Iser, Tomáš and Křivánek, Jaroslav and Šikudová, Elena},
title = {SkyGAN: Realistic Cloud Imagery for Image-based Lighting},
journal = {Computer Graphics Forum},
volume = {43},
number = {1},
pages = {e14990},
keywords = {modelling; natural phenomena, rendering; image-based rendering, rendering; atmospheric effects},
doi = {https://doi.org/10.1111/cgf.14990},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14990},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14990},
abstract = {Abstract Achieving photorealism when rendering virtual scenes in movies or architecture visualizations often depends on providing a realistic illumination and background. Typically, spherical environment maps serve both as a natural light source from the Sun and the sky, and as a background with clouds and a horizon. In practice, the input is either a static high-resolution HDR photograph manually captured on location in real conditions, or an analytical clear sky model that is dynamic, but cannot model clouds. Our approach bridges these two limited paradigms: a user can control the sun position and cloud coverage ratio, and generate a realistically looking environment map for these conditions. It is a hybrid data-driven analytical model based on a modified state-of-the-art GAN architecture, which is trained on matching pairs of physically-accurate clear sky radiance and HDR fisheye photographs of clouds. We demonstrate our results on renders of outdoor scenes under varying time, date and cloud covers. Our source code and a dataset of 39 000 HDR sky images are publicly available at https://github.com/CGGMFF/SkyGAN.},
year = {2024}
}

TODOs

• finish usage instructions (how to run inference - copy from the PC in S126)
• test if inference still works after code cleanup
• test if training still works after code cleanup
• link to trained pickles
• link to data processing scripts
• link to dataset
• update citation when volume+number are known
• maybe link to publication page on CGG website?