About Cross-reenactment and Free-view animation & Video length

[sunshinnnn]

Hi,

I've tried the novel view and novel pose fast demo, it works well. Thank you for the great work!

While as suggested in the Readme, In this quick demo, we do self-reenactment. If you want to do cross-reenactment, you need to transfer the SMPL shape of the source image to the target SMPL parameters., it seems to be not working with the current dataloader for novel pose, could you please also share how to run with cross-reenactment (novel pose + novel identity) and free-view animation (novel pose + novel identity + novel view).

Thanks!

[Lewis-lv0]

Hi, thanks for your kind words @sunshinnnn

cross-reenactment (novel pose + novel identity)

Let's say we'd like to apply assets/demo_images/demo1.png to assets/demo_images/demo1.png. A sample is shown below, where you are expected to do additional things on the places marked with TODO.

PROJECT_ROOT="$PWD"
VIDEO_PATH="assets/demo_videos/demo2.mp4"
VIDEO_DIR=$(dirname "$VIDEO_PATH")
mkdir -p "$VIDEO_DIR/images"
ffmpeg -i "$VIDEO_PATH" -r 12 -c:v png "$VIDEO_DIR/images/%04d.png"

# ------ TODO replace this path with your identity image --------
IMAGE_PATH="assets/demo_images/demo1.png"

# Resize the pose sequence images
python utils/resize.py "$VIDEO_DIR/images"

# Resize the identity image
python utils/resize.py "$IMAGE_PATH"

# Remove background for the identity image
python modules/grounded-sam/segment_human.py \
--img_path "$IMAGE_PATH" \
--grounded_checkpoint pretrained_models/Grounded_SAM/groundingdino_swint_ogc.pth \
--sam_checkpoint pretrained_models/Grounded_SAM/sam_vit_h_4b8939.pth

# Estimate SMPL parameters for the identity image and the pose seq
python scripts/data_processors/smpl/estimate_smpl.py --reference_img_path "$IMAGE_PATH" --driving_video_path "$VIDEO_DIR"


# ------ TODO ------
# Now the files under $VIDEO_DIR/smpl_results are the SMPL of the pose sequence
# Replace all their shape parameters with the shape parameter of the identity.
# You can decide whether you'd like to transfer the source view as well
# Remember to update the resulting vertices as well. Modify in place.
# This code can be helpful https://github.com/fudan-generative-vision/champ/blob/master/scripts/data_processors/smpl/smpl_transfer.py
# -------------

# render SMPL normals
blender scripts/data_processors/smpl/blend/smpl_rendering.blend --background --python scripts/data_processors/smpl/render_smpl_normal_map_np.py -- --driving_path "$VIDEO_DIR"


# ------ TODO ------
# An easy way to make it run is to hardcode the relevent paths for the identity image in modules/sherf/training/np_dataset.py
# 1. https://github.com/humansensinglab/GAS/blob/7788cc0df2b4aca914ada21ac80564463c6c17fa/modules/sherf/training/np_dataset.py#L265 obs_img_path = "assets/demo_images/demo1.png"
# 2. https://github.com/humansensinglab/GAS/blob/7788cc0df2b4aca914ada21ac80564463c6c17fa/modules/sherf/training/np_dataset.py#L276 annot_path = "assets/demo_images/smpl_results/demo1.npy"
# -------------

# render generalizable human nerf
cd modules/sherf
python -u train.py --outdir=logs/tiktok/ --cfg=TikTok --data="$PROJECT_ROOT/$(dirname "$VIDEO_PATH")" \
--gpus=1 --batch=1 --gamma=5 --aug=noaug --neural_rendering_resolution_initial=512 --gen_pose_cond=True --gpc_reg_prob=0.8 --kimg 800 --workers 0 \
--use_1d_feature True --use_2d_feature True --use_3d_feature True --use_sr_module False --sample_obs_view True --fix_obs_view False --use_nerf_decoder True --use_trans True \
--test_flag True \
--resume $PROJECT_ROOT/pretrained_models/sherf/network-snapshot.pkl \
--test_mode pose

# clear logs
rm -r logs



cd ../..

# ------ TODO ------
# configs/inference/novel_pose.yaml
# data.obs_img_path <- identity image path
# -------------

# run diffusion model
CUDA_VISIBLE_DEVICES=0 python inference_novel_pose.py --config configs/inference/novel_pose.yaml

[Lewis-lv0]

free-view animation (novel pose + novel identity + novel view)

Building upon the things above, we can also show the result from a novel view. For in-the-wild cases, the easiest way is to tranfer your novel camera view to the human pose estimation results (the rotation part). In our paper and project page, the cases are from a multi-view video dataset (MVHumanNet), which in nature provides various camera views to transfer.

Note that in our project page we also show a case titled free-view animation, which means both the camera and the human avatar are moving as if we are rendering from a 4d scene. For the diffusion model, we use the 'animation' mode though it's slightly ood.

[Lewis-lv0]

synchronized multi-view video generation

In our project page we also show a few results where we generate synchronized multi-view videos from a single input image. With all the geometry and appearance cues prepared, during the denoising process, we alternate between the novel view mode mv and novel pose mode 'animation' to get the results, as proposed by CAT4D.

Since I do not have access to the server anymore, I have not tested the things above yet
Hope it helps and please let me know if anything is not clear

[sunshinnnn]

Thanks for the detailed demonstration, the suggestion is very helpful!

After editing
https://github.com/humansensinglab/GAS/blob/7788cc0df2b4aca914ada21ac80564463c6c17fa/modules/sherf/training/np_dataset.py#L265 obs_img_path = "assets/demo_images/demo1.png"
https://github.com/humansensinglab/GAS/blob/7788cc0df2b4aca914ada21ac80564463c6c17fa/modules/sherf/training/np_dataset.py#L276 annot_path = "assets/demo_images/smpl_results/demo1.npy" and replacing beta parameters,
I managed to make it work for the cross-reenactment.

It's a very cool and smart strategy to transfrom camera views into the root rotation for the free-view animation, I will try this later.

For synchronized multi-view video generation, forgive me that I can't fully understand how is it implemented. It looks like we need to generate reference_images_pose_nerf and reference_images_pose_smpl arrays, but it's not intuitive how to make it step by step.
Does it implement in this way:

1. Run novel pose or novel view of sherf for all the pose and all the views to get the array pose_num * view_num images.
2. Run probably novel view of diffusion for each pose of frames, or run novel pose of diffusion for each view of frames. (I'm not sure how to do it alternatively with between the novel view mode and pose mode)

For the video length of diffusion, is it implemented in a single feedforward pass to give all the reference smpl and renference nerf. What's the longest video you ever tried?

[Lewis-lv0]

Yes you are right. it's implemented in a single feedforward pass to give all the reference smpl and renference nerf.

1. Run novel pose and novel view of sherf for all the pose and all the views to get the 2d grid array of pose_num * view_num images.
2. Diffusion sampling: I followed the basic idea of CAT4D sampling.

we alternate between multi-view sampling and temporal sampling with SDEdit from the previous iteration and use 25 steps for sampling from random noise and a reduces number of steps when initializing from lower noise levels.

Specifically,
(1) From the random noise, we denoise along the view axis using the novel view mode for 25 steps. By doing so we will get results that are multi-view consistent, but not temporal consistent.
(2) Add noise (noise level 16/25) to the denoised latent, we denoise along the temporal axis using the novel pose mode for 16 steps.
(3) Add noise (noise level 5/25) to the denoised latent, we denoise along the view axis using the novel view mode for 5 steps.

You can try to alternate more times to obtain better results along both dimensions.

In this view axis, I use 20 views, the same as that in the novel view task. In the temporal axis, I really don't remember how many frames I tried (probabaly 10 or 12). As far as I can remember, increasing the video length further will lead to OOM on a 80G gpu.
These will lead to short multi-view video clips like those shown on our project page.

Again, I apologize that I did not release this part when I still have access to the server.. Let me know if you have further questions

[sunshinnnn]

Thanks for the detailed logic illustration! I think I basically begin to understand what happened.