Sync2Gen
Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization
0. Environment
Environment: python 3.6 and cuda 10.0 on Ubuntu 18.04
- Pytorch 1.4.0
- tensorflow 1.14.0 (for tensorboard)
1. Dataset
├──dataset_3dfront/
├──data
├── bedroom
├── 0_abs.npy
├── 0_rel.pkl
├── ...
├── living
├── 0_abs.npy
├── 0_rel.pkl
├── ...
├── train_bedroom.txt
├── train_living.txt
├── val_bedroom.txt
└── val_living.txt
See 3D-FRONT Dataset for dataset generation.
2. VAE
2.1 Generate scenes from random noises
Download the pretrained model from https://drive.google.com/file/d/1VKNlEdUj1RBUOjBaBxE5xQvfsZodVjam/view?usp=sharing
Sync2Gen
└── log
└── 3dfront
├── bedroom
│ └── vaef_lr0001_w00001_B64
│ ├── checkpoint_eval799.tar
│ └── pairs
└── living
└── vaef_lr0001_w00001_B64
├── checkpoint_eval799.tar
└── pairs
type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./test_sparse.py --type $type --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=32 --variational --latent_dim 20 --abs_dim 16 --weight_kld 0.0001 --learning_rate 0.001 --use_dumped_pairs --dump_results --gen_from_noise --num_gen_from_noise 100
The predictions are dumped in ./dump/$type/vaef_lr0001_w00001_B64
2.2 Training
To train the network:
type='bedroom'; # or living
CUDA_VISIBLE_DEVICES=0 python ./train_sparse.py --data_path ./dataset_3dfront/data --type $type --log_dir ./log/3dfront/$type/vaef_lr0001_w00001_B64 --model_dict=model_scene_forward --max_parts=80 --num_class=20 --num_each_class=4 --batch_size=64 --variational --latent_dim 20 --abs_dim 16 --weight_kld 0.0001 --learning_rate 0.001
3. Bayesian optimization
cd optimization
3.1 Prior generation
See Prior generation.
3.2 Optimization
type=bedroom # or living;
bash opt.sh $type vaef_lr0001_w00001_B64 EXP_NAME
We use Pytorch-LBFGS for optimization.
3.3 Visualization
There is a simple visualization tool:
type=bedroom # or living
bash vis.sh $type vaef_lr0001_w00001_B64 EXP_NAME
The visualization is in ./vis. {i:04d}_2(3)d_pred.png is the initial prediction from VAE. {i:04d}_2(3)d_sync.png is the optimized layout after synchronization.
Acknowledgements
The repo is built based on:
We thank the authors for their great job.
Contact
If you have any questions, you can contact Haitao Yang (yanghtr [AT] outlook [DOT] com).
330 Jan 07, 2023
50 Nov 26, 2022
96 Dec 27, 2022
801 Jan 08, 2023
51 Nov 20, 2022
1 Nov 09, 2021
5 Nov 07, 2022
63 Jan 02, 2023
94 Dec 11, 2022
292 Dec 30, 2022
162 Nov 28, 2022
2 Jan 05, 2022
36 Dec 16, 2022
838 Jan 01, 2023
3 Oct 17, 2022
2.6k Jan 08, 2023
2 Mar 24, 2022
83 Oct 10, 2022
2.5k Jan 05, 2023
69 Dec 26, 2022