Data preparation¶
Datasets for supported algorithms
Folder structure
AGORA
COCO
COCO-WholeBody
CrowdPose
EFT
GTA-Human
Human3.6M
Human3.6M Mosh
HybrIK
LSP
LSPET
MPI-INF-3DHP
MPII
PoseTrack18
Penn Action
PW3D
SPIN
SURREAL
Overview¶
Our data pipeline use HumanData structure for storing and loading. The proprocessed npz files can be obtained from raw data using our data converters, and the supported configs can be found here.
These are our supported converters and their respective dataset-name:
AgoraConverter (
agora)AmassConverter (
amass)CocoConverter (
coco)CocoHybrIKConverter (
coco_hybrik)CocoWholebodyConverter (
coco_wholebody)CrowdposeConverter (
crowdpose)EftConverter (
eft)GTAHumanConverter (
gta_human)H36mConverter (
h36m_p1,h36m_p2)H36mHybrIKConverter (
h36m_hybrik)InstaVibeConverter (
instavariety_vibe)LspExtendedConverter (
lsp_extended)LspConverter (
lsp_original,lsp_dataset)MpiiConverter (
mpii)MpiInf3dhpConverter (
mpi_inf_3dhp)MpiInf3dhpHybrIKConverter (
mpi_inf_3dhp_hybrik)PennActionConverter (
penn_action)PosetrackConverter (
posetrack)Pw3dConverter (
pw3d)Pw3dHybrIKConverter (
pw3d_hybrik)SurrealConverter (
surreal)SpinConverter (
spin)Up3dConverter (
up3d)
Datasets for supported algorithms¶
For all algorithms, the root path for our datasets and output path for our preprocessed npz files are stored in data/datasets and data/preprocessed_datasets. As such, use this command with the listed dataset-names:
python tools/convert_datasets.py \
--datasets <dataset-name> \
--root_path data/datasets \
--output_path data/preprocessed_datasets
For HMR training and testing, the following datasets are required:
COCO
Human3.6M
Human3.6M Mosh
MPI-INF-3DHP
MPII
LSP
LSPET
PW3D
Convert datasets with the following dataset-names:
coco, pw3d, mpii, mpi_inf_3dhp, lsp_original, lsp_extended, h36m
Alternatively, you may download the preprocessed files directly:
Unfortunately, we are unable to distribute h36m_mosh_train.npz due to license limitations. However, we provide the
conversion tools should you possess the raw mosh data. Prefer refer to Human3.6M Mosh on details for conversion.
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── coco_2014_train.npz
├── h36m_train.npz or h36m_mosh_train.npz (if mosh is available)
├── lspet_train.npz
├── lsp_train.npz
├── mpi_inf_3dhp_train.npz
├── mpii_train.npz
└── pw3d_test.npz
For SPIN training, the following datasets are required:
COCO
Human3.6M
Human3.6M Mosh
MPI-INF-3DHP
MPII
LSP
LSPET
PW3D
SPIN
Convert datasets with the following dataset-names:
spin, h36m
Alternatively, you may download the preprocessed files directly:
Unfortunately, we are unable to distribute h36m_mosh_train.npz due to license limitations. However, we provide the
conversion tools should you posses the raw mosh data. Prefer refer to Human3.6M Mosh on details for conversion.
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── spin_coco_2014_train.npz
├── h36m_train.npz or h36m_mosh_train.npz (if mosh is available)
├── spin_lsp_train.npz
├── spin_lspet_train.npz
├── spin_mpi_inf_3dhp_train.npz
├── spin_mpii_train.npz
└── spin_pw3d_test.npz
For VIBE training and testing, the following datasets are required:
MPI-INF-3DHP
PW3D
The data converters are currently not available.
Alternatively, you may download the preprocessed files directly:
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── vibe_insta_variety.npz
├── vibe_mpi_inf_3dhp_train.npz
└── vibe_pw3d_test.npz
For HYBRIK training and testing, the following datasets are required:
HybrIK
COCO
Human3.6M
MPI-INF-3DHP
PW3D
Convert datasets with the following dataset-names:
h36m_hybrik, pw3d_hybrik, mpi_inf_3dhp_hybrik, coco_hybrik
Alternatively, you may download the preprocessed files directly:
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── hybriK_coco_2017_train.npz
├── hybrik_h36m_train.npz
├── hybrik_mpi_inf_3dhp_train.npz
└── hybrik_pw3d_test.npz
For PARE training, the following datasets are required:
Human3.6M
Human3.6M Mosh
MPI-INF-3DHP
EFT-COCO
EFT-MPII
EFT-LSPET
PW3D
Convert datasets with the following dataset-names:
h36m, coco, mpii, lspet, mpi-inf-3dhp, pw3d
Alternatively, you may download the preprocessed files directly:
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── h36m_mosh_train.npz
├── h36m_train.npz
├── mpi_inf_3dhp_train.npz
├── eft_mpii.npz
├── eft_lspet.npz
├── eft_coco_all.npz
└── pw3d_test.npz
For ExPose training, the following datasets are required:
Human3.6M
FreiHand
EHF
FFHQ
ExPose-Curated-fits
SPIN_SMPLX
Stirling-ESRC3D
PW3D
Convert datasets with the following dataset-names:
h36m, EHF, FreiHand, 3DPW, stirling, spin_in_smplx, ffhq, ExPose_curated_fits
Alternatively, you may download the preprocessed files directly:
The preprocessed datasets should have this structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── datasets
└── preprocessed_datasets
├── curated_fits_train.npz
├── ehf_val.npz
├── ffhq_flame_train.npz
├── freihand_test.npz
├── freihand_train.npz
├── freihand_val.npz
├── h36m_smplx_train.npz
├── pw3d_test.npz
├── spin_smplx_train.npz
└── stirling_ESRC3D_HQ.npz
Folder structure¶
AGORA¶
AGORA (CVPR'2021)
@inproceedings{Patel:CVPR:2021,
title = {{AGORA}: Avatars in Geography Optimized for Regression Analysis},
author = {Patel, Priyanka and Huang, Chun-Hao P. and Tesch, Joachim and Hoffmann, David T. and Tripathi, Shashank and Black, Michael J.},
booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition ({CVPR})},
month = jun,
year = {2021},
month_numeric = {6}
}
For AGORA, please download the dataset and place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── agora
├── camera_dataframe # smplx annotations
│ ├── train_0_withjv.pkl
│ ├── validation_0_withjv.pkl
│ └── ...
├── camera_dataframe_smpl # smpl annotations
│ ├── train_0_withjv.pkl
│ ├── validation_0_withjv.pkl
│ └── ...
├── images
│ ├── train
│ │ ├── ag_trainset_3dpeople_bfh_archviz_5_10_cam00_00000_1280x720.png
│ │ ├── ag_trainset_3dpeople_bfh_archviz_5_10_cam00_00001_1280x720.png
│ │ └── ...
│ ├── validation
│ └── test
├── smpl_gt
│ ├── trainset_3dpeople_adults_bfh
│ │ ├── 10004_w_Amaya_0_0.mtl
│ │ ├── 10004_w_Amaya_0_0.obj
│ │ ├── 10004_w_Amaya_0_0.pkl
│ │ └── ...
│ └── ...
└── smplx_gt
├── 10004_w_Amaya_0_0.obj
├── 10004_w_Amaya_0_0.pkl
└── ...
AMASS¶
AMASS (ICCV'2019)
@inproceedings{AMASS:2019,
title={AMASS: Archive of Motion Capture as Surface Shapes},
author={Mahmood, Naureen and Ghorbani, Nima and F. Troje, Nikolaus and Pons-Moll, Gerard and Black, Michael J.},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
year={2019},
month = {Oct},
url = {https://amass.is.tue.mpg.de},
month_numeric = {10}
}
Details for direct preprocessing will be added in the future.
Alternatively, you may download the preprocessed files directly:
COCO¶
COCO (ECCV'2014)
@inproceedings{lin2014microsoft,
title={Microsoft coco: Common objects in context},
author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
booktitle={European conference on computer vision},
pages={740--755},
year={2014},
organization={Springer}
}
For COCO data, please download from COCO download. COCO’2014 Train is needed for HMR training and COCO’2017 Train is needed for HybrIK trainig.
Download and extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── coco
├── annotations
| ├── person_keypoints_train2014.json
| ├── person_keypoints_val2014.json
├── train2014
│ ├── COCO_train2014_000000000009.jpg
│ ├── COCO_train2014_000000000025.jpg
│ ├── COCO_train2014_000000000030.jpg
| └── ...
└── train_2017
│── annotations
│ ├── person_keypoints_train2017.json
│ └── person_keypoints_val2017.json
│── train2017
│ ├── 000000000009.jpg
│ ├── 000000000025.jpg
│ ├── 000000000030.jpg
│ └── ...
└── val2017
├── 000000000139.jpg
├── 000000000285.jpg
├── 000000000632.jpg
└── ...
COCO-WholeBody¶
COCO-WholeBody (ECCV'2020)
@inproceedings{jin2020whole,
title={Whole-Body Human Pose Estimation in the Wild},
author={Jin, Sheng and Xu, Lumin and Xu, Jin and Wang, Can and Liu, Wentao and Qian, Chen and Ouyang, Wanli and Luo, Ping},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2020}
}
For COCO-WholeBody dataset, images can be downloaded from COCO download, 2017 Train/Val is needed for COCO keypoints training and validation.
Download and extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── coco
├── annotations
| ├── coco_wholebody_train_v1.0.json
| └── coco_wholebody_val_v1.0.json
└── train_2017
│── train2017
│ ├── 000000000009.jpg
│ ├── 000000000025.jpg
│ ├── 000000000030.jpg
│ └── ...
└── val2017
├── 000000000139.jpg
├── 000000000285.jpg
├── 000000000632.jpg
└── ...
CrowdPose¶
CrowdPose (CVPR'2019)
@article{li2018crowdpose,
title={CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark},
author={Li, Jiefeng and Wang, Can and Zhu, Hao and Mao, Yihuan and Fang, Hao-Shu and Lu, Cewu},
journal={arXiv preprint arXiv:1812.00324},
year={2018}
}
For CrowdPose data, please download from CrowdPose.
Download and extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── crowdpose
├── crowdpose_train.json
├── crowdpose_val.json
├── crowdpose_trainval.json
├── crowdpose_test.json
└── images
├── 100000.jpg
├── 100001.jpg
├── 100002.jpg
└── ...
EFT¶
EFT (3DV'2021)
@inproceedings{joo2020eft,
title={Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation},
author={Joo, Hanbyul and Neverova, Natalia and Vedaldi, Andrea},
booktitle={3DV},
year={2020}
}
For EFT data, please download from EFT.
Download and extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── eft
├── coco_2014_train_fit
| ├── COCO2014-All-ver01.json
| └── COCO2014-Part-ver01.json
|── LSPet_fit
| └── LSPet_ver01.json
└── MPII_fit
└── MPII_ver01.json
GTA-Human¶
GTA-Human (arXiv'2021)
@article{cai2021playing,
title={Playing for 3D Human Recovery},
author={Cai, Zhongang and Zhang, Mingyuan and Ren, Jiawei and Wei, Chen and Ren, Daxuan and Li, Jiatong and Lin, Zhengyu and Zhao, Haiyu and Yi, Shuai and Yang, Lei and others},
journal={arXiv preprint arXiv:2110.07588},
year={2021}
}
More details are coming soon!
Human3.6M¶
Human3.6M (TPAMI'2014)
@article{h36m_pami,
author = {Ionescu, Catalin and Papava, Dragos and Olaru, Vlad and Sminchisescu, Cristian},
title = {Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
publisher = {IEEE Computer Society},
volume = {36},
number = {7},
pages = {1325-1339},
month = {jul},
year = {2014}
}
For Human3.6M, please download from the official website and run the preprocessing script, which will extract pose annotations at downsampled framerate (10 FPS). The processed data should have the following structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── h36m
├── annot
├── S1
| ├── images
| | |── S1_Directions_1.54138969
| | | ├── S1_Directions_1.54138969_00001.jpg
| | | ├── S1_Directions_1.54138969_00006.jpg
| | | └── ...
| | └── ...
| ├── MyPoseFeatures
| | |── D2Positions
| | └── D3_Positions_Mono
| ├── MySegmentsMat
| | └── ground_truth_bs
| └── Videos
| |── Directions 1.54138969.mp4
| |── Directions 1.55011271.mp4
| └── ...
├── S5
├── S6
├── S7
├── S8
├── S9
├── S11
└── metadata.xml
To extract images from Human3.6M original videos, modify the h36m_p1 config in DATASET_CONFIG:
h36m_p1=dict(
type='H36mConverter',
modes=['train', 'valid'],
protocol=1,
extract_img=True, # set to true to extract images from raw videos
prefix='h36m'),
Human3.6M Mosh¶
For data preparation of Human3.6M for HMR, SPIN and PARE training, we use the MoShed data provided in HMR for training. However, due to license limitations, we are not allowed to redistribute the data. Even if you do not have access to these parameters, you can still generate the preprocessed h36m npz file without mosh parameters using our converter.
You will need to extract images from raw videos for training. Do note that preprocessing can take a long time if image extraction is required. To do so, modify the h36m_p1 config in DATASET_CONFIG:
Config without mosh:
h36m_p1=dict(
type='H36mConverter',
modes=['train', 'valid'],
protocol=1,
extract_img=True, # this is to specify you want to extract images from videos
prefix='h36m'),
Config with mosh:
h36m_p1=dict(
type='H36mConverter',
modes=['train', 'valid'],
protocol=1,
extract_img=True, # this is to specify you want to extract images from videos
mosh_dir='data/datasets/h36m_mosh', # supply the directory to the mosh if available
prefix='h36m'),
If you have MoShed data available, it should have the following structure:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── h36m_mosh
├── annot
├── S1
| ├── images
| | ├── Directions 1_cam0_aligned.pkl
| | ├── Directions 1_cam1_aligned.pkl
| | └── ...
├── S5
├── S6
├── S7
├── S8
├── S9
└── S11
HybrIK¶
HybrIK (CVPR'2021)
@inproceedings{li2020hybrikg,
author = {Li, Jiefeng and Xu, Chao and Chen, Zhicun and Bian, Siyuan and Yang, Lixin and Lu, Cewu},
title = {HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation},
booktitle={CVPR 2021},
pages={3383--3393},
year={2021},
organization={IEEE}
}
For HybrIK, please download the parsed json annotation files and place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── hybrik_data
├── Sample_5_train_Human36M_smpl_leaf_twist_protocol_2.json
├── Sample_20_test_Human36M_smpl_protocol_2.json
├── 3DPW_test_new.json
├── annotation_mpi_inf_3dhp_train_v2.json
└── annotation_mpi_inf_3dhp_test.json
To convert the preprocessed json files into npz files used for our pipeline, run the following preprocessing scripts:
LSP¶
LSP (BMVC'2010)
@inproceedings{johnson2010clustered,
title={Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation.},
author={Johnson, Sam and Everingham, Mark},
booktitle={bmvc},
volume={2},
number={4},
pages={5},
year={2010},
organization={Citeseer}
}
For LSP, please download the high resolution version
LSP dataset original.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── lsp
├── images
| ├── im0001.jpg
| ├── im0002.jpg
| └── ...
└── joints.mat
LSPET¶
LSP-Extended (CVPR'2011)
@inproceedings{johnson2011learning,
title={Learning effective human pose estimation from inaccurate annotation},
author={Johnson, Sam and Everingham, Mark},
booktitle={CVPR 2011},
pages={1465--1472},
year={2011},
organization={IEEE}
}
For LSPET, please download its high resolution form
HR-LSPET.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── lspet
├── im00001.jpg
├── im00002.jpg
├── im00003.jpg
├── ...
└── joints.mat
MPI-INF-3DHP¶
MPI_INF_3DHP (3DV'2017)
@inproceedings{mono-3dhp2017,
author = {Mehta, Dushyant and Rhodin, Helge and Casas, Dan and Fua, Pascal and Sotnychenko, Oleksandr and Xu, Weipeng and Theobalt, Christian},
title = {Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision},
booktitle = {3D Vision (3DV), 2017 Fifth International Conference on},
url = {http://gvv.mpi-inf.mpg.de/3dhp_dataset},
year = {2017},
organization={IEEE},
doi={10.1109/3dv.2017.00064},
}
You will need to extract images from raw videos for training. Do note that preprocessing can take a long time if image extraction is required. To do so, modify the mpi_inf_3dhp config in DATASET_CONFIG:
Config:
mpi_inf_3dhp=dict(
type='MpiInf3dhpConverter',
modes=['train', 'test'],
extract_img=True), # this is to specify you want to extract images from videos
For MPI-INF-3DHP, download and extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── mpi_inf_3dhp
├── mpi_inf_3dhp_test_set
│ ├── TS1
│ ├── TS2
│ ├── TS3
│ ├── TS4
│ ├── TS5
│ └── TS6
├── S1
│ ├── Seq1
│ └── Seq2
├── S2
│ ├── Seq1
│ └── Seq2
├── S3
│ ├── Seq1
│ └── Seq2
├── S4
│ ├── Seq1
│ └── Seq2
├── S5
│ ├── Seq1
│ └── Seq2
├── S6
│ ├── Seq1
│ └── Seq2
├── S7
│ ├── Seq1
│ └── Seq2
└── S8
├── Seq1
└── Seq2
MPII¶
MPII (CVPR'2014)
@inproceedings{andriluka14cvpr,
author = {Mykhaylo Andriluka and Leonid Pishchulin and Peter Gehler and Schiele, Bernt},
title = {2D Human Pose Estimation: New Benchmark and State of the Art Analysis},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2014},
month = {June}
}
For MPII data, please download images from [MPII Human Pose Dataset](http://human-pose.mpi-inf.mpg.de/ and annotations from here.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── mpii
|── train.h5
└── images
|── 000001163.jpg
|── 000003072.jpg
└── ...
PoseTrack18¶
PoseTrack18 (CVPR'2018)
@inproceedings{andriluka2018posetrack,
title={Posetrack: A benchmark for human pose estimation and tracking},
author={Andriluka, Mykhaylo and Iqbal, Umar and Insafutdinov, Eldar and Pishchulin, Leonid and Milan, Anton and Gall, Juergen and Schiele, Bernt},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={5167--5176},
year={2018}
}
For PoseTrack18 data, please download from PoseTrack18.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── posetrack
├── images
│ ├── train
│ │ ├── 000001_bonn_train
│ │ │ ├── 000000.jpg
│ │ │ ├── 000001.jpg
│ │ │ └── ...
│ │ └── ...
│ ├── val
│ │ ├── 000342_mpii_test
│ │ │ ├── 000000.jpg
│ │ │ ├── 000001.jpg
│ │ │ └── ...
│ │ └── ...
│ └── test
│ ├── 000001_mpiinew_test
│ │ ├── 000000.jpg
│ │ ├── 000001.jpg
│ │ └── ...
│ └── ...
└── posetrack_data
└── annotations
├── train
│ ├── 000001_bonn_train.json
│ ├── 000002_bonn_train.json
│ └── ...
├── val
│ ├── 000342_mpii_test.json
│ ├── 000522_mpii_test.json
│ └── ...
└── test
├── 000001_mpiinew_test.json
├── 000002_mpiinew_test.json
└── ...
Penn Action¶
Penn Action (ICCV'2013)
@inproceedings{zhang2013pennaction,
title={From Actemes to Action: A Strongly-supervised Representation for Detailed Action Understanding},
author={Zhang, Weiyu and Zhu, Menglong and Derpanis, Konstantinos},
booktitle={ICCV},
year={2013}
}
For Penn Action data, please download from Penn Action.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── penn_action
├── frames
│ ├── 0001
│ │ ├── 000001.jpg
│ │ ├── 000002.jpg
│ │ └── ...
│ └── ...
└── labels
├── 0001.mat
├── 0002.mat
└── ...
PW3D¶
PW3D (ECCV'2018)
@inproceedings{vonMarcard2018,
title = {Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera},
author = {von Marcard, Timo and Henschel, Roberto and Black, Michael and Rosenhahn, Bodo and Pons-Moll, Gerard},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2018},
month = {sep}
}
For PW3D data, please download from PW3D Dataset.
Extract them under $MMHUMAN3D/data/datasets, and make them look like this:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── pw3d
|── imageFiles
| | └── courtyard_arguing_00
| | ├── image_00000.jpg
| | ├── image_00001.jpg
| | └── ...
└── sequenceFiles
├── train
│ ├── downtown_arguing_00.pkl
│ └── ...
├── val
│ ├── courtyard_arguing_00.pkl
│ └── ...
└── test
├── courtyard_basketball_00.pkl
└── ...
SPIN¶
SPIN (ICCV'2019)
@inproceedings{kolotouros2019spin,
author = {Kolotouros, Nikos and Pavlakos, Georgios and Black, Michael J and Daniilidis, Kostas},
title = {Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop},
booktitle={ICCV},
year={2019}
}
For SPIN, please download the preprocessed npz files and place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── spin_data
├── coco_2014_train.npz
├── hr-lspet_train.npz
├── lsp_dataset_original_train.npz
├── mpi_inf_3dhp_train.npz
└── mpii_train.npz
SURREAL¶
SURREAL (CVPR'2017)
@inproceedings{varol17_surreal,
title = {Learning from Synthetic Humans},
author = {Varol, G{\"u}l and Romero, Javier and Martin, Xavier and Mahmood, Naureen and Black, Michael J. and Laptev, Ivan and Schmid, Cordelia},
booktitle = {CVPR},
year = {2017}
}
For SURREAL, please download the [dataset] (https://www.di.ens.fr/willow/research/surreal/data/) and place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── surreal
├── train
│ ├── run0
| | ├── 03_01
| | │ ├── 03_01_c0001_depth.mat
| | │ ├── 03_01_c0001_info.mat
| | │ ├── 03_01_c0001_segm.mat
| | │ ├── 03_01_c0001.mp4
| | │ └── ...
| | └── ...
│ ├── run1
│ └── run2
├── val
│ ├── run0
│ ├── run1
│ └── run2
└── test
├── run0
├── run1
└── run2
VIBE¶
VIBE (CVPR'2020)
@inproceedings{VIBE,
author = {Muhammed Kocabas and
Nikos Athanasiou and
Michael J. Black},
title = {{VIBE}: Video Inference for Human Body Pose and Shape Estimation},
booktitle = {CVPR},
year = {2020}
}
For VIBE, please download the preprocessed mpi_inf_3dhp and pw3d npz files from SPIN and pretrained frame feature extractor spin.pth. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
├── checkpoints
| └── spin.pth
└── datasets
└── vibe_data
├── mpi_inf_3dhp_train.npz
└── pw3d_test.npz
FreiHand¶
FreiHand (ICCV'2019)
@inproceedings{zimmermann2019freihand,
title={Freihand: A dataset for markerless capture of hand pose and shape from single rgb images},
author={Zimmermann, Christian and Ceylan, Duygu and Yang, Jimei and Russell, Bryan and Argus, Max and Brox, Thomas},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={813--822},
year={2019}
}
For FreiHand data, please download from FreiHand Dataset.
Extract them under $MMHUMAN3D/data/datasets. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── FreiHand
|── evaluation
| └── rgb
| ├── 00000000.jpg
| ├── 00000001.jpg
| └── ...
|── training
| └── rgb
| ├── 00000000.jpg
| ├── 00000001.jpg
| └── ...
|── evaluation_K.json
|── evaluation_mano.json
|── evaluation_scale.json
|── evaluation_verts.json
|── evaluation_xyz.json
|── training_K.json
|── training_mano.json
|── training_scale.json
|── training_verts.json
└── training_xyz.json
EHF¶
SMPLX (CVPR'2019)
@inproceedings{SMPL-X:2019,
title = {Expressive Body Capture: {3D} Hands, Face, and Body from a Single Image},
author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
pages = {10975--10985},
year = {2019}
}
For EHF data, please download from EHF Dataset.
Extract them under $MMHUMAN3D/data/datasets. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── EHF
|── 01_2Djnt.json
|── 01_2Djnt.png
|── 01_align.ply
|── 01_img.jpg
|── 01_img.png
|── 01_scan.obj
└── ...
FFHQ¶
FFHQ (CVPR'2019)
@inproceedings{karras2019style,
title={A style-based generator architecture for generative adversarial networks},
author={Karras, Tero and Laine, Samuli and Aila, Timo},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={4401--4410},
year={2019}
}
For FFHQ data, please download from FFHQ Dataset.
We present ffhq_annotations.npz by running RingNet on FFHQ and then fitting to FAN 2D landmarks by flame-fitting.
Extract them under $MMHUMAN3D/data/datasets. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── FFHQ
|── ffhq_global_images_1024
| ├── 00000.png
| ├── 00001.png
| └── ...
└── ffhq_annotations.npz
ExPose¶
ExPose (ECCV'2020)
@inproceedings{ExPose:2020,
title = {Monocular Expressive Body Regression through Body-Driven Attention},
author = {Choutas, Vasileios and Pavlakos, Georgios and Bolkart, Timo and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {European Conference on Computer Vision (ECCV)},
pages = {20--40},
year = {2020},
url = {https://expose.is.tue.mpg.de}
}
For ExPose data, please download from Curated Fits Dataset and SPIN IN SMPLX Dataset.
Extract them under $MMHUMAN3D/data/datasets. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
|── ExPose_curated_fits
| |── train.npz
| └── val.npz
└── spin_in_smplx
|── coco.npz
|── lsp.npz
|── lspet.npz
└── mpii.npz
Stirling¶
Stirling ESRC3D Face (FG'2018)
@inproceedings{feng2018evaluation,
title={Evaluation of dense 3D reconstruction from 2D face images in the wild},
author={Feng, Zhen-Hua and Huber, Patrik and Kittler, Josef and Hancock, Peter and Wu, Xiao-Jun and Zhao, Qijun and Koppen, Paul and R{\"a}tsch, Matthias},
booktitle={2018 13th IEEE International Conference on Automatic Face \& Gesture Recognition (FG 2018)},
pages={780--786},
year={2018},
organization={IEEE}
}
For Stirling ESRC3D Face data, please download from Stirling ESRC3D Face Dataset.
Extract them under $MMHUMAN3D/data/datasets. Place them in the folder structure below:
mmhuman3d
├── mmhuman3d
├── docs
├── tests
├── tools
├── configs
└── data
└── datasets
└── stirling
|── annotations
| ├── F_3D_N
| | ├── F1001_N.lnd
| | ├── F1002_N.lnd
| | └── ...
| └── M_3D_N
| ├── M1001_N.lnd
| ├── M1002_N.lnd
| └── ...
|── F_3D_N
| ├── F1001_N.obj
| ├── F1002_N.obj
| └── ...
|── M_3D_N
| ├── M1001_N.obj
| ├── M1002_N.obj
| └── ...
└── Subset_2D_FG2018
├── HQ
| ├── F1001_001.jpg
| ├── F1001_002.jpg
| └── ...
└── LQ
├── F1001_008.jpg
├── F1001_009.jpg
└── ...