Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Update dataset mim downloading source to OpenXLab #2614

Merged
merged 3 commits into from
Aug 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions dataset-index.yml
Original file line number Diff line number Diff line change
@@ -1,71 +1,72 @@
openxlab: true
coco2017:
dataset: COCO_2017
dataset: OpenDataLab/COCO_2017
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_coco2017.sh

mpii:
dataset: MPII_Human_Pose
dataset: OpenDataLab/MPII_Human_Pose
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_mpii.sh

aic:
dataset: AI_Challenger
dataset: OpenDataLab/AI_Challenger
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_aic.sh

crowdpose:
dataset: CrowdPose
dataset: OpenDataLab/CrowdPose
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_crowdpose.sh

halpe:
dataset: Halpe
dataset: OpenDataLab/Halpe
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_halpe.sh

lapa:
dataset: LaPa
dataset: OpenDataLab/LaPa
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_lapa.sh

300w:
dataset: 300w
dataset: OpenDataLab/300w
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_300w.sh

wflw:
dataset: WFLW
dataset: OpenDataLab/WFLW
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_wflw.sh

onehand10k:
dataset: OneHand10K
dataset: OpenDataLab/OneHand10K
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_onehand10k.sh

freihand:
dataset: FreiHAND
dataset: OpenDataLab/FreiHAND
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_freihand.sh

ap10k:
dataset: AP-10K
dataset: OpenDataLab/AP-10K
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_ap10k.sh

hagrid:
dataset: HaGRID
dataset: OpenDataLab/HaGRID
download_root: data
data_root: data/pose
script: tools/dataset_converters/scripts/preprocess_hagrid.sh
10 changes: 5 additions & 5 deletions docs/en/user_guides/prepare_datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,18 +158,18 @@ The heatmap target will be visualized together if it is generated in the pipelin

## Download dataset via MIM

By using [OpenDataLab](https://opendatalab.com/), you can obtain free formatted datasets in various fields. Through the search function of the platform, you may address the dataset they look for quickly and easily. Using the formatted datasets from the platform, you can efficiently conduct tasks across datasets.
By using [OpenXLab](https://openxlab.org.cn/datasets), you can obtain free formatted datasets in various fields. Through the search function of the platform, you may address the dataset they look for quickly and easily. Using the formatted datasets from the platform, you can efficiently conduct tasks across datasets.

If you use MIM to download, make sure that the version is greater than v0.3.8. You can use the following command to update, install, login and download the dataset:

```shell
# upgrade your MIM
pip install -U openmim

# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab, registry
odl login
# install OpenXLab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login

# download coco2017 and preprocess by MIM
mim download mmpose --dataset coco2017
Expand Down
10 changes: 5 additions & 5 deletions docs/zh_cn/user_guides/prepare_datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,18 +158,18 @@ python tools/misc/browse_dataset.py configs/body_2d_keypoint/topdown_heatmap/coc

## 用 MIM 下载数据集

通过使用 [OpenDataLab](https://opendatalab.com/),您可以直接下载开源数据集。通过平台的搜索功能,您可以快速轻松地找到他们正在寻找的数据集。使用平台上的格式化数据集,您可以高效地跨数据集执行任务。
通过使用 [OpenXLab](https://openxlab.org.cn/datasets),您可以直接下载开源数据集。通过平台的搜索功能,您可以快速轻松地找到他们正在寻找的数据集。使用平台上的格式化数据集,您可以高效地跨数据集执行任务。

如果您使用 MIM 下载,请确保版本大于 v0.3.8。您可以使用以下命令进行更新、安装、登录和数据集下载:

```shell
# upgrade your MIM
pip install -U openmim

# install OpenDataLab CLI tools
pip install -U opendatalab
# log in OpenDataLab, registry
odl login
# install OpenXLab CLI tools
pip install -U openxlab
# log in OpenXLab
openxlab login

# download coco2017 and preprocess by MIM
mim download mmpose --dataset coco2017
Expand Down
2 changes: 2 additions & 0 deletions model-index.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,3 +125,5 @@ Import:
- configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/vipnas_dark_coco-wholebody.yml
- configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/cspnext_udp_coco-wholebody.yml
- configs/fashion_2d_keypoint/topdown_heatmap/deepfashion2/res50_deepfasion2.yml
- configs/fashion_2d_keypoint/topdown_heatmap/deepfashion/hrnet_deepfashion.yml
- configs/fashion_2d_keypoint/topdown_heatmap/deepfashion/resnet_deepfashion.yml
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_300w.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/300w/raw/300w.tar.gz.00 -C $DOWNLOAD_DIR/
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___300w/raw/300w.tar.gz.00 -C $DOWNLOAD_DIR/
tar -xvf $DOWNLOAD_DIR/300w/300w.tar.00 -C $DATA_ROOT/
rm -rf $DOWNLOAD_DIR/300w
rm -rf $DOWNLOAD_DIR/300w $DOWNLOAD_DIR/OpenDataLab___300w
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_aic.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/AI_Challenger/raw/AI_Challenger.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/AI_Challenger
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___AI_Challenger/raw/AI_Challenger.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___AI_Challenger
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_ap10k.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/AP-10K/raw/AP-10K.tar.gz.00 -C $DOWNLOAD_DIR/
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___AP-10K/raw/AP-10K.tar.gz.00 -C $DOWNLOAD_DIR/
tar -xvf $DOWNLOAD_DIR/AP-10K/AP-10K.tar.00 -C $DATA_ROOT/
rm -rf $DOWNLOAD_DIR/AP-10K
rm -rf $DOWNLOAD_DIR/AP-10K $DOWNLOAD_DIR/OpenDataLab___AP-10K
8 changes: 4 additions & 4 deletions tools/dataset_converters/scripts/preprocess_coco2017.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

unzip $DOWNLOAD_DIR/COCO_2017/raw/Images/val2017.zip -d $DATA_ROOT
unzip $DOWNLOAD_DIR/COCO_2017/raw/Images/train2017.zip -d $DATA_ROOT
unzip $DOWNLOAD_DIR/COCO_2017/raw/Annotations/annotations_trainval2017.zip -d $DATA_ROOT
rm -rf $DOWNLOAD_DIR/COCO_2017
unzip $DOWNLOAD_DIR/OpenDataLab___COCO_2017/raw/Images/val2017.zip -d $DATA_ROOT
unzip $DOWNLOAD_DIR/OpenDataLab___COCO_2017/raw/Images/train2017.zip -d $DATA_ROOT
unzip $DOWNLOAD_DIR/OpenDataLab___COCO_2017/raw/Annotations/annotations_trainval2017.zip -d $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___COCO_2017
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_crowdpose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/CrowdPose/raw/CrowdPose.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/CrowdPose
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___CrowdPose/raw/CrowdPose.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___CrowdPose
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_freihand.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/FreiHAND/raw/FreiHAND.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/FreiHAND
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___FreiHAND/raw/FreiHAND.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___FreiHAND
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_hagrid.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

cat $DOWNLOAD_DIR/HaGRID/raw/*.tar.gz.* | tar -xvz -C $DATA_ROOT/..
cat $DOWNLOAD_DIR/OpenDataLab___HaGRID/raw/*.tar.gz.* | tar -xvz -C $DATA_ROOT/..
tar -xvf $DATA_ROOT/HaGRID.tar -C $DATA_ROOT/..
rm -rf $DOWNLOAD_DIR/HaGRID
rm -rf $DOWNLOAD_DIR/OpenDataLab___HaGRID
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_halpe.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/Halpe/raw/Halpe.tar.gz.00 -C $DOWNLOAD_DIR/
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___Halpe/raw/Halpe.tar.gz.00 -C $DOWNLOAD_DIR/
tar -xvf $DOWNLOAD_DIR/Halpe/Halpe.tar.00 -C $DATA_ROOT/
rm -rf $DOWNLOAD_DIR/Halpe
rm -rf $DOWNLOAD_DIR/Halpe $DOWNLOAD_DIR/OpenDataLab___Halpe
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_lapa.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/LaPa/raw/LaPa.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/LaPa
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___LaPa/raw/LaPa.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___LaPa
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_mpii.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/MPII_Human_Pose/raw/MPII_Human_Pose.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/MPII_Human_Pose
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___MPII_Human_Pose/raw/MPII_Human_Pose.tar.gz -C $DATA_ROOT
rm -rf $DOWNLOAD_DIR/OpenDataLab___MPII_Human_Pose
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_onehand10k.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/OneHand10K/raw/OneHand10K.tar.gz.00 -C $DOWNLOAD_DIR/
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___OneHand10K/raw/OneHand10K.tar.gz.00 -C $DOWNLOAD_DIR/
tar -xvf $DOWNLOAD_DIR/OneHand10K/OneHand10K.tar.00 -C $DATA_ROOT/
rm -rf $DOWNLOAD_DIR/OneHand10K
rm -rf $DOWNLOAD_DIR/OneHand10K $DOWNLOAD_DIR/OpenDataLab___OneHand10K
4 changes: 2 additions & 2 deletions tools/dataset_converters/scripts/preprocess_wflw.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
DOWNLOAD_DIR=$1
DATA_ROOT=$2

tar -zxvf $DOWNLOAD_DIR/WFLW/raw/WFLW.tar.gz.00 -C $DOWNLOAD_DIR/
tar -zxvf $DOWNLOAD_DIR/OpenDataLab___WFLW/raw/WFLW.tar.gz.00 -C $DOWNLOAD_DIR/
tar -xvf $DOWNLOAD_DIR/WFLW/WFLW.tar.00 -C $DATA_ROOT/
rm -rf $DOWNLOAD_DIR/WFLW
rm -rf $DOWNLOAD_DIR/WFLW $DOWNLOAD_DIR/OpenDataLab___WFLW
Loading