Enhance the preview function to get more comprehensive summary

[leeeizhang]

• csv preview [MRG] update the preview_csv_data function #249
• json preview
• yaml preview
• directory preview [MRG] enhance list_files function #252

[huangyz0918]

In my priority list, previewing the directory is the most important thing. Assume there is a dir containing a large number of pictures (or videos, audios, etc), we cannot simply return all pictures' path (but we need a part of them), how to draw a clear file structure/pattern for LLMs meanwhile maintaining a acceptable (not scale) RAG information?

[leeeizhang]

In my priority list, previewing the directory is the most important thing. Assume there is a dir containing a large number of pictures (or videos, audios, etc), we cannot simply return all pictures' path (but we need a part of them), how to draw a clear file structure/pattern for LLMs meanwhile maintaining a acceptable (not scale) RAG information?

Yes, let us make it easy:

1. Returning all file paths is not feasible: Most of the datast files are usually named under a fixed pattern. So we do not need to list all of them. Instead, giving some examples (3 to 5 files) in a directory is enough.
2. How to solve the directory recursion: We should limit the recursive depth for a dataset path. For the deeper directories, we only show the number of them.

Some example:

dataset/ (2 directories, 3 files)
├── train/ (89 directories)
│   ├── idx-0001/ (1203 files)
│   │   ├── 0001-01.png
│   │   ├── 0001-02.png
│   │   ├── 0001-03.png
│   │   └── ...
│   ├── idx-0002/ (1499 files)
│   │   ├── 0002-01.png
│   │   ├── 0002-02.png
│   │   ├── 0002-03.png
│   │   └── ...
│   └── idx-0003/ (1098 files)
│       ├── 0003-01.png
│       ├── 0003-02.png
│       ├── 0003-03.png
│       └── ...
├── eval/ (89 directories)
│   ├── idx-0001/ (1203 files)
│   │   ├── 0001-01.png
│   │   ├── 0001-02.png
│   │   ├── 0001-03.png
│   │   └── ...
│   ├── idx-0002/ (1499 files)
│   │   ├── 0002-01.png
│   │   ├── 0002-02.png
│   │   ├── 0002-03.png
│   │   └── ...
│   └── idx-0003/ (1098 files)
│       ├── 0003-01.png
│       ├── 0003-02.png
│       ├── 0003-03.png
│       └── ...
├── raw.zip (32,029,129 bytes)
├── README.md (1203 lines)
└── checksum.json (1203 lines)