Add Isna Persian Dataset (#3631)

The level of importance of this data is less than Wikipedia. So, I think [this pull request](#3629) should be merged first. I have uploaded the data to [huggingface](https://huggingface.co/datasets/pourmand1376/isna-news) according to Open-assistant's standard. So, it shouldn't need any processing. --------- Co-authored-by: Oliver Stanley <[email protected]>
LAION-AI · Aug 3, 2023 · 0d4adb5 · 0d4adb5
1 parent 65f5c2b
commit 0d4adb5
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 0 deletions.
diff --git a/data/datasets/__init__.py b/data/datasets/__init__.py
@@ -4,6 +4,7 @@
     "tv_dialogue": "sedthh/tv_dialogue",  # TV and Movie dialogues and transcripts
     "fd_dialogue": "sedthh/fd_dialogue",  # TV and Movie dialogues and transcripts from ForeverDreaming
     "tlcv2.0_oa": "pythainlp/tlcv2.0_oa",  # Thai classical literature texts
+    "fa-isna-news": "pourmand1376/isna-news",  # Isna Persian News
     "fa-wikipedia": "pourmand1376/fa-wikipedia",  # Farsi Wikipedia texts
 }
 

diff --git a/data/datasets/fa-isna-news/README.md b/data/datasets/fa-isna-news/README.md
@@ -0,0 +1,2 @@
+This text-only dataset is crawled from [Isna news](https://isna.ir/). This is
+biggest farsi news agency and thus the text is pretty clean.