Crawlit - Dofus & Dofus-Touch encyclopedia parser - Build your own Dofus API

In order to respond to a need expressed by the community, I create an encyclopedia parser for Dofus available for :

• Windows
• OSX
• Linux

Download : All versions are available here

⚠️ Because of new parsing method : 429 error limitation removed, You can parse as long as you want

Crawlit was built to be very light, it can run on almost every configurations.

In fact Crawlit is a tool :

• Developed by the dofus community, for the community
• 100% performance : Developed to be very light
• Simple to use | Plug and use
• Interactive prompt & Command prompt
• Json file & MongoDB importable file generated
• Resume parsing feature after errors (connection off / 429 error / ...)
• English & French encyclopedia supported (french item, english item and soon all other versions)
• Full Encyclopedia parse available : just execute the npm run dofus-all-fr/npm run dofus-all-en or npm run touch-all-fr/npm run touch-all-en command (be patient 5h & run it in bash shell)

Interactive prompt

App launched	App at the end

Command line prompt

You can run the program with those parameter (all lowercase):

Argument	Alias	Value
-g	--game	dofus or dofus-touch
-l	--language	french or english
-c	--category	categoryName (see list of categories)
-a or -m	--all for -a or --maxItem for -m	Number for -m/--maxItem (Amount of item you want)
-h	--help

Example : npm run crawlit && node ./dist/app.js --game dofus-touch --language english --category monster --maxItem 5

⚠️ If you want all equipments or all weapons : -category argument's value must be written like allequipments and allweapons.

Example : npm run crawlit && node ./dist/app.js --game dofus-touch --language english --category allequipments --all

JSON files also provided in the repository

In case you wan't directly the dofus JSON files, I oftenly upload up-to-date encyclopedia JSON in the data/ folder.

JSON format

An item result example

{  
  "_id":"null : represent your db id",
  "ankamaId": "item id",
  "name":"item name",
  "description":"item description.",
  "lvl":"item lvl",
  "type":"item type",
  "imgUrl":"image url of the item",
  "url":"Item's link",
  "stats":[ "many statistics line", "stat 2", ["..."], "stat n" ],
  "condition":[ "many conditions line", "condition 2", ["..."], "condition n" ],
  "set":{  
     "equipments":[itemId, itemId ...],
     "weapons":[itemId, itemId ...],
  }
}

Getting Started

Clone & access this repository locally :

git clone https://github.com/raczak/crawlit-dofus-encyclopedia-parser.git
cd crawlit-dofus-encyclopedia-parser

1) Install dependencies with npm :

npm install

2) Then at the root folder launch the programm with npm or node :

npm start

⚠️ There is an issue with npm start on some CLI, in this case prefere use npm run build && node lib/app.js.

Supported items

Items	state of progress
Equipment	100 %
Weapon	100 %
Set	100 %
Pet	100 %
Mount	100 %
Resource	100 %
Consumable	100 %
Professions	100 %
Recipe	25 %
Classe	100 %
Bestiary	100 %
Harnesse	100 %
Idol	100 %
Sidekick	0 %
Haven Bag	100 %

We also have done an API !

Visite the official non-official Dofus API : DOFAPI

Technologies choice

Crawlit is build with full javascript : NodeJs/Npm, javascript.

Bug & Crash || Proposal ?

For proposal don't hesitate to create a pool request or for a bug/crash an issue. You can also make us aware on bugs channel in discord.

Documentation if you want to contribute (dev)

English Version : Access the wiki to understand the project and contribute to it.

French Version : There is also a french version of the documentation 😃 !

Join the dev Team ? Discord

Contact us on discord (channel general or private message to theukid)

Project sponsors

Contributors

Zakaria RACHEDI	Le Corre Julien	Yannick Milanetto	Darkilen