[Epic]: Improve Documentation, Tutorials, and Examples

[andygrove]

“Write cool software and tell people about it” – Paul Dix @pauldix (Founder and CTO of InfluxData)

Call to action:

The DataFusion community has invested a lot in the cool software; Now is the time to do better on the “tell people about it” part.

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
DataFusion is too difficult to learn for new users. See https://towardsdev.com/writing-a-data-pipeline-in-rust-with-datafusion-25b5e45410ca for one users experience, which is summarized here:

• The API documentations are pretty bad, the less frequently used function does not provide any document or incomplete document and lacks examples that how to use them. So I had to guess and try out many things to use some of the function such as to_timestamp date_part when(...).otherwise()
• Data Reading example also lacks example, there is only 2–3 example mentioned in the API doc. But you might need another way of reading the data. For example, first define the schema and then use the schema to read the data without inferring the scheme by the framework. But this in not in the doc
• There is no tutorial like example in the doc, this also true for Rust API for Polars DataFrame
• From user point of view, I think documentation is the weakest part of the Framework, on top of that rust is not that easy itself.

Describe the solution you'd like
TBD. This issue is an EPIC to track tasks to improve the situation.

User Guide

• Document all scalar SQL functions in user guide #3065
• Add DataFrame section to user guide #3066
• Update SQL reference in user guide to cover all supported syntax #3091
• Add documentation on querying against files in object store such as S3 #3399
• Clarify DataFusion similarities and differences with duckdb, pola.rs and other similar systems #5498
• Automate build and publish of the user guide #5500
• Consolidate README.md and main documentation site #5755
• Improve expressions.md #5977
• Document Window functions #6338

Rust Docs (docs.rs)

• Automate production of SQL and DataFrame references for SQL functions / Expressions #3092
• Include the latest rustdocs (e.g. cargo doc output on https://arrow.apache.org/datafusion/) #5981
• rustdoc searches don't find all relevant classes  #6648

Developer/Contributor Guide

• Improve the docs for developers (Create a Developers / Hackers guide for DataFusion) #5501
• Create a presentation about DataFusion's architecture #5499
• Fix rustdoc errors #6042

Python Docs

• Add instructions and scripts for publishing the Python documentation datafusion-python#33

Blog posts

• Blog post about datafusion 16 release #4804
• Write a blog about parquet predicate pushdown #3464
• Blog post with DataFusion Jan - April 2023 #5812

Older Issues To Be Reviewed

• User Guide Improvements (aka give the userguide some love) #1814
• Add documentation on supported functions to datafusion website #1487
• documentations for datafusion-cli can be consolidated a bit more #1352

[kmitchener]

As a relatively new user of DataFusion, I agree the docs on the site are pretty bad, but not so bad that I wasn't able to understand it without being overwhelmed :) However, there's plenty of room for improvement and I've been doing a few things related to this:

• reviewing all the documentation-tagged issues in this project
• reviewing other similar project's documentation to see how it's structured, what's helpful to me as a newbie to their project, what's confusing about their documentation
• trying (locally) different structure for documentation based on the above

I've come up with a few changes that I think will help:

• realize that the DataFusion library itself is the product here, not DataFusion-CLI. DataFusion-CLI is a tool for demonstrating the power of the library quickly and easily for potential users (among other uses, but to me this seems to be the primary use case -- if it's not, it's worth clarifying what the intended use of DataFusion-CLI is).
  • as such, DataFusion-CLI should have a supporting role in the documentation
  • as such, there should be more examples how potential new users can use DataFusion-CLI to quickly demonstrate for themselves how DataFusion library can help them
  • therefore there should be multiple examples how to run DataFusion-CLI, how to register a variety of data into DataFusion context, and the power of the SQL -- examples should cover loading data from local, from object store like S3, partitioned and not, different formats, etc. Queries should be run. Explain plans and explain analyze should be shown.
  • probably CLI itself will need some changes to make it easier to use it against S3 or Azure Blob Storage. It should be able to parse the location given in a create external table command to automatically register an object store using sensible default authentication methods. for example, if I already have my environment configured to use the AWS CLI, then I should be able to startup DataFusion-CLI and run create external table test stored as parquet location 's3::/my-bucket/content' and it should "just work".
• there should be a separate User Guide and a Developer Guide (or maybe call it a "Contributor Guide"?)
• yes, in the User Guide, I think the functions we have in DataFusion should be documented. for new prospective users browsing docs, it's important to see the wide variety of useful functions that exist.
  • related to this, some of the documentation makes it appear that DataFusion is less capable than it actually is -- in particular, the bit that describes the very basic SQL syntax that works (implying that more complex SQL won't work)
• ETL is called out as particular use case for DataFusion, but none of the examples demonstrate ETL pipelines. In my opinion, definitely need some examples or even just write-ups about how DataFusion can work as an ETL tool.

It's still very much a WIP, but I have a structure that I think mostly makes sense in this branch: https://github.com/kmitchener/arrow-datafusion/tree/doc-improvements (related to PR #3005 )

The above incorporates some of the thoughts from #1821 and #1814 as well.

[andygrove]

Thanks @kmitchener that is great feedback

[alamb]

I had a conversation with @MrPowers today which inspired me to try and organize ideas to improve the datafusion documentation today.

I moved all the unfinished tickets into a new epic #7013 and am going to close this one so the current state of things is clearer. Let's continue the conversation there