fix: Install pandoc consistently, via Makefile recipe (version that supports .rtf files as input format)

[micmarty-deepsense]

Problem Description

In some cases you might find yourselves in a situation when pandoc won't be able to process an rtf as input file format, because older versions simply do not support that.

RuntimeError: Invalid input format! Got "rtf" but expected one of these: commonmark, creole, csv, docbook, docx, dokuwiki, epub, fb2, gfm, haddock, html, ipynb, jats, jira, json, latex, man, markdown, markdown_github, markdown_mmd, markdown_phpextra, markdown_strict, mediawiki, muse, native, odt, opml, org, rst, t2t, textile, tikiwiki, twiki, vimwiki

Basically, some user may install the wrong version. The README.md is not be precise enough when mentioning RTF files support:

unstructured/README.md

Lines 120 to 122 in 47b35cc

	- `tesseract-ocr` (images and PDFs, install `tesseract-lang` for additional language support)
	- `libreoffice` (MS Office docs)
	- `pandoc` (EPUBs, RTFs and Open Office docs)

Example

Installing pandoc from a stable repository, like Debian will give you 2.9 and the official documentation shows clearly that support for rtf was introduced in 2.14 https://pandoc.org/releases.html#pandoc-2.14.2-2021-08-21

Note that rtf is not there

More detail

Proposed Solution

• I've simply added/copied make install-pandoc calls, mimicking other recipes in order to ensure that 3.1.2 will be installed in all cases. Side note: make install-pandoc calls ./scripts/install-pandoc.sh under the hood.
• Update README file - mention that make install-pandoc is recommended (>=2.14.2)
• Verify tests that cover rtf cases:

  unstructured/test_unstructured/file_utils/test_file_conversion.py

  Line 14 in 47b35cc

  filename = os.path.join(DIRECTORY, "..", "..", "example-docs", "winter-sports.epub")

• Update setup_ubuntu.sh if needed?:

  unstructured/scripts/setup_ubuntu.sh

  Line 87 in 47b35cc

  $sudo $pac install -y libreoffice pandoc

[micmarty-deepsense]

Suggested error message

RuntimeError: Invalid input format! Got "rtf" but expected one of these: commonmark, creole, csv, docbook, docx, dokuwiki, epub, fb2, gfm, haddock, html, ipynb, jats, jira, json, latex, man, markdown, markdown_github, markdown_mmd, markdown_phpextra, markdown_strict, mediawiki, muse, native, odt, opml, org, rst, t2t, textile, tikiwiki, twiki, vimwiki

Support for RTF files is not available in the current pandoc installation. It was introduced in pandoc 2.14.2.
Reference: https://pandoc.org/releases.html#pandoc-2.14.2-2021-08-21

Current version of pandoc: 2.9.2.1
Please, follow the pandoc installation instructions in README.md to install the right version.

Running make test-extra-pypandoc will generate the following output

Reinstalling the right version makes all tests passing

sudo apt remove pandoc
make install-pandoc
make test-extra-pypandoc

[Klaijan]

A little related to this PR - Here at this line https://github.com/Unstructured-IO/unstructured/blob/main/.github/workflows/ci.yml#L438 just for cleanliness, I'm thinking is it ok if we add the pandoc installation to unstructured-inference's install-ci as well since CI will need to run against various file types including rtf. This way we can omit the make install-pandoc on later line.

But since it's not directly related to this PR so it's your call :)

Anyways, just need to update CHANGELOG and update the branch then all LGTM! Switch this to ready for review then I can approve it then!

[micmarty-deepsense]

I've verified if docker-build contains the right version of pandoc and it does:

make docker-build
make docker-start-bash
pandoc --version
pandoc --list-input-formats | grep rtf

[micmarty-deepsense]

A little related to this PR - Here at this line https://github.com/Unstructured-IO/unstructured/blob/main/.github/workflows/ci.yml#L438 just for cleanliness, I'm thinking is it ok if we add the pandoc installation to unstructured-inference's install-ci as well since CI will need to run against various file types including rtf. This way we can omit the make install-pandoc on later line.

But since it's not directly related to this PR so it's your call :)

Anyways, just need to update CHANGELOG and update the branch then all LGTM! Switch this to ready for review then I can approve it then!

Good catch @Klaijan ! I removed those install-pandoc calls from GH workflows and now it's part of the install-ci recipe

PR is no longer in draft mode, so if it's still legitimate, please approve ;) (when all the checks will pass)

[Klaijan]

LGTM!