amilib general TEST. Please run

[petermr]

amilib https://github.com/petermr/amilib/ is a problem-independent library of methods, mainly related to webscraping and document processing, It consists of a number of code modules and their methods, and also tests for most of the functionality. It will be very helpful if all current interns and any volunteers who have the time can test the code/compilation.

This is alpha code so we expect a few problems, mainly related to operating systems and versions of Python.

PLEASE DOCUMENT YOUR EXPERIENCE. ESPECIALLY:

• errors
• warnings
• timings
• unexpected behaviour

environment

• Please create a venv (virtual environment) so the enviroment is clean and accry out operations there.
• check that pip is installed
• git clone https://github.com/petermr/amilib
• verify that the toplevel contains

LICENSE			setup.py
README.md		quick_install.sh	temp
amilib			requirements.txt	test

ignore other files/directories

files in amilib/ directory

ls amilib

should show

__init__.py		ami_pdf_libs.py		file_lib.py		resources/
ami_bib.py		ami_svg.py		headless_lib.py		util.py
ami_csv.py		ami_util.py		html_extra.py		wikimedia.py
ami_html.py		amidriver.py		html_generator.py	xml_lib.py
ami_integrate.py	amix.py			html_marker.py
ami_nlp.py		bbox.py			pdf_args.py

TEST

The test directorytest

ls test

should contain:

__init__.py		test_file.py		test_pdf.py		test_wikidata.py
resources		test_headless.py	test_stat.py		test_xml.py
resources.py		test_html.py		test_svg.py

(ignore other files , if any)

[petermr]

running tests

Still in the toplevel directory, run 'pytest(you may have topip install it first). This is my output (there are no errors).

(base) pm286macbook-2:amilib pm286$ pytest
================================================ test session starts =================================================
platform darwin -- Python 3.8.3, pytest-6.2.5, py-1.9.0, pluggy-0.13.1
rootdir: /Users/pm286/workspace/amilib
plugins: cov-3.0.0
collected 218 items                                                                                                  

test/test_file.py ss                                                                                           [  0%]
test/test_headless.py .s..sssssss.....s                                                                        [  8%]
test/test_html.py ...s.s......s.s.....ssss...s.........s..ssssss..ss.ss..s.................................s.. [ 50%]
..s                                                                                                            [ 52%]
test/test_nlp.py .                                                                                             [ 52%]
test/test_pdf.py .ss........s.s.s.sssssss.s....ss.ss..s....ssssssssss..s....ss                                 [ 80%]
test/test_stat.py .                                                                                            [ 81%]
test/test_svg.py ...                                                                                           [ 82%]
test/test_util.py ss.....s...s...                                                                              [ 89%]
test/test_wikidata.py .s...........s.......                                                                    [ 99%]
test/test_xml.py ..                                                                                            [100%]

================================================== warnings summary ==================================================
../../../../opt/anaconda3/lib/python3.8/site-packages/numexpr/expressions.py:21
../../../../opt/anaconda3/lib/python3.8/site-packages/numexpr/expressions.py:21
  /opt/anaconda3/lib/python3.8/site-packages/numexpr/expressions.py:21: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    _np_version_forbids_neg_powint = LooseVersion(numpy.__version__) >= LooseVersion('1.12.0b1')

test/test_nlp.py::NLPTest::test_compute_text_similarity_STAT
  /opt/anaconda3/lib/python3.8/site-packages/sklearn/feature_extraction/text.py:525: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'
    warnings.warn(

test/test_nlp.py::NLPTest::test_compute_text_similarity_STAT
  /opt/anaconda3/lib/python3.8/site-packages/sklearn/feature_extraction/text.py:408: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'afterward', 'alon', 'alreadi', 'alway', 'ani', 'anoth', 'anyon', 'anyth', 'anywher', 'becam', 'becaus', 'becom', 'befor', 'besid', 'cri', 'describ', 'dure', 'els', 'elsewher', 'empti', 'everi', 'everyon', 'everyth', 'everywher', 'fifti', 'formerli', 'forti', 'ha', 'henc', 'hereaft', 'herebi', 'hi', 'howev', 'hundr', 'inde', 'latterli', 'mani', 'meanwhil', 'moreov', 'mostli', 'nobodi', 'noon', 'noth', 'nowher', 'onc', 'onli', 'otherwis', 'ourselv', 'perhap', 'pleas', 'seriou', 'sever', 'sinc', 'sincer', 'sixti', 'someon', 'someth', 'sometim', 'somewher', 'themselv', 'thenc', 'thereaft', 'therebi', 'therefor', 'thi', 'thu', 'togeth', 'twelv', 'twenti', 'veri', 'wa', 'whatev', 'whenc', 'whenev', 'wherea', 'whereaft', 'wherebi', 'wherev', 'whi', 'yourselv'] not in stop_words.
    warnings.warn(

test/test_pdf.py::PDFTest::test_create_text_lines_page6
test/test_pdf.py::PDFTest::test_get_text_attrib_vals
test/test_pdf.py::PDFTest::test_get_text_attribs
  /Users/pm286/workspace/amilib/amilib/ami_pdf_libs.py:195: FutureWarning: This search incorrectly ignores the root element, and will be fixed in a future version.  If you rely on the current behaviour, change it to './/{http://www.w3.org/2000/svg}text'
    self.text_elements = self.page_element.findall(f"//{{{SVG_NS}}}text")

test/test_stat.py::TestStat::test_plot_scatter_noel_oboyle_STAT_PLOT
  /opt/anaconda3/lib/python3.8/site-packages/sklearn/manifold/_mds.py:298: FutureWarning: The default value of `normalized_stress` will change to `'auto'` in version 1.4. To suppress this warning, manually set the value of `normalized_stress`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=============================== 147 passed, 71 skipped, 8 warnings in 91.53s (0:01:31) ===============================
(base) pm286macbook-2:amilib pm286$ 

NOTES

• the total; tests should be 218 (. is OK, s is skip, E or F is a fail which needs highlighlighting.
• a lot of tests are skipped. Some are obsolete, some are long and tested randomly.
• extensions:
  • read the pytest docs to see how to run indvidual modules or groups of tests
  • find out how to run tests in your IDE (pycharm or vscode)

Making amilib release on pypi

In production you will use a tested version of amilib on pypi . This might be done with (say)

pip install amilib==0.0.8

this number will increase as we develop more versions. Only PMR can create the version on PyPI but everyone can install it. We will need a consistent amilib version for amiclimate later.