Recent documents missing from results

[vanderburgt]

Describe the bug

Summary
Documents from the past month do not appear in the search results, only older documents are available.

URL / Environment
https://openbesluitvorming.nl/

Steps to reproduce

1. Go to a municipality's counsel information system (e.g. https://utrecht.bestuurlijkeinformatie.nl/Reports/Details/41b35454-6329-4f11-b803-d7e5a6141085)
2. Take note of a recent document (e.g. a weeks old)
3. Go to https://openbesluitvorming.nl/
4. Search for the document based on the title and organisation
5. View results
6. See no results for the document

Expected behavior

Documents should be available in the ORI database within 24 hours after they have been published in the counsel information system of a municipality.

Additional context

This issue reproduces for all municipalities that were tested:

• Schiermonnikoog (no documents at all)
• Utrecht (no document for the past month)
• Groningen (no docs for past 5-6 weeks): https://gemeenteraad.groningen.nl/Documenten/Raadsvoorstellen/ + https://openbesluitvorming.nl/?organisaties=%5B%22ori_groningen_20230704170843%22%5D&type=%5B%22MediaObject%22%5D&zoekterm=%22*%22
• Arnhem (no docs for past 4 weeks): https://openbesluitvorming.nl/?type=%5B%22MediaObject%22%5D&zoekterm=%22*%22&organisaties=%5B%22ori_arnhem_20230705033854%22%5D

[breyten]

I think shiermonnikoog has a different provider now?

[breyten]

loader_1    | [pipeline] ERROR [waadhoeke] Pipeline has failed. Setting status of run identifier pipeline_fcfca69b720b4c3a8b57e748d0cfbd5c to "error":
loader_1    | Unknown fault occured
loader_1    | [trace] ERROR Task ocd_backend.pipeline.setup_pipeline[a0c59f48-d57e-4909-83c0-e4567c62a0bf] raised unexpected: Fault('Unknown fault occured')
loader_1    | Traceback (most recent call last):
loader_1    |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
loader_1    |     R = retval = fun(*args, **kwargs)
loader_1    |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 650, in __protected_call__
loader_1    |     return self.run(*args, **kwargs)
loader_1    |   File "/opt/ori/ocd_backend/pipeline.py", line 108, in setup_pipeline
loader_1    |     for item in pipeline_extractors[pipeline['id']](
loader_1    |   File "/opt/ori/ocd_backend/extractors/ibabs.py", line 282, in run
loader_1    |     extra_info_item = self.client.service.GetListEntry(
loader_1    |   File "/usr/local/lib/python3.8/site-packages/zeep/proxy.py", line 40, in __call__
loader_1    |     return self._proxy._binding.send(
loader_1    |   File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 130, in send
loader_1    |     return self.process_reply(client, operation_obj, response)
loader_1    |   File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 195, in process_reply
loader_1    |     return self.process_error(doc, operation)
loader_1    |   File "/usr/local/lib/python3.8/site-packages/zeep/wsdl/bindings/soap.py", line 283, in process_error
loader_1    |     raise Fault(
loader_1    | zeep.exceptions.Fault: Unknown fault occured

Might be related to this.

[breyten]

loader_1    | [pipeline] ERROR [wassenaar] Pipeline has failed. Setting status of run identifier pipeline_b1d4e56336294cb38e0c063642c8e595 to "error":
loader_1    | Unable to parse date string 'R'
loader_1    | [trace] ERROR Task ocd_backend.pipeline.setup_pipeline[cb8ba3b2-b634-43cc-914c-f9721990d45e] raised unexpected: ParseError("Unable to parse date string 'R'")
loader_1    | Traceback (most recent call last):
loader_1    |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
loader_1    |     R = retval = fun(*args, **kwargs)
loader_1    |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 650, in __protected_call__
loader_1    |     return self.run(*args, **kwargs)
loader_1    |   File "/opt/ori/ocd_backend/pipeline.py", line 108, in setup_pipeline
loader_1    |     for item in pipeline_extractors[pipeline['id']](
loader_1    |   File "/opt/ori/ocd_backend/extractors/ibabs.py", line 309, in run
loader_1    |     if report_dict['datum'] and isinstance(report_dict['datum'], str) and report_dict['datum'] != 'V' and not start_date < iso8601.parse_date(report_dict['datum']).replace(tzinfo=None) < end_date:
loader_1    |   File "/usr/local/lib/python3.8/site-packages/iso8601/iso8601.py", line 195, in parse_date
loader_1    |     raise ParseError("Unable to parse date string %r" % datestring)
loader_1    | iso8601.iso8601.ParseError: Unable to parse date string 'R'

Same

[breyten]

De Ronde Venen (GM0736) is not at Notubiz anymore

[joepio]

Zoetermeer has a similar issue perhaps - no imports since Nov 2023.