Server fails (code 500) when private families are in DB

[ant20879g]

Gramps 5.2.2
Gramps Web API 2.3.1
Gramps Web Frontend 24.6.0
Gramps QL 0.3.0
locale: en
multi-tree: false
task queue: true

GET /api/families/?locale=en&profile=self&keys=gramps_id,profile,change&page=2&pagesize=20)
I marked few families as private, there're more than 20 non-families. Based on default filter all private families are at the end of the list or on the second page. GET works for first page and crashes for second one.

[DavidMStraub]

It can't be because of families being private, there must be another reason.

Please check the server logs and report them here. 500 always means a Python exception is thrown.

[DavidMStraub]

Please also try whether the problem persists after running check & repair on your database.

[ant20879g]

.... some lines are skipped for clarity....
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/webargs/core.py", line 649, in wrapper
grampsweb-1 | return func(*args, **kwargs)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 446, in get
grampsweb-1 | [self.full_object(obj, args, locale=locale) for obj in objects],
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 446, in
grampsweb-1 | [self.full_object(obj, args, locale=locale) for obj in objects],
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 93, in full_object
grampsweb-1 | obj = self.object_extend(obj, args, locale=locale)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/families.py", line 55, in object_extend
grampsweb-1 | obj.profile = get_family_profile_for_object(
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 505, in get_family_profile_for_object
grampsweb-1 | "father": get_person_profile_for_handle(
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 473, in get_person_profile_for_handle
grampsweb-1 | return get_person_profile_for_object(db_handle, obj, args, locale=locale)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 399, in get_person_profile_for_object
grampsweb-1 | birth, birth_event = get_birth_profile(
grampsweb-1 | ^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 266, in get_birth_profile
grampsweb-1 | event = get_birth_or_fallback(db_handle, person)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps/gen/utils/db.py", line 59, in get_birth_or_fallback
grampsweb-1 | birth_ref = person.get_birth_ref()
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | AttributeError: 'NoneType' object has no attribute 'get_birth_ref'

This happens when public family has private parents. gramps lib's sanitize_family() does not set parent handle when it's private, so family with null in one or two handles is legitimate case. For this particular case this line better to be in try/finally block.

[ant20879g]

When I make family private (for private parents) I'm getting slightly different error:

...some lines are skipped for clarity...
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/webargs/core.py", line 649, in wrapper
grampsweb-1 | return func(*args, **kwargs)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 446, in get
grampsweb-1 | [self.full_object(obj, args, locale=locale) for obj in objects],
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 446, in
grampsweb-1 | [self.full_object(obj, args, locale=locale) for obj in objects],
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/base.py", line 93, in full_object
grampsweb-1 | obj = self.object_extend(obj, args, locale=locale)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/families.py", line 55, in object_extend
grampsweb-1 | obj.profile = get_family_profile_for_object(
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 514, in get_family_profile_for_object
grampsweb-1 | "children": [
grampsweb-1 | ^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 515, in
grampsweb-1 | get_person_profile_for_handle(
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 473, in get_person_profile_for_handle
grampsweb-1 | return get_person_profile_for_object(db_handle, obj, args, locale=locale)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 399, in get_person_profile_for_object
grampsweb-1 | birth, birth_event = get_birth_profile(
grampsweb-1 | ^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/util.py", line 266, in get_birth_profile
grampsweb-1 | event = get_birth_or_fallback(db_handle, person)
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | File "/usr/local/lib/python3.11/dist-packages/gramps/gen/utils/db.py", line 59, in get_birth_or_fallback
grampsweb-1 | birth_ref = person.get_birth_ref()
grampsweb-1 | ^^^^^^^^^^^^^^^^^^^^
grampsweb-1 | AttributeError: 'NoneType' object has no attribute 'get_birth_ref'

I do not see families with private children in my DB, however, at the end it still crashes when get_person_profile_for_object() is called with null in the second parameter.

[ant20879g]

Forgot to mention before, DB check did not find any errors.

[DavidMStraub]

Thanks! I see the problem – Gramps' PrivateProxyDb returns None if a person is private rather than HandleError 😨 Why, actually?

[Nick-Hall]

The return value of None indicates that the handle exists, but has been filtered out by the proxy. The HandleError exception is a real error - the handle is referenced by an object, but doesn't actually exist.

In the past we used None for both cases, but this made tracking down errors very difficult and we had a large number of reports of associated NoneType errors.

[DavidMStraub]

and we had a large number of reports of associated NoneType errors.

Such a great argument in favour of type hints 😉

[Nick-Hall]

The main problem was that we didn't raise an exception for the errors. They just slipped through looking like proxy filtering.