Firehose - Convert commit contents to JSON format

[Nick-the-freak]

While it is possible to create a recursive function which parses all objects into JSON format, would be great to have an easier/official way.

Object of type XXX is not JSON serializable

[MarshalX]

Pls show the code when it tells about not setealiziblr objects

[Nick-the-freak]

This is the part of the code that tells about not serializable objects most frequently.

commit_info = {...}
...
elif op.action == 'create':
    record_raw_data = car.blocks.get(op.cid)
    if not record_raw_data:
        continue
    record = get_or_create(record_raw_data, strict=False)
    ...
    elif uri.collection == ids.AppBskyFeedPost and is_record_type(record, ids.AppBskyFeedPost):
        # check labels
        if record.labels:
            commit_info['labels'] = record.labels.__dict__⭐️
        # check embedings
        if record.embed:
            commit_info['embed'] = record.embed.__dict__ ⭐️
...
with open(output_filename, "a+") as json_file:
    json_file.write(f'{json.dumps(commit_info)}\n')⭐️

I created this function to parse/convert and works okay so far:

convert_to_json_serializable(record.labels)⭐️
convert_to_json_serializable(record.embed)⭐️

def convert_to_json_serializable(obj):
    if obj is None:
        return None
    elif isinstance(obj, atproto.xrpc_client.models.dot_dict.DotDict):
        return convert_to_json_serializable(obj.__dict__)
    elif isinstance(obj, list):
        return [convert_to_json_serializable(item) for item in obj]
    elif isinstance(obj, dict):
        return {key: convert_to_json_serializable(value) for key, value in obj.items()}
    else:
        try:
            return convert_to_json_serializable(obj.__dict__)
        except:
            return obj

[MarshalX]

The correct way is to use this method https://atproto.blue/en/latest/atproto/atproto.xrpc_client.models.utils.html#atproto.xrpc_client.models.utils.get_model_as_json

Of course you should manually handle lists

[Nick-the-freak]

Thanks for the link! I somehow missed the update to get_modle_as_json, the latest version works perfectly