Dictionary changed size during iteration

[mla-martin]

I ran this little test-script:

  import asyncio
  from asynctinydb import TinyDB, Query
  import sys
  import time
  
  async def create_db(db_path, num_elements):
      db = TinyDB(db_path)
      await db.truncate()  # Clear the database for a fresh start
  
      start_time = time.time()
      tasks = [db.insert({'type': 'item', 'id': i, 'name': f'Item {i}'}) for i in range(num_elements)]
      await asyncio.gather(*tasks)
      end_time = time.time()
  
      print(f"Database creation with {num_elements} elements took {end_time - start_time} seconds.")
      return db
  
  async def search_db(db, search_criteria):
      Item = Query()
      start_time = time.time()
      results = await db.search(Item.type == 'item' and Item.id == search_criteria)
      end_time = time.time()
  
      print(f"Search for item with ID {search_criteria} took {end_time - start_time} seconds, found {len(results)} items.")
      return results
  
  async def main(num_elements, search_id):
      db_path = 'async_test_db.json'
      db = await create_db(db_path, num_elements)
      search_results = await search_db(db, search_id)
  
  if __name__ == "__main__":
      if len(sys.argv) < 3:
          print("Usage: python script.py <number_of_elements> <search_id>")
          sys.exit(1)
  
      num_elements = int(sys.argv[1])
      search_id = int(sys.argv[2])
  
      asyncio.run(main(num_elements, search_id))

When running I get plenty of:

$ python3 benchmark.py 10000 3
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
asyncio.exceptions.InvalidStateError: invalid state
Traceback (most recent call last):
  File "/tmp/benchmark.py", line 40, in <module>
    asyncio.run(main(num_elements, search_id))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/tmp/benchmark.py", line 29, in main
    db = await create_db(db_path, num_elements)
  File "/tmp/benchmark.py", line 12, in create_db
    await asyncio.gather(*tasks)
  File "<...>/.local/lib/python3.10/site-packages/asynctinydb/table.py", line 363, in insert
    await self._update_table(updater)
  File "<...>/.local/lib/python3.10/site-packages/asynctinydb/table.py", line 899, in _update_table
    await self._storage.write(tables)
  File "<...>/.local/lib/python3.10/site-packages/asynctinydb/storages.py", line 221, in write
    data = await self._sink.run(stringify_keys, data)
  File "<...>/.local/lib/python3.10/site-packages/vermils/asynctools/asinkrunner.py", line 179, in _run
    ret = func()
  File "<...>/.local/lib/python3.10/site-packages/vermils/gadgets/misc.py", line 55, in stringify_keys
    memo[id(data)].update((str(k), stringify_keys(v, memo))
  File "<...>/.local/lib/python3.10/site-packages/vermils/gadgets/misc.py", line 55, in <genexpr>
    memo[id(data)].update((str(k), stringify_keys(v, memo))
  File "<...>/.local/lib/python3.10/site-packages/vermils/gadgets/misc.py", line 55, in stringify_keys
    memo[id(data)].update((str(k), stringify_keys(v, memo))
  File "<...>/.local/lib/python3.10/site-packages/vermils/gadgets/misc.py", line 55, in <genexpr>
    memo[id(data)].update((str(k), stringify_keys(v, memo))
RuntimeError: dictionary changed size during iteration

It does happen reproducible with 10000 entries to insert. With 1000 it happens sometimes and sometimes not.

[VermiIIi0n]

Thank you for your report!

I will fix it in the next release along with the other issue.

BTW, if you are doing a lot of writing, I suggest adding a CachingMiddleware. It took me 200+ seconds to finish this script with n = 10000.