What's the best way for me to reproduce this error?

Hi @hauntsaninja ,
I identified the issue, this issue is caused especially in Bazel and the reason the issue is coming is because of the creation of pycache module.

Here you can see the Code:

import importlib
import tiktoken_ext
import pkgutil

from pprint import pprint

if __name__ == "__main__":
    plugin_mods = pkgutil.iter_modules(tiktoken_ext.__path__, tiktoken_ext.__name__ + ".")

    for _, mod_name, _ in plugin_mods:
        mod = importlib.import_module(mod_name)
        try:
            constructors = mod.ENCODING_CONSTRUCTORS
            print(constructors)
        except Exception as e:
            print("Error", mod)
            pass

Here's the output:

First loop value
`Error <module 'tiktoken_ext.pycache' from '/tiktoken_0.3.1/site-packages/tiktoken_ext/pycache/init.py'>

Second loop value
{'gpt2': <function gpt2 at 0x104b74dc0>, 'r50k_base': <function r50k_base at 0x106386670>, 'p50k_base': <function p50k_base at 0x106386700>, 'p50k_edit': <function p50k_edit at 0x106386790>, 'cl100k_base': <function cl100k_base at 0x106386820>}`

Any update on this @hauntsaninja?

@hauntsaninja I get error in lambda while deploying through zappa

File "/var/task/summaryserver.py", line 13, in <module>
   encoding = tiktoken.get_encoding("cl100k_base")
 File "/var/task/tiktoken/registry.py", line 64, in get_encoding
   _find_constructors()
 File "/var/task/tiktoken/registry.py", line 44, in _find_constructors

The problem is not that __pycache__ exists, that’s normal. The problem is that pkgutil.iter_modules isn’t smart enough to skip it. So you should skip it yourself instead.

@flying-sheep please make them merge it.