⚡ Improve performance of `ln.connect()`, `lamin connect`, and `lamin load` for a notebook #84

falexwolf · 2024-09-29T13:28:07Z

This PR improves the CLI and documents profiling for the main PR below:

⚡ Improve performance of ln.connect(), lamin connect, and lamin load for a notebook lamindb#1998

The initial profiling run below shows that lamin load for a notebook takes 4.7s for an instance (laminlabs/lamindata) that's in us-east-1 when called from a laptop in Munich.

In there, ln.connect() contributes almost all waiting time with 4.3s.

Details

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000
→ connected lamindb: laminlabs/lamindata
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
% python -m line_profiler -rmt "lamin.lprof"
Timer unit: 1e-06 s

Total time: 4.70432 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 19

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    19                                           @profile                                                                                     
    20                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    21         1          1.0      1.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    22         1          0.0      0.0      0.0          url = entity                                                                         
    23         1          3.0      3.0      0.0          instance_slug, entity, uid = decompose_url(url)                                      
    24                                               elif entity not in {"artifact", "transform"}:                                            
    25                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    26                                               else:                                                                                    
    27                                                   instance_slug = None                                                                 
    28                                                                                                                                        
    29         1          0.0      0.0      0.0      if instance_slug is not None:                                                            
    30         1         18.0     18.0      0.0          auto_connect = ln_setup.settings.auto_connect                                        
    31                                                   # we don't want to auto-connect when importing lamindb                               
    32         1         80.0     80.0      0.0          ln_setup.settings.auto_connect = False                                               
    33                                                                                                                                        
    34         1       3677.0   3677.0      0.1          import lamindb as ln                                                                 
    35         1        415.0    415.0      0.0          from lamindb._finish import script_to_notebook                                       
    36                                                                                                                                        
    37         1         72.0     72.0      0.0          ln_setup.settings.auto_connect = auto_connect                                        
    38         1    4211566.0    4e+06     89.5          ln.connect(instance_slug)                                                            
    39                                               else:                                                                                    
    40                                                   import lamindb as ln                                                                 
    41                                                   from lamindb._finish import script_to_notebook                                       
    42                                                                                                                                        
    43                                               # below is to silence warnings about missing run inputs                                  
    44         1          2.0      2.0      0.0      ln.settings.track_run_inputs = False                                                     
    45                                                                                                                                        
    46         1          0.0      0.0      0.0      if entity == "transform":                                                                
    47         1          0.0      0.0      0.0          transform = (                                                                        
    48         1     133258.0 133258.0      2.8              ln.Transform.get(uid)                                                            
    49         1          0.0      0.0      0.0              if uid is not None                                                               
    50                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    51                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    52                                                       else ln.Transform.filter(key=key, source_code__isnull=False)                     
    53                                                       .order_by("-created_at")                                                         
    54                                                       .first()                                                                         
    55                                                   )                                                                                    
    56         1          0.0      0.0      0.0          target_filename = transform.key                                                      
    57                                                   # if Path(target_filename).exists():                                                 
    58                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    59                                                   #     if response != "y":                                                            
    60                                                   #         raise SystemExit("Aborted.")                                               
    61         1          2.0      2.0      0.0          if transform._source_code_artifact_id is not None:                                   
    62                                                       # backward compat                                                                
    63                                                       filepath_cache = transform._source_code_artifact.cache()                         
    64                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    65                                                           target_filename += transform._source_code_artifact.suffix                    
    66                                                       filepath_cache.rename(target_filename)                                           
    67         1          0.0      0.0      0.0          elif transform.source_code is not None:                                              
    68         1          1.0      1.0      0.0              if transform.key.endswith(".ipynb"):                                             
    69         1     355134.0 355134.0      7.5                  script_to_notebook(transform, target_filename, bump_revision=True)           
    70                                                       else:                                                                            
    71                                                           Path(target_filename).write_text(transform.source_code)                      
    72                                                   else:                                                                                
    73                                                       raise SystemExit("No source code available for this transform.")                 
    74         1         90.0     90.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    75         1          0.0      0.0      0.0          if with_env:                                                                         
    76                                                       if (                                                                             
    77                                                           transform.latest_run is not None                                             
    78                                                           and transform.latest_run.environment is not None                             
    79                                                       ):                                                                               
    80                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    81                                                           target_env_filename = (                                                      
    82                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    83                                                           )                                                                            
    84                                                           filepath_env_cache.rename(target_env_filename)                               
    85                                                           logger.important(target_env_filename)                                        
    86                                                       else:                                                                            
    87                                                           logger.warning("latest transform run with environment doesn't exist")        
    88         1          0.0      0.0      0.0          return target_filename                                                               
    89                                               elif entity == "artifact":                                                               
    90                                                   artifact = ln.Artifact.get(uid) if uid is not None else ln.Artifact.get(key=key)     
    91                                                   cache_path = artifact.cache()                                                        
    92                                                   logger.important(f"artifact is here: {cache_path}")                                  
    93                                                   return cache_path                                                                    


  4.70 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:19 - load

falexwolf · 2024-09-29T13:34:36Z

The roughly ~4sec have been what we've seen for a while.

https://laminlabs.slack.com/archives/C03P6D8U1PC/p1727270298402879?thread_ts=1727186286.045239&cid=C03P6D8U1PC
https://laminlabs.slack.com/archives/C03P6D8U1PC/p1727270024452489?thread_ts=1727186286.045239&cid=C03P6D8U1PC

falexwolf · 2024-09-29T13:40:03Z

Using the code here:

⚡️ Speed up ln.connect() lamindb-setup#871

Now profiling lamin connect and confirming ~4.3s for connecting to an instance (laminlabs/lamindata) that's in us-east-1 when called from a laptop in Munich.

From the below profiling run it's clear that loading bionty sources and loading all schema modules add up to ~3s. It doesn't seem necessary to have all schema modules imported off the bat. Hence, I'll try to only load lamindb.

Details

% kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"

% python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0.329013 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 109

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   109                                           @profile                                                        
   110                                           def reload_lamindb_itself(isettings) -> bool:                   
   111         1          3.0      3.0      0.0      reloaded = False                                            
   112         1          0.0      0.0      0.0      if "lamindb" in sys.modules:                                
   113         1          0.0      0.0      0.0          import lamindb                                          
   114                                                                                                           
   115         1     164950.0 164950.0     50.1          importlib.reload(lamindb)                               
   116         1          0.0      0.0      0.0          reloaded = True                                         
   117         1          4.0      4.0      0.0      if "bionty" in isettings.schema and "bionty" in sys.modules:
   118         1         18.0     18.0      0.0          schema_module = importlib.import_module("bionty")       
   119         1     164038.0 164038.0     49.9          importlib.reload(schema_module)                         
   120         1          0.0      0.0      0.0          reloaded = True                                         
   121         1          0.0      0.0      0.0      return reloaded                                             


Total time: 2.25075 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125                                               # only touch lamindb if we're operating from lamindb        
   126         1    1921495.0    2e+06     85.4      reload_schema_modules(isettings)                            
   127         1         22.0     22.0      0.0      log_message = settings.auto_connect                         
   128         1     329041.0 329041.0     14.6      if not reload_lamindb_itself(isettings):                    
   129                                                   log_message = True                                      
   130         1          0.0      0.0      0.0      if log_message:                                             
   131         1        190.0    190.0      0.0          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 3.20396 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 348

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   348                                           @profile                                                                             
   349                                           def load_from_isettings(                                                             
   350                                               isettings: InstanceSettings,                                                     
   351                                               *,                                                                               
   352                                               init: bool = False,                                                              
   353                                               user: UserSettings | None = None,                                                
   354                                               write_settings: bool = True,                                                     
   355                                           ) -> None:                                                                           
   356         1          5.0      5.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   357                                                                                                                                
   358         1          9.0      9.0      0.0      user = settings.user if user is None else user                                   
   359                                                                                                                                
   360         1          0.0      0.0      0.0      if init:                                                                         
   361                                                   # during init both user and storage need to be registered                    
   362                                                   register_user_and_storage_in_instance(isettings, user)                       
   363                                                   write_bionty_sources(isettings)                                              
   364                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   365                                               else:                                                                            
   366                                                   # when loading, django is already set up                                     
   367                                                   #                                                                            
   368                                                   # only register user if the instance is connected                            
   369                                                   # for the first time in an environment                                       
   370                                                   # this is our best proxy for that the user might not                         
   371                                                   # yet be registered                                                          
   372         1         52.0     52.0      0.0          if not isettings._get_settings_file().exists():                              
   373                                                       register_user(user)                                                      
   374         1     952192.0 952192.0     29.7          load_bionty_sources(isettings)                                               
   375         1        944.0    944.0      0.0      isettings._persist(write_to_disk=write_settings)                                 
   376         1    2250758.0    2e+06     70.2      reload_lamindb(isettings)                                                        


Total time: 4.47747 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 186

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   186                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   187                                           @profile                                                                               
   188                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   189                                               """Connect to instance.                                                            
   190                                                                                                                                  
   191                                               Args:                                                                              
   192                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   193                                                       If the instance is owned by you, it suffices to pass the instance name.    
   194                                               """                                                                                
   195                                               # validate kwargs                                                                  
   196         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   197                                                   "_db",                                                                         
   198                                                   "_write_settings",                                                             
   199                                                   "_raise_not_found_error",                                                      
   200                                                   "_test",                                                                       
   201                                                   "_user",                                                                       
   202                                               }                                                                                  
   203         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   204                                                   if kwarg not in valid_kwargs:                                                  
   205                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   206                                                                                                                                  
   207         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   208                                               # _db is still needed because it is called in init                                 
   209         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   210         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   211         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   212         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   213                                                                                                                                  
   214         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   215         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   216         1          0.0      0.0      0.0      if _user is not None:                                                              
   217                                                   access_token = _user.access_token                                              
   218                                                                                                                                  
   219         1          0.0      0.0      0.0      try:                                                                               
   220         1          6.0      6.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   221                                                                                                                                  
   222         1       3402.0   3402.0      0.1          if _check_instance_setup() and not _test:                                      
   223                                                       if (                                                                       
   224                                                           settings._instance_exists                                              
   225                                                           and f"{owner}/{name}" == settings.instance.slug                        
   226                                                       ):                                                                         
   227                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   228                                                           return None                                                            
   229                                                       else:                                                                      
   230                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   231         2          0.0      0.0      0.0          elif (                                                                         
   232         1          0.0      0.0      0.0              _write_settings                                                            
   233         1       2223.0   2223.0      0.0              and settings._instance_exists                                              
   234         1          4.0      4.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   235                                                   ):                                                                             
   236                                                       close_instance(mute=True)                                                  
   237                                                                                                                                  
   238         1          0.0      0.0      0.0          try:                                                                           
   239         2     430077.0 215038.5      9.6              isettings = _connect_instance(                                             
   240         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   241                                                       )                                                                          
   242                                                   except InstanceNotFoundError as e:                                             
   243                                                       if _raise_not_found_error:                                                 
   244                                                           raise e                                                                
   245                                                       else:                                                                      
   246                                                           return "instance-not-found"                                            
   247         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   248                                                       return isettings                                                           
   249                                                   # at this point we have checked already that isettings is not a string         
   250                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   251                                                   # has no effect if _user is None or if not cloud sqlite instance               
   252         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   253         1       1157.0   1157.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   254         1          0.0      0.0      0.0          if _test:                                                                      
   255                                                       return None                                                                
   256         1          2.0      2.0      0.0          silence_loggers()                                                              
   257         1     836557.0 836557.0     18.7          check, msg = isettings._load_db()                                              
   258         1          1.0      1.0      0.0          if not check:                                                                  
   259                                                       local_db = (                                                               
   260                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   261                                                       )                                                                          
   262                                                       if local_db:                                                               
   263                                                           logger.warning(                                                        
   264                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   265                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   266                                                               " call: lamin load --unload"                                       
   267                                                           )                                                                      
   268                                                       elif _raise_not_found_error:                                               
   269                                                           raise SystemExit(msg)                                                  
   270                                                       else:                                                                      
   271                                                           logger.warning(                                                        
   272                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   273                                                               " loadable: re-initializing"                                       
   274                                                           )                                                                      
   275                                                           return "instance-corrupted-or-deleted"                                 
   276                                                   # this is for testing purposes only                                            
   277         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   278                                                       raise RuntimeError("Technical testing error.")                             
   279                                                                                                                                  
   280                                                   # below is for backfilling the instance_uid value                              
   281                                                   # we'll enable it once more people migrated to 0.71.0                          
   282                                                   # ssettings_record = isettings.storage.record                                  
   283                                                   # if ssettings_record.instance_uid is None:                                    
   284                                                   #     ssettings_record.instance_uid = isettings.uid                            
   285                                                   #     # try saving if not read-only access                                     
   286                                                   #     try:                                                                     
   287                                                   #         ssettings_record.save()                                              
   288                                                   #     # raised by django when the access is denied                             
   289                                                   #     except ProgrammingError:                                                 
   290                                                   #         pass                                                                 
   291         1    3203984.0    3e+06     71.6          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   292                                               except Exception as e:                                                             
   293                                                   if isettings is not None:                                                      
   294                                                       if _write_settings:                                                        
   295                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   296                                                       settings._instance_settings = None                                         
   297                                                   raise e                                                                        
   298                                               # rename lnschema_bionty to bionty for sql tables                                  
   299         1          4.0      4.0      0.0      if "bionty" in isettings.schema:                                                   
   300         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   301         1         22.0     22.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   302                                                   )                                                                              
   303         1         28.0     28.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   304                                                       migrate_lnschema_bionty(                                                   
   305                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   306                                                       )                                                                          
   307         1          0.0      0.0      0.0      return None                                                                        


  0.33 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:109 - reload_lamindb_itself
  2.25 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  3.20 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:348 - load_from_isettings
  4.48 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:186 - connect

falexwolf · 2024-09-29T15:20:04Z

Refactored considerably, both in the lamindb-setup PR and in bionty.

Will now run profiling again hoping that bionty isn't reloaded upon merely importing lamindb.

falexwolf · 2024-09-29T16:18:31Z

With the below repo states checked out, we're now at ~2s for lamin connect.

Within lamin load however, the effect seems less drastic as lamindb is fully imported and there is only a ~0.7s improvement for no longer reloading all other schema modules.

One way to escape this would be to no longer fully import lamindb but only use Django + a few helper functions.

lamin connect 2.1s

% python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0               reloaded = False                         
   115         1          0.0      0.0               if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0               return reloaded                          


Total time: 0.000132 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1         12.0     12.0      9.1      log_message = settings.auto_connect                         
   126         1          4.0      4.0      3.0      if not reload_lamindb_itself(isettings):                    
   127         1          1.0      1.0      0.8          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1        115.0    115.0     87.1          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.963623 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                                             
   347                                           def load_from_isettings(                                                             
   348                                               isettings: InstanceSettings,                                                     
   349                                               *,                                                                               
   350                                               init: bool = False,                                                              
   351                                               user: UserSettings | None = None,                                                
   352                                               write_settings: bool = True,                                                     
   353                                           ) -> None:                                                                           
   354         1          4.0      4.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   355                                                                                                                                
   356         1          8.0      8.0      0.0      user = settings.user if user is None else user                                   
   357                                                                                                                                
   358         1          1.0      1.0      0.0      if init:                                                                         
   359                                                   # during init both user and storage need to be registered                    
   360                                                   register_user_and_storage_in_instance(isettings, user)                       
   361                                                   write_bionty_sources(isettings)                                              
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   363                                               else:                                                                            
   364                                                   # when loading, django is already set up                                     
   365                                                   #                                                                            
   366                                                   # only register user if the instance is connected                            
   367                                                   # for the first time in an environment                                       
   368                                                   # this is our best proxy for that the user might not                         
   369                                                   # yet be registered                                                          
   370         1         61.0     61.0      0.0          if not isettings._get_settings_file().exists():                              
   371                                                       register_user(user)                                                      
   372         1     962344.0 962344.0     99.9          load_bionty_sources(isettings)                                               
   373         1       1064.0   1064.0      0.1      isettings._persist(write_to_disk=write_settings)                                 
   374         1        141.0    141.0      0.0      reload_lamindb(isettings)                                                        


Total time: 2.11762 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          1.0      1.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          1.0      1.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          1.0      1.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          4.0      4.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2678.0   2678.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          1.0      0.5      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2095.0   2095.0      0.1              and settings._instance_exists                                              
   235         1          5.0      5.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     461988.0 230994.0     21.8              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1       1298.0   1298.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          1.0      1.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     685887.0 685887.0     32.4          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1     963637.0 963637.0     45.5          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          4.0      4.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1          9.0      9.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          7.0      7.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          1.0      1.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.96 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  2.12 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect

lamin connect 2.8s

% kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                   
   114                                           def reload_lamindb_itself(isettings) -> bool:              
   115         1          0.0      0.0               reloaded = False                                       
   116         1          0.0      0.0               if "lamindb" in sys.modules:                           
   117                                                   import lamindb                                     
   118                                                                                                      
   119                                                   print("reloading lamindb in reload_lamindb_itself")
   120                                                   importlib.reload(lamindb)                          
   121                                                   reloaded = True                                    
   122         1          0.0      0.0               return reloaded                                        


Total time: 0.000116 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 125

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   125                                           @profile                                                        
   126                                           def reload_lamindb(isettings: InstanceSettings):                
   127         1          8.0      8.0      6.9      log_message = settings.auto_connect                         
   128         1          4.0      4.0      3.4      if not reload_lamindb_itself(isettings):                    
   129         1          0.0      0.0      0.0          log_message = True                                      
   130         1          0.0      0.0      0.0      if log_message:                                             
   131         1        104.0    104.0     89.7          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 1.04037 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 348

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   348                                           @profile                                                                             
   349                                           def load_from_isettings(                                                             
   350                                               isettings: InstanceSettings,                                                     
   351                                               *,                                                                               
   352                                               init: bool = False,                                                              
   353                                               user: UserSettings | None = None,                                                
   354                                               write_settings: bool = True,                                                     
   355                                           ) -> None:                                                                           
   356         1          4.0      4.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   357                                                                                                                                
   358         1          9.0      9.0      0.0      user = settings.user if user is None else user                                   
   359                                                                                                                                
   360         1          1.0      1.0      0.0      if init:                                                                         
   361                                                   # during init both user and storage need to be registered                    
   362                                                   register_user_and_storage_in_instance(isettings, user)                       
   363                                                   write_bionty_sources(isettings)                                              
   364                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   365                                               else:                                                                            
   366                                                   # when loading, django is already set up                                     
   367                                                   #                                                                            
   368                                                   # only register user if the instance is connected                            
   369                                                   # for the first time in an environment                                       
   370                                                   # this is our best proxy for that the user might not                         
   371                                                   # yet be registered                                                          
   372         1         50.0     50.0      0.0          if not isettings._get_settings_file().exists():                              
   373                                                       register_user(user)                                                      
   374         1    1039501.0    1e+06     99.9          load_bionty_sources(isettings)                                               
   375         1        677.0    677.0      0.1      isettings._persist(write_to_disk=write_settings)                                 
   376         1        126.0    126.0      0.0      reload_lamindb(isettings)                                                        


Total time: 2.87169 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          1.0      1.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          4.0      4.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       3267.0   3267.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          1.0      0.5      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2136.0   2136.0      0.1              and settings._instance_exists                                              
   235         1          6.0      6.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2    1035505.0 517752.5     36.1              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1        733.0    733.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     789619.0 789619.0     27.5          check, msg = isettings._load_db()                                              
   259         1          1.0      1.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1    1040388.0    1e+06     36.2          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1         13.0     13.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          7.0      7.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:113 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:125 - reload_lamindb
  1.04 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:348 - load_from_isettings
  2.87 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect

lamin load ~4.9s

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000 
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"
Timer unit: 1e-06 s

Total time: 1.70289 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0      0.0      reloaded = False                         
   115         1          1.0      1.0      0.0      if "lamindb" in sys.modules:             
   116         1          1.0      1.0      0.0          import lamindb                       
   117                                                                                        
   118         1    1702889.0    2e+06    100.0          importlib.reload(lamindb)            
   119         1          0.0      0.0      0.0          reloaded = True                      
   120         1          0.0      0.0      0.0      return reloaded                          


Total time: 1.70292 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1         10.0     10.0      0.0      log_message = settings.auto_connect                         
   126         1    1702908.0    2e+06    100.0      if not reload_lamindb_itself(isettings):                    
   127                                                   log_message = True                                      
   128         1          1.0      1.0      0.0      if log_message:                                             
   129                                                   logger.important(f"connected lamindb: {isettings.slug}")


Total time: 2.74184 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                                             
   347                                           def load_from_isettings(                                                             
   348                                               isettings: InstanceSettings,                                                     
   349                                               *,                                                                               
   350                                               init: bool = False,                                                              
   351                                               user: UserSettings | None = None,                                                
   352                                               write_settings: bool = True,                                                     
   353                                           ) -> None:                                                                           
   354         1          5.0      5.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   355                                                                                                                                
   356         1          9.0      9.0      0.0      user = settings.user if user is None else user                                   
   357                                                                                                                                
   358         1          0.0      0.0      0.0      if init:                                                                         
   359                                                   # during init both user and storage need to be registered                    
   360                                                   register_user_and_storage_in_instance(isettings, user)                       
   361                                                   write_bionty_sources(isettings)                                              
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   363                                               else:                                                                            
   364                                                   # when loading, django is already set up                                     
   365                                                   #                                                                            
   366                                                   # only register user if the instance is connected                            
   367                                                   # for the first time in an environment                                       
   368                                                   # this is our best proxy for that the user might not                         
   369                                                   # yet be registered                                                          
   370         1         48.0     48.0      0.0          if not isettings._get_settings_file().exists():                              
   371                                                       register_user(user)                                                      
   372         1    1038260.0    1e+06     37.9          load_bionty_sources(isettings)                                               
   373         1        593.0    593.0      0.0      isettings._persist(write_to_disk=write_settings)                                 
   374         1    1702927.0    2e+06     62.1      reload_lamindb(isettings)                                                        


Total time: 4.55237 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          1.0      1.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          1.0      1.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          2.0      2.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2329.0   2329.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          1.0      1.0      0.0              _write_settings                                                            
   234         1       2201.0   2201.0      0.0              and settings._instance_exists                                              
   235         1          9.0      9.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2    1053804.0 526902.0     23.1              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1        735.0    735.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     751269.0 751269.0     16.5          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1    2741852.0    3e+06     60.2          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          7.0      7.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1         22.0     22.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1        132.0    132.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


Total time: 4.95805 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 19

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    19                                           @profile                                                                                     
    20                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    21         1          2.0      2.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    22         1          1.0      1.0      0.0          url = entity                                                                         
    23         1          3.0      3.0      0.0          instance_slug, entity, uid = decompose_url(url)                                      
    24                                               elif entity not in {"artifact", "transform"}:                                            
    25                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    26                                               else:                                                                                    
    27                                                   instance_slug = None                                                                 
    28                                                                                                                                        
    29         1          0.0      0.0      0.0      if instance_slug is not None:                                                            
    30         1        653.0    653.0      0.0          auto_connect = ln_setup.settings.auto_connect                                        
    31                                                   # we don't want to auto-connect when importing lamindb                               
    32         1          8.0      8.0      0.0          ln_setup.settings.auto_connect = False                                               
    33                                                                                                                                        
    34         1       5432.0   5432.0      0.1          import lamindb as ln                                                                 
    35         1        447.0    447.0      0.0          from lamindb._finish import script_to_notebook                                       
    36                                                                                                                                        
    37         1          8.0      8.0      0.0          ln_setup.settings.auto_connect = auto_connect                                        
    38         1    4552391.0    5e+06     91.8          ln.connect(instance_slug)                                                            
    39                                               else:                                                                                    
    40                                                   import lamindb as ln                                                                 
    41                                                   from lamindb._finish import script_to_notebook                                       
    42                                                                                                                                        
    43                                               # below is to silence warnings about missing run inputs                                  
    44         1          0.0      0.0      0.0      ln.settings.track_run_inputs = False                                                     
    45                                                                                                                                        
    46         1          1.0      1.0      0.0      if entity == "transform":                                                                
    47         1          0.0      0.0      0.0          transform = (                                                                        
    48         1     115429.0 115429.0      2.3              ln.Transform.get(uid)                                                            
    49         1          0.0      0.0      0.0              if uid is not None                                                               
    50                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    51                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    52                                                       else ln.Transform.filter(key=key, source_code__isnull=False)                     
    53                                                       .order_by("-created_at")                                                         
    54                                                       .first()                                                                         
    55                                                   )                                                                                    
    56         1          0.0      0.0      0.0          target_filename = transform.key                                                      
    57                                                   # if Path(target_filename).exists():                                                 
    58                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    59                                                   #     if response != "y":                                                            
    60                                                   #         raise SystemExit("Aborted.")                                               
    61         1          1.0      1.0      0.0          if transform._source_code_artifact_id is not None:                                   
    62                                                       # backward compat                                                                
    63                                                       filepath_cache = transform._source_code_artifact.cache()                         
    64                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    65                                                           target_filename += transform._source_code_artifact.suffix                    
    66                                                       filepath_cache.rename(target_filename)                                           
    67         1          0.0      0.0      0.0          elif transform.source_code is not None:                                              
    68         1          1.0      1.0      0.0              if transform.key.endswith(".ipynb"):                                             
    69         1     283613.0 283613.0      5.7                  script_to_notebook(transform, target_filename, bump_revision=True)           
    70                                                       else:                                                                            
    71                                                           Path(target_filename).write_text(transform.source_code)                      
    72                                                   else:                                                                                
    73                                                       raise SystemExit("No source code available for this transform.")                 
    74         1         65.0     65.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    75         1          0.0      0.0      0.0          if with_env:                                                                         
    76                                                       if (                                                                             
    77                                                           transform.latest_run is not None                                             
    78                                                           and transform.latest_run.environment is not None                             
    79                                                       ):                                                                               
    80                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    81                                                           target_env_filename = (                                                      
    82                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    83                                                           )                                                                            
    84                                                           filepath_env_cache.rename(target_env_filename)                               
    85                                                           logger.important(target_env_filename)                                        
    86                                                       else:                                                                            
    87                                                           logger.warning("latest transform run with environment doesn't exist")        
    88         1          0.0      0.0      0.0          return target_filename                                                               
    89                                               elif entity == "artifact":                                                               
    90                                                   artifact = ln.Artifact.get(uid) if uid is not None else ln.Artifact.get(key=key)     
    91                                                   cache_path = artifact.cache()                                                        
    92                                                   logger.important(f"artifact is here: {cache_path}")                                  
    93                                                   return cache_path                                                                    


  1.70 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  1.70 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  2.74 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  4.55 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  4.96 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:19 - load

lamin load 4.2s

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000
→ connected lamindb: laminlabs/lamindata
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                                            
Timer unit: 1e-06 s

Total time: 1.62664 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0      0.0      reloaded = False                         
   115         1          0.0      0.0      0.0      if "lamindb" in sys.modules:             
   116         1          1.0      1.0      0.0          import lamindb                       
   117                                                                                        
   118         1    1626636.0    2e+06    100.0          importlib.reload(lamindb)            
   119         1          1.0      1.0      0.0          reloaded = True                      
   120         1          0.0      0.0      0.0      return reloaded                          


Total time: 1.6268 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          7.0      7.0      0.0      log_message = settings.auto_connect                         
   126         1    1626657.0    2e+06    100.0      if not reload_lamindb_itself(isettings):                    
   127                                                   log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1        132.0    132.0      0.0          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 2.5877 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                                             
   347                                           def load_from_isettings(                                                             
   348                                               isettings: InstanceSettings,                                                     
   349                                               *,                                                                               
   350                                               init: bool = False,                                                              
   351                                               user: UserSettings | None = None,                                                
   352                                               write_settings: bool = True,                                                     
   353                                           ) -> None:                                                                           
   354         1          5.0      5.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   355                                                                                                                                
   356         1          9.0      9.0      0.0      user = settings.user if user is None else user                                   
   357                                                                                                                                
   358         1          0.0      0.0      0.0      if init:                                                                         
   359                                                   # during init both user and storage need to be registered                    
   360                                                   register_user_and_storage_in_instance(isettings, user)                       
   361                                                   write_bionty_sources(isettings)                                              
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   363                                               else:                                                                            
   364                                                   # when loading, django is already set up                                     
   365                                                   #                                                                            
   366                                                   # only register user if the instance is connected                            
   367                                                   # for the first time in an environment                                       
   368                                                   # this is our best proxy for that the user might not                         
   369                                                   # yet be registered                                                          
   370         1         44.0     44.0      0.0          if not isettings._get_settings_file().exists():                              
   371                                                       register_user(user)                                                      
   372         1     960212.0 960212.0     37.1          load_bionty_sources(isettings)                                               
   373         1        624.0    624.0      0.0      isettings._persist(write_to_disk=write_settings)                                 
   374         1    1626809.0    2e+06     62.9      reload_lamindb(isettings)                                                        


Total time: 3.82277 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          2.0      2.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2145.0   2145.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2074.0   2074.0      0.1              and settings._instance_exists                                              
   235         1          5.0      5.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     540425.0 270212.5     14.1              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          1.0      1.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1        602.0    602.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   255         1          1.0      1.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     689757.0 689757.0     18.0          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1    2587718.0    3e+06     67.7          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          5.0      5.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1         16.0     16.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1         12.0     12.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


Total time: 4.21651 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 19

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    19                                           @profile                                                                                     
    20                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    21         1          2.0      2.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    22         1          0.0      0.0      0.0          url = entity                                                                         
    23         1          4.0      4.0      0.0          instance_slug, entity, uid = decompose_url(url)                                      
    24                                               elif entity not in {"artifact", "transform"}:                                            
    25                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    26                                               else:                                                                                    
    27                                                   instance_slug = None                                                                 
    28                                                                                                                                        
    29         1          0.0      0.0      0.0      if instance_slug is not None:                                                            
    30         1         15.0     15.0      0.0          auto_connect = ln_setup.settings.auto_connect                                        
    31                                                   # we don't want to auto-connect when importing lamindb                               
    32         1         90.0     90.0      0.0          ln_setup.settings.auto_connect = False                                               
    33                                                                                                                                        
    34         1       3791.0   3791.0      0.1          import lamindb as ln                                                                 
    35         1        502.0    502.0      0.0          from lamindb._finish import script_to_notebook                                       
    36                                                                                                                                        
    37         1         76.0     76.0      0.0          ln_setup.settings.auto_connect = auto_connect                                        
    38         1    3822786.0    4e+06     90.7          ln.connect(instance_slug)                                                            
    39                                               else:                                                                                    
    40                                                   import lamindb as ln                                                                 
    41                                                   from lamindb._finish import script_to_notebook                                       
    42                                                                                                                                        
    43                                               # below is to silence warnings about missing run inputs                                  
    44         1          0.0      0.0      0.0      ln.settings.track_run_inputs = False                                                     
    45                                                                                                                                        
    46         1          0.0      0.0      0.0      if entity == "transform":                                                                
    47         1          0.0      0.0      0.0          transform = (                                                                        
    48         1     113723.0 113723.0      2.7              ln.Transform.get(uid)                                                            
    49         1          1.0      1.0      0.0              if uid is not None                                                               
    50                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    51                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    52                                                       else ln.Transform.filter(key=key, source_code__isnull=False)                     
    53                                                       .order_by("-created_at")                                                         
    54                                                       .first()                                                                         
    55                                                   )                                                                                    
    56         1          0.0      0.0      0.0          target_filename = transform.key                                                      
    57                                                   # if Path(target_filename).exists():                                                 
    58                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    59                                                   #     if response != "y":                                                            
    60                                                   #         raise SystemExit("Aborted.")                                               
    61         1          2.0      2.0      0.0          if transform._source_code_artifact_id is not None:                                   
    62                                                       # backward compat                                                                
    63                                                       filepath_cache = transform._source_code_artifact.cache()                         
    64                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    65                                                           target_filename += transform._source_code_artifact.suffix                    
    66                                                       filepath_cache.rename(target_filename)                                           
    67         1          0.0      0.0      0.0          elif transform.source_code is not None:                                              
    68         1          1.0      1.0      0.0              if transform.key.endswith(".ipynb"):                                             
    69         1     275443.0 275443.0      6.5                  script_to_notebook(transform, target_filename, bump_revision=True)           
    70                                                       else:                                                                            
    71                                                           Path(target_filename).write_text(transform.source_code)                      
    72                                                   else:                                                                                
    73                                                       raise SystemExit("No source code available for this transform.")                 
    74         1         72.0     72.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    75         1          0.0      0.0      0.0          if with_env:                                                                         
    76                                                       if (                                                                             
    77                                                           transform.latest_run is not None                                             
    78                                                           and transform.latest_run.environment is not None                             
    79                                                       ):                                                                               
    80                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    81                                                           target_env_filename = (                                                      
    82                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    83                                                           )                                                                            
    84                                                           filepath_env_cache.rename(target_env_filename)                               
    85                                                           logger.important(target_env_filename)                                        
    86                                                       else:                                                                            
    87                                                           logger.warning("latest transform run with environment doesn't exist")        
    88         1          0.0      0.0      0.0          return target_filename                                                               
    89                                               elif entity == "artifact":                                                               
    90                                                   artifact = ln.Artifact.get(uid) if uid is not None else ln.Artifact.get(key=key)     
    91                                                   cache_path = artifact.cache()                                                        
    92                                                   logger.important(f"artifact is here: {cache_path}")                                  
    93                                                   return cache_path                                                                    


  1.63 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  1.63 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  2.59 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  3.82 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  4.22 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:19 - load

falexwolf · 2024-09-29T20:13:05Z

With this refactor, we're now down to ~2.6s for lamin load.

Let's now move the bionty source load into bionty.

lamin load 2.58s

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000
→ connected lamindb: laminlabs/lamindata
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                                            
Timer unit: 1e-06 s

Total time: 1e-06 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0      0.0      reloaded = False                         
   115         1          1.0      1.0    100.0      if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0      0.0      return reloaded                          


Total time: 0.000149 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1         13.0     13.0      8.7      log_message = settings.auto_connect                         
   126         1          6.0      6.0      4.0      if not reload_lamindb_itself(isettings):                    
   127         1          0.0      0.0      0.0          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1        130.0    130.0     87.2          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.928304 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                                             
   347                                           def load_from_isettings(                                                             
   348                                               isettings: InstanceSettings,                                                     
   349                                               *,                                                                               
   350                                               init: bool = False,                                                              
   351                                               user: UserSettings | None = None,                                                
   352                                               write_settings: bool = True,                                                     
   353                                           ) -> None:                                                                           
   354         1          7.0      7.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   355                                                                                                                                
   356         1          8.0      8.0      0.0      user = settings.user if user is None else user                                   
   357                                                                                                                                
   358         1          0.0      0.0      0.0      if init:                                                                         
   359                                                   # during init both user and storage need to be registered                    
   360                                                   register_user_and_storage_in_instance(isettings, user)                       
   361                                                   write_bionty_sources(isettings)                                              
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   363                                               else:                                                                            
   364                                                   # when loading, django is already set up                                     
   365                                                   #                                                                            
   366                                                   # only register user if the instance is connected                            
   367                                                   # for the first time in an environment                                       
   368                                                   # this is our best proxy for that the user might not                         
   369                                                   # yet be registered                                                          
   370         1         45.0     45.0      0.0          if not isettings._get_settings_file().exists():                              
   371                                                       register_user(user)                                                      
   372         1     927186.0 927186.0     99.9          load_bionty_sources(isettings)                                               
   373         1        896.0    896.0      0.1      isettings._persist(write_to_disk=write_settings)                                 
   374         1        162.0    162.0      0.0      reload_lamindb(isettings)                                                        


Total time: 2.2474 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          0.0      0.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          1.0      1.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2764.0   2764.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2162.0   2162.0      0.1              and settings._instance_exists                                              
   235         1          6.0      6.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     645239.0 322619.5     28.7              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          1.0      1.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1        670.0    670.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          2.0      2.0      0.0          silence_loggers()                                                              
   258         1     668188.0 668188.0     29.7          check, msg = isettings._load_db()                                              
   259         1          1.0      1.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1     928325.0 928325.0     41.3          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          4.0      4.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1         13.0     13.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1         15.0     15.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


Total time: 2.58448 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 18

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    18                                           @profile                                                                                     
    19                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    20         1          1.0      1.0      0.0      import lamindb_setup as ln_setup                                                         
    21                                                                                                                                        
    22         1          1.0      1.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    23         1          0.0      0.0      0.0          url = entity                                                                         
    24         1          3.0      3.0      0.0          instance, entity, uid = decompose_url(url)                                           
    25                                               elif entity not in {"artifact", "transform"}:                                            
    26                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    27                                               else:                                                                                    
    28                                                   instance = ln_setup.settings.instance.slug                                           
    29                                                                                                                                        
    30         1    2247421.0    2e+06     87.0      ln_setup.connect(instance)                                                               
    31         1          6.0      6.0      0.0      from lnschema_core import models as ln                                                   
    32                                                                                                                                        
    33         1          1.0      1.0      0.0      def script_to_notebook(                                                                  
    34         1          0.0      0.0      0.0          transform: ln.Transform, notebook_path: Path, bump_revision: bool = False            
    35                                               ) -> None:                                                                               
    36                                                   import jupytext                                                                      
    37                                                   from lamin_utils._base62 import increment_base62                                     
    38                                                                                                                                        
    39                                                   py_content = transform.source_code.replace(                                          
    40                                                       "# # transform.name", f"# # {transform.name}"                                    
    41                                                   )                                                                                    
    42                                                   if bump_revision:                                                                    
    43                                                       uid = transform.uid                                                              
    44                                                       new_uid = f"{uid[:-4]}{increment_base62(uid[-4:])}"                              
    45                                                       py_content = py_content.replace(uid, new_uid)                                    
    46                                                       logger.important(f"updated uid: {uid} → {new_uid}")                              
    47                                                   notebook = jupytext.reads(py_content, fmt="py:percent")                              
    48                                                   jupytext.write(notebook, notebook_path)                                              
    49                                                                                                                                        
    50         1          0.0      0.0      0.0      if entity == "transform":                                                                
    51         1          0.0      0.0      0.0          transform = (                                                                        
    52         1     114732.0 114732.0      4.4              ln.Transform.objects.get(uid=uid)                                                
    53         1          1.0      1.0      0.0              if uid is not None                                                               
    54                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    55                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    56                                                       else ln.Transform.objects.filter(key=key, source_code__isnull=False)             
    57                                                       .order_by("-created_at")                                                         
    58                                                       .first()                                                                         
    59                                                   )                                                                                    
    60         1          0.0      0.0      0.0          target_filename = transform.key                                                      
    61                                                   # if Path(target_filename).exists():                                                 
    62                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    63                                                   #     if response != "y":                                                            
    64                                                   #         raise SystemExit("Aborted.")                                               
    65         1          4.0      4.0      0.0          if transform._source_code_artifact_id is not None:  # backward compat                
    66                                                       # need lamindb here to have .cache() available                                   
    67                                                       import lamindb as ln                                                             
    68                                                                                                                                        
    69                                                       ln.settings.track_run_inputs = False                                             
    70                                                       filepath_cache = transform._source_code_artifact.cache()                         
    71                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    72                                                           target_filename += transform._source_code_artifact.suffix                    
    73                                                       filepath_cache.rename(target_filename)                                           
    74         1          0.0      0.0      0.0          elif transform.source_code is not None:                                              
    75         1          3.0      3.0      0.0              if transform.key.endswith(".ipynb"):                                             
    76         1     222236.0 222236.0      8.6                  script_to_notebook(transform, target_filename, bump_revision=True)           
    77                                                       else:                                                                            
    78                                                           Path(target_filename).write_text(transform.source_code)                      
    79                                                   else:                                                                                
    80                                                       raise SystemExit("No source code available for this transform.")                 
    81         1         72.0     72.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    82         1          1.0      1.0      0.0          if with_env:                                                                         
    83                                                       import lamindb as ln                                                             
    84                                                                                                                                        
    85                                                       ln.settings.track_run_inputs = False                                             
    86                                                       if (                                                                             
    87                                                           transform.latest_run is not None                                             
    88                                                           and transform.latest_run.environment is not None                             
    89                                                       ):                                                                               
    90                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    91                                                           target_env_filename = (                                                      
    92                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    93                                                           )                                                                            
    94                                                           filepath_env_cache.rename(target_env_filename)                               
    95                                                           logger.important(target_env_filename)                                        
    96                                                       else:                                                                            
    97                                                           logger.warning("latest transform run with environment doesn't exist")        
    98         1          0.0      0.0      0.0          return target_filename                                                               
    99                                               elif entity == "artifact":                                                               
   100                                                   import lamindb as ln                                                                 
   101                                                                                                                                        
   102                                                   ln.settings.track_run_inputs = False                                                 
   103                                                   artifact = (                                                                         
   104                                                       ln.Artifact.get(uid)                                                             
   105                                                       if uid is not None                                                               
   106                                                       else ln.Artifact.filter(key=key, source_code__isnull=False)                      
   107                                                       .order_by("-created_at")                                                         
   108                                                       .first()                                                                         
   109                                                   )                                                                                    
   110                                                   cache_path = artifact.cache()                                                        
   111                                                   logger.important(f"artifact is here: {cache_path}")                                  
   112                                                   return cache_path                                                                    


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.93 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  2.25 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  2.58 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:18 - load

lamin load 2.67s

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000
→ connected lamindb: laminlabs/lamindata
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                                            
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0               reloaded = False                         
   115         1          0.0      0.0               if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0               return reloaded                          


Total time: 0.000109 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          9.0      9.0      8.3      log_message = settings.auto_connect                         
   126         1          4.0      4.0      3.7      if not reload_lamindb_itself(isettings):                    
   127         1          0.0      0.0      0.0          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1         96.0     96.0     88.1          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.957393 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                                             
   347                                           def load_from_isettings(                                                             
   348                                               isettings: InstanceSettings,                                                     
   349                                               *,                                                                               
   350                                               init: bool = False,                                                              
   351                                               user: UserSettings | None = None,                                                
   352                                               write_settings: bool = True,                                                     
   353                                           ) -> None:                                                                           
   354         1          4.0      4.0      0.0      from .core._setup_bionty_sources import load_bionty_sources, write_bionty_sources
   355                                                                                                                                
   356         1          8.0      8.0      0.0      user = settings.user if user is None else user                                   
   357                                                                                                                                
   358         1          0.0      0.0      0.0      if init:                                                                         
   359                                                   # during init both user and storage need to be registered                    
   360                                                   register_user_and_storage_in_instance(isettings, user)                       
   361                                                   write_bionty_sources(isettings)                                              
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)               
   363                                               else:                                                                            
   364                                                   # when loading, django is already set up                                     
   365                                                   #                                                                            
   366                                                   # only register user if the instance is connected                            
   367                                                   # for the first time in an environment                                       
   368                                                   # this is our best proxy for that the user might not                         
   369                                                   # yet be registered                                                          
   370         1         48.0     48.0      0.0          if not isettings._get_settings_file().exists():                              
   371                                                       register_user(user)                                                      
   372         1     956551.0 956551.0     99.9          load_bionty_sources(isettings)                                               
   373         1        665.0    665.0      0.1      isettings._persist(write_to_disk=write_settings)                                 
   374         1        117.0    117.0      0.0      reload_lamindb(isettings)                                                        


Total time: 2.2763 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          0.0      0.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          2.0      2.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       3298.0   3298.0      0.1          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          1.0      0.5      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2168.0   2168.0      0.1              and settings._instance_exists                                              
   235         1          8.0      8.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     501835.0 250917.5     22.0              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          1.0      1.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1       1566.0   1566.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          2.0      2.0      0.0          silence_loggers()                                                              
   258         1     809988.0 809988.0     35.6          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1     957407.0 957407.0     42.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1         11.0     11.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          6.0      6.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


Total time: 2.66967 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 36

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    36                                           @profile                                                                                                                                              
    37                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):                                                                      
    38         1          1.0      1.0      0.0      import lamindb_setup as ln_setup                                                                                                                  
    39                                                                                                                                                                                                 
    40         1          1.0      1.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                                                                           
    41         1          0.0      0.0      0.0          url = entity                                                                                                                                  
    42         1          4.0      4.0      0.0          instance, entity, uid = decompose_url(url)                                                                                                    
    43                                               elif entity not in {"artifact", "transform"}:                                                                                                     
    44                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")                                                              
    45                                               else:                                                                                                                                             
    46                                                   instance = ln_setup.settings.instance.slug                                                                                                    
    47                                                                                                                                                                                                 
    48         1    2276315.0    2e+06     85.3      ln_setup.connect(instance)                                                                                                                        
    49         1          6.0      6.0      0.0      from lnschema_core import models as ln                                                                                                            
    50                                                                                                                                                                                                 
    51         1          0.0      0.0      0.0      if entity == "transform":                                                                                                                         
    52         1          0.0      0.0      0.0          transform = (                                                                                                                                 
    53         1     113361.0 113361.0      4.2              ln.Transform.objects.get(uid=uid)                                                                                                         
    54         1          0.0      0.0      0.0              if uid is not None                                                                                                                        
    55                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks                                                         
    56                                                       # hence, we use source_code__isnull=False and order by created_at instead                                                                 
    57                                                       else ln.Transform.objects.filter(key=key, source_code__isnull=False).order_by("-created_at").first()                                      
    58                                                   )                                                                                                                                             
    59         1          0.0      0.0      0.0          target_filename = transform.key                                                                                                               
    60                                                   # if Path(target_filename).exists():                                                                                                          
    61                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                                                                         
    62                                                   #     if response != "y":                                                                                                                     
    63                                                   #         raise SystemExit("Aborted.")                                                                                                        
    64         1          7.0      7.0      0.0          if transform._source_code_artifact_id is not None:  # backward compat                                                                         
    65                                                       # need lamindb here to have .cache() available                                                                                            
    66                                                       import lamindb as ln                                                                                                                      
    67                                                                                                                                                                                                 
    68                                                       ln.settings.track_run_inputs = False                                                                                                      
    69                                                       filepath_cache = transform._source_code_artifact.cache()                                                                                  
    70                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):                                                                  
    71                                                           target_filename += transform._source_code_artifact.suffix                                                                             
    72                                                       filepath_cache.rename(target_filename)                                                                                                    
    73         1          0.0      0.0      0.0          elif transform.source_code is not None:                                                                                                       
    74         1          3.0      3.0      0.0              if transform.key.endswith(".ipynb"):                                                                                                      
    75         1     279888.0 279888.0     10.5                  script_to_notebook(transform, target_filename, bump_revision=True)                                                                    
    76                                                       else:                                                                                                                                     
    77                                                           Path(target_filename).write_text(transform.source_code)                                                                               
    78                                                   else:                                                                                                                                         
    79                                                       raise SystemExit("No source code available for this transform.")                                                                          
    80         1         88.0     88.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                                                                              
    81         1          0.0      0.0      0.0          if with_env:                                                                                                                                  
    82                                                       import lamindb as ln                                                                                                                      
    83                                                                                                                                                                                                 
    84                                                       ln.settings.track_run_inputs = False                                                                                                      
    85                                                       if (                                                                                                                                      
    86                                                           transform.latest_run is not None                                                                                                      
    87                                                           and transform.latest_run.environment is not None                                                                                      
    88                                                       ):                                                                                                                                        
    89                                                           filepath_env_cache = transform.latest_run.environment.cache()                                                                         
    90                                                           target_env_filename = (                                                                                                               
    91                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"                                                                  
    92                                                           )                                                                                                                                     
    93                                                           filepath_env_cache.rename(target_env_filename)                                                                                        
    94                                                           logger.important(target_env_filename)                                                                                                 
    95                                                       else:                                                                                                                                     
    96                                                           logger.warning("latest transform run with environment doesn't exist")                                                                 
    97         1          0.0      0.0      0.0          return target_filename                                                                                                                        
    98                                               elif entity == "artifact":                                                                                                                        
    99                                                   import lamindb as ln                                                                                                                          
   100                                                                                                                                                                                                 
   101                                                   ln.settings.track_run_inputs = False                                                                                                          
   102                                                   artifact = ln.Artifact.get(uid) if uid is not None else ln.Artifact.filter(key=key, source_code__isnull=False).order_by("-created_at").first()
   103                                                   cache_path = artifact.cache()                                                                                                                 
   104                                                   logger.important(f"artifact is here: {cache_path}")                                                                                           
   105                                                   return cache_path                                                                                                                             


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.96 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  2.28 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  2.67 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:36 - load

falexwolf · 2024-09-29T20:37:24Z

Moved loading bionty sources to bionty brings us to ~2.2s

ln.connect even went down from 2s to 1s, but we don't see all of this speed up in lamin load because Transform.objects.get() runtime increase from 0.1s to 0.9s.

This is because before we paid the price for establishing the very first database connection through Django during ln.connect(), and now we pay it during Transform.objects.get().

The two biggest attack points are now:

not having django.setup() set up all schema modules; it contributes 0.6s to lamin connect now
removing the need to establish a Django connection to get the source code, which will likely give us another 0.9s

lamin connect 1.2s

% kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0               reloaded = False                         
   115         1          0.0      0.0               if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0               return reloaded                          


Total time: 0.000108 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          8.0      8.0      7.4      log_message = settings.auto_connect                         
   126         1          4.0      4.0      3.7      if not reload_lamindb_itself(isettings):                    
   127         1          0.0      0.0      0.0          log_message = True                                      
   128         1          1.0      1.0      0.9      if log_message:                                             
   129         1         95.0     95.0     88.0          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.00077 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                              
   347                                           def load_from_isettings(                                              
   348                                               isettings: InstanceSettings,                                      
   349                                               *,                                                                
   350                                               init: bool = False,                                               
   351                                               user: UserSettings | None = None,                                 
   352                                               write_settings: bool = True,                                      
   353                                           ) -> None:                                                            
   354         1          4.0      4.0      0.5      from .core._setup_bionty_sources import write_bionty_sources      
   355                                                                                                                 
   356         1          8.0      8.0      1.0      user = settings.user if user is None else user                    
   357                                                                                                                 
   358         1          0.0      0.0      0.0      if init:                                                          
   359                                                   # during init both user and storage need to be registered     
   360                                                   register_user_and_storage_in_instance(isettings, user)        
   361                                                   write_bionty_sources(isettings)                               
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)
   363                                               else:                                                             
   364                                                   # when loading, django is already set up                      
   365                                                   #                                                             
   366                                                   # only register user if the instance is connected             
   367                                                   # for the first time in an environment                        
   368                                                   # this is our best proxy for that the user might not          
   369                                                   # yet be registered                                           
   370         1         40.0     40.0      5.2          if not isettings._get_settings_file().exists():               
   371                                                       register_user(user)                                       
   372         1        604.0    604.0     78.4      isettings._persist(write_to_disk=write_settings)                  
   373         1        114.0    114.0     14.8      reload_lamindb(isettings)                                         


Total time: 1.20296 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          3.0      3.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          5.0      5.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       3186.0   3186.0      0.3          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2170.0   2170.0      0.2              and settings._instance_exists                                              
   235         1          7.0      7.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     426542.0 213271.0     35.5              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1       1165.0   1165.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          1.0      1.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     769072.0 769072.0     63.9          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1        784.0    784.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          2.0      2.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1          8.0      8.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1         11.0     11.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  1.20 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect

lamin connect 1.35s

% python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0               reloaded = False                         
   115         1          0.0      0.0               if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0               return reloaded                          


Total time: 0.000101 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          6.0      6.0      5.9      log_message = settings.auto_connect                         
   126         1          4.0      4.0      4.0      if not reload_lamindb_itself(isettings):                    
   127         1          1.0      1.0      1.0          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1         90.0     90.0     89.1          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.000829 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                              
   347                                           def load_from_isettings(                                              
   348                                               isettings: InstanceSettings,                                      
   349                                               *,                                                                
   350                                               init: bool = False,                                               
   351                                               user: UserSettings | None = None,                                 
   352                                               write_settings: bool = True,                                      
   353                                           ) -> None:                                                            
   354         1          5.0      5.0      0.6      from .core._setup_bionty_sources import write_bionty_sources      
   355                                                                                                                 
   356         1         11.0     11.0      1.3      user = settings.user if user is None else user                    
   357                                                                                                                 
   358         1          0.0      0.0      0.0      if init:                                                          
   359                                                   # during init both user and storage need to be registered     
   360                                                   register_user_and_storage_in_instance(isettings, user)        
   361                                                   write_bionty_sources(isettings)                               
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)
   363                                               else:                                                             
   364                                                   # when loading, django is already set up                      
   365                                                   #                                                             
   366                                                   # only register user if the instance is connected             
   367                                                   # for the first time in an environment                        
   368                                                   # this is our best proxy for that the user might not          
   369                                                   # yet be registered                                           
   370         1         49.0     49.0      5.9          if not isettings._get_settings_file().exists():               
   371                                                       register_user(user)                                       
   372         1        660.0    660.0     79.6      isettings._persist(write_to_disk=write_settings)                  
   373         1        104.0    104.0     12.5      reload_lamindb(isettings)                                         


Total time: 1.34683 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          1.0      1.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2712.0   2712.0      0.2          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2086.0   2086.0      0.2              and settings._instance_exists                                              
   235         1          7.0      7.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     628111.0 314055.5     46.6              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1       1705.0   1705.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     711352.0 711352.0     52.8          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1        837.0    837.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1          7.0      7.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          6.0      6.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          1.0      1.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  1.35 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
(py310) falexwolf@mbpalex lamindb-setup %

lamin load 2.2s

% python -m line_profiler -rmt "lamin.lprof"                                            
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0               reloaded = False                         
   115         1          0.0      0.0               if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0               return reloaded                          


Total time: 0.00011 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          8.0      8.0      7.3      log_message = settings.auto_connect                         
   126         1          4.0      4.0      3.6      if not reload_lamindb_itself(isettings):                    
   127         1          0.0      0.0      0.0          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1         98.0     98.0     89.1          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.00084 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                              
   347                                           def load_from_isettings(                                              
   348                                               isettings: InstanceSettings,                                      
   349                                               *,                                                                
   350                                               init: bool = False,                                               
   351                                               user: UserSettings | None = None,                                 
   352                                               write_settings: bool = True,                                      
   353                                           ) -> None:                                                            
   354         1          4.0      4.0      0.5      from .core._setup_bionty_sources import write_bionty_sources      
   355                                                                                                                 
   356         1         10.0     10.0      1.2      user = settings.user if user is None else user                    
   357                                                                                                                 
   358         1          0.0      0.0      0.0      if init:                                                          
   359                                                   # during init both user and storage need to be registered     
   360                                                   register_user_and_storage_in_instance(isettings, user)        
   361                                                   write_bionty_sources(isettings)                               
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)
   363                                               else:                                                             
   364                                                   # when loading, django is already set up                      
   365                                                   #                                                             
   366                                                   # only register user if the instance is connected             
   367                                                   # for the first time in an environment                        
   368                                                   # this is our best proxy for that the user might not          
   369                                                   # yet be registered                                           
   370         1         52.0     52.0      6.2          if not isettings._get_settings_file().exists():               
   371                                                       register_user(user)                                       
   372         1        656.0    656.0     78.1      isettings._persist(write_to_disk=write_settings)                  
   373         1        118.0    118.0     14.0      reload_lamindb(isettings)                                         


Total time: 1.08633 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          0.0      0.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          1.0      1.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2725.0   2725.0      0.3          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          2.0      1.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2117.0   2117.0      0.2              and settings._instance_exists                                              
   235         1          5.0      5.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     425182.0 212591.0     39.1              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          1.0      1.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1        857.0    857.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     654560.0 654560.0     60.3          check, msg = isettings._load_db()                                              
   259         1          2.0      2.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1        850.0    850.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          1.0      1.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1          8.0      8.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          6.0      6.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          1.0      1.0      0.0      return None                                                                        


Total time: 2.26622 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 18

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    18                                           @profile                                                                                     
    19                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    20         1          1.0      1.0      0.0      import lamindb_setup as ln_setup                                                         
    21                                                                                                                                        
    22         1          1.0      1.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    23         1          0.0      0.0      0.0          url = entity                                                                         
    24         1          3.0      3.0      0.0          instance, entity, uid = decompose_url(url)                                           
    25                                               elif entity not in {"artifact", "transform"}:                                            
    26                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    27                                               else:                                                                                    
    28                                                   instance = ln_setup.settings.instance.slug                                           
    29                                                                                                                                        
    30         1    1086343.0    1e+06     47.9      ln_setup.connect(instance)                                                               
    31         1          2.0      2.0      0.0      from lnschema_core import models as ln                                                   
    32                                                                                                                                        
    33         1          0.0      0.0      0.0      def script_to_notebook(                                                                  
    34         1          1.0      1.0      0.0          transform: ln.Transform, notebook_path: Path, bump_revision: bool = False            
    35                                               ) -> None:                                                                               
    36                                                   import jupytext                                                                      
    37                                                   from lamin_utils._base62 import increment_base62                                     
    38                                                                                                                                        
    39                                                   py_content = transform.source_code.replace(                                          
    40                                                       "# # transform.name", f"# # {transform.name}"                                    
    41                                                   )                                                                                    
    42                                                   if bump_revision:                                                                    
    43                                                       uid = transform.uid                                                              
    44                                                       new_uid = f"{uid[:-4]}{increment_base62(uid[-4:])}"                              
    45                                                       py_content = py_content.replace(uid, new_uid)                                    
    46                                                       logger.important(f"updated uid: {uid} → {new_uid}")                              
    47                                                   notebook = jupytext.reads(py_content, fmt="py:percent")                              
    48                                                   jupytext.write(notebook, notebook_path)                                              
    49                                                                                                                                        
    50         1          1.0      1.0      0.0      if entity == "transform":                                                                
    51         1          0.0      0.0      0.0          transform = (                                                                        
    52         1     943167.0 943167.0     41.6              ln.Transform.objects.get(uid=uid)                                                
    53         1          0.0      0.0      0.0              if uid is not None                                                               
    54                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    55                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    56                                                       else ln.Transform.objects.filter(key=key, source_code__isnull=False)             
    57                                                       .order_by("-created_at")                                                         
    58                                                       .first()                                                                         
    59                                                   )                                                                                    
    60         1          0.0      0.0      0.0          target_filename = transform.key                                                      
    61                                                   # if Path(target_filename).exists():                                                 
    62                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    63                                                   #     if response != "y":                                                            
    64                                                   #         raise SystemExit("Aborted.")                                               
    65         1          5.0      5.0      0.0          if transform._source_code_artifact_id is not None:  # backward compat                
    66                                                       # need lamindb here to have .cache() available                                   
    67                                                       import lamindb as ln                                                             
    68                                                                                                                                        
    69                                                       ln.settings.track_run_inputs = False                                             
    70                                                       filepath_cache = transform._source_code_artifact.cache()                         
    71                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    72                                                           target_filename += transform._source_code_artifact.suffix                    
    73                                                       filepath_cache.rename(target_filename)                                           
    74         1          1.0      1.0      0.0          elif transform.source_code is not None:                                              
    75         1          3.0      3.0      0.0              if transform.key.endswith(".ipynb"):                                             
    76         1     236625.0 236625.0     10.4                  script_to_notebook(transform, target_filename, bump_revision=True)           
    77                                                       else:                                                                            
    78                                                           Path(target_filename).write_text(transform.source_code)                      
    79                                                   else:                                                                                
    80                                                       raise SystemExit("No source code available for this transform.")                 
    81         1         68.0     68.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    82         1          0.0      0.0      0.0          if with_env:                                                                         
    83                                                       import lamindb as ln                                                             
    84                                                                                                                                        
    85                                                       ln.settings.track_run_inputs = False                                             
    86                                                       if (                                                                             
    87                                                           transform.latest_run is not None                                             
    88                                                           and transform.latest_run.environment is not None                             
    89                                                       ):                                                                               
    90                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    91                                                           target_env_filename = (                                                      
    92                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    93                                                           )                                                                            
    94                                                           filepath_env_cache.rename(target_env_filename)                               
    95                                                           logger.important(target_env_filename)                                        
    96                                                       else:                                                                            
    97                                                           logger.warning("latest transform run with environment doesn't exist")        
    98         1          0.0      0.0      0.0          return target_filename                                                               
    99                                               elif entity == "artifact":                                                               
   100                                                   import lamindb as ln                                                                 
   101                                                                                                                                        
   102                                                   ln.settings.track_run_inputs = False                                                 
   103                                                   artifact = (                                                                         
   104                                                       ln.Artifact.get(uid)                                                             
   105                                                       if uid is not None                                                               
   106                                                       else ln.Artifact.filter(key=key, source_code__isnull=False)                      
   107                                                       .order_by("-created_at")                                                         
   108                                                       .first()                                                                         
   109                                                   )                                                                                    
   110                                                   cache_path = artifact.cache()                                                        
   111                                                   logger.important(f"artifact is here: {cache_path}")                                  
   112                                                   return cache_path                                                                    


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  1.09 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  2.27 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:18 - load

lamin load 2.43s

% kernprof -l lamin load https://lamin.ai/laminlabs/lamindata/transform/3qIZOmDUVBwc0000
→ connected lamindb: laminlabs/lamindata
→ updated uid: 3qIZOmDUVBwc0000 → 3qIZOmDUVBwc0001
→ notebook is here: demo.ipynb
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                                            
Timer unit: 1e-06 s

Total time: 1e-06 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb_itself at line 112

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   112                                           @profile                                     
   113                                           def reload_lamindb_itself(isettings) -> bool:
   114         1          0.0      0.0      0.0      reloaded = False                         
   115         1          1.0      1.0    100.0      if "lamindb" in sys.modules:             
   116                                                   import lamindb                       
   117                                                                                        
   118                                                   importlib.reload(lamindb)            
   119                                                   reloaded = True                      
   120         1          0.0      0.0      0.0      return reloaded                          


Total time: 0.0001 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: reload_lamindb at line 123

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   123                                           @profile                                                        
   124                                           def reload_lamindb(isettings: InstanceSettings):                
   125         1          8.0      8.0      8.0      log_message = settings.auto_connect                         
   126         1          4.0      4.0      4.0      if not reload_lamindb_itself(isettings):                    
   127         1          0.0      0.0      0.0          log_message = True                                      
   128         1          0.0      0.0      0.0      if log_message:                                             
   129         1         88.0     88.0     88.0          logger.important(f"connected lamindb: {isettings.slug}")


Total time: 0.000726 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py
Function: load_from_isettings at line 346

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   346                                           @profile                                                              
   347                                           def load_from_isettings(                                              
   348                                               isettings: InstanceSettings,                                      
   349                                               *,                                                                
   350                                               init: bool = False,                                               
   351                                               user: UserSettings | None = None,                                 
   352                                               write_settings: bool = True,                                      
   353                                           ) -> None:                                                            
   354         1          3.0      3.0      0.4      from .core._setup_bionty_sources import write_bionty_sources      
   355                                                                                                                 
   356         1          6.0      6.0      0.8      user = settings.user if user is None else user                    
   357                                                                                                                 
   358         1          0.0      0.0      0.0      if init:                                                          
   359                                                   # during init both user and storage need to be registered     
   360                                                   register_user_and_storage_in_instance(isettings, user)        
   361                                                   write_bionty_sources(isettings)                               
   362                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)
   363                                               else:                                                             
   364                                                   # when loading, django is already set up                      
   365                                                   #                                                             
   366                                                   # only register user if the instance is connected             
   367                                                   # for the first time in an environment                        
   368                                                   # this is our best proxy for that the user might not          
   369                                                   # yet be registered                                           
   370         1         38.0     38.0      5.2          if not isettings._get_settings_file().exists():               
   371                                                       register_user(user)                                       
   372         1        573.0    573.0     78.9      isettings._persist(write_to_disk=write_settings)                  
   373         1        106.0    106.0     14.6      reload_lamindb(isettings)                                         


Total time: 1.28602 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 187

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   187                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   188                                           @profile                                                                               
   189                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   190                                               """Connect to instance.                                                            
   191                                                                                                                                  
   192                                               Args:                                                                              
   193                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   194                                                       If the instance is owned by you, it suffices to pass the instance name.    
   195                                               """                                                                                
   196                                               # validate kwargs                                                                  
   197         1          0.0      0.0      0.0      valid_kwargs = {                                                                   
   198                                                   "_db",                                                                         
   199                                                   "_write_settings",                                                             
   200                                                   "_raise_not_found_error",                                                      
   201                                                   "_test",                                                                       
   202                                                   "_user",                                                                       
   203                                               }                                                                                  
   204         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   205                                                   if kwarg not in valid_kwargs:                                                  
   206                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   207                                                                                                                                  
   208         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   209                                               # _db is still needed because it is called in init                                 
   210         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   211         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   212         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   213         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   216         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   217         1          0.0      0.0      0.0      if _user is not None:                                                              
   218                                                   access_token = _user.access_token                                              
   219                                                                                                                                  
   220         1          0.0      0.0      0.0      try:                                                                               
   221         1          5.0      5.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   222                                                                                                                                  
   223         1       2727.0   2727.0      0.2          if _check_instance_setup() and not _test:                                      
   224                                                       if (                                                                       
   225                                                           settings._instance_exists                                              
   226                                                           and f"{owner}/{name}" == settings.instance.slug                        
   227                                                       ):                                                                         
   228                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   229                                                           return None                                                            
   230                                                       else:                                                                      
   231                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   232         2          0.0      0.0      0.0          elif (                                                                         
   233         1          0.0      0.0      0.0              _write_settings                                                            
   234         1       2118.0   2118.0      0.2              and settings._instance_exists                                              
   235         1          6.0      6.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   236                                                   ):                                                                             
   237                                                       close_instance(mute=True)                                                  
   238                                                                                                                                  
   239         1          0.0      0.0      0.0          try:                                                                           
   240         2     602996.0 301498.0     46.9              isettings = _connect_instance(                                             
   241         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   242                                                       )                                                                          
   243                                                   except InstanceNotFoundError as e:                                             
   244                                                       if _raise_not_found_error:                                                 
   245                                                           raise e                                                                
   246                                                       else:                                                                      
   247                                                           return "instance-not-found"                                            
   248         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   249                                                       return isettings                                                           
   250                                                   # at this point we have checked already that isettings is not a string         
   251                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   252                                                   # has no effect if _user is None or if not cloud sqlite instance               
   253         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   254         1       1499.0   1499.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   255         1          0.0      0.0      0.0          if _test:                                                                      
   256                                                       return None                                                                
   257         1          1.0      1.0      0.0          silence_loggers()                                                              
   258         1     675913.0 675913.0     52.6          check, msg = isettings._load_db()                                              
   259         1          0.0      0.0      0.0          if not check:                                                                  
   260                                                       local_db = (                                                               
   261                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   262                                                       )                                                                          
   263                                                       if local_db:                                                               
   264                                                           logger.warning(                                                        
   265                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   266                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   267                                                               " call: lamin load --unload"                                       
   268                                                           )                                                                      
   269                                                       elif _raise_not_found_error:                                               
   270                                                           raise SystemExit(msg)                                                  
   271                                                       else:                                                                      
   272                                                           logger.warning(                                                        
   273                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   274                                                               " loadable: re-initializing"                                       
   275                                                           )                                                                      
   276                                                           return "instance-corrupted-or-deleted"                                 
   277                                                   # this is for testing purposes only                                            
   278         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   279                                                       raise RuntimeError("Technical testing error.")                             
   280                                                                                                                                  
   281                                                   # below is for backfilling the instance_uid value                              
   282                                                   # we'll enable it once more people migrated to 0.71.0                          
   283                                                   # ssettings_record = isettings.storage.record                                  
   284                                                   # if ssettings_record.instance_uid is None:                                    
   285                                                   #     ssettings_record.instance_uid = isettings.uid                            
   286                                                   #     # try saving if not read-only access                                     
   287                                                   #     try:                                                                     
   288                                                   #         ssettings_record.save()                                              
   289                                                   #     # raised by django when the access is denied                             
   290                                                   #     except ProgrammingError:                                                 
   291                                                   #         pass                                                                 
   292         1        741.0    741.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   293                                               except Exception as e:                                                             
   294                                                   if isettings is not None:                                                      
   295                                                       if _write_settings:                                                        
   296                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   297                                                       settings._instance_settings = None                                         
   298                                                   raise e                                                                        
   299                                               # rename lnschema_bionty to bionty for sql tables                                  
   300         1          2.0      2.0      0.0      if "bionty" in isettings.schema:                                                   
   301         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   302         1          7.0      7.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   303                                                   )                                                                              
   304         1          7.0      7.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   305                                                       migrate_lnschema_bionty(                                                   
   306                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   307                                                       )                                                                          
   308         1          0.0      0.0      0.0      return None                                                                        


Total time: 2.43147 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py
Function: load at line 18

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    18                                           @profile                                                                                     
    19                                           def load(entity: str, uid: str = None, key: str = None, with_env: bool = False):             
    20         1          1.0      1.0      0.0      import lamindb_setup as ln_setup                                                         
    21                                                                                                                                        
    22         1          1.0      1.0      0.0      if entity.startswith("https://") and "lamin" in entity:                                  
    23         1          0.0      0.0      0.0          url = entity                                                                         
    24         1          4.0      4.0      0.0          instance, entity, uid = decompose_url(url)                                           
    25                                               elif entity not in {"artifact", "transform"}:                                            
    26                                                   raise SystemExit("Entity has to be a laminhub URL or 'artifact' or 'transform'")     
    27                                               else:                                                                                    
    28                                                   instance = ln_setup.settings.instance.slug                                           
    29                                                                                                                                        
    30         1    1286047.0    1e+06     52.9      ln_setup.connect(instance)                                                               
    31         1          2.0      2.0      0.0      from lnschema_core import models as ln                                                   
    32                                                                                                                                        
    33         1          1.0      1.0      0.0      def script_to_notebook(                                                                  
    34         1          0.0      0.0      0.0          transform: ln.Transform, notebook_path: Path, bump_revision: bool = False            
    35                                               ) -> None:                                                                               
    36                                                   import jupytext                                                                      
    37                                                   from lamin_utils._base62 import increment_base62                                     
    38                                                                                                                                        
    39                                                   py_content = transform.source_code.replace(                                          
    40                                                       "# # transform.name", f"# # {transform.name}"                                    
    41                                                   )                                                                                    
    42                                                   if bump_revision:                                                                    
    43                                                       uid = transform.uid                                                              
    44                                                       new_uid = f"{uid[:-4]}{increment_base62(uid[-4:])}"                              
    45                                                       py_content = py_content.replace(uid, new_uid)                                    
    46                                                       logger.important(f"updated uid: {uid} → {new_uid}")                              
    47                                                   notebook = jupytext.reads(py_content, fmt="py:percent")                              
    48                                                   jupytext.write(notebook, notebook_path)                                              
    49                                                                                                                                        
    50         1          0.0      0.0      0.0      if entity == "transform":                                                                
    51         1          1.0      1.0      0.0          transform = (                                                                        
    52         1     923597.0 923597.0     38.0              ln.Transform.objects.get(uid=uid)                                                
    53         1          0.0      0.0      0.0              if uid is not None                                                               
    54                                                       # if below, we take is_latest=True as the criterion, we might get draft notebooks
    55                                                       # hence, we use source_code__isnull=False and order by created_at instead        
    56                                                       else ln.Transform.objects.filter(key=key, source_code__isnull=False)             
    57                                                       .order_by("-created_at")                                                         
    58                                                       .first()                                                                         
    59                                                   )                                                                                    
    60         1          1.0      1.0      0.0          target_filename = transform.key                                                      
    61                                                   # if Path(target_filename).exists():                                                 
    62                                                   #     response = input(f"! {target_filename} exists: replace? (y/n)")                
    63                                                   #     if response != "y":                                                            
    64                                                   #         raise SystemExit("Aborted.")                                               
    65         1          5.0      5.0      0.0          if transform._source_code_artifact_id is not None:  # backward compat                
    66                                                       # need lamindb here to have .cache() available                                   
    67                                                       import lamindb as ln                                                             
    68                                                                                                                                        
    69                                                       ln.settings.track_run_inputs = False                                             
    70                                                       filepath_cache = transform._source_code_artifact.cache()                         
    71                                                       if not target_filename.endswith(transform._source_code_artifact.suffix):         
    72                                                           target_filename += transform._source_code_artifact.suffix                    
    73                                                       filepath_cache.rename(target_filename)                                           
    74         1          0.0      0.0      0.0          elif transform.source_code is not None:                                              
    75         1          3.0      3.0      0.0              if transform.key.endswith(".ipynb"):                                             
    76         1     221746.0 221746.0      9.1                  script_to_notebook(transform, target_filename, bump_revision=True)           
    77                                                       else:                                                                            
    78                                                           Path(target_filename).write_text(transform.source_code)                      
    79                                                   else:                                                                                
    80                                                       raise SystemExit("No source code available for this transform.")                 
    81         1         64.0     64.0      0.0          logger.important(f"{transform.type} is here: {target_filename}")                     
    82         1          0.0      0.0      0.0          if with_env:                                                                         
    83                                                       import lamindb as ln                                                             
    84                                                                                                                                        
    85                                                       ln.settings.track_run_inputs = False                                             
    86                                                       if (                                                                             
    87                                                           transform.latest_run is not None                                             
    88                                                           and transform.latest_run.environment is not None                             
    89                                                       ):                                                                               
    90                                                           filepath_env_cache = transform.latest_run.environment.cache()                
    91                                                           target_env_filename = (                                                      
    92                                                               ".".join(target_filename.split(".")[:-1]) + "__requirements.txt"         
    93                                                           )                                                                            
    94                                                           filepath_env_cache.rename(target_env_filename)                               
    95                                                           logger.important(target_env_filename)                                        
    96                                                       else:                                                                            
    97                                                           logger.warning("latest transform run with environment doesn't exist")        
    98         1          0.0      0.0      0.0          return target_filename                                                               
    99                                               elif entity == "artifact":                                                               
   100                                                   import lamindb as ln                                                                 
   101                                                                                                                                        
   102                                                   ln.settings.track_run_inputs = False                                                 
   103                                                   artifact = (                                                                         
   104                                                       ln.Artifact.get(uid)                                                             
   105                                                       if uid is not None                                                               
   106                                                       else ln.Artifact.filter(key=key, source_code__isnull=False)                      
   107                                                       .order_by("-created_at")                                                         
   108                                                       .first()                                                                         
   109                                                   )                                                                                    
   110                                                   cache_path = artifact.cache()                                                        
   111                                                   logger.important(f"artifact is here: {cache_path}")                                  
   112                                                   return cache_path                                                                    


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:112 - reload_lamindb_itself
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:123 - reload_lamindb
  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_init_instance.py:346 - load_from_isettings
  1.29 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:187 - connect
  2.43 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamin-cli/lamin_cli/_load.py:18 - load

falexwolf · 2024-09-29T21:15:08Z

Now removing profiling 3 functions that are no longer bottlenecks and adding new functions.

laminlabs/lamindb-setup@594ec16

First question is the time it takes to connect via the edge function or not.

The edge function seems to slow down things by 1sec.

I will work with the non-edge function implementation for now.

Query through edge function: ~2s

% kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0.668269 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py
Function: setup_django at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @profile                                                                               
    29                                           def setup_django(                                                                      
    30                                               isettings: InstanceSettings,                                                       
    31                                               deploy_migrations: bool = False,                                                   
    32                                               create_migrations: bool = False,                                                   
    33                                               configure_only: bool = False,                                                      
    34                                               init: bool = False,                                                                
    35                                               view_schema: bool = False,                                                         
    36                                           ):                                                                                     
    37         1          0.0      0.0      0.0      if IS_RUN_FROM_IPYTHON:                                                            
    38                                                   os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"                               
    39                                                                                                                                  
    40         1        388.0    388.0      0.1      import dj_database_url                                                             
    41         1          1.0      1.0      0.0      import django                                                                      
    42         1          2.0      2.0      0.0      from django.conf import settings                                                   
    43         1       4362.0   4362.0      0.7      from django.core.management import call_command                                    
    44                                                                                                                                  
    45                                               # configuration                                                                    
    46         1          3.0      3.0      0.0      if not settings.configured:                                                        
    47         2         29.0     14.5      0.0          default_db = dj_database_url.config(                                           
    48         1          0.0      0.0      0.0              env="LAMINDB_DJANGO_DATABASE_URL",                                         
    49         1          5.0      5.0      0.0              default=isettings.db,                                                      
    50                                                       # see comment next to patching BaseDatabaseWrapper below                   
    51         1          1.0      1.0      0.0              conn_max_age=CONN_MAX_AGE,                                                 
    52         1          0.0      0.0      0.0              conn_health_checks=True,                                                   
    53                                                   )                                                                              
    54         1          0.0      0.0      0.0          DATABASES = {                                                                  
    55         1          0.0      0.0      0.0              "default": default_db,                                                     
    56                                                   }                                                                              
    57         1          2.0      2.0      0.0          from .._init_instance import get_schema_module_name                            
    58                                                                                                                                  
    59         1          3.0      3.0      0.0          schema_names = ["core"] + list(isettings.schema)                               
    60         1        986.0    986.0      0.1          installed_apps = [get_schema_module_name(n) for n in schema_names]             
    61         1          0.0      0.0      0.0          if view_schema:                                                                
    62                                                       installed_apps = installed_apps[::-1]  # to fix how apps appear            
    63                                                       installed_apps += ["schema_graph", "django.contrib.staticfiles"]           
    64                                                                                                                                  
    65         2          0.0      0.0      0.0          kwargs = dict(                                                                 
    66         1          0.0      0.0      0.0              INSTALLED_APPS=installed_apps,                                             
    67         1          0.0      0.0      0.0              DATABASES=DATABASES,                                                       
    68         1          0.0      0.0      0.0              DEFAULT_AUTO_FIELD="django.db.models.BigAutoField",                        
    69         1          0.0      0.0      0.0              TIME_ZONE="UTC",                                                           
    70         1          1.0      1.0      0.0              USE_TZ=True,                                                               
    71                                                   )                                                                              
    72         1          0.0      0.0      0.0          if view_schema:                                                                
    73                                                       kwargs.update(                                                             
    74                                                           DEBUG=True,                                                            
    75                                                           ROOT_URLCONF="lamindb_setup._schema",                                  
    76                                                           SECRET_KEY="dummy",                                                    
    77                                                           TEMPLATES=[                                                            
    78                                                               {                                                                  
    79                                                                   "BACKEND": "django.template.backends.django.DjangoTemplates",  
    80                                                                   "APP_DIRS": True,                                              
    81                                                               },                                                                 
    82                                                           ],                                                                     
    83                                                           STATIC_ROOT=f"{Path.home().as_posix()}/.lamin/",                       
    84                                                           STATICFILES_FINDERS=[                                                  
    85                                                               "django.contrib.staticfiles.finders.AppDirectoriesFinder",         
    86                                                           ],                                                                     
    87                                                           STATIC_URL="static/",                                                  
    88                                                       )                                                                          
    89         1         16.0     16.0      0.0          settings.configure(**kwargs)                                                   
    90         1     662464.0 662464.0     99.1          django.setup(set_prefix=False)                                                 
    91                                                   # https://laminlabs.slack.com/archives/C04FPE8V01W/p1698239551460289           
    92         1          2.0      2.0      0.0          from django.db.backends.base.base import BaseDatabaseWrapper                   
    93                                                                                                                                  
    94         1          2.0      2.0      0.0          BaseDatabaseWrapper.close_if_health_check_failed = close_if_health_check_failed
    95                                                                                                                                  
    96         1          0.0      0.0      0.0      if configure_only:                                                                 
    97                                                   return None                                                                    
    98                                                                                                                                  
    99                                               # migrations management                                                            
   100         1          0.0      0.0      0.0      if create_migrations:                                                              
   101                                                   call_command("makemigrations")                                                 
   102                                                   return None                                                                    
   103                                                                                                                                  
   104         1          0.0      0.0      0.0      if deploy_migrations:                                                              
   105                                                   call_command("migrate", verbosity=2)                                           
   106                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)                 
   107         1          0.0      0.0      0.0      elif init:                                                                         
   108                                                   global IS_MIGRATING                                                            
   109                                                   IS_MIGRATING = True                                                            
   110                                                   call_command("migrate", verbosity=0)                                           
   111                                                   IS_MIGRATING = False                                                           
   112                                                                                                                                  
   113                                               global IS_SETUP                                                                    
   114         1          0.0      0.0      0.0      IS_SETUP = True                                                                    
   115                                                                                                                                  
   116         1          2.0      2.0      0.0      if isettings.keep_artifacts_local:                                                 
   117                                                   isettings._search_local_root()                                                 


Total time: 0.668303 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py
Function: _load_db at line 459

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   459                                               @profile                                                                           
   460                                               def _load_db(self) -> tuple[bool, str]:                                            
   461                                                   # Is the database available and initialized as LaminDB?                        
   462                                                   # returns a tuple of status code and message                                   
   463         1          2.0      2.0      0.0          if self.dialect == "sqlite" and not self._sqlite_file.exists():                
   464                                                       legacy_file = self.storage.key_to_filepath(f"{self.name}.lndb")            
   465                                                       if legacy_file.exists():                                                   
   466                                                           raise RuntimeError(                                                    
   467                                                               "The SQLite file has been renamed!\nPlease rename your SQLite file"
   468                                                               f" {legacy_file} to {self._sqlite_file}"                           
   469                                                           )                                                                      
   470                                                       return False, f"SQLite file {self._sqlite_file} does not exist"            
   471         1          4.0      4.0      0.0          from lamindb_setup import settings  # to check user                            
   472                                                                                                                                  
   473         1          2.0      2.0      0.0          from .django import setup_django                                               
   474                                                                                                                                  
   475                                                   # we need the local sqlite to setup django                                     
   476         1          3.0      3.0      0.0          self._update_local_sqlite_file(lock_cloud_sqlite=self._is_cloud_sqlite)        
   477                                                   # setting up django also performs a check for migrations & prints them         
   478                                                   # as warnings                                                                  
   479                                                   # this should fail, e.g., if the db is not reachable                           
   480         1     668292.0 668292.0    100.0          setup_django(self)                                                             
   481         1          0.0      0.0      0.0          return True, ""                                                                


Total time: 2.11332 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py
Function: _connect_instance_new at line 389

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   389                                           @profile                                                                          
   390                                           def _connect_instance_new(                                                        
   391                                               owner: str,  # account_handle                                                 
   392                                               name: str,  # instance_name                                                   
   393                                               client: Client,                                                               
   394                                           ) -> tuple[dict, dict] | str:                                                     
   395         2    2113266.0    1e+06    100.0      response = client.functions.invoke(                                           
   396         1          0.0      0.0      0.0          "get-instance-settings-v1",                                               
   397         1          1.0      1.0      0.0          invoke_options={"body": {"owner": owner, "name": name}},                  
   398                                               )                                                                             
   399                                               # no instance found, check why is that                                        
   400         1          0.0      0.0      0.0      if response == b"{}":                                                         
   401                                                   # try the via single requests, will take more time                        
   402                                                   account = select_account_by_handle(owner, client)                         
   403                                                   if account is None:                                                       
   404                                                       return "account-not-exists"                                           
   405                                                   instance = select_instance_by_name(account["id"], name, client)           
   406                                                   if instance is None:                                                      
   407                                                       return "instance-not-found"                                           
   408                                                   # get default storage                                                     
   409                                                   storage = select_default_storage_by_instance_id(instance["id"], client)   
   410                                                   if storage is None:                                                       
   411                                                       return "default-storage-does-not-exist-on-hub"                        
   412                                                   logger.warning(                                                           
   413                                                       "Could not find instance via API, but found directly querying hub."   
   414                                                   )                                                                         
   415                                               else:                                                                         
   416         1         31.0     31.0      0.0          instance = json.loads(response)                                           
   417         1          1.0      1.0      0.0          storage = instance.pop("storage")                                         
   418                                                                                                                             
   419         1          0.0      0.0      0.0      if instance["db_scheme"] is not None:                                         
   420         1          0.0      0.0      0.0          db_user_name, db_user_password = None, None                               
   421         1          1.0      1.0      0.0          if "db_user_name" in instance and "db_user_password" in instance:         
   422         1          1.0      1.0      0.0              db_user_name, db_user_password = (                                    
   423         1          0.0      0.0      0.0                  instance["db_user_name"],                                         
   424         1          0.0      0.0      0.0                  instance["db_user_password"],                                     
   425                                                       )                                                                     
   426                                                   else:                                                                     
   427                                                       db_user = select_db_user_by_instance(instance["id"], client)          
   428                                                       if db_user is not None:                                               
   429                                                           db_user_name, db_user_password = (                                
   430                                                               db_user["db_user_name"],                                      
   431                                                               db_user["db_user_password"],                                  
   432                                                           )                                                                 
   433         2         14.0      7.0      0.0          db_dsn = LaminDsn.build(                                                  
   434         1          0.0      0.0      0.0              scheme=instance["db_scheme"],                                         
   435         1          0.0      0.0      0.0              user=db_user_name if db_user_name is not None else "none",            
   436         1          5.0      5.0      0.0              password=db_user_password if db_user_password is not None else "none",
   437         1          0.0      0.0      0.0              host=instance["db_host"],                                             
   438         1          0.0      0.0      0.0              port=instance["db_port"],                                             
   439         1          0.0      0.0      0.0              database=instance["db_database"],                                     
   440                                                   )                                                                         
   441         1          0.0      0.0      0.0          instance["db"] = db_dsn                                                   
   442         1          0.0      0.0      0.0      return instance, storage  # type: ignore                                      


Total time: 2.16498 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: _connect_instance at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                                                       
   114                                           def _connect_instance(                                                                         
   115                                               owner: str,                                                                                
   116                                               name: str,                                                                                 
   117                                               *,                                                                                         
   118                                               db: str | None = None,                                                                     
   119                                               raise_permission_error: bool = True,                                                       
   120                                               access_token: str | None = None,                                                           
   121                                           ) -> InstanceSettings:                                                                         
   122         1          8.0      8.0      0.0      settings_file = instance_settings_file(name, owner)                                        
   123         1          0.0      0.0      0.0      make_hub_request = True                                                                    
   124         1          8.0      8.0      0.0      if settings_file.exists():                                                                 
   125         1       2068.0   2068.0      0.1          isettings = load_instance_settings(settings_file)                                      
   126                                                   # skip hub request for a purely local instance                                         
   127         1        159.0    159.0      0.0          make_hub_request = isettings.is_remote                                                 
   128         1          0.0      0.0      0.0      if make_hub_request:                                                                       
   129                                                   # the following will return a string if the instance does not exist                    
   130                                                   # on the hub                                                                           
   131                                                   # do not call hub if the user is anonymous                                             
   132         1          0.0      0.0      0.0          if owner != "anonymous":                                                               
   133         2    2160146.0    1e+06     99.8              hub_result = connect_instance_from_hub(                                            
   134         1          0.0      0.0      0.0                  owner=owner, name=name, access_token=access_token                              
   135                                                       )                                                                                  
   136                                                   else:                                                                                  
   137                                                       hub_result = "anonymous-user"                                                      
   138                                                   # if hub_result is not a string, it means it made a request                            
   139                                                   # that successfully returned metadata                                                  
   140         1          1.0      1.0      0.0          if not isinstance(hub_result, str):                                                    
   141         1          0.0      0.0      0.0              instance_result, storage_result = hub_result                                       
   142         2       2533.0   1266.5      0.1              db_updated = update_db_using_local(                                                
   143         1          0.0      0.0      0.0                  instance_result,                                                               
   144         1          0.0      0.0      0.0                  settings_file,                                                                 
   145         1          0.0      0.0      0.0                  db=db,                                                                         
   146         1          0.0      0.0      0.0                  raise_permission_error=raise_permission_error,                                 
   147                                                       )                                                                                  
   148         2         36.0     18.0      0.0              ssettings = StorageSettings(                                                       
   149         1          0.0      0.0      0.0                  root=storage_result["root"],                                                   
   150         1          0.0      0.0      0.0                  region=storage_result["region"],                                               
   151         1          0.0      0.0      0.0                  uid=storage_result["lnid"],                                                    
   152         1          3.0      3.0      0.0                  uuid=UUID(storage_result["id"]),                                               
   153         1          2.0      2.0      0.0                  instance_id=UUID(instance_result["id"]),                                       
   154                                                       )                                                                                  
   155         2         11.0      5.5      0.0              isettings = InstanceSettings(                                                      
   156         1          3.0      3.0      0.0                  id=UUID(instance_result["id"]),                                                
   157         1          0.0      0.0      0.0                  owner=owner,                                                                   
   158         1          0.0      0.0      0.0                  name=name,                                                                     
   159         1          0.0      0.0      0.0                  storage=ssettings,                                                             
   160         1          0.0      0.0      0.0                  db=db_updated,                                                                 
   161         1          0.0      0.0      0.0                  schema=instance_result["schema_str"],                                          
   162         1          0.0      0.0      0.0                  git_repo=instance_result["git_repo"],                                          
   163         1          0.0      0.0      0.0                  keep_artifacts_local=bool(instance_result["keep_artifacts_local"]),            
   164         1          0.0      0.0      0.0                  is_on_hub=True,                                                                
   165                                                       )                                                                                  
   166                                                   else:                                                                                  
   167                                                       if hub_result != "anonymous-user":                                                 
   168                                                           message = INSTANCE_NOT_FOUND_MESSAGE.format(                                   
   169                                                               owner=owner, name=name, hub_result=hub_result                              
   170                                                           )                                                                              
   171                                                       else:                                                                              
   172                                                           message = "It is not possible to load an anonymous-owned instance from the hub"
   173                                                       if settings_file.exists():                                                         
   174                                                           isettings = load_instance_settings(settings_file)                              
   175                                                           if isettings.is_remote:                                                        
   176                                                               raise InstanceNotFoundError(message)                                       
   177                                                       else:                                                                              
   178                                                           raise InstanceNotFoundError(message)                                           
   179         1          0.0      0.0      0.0      return isettings                                                                           


Total time: 2.84026 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 182

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   182                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   183                                           @profile                                                                               
   184                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   185                                               """Connect to instance.                                                            
   186                                                                                                                                  
   187                                               Args:                                                                              
   188                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   189                                                       If the instance is owned by you, it suffices to pass the instance name.    
   190                                               """                                                                                
   191                                               # validate kwargs                                                                  
   192         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   193                                                   "_db",                                                                         
   194                                                   "_write_settings",                                                             
   195                                                   "_raise_not_found_error",                                                      
   196                                                   "_test",                                                                       
   197                                                   "_user",                                                                       
   198                                               }                                                                                  
   199         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   200                                                   if kwarg not in valid_kwargs:                                                  
   201                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   202                                                                                                                                  
   203         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   204                                               # _db is still needed because it is called in init                                 
   205         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   206         1          1.0      1.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   207         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   208         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   209                                                                                                                                  
   210         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   211         1          1.0      1.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   212         1          0.0      0.0      0.0      if _user is not None:                                                              
   213                                                   access_token = _user.access_token                                              
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      try:                                                                               
   216         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   217                                                                                                                                  
   218         1       2738.0   2738.0      0.1          if _check_instance_setup() and not _test:                                      
   219                                                       if (                                                                       
   220                                                           settings._instance_exists                                              
   221                                                           and f"{owner}/{name}" == settings.instance.slug                        
   222                                                       ):                                                                         
   223                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   224                                                           return None                                                            
   225                                                       else:                                                                      
   226                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   227         2          0.0      0.0      0.0          elif (                                                                         
   228         1          0.0      0.0      0.0              _write_settings                                                            
   229         1       2121.0   2121.0      0.1              and settings._instance_exists                                              
   230         1          8.0      8.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   231                                                   ):                                                                             
   232                                                       close_instance(mute=True)                                                  
   233                                                                                                                                  
   234         1          0.0      0.0      0.0          try:                                                                           
   235         2    2164999.0    1e+06     76.2              isettings = _connect_instance(                                             
   236         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   237                                                       )                                                                          
   238                                                   except InstanceNotFoundError as e:                                             
   239                                                       if _raise_not_found_error:                                                 
   240                                                           raise e                                                                
   241                                                       else:                                                                      
   242                                                           return "instance-not-found"                                            
   243         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   244                                                       return isettings                                                           
   245                                                   # at this point we have checked already that isettings is not a string         
   246                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   247                                                   # has no effect if _user is None or if not cloud sqlite instance               
   248         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   249         1        881.0    881.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   250         1          0.0      0.0      0.0          if _test:                                                                      
   251                                                       return None                                                                
   252         1          1.0      1.0      0.0          silence_loggers()                                                              
   253         1     668307.0 668307.0     23.5          check, msg = isettings._load_db()                                              
   254         1          0.0      0.0      0.0          if not check:                                                                  
   255                                                       local_db = (                                                               
   256                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   257                                                       )                                                                          
   258                                                       if local_db:                                                               
   259                                                           logger.warning(                                                        
   260                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   261                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   262                                                               " call: lamin load --unload"                                       
   263                                                           )                                                                      
   264                                                       elif _raise_not_found_error:                                               
   265                                                           raise SystemExit(msg)                                                  
   266                                                       else:                                                                      
   267                                                           logger.warning(                                                        
   268                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   269                                                               " loadable: re-initializing"                                       
   270                                                           )                                                                      
   271                                                           return "instance-corrupted-or-deleted"                                 
   272                                                   # this is for testing purposes only                                            
   273         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   274                                                       raise RuntimeError("Technical testing error.")                             
   275                                                                                                                                  
   276                                                   # below is for backfilling the instance_uid value                              
   277                                                   # we'll enable it once more people migrated to 0.71.0                          
   278                                                   # ssettings_record = isettings.storage.record                                  
   279                                                   # if ssettings_record.instance_uid is None:                                    
   280                                                   #     ssettings_record.instance_uid = isettings.uid                            
   281                                                   #     # try saving if not read-only access                                     
   282                                                   #     try:                                                                     
   283                                                   #         ssettings_record.save()                                              
   284                                                   #     # raised by django when the access is denied                             
   285                                                   #     except ProgrammingError:                                                 
   286                                                   #         pass                                                                 
   287         1       1175.0   1175.0      0.0          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   288                                               except Exception as e:                                                             
   289                                                   if isettings is not None:                                                      
   290                                                       if _write_settings:                                                        
   291                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   292                                                       settings._instance_settings = None                                         
   293                                                   raise e                                                                        
   294                                               # rename lnschema_bionty to bionty for sql tables                                  
   295         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   296         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   297         1          8.0      8.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   298                                                   )                                                                              
   299         1         11.0     11.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   300                                                       migrate_lnschema_bionty(                                                   
   301                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   302                                                       )                                                                          
   303         1          0.0      0.0      0.0      return None                                                                        


  0.67 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py:28 - setup_django
  0.67 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py:459 - _load_db
  2.11 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py:389 - _connect_instance_new
  2.16 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:113 - _connect_instance
  2.84 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:182 - connect
(py310) falexwolf@mbpalex lamindb-setup % kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0.658284 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py
Function: setup_django at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @profile                                                                               
    29                                           def setup_django(                                                                      
    30                                               isettings: InstanceSettings,                                                       
    31                                               deploy_migrations: bool = False,                                                   
    32                                               create_migrations: bool = False,                                                   
    33                                               configure_only: bool = False,                                                      
    34                                               init: bool = False,                                                                
    35                                               view_schema: bool = False,                                                         
    36                                           ):                                                                                     
    37         1          1.0      1.0      0.0      if IS_RUN_FROM_IPYTHON:                                                            
    38                                                   os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"                               
    39                                                                                                                                  
    40         1        405.0    405.0      0.1      import dj_database_url                                                             
    41         1          1.0      1.0      0.0      import django                                                                      
    42         1          3.0      3.0      0.0      from django.conf import settings                                                   
    43         1       4150.0   4150.0      0.6      from django.core.management import call_command                                    
    44                                                                                                                                  
    45                                               # configuration                                                                    
    46         1          5.0      5.0      0.0      if not settings.configured:                                                        
    47         2         36.0     18.0      0.0          default_db = dj_database_url.config(                                           
    48         1          0.0      0.0      0.0              env="LAMINDB_DJANGO_DATABASE_URL",                                         
    49         1          5.0      5.0      0.0              default=isettings.db,                                                      
    50                                                       # see comment next to patching BaseDatabaseWrapper below                   
    51         1          1.0      1.0      0.0              conn_max_age=CONN_MAX_AGE,                                                 
    52         1          0.0      0.0      0.0              conn_health_checks=True,                                                   
    53                                                   )                                                                              
    54         1          0.0      0.0      0.0          DATABASES = {                                                                  
    55         1          0.0      0.0      0.0              "default": default_db,                                                     
    56                                                   }                                                                              
    57         1          2.0      2.0      0.0          from .._init_instance import get_schema_module_name                            
    58                                                                                                                                  
    59         1          3.0      3.0      0.0          schema_names = ["core"] + list(isettings.schema)                               
    60         1        989.0    989.0      0.2          installed_apps = [get_schema_module_name(n) for n in schema_names]             
    61         1          0.0      0.0      0.0          if view_schema:                                                                
    62                                                       installed_apps = installed_apps[::-1]  # to fix how apps appear            
    63                                                       installed_apps += ["schema_graph", "django.contrib.staticfiles"]           
    64                                                                                                                                  
    65         2          1.0      0.5      0.0          kwargs = dict(                                                                 
    66         1          0.0      0.0      0.0              INSTALLED_APPS=installed_apps,                                             
    67         1          0.0      0.0      0.0              DATABASES=DATABASES,                                                       
    68         1          0.0      0.0      0.0              DEFAULT_AUTO_FIELD="django.db.models.BigAutoField",                        
    69         1          0.0      0.0      0.0              TIME_ZONE="UTC",                                                           
    70         1          0.0      0.0      0.0              USE_TZ=True,                                                               
    71                                                   )                                                                              
    72         1          0.0      0.0      0.0          if view_schema:                                                                
    73                                                       kwargs.update(                                                             
    74                                                           DEBUG=True,                                                            
    75                                                           ROOT_URLCONF="lamindb_setup._schema",                                  
    76                                                           SECRET_KEY="dummy",                                                    
    77                                                           TEMPLATES=[                                                            
    78                                                               {                                                                  
    79                                                                   "BACKEND": "django.template.backends.django.DjangoTemplates",  
    80                                                                   "APP_DIRS": True,                                              
    81                                                               },                                                                 
    82                                                           ],                                                                     
    83                                                           STATIC_ROOT=f"{Path.home().as_posix()}/.lamin/",                       
    84                                                           STATICFILES_FINDERS=[                                                  
    85                                                               "django.contrib.staticfiles.finders.AppDirectoriesFinder",         
    86                                                           ],                                                                     
    87                                                           STATIC_URL="static/",                                                  
    88                                                       )                                                                          
    89         1         18.0     18.0      0.0          settings.configure(**kwargs)                                                   
    90         1     652659.0 652659.0     99.1          django.setup(set_prefix=False)                                                 
    91                                                   # https://laminlabs.slack.com/archives/C04FPE8V01W/p1698239551460289           
    92         1          2.0      2.0      0.0          from django.db.backends.base.base import BaseDatabaseWrapper                   
    93                                                                                                                                  
    94         1          1.0      1.0      0.0          BaseDatabaseWrapper.close_if_health_check_failed = close_if_health_check_failed
    95                                                                                                                                  
    96         1          0.0      0.0      0.0      if configure_only:                                                                 
    97                                                   return None                                                                    
    98                                                                                                                                  
    99                                               # migrations management                                                            
   100         1          0.0      0.0      0.0      if create_migrations:                                                              
   101                                                   call_command("makemigrations")                                                 
   102                                                   return None                                                                    
   103                                                                                                                                  
   104         1          0.0      0.0      0.0      if deploy_migrations:                                                              
   105                                                   call_command("migrate", verbosity=2)                                           
   106                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)                 
   107         1          0.0      0.0      0.0      elif init:                                                                         
   108                                                   global IS_MIGRATING                                                            
   109                                                   IS_MIGRATING = True                                                            
   110                                                   call_command("migrate", verbosity=0)                                           
   111                                                   IS_MIGRATING = False                                                           
   112                                                                                                                                  
   113                                               global IS_SETUP                                                                    
   114         1          0.0      0.0      0.0      IS_SETUP = True                                                                    
   115                                                                                                                                  
   116         1          2.0      2.0      0.0      if isettings.keep_artifacts_local:                                                 
   117                                                   isettings._search_local_root()                                                 


Total time: 0.658317 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py
Function: _load_db at line 459

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   459                                               @profile                                                                           
   460                                               def _load_db(self) -> tuple[bool, str]:                                            
   461                                                   # Is the database available and initialized as LaminDB?                        
   462                                                   # returns a tuple of status code and message                                   
   463         1          2.0      2.0      0.0          if self.dialect == "sqlite" and not self._sqlite_file.exists():                
   464                                                       legacy_file = self.storage.key_to_filepath(f"{self.name}.lndb")            
   465                                                       if legacy_file.exists():                                                   
   466                                                           raise RuntimeError(                                                    
   467                                                               "The SQLite file has been renamed!\nPlease rename your SQLite file"
   468                                                               f" {legacy_file} to {self._sqlite_file}"                           
   469                                                           )                                                                      
   470                                                       return False, f"SQLite file {self._sqlite_file} does not exist"            
   471         1          4.0      4.0      0.0          from lamindb_setup import settings  # to check user                            
   472                                                                                                                                  
   473         1          2.0      2.0      0.0          from .django import setup_django                                               
   474                                                                                                                                  
   475                                                   # we need the local sqlite to setup django                                     
   476         1          2.0      2.0      0.0          self._update_local_sqlite_file(lock_cloud_sqlite=self._is_cloud_sqlite)        
   477                                                   # setting up django also performs a check for migrations & prints them         
   478                                                   # as warnings                                                                  
   479                                                   # this should fail, e.g., if the db is not reachable                           
   480         1     658307.0 658307.0    100.0          setup_django(self)                                                             
   481         1          0.0      0.0      0.0          return True, ""                                                                


Total time: 1.42614 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py
Function: _connect_instance_new at line 389

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   389                                           @profile                                                                          
   390                                           def _connect_instance_new(                                                        
   391                                               owner: str,  # account_handle                                                 
   392                                               name: str,  # instance_name                                                   
   393                                               client: Client,                                                               
   394                                           ) -> tuple[dict, dict] | str:                                                     
   395         2    1426087.0 713043.5    100.0      response = client.functions.invoke(                                           
   396         1          0.0      0.0      0.0          "get-instance-settings-v1",                                               
   397         1          0.0      0.0      0.0          invoke_options={"body": {"owner": owner, "name": name}},                  
   398                                               )                                                                             
   399                                               # no instance found, check why is that                                        
   400         1          0.0      0.0      0.0      if response == b"{}":                                                         
   401                                                   # try the via single requests, will take more time                        
   402                                                   account = select_account_by_handle(owner, client)                         
   403                                                   if account is None:                                                       
   404                                                       return "account-not-exists"                                           
   405                                                   instance = select_instance_by_name(account["id"], name, client)           
   406                                                   if instance is None:                                                      
   407                                                       return "instance-not-found"                                           
   408                                                   # get default storage                                                     
   409                                                   storage = select_default_storage_by_instance_id(instance["id"], client)   
   410                                                   if storage is None:                                                       
   411                                                       return "default-storage-does-not-exist-on-hub"                        
   412                                                   logger.warning(                                                           
   413                                                       "Could not find instance via API, but found directly querying hub."   
   414                                                   )                                                                         
   415                                               else:                                                                         
   416         1         30.0     30.0      0.0          instance = json.loads(response)                                           
   417         1          0.0      0.0      0.0          storage = instance.pop("storage")                                         
   418                                                                                                                             
   419         1          0.0      0.0      0.0      if instance["db_scheme"] is not None:                                         
   420         1          0.0      0.0      0.0          db_user_name, db_user_password = None, None                               
   421         1          0.0      0.0      0.0          if "db_user_name" in instance and "db_user_password" in instance:         
   422         1          0.0      0.0      0.0              db_user_name, db_user_password = (                                    
   423         1          1.0      1.0      0.0                  instance["db_user_name"],                                         
   424         1          0.0      0.0      0.0                  instance["db_user_password"],                                     
   425                                                       )                                                                     
   426                                                   else:                                                                     
   427                                                       db_user = select_db_user_by_instance(instance["id"], client)          
   428                                                       if db_user is not None:                                               
   429                                                           db_user_name, db_user_password = (                                
   430                                                               db_user["db_user_name"],                                      
   431                                                               db_user["db_user_password"],                                  
   432                                                           )                                                                 
   433         2         14.0      7.0      0.0          db_dsn = LaminDsn.build(                                                  
   434         1          0.0      0.0      0.0              scheme=instance["db_scheme"],                                         
   435         1          1.0      1.0      0.0              user=db_user_name if db_user_name is not None else "none",            
   436         1          0.0      0.0      0.0              password=db_user_password if db_user_password is not None else "none",
   437         1          4.0      4.0      0.0              host=instance["db_host"],                                             
   438         1          0.0      0.0      0.0              port=instance["db_port"],                                             
   439         1          0.0      0.0      0.0              database=instance["db_database"],                                     
   440                                                   )                                                                         
   441         1          1.0      1.0      0.0          instance["db"] = db_dsn                                                   
   442         1          0.0      0.0      0.0      return instance, storage  # type: ignore                                      


Total time: 1.47764 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: _connect_instance at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                                                       
   114                                           def _connect_instance(                                                                         
   115                                               owner: str,                                                                                
   116                                               name: str,                                                                                 
   117                                               *,                                                                                         
   118                                               db: str | None = None,                                                                     
   119                                               raise_permission_error: bool = True,                                                       
   120                                               access_token: str | None = None,                                                           
   121                                           ) -> InstanceSettings:                                                                         
   122         1          8.0      8.0      0.0      settings_file = instance_settings_file(name, owner)                                        
   123         1          0.0      0.0      0.0      make_hub_request = True                                                                    
   124         1          7.0      7.0      0.0      if settings_file.exists():                                                                 
   125         1       2062.0   2062.0      0.1          isettings = load_instance_settings(settings_file)                                      
   126                                                   # skip hub request for a purely local instance                                         
   127         1        160.0    160.0      0.0          make_hub_request = isettings.is_remote                                                 
   128         1          0.0      0.0      0.0      if make_hub_request:                                                                       
   129                                                   # the following will return a string if the instance does not exist                    
   130                                                   # on the hub                                                                           
   131                                                   # do not call hub if the user is anonymous                                             
   132         1          1.0      1.0      0.0          if owner != "anonymous":                                                               
   133         2    1472850.0 736425.0     99.7              hub_result = connect_instance_from_hub(                                            
   134         1          0.0      0.0      0.0                  owner=owner, name=name, access_token=access_token                              
   135                                                       )                                                                                  
   136                                                   else:                                                                                  
   137                                                       hub_result = "anonymous-user"                                                      
   138                                                   # if hub_result is not a string, it means it made a request                            
   139                                                   # that successfully returned metadata                                                  
   140         1          1.0      1.0      0.0          if not isinstance(hub_result, str):                                                    
   141         1          0.0      0.0      0.0              instance_result, storage_result = hub_result                                       
   142         2       2496.0   1248.0      0.2              db_updated = update_db_using_local(                                                
   143         1          0.0      0.0      0.0                  instance_result,                                                               
   144         1          0.0      0.0      0.0                  settings_file,                                                                 
   145         1          0.0      0.0      0.0                  db=db,                                                                         
   146         1          0.0      0.0      0.0                  raise_permission_error=raise_permission_error,                                 
   147                                                       )                                                                                  
   148         2         37.0     18.5      0.0              ssettings = StorageSettings(                                                       
   149         1          1.0      1.0      0.0                  root=storage_result["root"],                                                   
   150         1          0.0      0.0      0.0                  region=storage_result["region"],                                               
   151         1          0.0      0.0      0.0                  uid=storage_result["lnid"],                                                    
   152         1          4.0      4.0      0.0                  uuid=UUID(storage_result["id"]),                                               
   153         1          2.0      2.0      0.0                  instance_id=UUID(instance_result["id"]),                                       
   154                                                       )                                                                                  
   155         2         12.0      6.0      0.0              isettings = InstanceSettings(                                                      
   156         1          2.0      2.0      0.0                  id=UUID(instance_result["id"]),                                                
   157         1          0.0      0.0      0.0                  owner=owner,                                                                   
   158         1          0.0      0.0      0.0                  name=name,                                                                     
   159         1          0.0      0.0      0.0                  storage=ssettings,                                                             
   160         1          0.0      0.0      0.0                  db=db_updated,                                                                 
   161         1          0.0      0.0      0.0                  schema=instance_result["schema_str"],                                          
   162         1          0.0      0.0      0.0                  git_repo=instance_result["git_repo"],                                          
   163         1          0.0      0.0      0.0                  keep_artifacts_local=bool(instance_result["keep_artifacts_local"]),            
   164         1          0.0      0.0      0.0                  is_on_hub=True,                                                                
   165                                                       )                                                                                  
   166                                                   else:                                                                                  
   167                                                       if hub_result != "anonymous-user":                                                 
   168                                                           message = INSTANCE_NOT_FOUND_MESSAGE.format(                                   
   169                                                               owner=owner, name=name, hub_result=hub_result                              
   170                                                           )                                                                              
   171                                                       else:                                                                              
   172                                                           message = "It is not possible to load an anonymous-owned instance from the hub"
   173                                                       if settings_file.exists():                                                         
   174                                                           isettings = load_instance_settings(settings_file)                              
   175                                                           if isettings.is_remote:                                                        
   176                                                               raise InstanceNotFoundError(message)                                       
   177                                                       else:                                                                              
   178                                                           raise InstanceNotFoundError(message)                                           
   179         1          1.0      1.0      0.0      return isettings                                                                           


Total time: 2.1427 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 182

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   182                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   183                                           @profile                                                                               
   184                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   185                                               """Connect to instance.                                                            
   186                                                                                                                                  
   187                                               Args:                                                                              
   188                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   189                                                       If the instance is owned by you, it suffices to pass the instance name.    
   190                                               """                                                                                
   191                                               # validate kwargs                                                                  
   192         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   193                                                   "_db",                                                                         
   194                                                   "_write_settings",                                                             
   195                                                   "_raise_not_found_error",                                                      
   196                                                   "_test",                                                                       
   197                                                   "_user",                                                                       
   198                                               }                                                                                  
   199         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   200                                                   if kwarg not in valid_kwargs:                                                  
   201                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   202                                                                                                                                  
   203         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   204                                               # _db is still needed because it is called in init                                 
   205         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   206         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   207         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   208         1          1.0      1.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   209                                                                                                                                  
   210         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   211         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   212         1          1.0      1.0      0.0      if _user is not None:                                                              
   213                                                   access_token = _user.access_token                                              
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      try:                                                                               
   216         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   217                                                                                                                                  
   218         1       2780.0   2780.0      0.1          if _check_instance_setup() and not _test:                                      
   219                                                       if (                                                                       
   220                                                           settings._instance_exists                                              
   221                                                           and f"{owner}/{name}" == settings.instance.slug                        
   222                                                       ):                                                                         
   223                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   224                                                           return None                                                            
   225                                                       else:                                                                      
   226                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   227         2          0.0      0.0      0.0          elif (                                                                         
   228         1          0.0      0.0      0.0              _write_settings                                                            
   229         1       2137.0   2137.0      0.1              and settings._instance_exists                                              
   230         1         10.0     10.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   231                                                   ):                                                                             
   232                                                       close_instance(mute=True)                                                  
   233                                                                                                                                  
   234         1          0.0      0.0      0.0          try:                                                                           
   235         2    1477667.0 738833.5     69.0              isettings = _connect_instance(                                             
   236         1          1.0      1.0      0.0                  owner, name, db=_db, access_token=access_token                         
   237                                                       )                                                                          
   238                                                   except InstanceNotFoundError as e:                                             
   239                                                       if _raise_not_found_error:                                                 
   240                                                           raise e                                                                
   241                                                       else:                                                                      
   242                                                           return "instance-not-found"                                            
   243         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   244                                                       return isettings                                                           
   245                                                   # at this point we have checked already that isettings is not a string         
   246                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   247                                                   # has no effect if _user is None or if not cloud sqlite instance               
   248         1          1.0      1.0      0.0          isettings._locker_user = _user                                                 
   249         1        654.0    654.0      0.0          isettings._persist(write_to_disk=_write_settings)                              
   250         1          1.0      1.0      0.0          if _test:                                                                      
   251                                                       return None                                                                
   252         1          1.0      1.0      0.0          silence_loggers()                                                              
   253         1     658325.0 658325.0     30.7          check, msg = isettings._load_db()                                              
   254         1          0.0      0.0      0.0          if not check:                                                                  
   255                                                       local_db = (                                                               
   256                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   257                                                       )                                                                          
   258                                                       if local_db:                                                               
   259                                                           logger.warning(                                                        
   260                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   261                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   262                                                               " call: lamin load --unload"                                       
   263                                                           )                                                                      
   264                                                       elif _raise_not_found_error:                                               
   265                                                           raise SystemExit(msg)                                                  
   266                                                       else:                                                                      
   267                                                           logger.warning(                                                        
   268                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   269                                                               " loadable: re-initializing"                                       
   270                                                           )                                                                      
   271                                                           return "instance-corrupted-or-deleted"                                 
   272                                                   # this is for testing purposes only                                            
   273         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   274                                                       raise RuntimeError("Technical testing error.")                             
   275                                                                                                                                  
   276                                                   # below is for backfilling the instance_uid value                              
   277                                                   # we'll enable it once more people migrated to 0.71.0                          
   278                                                   # ssettings_record = isettings.storage.record                                  
   279                                                   # if ssettings_record.instance_uid is None:                                    
   280                                                   #     ssettings_record.instance_uid = isettings.uid                            
   281                                                   #     # try saving if not read-only access                                     
   282                                                   #     try:                                                                     
   283                                                   #         ssettings_record.save()                                              
   284                                                   #     # raised by django when the access is denied                             
   285                                                   #     except ProgrammingError:                                                 
   286                                                   #         pass                                                                 
   287         1       1094.0   1094.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   288                                               except Exception as e:                                                             
   289                                                   if isettings is not None:                                                      
   290                                                       if _write_settings:                                                        
   291                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   292                                                       settings._instance_settings = None                                         
   293                                                   raise e                                                                        
   294                                               # rename lnschema_bionty to bionty for sql tables                                  
   295         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   296         1          1.0      1.0      0.0          no_lnschema_bionty_file = (                                                    
   297         1         10.0     10.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   298                                                   )                                                                              
   299         1          8.0      8.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   300                                                       migrate_lnschema_bionty(                                                   
   301                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   302                                                       )                                                                          
   303         1          0.0      0.0      0.0      return None                                                                        


  0.66 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py:28 - setup_django
  0.66 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py:459 - _load_db
  1.43 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py:389 - _connect_instance_new
  1.48 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:113 - _connect_instance
  2.14 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:182 - connect

Query through postgrest directly: ~1s

(py310) falexwolf@mbpalex lamindb-setup % kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py
Function: _connect_instance_new at line 389

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   389                                           @profile                                                                          
   390                                           def _connect_instance_new(                                                        
   391                                               owner: str,  # account_handle                                                 
   392                                               name: str,  # instance_name                                                   
   393                                               client: Client,                                                               
   394                                           ) -> tuple[dict, dict] | str:                                                     
   395                                               response = client.functions.invoke(                                           
   396                                                   "get-instance-settings-v1",                                               
   397                                                   invoke_options={"body": {"owner": owner, "name": name}},                  
   398                                               )                                                                             
   399                                               # no instance found, check why is that                                        
   400                                               if response == b"{}":                                                         
   401                                                   # try the via single requests, will take more time                        
   402                                                   account = select_account_by_handle(owner, client)                         
   403                                                   if account is None:                                                       
   404                                                       return "account-not-exists"                                           
   405                                                   instance = select_instance_by_name(account["id"], name, client)           
   406                                                   if instance is None:                                                      
   407                                                       return "instance-not-found"                                           
   408                                                   # get default storage                                                     
   409                                                   storage = select_default_storage_by_instance_id(instance["id"], client)   
   410                                                   if storage is None:                                                       
   411                                                       return "default-storage-does-not-exist-on-hub"                        
   412                                                   logger.warning(                                                           
   413                                                       "Could not find instance via API, but found directly querying hub."   
   414                                                   )                                                                         
   415                                               else:                                                                         
   416                                                   instance = json.loads(response)                                           
   417                                                   storage = instance.pop("storage")                                         
   418                                                                                                                             
   419                                               if instance["db_scheme"] is not None:                                         
   420                                                   db_user_name, db_user_password = None, None                               
   421                                                   if "db_user_name" in instance and "db_user_password" in instance:         
   422                                                       db_user_name, db_user_password = (                                    
   423                                                           instance["db_user_name"],                                         
   424                                                           instance["db_user_password"],                                     
   425                                                       )                                                                     
   426                                                   else:                                                                     
   427                                                       db_user = select_db_user_by_instance(instance["id"], client)          
   428                                                       if db_user is not None:                                               
   429                                                           db_user_name, db_user_password = (                                
   430                                                               db_user["db_user_name"],                                      
   431                                                               db_user["db_user_password"],                                  
   432                                                           )                                                                 
   433                                                   db_dsn = LaminDsn.build(                                                  
   434                                                       scheme=instance["db_scheme"],                                         
   435                                                       user=db_user_name if db_user_name is not None else "none",            
   436                                                       password=db_user_password if db_user_password is not None else "none",
   437                                                       host=instance["db_host"],                                             
   438                                                       port=instance["db_port"],                                             
   439                                                       database=instance["db_database"],                                     
   440                                                   )                                                                         
   441                                                   instance["db"] = db_dsn                                                   
   442                                               return instance, storage  # type: ignore                                      


Total time: 0.422809 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: _connect_instance at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                                                       
   114                                           def _connect_instance(                                                                         
   115                                               owner: str,                                                                                
   116                                               name: str,                                                                                 
   117                                               *,                                                                                         
   118                                               db: str | None = None,                                                                     
   119                                               raise_permission_error: bool = True,                                                       
   120                                               access_token: str | None = None,                                                           
   121                                           ) -> InstanceSettings:                                                                         
   122         1          9.0      9.0      0.0      settings_file = instance_settings_file(name, owner)                                        
   123         1          0.0      0.0      0.0      make_hub_request = True                                                                    
   124         1          7.0      7.0      0.0      if settings_file.exists():                                                                 
   125         1       2060.0   2060.0      0.5          isettings = load_instance_settings(settings_file)                                      
   126                                                   # skip hub request for a purely local instance                                         
   127         1        156.0    156.0      0.0          make_hub_request = isettings.is_remote                                                 
   128         1          0.0      0.0      0.0      if make_hub_request:                                                                       
   129                                                   # the following will return a string if the instance does not exist                    
   130                                                   # on the hub                                                                           
   131                                                   # do not call hub if the user is anonymous                                             
   132         1          0.0      0.0      0.0          if owner != "anonymous":                                                               
   133         2     417823.0 208911.5     98.8              hub_result = connect_instance_from_hub(                                            
   134         1          0.0      0.0      0.0                  owner=owner, name=name, access_token=access_token                              
   135                                                       )                                                                                  
   136                                                   else:                                                                                  
   137                                                       hub_result = "anonymous-user"                                                      
   138                                                   # if hub_result is not a string, it means it made a request                            
   139                                                   # that successfully returned metadata                                                  
   140         1          2.0      2.0      0.0          if not isinstance(hub_result, str):                                                    
   141         1          1.0      1.0      0.0              instance_result, storage_result = hub_result                                       
   142         2       2692.0   1346.0      0.6              db_updated = update_db_using_local(                                                
   143         1          0.0      0.0      0.0                  instance_result,                                                               
   144         1          0.0      0.0      0.0                  settings_file,                                                                 
   145         1          0.0      0.0      0.0                  db=db,                                                                         
   146         1          0.0      0.0      0.0                  raise_permission_error=raise_permission_error,                                 
   147                                                       )                                                                                  
   148         2         38.0     19.0      0.0              ssettings = StorageSettings(                                                       
   149         1          1.0      1.0      0.0                  root=storage_result["root"],                                                   
   150         1          0.0      0.0      0.0                  region=storage_result["region"],                                               
   151         1          0.0      0.0      0.0                  uid=storage_result["lnid"],                                                    
   152         1          3.0      3.0      0.0                  uuid=UUID(storage_result["id"]),                                               
   153         1          2.0      2.0      0.0                  instance_id=UUID(instance_result["id"]),                                       
   154                                                       )                                                                                  
   155         2         11.0      5.5      0.0              isettings = InstanceSettings(                                                      
   156         1          3.0      3.0      0.0                  id=UUID(instance_result["id"]),                                                
   157         1          0.0      0.0      0.0                  owner=owner,                                                                   
   158         1          0.0      0.0      0.0                  name=name,                                                                     
   159         1          0.0      0.0      0.0                  storage=ssettings,                                                             
   160         1          0.0      0.0      0.0                  db=db_updated,                                                                 
   161         1          1.0      1.0      0.0                  schema=instance_result["schema_str"],                                          
   162         1          0.0      0.0      0.0                  git_repo=instance_result["git_repo"],                                          
   163         1          0.0      0.0      0.0                  keep_artifacts_local=bool(instance_result["keep_artifacts_local"]),            
   164         1          0.0      0.0      0.0                  is_on_hub=True,                                                                
   165                                                       )                                                                                  
   166                                                   else:                                                                                  
   167                                                       if hub_result != "anonymous-user":                                                 
   168                                                           message = INSTANCE_NOT_FOUND_MESSAGE.format(                                   
   169                                                               owner=owner, name=name, hub_result=hub_result                              
   170                                                           )                                                                              
   171                                                       else:                                                                              
   172                                                           message = "It is not possible to load an anonymous-owned instance from the hub"
   173                                                       if settings_file.exists():                                                         
   174                                                           isettings = load_instance_settings(settings_file)                              
   175                                                           if isettings.is_remote:                                                        
   176                                                               raise InstanceNotFoundError(message)                                       
   177                                                       else:                                                                              
   178                                                           raise InstanceNotFoundError(message)                                           
   179         1          0.0      0.0      0.0      return isettings                                                                           


Total time: 0.655027 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py
Function: setup_django at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @profile                                                                               
    29                                           def setup_django(                                                                      
    30                                               isettings: InstanceSettings,                                                       
    31                                               deploy_migrations: bool = False,                                                   
    32                                               create_migrations: bool = False,                                                   
    33                                               configure_only: bool = False,                                                      
    34                                               init: bool = False,                                                                
    35                                               view_schema: bool = False,                                                         
    36                                           ):                                                                                     
    37         1          1.0      1.0      0.0      if IS_RUN_FROM_IPYTHON:                                                            
    38                                                   os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"                               
    39                                                                                                                                  
    40         1        434.0    434.0      0.1      import dj_database_url                                                             
    41         1          3.0      3.0      0.0      import django                                                                      
    42         1          3.0      3.0      0.0      from django.conf import settings                                                   
    43         1       3719.0   3719.0      0.6      from django.core.management import call_command                                    
    44                                                                                                                                  
    45                                               # configuration                                                                    
    46         1          7.0      7.0      0.0      if not settings.configured:                                                        
    47         2         29.0     14.5      0.0          default_db = dj_database_url.config(                                           
    48         1          0.0      0.0      0.0              env="LAMINDB_DJANGO_DATABASE_URL",                                         
    49         1          5.0      5.0      0.0              default=isettings.db,                                                      
    50                                                       # see comment next to patching BaseDatabaseWrapper below                   
    51         1          0.0      0.0      0.0              conn_max_age=CONN_MAX_AGE,                                                 
    52         1          0.0      0.0      0.0              conn_health_checks=True,                                                   
    53                                                   )                                                                              
    54         1          0.0      0.0      0.0          DATABASES = {                                                                  
    55         1          0.0      0.0      0.0              "default": default_db,                                                     
    56                                                   }                                                                              
    57         1          4.0      4.0      0.0          from .._init_instance import get_schema_module_name                            
    58                                                                                                                                  
    59         1          3.0      3.0      0.0          schema_names = ["core"] + list(isettings.schema)                               
    60         1       1014.0   1014.0      0.2          installed_apps = [get_schema_module_name(n) for n in schema_names]             
    61         1          0.0      0.0      0.0          if view_schema:                                                                
    62                                                       installed_apps = installed_apps[::-1]  # to fix how apps appear            
    63                                                       installed_apps += ["schema_graph", "django.contrib.staticfiles"]           
    64                                                                                                                                  
    65         2          0.0      0.0      0.0          kwargs = dict(                                                                 
    66         1          0.0      0.0      0.0              INSTALLED_APPS=installed_apps,                                             
    67         1          0.0      0.0      0.0              DATABASES=DATABASES,                                                       
    68         1          0.0      0.0      0.0              DEFAULT_AUTO_FIELD="django.db.models.BigAutoField",                        
    69         1          0.0      0.0      0.0              TIME_ZONE="UTC",                                                           
    70         1          1.0      1.0      0.0              USE_TZ=True,                                                               
    71                                                   )                                                                              
    72         1          0.0      0.0      0.0          if view_schema:                                                                
    73                                                       kwargs.update(                                                             
    74                                                           DEBUG=True,                                                            
    75                                                           ROOT_URLCONF="lamindb_setup._schema",                                  
    76                                                           SECRET_KEY="dummy",                                                    
    77                                                           TEMPLATES=[                                                            
    78                                                               {                                                                  
    79                                                                   "BACKEND": "django.template.backends.django.DjangoTemplates",  
    80                                                                   "APP_DIRS": True,                                              
    81                                                               },                                                                 
    82                                                           ],                                                                     
    83                                                           STATIC_ROOT=f"{Path.home().as_posix()}/.lamin/",                       
    84                                                           STATICFILES_FINDERS=[                                                  
    85                                                               "django.contrib.staticfiles.finders.AppDirectoriesFinder",         
    86                                                           ],                                                                     
    87                                                           STATIC_URL="static/",                                                  
    88                                                       )                                                                          
    89         1         17.0     17.0      0.0          settings.configure(**kwargs)                                                   
    90         1     649781.0 649781.0     99.2          django.setup(set_prefix=False)                                                 
    91                                                   # https://laminlabs.slack.com/archives/C04FPE8V01W/p1698239551460289           
    92         1          2.0      2.0      0.0          from django.db.backends.base.base import BaseDatabaseWrapper                   
    93                                                                                                                                  
    94         1          2.0      2.0      0.0          BaseDatabaseWrapper.close_if_health_check_failed = close_if_health_check_failed
    95                                                                                                                                  
    96         1          0.0      0.0      0.0      if configure_only:                                                                 
    97                                                   return None                                                                    
    98                                                                                                                                  
    99                                               # migrations management                                                            
   100         1          0.0      0.0      0.0      if create_migrations:                                                              
   101                                                   call_command("makemigrations")                                                 
   102                                                   return None                                                                    
   103                                                                                                                                  
   104         1          0.0      0.0      0.0      if deploy_migrations:                                                              
   105                                                   call_command("migrate", verbosity=2)                                           
   106                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)                 
   107         1          0.0      0.0      0.0      elif init:                                                                         
   108                                                   global IS_MIGRATING                                                            
   109                                                   IS_MIGRATING = True                                                            
   110                                                   call_command("migrate", verbosity=0)                                           
   111                                                   IS_MIGRATING = False                                                           
   112                                                                                                                                  
   113                                               global IS_SETUP                                                                    
   114         1          0.0      0.0      0.0      IS_SETUP = True                                                                    
   115                                                                                                                                  
   116         1          2.0      2.0      0.0      if isettings.keep_artifacts_local:                                                 
   117                                                   isettings._search_local_root()                                                 


Total time: 0.655056 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py
Function: _load_db at line 459

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   459                                               @profile                                                                           
   460                                               def _load_db(self) -> tuple[bool, str]:                                            
   461                                                   # Is the database available and initialized as LaminDB?                        
   462                                                   # returns a tuple of status code and message                                   
   463         1          2.0      2.0      0.0          if self.dialect == "sqlite" and not self._sqlite_file.exists():                
   464                                                       legacy_file = self.storage.key_to_filepath(f"{self.name}.lndb")            
   465                                                       if legacy_file.exists():                                                   
   466                                                           raise RuntimeError(                                                    
   467                                                               "The SQLite file has been renamed!\nPlease rename your SQLite file"
   468                                                               f" {legacy_file} to {self._sqlite_file}"                           
   469                                                           )                                                                      
   470                                                       return False, f"SQLite file {self._sqlite_file} does not exist"            
   471         1          2.0      2.0      0.0          from lamindb_setup import settings  # to check user                            
   472                                                                                                                                  
   473         1          4.0      4.0      0.0          from .django import setup_django                                               
   474                                                                                                                                  
   475                                                   # we need the local sqlite to setup django                                     
   476         1          4.0      4.0      0.0          self._update_local_sqlite_file(lock_cloud_sqlite=self._is_cloud_sqlite)        
   477                                                   # setting up django also performs a check for migrations & prints them         
   478                                                   # as warnings                                                                  
   479                                                   # this should fail, e.g., if the db is not reachable                           
   480         1     655044.0 655044.0    100.0          setup_django(self)                                                             
   481         1          0.0      0.0      0.0          return True, ""                                                                


Total time: 1.08506 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 182

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   182                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   183                                           @profile                                                                               
   184                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   185                                               """Connect to instance.                                                            
   186                                                                                                                                  
   187                                               Args:                                                                              
   188                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   189                                                       If the instance is owned by you, it suffices to pass the instance name.    
   190                                               """                                                                                
   191                                               # validate kwargs                                                                  
   192         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   193                                                   "_db",                                                                         
   194                                                   "_write_settings",                                                             
   195                                                   "_raise_not_found_error",                                                      
   196                                                   "_test",                                                                       
   197                                                   "_user",                                                                       
   198                                               }                                                                                  
   199         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   200                                                   if kwarg not in valid_kwargs:                                                  
   201                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   202                                                                                                                                  
   203         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   204                                               # _db is still needed because it is called in init                                 
   205         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   206         1          1.0      1.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   207         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   208         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   209                                                                                                                                  
   210         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   211         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   212         1          1.0      1.0      0.0      if _user is not None:                                                              
   213                                                   access_token = _user.access_token                                              
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      try:                                                                               
   216         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   217                                                                                                                                  
   218         1       2818.0   2818.0      0.3          if _check_instance_setup() and not _test:                                      
   219                                                       if (                                                                       
   220                                                           settings._instance_exists                                              
   221                                                           and f"{owner}/{name}" == settings.instance.slug                        
   222                                                       ):                                                                         
   223                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   224                                                           return None                                                            
   225                                                       else:                                                                      
   226                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   227         2          1.0      0.5      0.0          elif (                                                                         
   228         1          0.0      0.0      0.0              _write_settings                                                            
   229         1       2123.0   2123.0      0.2              and settings._instance_exists                                              
   230         1          7.0      7.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   231                                                   ):                                                                             
   232                                                       close_instance(mute=True)                                                  
   233                                                                                                                                  
   234         1          0.0      0.0      0.0          try:                                                                           
   235         2     422850.0 211425.0     39.0              isettings = _connect_instance(                                             
   236         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   237                                                       )                                                                          
   238                                                   except InstanceNotFoundError as e:                                             
   239                                                       if _raise_not_found_error:                                                 
   240                                                           raise e                                                                
   241                                                       else:                                                                      
   242                                                           return "instance-not-found"                                            
   243         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   244                                                       return isettings                                                           
   245                                                   # at this point we have checked already that isettings is not a string         
   246                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   247                                                   # has no effect if _user is None or if not cloud sqlite instance               
   248         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   249         1       1013.0   1013.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   250         1          0.0      0.0      0.0          if _test:                                                                      
   251                                                       return None                                                                
   252         1          1.0      1.0      0.0          silence_loggers()                                                              
   253         1     655061.0 655061.0     60.4          check, msg = isettings._load_db()                                              
   254         1          0.0      0.0      0.0          if not check:                                                                  
   255                                                       local_db = (                                                               
   256                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   257                                                       )                                                                          
   258                                                       if local_db:                                                               
   259                                                           logger.warning(                                                        
   260                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   261                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   262                                                               " call: lamin load --unload"                                       
   263                                                           )                                                                      
   264                                                       elif _raise_not_found_error:                                               
   265                                                           raise SystemExit(msg)                                                  
   266                                                       else:                                                                      
   267                                                           logger.warning(                                                        
   268                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   269                                                               " loadable: re-initializing"                                       
   270                                                           )                                                                      
   271                                                           return "instance-corrupted-or-deleted"                                 
   272                                                   # this is for testing purposes only                                            
   273         1          0.0      0.0      0.0          if _TEST_FAILED_LOAD:                                                          
   274                                                       raise RuntimeError("Technical testing error.")                             
   275                                                                                                                                  
   276                                                   # below is for backfilling the instance_uid value                              
   277                                                   # we'll enable it once more people migrated to 0.71.0                          
   278                                                   # ssettings_record = isettings.storage.record                                  
   279                                                   # if ssettings_record.instance_uid is None:                                    
   280                                                   #     ssettings_record.instance_uid = isettings.uid                            
   281                                                   #     # try saving if not read-only access                                     
   282                                                   #     try:                                                                     
   283                                                   #         ssettings_record.save()                                              
   284                                                   #     # raised by django when the access is denied                             
   285                                                   #     except ProgrammingError:                                                 
   286                                                   #         pass                                                                 
   287         1       1161.0   1161.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   288                                               except Exception as e:                                                             
   289                                                   if isettings is not None:                                                      
   290                                                       if _write_settings:                                                        
   291                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   292                                                       settings._instance_settings = None                                         
   293                                                   raise e                                                                        
   294                                               # rename lnschema_bionty to bionty for sql tables                                  
   295         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   296         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   297         1         10.0     10.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   298                                                   )                                                                              
   299         1          7.0      7.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   300                                                       migrate_lnschema_bionty(                                                   
   301                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   302                                                       )                                                                          
   303         1          0.0      0.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py:389 - _connect_instance_new
  0.42 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:113 - _connect_instance
  0.66 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py:28 - setup_django
  0.66 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py:459 - _load_db
  1.09 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:182 - connect
(py310) falexwolf@mbpalex lamindb-setup % kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py
Function: _connect_instance_new at line 389

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   389                                           @profile                                                                          
   390                                           def _connect_instance_new(                                                        
   391                                               owner: str,  # account_handle                                                 
   392                                               name: str,  # instance_name                                                   
   393                                               client: Client,                                                               
   394                                           ) -> tuple[dict, dict] | str:                                                     
   395                                               response = client.functions.invoke(                                           
   396                                                   "get-instance-settings-v1",                                               
   397                                                   invoke_options={"body": {"owner": owner, "name": name}},                  
   398                                               )                                                                             
   399                                               # no instance found, check why is that                                        
   400                                               if response == b"{}":                                                         
   401                                                   # try the via single requests, will take more time                        
   402                                                   account = select_account_by_handle(owner, client)                         
   403                                                   if account is None:                                                       
   404                                                       return "account-not-exists"                                           
   405                                                   instance = select_instance_by_name(account["id"], name, client)           
   406                                                   if instance is None:                                                      
   407                                                       return "instance-not-found"                                           
   408                                                   # get default storage                                                     
   409                                                   storage = select_default_storage_by_instance_id(instance["id"], client)   
   410                                                   if storage is None:                                                       
   411                                                       return "default-storage-does-not-exist-on-hub"                        
   412                                                   logger.warning(                                                           
   413                                                       "Could not find instance via API, but found directly querying hub."   
   414                                                   )                                                                         
   415                                               else:                                                                         
   416                                                   instance = json.loads(response)                                           
   417                                                   storage = instance.pop("storage")                                         
   418                                                                                                                             
   419                                               if instance["db_scheme"] is not None:                                         
   420                                                   db_user_name, db_user_password = None, None                               
   421                                                   if "db_user_name" in instance and "db_user_password" in instance:         
   422                                                       db_user_name, db_user_password = (                                    
   423                                                           instance["db_user_name"],                                         
   424                                                           instance["db_user_password"],                                     
   425                                                       )                                                                     
   426                                                   else:                                                                     
   427                                                       db_user = select_db_user_by_instance(instance["id"], client)          
   428                                                       if db_user is not None:                                               
   429                                                           db_user_name, db_user_password = (                                
   430                                                               db_user["db_user_name"],                                      
   431                                                               db_user["db_user_password"],                                  
   432                                                           )                                                                 
   433                                                   db_dsn = LaminDsn.build(                                                  
   434                                                       scheme=instance["db_scheme"],                                         
   435                                                       user=db_user_name if db_user_name is not None else "none",            
   436                                                       password=db_user_password if db_user_password is not None else "none",
   437                                                       host=instance["db_host"],                                             
   438                                                       port=instance["db_port"],                                             
   439                                                       database=instance["db_database"],                                     
   440                                                   )                                                                         
   441                                                   instance["db"] = db_dsn                                                   
   442                                               return instance, storage  # type: ignore                                      


Total time: 0.404152 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: _connect_instance at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                                                       
   114                                           def _connect_instance(                                                                         
   115                                               owner: str,                                                                                
   116                                               name: str,                                                                                 
   117                                               *,                                                                                         
   118                                               db: str | None = None,                                                                     
   119                                               raise_permission_error: bool = True,                                                       
   120                                               access_token: str | None = None,                                                           
   121                                           ) -> InstanceSettings:                                                                         
   122         1          8.0      8.0      0.0      settings_file = instance_settings_file(name, owner)                                        
   123         1          0.0      0.0      0.0      make_hub_request = True                                                                    
   124         1         10.0     10.0      0.0      if settings_file.exists():                                                                 
   125         1       2077.0   2077.0      0.5          isettings = load_instance_settings(settings_file)                                      
   126                                                   # skip hub request for a purely local instance                                         
   127         1        175.0    175.0      0.0          make_hub_request = isettings.is_remote                                                 
   128         1          0.0      0.0      0.0      if make_hub_request:                                                                       
   129                                                   # the following will return a string if the instance does not exist                    
   130                                                   # on the hub                                                                           
   131                                                   # do not call hub if the user is anonymous                                             
   132         1          1.0      1.0      0.0          if owner != "anonymous":                                                               
   133         2     399313.0 199656.5     98.8              hub_result = connect_instance_from_hub(                                            
   134         1          0.0      0.0      0.0                  owner=owner, name=name, access_token=access_token                              
   135                                                       )                                                                                  
   136                                                   else:                                                                                  
   137                                                       hub_result = "anonymous-user"                                                      
   138                                                   # if hub_result is not a string, it means it made a request                            
   139                                                   # that successfully returned metadata                                                  
   140         1          2.0      2.0      0.0          if not isinstance(hub_result, str):                                                    
   141         1          0.0      0.0      0.0              instance_result, storage_result = hub_result                                       
   142         2       2503.0   1251.5      0.6              db_updated = update_db_using_local(                                                
   143         1          1.0      1.0      0.0                  instance_result,                                                               
   144         1          0.0      0.0      0.0                  settings_file,                                                                 
   145         1          0.0      0.0      0.0                  db=db,                                                                         
   146         1          0.0      0.0      0.0                  raise_permission_error=raise_permission_error,                                 
   147                                                       )                                                                                  
   148         2         41.0     20.5      0.0              ssettings = StorageSettings(                                                       
   149         1          0.0      0.0      0.0                  root=storage_result["root"],                                                   
   150         1          0.0      0.0      0.0                  region=storage_result["region"],                                               
   151         1          1.0      1.0      0.0                  uid=storage_result["lnid"],                                                    
   152         1          3.0      3.0      0.0                  uuid=UUID(storage_result["id"]),                                               
   153         1          2.0      2.0      0.0                  instance_id=UUID(instance_result["id"]),                                       
   154                                                       )                                                                                  
   155         2         13.0      6.5      0.0              isettings = InstanceSettings(                                                      
   156         1          2.0      2.0      0.0                  id=UUID(instance_result["id"]),                                                
   157         1          0.0      0.0      0.0                  owner=owner,                                                                   
   158         1          0.0      0.0      0.0                  name=name,                                                                     
   159         1          0.0      0.0      0.0                  storage=ssettings,                                                             
   160         1          0.0      0.0      0.0                  db=db_updated,                                                                 
   161         1          0.0      0.0      0.0                  schema=instance_result["schema_str"],                                          
   162         1          0.0      0.0      0.0                  git_repo=instance_result["git_repo"],                                          
   163         1          0.0      0.0      0.0                  keep_artifacts_local=bool(instance_result["keep_artifacts_local"]),            
   164         1          0.0      0.0      0.0                  is_on_hub=True,                                                                
   165                                                       )                                                                                  
   166                                                   else:                                                                                  
   167                                                       if hub_result != "anonymous-user":                                                 
   168                                                           message = INSTANCE_NOT_FOUND_MESSAGE.format(                                   
   169                                                               owner=owner, name=name, hub_result=hub_result                              
   170                                                           )                                                                              
   171                                                       else:                                                                              
   172                                                           message = "It is not possible to load an anonymous-owned instance from the hub"
   173                                                       if settings_file.exists():                                                         
   174                                                           isettings = load_instance_settings(settings_file)                              
   175                                                           if isettings.is_remote:                                                        
   176                                                               raise InstanceNotFoundError(message)                                       
   177                                                       else:                                                                              
   178                                                           raise InstanceNotFoundError(message)                                           
   179         1          0.0      0.0      0.0      return isettings                                                                           


Total time: 0.680914 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py
Function: setup_django at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @profile                                                                               
    29                                           def setup_django(                                                                      
    30                                               isettings: InstanceSettings,                                                       
    31                                               deploy_migrations: bool = False,                                                   
    32                                               create_migrations: bool = False,                                                   
    33                                               configure_only: bool = False,                                                      
    34                                               init: bool = False,                                                                
    35                                               view_schema: bool = False,                                                         
    36                                           ):                                                                                     
    37         1          1.0      1.0      0.0      if IS_RUN_FROM_IPYTHON:                                                            
    38                                                   os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"                               
    39                                                                                                                                  
    40         1        381.0    381.0      0.1      import dj_database_url                                                             
    41         1          1.0      1.0      0.0      import django                                                                      
    42         1          2.0      2.0      0.0      from django.conf import settings                                                   
    43         1       3580.0   3580.0      0.5      from django.core.management import call_command                                    
    44                                                                                                                                  
    45                                               # configuration                                                                    
    46         1          4.0      4.0      0.0      if not settings.configured:                                                        
    47         2         30.0     15.0      0.0          default_db = dj_database_url.config(                                           
    48         1          0.0      0.0      0.0              env="LAMINDB_DJANGO_DATABASE_URL",                                         
    49         1          5.0      5.0      0.0              default=isettings.db,                                                      
    50                                                       # see comment next to patching BaseDatabaseWrapper below                   
    51         1          0.0      0.0      0.0              conn_max_age=CONN_MAX_AGE,                                                 
    52         1          0.0      0.0      0.0              conn_health_checks=True,                                                   
    53                                                   )                                                                              
    54         1          0.0      0.0      0.0          DATABASES = {                                                                  
    55         1          0.0      0.0      0.0              "default": default_db,                                                     
    56                                                   }                                                                              
    57         1          4.0      4.0      0.0          from .._init_instance import get_schema_module_name                            
    58                                                                                                                                  
    59         1          3.0      3.0      0.0          schema_names = ["core"] + list(isettings.schema)                               
    60         1        971.0    971.0      0.1          installed_apps = [get_schema_module_name(n) for n in schema_names]             
    61         1          0.0      0.0      0.0          if view_schema:                                                                
    62                                                       installed_apps = installed_apps[::-1]  # to fix how apps appear            
    63                                                       installed_apps += ["schema_graph", "django.contrib.staticfiles"]           
    64                                                                                                                                  
    65         2          0.0      0.0      0.0          kwargs = dict(                                                                 
    66         1          0.0      0.0      0.0              INSTALLED_APPS=installed_apps,                                             
    67         1          0.0      0.0      0.0              DATABASES=DATABASES,                                                       
    68         1          0.0      0.0      0.0              DEFAULT_AUTO_FIELD="django.db.models.BigAutoField",                        
    69         1          0.0      0.0      0.0              TIME_ZONE="UTC",                                                           
    70         1          0.0      0.0      0.0              USE_TZ=True,                                                               
    71                                                   )                                                                              
    72         1          0.0      0.0      0.0          if view_schema:                                                                
    73                                                       kwargs.update(                                                             
    74                                                           DEBUG=True,                                                            
    75                                                           ROOT_URLCONF="lamindb_setup._schema",                                  
    76                                                           SECRET_KEY="dummy",                                                    
    77                                                           TEMPLATES=[                                                            
    78                                                               {                                                                  
    79                                                                   "BACKEND": "django.template.backends.django.DjangoTemplates",  
    80                                                                   "APP_DIRS": True,                                              
    81                                                               },                                                                 
    82                                                           ],                                                                     
    83                                                           STATIC_ROOT=f"{Path.home().as_posix()}/.lamin/",                       
    84                                                           STATICFILES_FINDERS=[                                                  
    85                                                               "django.contrib.staticfiles.finders.AppDirectoriesFinder",         
    86                                                           ],                                                                     
    87                                                           STATIC_URL="static/",                                                  
    88                                                       )                                                                          
    89         1         19.0     19.0      0.0          settings.configure(**kwargs)                                                   
    90         1     675906.0 675906.0     99.3          django.setup(set_prefix=False)                                                 
    91                                                   # https://laminlabs.slack.com/archives/C04FPE8V01W/p1698239551460289           
    92         1          2.0      2.0      0.0          from django.db.backends.base.base import BaseDatabaseWrapper                   
    93                                                                                                                                  
    94         1          2.0      2.0      0.0          BaseDatabaseWrapper.close_if_health_check_failed = close_if_health_check_failed
    95                                                                                                                                  
    96         1          0.0      0.0      0.0      if configure_only:                                                                 
    97                                                   return None                                                                    
    98                                                                                                                                  
    99                                               # migrations management                                                            
   100         1          0.0      0.0      0.0      if create_migrations:                                                              
   101                                                   call_command("makemigrations")                                                 
   102                                                   return None                                                                    
   103                                                                                                                                  
   104         1          0.0      0.0      0.0      if deploy_migrations:                                                              
   105                                                   call_command("migrate", verbosity=2)                                           
   106                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)                 
   107         1          1.0      1.0      0.0      elif init:                                                                         
   108                                                   global IS_MIGRATING                                                            
   109                                                   IS_MIGRATING = True                                                            
   110                                                   call_command("migrate", verbosity=0)                                           
   111                                                   IS_MIGRATING = False                                                           
   112                                                                                                                                  
   113                                               global IS_SETUP                                                                    
   114         1          0.0      0.0      0.0      IS_SETUP = True                                                                    
   115                                                                                                                                  
   116         1          2.0      2.0      0.0      if isettings.keep_artifacts_local:                                                 
   117                                                   isettings._search_local_root()                                                 


Total time: 0.680939 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py
Function: _load_db at line 459

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   459                                               @profile                                                                           
   460                                               def _load_db(self) -> tuple[bool, str]:                                            
   461                                                   # Is the database available and initialized as LaminDB?                        
   462                                                   # returns a tuple of status code and message                                   
   463         1          2.0      2.0      0.0          if self.dialect == "sqlite" and not self._sqlite_file.exists():                
   464                                                       legacy_file = self.storage.key_to_filepath(f"{self.name}.lndb")            
   465                                                       if legacy_file.exists():                                                   
   466                                                           raise RuntimeError(                                                    
   467                                                               "The SQLite file has been renamed!\nPlease rename your SQLite file"
   468                                                               f" {legacy_file} to {self._sqlite_file}"                           
   469                                                           )                                                                      
   470                                                       return False, f"SQLite file {self._sqlite_file} does not exist"            
   471         1          3.0      3.0      0.0          from lamindb_setup import settings  # to check user                            
   472                                                                                                                                  
   473         1          2.0      2.0      0.0          from .django import setup_django                                               
   474                                                                                                                                  
   475                                                   # we need the local sqlite to setup django                                     
   476         1          3.0      3.0      0.0          self._update_local_sqlite_file(lock_cloud_sqlite=self._is_cloud_sqlite)        
   477                                                   # setting up django also performs a check for migrations & prints them         
   478                                                   # as warnings                                                                  
   479                                                   # this should fail, e.g., if the db is not reachable                           
   480         1     680929.0 680929.0    100.0          setup_django(self)                                                             
   481         1          0.0      0.0      0.0          return True, ""                                                                


Total time: 1.09231 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 182

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   182                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   183                                           @profile                                                                               
   184                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   185                                               """Connect to instance.                                                            
   186                                                                                                                                  
   187                                               Args:                                                                              
   188                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   189                                                       If the instance is owned by you, it suffices to pass the instance name.    
   190                                               """                                                                                
   191                                               # validate kwargs                                                                  
   192         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   193                                                   "_db",                                                                         
   194                                                   "_write_settings",                                                             
   195                                                   "_raise_not_found_error",                                                      
   196                                                   "_test",                                                                       
   197                                                   "_user",                                                                       
   198                                               }                                                                                  
   199         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   200                                                   if kwarg not in valid_kwargs:                                                  
   201                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   202                                                                                                                                  
   203         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   204                                               # _db is still needed because it is called in init                                 
   205         1          1.0      1.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   206         1          0.0      0.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   207         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   208         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   209                                                                                                                                  
   210         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   211         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   212         1          0.0      0.0      0.0      if _user is not None:                                                              
   213                                                   access_token = _user.access_token                                              
   214                                                                                                                                  
   215         1          1.0      1.0      0.0      try:                                                                               
   216         1          6.0      6.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   217                                                                                                                                  
   218         1       2809.0   2809.0      0.3          if _check_instance_setup() and not _test:                                      
   219                                                       if (                                                                       
   220                                                           settings._instance_exists                                              
   221                                                           and f"{owner}/{name}" == settings.instance.slug                        
   222                                                       ):                                                                         
   223                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   224                                                           return None                                                            
   225                                                       else:                                                                      
   226                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   227         2          0.0      0.0      0.0          elif (                                                                         
   228         1          0.0      0.0      0.0              _write_settings                                                            
   229         1       2172.0   2172.0      0.2              and settings._instance_exists                                              
   230         1         14.0     14.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   231                                                   ):                                                                             
   232                                                       close_instance(mute=True)                                                  
   233                                                                                                                                  
   234         1          0.0      0.0      0.0          try:                                                                           
   235         2     404173.0 202086.5     37.0              isettings = _connect_instance(                                             
   236         1          1.0      1.0      0.0                  owner, name, db=_db, access_token=access_token                         
   237                                                       )                                                                          
   238                                                   except InstanceNotFoundError as e:                                             
   239                                                       if _raise_not_found_error:                                                 
   240                                                           raise e                                                                
   241                                                       else:                                                                      
   242                                                           return "instance-not-found"                                            
   243         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   244                                                       return isettings                                                           
   245                                                   # at this point we have checked already that isettings is not a string         
   246                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   247                                                   # has no effect if _user is None or if not cloud sqlite instance               
   248         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   249         1        989.0    989.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   250         1          0.0      0.0      0.0          if _test:                                                                      
   251                                                       return None                                                                
   252         1          1.0      1.0      0.0          silence_loggers()                                                              
   253         1     680944.0 680944.0     62.3          check, msg = isettings._load_db()                                              
   254         1          0.0      0.0      0.0          if not check:                                                                  
   255                                                       local_db = (                                                               
   256                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   257                                                       )                                                                          
   258                                                       if local_db:                                                               
   259                                                           logger.warning(                                                        
   260                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   261                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   262                                                               " call: lamin load --unload"                                       
   263                                                           )                                                                      
   264                                                       elif _raise_not_found_error:                                               
   265                                                           raise SystemExit(msg)                                                  
   266                                                       else:                                                                      
   267                                                           logger.warning(                                                        
   268                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   269                                                               " loadable: re-initializing"                                       
   270                                                           )                                                                      
   271                                                           return "instance-corrupted-or-deleted"                                 
   272                                                   # this is for testing purposes only                                            
   273         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   274                                                       raise RuntimeError("Technical testing error.")                             
   275                                                                                                                                  
   276                                                   # below is for backfilling the instance_uid value                              
   277                                                   # we'll enable it once more people migrated to 0.71.0                          
   278                                                   # ssettings_record = isettings.storage.record                                  
   279                                                   # if ssettings_record.instance_uid is None:                                    
   280                                                   #     ssettings_record.instance_uid = isettings.uid                            
   281                                                   #     # try saving if not read-only access                                     
   282                                                   #     try:                                                                     
   283                                                   #         ssettings_record.save()                                              
   284                                                   #     # raised by django when the access is denied                             
   285                                                   #     except ProgrammingError:                                                 
   286                                                   #         pass                                                                 
   287         1       1176.0   1176.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   288                                               except Exception as e:                                                             
   289                                                   if isettings is not None:                                                      
   290                                                       if _write_settings:                                                        
   291                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   292                                                       settings._instance_settings = None                                         
   293                                                   raise e                                                                        
   294                                               # rename lnschema_bionty to bionty for sql tables                                  
   295         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   296         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   297         1          9.0      9.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   298                                                   )                                                                              
   299         1          8.0      8.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   300                                                       migrate_lnschema_bionty(                                                   
   301                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   302                                                       )                                                                          
   303         1          0.0      0.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py:389 - _connect_instance_new
  0.40 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:113 - _connect_instance
  0.68 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py:28 - setup_django
  0.68 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py:459 - _load_db
  1.09 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:182 - connect
(py310) falexwolf@mbpalex lamindb-setup % kernprof -l lamin connect https://lamin.ai/laminlabs/lamindata
→ connected lamindb: laminlabs/lamindata
Wrote profile results to lamin.lprof
Inspect results with:
python -m line_profiler -rmt "lamin.lprof"
(py310) falexwolf@mbpalex lamindb-setup % python -m line_profiler -rmt "lamin.lprof"                    
Timer unit: 1e-06 s

Total time: 0 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py
Function: _connect_instance_new at line 389

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   389                                           @profile                                                                          
   390                                           def _connect_instance_new(                                                        
   391                                               owner: str,  # account_handle                                                 
   392                                               name: str,  # instance_name                                                   
   393                                               client: Client,                                                               
   394                                           ) -> tuple[dict, dict] | str:                                                     
   395                                               response = client.functions.invoke(                                           
   396                                                   "get-instance-settings-v1",                                               
   397                                                   invoke_options={"body": {"owner": owner, "name": name}},                  
   398                                               )                                                                             
   399                                               # no instance found, check why is that                                        
   400                                               if response == b"{}":                                                         
   401                                                   # try the via single requests, will take more time                        
   402                                                   account = select_account_by_handle(owner, client)                         
   403                                                   if account is None:                                                       
   404                                                       return "account-not-exists"                                           
   405                                                   instance = select_instance_by_name(account["id"], name, client)           
   406                                                   if instance is None:                                                      
   407                                                       return "instance-not-found"                                           
   408                                                   # get default storage                                                     
   409                                                   storage = select_default_storage_by_instance_id(instance["id"], client)   
   410                                                   if storage is None:                                                       
   411                                                       return "default-storage-does-not-exist-on-hub"                        
   412                                                   logger.warning(                                                           
   413                                                       "Could not find instance via API, but found directly querying hub."   
   414                                                   )                                                                         
   415                                               else:                                                                         
   416                                                   instance = json.loads(response)                                           
   417                                                   storage = instance.pop("storage")                                         
   418                                                                                                                             
   419                                               if instance["db_scheme"] is not None:                                         
   420                                                   db_user_name, db_user_password = None, None                               
   421                                                   if "db_user_name" in instance and "db_user_password" in instance:         
   422                                                       db_user_name, db_user_password = (                                    
   423                                                           instance["db_user_name"],                                         
   424                                                           instance["db_user_password"],                                     
   425                                                       )                                                                     
   426                                                   else:                                                                     
   427                                                       db_user = select_db_user_by_instance(instance["id"], client)          
   428                                                       if db_user is not None:                                               
   429                                                           db_user_name, db_user_password = (                                
   430                                                               db_user["db_user_name"],                                      
   431                                                               db_user["db_user_password"],                                  
   432                                                           )                                                                 
   433                                                   db_dsn = LaminDsn.build(                                                  
   434                                                       scheme=instance["db_scheme"],                                         
   435                                                       user=db_user_name if db_user_name is not None else "none",            
   436                                                       password=db_user_password if db_user_password is not None else "none",
   437                                                       host=instance["db_host"],                                             
   438                                                       port=instance["db_port"],                                             
   439                                                       database=instance["db_database"],                                     
   440                                                   )                                                                         
   441                                                   instance["db"] = db_dsn                                                   
   442                                               return instance, storage  # type: ignore                                      


Total time: 0.386866 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: _connect_instance at line 113

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   113                                           @profile                                                                                       
   114                                           def _connect_instance(                                                                         
   115                                               owner: str,                                                                                
   116                                               name: str,                                                                                 
   117                                               *,                                                                                         
   118                                               db: str | None = None,                                                                     
   119                                               raise_permission_error: bool = True,                                                       
   120                                               access_token: str | None = None,                                                           
   121                                           ) -> InstanceSettings:                                                                         
   122         1          8.0      8.0      0.0      settings_file = instance_settings_file(name, owner)                                        
   123         1          0.0      0.0      0.0      make_hub_request = True                                                                    
   124         1         10.0     10.0      0.0      if settings_file.exists():                                                                 
   125         1       2077.0   2077.0      0.5          isettings = load_instance_settings(settings_file)                                      
   126                                                   # skip hub request for a purely local instance                                         
   127         1        154.0    154.0      0.0          make_hub_request = isettings.is_remote                                                 
   128         1          1.0      1.0      0.0      if make_hub_request:                                                                       
   129                                                   # the following will return a string if the instance does not exist                    
   130                                                   # on the hub                                                                           
   131                                                   # do not call hub if the user is anonymous                                             
   132         1          0.0      0.0      0.0          if owner != "anonymous":                                                               
   133         2     381994.0 190997.0     98.7              hub_result = connect_instance_from_hub(                                            
   134         1          0.0      0.0      0.0                  owner=owner, name=name, access_token=access_token                              
   135                                                       )                                                                                  
   136                                                   else:                                                                                  
   137                                                       hub_result = "anonymous-user"                                                      
   138                                                   # if hub_result is not a string, it means it made a request                            
   139                                                   # that successfully returned metadata                                                  
   140         1          0.0      0.0      0.0          if not isinstance(hub_result, str):                                                    
   141         1          0.0      0.0      0.0              instance_result, storage_result = hub_result                                       
   142         2       2566.0   1283.0      0.7              db_updated = update_db_using_local(                                                
   143         1          0.0      0.0      0.0                  instance_result,                                                               
   144         1          0.0      0.0      0.0                  settings_file,                                                                 
   145         1          0.0      0.0      0.0                  db=db,                                                                         
   146         1          0.0      0.0      0.0                  raise_permission_error=raise_permission_error,                                 
   147                                                       )                                                                                  
   148         2         38.0     19.0      0.0              ssettings = StorageSettings(                                                       
   149         1          0.0      0.0      0.0                  root=storage_result["root"],                                                   
   150         1          0.0      0.0      0.0                  region=storage_result["region"],                                               
   151         1          0.0      0.0      0.0                  uid=storage_result["lnid"],                                                    
   152         1          3.0      3.0      0.0                  uuid=UUID(storage_result["id"]),                                               
   153         1          2.0      2.0      0.0                  instance_id=UUID(instance_result["id"]),                                       
   154                                                       )                                                                                  
   155         2         11.0      5.5      0.0              isettings = InstanceSettings(                                                      
   156         1          2.0      2.0      0.0                  id=UUID(instance_result["id"]),                                                
   157         1          0.0      0.0      0.0                  owner=owner,                                                                   
   158         1          0.0      0.0      0.0                  name=name,                                                                     
   159         1          0.0      0.0      0.0                  storage=ssettings,                                                             
   160         1          0.0      0.0      0.0                  db=db_updated,                                                                 
   161         1          0.0      0.0      0.0                  schema=instance_result["schema_str"],                                          
   162         1          0.0      0.0      0.0                  git_repo=instance_result["git_repo"],                                          
   163         1          0.0      0.0      0.0                  keep_artifacts_local=bool(instance_result["keep_artifacts_local"]),            
   164         1          0.0      0.0      0.0                  is_on_hub=True,                                                                
   165                                                       )                                                                                  
   166                                                   else:                                                                                  
   167                                                       if hub_result != "anonymous-user":                                                 
   168                                                           message = INSTANCE_NOT_FOUND_MESSAGE.format(                                   
   169                                                               owner=owner, name=name, hub_result=hub_result                              
   170                                                           )                                                                              
   171                                                       else:                                                                              
   172                                                           message = "It is not possible to load an anonymous-owned instance from the hub"
   173                                                       if settings_file.exists():                                                         
   174                                                           isettings = load_instance_settings(settings_file)                              
   175                                                           if isettings.is_remote:                                                        
   176                                                               raise InstanceNotFoundError(message)                                       
   177                                                       else:                                                                              
   178                                                           raise InstanceNotFoundError(message)                                           
   179         1          0.0      0.0      0.0      return isettings                                                                           


Total time: 0.64292 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py
Function: setup_django at line 28

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    28                                           @profile                                                                               
    29                                           def setup_django(                                                                      
    30                                               isettings: InstanceSettings,                                                       
    31                                               deploy_migrations: bool = False,                                                   
    32                                               create_migrations: bool = False,                                                   
    33                                               configure_only: bool = False,                                                      
    34                                               init: bool = False,                                                                
    35                                               view_schema: bool = False,                                                         
    36                                           ):                                                                                     
    37         1          0.0      0.0      0.0      if IS_RUN_FROM_IPYTHON:                                                            
    38                                                   os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"                               
    39                                                                                                                                  
    40         1        411.0    411.0      0.1      import dj_database_url                                                             
    41         1          1.0      1.0      0.0      import django                                                                      
    42         1          2.0      2.0      0.0      from django.conf import settings                                                   
    43         1       3710.0   3710.0      0.6      from django.core.management import call_command                                    
    44                                                                                                                                  
    45                                               # configuration                                                                    
    46         1          4.0      4.0      0.0      if not settings.configured:                                                        
    47         2         30.0     15.0      0.0          default_db = dj_database_url.config(                                           
    48         1          0.0      0.0      0.0              env="LAMINDB_DJANGO_DATABASE_URL",                                         
    49         1          4.0      4.0      0.0              default=isettings.db,                                                      
    50                                                       # see comment next to patching BaseDatabaseWrapper below                   
    51         1          1.0      1.0      0.0              conn_max_age=CONN_MAX_AGE,                                                 
    52         1          0.0      0.0      0.0              conn_health_checks=True,                                                   
    53                                                   )                                                                              
    54         1          1.0      1.0      0.0          DATABASES = {                                                                  
    55         1          0.0      0.0      0.0              "default": default_db,                                                     
    56                                                   }                                                                              
    57         1          2.0      2.0      0.0          from .._init_instance import get_schema_module_name                            
    58                                                                                                                                  
    59         1          3.0      3.0      0.0          schema_names = ["core"] + list(isettings.schema)                               
    60         1        986.0    986.0      0.2          installed_apps = [get_schema_module_name(n) for n in schema_names]             
    61         1          0.0      0.0      0.0          if view_schema:                                                                
    62                                                       installed_apps = installed_apps[::-1]  # to fix how apps appear            
    63                                                       installed_apps += ["schema_graph", "django.contrib.staticfiles"]           
    64                                                                                                                                  
    65         2          1.0      0.5      0.0          kwargs = dict(                                                                 
    66         1          0.0      0.0      0.0              INSTALLED_APPS=installed_apps,                                             
    67         1          0.0      0.0      0.0              DATABASES=DATABASES,                                                       
    68         1          0.0      0.0      0.0              DEFAULT_AUTO_FIELD="django.db.models.BigAutoField",                        
    69         1          0.0      0.0      0.0              TIME_ZONE="UTC",                                                           
    70         1          0.0      0.0      0.0              USE_TZ=True,                                                               
    71                                                   )                                                                              
    72         1          0.0      0.0      0.0          if view_schema:                                                                
    73                                                       kwargs.update(                                                             
    74                                                           DEBUG=True,                                                            
    75                                                           ROOT_URLCONF="lamindb_setup._schema",                                  
    76                                                           SECRET_KEY="dummy",                                                    
    77                                                           TEMPLATES=[                                                            
    78                                                               {                                                                  
    79                                                                   "BACKEND": "django.template.backends.django.DjangoTemplates",  
    80                                                                   "APP_DIRS": True,                                              
    81                                                               },                                                                 
    82                                                           ],                                                                     
    83                                                           STATIC_ROOT=f"{Path.home().as_posix()}/.lamin/",                       
    84                                                           STATICFILES_FINDERS=[                                                  
    85                                                               "django.contrib.staticfiles.finders.AppDirectoriesFinder",         
    86                                                           ],                                                                     
    87                                                           STATIC_URL="static/",                                                  
    88                                                       )                                                                          
    89         1         17.0     17.0      0.0          settings.configure(**kwargs)                                                   
    90         1     637743.0 637743.0     99.2          django.setup(set_prefix=False)                                                 
    91                                                   # https://laminlabs.slack.com/archives/C04FPE8V01W/p1698239551460289           
    92         1          1.0      1.0      0.0          from django.db.backends.base.base import BaseDatabaseWrapper                   
    93                                                                                                                                  
    94         1          1.0      1.0      0.0          BaseDatabaseWrapper.close_if_health_check_failed = close_if_health_check_failed
    95                                                                                                                                  
    96         1          0.0      0.0      0.0      if configure_only:                                                                 
    97                                                   return None                                                                    
    98                                                                                                                                  
    99                                               # migrations management                                                            
   100         1          0.0      0.0      0.0      if create_migrations:                                                              
   101                                                   call_command("makemigrations")                                                 
   102                                                   return None                                                                    
   103                                                                                                                                  
   104         1          0.0      0.0      0.0      if deploy_migrations:                                                              
   105                                                   call_command("migrate", verbosity=2)                                           
   106                                                   isettings._update_cloud_sqlite_file(unlock_cloud_sqlite=False)                 
   107         1          0.0      0.0      0.0      elif init:                                                                         
   108                                                   global IS_MIGRATING                                                            
   109                                                   IS_MIGRATING = True                                                            
   110                                                   call_command("migrate", verbosity=0)                                           
   111                                                   IS_MIGRATING = False                                                           
   112                                                                                                                                  
   113                                               global IS_SETUP                                                                    
   114         1          0.0      0.0      0.0      IS_SETUP = True                                                                    
   115                                                                                                                                  
   116         1          2.0      2.0      0.0      if isettings.keep_artifacts_local:                                                 
   117                                                   isettings._search_local_root()                                                 


Total time: 0.642954 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py
Function: _load_db at line 459

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   459                                               @profile                                                                           
   460                                               def _load_db(self) -> tuple[bool, str]:                                            
   461                                                   # Is the database available and initialized as LaminDB?                        
   462                                                   # returns a tuple of status code and message                                   
   463         1          3.0      3.0      0.0          if self.dialect == "sqlite" and not self._sqlite_file.exists():                
   464                                                       legacy_file = self.storage.key_to_filepath(f"{self.name}.lndb")            
   465                                                       if legacy_file.exists():                                                   
   466                                                           raise RuntimeError(                                                    
   467                                                               "The SQLite file has been renamed!\nPlease rename your SQLite file"
   468                                                               f" {legacy_file} to {self._sqlite_file}"                           
   469                                                           )                                                                      
   470                                                       return False, f"SQLite file {self._sqlite_file} does not exist"            
   471         1          3.0      3.0      0.0          from lamindb_setup import settings  # to check user                            
   472                                                                                                                                  
   473         1          2.0      2.0      0.0          from .django import setup_django                                               
   474                                                                                                                                  
   475                                                   # we need the local sqlite to setup django                                     
   476         1          2.0      2.0      0.0          self._update_local_sqlite_file(lock_cloud_sqlite=self._is_cloud_sqlite)        
   477                                                   # setting up django also performs a check for migrations & prints them         
   478                                                   # as warnings                                                                  
   479                                                   # this should fail, e.g., if the db is not reachable                           
   480         1     642943.0 642943.0    100.0          setup_django(self)                                                             
   481         1          1.0      1.0      0.0          return True, ""                                                                


Total time: 1.03691 s
File: /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py
Function: connect at line 182

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   182                                           @unlock_cloud_sqlite_upon_exception(ignore_prev_locker=True)                           
   183                                           @profile                                                                               
   184                                           def connect(slug: str, **kwargs) -> str | tuple | None:                                
   185                                               """Connect to instance.                                                            
   186                                                                                                                                  
   187                                               Args:                                                                              
   188                                                   slug: The instance slug `account_handle/instance_name` or URL.                 
   189                                                       If the instance is owned by you, it suffices to pass the instance name.    
   190                                               """                                                                                
   191                                               # validate kwargs                                                                  
   192         1          1.0      1.0      0.0      valid_kwargs = {                                                                   
   193                                                   "_db",                                                                         
   194                                                   "_write_settings",                                                             
   195                                                   "_raise_not_found_error",                                                      
   196                                                   "_test",                                                                       
   197                                                   "_user",                                                                       
   198                                               }                                                                                  
   199         1          1.0      1.0      0.0      for kwarg in kwargs:                                                               
   200                                                   if kwarg not in valid_kwargs:                                                  
   201                                                       raise TypeError(f"connect() got unexpected keyword argument '{kwarg}'")    
   202                                                                                                                                  
   203         1          0.0      0.0      0.0      isettings: InstanceSettings = None  # type: ignore                                 
   204                                               # _db is still needed because it is called in init                                 
   205         1          0.0      0.0      0.0      _db: str | None = kwargs.get("_db", None)                                          
   206         1          1.0      1.0      0.0      _write_settings: bool = kwargs.get("_write_settings", True)                        
   207         1          0.0      0.0      0.0      _raise_not_found_error: bool = kwargs.get("_raise_not_found_error", True)          
   208         1          0.0      0.0      0.0      _test: bool = kwargs.get("_test", False)                                           
   209                                                                                                                                  
   210         1          0.0      0.0      0.0      access_token: str | None = None                                                    
   211         1          0.0      0.0      0.0      _user: UserSettings | None = kwargs.get("_user", None)                             
   212         1          0.0      0.0      0.0      if _user is not None:                                                              
   213                                                   access_token = _user.access_token                                              
   214                                                                                                                                  
   215         1          0.0      0.0      0.0      try:                                                                               
   216         1          3.0      3.0      0.0          owner, name = get_owner_name_from_identifier(slug)                             
   217                                                                                                                                  
   218         1       2772.0   2772.0      0.3          if _check_instance_setup() and not _test:                                      
   219                                                       if (                                                                       
   220                                                           settings._instance_exists                                              
   221                                                           and f"{owner}/{name}" == settings.instance.slug                        
   222                                                       ):                                                                         
   223                                                           logger.info(f"connected lamindb: {settings.instance.slug}")            
   224                                                           return None                                                            
   225                                                       else:                                                                      
   226                                                           raise RuntimeError(MESSAGE_NO_MULTIPLE_INSTANCE)                       
   227         2          0.0      0.0      0.0          elif (                                                                         
   228         1          0.0      0.0      0.0              _write_settings                                                            
   229         1       2129.0   2129.0      0.2              and settings._instance_exists                                              
   230         1         10.0     10.0      0.0              and f"{owner}/{name}" != settings.instance.slug                            
   231                                                   ):                                                                             
   232                                                       close_instance(mute=True)                                                  
   233                                                                                                                                  
   234         1          0.0      0.0      0.0          try:                                                                           
   235         2     386904.0 193452.0     37.3              isettings = _connect_instance(                                             
   236         1          0.0      0.0      0.0                  owner, name, db=_db, access_token=access_token                         
   237                                                       )                                                                          
   238                                                   except InstanceNotFoundError as e:                                             
   239                                                       if _raise_not_found_error:                                                 
   240                                                           raise e                                                                
   241                                                       else:                                                                      
   242                                                           return "instance-not-found"                                            
   243         1          0.0      0.0      0.0          if isinstance(isettings, str):                                                 
   244                                                       return isettings                                                           
   245                                                   # at this point we have checked already that isettings is not a string         
   246                                                   # _user is passed to lock cloud sqite for this user in isettings._load_db()    
   247                                                   # has no effect if _user is None or if not cloud sqlite instance               
   248         1          0.0      0.0      0.0          isettings._locker_user = _user                                                 
   249         1        982.0    982.0      0.1          isettings._persist(write_to_disk=_write_settings)                              
   250         1          0.0      0.0      0.0          if _test:                                                                      
   251                                                       return None                                                                
   252         1          1.0      1.0      0.0          silence_loggers()                                                              
   253         1     642959.0 642959.0     62.0          check, msg = isettings._load_db()                                              
   254         1          0.0      0.0      0.0          if not check:                                                                  
   255                                                       local_db = (                                                               
   256                                                           isettings._is_cloud_sqlite and isettings._sqlite_file_local.exists()   
   257                                                       )                                                                          
   258                                                       if local_db:                                                               
   259                                                           logger.warning(                                                        
   260                                                               "SQLite file does not exist in the cloud, but exists locally:"     
   261                                                               f" {isettings._sqlite_file_local}\nTo push the file to the cloud," 
   262                                                               " call: lamin load --unload"                                       
   263                                                           )                                                                      
   264                                                       elif _raise_not_found_error:                                               
   265                                                           raise SystemExit(msg)                                                  
   266                                                       else:                                                                      
   267                                                           logger.warning(                                                        
   268                                                               f"instance exists with id {isettings._id.hex}, but database is not"
   269                                                               " loadable: re-initializing"                                       
   270                                                           )                                                                      
   271                                                           return "instance-corrupted-or-deleted"                                 
   272                                                   # this is for testing purposes only                                            
   273         1          1.0      1.0      0.0          if _TEST_FAILED_LOAD:                                                          
   274                                                       raise RuntimeError("Technical testing error.")                             
   275                                                                                                                                  
   276                                                   # below is for backfilling the instance_uid value                              
   277                                                   # we'll enable it once more people migrated to 0.71.0                          
   278                                                   # ssettings_record = isettings.storage.record                                  
   279                                                   # if ssettings_record.instance_uid is None:                                    
   280                                                   #     ssettings_record.instance_uid = isettings.uid                            
   281                                                   #     # try saving if not read-only access                                     
   282                                                   #     try:                                                                     
   283                                                   #         ssettings_record.save()                                              
   284                                                   #     # raised by django when the access is denied                             
   285                                                   #     except ProgrammingError:                                                 
   286                                                   #         pass                                                                 
   287         1       1126.0   1126.0      0.1          load_from_isettings(isettings, user=_user, write_settings=_write_settings)     
   288                                               except Exception as e:                                                             
   289                                                   if isettings is not None:                                                      
   290                                                       if _write_settings:                                                        
   291                                                           isettings._get_settings_file().unlink(missing_ok=True)  # type: ignore 
   292                                                       settings._instance_settings = None                                         
   293                                                   raise e                                                                        
   294                                               # rename lnschema_bionty to bionty for sql tables                                  
   295         1          3.0      3.0      0.0      if "bionty" in isettings.schema:                                                   
   296         1          0.0      0.0      0.0          no_lnschema_bionty_file = (                                                    
   297         1          8.0      8.0      0.0              settings_dir / f"no_lnschema_bionty-{isettings.slug.replace('/', '')}"     
   298                                                   )                                                                              
   299         1          7.0      7.0      0.0          if not no_lnschema_bionty_file.exists():                                       
   300                                                       migrate_lnschema_bionty(                                                   
   301                                                           isettings, no_lnschema_bionty_file, write_file=_write_settings         
   302                                                       )                                                                          
   303         1          0.0      0.0      0.0      return None                                                                        


  0.00 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_hub_core.py:389 - _connect_instance_new
  0.39 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:113 - _connect_instance
  0.64 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/django.py:28 - setup_django
  0.64 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_instance.py:459 - _load_db
  1.04 seconds - /Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/_connect_instance.py:182 - connect

falexwolf · 2024-09-29T21:23:59Z

Looking at these new data it's clear that not relying on Django at all for this will spare us

0.6s for setting up Django
0.9s for establishing the first connection

There is another 0.4s for getting instance metadata though connect_hub(), but this is not the big attack point right now.

falexwolf · 2024-09-29T21:29:04Z

Hence, within the Django framework, we're close to what's possible now. I'll end this current optimization and will make a another PR to refactor via LaminHub REST.

⚡️ Profile lamin load for a notebook

40f5f86

falexwolf changed the title ~~⚡️ Profile lamin load for a notebook~~ ⚡️ Profile lamin load for a notebook and ln.connect() for an instance Sep 29, 2024

falexwolf changed the title ~~⚡️ Profile lamin load for a notebook and ln.connect() for an instance~~ ⚡️ Profile lamin load for a notebook and ln.connect() Sep 29, 2024

This was referenced Sep 29, 2024

⚡ ln.connect() is rather slow and affects lamin load for a notebook or script laminlabs/lamindb#1997

Closed

⚡️ Speed up ln.connect() laminlabs/lamindb-setup#871

Merged

⚡️ Refactor so that lamin load doesn't require full-scale lamindb import

7e19219

falexwolf changed the title ~~⚡️ Profile lamin load for a notebook and ln.connect()~~ ⚡️ Profile ln.connect(), lamin load and lamin load for a notebook Sep 29, 2024

falexwolf changed the title ~~⚡️ Profile ln.connect(), lamin load and lamin load for a notebook~~ ⚡️ Profile ln.connect(), lamin connect and lamin load for a notebook Sep 29, 2024

falexwolf changed the title ~~⚡️ Profile ln.connect(), lamin connect and lamin load for a notebook~~ ⚡ Improve performance of ln.connect(), lamin connect, and lamin load for a notebook Sep 29, 2024

falexwolf mentioned this pull request Sep 29, 2024

⚡ Improve performance of ln.connect(), lamin connect, and lamin load for a notebook laminlabs/lamindb#1998

Merged

🔥 Remove profiling statement

696f389

falexwolf merged commit dd5a6dd into main Sep 29, 2024

falexwolf deleted the profileload branch September 29, 2024 21:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

⚡ Improve performance of `ln.connect()`, `lamin connect`, and `lamin load` for a notebook #84

⚡ Improve performance of `ln.connect()`, `lamin connect`, and `lamin load` for a notebook #84

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024

falexwolf commented Sep 29, 2024

⚡ Improve performance of ln.connect(), lamin connect, and lamin load for a notebook #84

⚡ Improve performance of ln.connect(), lamin connect, and lamin load for a notebook #84

Conversation

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024 • edited Loading

falexwolf commented Sep 29, 2024

falexwolf commented Sep 29, 2024

⚡ Improve performance of `ln.connect()`, `lamin connect`, and `lamin load` for a notebook #84

⚡ Improve performance of `ln.connect()`, `lamin connect`, and `lamin load` for a notebook #84

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading

falexwolf commented Sep 29, 2024 •

edited

Loading