[bug] ilock sometimes fails due to race condition

[parasj]

This is a known issue symonsoft/ilock#4

$ skylark cp s3://exps-paras-skylark-us-east-1/fake_imagenet/ s3://exps-paras-skylark-us-east-2

=================================================
  ______  _             _                 _
 / _____)| |           | |               | |
( (____  | |  _  _   _ | |  _____   ____ | |  _
 \____ \ | |_/ )| | | || | (____ | / ___)| |_/ )
 _____) )|  _ ( | |_| || | / ___ || |    |  _ (
(______/ |_| \_) \__  | \_)\_____||_|    |_| \_)
                (____/
=================================================

Storing debug information for transfer in /tmp/skylark/transfer_logs/20220513_215915/client.log
✔ Initializing cloud keys (5/5) in 2.08s
✔ Provisioning gateway instances (2/2) in 28.77s
⠴ Applying firewall rules (3/4)21:59:47 [ERROR] Error running <lambda> with args functools.partial(<bound method AWSCloudProvider.add_ip_to_security_group of <skylark.compute.aws.aws_cloud_provider.AWSCloudProvider object at 0x7fa7040554f0>>, 'us-east-1', '54.198.49.51'): [Errno 2] No such file or directory: '/tmp/ilock-9d018be7ad26629d7951b129d3de8e9ebb1aca77bd9f42919bdabffc5e111c38.lock'
Traceback (most recent call last):
  File "/home/ubuntu/miniconda/bin/skylark", line 33, in <module>
    sys.exit(load_entry_point('skylark', 'console_scripts', 'skylark')())
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/ubuntu/skylark/skylark/cli/cli.py", line 199, in cp
    replicate_helper(
  File "/home/ubuntu/skylark/skylark/cli/cli_helper.py", line 303, in replicate_helper
    rc.provision_gateways(reuse_gateways, use_bbr=use_bbr)
  File "/home/ubuntu/skylark/skylark/replicate/replicator_client.py", line 207, in provision_gateways
    do_parallel(lambda fn: fn(), aws_jobs, spinner=True, desc="Applying firewall rules")
  File "/home/ubuntu/skylark/skylark/utils/utils.py", line 94, in do_parallel
    args, result = future.result()
  File "/home/ubuntu/miniconda/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/home/ubuntu/miniconda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/ubuntu/miniconda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ubuntu/skylark/skylark/utils/utils.py", line 82, in wrapped_fn
    raise e
  File "/home/ubuntu/skylark/skylark/utils/utils.py", line 79, in wrapped_fn
    return args, func(args)
  File "/home/ubuntu/skylark/skylark/replicate/replicator_client.py", line 207, in <lambda>
    do_parallel(lambda fn: fn(), aws_jobs, spinner=True, desc="Applying firewall rules")
  File "/home/ubuntu/skylark/skylark/compute/aws/aws_cloud_provider.py", line 281, in add_ip_to_security_group
    logger.warn("[AWS] Error adding IP to security group, since it already exits")
  File "/home/ubuntu/miniconda/lib/python3.8/site-packages/ilock/__init__.py", line 59, in __exit__
    os.unlink(self._filepath)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ilock-9d018be7ad26629d7951b129d3de8e9ebb1aca77bd9f42919bdabffc5e111c38.lock'