sam fails to build when build-dir or template-file exists on non-C drive

[feldschlachtiv]

Description

Sam fails when template or build-dir exists on non-C drive, and because of that pycharm cannot run lambda function locally.

Looking at SAM source code, it seems sam tries to use relative path on L161 in command.py. I guess this is to shorten the command to avoid hitting command length limit, however, it doesn't work when either ctx.build_dir or ctx.output_template_path is on non-C drive as there's no relative path exist, assuming sam is installed in C drive.

       msg = gen_success_msg(os.path.relpath(ctx.build_dir),
                             os.path.relpath(ctx.output_template_path),
                             os.path.abspath(ctx.build_dir) == os.path.abspath(DEFAULT_BUILD_DIR))

I replaced the two os.path.relpath calls above with os.path.abspath, and sam succeeded to build my function without error (and hence pycharm was able to run the function locally)

Steps to reproduce

Run the following command in windows 10 command prompt. (content of template.yaml is the "Hello World" one)
sam build --template N:\projects\git\kuraberu\template.yaml --build-dir N:\projects\git\kuraberu.aws-sam\build HelloWorldFunction --debug

Observed result

2019-08-14 00:56:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-14 00:56:25 Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-14 00:56:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-14 00:56:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-14 00:56:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-14 00:56:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-14 00:56:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-14 00:56:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-14 00:56:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
2019-08-14 00:56:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-14 00:56:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
2019-08-14 00:56:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-08-14 00:56:25 Changing event name from before-call.apigateway to before-call.api-gateway
2019-08-14 00:56:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-08-14 00:56:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-08-14 00:56:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-08-14 00:56:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-08-14 00:56:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
2019-08-14 00:56:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-08-14 00:56:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
2019-08-14 00:56:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-08-14 00:56:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
2019-08-14 00:56:25 Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2019-08-14 00:56:25 'build' command is called
2019-08-14 00:56:25 No Parameters detected in the template
2019-08-14 00:56:25 2 resources found in the template
2019-08-14 00:56:25 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-08-14 00:56:25 Building resource 'HelloWorldFunction'
2019-08-14 00:56:25 Loading workflow module 'aws_lambda_builders.workflows'
2019-08-14 00:56:25 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-08-14 00:56:25 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-08-14 00:56:25 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-08-14 00:56:26 Running workflow 'PythonPipBuilder'
2019-08-14 00:56:26 Running PythonPipBuilder:ResolveDependencies
2019-08-14 00:56:26 calling pip download -r N:\projects\git\kuraberu\hello_world\requirements.txt --dest C:\Users\yamshu\AppData\Local\Temp\tmp2o0n2xjb
2019-08-14 00:56:27 Full dependency closure: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)}
2019-08-14 00:56:27 initial compatible: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)}
2019-08-14 00:56:27 initial incompatible: set()
2019-08-14 00:56:27 Downloading missing wheels: set()
2019-08-14 00:56:27 compatible wheels after second download pass: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)}
2019-08-14 00:56:27 Build missing wheels from sdists (C compiling True): set()
2019-08-14 00:56:27 compatible after building wheels (no C compiling): {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)}
2019-08-14 00:56:27 Build missing wheels from sdists (C compiling False): set()
2019-08-14 00:56:27 compatible after building wheels (C compiling): {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)}
2019-08-14 00:56:27 Final compatible: {requests==2.22.0(wheel), chardet==3.0.4(wheel), idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel)}
2019-08-14 00:56:27 Final incompatible: set()
2019-08-14 00:56:27 Final missing wheels: set()
2019-08-14 00:56:28 PythonPipBuilder:ResolveDependencies succeeded
2019-08-14 00:56:28 Running PythonPipBuilder:CopySource
2019-08-14 00:56:28 PythonPipBuilder:CopySource succeeded

Build Succeeded
2019-08-14 00:56:28 Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2490, 'exitReason': 'ValueError', 'exitCode': 255, 'requestId': 'b5bac6f2-d2fc-4477-a1bd-1262c7838d88', 'installationId': 'a161c6e6-1e2c-4a3a-9c82-6034f85d4100', 'sessionId': '43fe2ffd-e622-414b-9a97-62b7c7582762', 'executionEnvironment': 'CLI', 'pyversion': '3.6.7', 'samcliVersion': '0.19.0'}}]}
2019-08-14 00:56:28 Starting new HTTPS connection (1): aws-serverless-tools-telemetry.us-west-2.amazonaws.com:443
2019-08-14 00:56:28 HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
File "runpy.py", line 193, in _run_module_as_main
File "runpy.py", line 85, in run_code
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli_main.py", line 12, in
cli(prog_name="sam")
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 722, in call
return self.main(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 535, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 535, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 94, in wrapped
raise exception # pylint: disable=raising-bad-type
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 65, in wrapped
return_value = func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 105, in cli
skip_pull_image, parameter_overrides, mode) # pragma: no cover
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 161, in do_cli
msg = gen_success_msg(os.path.relpath(ctx.build_dir),
File "ntpath.py", line 584, in relpath
ValueError: path is on mount 'N:', start on mount 'C:'

Expected result

I expected sam can build functions located on non-C drive.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10 Professional
2. sam --version: SAM CLI, version 0.19.0

Add --debug flag to command you are running

[jfuss]

@feldschlachtiv Can you describe how you have everything setup? I want to make sure I understand what is going on before going further.

It looks like you have your code on one drive (N) but are running command and PyCharm from a different drive (C)? I am not a windows person but why would you do that? Is this common?

We do relative paths here to make the output of the command easier to read and understand.

[feldschlachtiv]

@jfuss - thank you for your reply! Here is what I can tell as a Windows user at home, though I mostly use Linux at work.

• I have my code in N drive - when you have multiple physical disks (SSD or HDD) in your PC, Windows assigns the drive letter(such as C, D, ..) per disk. You can confirm this by googling something like 'SSD "D drive"' or for example, https://qph.fs.quoracdn.net/main-qimg-9713ca1340c5e3f5cd92631f5d8ae0c0.webp. You can change the drive letter for disk, and 'N' is just my longtime favorite.
• I am running sam command and pycharm from C drive (because aws-sam-cli and pycharm are installed in C drive) - this is also almost mandatory in Windows. People store OS and all applications in C drive (C:\Windows, and C:\Program Files<app name> are the convention). You need to conduct a series of non-trivial tasks to move them to another drive (such as https://www.backup-utility.com/backup-restore/move-program-files-to-another-drive-windows-10.html)

[feldschlachtiv]

@jfuss - let me know if you need any further info from me. I am happy to provide. Thank you!

[jfuss]

@feldschlachtiv Thank you for the explanation, really helped me understand your setup. So it looks like we need further support in our output. If I recall correctly, we did relative paths here to make the output easier to understand, @sanathkr might remember the exact reasoning. For this case, paths that are absolute makes sense. To get the best of both worlds, I think we should try to do relative paths and if that errors out, fall back to absolute paths. Thoughts?

[feldschlachtiv]

@jfuss - I think it's a good idea to try relative paths then absolute paths. I don't have bias on neither - as long as it works.

I made the change to my pull request. Can you have a look when you get a chance?

Thank you for the feedback by the way!

[eth0izzle]

Would be good to get this merged as I'm also experiencing the same issue. Workaround for now is to copy my code to the C:\ drive.

[feldschlachtiv]

@jfuss - did you, or anyone who can review, have a chance to look at my change in pull request #1337?

[sanathkr]

@feldschlachtiv I have commented on your PR. Should be a quick fix to get this merged.

[feldschlachtiv]

@sanathkr - thank you for looking into it. I made a couple of changes - can you have a look at the PR (#1337) when you get a chance?

[sriram-mv]

Closing, as #1337 is merged and released.