Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use S3 to upload tarball needed for DryRun #6465

Closed
belforte opened this issue Mar 15, 2021 · 2 comments
Closed

use S3 to upload tarball needed for DryRun #6465

belforte opened this issue Mar 15, 2021 · 2 comments
Assignees

Comments

@belforte
Copy link
Member

self.logger.info('Uploading dry run tarball to the user file cache')
ufc = UserFileCache(mydict={'cert': kw['task']['user_proxy'], 'key': kw['task']['user_proxy'], 'endpoint': kw['task']['tm_cache_url']})
result = ufc.uploadLog('dry-run-sandbox.tar.gz')
os.remove('dry-run-sandbox.tar.gz')
if 'hashkey' not in result:
raise TaskWorkerException('Failed to upload dry-run-sandbox.tar.gz to the user file cache: ' + str(result))
else:
self.logger.info('Uploaded dry run tarball to the user file cache: ' + str(result))
update = {'workflow': kw['task']['tm_taskname'], 'subresource': 'state', 'status': 'UPLOADED'}
self.logger.debug('Updating task status: %s' % str(update))
self.server.post(self.resturi, data=urllib.urlencode(update))

@belforte
Copy link
Member Author

for this needs to introduce a new objecttype in S3 CRABCache

@belforte
Copy link
Member Author

will start with current dry run sandbox even if too large and leave to a later time a more sensible optimization along the lines in #6544
But should start with a proper objecttype so we do not change again: #6552

belforte added a commit to belforte/CRABServer that referenced this issue Apr 20, 2021
belforte added a commit to belforte/CRABServer that referenced this issue Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant