First step in porting to python3 - code on sched #6818
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249959 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249958 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249958 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249955 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249954 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249952 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249950 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249949 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249943 +0200 parent 23707a1 author belforte <[email protected]> 1633608992 +0200 committer belforte <[email protected]> 1635249911 +0200 change python to python3 fix import of HTTPException use system curl use latest curl pick urlencode from urllib.parse use new pycurl define pythonpath for pycurl in correct script use py3 and new pycurl also for cache_status in task wrapper open file in wb for pickle and cleanup a bit remove old dashboard code remove setDashboardLogs from ServerUtilities remove calls to ApmonIf remove calls to DashboardAPI remove last calls to old Dashboard code remove calls to Apmon from sh scripts as well syntax error in previous commits import HTTPExcetion from http.client import urlencode form urllib.parse Revert "import urlencode form urllib.parse" This reverts commit 9221c46. import urlencode form urllib.parse remove unused and failing unittests remove unused and failing unittests binary files for pickle in cache_status_jel use py3 version of rucio define pycurl for dag_bootstrap.sh as well consistently open defe_ info_file in PJ as text, not binary change iteritems to items for py3 replace all occurrences of iteritems with items use getHaslLfn also in PostJonb and py3-ize it use getHaslLfn also in PostJonb and py3-ize it add comments and better parsing of fjr_parse_results.txt add comments and port to py3 code in cache_status_jel/parseErrorReport fix indentation import json add back line removed by mistake fix import of HTTPException pick urlencode from urllib.parse use new pycurl define pythonpath for pycurl in correct script use py3 and new pycurl also for cache_status in task wrapper open file in wb for pickle and cleanup a bit remove old dashboard code remove setDashboardLogs from ServerUtilities remove calls to ApmonIf remove calls to DashboardAPI remove last calls to old Dashboard code remove calls to Apmon from sh scripts as well import urlencode form urllib.parse remove unused and failing unittests remove unused and failing unittests binary files for pickle in cache_status_jel use py3 version of rucio define pycurl for dag_bootstrap.sh as well consistently open defe_ info_file in PJ as text, not binary change iteritems to items for py3 replace all occurrences of iteritems with items use getHaslLfn also in PostJonb and py3-ize it add comments and port to py3 code in cache_status_jel/parseErrorReport import json add back line removed by mistake tentative fix for upload_input_metadata in PostJob add comments this looks proper fix for input_file_metadata apply same fix to output_file_metadata
|
just in case something went wrong with rebasing, let's put this on crab-dev-tw01 and check again that tasks run on vocms059 (the only schedd with the new pycurl, atm) |
|
Jenkins results:
|
mapellidario
added a commit
to mapellidario/CRABServer
that referenced
this pull request
Feb 28, 2022
* parent 23707a1 (dmwm#6818) Initial changes for python3. Make it possible to run with python3 on sched. * use gocurl from CVMFS Fix dmwm#6822 (dmwm#6824) * Belforte patch 1 (dmwm#6825) * use gocurl from CVMFS Fix dmwm#6822 (dmwm#6823) * add comment about py2/3 compatibility needs * use status_cache in pickle format/. Fix dmwm#6820 (dmwm#6829) * Remove most old "Panda" code (dmwm#6835) * remove PandaServerInterface. for dmwm#6542 * remove unused taskbuffer. For dmwm#6542 * remove useless comment about Panda. For dmwm#6542 * remove PanDAExceptions. For dmwm#6542 * disallow panda scheduler in regexp. for dmwm#6542 * Remove old crab cache code (dmwm#6833) * remove code in UserFileCache. for dmwm#6776 * remove reference to UserFileCache in setup.py. For dmwm#6776 * remove all code references to UserFileCache. For dmwm#6776 * remove all calls to panda stuff in the code (dmwm#6836) * remove pada fields. For dmwm#6542 * remove references to pandajobid DB column in code. For dmwm#6542 * remove panda-related JobGroup. For dmwm#6542 * remove useless calls to JobGroup. For dmwm#6542 * remove all references in code to panda, jobset and jobgroups. For dmwm#6542 * Move away mysql fix 6837 (dmwm#6838) * add a place for obsolete code * move MYSQL code to obsolete dir. Fix dmwm#6837 * remove Databases/TaskDB/Oracle/JobGroup from build. Fix dmwm#6839 (dmwm#6840) * use urllib3 in place of urllib2 (dmwm#6841) * remove couchDb related code. Easy part for dmwm#6834 (dmwm#6842) * Proper fix for autom split (dmwm#6843) * py3 fix for hashblib * proper py3 porting of urllib2.urlopen * remove old code. For dmwm#6845 (dmwm#6847) * Remove couch db code (dmwm#6848) * remove couchDb related code. Easy part for dmwm#6834 * remove CouchDB code from DagmanResubmitter. For dmwm#6845 * remove CouchDB code from PostJob. For dmwm#6845 * remove isCouchDBURL, now unused. For dmwm#6845 * one more cleanup in PostJob. For dmwm#6845 * one more cleanup in PostJob. For dmwm#6845 * restore code deleted by mistake * [py3] src/python/Databases suports py2 and py3 (dmwm#6828) * scr/pytohn/CRABInterface supports py3 (dmwm#6831) * [py3] src/python/CRABInterface - changes suggested by futurize * removed uses of deprecated panda code * validate_str instead of validate_ustr, deprecated in WMCore * a hack to make it run for minimal purposes (dmwm#6850) * complete removal of unused taskbuffer * stop trying to remove failed migrations from 2019. Fix dmwm#6854 (dmwm#6856) * Port to python3 recent small fixes from master (dmwm#6858) * use gocurl from CVMFS Fix dmwm#6822 (dmwm#6823) * add comment about py2/3 compatibility needs (dmwm#6826) * add GH remote for Diego * upload new config version (dmwm#6852) * stop trying to remove failed migrations from 2019. Fix dmwm#6854 (dmwm#6855) Co-authored-by: Daina <[email protected]> * better logging of acquired publication files. Fix dmwm#6860 (dmwm#6861) * remove unused/undef variable. fix dmwm#6864 (dmwm#6865) * Second batch of fixes for crabserver REST in py3. (dmwm#6873) * HTCondorWorkflow: decode to str before parsing * HTCondorWorkflow: convert to str output of literal eval * slight improve to stefano's `horrible hack` * updated version of wmcore to 1.5.5 in requirements.txt * Add more logging (dmwm#6877) * add logging of tmp file removal * avoid duplicating ids. Fix dmwm#6800 * get task (DAG) status form sched. Fix dmwm#6869 (dmwm#6874) * get task (DAG) status form sched. Fix dmwm#6869 * improve comments * rename cache_status_jel to cache_status and use it. Fix dmwm#6411 (dmwm#6878) * validate both temp and final output LFNs. Fix dmwm#6871 (dmwm#6879) * change back to use py3 for cache_status last commit had changed by mistake to use python2 for cache_status * make migration dbg Utils worn in container. Fix dmwm#6853 (dmwm#6886) * Py3 for publisher (dmwm#6887) * ensure tasks is a list * basestring -> string * no need to cast to unicode * use python3 to start TaskPublish * REST and TW - correctly encode/decode input/outputs of b64encode/b64decode * stop inserting nose in TW tarball. Fix dmwm#6455 (dmwm#6888) * stop inserting nose in TW tarball. Fix dmwm#6455 * make sure CRAB3.zip exists, improve comments * improve log * port to python3 branch of dmwm@87ada3b * port to python3 branch of dmwm@9a72d9e * Make new publisher default (dmwm#6892) * make NEW_PUBLISHER the default, fix dmwm#6412 * remove code swithing NEW_PUBLISHER. Fix dmwm#6410 * add comments * start Publisher in py3 env (dmwm#6894) * stupid typo * py3 crabserver compatible with tasks submitted by py2 crabserver (dmwm#6907) - tm_split_args: convert to unicode the values in the lists: 'lumis' and 'runs'wq! * crabserver py3 - change tag for build with jenkins (dmwm#6908) * Make tw work in py3 for dmwm#6899 (dmwm#6901) * Queue is now lowercase, xrange -> range * use python3 to start TW * start TW from python3.8 dir * workaround ldap currently missing in py3 build * basestring --> str * use binary files for pickle * make sure to hande classAd defined as bytes as well * remove MonALISA code. Fix dmwm#6911 (dmwm#6913) * TW - new tag of WMCore with fix to credential/proxy (dmwm#6915) * TW - remove Logger and ProcInfo from setup.py and from bin/htcondor_make_runtime.sh (dmwm#6916) * TW - remove Logger and ProcInfo from setup.py * TW - remove Logger and ProcInfo from bin/htcondor_make_runtime.sh * TW - remove apmon from setup.py * TW - update tag of WMCore to mapellidario/py3.211214patch1 * setup.py - remove RESTInteractions from CRABClient build (dmwm#6919) * generate Error on bad extconfig format, remove old code, cleanup. Fix dmwm#6897 See also dmwm#6897 (comment) (dmwm#6910) * better py3 comp. for authenticatedSubprocess. fix dmwm#6899 (comment) (dmwm#6927) * remove references to asourl/asodb in TW (dmwm#6929) * [py3] apply py3-modernization changes to whole dmwm/CRABServer (dmwm#6921) * [py3] migrated TW/Actions/ to py3 * [py3] fix open() mode: str for json, bytes for pickle * [py3] fix use of hashlib.sha1(): input must be bytes * TaskWorker/Actions/StageoutCheck: use execute_command, not executeCommand * Publish utils for py3 (dmwm#6941) * use python3 to run DebugFailedBlockPublication * use python3 to run FindFailedBlockPublication * make py3 compat and improve printout. Fix dmwm#6939 * optionally create new publication * Fix task publish 6940 (dmwm#6942) * avoid using undefined variable. Fix dmwm#6940 * make sure all calls to DBS are in try/excect for dmwm#6940 * use Rucio client py2 for FTS_transfer.py. Fix dmwm#6948 (dmwm#6949) * use Rucio client py2 for FTS_transfer.py. Fix dmwm#6948 * add comment about python version * pass $XrdSecGSISRVNAMES to cmsRun. Fix dmwm#6953 (dmwm#6955) (dmwm#6956) * Pre dag divide by zero fix 6926 (dmwm#6959) * protect against probe jobs returning no events. Fix dmwm#6926 * some pylint cleanups * Cleanup userproxy from rest fix 6931 (dmwm#6960) * remove unused retrieveUserCert for dmwm#6931 * cleanup unused userproxy from REST fix dmwm#6931 * remove unused imports * cleanup serverdn/serverproxy/serverkey from REST code. Fix dmwm#6961 * correct kill arguments. Fix dmwm#6928 (dmwm#6964) * requirements.txt: update wmcore tag (dmwm#6966) * REST-py3 backward compatibile with publisher-py2 (dmwm#6967) * Fix mkruntime 6970 (dmwm#6971) * non need for cherrypy in TW tarball. Fix dmwm#6970 * place dummyFile in local dir and clenaup * remove useless encode. fix dmwm#6972 (dmwm#6973) * use $STARTDIR for dummyFile. (dmwm#6974) * enable TaskWorker to use IDTOKENS. Fix dmwm#6903 (dmwm#6975) * update requirements.txt to dmwm/WMCore 1.5.7 (dmwm#6982) * use different WEB_DIR for token auth. Fix dmwm#6905 (dmwm#6983) * correct check for classAd existance. Fix dmwm#6986 (dmwm#6987) * define CRAB_UserHN ad for task_wrapper. Fix dmwm#6981 (dmwm#6988) * no spaces around = in bash. properly fix dmwm#6981 * fix not py3-compatible pycurl.error handling in RESTInteractions (dmwm#6996) * make Pre/Post/RetryJob use existing WEB_DIR. Fix dmwm#6994 (dmwm#6998) * remove extra / in API name. Fix dmwm#7004 (dmwm#7005) * Remove extra slash fix 7004 (dmwm#7006) * remove extra / in API name. Fix dmwm#7004 * remove extra / in API name. Fix dmwm#7004 * restore NoAvailableStie exception for TW. Fix dmwm#7038 (dmwm#7039) * make sure classAds for matching are ORDERED lists, fix dmwm#7043 (dmwm#7044) * make sure eventsThr and eventsSize are not used if not initialized. Fix dmwm#7065 (dmwm#7066) * Adjust code to work with new DBS Go based server (dmwm#6969) (dmwm#7074) Co-authored-by: Valentin Kuznetsov <[email protected]> * user python3 for FTS_transfers. Fix dmwm#6909 (dmwm#7052) * adapt to new DBS serverinfo API (dmwm#7093) * use WMCore 2.0.1.pre3 - Fix dmwm#7096 (dmwm#7097) * point user feedback to CmsTalk. Fix dmwm#7100 (dmwm#7101) Co-authored-by: Stefano Belforte <[email protected]> Co-authored-by: Daina <[email protected]> Co-authored-by: Valentin Kuznetsov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Massive (sorry) set of changes for python3 compatibility.
Besides some changes like fix import of HTTPException or use of iteritems in dictionaries which are applied everywhere, the main thrust of this PR is a version of the "things which run on sched" which works with python3. Since this work evolved over some time while other changes were pushed in master, this PR has a horrible history also after squashing. Let's take it as is
Initial tests are OK, some tasks work with this.
Known shortcomings:
Incomplete list of changes:
change python to python3 when calling scripts from bash
fix import of HTTPException
use our own pycurl on sched
pick urlencode from urllib.parse
define pythonpath for pycurl in correct script
open file in wb for pickle and cleanup a bit
remove unused and failing unittests
binary files for pickle in cache_status_jel
use py3 version of rucio
define pycurl for dag_bootstrap.sh as well
consistently open defe_ info_file in PJ as text, not binary
replace all occurrences of iteritems with items
use getHaslLfn also in PostJonb and py3-ize it
add comments and better parsing of fjr_parse_results.txt
add comments and port to py3 code in cache_status_jel/parseErrorReport