adapt Publisher to latest Rucio_transfer

[belforte]

a few changes to current Puvlisher are needed.
First one is to take DBS block name from Rucio_transfer, not create it in Publisher.
So we need a place in TRANSFERSDB.

with ref. to #7087
in TRANSFERSDB table we have these two which are not useful anymore, even if PostJob is currently filling them, there is no place where we use them:

tm_rest_host VARCHAR(50) NOT NULL,
tm_rest_uri VARCHAR(255) NOT NULL,

I propose to change the latter to VARCHAR(1000) and rename to tm_publish_block

We can do the VARCHAR change immediately, so that we can write using the tm_rest_uri name for tests, but of course we want to change REST code to have a good name.

@dciangot what do you think ?

[dciangot]

Fine with me. So we are going to need also a patch for the update transfer on the REST end right?

[belforte]

Right. I will take care.

[belforte]

I did on lxplus8

belforte@lxplus8s22/~> export ORACLE_CERN=/afs/cern.ch/project/oracle; . $ORACLE_CERN/script/profile_oracle.sh; setoraenv -s prod; 
belforte@lxplus8s22/~> sqlplus cmsweb_analysis_dev@int2r

SQL*Plus: Release 11.2.0.3.0 Production on Mon May 2 11:56:14 2022

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Enter password: 

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

SP2-0310: unable to open file "?/sqlplus/admin/glogin.sql"
SQL> describe filetransfersdb
 Name					   Null?    Type
 ----------------------------------------- -------- ----------------------------
 TM_ID					   NOT NULL VARCHAR2(60)
[...]
 TM_REST_URI				   NOT NULL VARCHAR2(255)
[...]

SQL> alter table filetransfersdb modify TM_REST_URI varchar(1000) NULL;         

Table altered.

SQL>    quit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
belforte@lxplus8s22/~> 

and I verrified that TM_REST_URI is listed as VARCHAR2(1000) w/ot the NOT NULL so now empty values are acceptable. We sill stop filling this column in PostJob and let Rucio_transfers use it to store dbs-block/rucio-dataset name

[belforte]

I used same commad to remove NOT_NULL constrain also from TM_REST_HOST
and am going to apply same changes to preprod DB

[belforte]

changes appear harmless. I am ready to test writing from RUCIO_trasnfers.py into tm_rest_uri. Progress tracked in my branch https://github.com/belforte/CRABServer/tree/new-publisher-for-rucio-7223

[belforte]

pff.. we can PUT a new record in filetransfersdb with whatever tm_rest_uri but when it comes to updating POST does not accept all args. atm.

CRABServer/src/python/CRABInterface/RESTFileUserTransfers.py

Lines 61 to 80 in 093d8ff

	if method in ['POST']:
	# POST is for update, so we should allow anyone anything?
	# Of Course no, but there are multiple combinations, so we are not validating here
	# and all validation is in post function
	validate_str("subresource", param, safe, RX_SUBPOSTUSERTRANSFER, optional=False)
	validate_str("id", param, safe, RX_ANYTHING, optional=True)
	validate_str("username", param, safe, RX_USERNAME, optional=True)
	validate_str("taskname", param, safe, RX_TASKNAME, optional=True)
	validate_num("start_time", param, safe, optional=True)
	validate_str("source", param, safe, RX_ANYTHING, optional=True)
	validate_str("source_lfn", param, safe, RX_ANYTHING, optional=True)
	validate_num("filesize", param, safe, optional=True)
	validate_str("transfer_state", param, safe, RX_ANYTHING, optional=True)
	validate_num("transfer_retry_count", param, safe, optional=True)
	validate_num("publish", param, safe, optional=False)
	validate_str("publication_state", param, safe, RX_ANYTHING, optional=True)
	validate_str("job_id", param, safe, RX_JOBID, optional=True)
	validate_num("job_retry_count", param, safe, optional=True)
	validate_strlist("listOfIds", param, safe, RX_ANYTHING) # Interesting... TODO. Have optional in strlist
	elif method in ['GET']:

CRABServer/src/python/CRABInterface/RESTFileUserTransfers.py

Lines 145 to 165 in 093d8ff

	def post(self, subresource, id, username, taskname, start_time, source, source_lfn, filesize,
	transfer_state, transfer_retry_count, publish, publication_state, job_id, job_retry_count, listOfIds):
	"""This is used for user to allow kill transfers for specific task, retryPublication or retryTransfers.
	So far we do not allow retryPublications or retryTransfers for themselfs."""
	binds = {}
	binds['last_update'] = [int(time.time())]
	if subresource == 'updateDoc':
	binds['id'] = [id]
	binds['taskname'] = [taskname]
	binds['username'] = [username]
	binds['start_time'] = [start_time]
	binds['source'] = [source]
	binds['source_lfn'] = [source_lfn]
	binds['filesize'] = [filesize]
	binds['transfer_state'] = [TRANSFERDB_STATUSES[transfer_state]]
	binds['publish'] = [publish]
	binds['publication_state'] = [PUBLICATIONDB_STATUSES[publication_state]]
	binds['transfer_retry_count'] = [transfer_retry_count]
	binds['job_id'] = [job_id]
	binds['job_retry_count'] = [job_retry_count]
	return self.api.modify(self.transferDB.UpdateUserTransfersById_sql, **binds)

[belforte]

we need a new REST in order to make progress :-(

[belforte]

since I need to make a new REST version, will deal first with completing asourl/db cleanup #6930

[belforte]

the desired cleanup (#6930) is done

[belforte]

High level design of this functionality now in https://github.com/dmwm/CRABServer/wiki/ASO-via-Rucio/

[belforte]

I need a clean action plan, stuff written initially is no good. I'd rather add new columns than mess around with reusing old ones. #7480

[belforte]

replaced by #7480