[BUG] - core:RecordsTransform; only working for last element in array #953
Comments
|
Hello, @brendanmcdaniel Thank you for reporting a bug. Cheers |
|
Hello |
|
Hi, I have checked the export.json and am still seeing the issue. My expectation is that ExternalId__c should always have a value when retrieved. export.json [08:17:54.580] Execution of the command sfdmu:run has been started.
LOADING AND VALIDATING THE EXPORT.JSON SCRIPT ...
OBJECT SET #1 STARTED
{
"Running Plugin version": "4.38.0"
}
{
"Running Add-On API version": "1.6.0"
}
[08:17:54.612] Loading Core Add-On Module declarations ...
[08:17:54.612] Loading Add-On module core:RecordsTransform ...
[08:17:54.648] Connecting to the myuser@example.com.sandbox using SFDX sf org display ...
[08:17:57.423] Successfully connected to the myuser@example.com.sandbox
{
"Source": "[email protected].",
"Target": "csvfile.",
"Package script": "/test/export.json."
}
GETTING AND ANALYSING OBJECT METADATA ...
[08:17:59.50] Processing the object Account ...
[08:17:59.50] Fetching metadata of the Account (Source) ...
[08:18:02.308] Processing the object BroadcastCommunication ...
[08:18:02.309] Fetching metadata of the BroadcastCommunication (Source) ...
[08:18:02.892] Fetching metadata of the Group (Source) ...
[08:18:03.956] Processing the object BroadcastCommunication ...
[08:18:03.957] Processing the object Account ...
PREPARING DATA MIGRATION JOB ...
[08:18:03.958] Building the migration strategy ...
{
"Order of objects to query": "Account; BroadcastCommunication"
}
{
"Order of objects to delete": "BroadcastCommunication; Account"
}
{
"Order of objects to update": "Account; BroadcastCommunication"
}
[08:18:03.959] Preparing of the data migration job has been completed.
ANALYSING DATA...
[08:18:04.120] {Account} The original query string of this object is returning 18 records from the Source org.
[08:18:04.271] {BroadcastCommunication} The original query string of this object is returning 4 records from the Source org.
TRIGGERING ADD-ON EVENTS ...
[08:18:04.272] No Add-On modules found to run.
[08:18:04.272] Ready to process the data.
DELETING OLD DATA FROM THE TARGET ...
[08:18:04.273] {BroadcastCommunication} No records to delete.
[08:18:04.273] {Account} No records to delete.
[08:18:04.273] Deleting data was skipped.
SOURCE:
===========
FETCHING THE DATA (STAGE 1) ...
[08:18:04.274] {Account} Fetching the Source data from Org (STAGE 1: all records) ...
[08:18:04.274] {Account} Query string: SELECT Id, Name, ExternalId__c FROM Account.
[08:18:04.275] {Account} Using REST API to retrieve the data ...
[08:18:04.457] {Account} Data retrieval (Source) has been completed. Got 18 new records.
[08:18:04.458] {BroadcastCommunication} Fetching the Source data from Org (STAGE 1: all records) ...
[08:18:04.458] {BroadcastCommunication} Query string: SELECT Id, ExternalId__c FROM BroadcastCommunication.
[08:18:04.459] {BroadcastCommunication} Using REST API to retrieve the data ...
[08:18:04.609] {BroadcastCommunication} Data retrieval (Source) has been completed. Got 4 new records.
[08:18:04.610] Data retrieval (STAGE 1) has been completed.
FETCHING THE DATA (STAGE 2) ...
[08:18:04.611] PASS 1
===========
[08:18:04.612] No records
[08:18:04.612] PASS 2
===========
[08:18:04.612] No records
[08:18:04.612] PASS 3
===========
[08:18:04.612] No records
[08:18:04.612] PASS 4
===========
[08:18:04.613] No records
TARGET:
===========
[08:18:04.613] No records
[08:18:04.613] Data retrieval (STAGE 2) has been completed.
TRIGGERING ADD-ON EVENTS ...
[08:18:04.613] No Add-On modules found to run.
DATA RETRIEVAL SUMMARY.
[08:18:04.614] {Account} The total amount of the retrieved records: 18/0.
[08:18:04.614] {BroadcastCommunication} The total amount of the retrieved records: 4/0.
[08:18:04.614] {Global} Add-On event:onDataRetrieved has been triggered.
[08:18:04.614] [core:RecordsTransform] {Global} The Add-On module has started ...
[08:18:04.614] [core:RecordsTransform] {Global} Checking the args ...
[08:18:04.614] [core:RecordsTransform] {Global} Creating the transformation map ...
[08:18:04.615] [core:RecordsTransform] {Global} Transforming records ...
[08:18:04.615] [core:RecordsTransform] {Global} Totally transformed 4 records.
[08:18:04.615] [core:RecordsTransform] {Global} The Add-On module has stopped.
[08:18:04.615] {Global} Add-On event:onDataRetrieved has been completed.
UPDATING THE TARGET (STAGE 1) ...
[08:18:04.616] {Account} Creating the file /test/Account.csv ...
[08:18:04.618] {Account} The Target has been updated. Totally processed 18 records.
[08:18:04.619] {BroadcastCommunication} Creating the file /test/BroadcastCommunication.csv ...
[08:18:04.619] {BroadcastCommunication} The Target has been updated. Totally processed 4 records.
[08:18:04.620] The Target (STAGE 1) has been updated. Totally processed 22 records.
UPDATING THE TARGET (STAGE 2) ...
[08:18:04.620] Nothing was updated.
TRIGGERING ADD-ON EVENTS ...
[08:18:04.620] No Add-On modules found to run.
DATA PROCESSING SUMMARY.
[08:18:04.620] {Account} Totally processed 18 records.
[08:18:04.620] {BroadcastCommunication} Totally processed 4 records.
[08:18:04.620] Writing to /test/MissingParentRecordsReport.csv ...
TRIGGERING ADD-ON EVENTS ...
[08:18:04.621] No Add-On modules found to run.
[08:18:04.621] Command succeeded.
[08:18:04.621] Execution of the command sfdmu:run has been completed. Exit code 0 (SUCCESS).
[08:18:04.622] Total time elapsed: 00h 00m 10s 042ms .We can see in the log that a total of 22 records have been retrieved over 2 objects (Account 18, BroadCastCommunication 4). in the resultant files only that last specified object in the export.json receives the transformation. Account 
BroadCastCommunication 
|
|
Hi, I don’t have access to your data but have tested the module in common scenarios. I don’t see a bug with SFDMU and am unable to assist further. Alternatively, you can debug the plugin on your side and let me know if you find anything. Thanks. |
|
Interesting.. the source data neither of the objects have values present for ExternalID__c the think was that the transformation is testing if null the use id+date for that value. I will debug here but as far as I can tell this is not expected behaviour? |
|
Would it be possible for you to share a sample export.json where this applies to all retrieved csv's? |
|
The key point that the transformation is made after records are retrieved so, for records to be retrieved successfully you should have external ids prepopulated. Transformation on external id field does not make sense. |
Yes Understood... the external id situation may or may not have a value. If it doesn't then a transform should occur if it does the value should remain unchanged. My target in all of this is csvfile, the functionality works but like i said only on the last element in the array. You can consider the ExrternalId__c field as an arbitrary string field |
|
For the update / upset the external id values MUST PRESENT. Population them later via transform module does not supported. |
|
This case has been marked as 'to-be-closed', since it has no activity for the 3 days. |
Describe the bug
When using the core:RecordsTransform as a module on dataRetrievedAddons I notice that only the last object in transformations actually receives the manupulation
To Reproduce
Sample export.json
{
"objectSets": [
{
"objects": [
{
"query": "SELECT Id, Name, Type, RecordTypeId FROM Account",
"operation": "Upsert",
"externalId": "Name",
"excludedFields": []
},
{
"query": "SELECT Id, ExternalId__c, createable_true FROM BroadcastCommunication",
"operation": "Upsert",
"externalId": "ExternalId__c",
"excludedFields": []
},
{
"query": "SELECT Id, ExternalId__c, createable_true FROM Carousel_Configuration__c",
"operation": "Upsert",
"externalId": "ExternalId__c"
}
]
}
],
"dataRetrievedAddons": [
{
"module": "core:RecordsTransform",
"args": {
"fields": [
{
"alias": "AccountId",
"sourceObject": "Account",
"sourceField": "Id"
},
{
"alias": "AccountExternalId",
"sourceObject": "Account",
"sourceField": "ExternalId__c"
},
{
"alias": "BroadcastCommunicationId",
"sourceObject": "BroadcastCommunication",
"sourceField": "Id"
},
{
"alias": "BroadcastCommunicationExternalId",
"sourceObject": "BroadcastCommunication",
"sourceField": "ExternalId__c"
},
{
"alias": "CarouselConfigurationId",
"sourceObject": "Carousel_Configuration__c",
"sourceField": "Id"
},
{
"alias": "CarouselConfigurationExternalId",
"sourceObject": "Carousel_Configuration__c",
"sourceField": "ExternalId__c"
}
],
"transformations": [
{
"targetObject": "Account",
"targetField": "ExternalId__c",
"formula": "formula.AccountExternalId ? formula.AccountExternalId : formula.AccountId + - + Date.now()"
},
{
"targetObject": "BroadcastCommunication",
"targetField": "ExternalId__c",
"formula": "formula.BroadcastCommunicationExternalId ? formula.BroadcastCommunicationExternalId : formula.BroadcastCommunicationId + - + Date.now()"
},
{
"targetObject": "Carousel_Configuration__c",
"targetField": "ExternalId__c",
"formula": "formula.CarouselConfigurationExternalId ? formula.CarouselConfigurationExternalId : formula.CarouselConfigurationId + - + Date.now()"
]
}
}
]
}
Expected behavior
The retrieved CSVs should all have a value on attribute ExternalId__c
Log file
I have minified the export.json and excluded some fields for sensitivity purposes but from the logs we see
with the totally transformed records only being that of the last element, in this case Carousel_Configuration__c
The text was updated successfully, but these errors were encountered: