Problem when parsing some workflow and with the function addControlDenpency

[GnimEd]

I started a simulation with WRENCH on the Montage_25 workflow. I use the function getTaskParents (in the Workflow class) to display the parents of a task, We see that task ID00003 does not appear among the parents of task ID00019.
However, looking closely at the file, task ID00003 should be among the parents of ID00019.
Could you please look at the cause of this problem (here you will find the Montage_25 Workflow link)
On the other hand, we wanted to use the addControlDependency function, in a static mapping algorithm, this caused a segmentation fault error. Can you check if this function works as it should?

You will find Montage_25 here.

[rafaelfsilva]

Hi @GnimEd,

Actually, this is the expected behavior and everything looks good. Although the DAX file explicitly shows the dependency between tasks ID00003 and ID00019 as follows:

  <child ref="ID00019">
    <parent ref="ID00003"/>
    <parent ref="ID00015"/>
  </child>

In our code we remove redundant dependencies between tasks. In this case, the dependency between these tasks are not direct (or weak), thus they can be removed since the actual dependency between these tasks are resolved by:

ID00003 -> ID00013 -> ID00014 -> ID00015 -> ID00019

This means that task ID00019 cannot start running until all previous dependencies (including the 4 ones shown above) are completed.

[rafaelfsilva]

I will be investigating your second issue commented above soon.

[rafaelfsilva]

@GnimEd,

I have investigated the issue regarding the addControlDependency() function. The function works well when the parent task (first argument of the function, also called source task) has a directed dependency to the child task (second argument of the function, also called destination task). The example below (from the Montage_25) works as expected:

wrench::Workflow workflow;
workflow.loadFromDAXorJSON(workflow_file, "1000Gf");
wrench::WorkflowTask *task1 = workflow.getTaskByID("ID00001");
wrench::WorkflowTask *task2 = workflow.getTaskByID("ID00002");
workflow.addControlDependency(task1, task2);

However, if the attempt to add a dependency will break the no directed cycles condition, a segmentation fault error occurs. The example below for the Montage_25 will generate a Segmentation fault: 11 (in the workflow DAX file there is already a dependency from task ID00001 to task ID00005):

wrench::WorkflowTask *task5 = workflow.getTaskByID("ID00005");
workflow.addControlDependency(task5, task1);

Could you please confirm that this is the case for you?

If this is not the case, could you please put a code snippet of how you are trying to add the dependency that is generating this segmentation fault error?

Thanks

[rafaelfsilva]

@henricasanova, I will try to fix the issue to properly catch and handle the exception mentioned above.

[GnimEd]

Hi @rafaelfsilva

It would have been interesting to leave the links redundant, because apparently each link means that the parent job provides input files at the child job. If these extra links are removed and if in an algorithm, we want to know from which parents comes an input file, we are forced to do a search on all tasks, rather than on only parents.

For task ID00016, which is on the same level as task ID00019 and has parent ID00000 and ID00015, the getParentTasks method provides these two parents well, yet the indirect link also exists :

ID00000 ->ID00005 ->ID00014 ->ID00015
ID00000 ->ID00006 ->ID00014 ->ID00015
ID00000 ->ID00008 ->ID00014 ->ID00015

Although the DAX file explicitly shows the dependency between tasks ID00000 and ID00016 as follows:

<child ref="ID00016">
    <parent ref="ID00000"/>
    <parent ref="ID00015"/>
  </child>

Regarding the problem of addControleDependency, we wanted to use it to prevent a job t_j from being ready until another job t_i did not finish its execution during the simulation, t_i being a priority and knowing that t_j and t_i are not linked in the initial workflow.
May we know the behavior of the simulator, we will adapt our code while waiting for your decision.

[rafaelfsilva]

@GnimEd, we have discussed and we agree that it would be good to provide such functionality (showing the redundant dependency). I will be working on that fix very soon. In the meantime, you can find which task generated a file by using the getOutputOf() function from the WorkflowFile object (it will return a pointer to the WorkflowTask object that generated the file.

Regarding the issue of adding control dependency, we could not reproduce your error. Could you please share with us some code snippet so we can try to reproduce your issue?

[GnimEd]

Hi @rafaelfsilva
For the segmentation fault error produced by the addControleDependency () method, we first sort all the tasks by decreasing rank, and we add the dependency check to prevent a less priory task from starting before a higher priority. sortByRank is the comparator.

wrench::Workflow workflow;
std::sort(workflow.getTasks().begin, workflow.getTasks().end, sortByRank);

for (unsigned long i = 1; i < workflow.getTasks().size() - 1; i++) {
     WorkflowTask *ti = workflow.getTasks().at(i);
     if (startTime(ti) > earliestStartTime(ti)) {
          for (unsigned long j = 0; j < i; j++) {
               WorkflowTask *tj = workflow.getTasks().at(j);
                if (finishTime(tj) == startTime(ti) && there are scheduled on the same ComputeService) {
                    addControleDependency(tj, ti);
               }
          }
     }
}

Thanks for the resolution track with the getOutputOf() function from the WorkflowFile object.

[GnimEd]

I apologize for the previous message, I did not understand your question.
For the Montage_25 workflow, if you want to add a dependency between the tasks tj and ti, such that ID of tj is ID00010 and ID of ti is ID00011, this produces a segmentation error: addControlDependency (tj, ti).

wrench::Workflow workflow;
wrench::WorkflowTask *ti = workflow.getTaskByID("ID00011");
wrench::WorkflowTask *tj = workflow.getTaskByID("ID00010");
workflow.addControlDependency(tj, ti);

NB: There is no dependence between these two tasks (ID00010, ID00011), which would break the no directed cycles condition.

[GnimEd]

Hi @rafaelfsilva
I had not noticed that we could use getOutputOf (), I use it now in my algorithm, and it is even faster because it considerably reduces the complexity of my algorithm. Concerning addControlDependency, I can simulate the desired behavior without using this function.
I want to thank you for your help.

[rafaelfsilva]

Hi @GnimEd,

Thank you for your feedback. I am glad you algorithms is working fine now. I am still not able to reproduce the segmentation fault error. I am using this Montage_25.dax.zip DAX file. Could you please let me know if you have the same problem with this file?

In the meantime, I will be adding the support for exposing all dependencies in the DAX.

[rafaelfsilva]

Hi @GnimEd,

We have added the support for exposing all dependencies in the DAX (bfadcfe). When using any of the following functions below, you can set the parameter redundant_dependencies to true so all dependencies will be kept in the DAG.

Functions:

void Workflow::addControlDependency(WorkflowTask *src, WorkflowTask *dest, bool redundant_dependencies);
void Workflow::loadFromDAX(const std::string &filename, const std::string &reference_flop_rate, bool redundant_dependencies);
void Workflow::loadFromJSON(const std::string &filename, const std::string &reference_flop_rate, bool redundant_dependencies);
void Workflow::loadFromDAXorJSON(const std::string &filename, const std::string &reference_flop_rate, bool redundant_dependencies);

Thank you!