[cxxmodules] Add initial implementation of the semantic GMI. #5094

vgvassilev · 2020-03-06T14:07:33Z

No description provided.

phsft-bot · 2020-03-06T14:07:57Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-fedora30/cxx14, ROOT-ubuntu16/nortcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-03-06T14:25:14Z

Build failed on ROOT-performance-centos7-multicore/default.
See console output.

Failing tests:

phsft-bot · 2020-03-06T14:26:27Z

Build failed on ROOT-fedora30/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-06T14:28:17Z

Build failed on ROOT-fedora27/noimt.
See console output.

Failing tests:

phsft-bot · 2020-03-06T14:28:37Z

Build failed on ROOT-fedora29/python3.
See console output.

Failing tests:

phsft-bot · 2020-03-06T14:43:04Z

Build failed on ROOT-ubuntu18.04-i386/cxx14.
See console output.

Failing tests:

vgvassilev · 2020-03-26T19:55:00Z

@phsft-bot build!

phsft-bot · 2020-03-26T19:55:09Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora29/python3, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-03-26T20:13:58Z

Build failed on ROOT-fedora31/noimt.
See console output.

Failing tests:

phsft-bot · 2020-03-26T20:19:27Z

Build failed on ROOT-fedora29/python3.
See console output.

Failing tests:

phsft-bot · 2020-03-26T20:20:15Z

Build failed on ROOT-performance-centos7-multicore/default.
See console output.

Failing tests:

phsft-bot · 2020-03-26T20:34:43Z

Build failed on ROOT-fedora30/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-26T20:45:43Z

Build failed on ROOT-debian10-i386/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-28T15:39:44Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora29/python3, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-03-28T15:54:32Z

Build failed on ROOT-fedora31/noimt.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:04:46Z

Build failed on ROOT-fedora30/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:08:28Z

Build failed on ROOT-performance-centos7-multicore/default.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:09:21Z

Build failed on ROOT-fedora29/python3.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:10:23Z

Build failed on ROOT-ubuntu16/nortcxxmod.
See console output.

Failing tests:

And 45 more

phsft-bot · 2020-03-28T16:28:04Z

Build failed on ROOT-debian10-i386/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:28:54Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora29/python3, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-03-28T16:40:16Z

Build failed on ROOT-fedora31/noimt.
See console output.

Failing tests:

phsft-bot · 2020-03-28T16:48:21Z

Build failed on ROOT-fedora29/python3.
See console output.

Failing tests:

phsft-bot · 2020-03-28T17:03:07Z

Build failed on ROOT-performance-centos7-multicore/default.
See console output.

Failing tests:

phsft-bot · 2020-03-28T17:14:35Z

Build failed on ROOT-fedora30/cxx14.
See console output.

Failing tests:

phsft-bot · 2020-03-28T17:33:52Z

Build failed on ROOT-debian10-i386/cxx14.
See console output.

Failing tests:

vgvassilev · 2020-04-01T07:27:44Z

We seem to fail to generate the dtor of ParamFunctorHandler via instantiating the ctor ~ParamFunctorHandler -> ~ParamFunctionBase-> ~ParamFunctorHandler. For some reason the template instantiator thinks that it does not need to instantiate the dtor.

Minimal reproducer:

cat stressMC_reduced.cxx 
#include <TTree.h>
#include <TF1.h>
using namespace ROOT::Math;

class StatFunction : public ROOT::Math::IParamFunction {
public:

   unsigned int NPar() const override { return 0; }
   const double * Parameters() const override { return nullptr; }
   ROOT::Math::IGenFunction * Clone() const override { return nullptr; }
   void SetParameters(const double * p) override { }

   // test cumulative function
  void Test() {
   TF1 * f = new TF1("ftemp",ParamFunctor(*this),0,0,0);
  }


private:


   double DoEvalPar(double x, const double * ) const override {
     return 0;
   }

};


typedef StatFunction Dist_beta;
int testStatFunctions() {
   Dist_beta dist;
   dist.Test();
   return 0;
}


class VectorTest {
public:
   template<class V>
   double testWrite(V & dataV, std::string typeName="") {
      TTree tree("VectorTree","");

      // need to add namespace to full type name
      if (typeName == "") {
      }

      tree.Fill();

      return 0;
   }
};

int stressMC_reduced(double nscale = 1) {
   return 0;
}

phsft-bot · 2020-06-16T12:02:30Z

Build failed on ROOT-fedora31/noimt.
Running on root-fedora-31-1.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-16T12:09:55Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-16T12:10:50Z

Build failed on ROOT-fedora30/cxx14.
Running on root-fedora30-1.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-16T13:31:12Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-16T14:43:57Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

vgvassilev · 2020-06-16T14:44:53Z

@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On

phsft-bot · 2020-06-16T14:45:01Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14 with flags -DCTEST_TEST_EXCLUDE_NONE=On
How to customize builds

phsft-bot · 2020-06-16T15:25:26Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

vgvassilev · 2020-06-16T15:50:00Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

* [projectroot.test.test_stressIOPlugins_xroot](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot/test/test_stressIOPlugins_xroot/)

* [projectroot.tree.treeplayer.test.gtest_tree_treeplayer_test_treeprocessormt_remotefiles](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot.tree.treeplayer/test/gtest_tree_treeplayer_test_treeprocessormt_remotefiles/)

* [projectroot.runtutorials.tutorial_dataframe_df102_NanoAODDimuonAnalysis](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot/runtutorials/tutorial_dataframe_df102_NanoAODDimuonAnalysis/)

* [projectroot.runtutorials.tutorial_dataframe_df101_h1Analysis](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot/runtutorials/tutorial_dataframe_df101_h1Analysis/)

* [projectroot.runtutorials.tutorial_dataframe_df103_NanoAODHiggsAnalysis](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot/runtutorials/tutorial_dataframe_df103_NanoAODHiggsAnalysis/)

* [projectroot.runtutorials.tutorial_tmva_tmva103_Application](https://lcgapp-services.cern.ch/root-jenkins/job/root-pullrequests-build/87069/testReport/projectroot/runtutorials/tutorial_tmva_tmva103_Application/)

Those fail all over the place.

core/metacling/src/CMakeLists.txt

pcanal · 2020-06-16T16:04:35Z

core/metacling/src/TCling.cxx

+                  } else if (NamespaceDecl *NSD = llvm::dyn_cast<NamespaceDecl>(R))
+                     Register(NSD, /*AddSingleEntry=*/ false);
+                  else if (TypedefNameDecl *TND = dyn_cast<TypedefNameDecl>(R))
+                     Register(TND);


Can't typedef decl be repeated (in several header files and thus several modules)?

Yes, but it should define the same entity, otherwise modules ODR hash will complain. So whichever we pick should be fine. I suspect that may lead to loading the 'wrong' module to resolve the intended typedef but cannot do anything better.

Alternatively, we could load all modules which have this typedef name -- that would be redundant because if the typedefs are all the same what's the point; if the typedefs are different we will get an ODR violation diagnostic.

Do we need to load the pcm for the target of the typedef?

Yes, we need to load something that has a clang::Decl for the missing identifier

core/metacling/src/TCling.cxx

pcanal · 2020-06-16T16:24:12Z

interpreter/llvm/src/tools/clang/lib/Serialization/GlobalModuleIndex.cpp

+        // ... unless it's a .pcm.lock file, which indicates that someone is
+        // in the process of rebuilding a module. They'll rebuild the index
+        // at the end of that translation unit, so we don't have to.
+        if (llvm::sys::path::extension(D->path()) == ".pcm.lock")


What happens in the follow scenario:

(1) Process A starts and build a module and thus creates a .pcm.lock (2) Process A is put to sleep (and/or is very very slow) (3) some other pcm is created fully (and have names such it is early in the directory iteration). (4) Process B starts and try to write the index and gives ups (5) Process B finishes (6) Process A wakes ups and finish write the module.idx.

Are the extra pcm creates in step (3) included in the module.idx (create by Process A)

We will not add it in the index probably.

Note that this is pre-existing llvm code and the ROOT GMI does not actually use it. It provides a table with ExternalIDs

We will not add it in the index probably.

I see. Then the comment is not appropriate for our use case [And the result is (silently) sub-optimal, isn't it?]

I agree. Let’s see if this will improve when we upgrade llvm.

phsft-bot · 2020-06-16T16:40:58Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos7-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac1015/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-06-16T17:01:43Z

Build failed on ROOT-performance-centos7-multicore/default.
Running on olhswep22.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See cdash.
See console output.

Failing tests:

phsft-bot · 2020-06-17T05:08:37Z

Build failed on ROOT-fedora31/noimt.
Running on root-fedora-31-1.cern.ch:/build/workspace/root-pullrequests-build
See cdash.
See console output.

Errors:

[2020-06-17T05:08:36.625Z] CMake Error at /build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:950 (message):

vgvassilev · 2020-06-23T18:24:21Z

@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On

vgvassilev requested a review from pcanal as a code owner March 6, 2020 14:07

eguiraud assigned vgvassilev Mar 10, 2020

vgvassilev force-pushed the cxxmodules_enable_gmi_by_default branch from fd9cca9 to a7869de Compare March 28, 2020 16:28

vgvassilev force-pushed the cxxmodules_enable_gmi_by_default branch from ce22a8f to 41189a1 Compare June 16, 2020 13:31

vgvassilev marked this pull request as ready for review June 16, 2020 13:31

vgvassilev requested a review from etejedor as a code owner June 16, 2020 13:31

vgvassilev force-pushed the cxxmodules_enable_gmi_by_default branch from 41189a1 to 1ac186c Compare June 16, 2020 14:43

vgvassilev changed the title ~~[cxxmodules] Enable global module indexing by default.~~ [cxxmodules] Add initial implementation of the semantic GMI. Jun 16, 2020

pcanal reviewed Jun 16, 2020

View reviewed changes

core/metacling/src/CMakeLists.txt Show resolved Hide resolved

pcanal reviewed Jun 16, 2020

View reviewed changes

core/metacling/src/TCling.cxx Show resolved Hide resolved

pcanal reviewed Jun 16, 2020

View reviewed changes

vgvassilev added 4 commits June 16, 2020 16:40

Add initial implementation of the semantic GMI.

cfbaeae

Add Rint to the core modules.

e4e82e6

[gmi] TCling::Declare is not allowed to autoload missing libraries.

11bbc24

LoadPCMImpl may need to deserialize.

6c26e5f

vgvassilev force-pushed the cxxmodules_enable_gmi_by_default branch from 1ac186c to 6c26e5f Compare June 16, 2020 16:40

pcanal approved these changes Jun 16, 2020

View reviewed changes

vgvassilev merged commit f6e69e8 into root-project:master Jun 17, 2020

vgvassilev deleted the cxxmodules_enable_gmi_by_default branch June 17, 2020 04:36

[cxxmodules] Add initial implementation of the semantic GMI. #5094

[cxxmodules] Add initial implementation of the semantic GMI. #5094

Conversation

vgvassilev commented Mar 6, 2020

phsft-bot commented Mar 6, 2020

phsft-bot commented Mar 6, 2020

Failing tests:

phsft-bot commented Mar 6, 2020

Failing tests:

phsft-bot commented Mar 6, 2020

Failing tests:

phsft-bot commented Mar 6, 2020

Failing tests:

phsft-bot commented Mar 6, 2020

Failing tests:

vgvassilev commented Mar 26, 2020

phsft-bot commented Mar 26, 2020

phsft-bot commented Mar 26, 2020

Failing tests:

phsft-bot commented Mar 26, 2020

Failing tests:

phsft-bot commented Mar 26, 2020

Failing tests:

phsft-bot commented Mar 26, 2020

Failing tests:

phsft-bot commented Mar 26, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

phsft-bot commented Mar 28, 2020

Failing tests:

vgvassilev commented Apr 1, 2020

phsft-bot commented Jun 16, 2020

Failing tests:

phsft-bot commented Jun 16, 2020

Failing tests:

phsft-bot commented Jun 16, 2020

Failing tests:

phsft-bot commented Jun 16, 2020

phsft-bot commented Jun 16, 2020

vgvassilev commented Jun 16, 2020

phsft-bot commented Jun 16, 2020

phsft-bot commented Jun 16, 2020

Failing tests:

vgvassilev commented Jun 16, 2020

Failing tests:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phsft-bot commented Jun 16, 2020

phsft-bot commented Jun 16, 2020

Failing tests:

phsft-bot commented Jun 17, 2020

Errors:

vgvassilev commented Jun 23, 2020