Provide new API for Adapters that make use of Kokkos::Views (both Host and Device spaces)

[JacobDomagala]

Provide new API for Zoltan2's Adapters. New API should consist of new get functions (that currently use raw pointers or ArrayViews) that return Kokkos::View in Host or Device space.
Example:

// Old (deprecated) API
virtual void getIDsView(const gno_t *&ids) const;
virtual void getIDsKokkosView(ConstIdsDeviceView &ids) const;

// New API
virtual void getIDsHostView(ConstIdsHostView &hostIds) const;
virtual void getIDsDeviceView(ConstIdsDeviceView &deviceIds) const; 

[JacobDomagala]

Just the idea:

• As we add new API, we sometimes double the memory footprint by filling both legacy API data members and newly added Kokkos ones. Should we add CMake flag (something like Zoltan2_USE_KOKKOS_REFACTOR) that would help us to #ifdef parts of code?

Example:

templte <typename User>
SomeAdapter(const RCP<const User> &input)
 {
#ifndef Zoltan2_USE_KOKKOS_REFACTOR
  legacyMember_ = copyFromInput(input);
  // initialize all (old) members
#else
  newMember_ = copyFromInput(input);
  // initialize all (new) members
#endif

}

or

templte <typename User>
SomeAdapter(const RCP<const User> &input)
 {
  legacyMember_ = copyFromInput(input);
  // initialize all (old) members
#ifdef Zoltan2_USE_KOKKOS_REFACTOR
  newMember_ = copyFromInput(input);
  // initialize all (new) members
#endif

}

For the time being (untill we fully adapt Kokkos into Zoltan2) I would suggest having this flag to false by default.

[JacobDomagala]

Also, should we care how we return data from Adapters?
Originally all data returned by Adapter was const and Models would (for the most part) either forward it to Algorithm or create new variable and copy the Adapter's data (with/without modifications).
The new (Kokkos-friendly) API doesn't follow that pattern fully (we return View to const data only for IDs, for weights and parts we return View to non-const, which allows Model to modify Adapter's internal data).

Adapter (old) API

virtual void getIDsView(const gno_t *&ids) const;
virtual void getWeightsView(const scalar_t *&wgt, int &stride, int idx = 0) const;
virtual void getPartsView(const part_t *&inputPart) const;

Adapter (new) API

virtual void getIDsKokkosView(Kokkos::View<const gno_t *, device_t> &ids) const;     // const
virtual void getWeightsKokkosView(Kokkos::View<scalar_t **, device_t> & wgt) const;  // non-const!
virtual void getPartsDeviceView(Kokkos::View<part_t*, device_t> &inputPart) const;   // non-const!

[JacobDomagala]

As for the Weights getter, I think we could provide both versions:

1. Let user query by weight index and return 1D view

void getWeightsHostView(WeightsHostView1D& weights, int idx = 0) const;

// example use case (GraphModel):
{ 
    // (...) initialize other members
   for (int idx=0; idx < nWeightsPerVertex_; idx++){
      const auto useNumNZ = adapter->useDegreeAsWeight(idx);
      if (useNumNZ){
        WeightsHostView1D weights;
        for (size_t i=0; i < nLocalVertices_; i++) {
          weights[i] = eOffsets_[i+1] - eOffsets_[i];
        }
        weightInfo[idx] = weights;
      }
      else{
        WeightsHostView1D weights;
        adapter->getWeightsHostView(weights, idx);
        weightInfo[idx] = weights;
      }
    }
}

2. Return 2D view of all weights

void getWeightsHostView(WeightsHostView& weights) const

// example use case (CoordinateModel):
{ 
    // (...) initialize other members
    WeightsHostView wgts;
    if(userNumWeights_ > 0) {
      adapter->getWeightsHostView(wgts);
    }
}