Hydrate: restore populated data

[rubenstolk]

Currently when you use either new Model(data) or Model.hydrate(data) and data contains pre-populated stuff, the model instance won't have these populated fields.

Would it be possible to restore a model from a plain object while restoring populated data as well?

[vkarpov15]

Not really supported at the moment. You'd have to call populated() on each path you want to populate, instantiate instances of the child model, and set each path.

[rubenstolk]

@vkarpov15 any samples ready? Looking forward to 4.8 👍

[vkarpov15]

Actually you don't need to do populated(), as long as you just instantiate child model instances it'll work

var userSchema = new Schema({
  name: String
});

var companySchema = new Schema({
  name: String,
  users: [{ ref: 'User', type: Schema.Types.ObjectId }]
});

var User = mongoose.model('User', userSchema);
var Company = mongoose.model('Company', companySchema);

var users = [User.hydrate({ _id: new mongoose.mongo.ObjectId(), name: 'Val' })];
var company = { _id: new mongoose.mongo.ObjectId(), name: 'Booster', users: [users[0]._id] };

// How to hydrate
var c = Company.hydrate(company);
c.users = users;

console.log(c.toObject({ virtuals: true }), c.populated('users'));

$ node gh-4727.js 
{ _id: 5838c535cd064350dab4d717,
  name: 'Booster',
  users: 
   [ { _id: 5838c535cd064350dab4d716,
       name: 'Val',
       id: '5838c535cd064350dab4d716' } ],
  id: '5838c535cd064350dab4d717' } [ 5838c535cd064350dab4d716 ]
^C
$ 

It's actually pretty easy, the general process is to hydrate the child models first, then hydrate the parent model, and set the desired paths to the hydrated child models

[rubenstolk]

Thanks, that's quite clear. Would be really nice to have this done automatically.

[vkarpov15]

Yeah it would be nice, but not a high priority atm. Till then, you can just do it manually

[AndrewBarba]

I ended up writing a small function to do this, could easily be adopted to support arrays as well:

/**
 * @method hydratePopulated
 * @param {Object} json
 * @param {Array} [populated]
 * @return {Document}
 */
Model.hydratePopulated = function(json, populated=[]) {
  let object = this.hydrate(json)
  for (let path of populated) {
    let { ref } = this.schema.paths[path].options
    object[path] = mongoose.model(ref).hydrate(json[path])
  }
  return object
}

[IcanDivideBy0]

I've written another version of it, arrays are still not supported, but this one attempt to automatically detect path that have been populated based on the schema. I didn't tested it much right now, but it seems to work pretty well.

const mongoose = require("mongoose");
const { getValue, setValue } = require("mongoose/lib/utils");

/**
 * @method hydratePopulated
 * @param {Object} json
 * @return {Document}
 */
mongoose.Model.hydratePopulated = function (json) {
  let object = this.hydrate(json);

  for (const [path, type] of Object.entries(this.schema.singleNestedPaths)) {
    const { ref } = type.options;
    if (!ref) continue;

    const value = getValue(path, json);
    if (value == null || value instanceof mongoose.Types.ObjectId) continue;

    setValue(path, mongoose.model(ref).hydratePopulated(value), object);
  }

  return object;
};

If anyone find a better way to get/set values (maybe in lib/helpers/populate?) that might do the trick for arrays too

[rubenvereecken]

I've written the following in Typescript based on what @IcanDivideBy0 had. It supports arrays and probably does not support nested populated calls. Sharing mainly for inspiration

/**
 * Hydrates a lean document with populated refs, like meetup.creator or meetup.attendees
 *
 * Probably doesn't work with nested populated yet
 */
function hydrateDeeply<T extends CommonSchema<S>, S extends RefType>(
  obj: T,
  model?: ReturnModelType<any>
) {
  model = model ?? getModelFor(obj);

  let hydrated = model.hydrate(obj);

  // Gives nested paths like clubData.images.main -- so this is not actually recursive
  // Using both `paths` and `singleNestedPaths` because one is 1 level deep and the other 2+ only
  for (const [path, type] of [
    ...Object.entries(model.schema.paths),
    ...Object.entries(model.schema.singleNestedPaths),
  ]) {
    // @ts-ignore
    const options = type.options;

    // Non-array case
    if (options.ref) {
      const ref = options.ref;

      const value = getValue(path, obj);

      // Not actually populated -- ignore it
      if (!isDocLike(value)) continue;

      // Doesn't really have to be a deep call because we flatten everything with `singleNestedPaths
      const hydratedValue = hydrateDeeply(value, mongoose.model(ref));
      setValue(path, hydratedValue, hydrated);
    } else if (_.isArray(options.type) && options.type[0].ref) {
      const ref = options.type[0].ref;

      const value = getValue(path, obj);

      // Not set properly, empty, or not populated
      if (!_.isArray(value) || value.length == 0 || !isDocLike(value[0])) continue;

      // Doesn't really have to be a deep call because we flatten everything with `singleNestedPaths
      const hydratedValue = value.map(el => hydrateDeeply(el, mongoose.model(ref)), hydrated);
      setValue(path, hydratedValue, hydrated);
    }
  }

  return hydrated;
}

[Mifrill]

📓 For folks who faced this issue, check the workaround: https://github.com/Mifrill/mongoose-hydrate-populated-data

[yamaha252]

Deep hydration still doesn't work properly. My solution is:

import {Model, SchemaType, Types} from 'mongoose';

export function hydratePopulated(model: Model<any>, data: unknown) {
  if (typeof data === 'string') {
    return new Types.ObjectId(data);
  }

  if (Array.isArray(data)) {
    return data.map((v) => hydratePopulated(model, v));
  }

  if (data instanceof Types.ObjectId) {
    return data;
  }

  if (typeof data === 'object') {
    const doc = data instanceof Model ? data : model.hydrate({...data}, undefined, {
      hydratedPopulatedDocs: true,
    });

    const fields: SchemaType[] = Object.values({
      ...model.schema.paths,
      ...model.schema.virtuals,
    });

    for (const field of fields) {
      const ref = field.options.ref || (Array.isArray(field.options.type) && field.options.type[0].ref);
      const value = data[field.path];
      if (ref && value) {
        doc.set(field.path, hydratePopulated(model.db.model(value.__t || ref), value));
      }
    }

    return doc;
  }

  return data;
}

But the package should be fixed to make virtuals work: yamaha252@ab3158a

[vkarpov15]

@yamaha252 can you please open a new issue with code samples so we can understand exactly what isn't working properly?