CHANGES.txt

=======
CHANGES
=======

0.9.0 (unreleased)
------------------

- Feature: Allow object state to be stored on the ghost via the
  ``_p_mongo_state`` attribute. This allows some parent object to control
  serialization with lazy loading (most useful for sub-document objects).

- Feature: Allow to store objects using an explicit obejct id, instead of
  letting MongoDB generate one for us. The``IdNamesMongoContainer`` class has
  been updated to allow not ignore the key while setting items.

- Feature: Support serialization of objects that return a single string from
  ``__reduce__()``.

- Feature: Added a container-level cache to the ``MongoContainer``
  object. This cache is only used within the transaction, but it reduced calls
  to Mongo by 40% in a real world use. The cache can be turned off by setting
  ``mongopersist.zope.container.USE_CONTAINER_CACHE`` to false.

- Feature: Added serializer for ``datetime.time``, since the loaded reduced
  state is not usable (due to string to unicode conversion).

- Feature: Added ability for types not to paricipate in circular reference
  detection. Mongo-native types already did not participate, but now other
  non-persistent types can declarare themselves as "reference-safe" using the
  `_m_reference_safe` flag. This is particularly useful for objects serialize
  into a set of trivial values that cannot have complex references. (You
  should rarely need this feature.)

- Feature: The connection pool client connections now block writes until at
  least 2 replications have been successfully been reported (w=3). It used to
  be set to w=1, which blocks until master confirms the write.

- Bug: PersistentDict ``__eq__()`` and ``__neq__()`` methods do not rely on
  ``__cmp__()`` anymore.

- Bug: When a persistent object which is a subobject is modified, on
  abort restore the state of the parent object.

- Bug: When a sub-document object is changed and then the main object is
  removed, the removed object would re-appear, since the sub-document object
  was still registered as changed and caused a write undoing the removal. The
  issue was fixed by removing all registered sub-document objects as well from
  the list during removal.

- Bug: Objects where compared within the transaction manager for different
  reasons. This could create complicated scenarios (including data loss) when
  the ``__cmp__()`` method of the object was non-trivial. Now the result of
  the ``id(obj)`` function is used to compare objects, which is very save,
  since the behavior of this function call cannot be changed for an object.

- Bug: Ensure that only basic types and not objects derived from any basic
  type are treated as such.

- Bug: It can happen that comparing similar dicts can cause decode
  errors. This in turn caused some object updates to fail in the data manager.

- Bug: ObjectWriter ``get_collection_name()`` now works properly if the object
  writer has no jar. This is the case when we simply want to serialize any
  object to JSON, but one of the objects is a proper MongoDB persisted object.

- Bug: ``__reduce__()`` may return ``None`` as object state. By handling this
  case porperly, sets can now be serialized.

- Bug: Removed objects could return, if they were modified after
  removal. Removed objects are now explicitely ignored when registering
  modified objects.

- Bug: Make absolutely sure that ``ConflictError`` cannot be raised during
  abort when using the ``SerialConflictHandler`` conflict handlers.

- Bug: Fix circular reference detection of non-persistent object that were
  different instances but equated.

- Bug: Fixed a bug where shared collection among derived classes would not
  work consistently.

- Bug: Fixed a bug during initialization of persistent sub-objects in that
  changes would not be seen in the transaction once the datamanger was
  flushed.

- Bug: serializing a dict with all-string keys that contained invalid
  MongoDB field name characters ('.', '$' or '\0') used to cause an error.

- Bug: Do not let objects, marked with _p_mongo_sub_object be stored separately
  from main document they belong to.


0.8.4 (2013-06-13)
------------------

- Bug: Fix insert followed by remove in the same transaction. The document was
  not removed from mongo.

- Bug: Fix ``transaction.abort()`` behaviour for complex objects. ``_py_type``
  information is not lost after transaction abort.


0.8.3 (2013-04-09)
------------------

- Bug: Fixed ``MongoContainer`` vs ``IdNamesMongoContainer`` ``add`` and
  ``__setitem__`` behavour on ``None`` keys.  ``ObjectAddedEvent`` or
  ``ObjectMovedEvent`` were not fired because
  ``zope.container.contained.setitem`` got the just inserted object back.

  ``MongoContainer`` always requires ``_m_mapping_key`` and uses that attribute
  of the object to determine the new key.

  ``IdNamesMongoContainer`` requires ``_m_mapping_key`` ``None`` and uses
  ``_id`` to determine the new key.



0.8.2 (2013-04-03)
------------------

- Bug: Fixed ``check_conflict``: make sure we use the same db and collection
  as in the object


0.8.1 (2013-03-19)
------------------

- Bug: Fixed ``_p_changed`` setting on object loading which was caused by
  assigning directly to ``__name__``. That caused all objects read from
  containers to be marked changed on load. That wrecked cache performance too.


0.8.0 (2013-02-09)
------------------

- Feature: Added ``find_objects()`` and ``find_one_object()`` to the collection
  wrapper, so that whenever you get a collection from the data manager, you
  can load objects directly through the find API.

- Feature: Added the ability for MongoContained objects to fully reference and
  load their parents. This allows one to query mongo directly and create the
  object from the doc without going through the right container, which you
  might not know easily.


0.7.7 (2013-02-08)
------------------

- Bug: Do not fail if we cannot delete the parent and name attributes.


0.7.6 (2013-02-08)
------------------

- Feature: Switch to ``pymongo.MongoClient``, set default write concern values,
  allow override of write concern values.


0.7.5 (2013-02-06)
------------------

- Tests: Added, cleaned tests

- Bug: Re-release after missing files in 0.7.4

0.7.4 (2013-02-05)
------------------

- Bug: Due to ``UserDict`` implementing ``dict`` comparison semantics, any
  empty ``MongoContainer`` would equate to another empty one. This behavior
  would cause object changes to not be properly recognzed by the mongo data
  manager. The implemented solution is to implement default object comparison
  behavior for mongo containers.


0.7.3 (2013-01-29)
------------------

- Feature: Update to latest package versions, specifically pymongo 2.4.x. In
  this release, ``pymongo`` does not reexport ``objectid`` and ``dbref``.

0.7.2 (2012-04-19)
------------------

- Bug: avoid caching MongoDataManager instances in mongo container to avoid
  multiple MongoDataManagers in the single transaction in multithreaded
  environment. Cache IMongoDataManagerProvider instead.

0.7.1 (2012-04-13)
------------------

- Performance: Improved the profiler a bit by allowing to disable modification
  of records as well.

- Performance: Added caching of ``_m_jar`` lookups in Mongo Containers, since
  the computation turned out to be significantly expensive.

- Performance: Use lazy hash computation for DBRef. Also, disable support for
  arbitrary keyword arguments. This makes roughly a 2-4% difference in object
  loading time.

- Bug: An error occurred when ``_py_serial`` was missing. This was possible
  due to a bug in version 0.6. It also protects against third party software
  which is not aware of our meta-data.

- Performance: Switched to ``repoze.lru`` (from ``lru``), which is much
  faster.

- Performance: To avoid excessive hash computations, we now use the hash of
  the ``DBRef`` references as cache keys.

- Bug: ``ObjectId`` ids are not guaranteed to be unique across
  collections. Thus they are a bad key for global caches. So we use full
  ``DBRef`` references instead.

0.7.0 (2012-04-02)
------------------

- Feature: A new ``IConflictHandler`` interface now controls all aspects of
  conflict resolution. The following implementations are provided:

  * ``NoCheckConflictHandler``: This handler does nothing and when used, the
    system behaves as before when the ``detect_conflicts`` flag was set to
    ``False``.

  * ``SimpleSerialConflictHandler``: This handler uses serial numbers on each
    document to keep track of versions and then to detect conflicts. When a
    conflict is detected, a ``ConflictError`` is raised. This handler is
    identical to ``detect_conflicts`` being set to ``True``.

  * ``ResolvingSerialConflictHandler``: Another serial handler, but it has the
    ability to resolve a conflict. For this to happen, a persistent object
    must implement ``_p_resolveConflict(orig_state, cur_state, new_state)``,
    which returns the new, merged state. (Experimental)

  As a result, the ``detect_conflicts`` flag of the data manager was removed
  and replaced with the ``conflict_handler`` attribute. One can pass in the
  ``conflict_handler_factory`` to the data manager constructor. The factory
  needs to expect on argument, the data manager.

- Feature: The new ``IdNamesMongoContainer`` class uses the natural Mongo
  ObjectId as the name/key for the items in the container. No more messing
  around with coming up or generating a name. Of course, if you specified
  ``None`` as a key in the past, it already used the object id, but it was
  sotred again in the mapping key field. Now the object id is used directly
  everywhere.

- Feature: Whenever ``setattr()`` is called on a persistent object, it is
  marked as changed even if the new value equals the old one. To minimize
  writes to MongoDB, the latest database state is compared to the new state
  and the new state is only written when changes are detected. A flag called
  ``serialize.IGNORE_IDENTICAL_DOCUMENTS`` (default: ``True``) is used to
  control the feature. (Experimental)

- Feature: ``ConflictError`` has now a much more meaningful API. Instead of
  just referencing the object and different serials, it now actual has the
  original, current and new state documents.

- Feature: Conflicts are now detected while aborting a transaction. The
  implemented policy will not reset the document state, if a conflict is
  detected.

- Feature: Provide a flag to turn on MongoDB access logging. The flag is false
  by default, since access logging is very expensive.

- Feature: Added transaction ID to LoggingDecorator.

- Feature: Added a little script to test performance. It is not very
  sophisticated, but it is sufficient for a first round of optimizations.

- Feature: Massively improved performance on all levels. This was mainly
  accomplished by removing unnecessary database accesses, better caching and
  more efficient algorithms. This results in speedups between 4-25 times.

  - When resolving the path to a class, the result is now cached. More
    importantly, lookup failures are also cached mapping path ->
    ``None``. This is important, since an optimization the ``resolve()``
    method causes a lot of failing lookups.

  - When resolving the dbref to a type, we try to resolve the dbref early
    using the document, if we know that the documents within the collection
    store their type path. This avoids frequent queries of the name map
    collection when it is not needed.

  - When getting the object document to read the class path, it will now read
    the entire document and store it in the ``_latest_states`` dictionary, so
    that other code may pick it up and use it. This should avoid superflous
    reads from MongoDB.

  - Drastically improved performance for collections that store only one type
    of object and where the documents do not store the type (i.e. it is
    stored in the name map collection).

  - The Mongo Container fast load via find() did not work correctly, since
    setstate() did not change the state from ghost to active and thus the
    state was loaded again from MongoDB and set on the object. Now we use the
    new ``_latest_states`` cache to lookup a document when ``setstate()`` is
    called through the proper channels. Now this "fast load" method truly
    causes O(1) database lookups.

  - Implemented several more mapping methods for the Mongo Container, so that
    all methods getting the full list of items are fast now.

  - Whenever the Mongo Object Id is used as a hash key, use the hash of the id
    instead. The ``__cmp__()`` method of the ``ObjectId`` class is way too
    slow.

  - Cache collection name lookup from objects in the ``ObjectWriter`` class.

- Bug: We have seen several occasions in production where we suddenly lost
  some state in some documents, which prohibited the objects from being
  loadable again. The cause was that the ``_original_states`` attribute did not
  store the raw MongoDB document, but a modified one. Since those states are
  used during abort to reset the state, however, the modified document got
  stored making the affected objects inaccessible.

- Bug: When a transaction was aborted, the states of all *loaded* objects were
  reset. Now, only *modified* object states are reset. This should drastically
  lower problems (by the ratio of read over modified objects) due to lack of
  full MVCC.

- Bug: When looking for an item by key/name (``find_*()`` methods) , you would
  never get the right object back, but the first one found in the
  database. This was due to clobbering the search filter with more general
  parameters.


0.6.1 (2012-03-28)
------------------

- Feature: Added quite detailed debug logging around collection methods

0.6.0 (2012-03-12)
------------------

- Feature: Switched to optimisitc data dumping, which approaches transactions
  by dumping early and as the data comes. All changes are undone when the
  transaction fails/aborts. See ``optimistic-data-dumping.txt`` for
  details. Here are some of the new features:

  * Data manager keeps track of all original docs before their objects are
    modified, so any change can be done.

  * Added an API to data manager (``DataManager.insert(obj)``) to insert an
    object in the database.

  * Added an API to data manager (``DataManager.remove(obj)``) to remove an
    object from the database.

  * Data can be flushed to Mongo (``DataManager.flush()``) at any point of the
    transaction retaining the ability to completely undo all changes. Flushing
    features the following characteristics:

    + During a given transaction, we guarantee that the user will always receive
      the same Python object. This requires that flush does not reset the object
      cache.

    + The ``_p_serial`` is increased by one. (Automatically done in object
      writer.)

    + The object is removed from the registered objects and the ``_p_changed``
      flag is set to ``False``.

    + Before flushing, potential conflicts are detected.

  * Implemented a flushing policy: Changes are always flushed before any query
    is made. A simple wrapper for the ``pymongo`` collection
    (``CollectionWrapper``) ensures that flush is called before the correct
    method calls. Two new API methods ``DataManager.get_collection(db_name,
    coll_name)`` and ``DataManager.get_collection_from_object(obj)``
    allows one to quickly get a wrapped collection.

- Feature: Renamed ``processSpec()`` to ``process_spec()`` to adhere to
  package nameing convention.

- Feature: Created a ``ProcessSpecDecorator`` that is used in the
  ``CollectionWrapper`` class to process the specs of the ``find()``,
  ``find_one()`` and ``find_and_modify()`` collection methods.

- Feature: The ``MongoContainer`` class now removes objects from the database
  upon container removal is ``_m_remove_documents`` is ``True``. The default
  is ``True``.

- Feature: When adding an item to ``MongoContainer`` and the key is ``None``,
  then the OID is chosen as the key. Ids are perfect keys, because they are
  guaranteed to be unique within the collection.

- Feature: Since people did not like the setitem with ``None`` key
  implementation, I also added the ``MongoContainer.add(value, key=None)``
  method, which makes specifying the key optional. The default implementation
  is to use the OID, if the key is ``None``.

- Feature: Removed ``fields`` argument from the ``MongoContainer.find(...)``
  and ``MongoContainer.find_one(...)`` methods, since it was not used.

- Feature: If a container has N items, it took N+1 queries to load the list of
  items completely. This was due to one query returning all DBRefs and then
  using one query to load the state for each. Now, the first query loads all
  full states and uses an extension to ``DataManager.setstate(obj, doc=None)``
  to load the state of the object with the previously queried data.

- Feature: Changed ``MongoContainer.get_collection()`` to return a
  ``CollectionWrapper`` instance.


0.5.5 (2012-03-09)
------------------

- Feature: Moved ZODB dependency to test dependency

- Bug: When an object has a SimpleContainer as attribute, then simply loading
  this object would cause it to written at the end of the transaction. The
  culprit was a persistent dictionary containing the SimpleContainer
  state. This dictionary got modified during state load and caused it to be
  registered as a changed object and it was marked as a ``_p_mongo_sub_object``
  and had the original object as ``_p_mongo_doc_object``.


0.5.4 (2012-03-05)
------------------

- Feature: Added a hook via the IMongoSpecProcessor adapter that gets called
  before each find to process/log spec.

0.5.3 (2012/01/16)
------------------

- Bug: ``MongoContainer`` did not emit any Zope container or lifecycle
  events. This has been fixed by using the ``zope.container.contained``
  helper functions.

0.5.2 (2012-01-13)
------------------

- Feature: Added an interface for the ``MongoContainer`` class describing the
  additional attributes and methods.

0.5.1 (2011-12-22)
------------------

- Bug: The ``MongoContainer`` class did not implement the ``IContainer``
  interface.

0.5.0 (2011-11-04)
------------------

- Initial Release