Shift ingestion feature metrics to after store write

[zhilingc]

Is your feature request related to a problem? Please describe.
Existing metrics collection for ingestion is done in parallel to store writes:

               +
               |
       +-------+--------+
       |                |
+------+-----+   +------+-----+
|            |   |            |
|   store    |   |  metrics   |
|   write    |   |   write    |
|            |   |            |
+------------+   +------------+

While this does give a rough gauge as to what data looks like in flight, it doesn't actually reflect the status of the data in the store, which is ultimately what we are concerned about, since write failures are common.

Describe the solution you'd like
Ideally we want to write metrics POST store writes, as so:

              ...
               +
               |
         +-----+-----+
         |           |
         |   store   |
         |   write   |
         |           |
         +-----+-----+
               |
      +--------+--------+
      +                 +
   success           failure
      +                 +
      |                 |
+-----+-----+     +-----+-----+
|  write    |     |  write    |
|  sucess   |     |  failure  |
|  metrics  |     |  metrics  |
+-----------+     +-----------+

Metrics collected post store writes would more accurately reflect the data in the stores.