Stateful processing #244

superbock · 2017-01-20T12:31:18Z

This PR adds proper stateful processing.

The process() method of stateful processors accepts an additional/new reset argument, which defaults to True and thus does not change existing behaviour. This is implemented for HMMs and RNNs.

To be able to process sequences either as a whole or on a frame-by-frame basis, process() expects always the same data dimensionality. NeuralNetwork previously corrected the dimensions if a 1-dimensional input feature is expected (e.g. SuperFluxNN). This conversion is moved to the executable program instead.

If data is processed framewise, reset=False must be passed to the processors, otherwise stateful processors get reseted constantly. To be able to pass additional keyword arguments, madmom.processors was adapted accordingly.

fixes #239

fdlm · 2017-01-20T12:56:03Z

madmom/ml/hmm.pyx

@@ -394,6 +394,38 @@ class HiddenMarkovModel(object):
            raise ValueError('Initial distribution is not a probability '
                             'distribution.')
        self.initial_distribution = initial_distribution
+        # attributes needed for stateful processing (i.e. forward_step())
+        self._prev = self.initial_distribution.copy()
+        self._cur = np.zeros_like(self._prev)


Maybe just call reset() instead?

Also, we could use the variables here in forward_generator, instead of allocating additional memory (can't comment there, though, because it didn't change).

Yes, calling reset() does the same.

IMHO, we should get rid of forward_generator altogether in the long run, since it basically does the same as forward() with reset=False called with individual observations. What the current implementation of forward() doesn't support is handling of block_size other than 1. Thus I did not remove it for now.

EDIT: the block_size only affects the computation of the densities given the observations. So maybe we could get rid of forward_generator sooner than thought if we don't care about the additional CPU cycles if we compute the densities frame by frame, too. If we don't want to remove it completely (since it is not exactly the same as calling forward() multiple times), we can at least replace the duplicated logic inside this method and call forward() instead.

Ok, I opted against calling reset(), because I think it is more explicit to initialise variables in __init__() and not in any other method.

fdlm · 2017-01-20T12:56:32Z

madmom/ml/hmm.pyx

+        self.__dict__.update(state)
+        # add non-pickled attributes needed for stateful processing
+        self._prev = self.initial_distribution.copy()
+        self._cur = np.zeros_like(self._prev)


Same here, just call reset()?

fdlm · 2017-02-02T08:36:12Z

bin/SuperFluxNN

-        out_processor = [rnn, pp, writer]
+        # Note: we need np.atleast_2d and np.transpose before the RNN, since
+        #       it expects the data in 2D (1D means framewise processing)
+        out_processor = [np.atleast_2d, np.transpose, rnn, pp, writer]


Wouldn't it be better to have the SpectralOnsetProcessor return the correct shape (n_frames, 1)?
Having to use np.atleast_2d and np.transpose every time I use the SpectralOnsetProcessor seems a bit unintuitive (although I also see returning a 2d-array when the data is 1d also as a bit unintuitive - not sure which one to prefer).

Yes and no. I thought about this as well, but opted for the way of least invasion to existing code.
As you state, returning a 2d-array is a bit unintuitive.

There's a TODO item in #185 which says: "rewrite all features as processors (mostly madmom.features.onsets.*)". I think it is better to address this separately and not in this PR.

fdlm · 2017-02-02T08:37:26Z

madmom/ml/hmm.pyx

        cdef double [::1] tm_probabilities = tm.log_probabilities
-        cdef unsigned int num_states = tm.num_states
+        cdef int num_states = tm.num_states


Why not unsigned int?

Basically it shouldn't matter, at least for num_states. I was hoping to get rid of the problems on windows (#178) as well. I can revert line 451, but the others should be uint32_t since it is supposed to be safer for multiple platforms.

fdlm · 2017-02-02T08:37:35Z

madmom/ml/hmm.pyx


        # observation model stuff
        om = self.observation_model
-        cdef unsigned int num_observations = len(observations)
-        cdef unsigned int [::1] om_pointers = om.pointers
+        cdef int num_observations = len(observations)


Why not unsigned int?

fdlm · 2017-02-02T08:37:57Z

madmom/ml/hmm.pyx

@@ -440,7 +468,7 @@ class HiddenMarkovModel(object):
                                                       num_states),
                                                      dtype=np.uint32)
        # define counters etc.
-        cdef unsigned int state, frame, prev_state, pointer
+        cdef int state, frame, prev_state, pointer


Why not unsigned int? (I'm repeating myself... :) )

fdlm · 2017-02-02T08:41:53Z

madmom/ml/hmm.pyx

            # keep track of the normalisation sum
            prob_sum = 0
            # iterate over all states
            for state in range(num_states):
+                # we need to reset the current state in case this method
+                # gets called multiple times (i.e. framewise processing)
+                fwd_cur[frame, state] = 0


You don't need this, because you initialise fwd_cur with np.zeros when calling this function.

Yes, this is a leftover from a previous code version. Back then I also initialised fwd_cur of length 1 during initialisation and only re-allocated it if called with observations with length > 1.

fdlm · 2017-02-02T08:44:08Z

madmom/ml/hmm.pyx

-                                           dtype=np.float)
+        cdef double[::1] fwd_prev = self._prev
+        cdef double[:, ::1] fwd_cur = \
+            np.zeros((num_observations, num_states), dtype=np.float)


Since fwd_cur does not only hold the current forward values, but all of them (shape (num_observations, num_states)), I would suggest to keep the old name, fwd.

I don't understand why you need fwd_prev, if all the values are stored in fwd_cur anyways. You can access the previous values with fwd_cur[frame - 1].

ok, this makes sense.

fwd_cur holds only num_observations, not num_observations + 1 variables any more. The previous variables are stored in self._prev. This change was necessary (at least I think it is) because during initialisation we know how many states we have and can thus init self._prev, but don't know the lengths of the observations yet. If we want to be able to call forward() with observations of variable length (i.e. both a complete sequence or frame-by-frame), we have to init fwd_cur (or fwd, see 1.) inside forward() but need to keep the previous variables somewhere, hence self._prev.

needed to pass reset=False or other keywords to process() as proposed in #33 to distinguish between stateful and stateless processing (e.g. for HMMs or RNNs)

revert the separate computation/activation of complete sequences vs. stepwise processing; layers get resetted automatically when a whole sequence is processed

superbock force-pushed the stateful_hmms branch from 440ec31 to 1139eaf Compare January 20, 2017 12:38

superbock pushed a commit that referenced this pull request Jan 20, 2017

stateful processing of HMMs (#244)

1139eaf

superbock force-pushed the stateful_hmms branch from 1139eaf to dfe5756 Compare January 20, 2017 12:38

superbock pushed a commit that referenced this pull request Jan 20, 2017

stateful processing of HMMs (#244)

dfe5756

superbock force-pushed the stateful_hmms branch from dfe5756 to d31ea5c Compare January 20, 2017 12:48

superbock pushed a commit that referenced this pull request Jan 20, 2017

stateful processing of HMMs (#244)

d31ea5c

superbock requested a review from fdlm January 20, 2017 12:49

superbock force-pushed the stateful_hmms branch from d31ea5c to 370fb38 Compare January 21, 2017 09:37

superbock pushed a commit that referenced this pull request Jan 21, 2017

stateful processing of HMMs (#244)

370fb38

superbock mentioned this pull request Jan 21, 2017

Proper handling of state-ful processors #237

Closed

4 tasks

superbock force-pushed the stateful_hmms branch from 370fb38 to 00a9d65 Compare January 21, 2017 22:09

superbock pushed a commit that referenced this pull request Jan 21, 2017

stateful processing of HMMs (#244)

00a9d65

superbock mentioned this pull request Jan 23, 2017

Streaming mode #185

Merged

17 tasks

superbock force-pushed the stateful_hmms branch from 56147ce to d918205 Compare January 27, 2017 09:43

superbock pushed a commit that referenced this pull request Jan 27, 2017

stateful processing of HMMs (#244)

d918205

superbock force-pushed the stateful_hmms branch from d918205 to 2903951 Compare January 27, 2017 09:44

superbock pushed a commit that referenced this pull request Jan 27, 2017

stateful processing of HMMs (#244)

2903951

fixes #239

superbock force-pushed the stateful_hmms branch 3 times, most recently from 8d5cd01 to 372a18a Compare January 29, 2017 15:19

superbock changed the title ~~Stateful processing of HMMs~~ Stateful processing Jan 29, 2017

superbock requested review from fdlm and removed request for fdlm January 29, 2017 15:40

superbock force-pushed the stateful_hmms branch 3 times, most recently from 6e3a374 to d93a5c8 Compare January 31, 2017 12:26

fdlm reviewed Feb 2, 2017

View reviewed changes

Sebastian Böck added 3 commits February 3, 2017 09:14

stateful processing of HMMs; fixes #239

3e3bc15

enable passing of process(**kwargs) through Processors

58aa6ef

needed to pass reset=False or other keywords to process() as proposed in #33 to distinguish between stateful and stateless processing (e.g. for HMMs or RNNs)

neural networks and layers expect 2D input and can be resetted

5a9b3c7

revert the separate computation/activation of complete sequences vs. stepwise processing; layers get resetted automatically when a whole sequence is processed

superbock force-pushed the stateful_hmms branch from d93a5c8 to 5a9b3c7 Compare February 3, 2017 08:21

superbock merged commit 5e3879d into master Feb 3, 2017

superbock deleted the stateful_hmms branch February 6, 2017 04:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stateful processing #244

Stateful processing #244

superbock commented Jan 20, 2017 •

edited

Loading

fdlm Jan 20, 2017

fdlm Jan 20, 2017

superbock Feb 2, 2017 •

edited

Loading

superbock Feb 3, 2017

fdlm Jan 20, 2017

fdlm Feb 2, 2017

superbock Feb 2, 2017

fdlm Feb 2, 2017

superbock Feb 2, 2017

fdlm Feb 2, 2017

fdlm Feb 2, 2017

fdlm Feb 2, 2017

superbock Feb 2, 2017

fdlm Feb 2, 2017

superbock Feb 2, 2017

Stateful processing #244

Stateful processing #244

Conversation

superbock commented Jan 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superbock Feb 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superbock commented Jan 20, 2017 •

edited

Loading

superbock Feb 2, 2017 •

edited

Loading