From ebb040cc26697e8b98ac11652e99f50066fdd45c Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Thu, 2 Jul 2020 18:53:30 +0500 Subject: [PATCH 1/8] Specify operation CTCLoss-4 Signed-off-by: Roman Kazantsev --- docs/ops/opset4.md | 1 + docs/ops/sequence/CTCLoss_4.md | 108 +++++++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100644 docs/ops/sequence/CTCLoss_4.md diff --git a/docs/ops/opset4.md b/docs/ops/opset4.md index ced7fe1e76dc22..e377428d38bcd4 100644 --- a/docs/ops/opset4.md +++ b/docs/ops/opset4.md @@ -25,6 +25,7 @@ declared in `namespace opset4`. * [Broadcast](movement/Broadcast_3.md) * [Bucketize](condition/Bucketize_3.md) * [CTCGreedyDecoder](sequence/CTCGreedyDecoder_1.md) +* [CTCLoss](sequence/CTCLoss_4.md) * [Ceiling](arithmetic/Ceiling_1.md) * [Clamp](activation/Clamp_1.md) * [Concat](movement/Concat_1.md) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md new file mode 100644 index 00000000000000..6b6effe760f7db --- /dev/null +++ b/docs/ops/sequence/CTCLoss_4.md @@ -0,0 +1,108 @@ +## CTCLoss + +**Versioned name**: *CTCLoss-4* + +**Category**: Sequence processing + +**Short description**: *CTCLoss* computes the CTC (Connectionist Temporal Classification) Loss. + +**Detailed description**: + +This operation is similar to the TensorFlow* operation [CTCLoss](https://www.tensorflow.org/api_docs/python/tf/compat/v1/nn/ctc_loss) + +*CTCLoss* calculates loss between input and target sequences. It sums over the probability of possible alignments of input to target. + +Input sequences `inputs[:,i,:]` can have different length. The lengths of sequences are coded as values 1 and 0 in the second input tensor `sequence_mask`. +Value `sequence_mask[j, i]` specifies whether there is a sequence symbol at index `j` in the sequence `i` in the batch of sequences. +If there is no symbol at `j`-th position `sequence_mask[j, i] = 0`, and `sequence_mask[j, i] = 1` otherwise. +Starting from `j = 0`, `sequence_mass[j, i]` are equal to 1 up to the particular index `j = last_sequence_symbol`, +which is defined independently for each sequence `i`. For `j > last_sequence_symbol`, values in `sequence_mask[j, i]` are all zeros. +A length of target sequence from `labels[i,:]` must not be greater than a lenght of corresponding input sequence. Otherwise, the operation behaviour is undefined. + +*CTCLoss* calculation scheme: + +1. Compute probability of `j`-th character at time step `t` for `i`-th input sequence from `inputs` using softmax formula: +\f[ +p_{t,i,j} = \frac{\exp(inputs[t,i,j])}{\sum^{K}_{k=0}{\exp(inputs[t,i,k])}} +\f] + +2. For a given `i`-th target from `labels[i,:]` find all aligned paths. +A path `S = (c1,c2,...,cT)` is aligned with a target `G=(g1,g2,...,gT)` if both chains are equal after decoding. +The decoding removes pads `-1` from a target `G` and merges repeated characters in `G` in case *preprocess_collapse_repeated* equal to True. +The decoding merges repeated characters in `S` in case *merge_repeated* equal to True and removes blank characters represented by last class index `K-1`, +where `K` is a number of classes including a blank symbol. +For example, in case default *merge_repeated* and *preprocess_collapse_repeated* a target sequence `(0,3,2,2,-1,-1,-1,-1,-1)` is processed to `(0,3,2,2)` and +an input `(0,0,4,3,2,2,4,2,4)` is also processed to `(0,3,2,2)`, where `K=5`. +Compute probabilities of these alignments as follows: +\f[ +p(S) = \prod_{t=1}^{T} p_{t,i,ct} +\f] + +3. Finally sum up probabilities of all aligned paths for a given target and compute negative logarithm of it: +\f[ +CTCLoss = -\log \sum_{S} p(S) +\f] + + +**Attributes** + +* *preprocess_collapse_repeated* + + * **Description**: *preprocess_collapse_repeated* is a flag for a preprocessing step before loss calculation, wherein repeated labels passed to the loss are merged into single labels. + * **Range of values**: True or False + * **Type**: `boolean` + * **Default value**: False + * **Required**: *no* + +* *merge_repeated* + + * **Description**: *merge_repeated* is a flag for merging repeated labels during the CTC loss calculation. + * **Range of values**: True or False + * **Type**: `boolean` + * **Default value**: True + * **Required**: *no* + +**Inputs** + +* **1**: `inputs` - Input tensor with a batch of sequences. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes. Required. + +* **2**: `sequence_mask` - 2D input tensor of type *T_IND* with sequence masks for each sequence in the batch. Populated with values 0 and 1. Shape of this input is `[T, N]`. Required. + +* **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A sequence can be shorter than the size `T` of the tensor, all elements that do not code sequence classes are filled with -1. Required. + +**Output** + +* **1**: Output tensor with shape `[N]`, negative log of summed up probabilities for aligned paths. Type of elements is *T_F*. + +**Types** + +* *T_F*: any supported floating point type. + +* *T_IND*: any supported integer type. + +**Example** + +```xml + + + + 20 + 8 + 128 + + + 20 + 8 + + + 8 + 20 + + + + + 8 + + + +``` From 65c957e5c2d873c1003793a426e7bfeab464b57f Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Sat, 11 Jul 2020 00:50:01 +0500 Subject: [PATCH 2/8] Correct documentation for CTCLoss after #1 review Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 55 ++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index 6b6effe760f7db..2288413322d998 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -10,14 +10,13 @@ This operation is similar to the TensorFlow* operation [CTCLoss](https://www.tensorflow.org/api_docs/python/tf/compat/v1/nn/ctc_loss) -*CTCLoss* calculates loss between input and target sequences. It sums over the probability of possible alignments of input to target. +*CTCLoss* estimates a chance that a target can occur (or is real) for given input sequence of logits. +Briefly, *CTCLoss* operation finds all sequences aligned with a target sequence `labels[i,:]`, computes log-probabilities of these aligned sequences using `inputs[:,i,:]` of logits +and computes a negative sum of these log-probabilies. -Input sequences `inputs[:,i,:]` can have different length. The lengths of sequences are coded as values 1 and 0 in the second input tensor `sequence_mask`. -Value `sequence_mask[j, i]` specifies whether there is a sequence symbol at index `j` in the sequence `i` in the batch of sequences. -If there is no symbol at `j`-th position `sequence_mask[j, i] = 0`, and `sequence_mask[j, i] = 1` otherwise. -Starting from `j = 0`, `sequence_mass[j, i]` are equal to 1 up to the particular index `j = last_sequence_symbol`, -which is defined independently for each sequence `i`. For `j > last_sequence_symbol`, values in `sequence_mask[j, i]` are all zeros. -A length of target sequence from `labels[i,:]` must not be greater than a lenght of corresponding input sequence. Otherwise, the operation behaviour is undefined. +Input sequences of logits from `inputs` can have different length. The length of each sequence `inputs[:,i,:]` equals `input_length[i]`. +A length of target sequence from `labels[i,:]` is determined by a pad `-1`. The length must not be greater than a lenght of corresponding input sequence `inputs[:,i,:]`. +Otherwise, the operation behaviour is undefined. *CTCLoss* calculation scheme: @@ -29,18 +28,19 @@ p_{t,i,j} = \frac{\exp(inputs[t,i,j])}{\sum^{K}_{k=0}{\exp(inputs[t,i,k])}} 2. For a given `i`-th target from `labels[i,:]` find all aligned paths. A path `S = (c1,c2,...,cT)` is aligned with a target `G=(g1,g2,...,gT)` if both chains are equal after decoding. The decoding removes pads `-1` from a target `G` and merges repeated characters in `G` in case *preprocess_collapse_repeated* equal to True. -The decoding merges repeated characters in `S` in case *merge_repeated* equal to True and removes blank characters represented by last class index `K-1`, -where `K` is a number of classes including a blank symbol. -For example, in case default *merge_repeated* and *preprocess_collapse_repeated* a target sequence `(0,3,2,2,-1,-1,-1,-1,-1)` is processed to `(0,3,2,2)` and -an input `(0,0,4,3,2,2,4,2,4)` is also processed to `(0,3,2,2)`, where `K=5`. +The decoding merges repeated characters in `S` in case *ctc_merge_repeated* equal to True and removes blank characters represented by `blank_index`. +By default, `blank_index` is equal to `C-1`, where `C` is a number of classes including the blank. +For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique*, and `blank_index` a target sequence `(0,3,2,2,-1,-1,-1,-1,-1)` is processed to `(0,3,2,2)` and +a path `(0,0,4,3,2,2,4,2,4)` is also processed to `(0,3,2,2)`, where `C=5`. There exist other paths that are also aligned, for instance, `0,4,3,3,2,4,2,2,2`. +Paths checked for alignment with `label[:,i]` must be of length `input_length[i] = L_i`. Compute probabilities of these alignments as follows: \f[ -p(S) = \prod_{t=1}^{T} p_{t,i,ct} +p(S) = \prod_{t=1}^{L_i} p_{t,i,ct} \f] -3. Finally sum up probabilities of all aligned paths for a given target and compute negative logarithm of it: +3. Finally sum up logarithms of probabilities of all aligned paths with negative sign: \f[ -CTCLoss = -\log \sum_{S} p(S) +CTCLoss = - \sum_{S} \ln p(S) \f] @@ -54,22 +54,32 @@ CTCLoss = -\log \sum_{S} p(S) * **Default value**: False * **Required**: *no* -* *merge_repeated* +* *ctc_merge_repeated* - * **Description**: *merge_repeated* is a flag for merging repeated labels during the CTC loss calculation. + * **Description**: *ctc_merge_repeated* is a flag for merging repeated labels during the CTC loss calculation. * **Range of values**: True or False * **Type**: `boolean` * **Default value**: True * **Required**: *no* +* *unique* + + * **Description**: *unique* is a flag to find unique elements for each `labels[i,:]` before matching with potential alignments2. + * **Range of values**: True or False + * **Type**: `boolean` + * **Default value**: False + * **Required**: *no* + **Inputs** -* **1**: `inputs` - Input tensor with a batch of sequences. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes. Required. +* **1**: `inputs` - Input tensor with a batch of sequences. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes including the blank. Required. -* **2**: `sequence_mask` - 2D input tensor of type *T_IND* with sequence masks for each sequence in the batch. Populated with values 0 and 1. Shape of this input is `[T, N]`. Required. +* **2**: `input_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of values not greater than `T`. Lengths of input sequences `inputs[:,i,:]`. Required. * **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A sequence can be shorter than the size `T` of the tensor, all elements that do not code sequence classes are filled with -1. Required. +* **4**: `blank_index` - Scalar. Set the class index to use for the blank label. Default value is `C-1`. Optional. + **Output** * **1**: Output tensor with shape `[N]`, negative log of summed up probabilities for aligned paths. Type of elements is *T_F*. @@ -78,7 +88,7 @@ CTCLoss = -\log \sum_{S} p(S) * *T_F*: any supported floating point type. -* *T_IND*: any supported integer type. +* *T_IND*: any supported signed integer type. **Example** @@ -89,7 +99,7 @@ CTCLoss = -\log \sum_{S} p(S) 20 8 128 - + 20 8 @@ -97,12 +107,13 @@ CTCLoss = -\log \sum_{S} p(S) 8 20 - + + 8 - + ``` From b43636430b5788bf18589d193e7dc0720a9308cc Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Mon, 13 Jul 2020 19:56:19 +0500 Subject: [PATCH 3/8] Correct documentation for CTCLoss after #2 review Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index 2288413322d998..5e9b26ef5f720d 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -8,7 +8,7 @@ **Detailed description**: -This operation is similar to the TensorFlow* operation [CTCLoss](https://www.tensorflow.org/api_docs/python/tf/compat/v1/nn/ctc_loss) +*CTCLoss* operation is presented in [Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks: Graves et al., 2016](http://www.cs.toronto.edu/~graves/icml_2006.pdf) *CTCLoss* estimates a chance that a target can occur (or is real) for given input sequence of logits. Briefly, *CTCLoss* operation finds all sequences aligned with a target sequence `labels[i,:]`, computes log-probabilities of these aligned sequences using `inputs[:,i,:]` of logits @@ -32,15 +32,15 @@ The decoding merges repeated characters in `S` in case *ctc_merge_repeated* equa By default, `blank_index` is equal to `C-1`, where `C` is a number of classes including the blank. For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique*, and `blank_index` a target sequence `(0,3,2,2,-1,-1,-1,-1,-1)` is processed to `(0,3,2,2)` and a path `(0,0,4,3,2,2,4,2,4)` is also processed to `(0,3,2,2)`, where `C=5`. There exist other paths that are also aligned, for instance, `0,4,3,3,2,4,2,2,2`. -Paths checked for alignment with `label[:,i]` must be of length `input_length[i] = L_i`. -Compute probabilities of these alignments as follows: +Paths checked for alignment with a target `label[:,i]` must be of length `input_length[i] = L_i`. +Compute probabilities of these aligned paths (alignments) as follows: \f[ p(S) = \prod_{t=1}^{L_i} p_{t,i,ct} \f] -3. Finally sum up logarithms of probabilities of all aligned paths with negative sign: +3. Finally, compute negative sum of log-probabilities of all alignments: \f[ -CTCLoss = - \sum_{S} \ln p(S) +CTCLoss = \minus \sum_{S} \ln p(S) \f] @@ -64,7 +64,7 @@ CTCLoss = - \sum_{S} \ln p(S) * *unique* - * **Description**: *unique* is a flag to find unique elements for each `labels[i,:]` before matching with potential alignments2. + * **Description**: *unique* is a flag to find unique elements for a target `labels[i,:]` before matching with potential alignments. * **Range of values**: True or False * **Type**: `boolean` * **Default value**: False @@ -78,17 +78,17 @@ CTCLoss = - \sum_{S} \ln p(S) * **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A sequence can be shorter than the size `T` of the tensor, all elements that do not code sequence classes are filled with -1. Required. -* **4**: `blank_index` - Scalar. Set the class index to use for the blank label. Default value is `C-1`. Optional. +* **4**: `blank_index` - Scalar of type *T_IND*. Set the class index to use for the blank label. Default value is `C-1`. Optional. **Output** -* **1**: Output tensor with shape `[N]`, negative log of summed up probabilities for aligned paths. Type of elements is *T_F*. +* **1**: Output tensor with shape `[N]`, negative sum of log-probabilities of alignments. Type of elements is *T_F*. **Types** * *T_F*: any supported floating point type. -* *T_IND*: any supported signed integer type. +* *T_IND*: any supported integer type. **Example** From a39904b2845d7971ed0ff4e195ba3de1fd7fcb02 Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Tue, 14 Jul 2020 14:55:45 +0500 Subject: [PATCH 4/8] Correct documentation for CTCLoss after #3 review --- docs/ops/sequence/CTCLoss_4.md | 37 +++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index 5e9b26ef5f720d..d40e262d55ce4a 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -10,35 +10,36 @@ *CTCLoss* operation is presented in [Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks: Graves et al., 2016](http://www.cs.toronto.edu/~graves/icml_2006.pdf) -*CTCLoss* estimates a chance that a target can occur (or is real) for given input sequence of logits. -Briefly, *CTCLoss* operation finds all sequences aligned with a target sequence `labels[i,:]`, computes log-probabilities of these aligned sequences using `inputs[:,i,:]` of logits +*CTCLoss* estimates likelyhood that a target `labels[i,:]` can occur (or is real) for given input sequence of logits `logits[:,i,:]`. +Briefly, *CTCLoss* operation finds all sequences aligned with a target `labels[i,:]`, computes log-probabilities of the aligned sequences using `logits[:,i,:]` and computes a negative sum of these log-probabilies. -Input sequences of logits from `inputs` can have different length. The length of each sequence `inputs[:,i,:]` equals `input_length[i]`. -A length of target sequence from `labels[i,:]` is determined by a pad `-1`. The length must not be greater than a lenght of corresponding input sequence `inputs[:,i,:]`. +Input sequences of logits `logits` can have different length. The length of each sequence `logits[:,i,:]` equals `logit_length[i]`. +A length of target sequence `labels[i,:]` equals `label_length[i]`. The length of the target sequence must not be greater than the length of corresponding input sequence `logits[:,i,:]`. Otherwise, the operation behaviour is undefined. *CTCLoss* calculation scheme: -1. Compute probability of `j`-th character at time step `t` for `i`-th input sequence from `inputs` using softmax formula: +1. Compute probability of `j`-th character at time step `t` for `i`-th input sequence from `logits` using softmax formula: \f[ -p_{t,i,j} = \frac{\exp(inputs[t,i,j])}{\sum^{K}_{k=0}{\exp(inputs[t,i,k])}} +p_{t,i,j} = \frac{\exp(logits[t,i,j])}{\sum^{K}_{k=0}{\exp(logits[t,i,k])}} \f] 2. For a given `i`-th target from `labels[i,:]` find all aligned paths. A path `S = (c1,c2,...,cT)` is aligned with a target `G=(g1,g2,...,gT)` if both chains are equal after decoding. -The decoding removes pads `-1` from a target `G` and merges repeated characters in `G` in case *preprocess_collapse_repeated* equal to True. +The decoding extracts substring of length `label_length[i]` from a target `G`, merges repeated characters in `G` in case *preprocess_collapse_repeated* equal to True and +finds unique elements in the order of character occurence in case *unique* equal to True. The decoding merges repeated characters in `S` in case *ctc_merge_repeated* equal to True and removes blank characters represented by `blank_index`. By default, `blank_index` is equal to `C-1`, where `C` is a number of classes including the blank. -For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique*, and `blank_index` a target sequence `(0,3,2,2,-1,-1,-1,-1,-1)` is processed to `(0,3,2,2)` and -a path `(0,0,4,3,2,2,4,2,4)` is also processed to `(0,3,2,2)`, where `C=5`. There exist other paths that are also aligned, for instance, `0,4,3,3,2,4,2,2,2`. -Paths checked for alignment with a target `label[:,i]` must be of length `input_length[i] = L_i`. +For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique*, and `blank_index` a target sequence `G=(0,3,2,2,2,2,2,4,3)` of a length `label_length[i]=4` is processed +to `(0,3,2,2)` and a path `S=(0,0,4,3,2,2,4,2,4)` of a length `logit_length[i]=9` is also processed to `(0,3,2,2)`, where `C=5`. +There exist other paths that are also aligned with `G`, for instance, `0,4,3,3,2,4,2,2,2`. Paths checked for alignment with a target `label[:,i]` must be of length `logit_length[i] = L_i`. Compute probabilities of these aligned paths (alignments) as follows: \f[ p(S) = \prod_{t=1}^{L_i} p_{t,i,ct} \f] -3. Finally, compute negative sum of log-probabilities of all alignments: +3. Finally, compute negative sum of log-probabilities of all found alignments: \f[ CTCLoss = \minus \sum_{S} \ln p(S) \f] @@ -72,13 +73,15 @@ CTCLoss = \minus \sum_{S} \ln p(S) **Inputs** -* **1**: `inputs` - Input tensor with a batch of sequences. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes including the blank. Required. +* **1**: `logits` - Input tensor with a batch of sequences of logits. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes including the blank. Required. -* **2**: `input_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of values not greater than `T`. Lengths of input sequences `inputs[:,i,:]`. Required. +* **2**: `logit_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T`. Lengths of input sequences of logits `logits[:,i,:]`. Required. -* **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A sequence can be shorter than the size `T` of the tensor, all elements that do not code sequence classes are filled with -1. Required. +* **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A length of a target sequence `labels[i,:]` is equal to `label_length[i]` and must contain of integers from a range `[0; C-1]` except `blank_index`. Required. -* **4**: `blank_index` - Scalar of type *T_IND*. Set the class index to use for the blank label. Default value is `C-1`. Optional. +* **4**: `label_length` - 1D tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T` and `label_length[i] <= logit_length[i]` for all possible `i`. Required. + +* **5**: `blank_index` - Scalar of type *T_IND*. Set the class index to use for the blank label. Default value is `C-1`. Optional. **Output** @@ -101,7 +104,6 @@ CTCLoss = \minus \sum_{S} \ln p(S) 128 - 20 8 @@ -109,6 +111,9 @@ CTCLoss = \minus \sum_{S} \ln p(S) 20 + 8 + + From 57b9e560f1f6a8662e9cb83636ac765732b84af1 Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Tue, 14 Jul 2020 15:09:28 +0500 Subject: [PATCH 5/8] Correct documentation for CTCLoss after #4 review Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index d40e262d55ce4a..bcaafa4c2d815a 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -14,7 +14,7 @@ Briefly, *CTCLoss* operation finds all sequences aligned with a target `labels[i,:]`, computes log-probabilities of the aligned sequences using `logits[:,i,:]` and computes a negative sum of these log-probabilies. -Input sequences of logits `logits` can have different length. The length of each sequence `logits[:,i,:]` equals `logit_length[i]`. +Input sequences of logits `logits` can have different lengths. The length of each sequence `logits[:,i,:]` equals `logit_length[i]`. A length of target sequence `labels[i,:]` equals `label_length[i]`. The length of the target sequence must not be greater than the length of corresponding input sequence `logits[:,i,:]`. Otherwise, the operation behaviour is undefined. @@ -44,12 +44,13 @@ p(S) = \prod_{t=1}^{L_i} p_{t,i,ct} CTCLoss = \minus \sum_{S} \ln p(S) \f] +**Note**: This calculation scheme does not provide steps for optimal implementation and primarily serves for better explanation. **Attributes** * *preprocess_collapse_repeated* - * **Description**: *preprocess_collapse_repeated* is a flag for a preprocessing step before loss calculation, wherein repeated labels passed to the loss are merged into single labels. + * **Description**: *preprocess_collapse_repeated* is a flag for a preprocessing step before loss calculation, wherein repeated labels in `labels[i,:]` passed to the loss are merged into single labels. * **Range of values**: True or False * **Type**: `boolean` * **Default value**: False @@ -57,7 +58,7 @@ CTCLoss = \minus \sum_{S} \ln p(S) * *ctc_merge_repeated* - * **Description**: *ctc_merge_repeated* is a flag for merging repeated labels during the CTC loss calculation. + * **Description**: *ctc_merge_repeated* is a flag for merging repeated characters in a potential alignment during the CTC loss calculation. * **Range of values**: True or False * **Type**: `boolean` * **Default value**: True @@ -113,7 +114,7 @@ CTCLoss = \minus \sum_{S} \ln p(S) 8 - + From 1fc2f752113abf810eeb906247d4b2e9efe7cb46 Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Wed, 15 Jul 2020 14:11:14 +0500 Subject: [PATCH 6/8] Correct layout for logits and add more description for unique attribute Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index bcaafa4c2d815a..a92a8690dee61d 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -10,19 +10,19 @@ *CTCLoss* operation is presented in [Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks: Graves et al., 2016](http://www.cs.toronto.edu/~graves/icml_2006.pdf) -*CTCLoss* estimates likelyhood that a target `labels[i,:]` can occur (or is real) for given input sequence of logits `logits[:,i,:]`. -Briefly, *CTCLoss* operation finds all sequences aligned with a target `labels[i,:]`, computes log-probabilities of the aligned sequences using `logits[:,i,:]` +*CTCLoss* estimates likelyhood that a target `labels[i,:]` can occur (or is real) for given input sequence of logits `logits[i,:,:]`. +Briefly, *CTCLoss* operation finds all sequences aligned with a target `labels[i,:]`, computes log-probabilities of the aligned sequences using `logits[i,:,:]` and computes a negative sum of these log-probabilies. -Input sequences of logits `logits` can have different lengths. The length of each sequence `logits[:,i,:]` equals `logit_length[i]`. -A length of target sequence `labels[i,:]` equals `label_length[i]`. The length of the target sequence must not be greater than the length of corresponding input sequence `logits[:,i,:]`. +Input sequences of logits `logits` can have different lengths. The length of each sequence `logits[i,:,:]` equals `logit_length[i]`. +A length of target sequence `labels[i,:]` equals `label_length[i]`. The length of the target sequence must not be greater than the length of corresponding input sequence `logits[i,:,:]`. Otherwise, the operation behaviour is undefined. *CTCLoss* calculation scheme: 1. Compute probability of `j`-th character at time step `t` for `i`-th input sequence from `logits` using softmax formula: \f[ -p_{t,i,j} = \frac{\exp(logits[t,i,j])}{\sum^{K}_{k=0}{\exp(logits[t,i,k])}} +p_{t,i,j} = \frac{\exp(logits[i,t,j])}{\sum^{K}_{k=0}{\exp(logits[i,t,k])}} \f] 2. For a given `i`-th target from `labels[i,:]` find all aligned paths. @@ -66,7 +66,7 @@ CTCLoss = \minus \sum_{S} \ln p(S) * *unique* - * **Description**: *unique* is a flag to find unique elements for a target `labels[i,:]` before matching with potential alignments. + * **Description**: *unique* is a flag to find unique elements for a target `labels[i,:]` before matching with potential alignments. Unique elements in the processed `labels[i,:]` are sorted in the order of their occurence in original `labels[i,:]`. For example, the processed sequence for `labels[i,:]=(0,1,1,0,1,3,3,2,2,3)` of length `label_length[i]=10` will be `(0,1,3,2)` in case *unique* equal to True. * **Range of values**: True or False * **Type**: `boolean` * **Default value**: False @@ -74,9 +74,9 @@ CTCLoss = \minus \sum_{S} \ln p(S) **Inputs** -* **1**: `logits` - Input tensor with a batch of sequences of logits. Type of elements is *T_F*. Shape of the tensor is `[T, N, C]`, where `T` is the maximum sequence length, `N` is the batch size and `C` is the number of classes including the blank. Required. +* **1**: `logits` - Input tensor with a batch of sequences of logits. Type of elements is *T_F*. Shape of the tensor is `[N, T, C]`, where `N` is the batch size, `T` is the maximum sequence length and `C` is the number of classes including the blank. Required. -* **2**: `logit_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T`. Lengths of input sequences of logits `logits[:,i,:]`. Required. +* **2**: `logit_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T`. Lengths of input sequences of logits `logits[i,:,:]`. Required. * **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A length of a target sequence `labels[i,:]` is equal to `label_length[i]` and must contain of integers from a range `[0; C-1]` except `blank_index`. Required. @@ -100,8 +100,8 @@ CTCLoss = \minus \sum_{S} \ln p(S) - 20 8 + 20 128 From a7ae34b6b106887d5b1e601c5c2b7d1931ac265e Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Wed, 15 Jul 2020 16:52:16 +0500 Subject: [PATCH 7/8] Correct types for length and indices tensors Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index a92a8690dee61d..ced3e3b44012f6 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -76,13 +76,13 @@ CTCLoss = \minus \sum_{S} \ln p(S) * **1**: `logits` - Input tensor with a batch of sequences of logits. Type of elements is *T_F*. Shape of the tensor is `[N, T, C]`, where `N` is the batch size, `T` is the maximum sequence length and `C` is the number of classes including the blank. Required. -* **2**: `logit_length` - 1D input tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T`. Lengths of input sequences of logits `logits[i,:,:]`. Required. +* **2**: `logit_length` - 1D input tensor of type *T1* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T`. Lengths of input sequences of logits `logits[i,:,:]`. Required. -* **3**: `labels` - 2D tensor with shape `[N, T]` of type *T_IND*. A length of a target sequence `labels[i,:]` is equal to `label_length[i]` and must contain of integers from a range `[0; C-1]` except `blank_index`. Required. +* **3**: `labels` - 2D tensor with shape `[N, T]` of type *T2*. A length of a target sequence `labels[i,:]` is equal to `label_length[i]` and must contain of integers from a range `[0; C-1]` except `blank_index`. Required. -* **4**: `label_length` - 1D tensor of type *T_IND* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T` and `label_length[i] <= logit_length[i]` for all possible `i`. Required. +* **4**: `label_length` - 1D tensor of type *T1* and of a shape `[N]`. The tensor must consist of non-negative values not greater than `T` and `label_length[i] <= logit_length[i]` for all possible `i`. Required. -* **5**: `blank_index` - Scalar of type *T_IND*. Set the class index to use for the blank label. Default value is `C-1`. Optional. +* **5**: `blank_index` - Scalar of type *T2*. Set the class index to use for the blank label. Default value is `C-1`. Optional. **Output** @@ -92,7 +92,7 @@ CTCLoss = \minus \sum_{S} \ln p(S) * *T_F*: any supported floating point type. -* *T_IND*: any supported integer type. +* *T1*, *T2*: `int32` or `int64`. **Example** From ef4d58dbdce3719871ef370e2eba57d10acdeba6 Mon Sep 17 00:00:00 2001 From: Roman Kazantsev Date: Wed, 15 Jul 2020 20:15:26 +0500 Subject: [PATCH 8/8] Correct formulas and punctuation Signed-off-by: Roman Kazantsev --- docs/ops/sequence/CTCLoss_4.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/ops/sequence/CTCLoss_4.md b/docs/ops/sequence/CTCLoss_4.md index ced3e3b44012f6..0fa47ad66af2ee 100644 --- a/docs/ops/sequence/CTCLoss_4.md +++ b/docs/ops/sequence/CTCLoss_4.md @@ -22,7 +22,7 @@ Otherwise, the operation behaviour is undefined. 1. Compute probability of `j`-th character at time step `t` for `i`-th input sequence from `logits` using softmax formula: \f[ -p_{t,i,j} = \frac{\exp(logits[i,t,j])}{\sum^{K}_{k=0}{\exp(logits[i,t,k])}} +p_{i,t,j} = \frac{\exp(logits[i,t,j])}{\sum^{K}_{k=0}{\exp(logits[i,t,k])}} \f] 2. For a given `i`-th target from `labels[i,:]` find all aligned paths. @@ -31,12 +31,12 @@ The decoding extracts substring of length `label_length[i]` from a target `G`, m finds unique elements in the order of character occurence in case *unique* equal to True. The decoding merges repeated characters in `S` in case *ctc_merge_repeated* equal to True and removes blank characters represented by `blank_index`. By default, `blank_index` is equal to `C-1`, where `C` is a number of classes including the blank. -For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique*, and `blank_index` a target sequence `G=(0,3,2,2,2,2,2,4,3)` of a length `label_length[i]=4` is processed +For example, in case default *ctc_merge_repeated*, *preprocess_collapse_repeated*, *unique* and `blank_index` a target sequence `G=(0,3,2,2,2,2,2,4,3)` of a length `label_length[i]=4` is processed to `(0,3,2,2)` and a path `S=(0,0,4,3,2,2,4,2,4)` of a length `logit_length[i]=9` is also processed to `(0,3,2,2)`, where `C=5`. There exist other paths that are also aligned with `G`, for instance, `0,4,3,3,2,4,2,2,2`. Paths checked for alignment with a target `label[:,i]` must be of length `logit_length[i] = L_i`. Compute probabilities of these aligned paths (alignments) as follows: \f[ -p(S) = \prod_{t=1}^{L_i} p_{t,i,ct} +p(S) = \prod_{t=1}^{L_i} p_{i,t,ct} \f] 3. Finally, compute negative sum of log-probabilities of all found alignments: