Missing pieces for first release

[albertz]

Also check for a complete list: https://github.com/rwth-i6/returnn_common/milestone/1

• Rec design for recurrent definitions / loops #16
• Design/Handling of dimension tags #17
• How to define whether search (or train flag) is enabled? #18
• Functional layer API, conventions #21
• Masked computation wrapper #23
• Cond wrapper #24
• Consistent variable name scopes for modules #25
• RETURNN layers with hidden state should make it explicit #31
• Sequential module #33
• ModuleList module #34
• Loop.State incomplete #35
• Losses to implement and losses naming conventions #38
• Reorganization of code and module names, and user network code conventions #39
• Remove automatic layer input concatenation? #41
• Make minimum Python version explicit #43
• Better way to define main network dict (make_root_net_dict) #44
• Replace reduce(x, mode="mean", ...) by reduce_mean(x, ...), etc? #46
• Make shape and dims available? #47
• Enforce dim tags to be unique? #48
• Higher-level encoder decoder interfaces for transducer, attention, LM, ILM, etc #49
• Name scope for custom module functions? #50
• How to handle Sisyphus hashes #51
• Implement standard attention and self-attention module #52
• Implement standard Transformer encoder and decoder #53
• Implement standard Conformer encoder #54
• Definition of losses, mark_as_loss? #56
• Collecting frame error, label error, edit distance? #57
• How to define the API for parameter initialization, regularization (L2, weight dropout, etc), maybe updater opts per-param #59
• Have Conv1d, Conv2d, Conv3d instead of Conv? Same for Pool? #61
• Remove activation option from Linear, Conv #62
• Make activation function private, prefer all direct wrappers like relu etc #63
• Make eval function private, or rename it? #64
• Implement SpecAugment #65
• nn.dot should have better signature #67
• How to flatten (pack) sequence before softmax + loss or just loss #68
• Unexpected default for nn.softmax #69
• RecLayer time dim explicit instead of implicit via RecUnstackLayer #72
• Transformer example with relative positional encoding #74
• Inconsistency: input dim needed for custom modules with custom parameters but not builtin layer wrappers? #75
• att_dropout is probably applied wrongly (on the feature) #76
• nn.dropout should have axis mandatory #77
• Change nn.SelfAttention default att_dropout? #78
• SelfAttentionStep lacks initial_state #79
• Implement Loop.last (or maybe only for states) #80
• Unify rec ...Step variants with on-seq variants? #81
• Only functional wrapped layers except VariableLayer? #82
• Fix batch norm defaults #83
• Make sure a recent RETURNN behavior version is used? #84
• How to have custom updates for parameters #90
• How to handle parameter initialization #92
• Model checkpoint load and store logic #93
• What param init defaults should we use #94
• Use stateless random ops #95
• Parameter random init outside name ctx does not work #109
• Remove unused layers #111
• name_scope layer option not used ideally #112
• Dim tag var names in serialized code are too long #113
• Loop max_seq_len support incomplete #114
• Out spatial dim tags arguments not needed #115
• Out dim tags potentially needs duplicated logic #116
• Dim tag description and identifier name inconsistent and not optimal  #119
• Inconsistency for axis or in_spatial_dim #120
• Dim internals and API should be refactored returnn#975
• Improve verify_out_shape, ignore unrelated dims #121
• Root module, naming logic API #125
• Conformer misses relative pos encoding #132
• Broadcasting with standard operators #136
• Introduce DType class or so? #137
• constant, random, reduce: shape requires ordering #138
• Allow anonymous parameters? #147
• nn.Random for multiple ops #148
• Weight Decay #149
• Serialization of big Numpy arrays #150
• Loop not simple enough in some cases #151
• Remove nn.scoped #159
• mark_as_loss: extension for LR scheduling? #211
• Lazy init causes unexpected behavior? #212
• Conformer/Transformer has same initial param value in each layer #216
• SelfAttention misses Linear after attention, wrong for Conformer, Transformer #221