VDCNN

Tensorflow Implementation of Very Deep Convolutional Neural Network for Text Classification, proposed by Conneau et al.

Archiecture for VDCNN is now correctly re-implemented with Tensorflow 2 and tf.keras support. A simple training interface is implemented following Tensorflow 2 Expert Tutorial. Feel free to contribute additional utilities like TensorBoard support.

Side Note, if you are a newcomer for NLP text classification:

• Please checkout new SOTA NLP methods like transformers or Bert.

• Check out PyTorch for MUCH BETTER dynamic graphing and dataset object support.

  • Current VDCNN implementation is also extremely easy to be ported onto PyTorch.

Prerequisites

• Python3
• Tensorflow >= 2.0
• tensorflow-datasets
• numpy

Datasets

The original paper tests several NLP datasets, including DBPedia, AG's News, Sogou News and etc.

tensorflow-datasets is used to support AG's News dataset.

Downloads of those NLP text classification datasets can be found here (Many thanks to ArdalanM):

Dataset	Classes	Train samples	Test samples	source
AG’s News	4	120 000	7 600	link
Sogou News	5	450 000	60 000	link
DBPedia	14	560 000	70 000	link
Yelp Review Polarity	2	560 000	38 000	link
Yelp Review Full	5	650 000	50 000	link
Yahoo! Answers	10	1 400 000	60 000	link
Amazon Review Full	5	3 000 000	650 000	link
Amazon Review Polarity	2	3 600 000	400 000	link

Parameters Setting

The original paper suggests the following details for training:

• SGD optimizer with lr 1e-2, decay 0.9.
• 10 - 15 epochs for convergence.
• He Initialization.

Some additional parameter settings for this repo:

• Gradient clipping with norm_value of 7.0, to stablize the training.

Skip connections and pooling are correctly implemented now:

• k-maxpooling.
• maxpooling with kernel size of 3 and strides 2.
• conv pooling with K_i convolutional layer.

For dotted skip connections:

• Identity with zero padding.
• Conv1D with kernel size of 1.

Please refer to Conneau et al for their methodology and experiment section in more detail.

Experiments

Results are reported as follows: (i) / (ii)

• (i): Test set accuracy reported by the paper (acc = 100% - error_rate)
• (ii): Test set accuracy reproduced by this Keras implementation

TODO: Feel free to report your own experimental results in the following format:

Results for "Identity" Shortcut, "k-max" Pooling:

Depth	ag_news	DBPedia	Sogou News
9 layers	90.17 / xx.xxxx	98.44 / xx.xxxx	96.42 / xx.xxxx
17 layers	90.61 / xx.xxxx	98.39 / xx.xxxx	96.49 / xx.xxxx
29 layers	91.33 / xx.xxxx	98.59 / xx.xxxx	96.82 / xx.xxxx
49 layers	xx.xx / xx.xxxx	xx.xx / xx.xxxx	xx.xx / xx.xxxx

Reference

Original preprocessing codes and VDCNN Implementation By geduo15

Train Script and data iterator from Convolutional Neural Network for Text Classification

NLP Datasets Gathered by ArdalanM and Others