Skip to content

Latest commit

 

History

History
69 lines (50 loc) · 2.36 KB

Readme.md

File metadata and controls

69 lines (50 loc) · 2.36 KB
Raw

Visual Question Generation for Class Acquisition of Unknown Objects

This is the implementation of Visual Question Generation for Class Acquisition of Unknown Objects by Kohei Uehara, Antonio Tejero-De-Pablos, Yoshitaka Ushiku and Tatsuya Harada (ECCV 2018).

The link to the paper is here: https://arxiv.org/abs/1808.01821

Requirements

  • Python 3.6
  • Chainer 3.2.0
  • cupy 2.2.0
  • matplotlib 2.1.0
  • scikit-image
  • selectivesearch
  • pyimagesaliency
    • This code is for python2, so you need to add option use_2to3 = True to setup.py.
  • opencv-python
  • nltk
  • numpy

Usage

Download

You can download files for this code, and if you want to use them, put them in the /data folder.

  • pretrained ResNet model [Download]
  • pretrained VQG model [Download]
  • our Dataset [Download]
    • This contains questions, answers, and question target for each of the images from Visual Genome dataset.
  • word embeddings [Download]
    • This contains word vectors by poincare embeddings for each of the target words from wordnet synset.
  • word id mappings [Download]

Also, you need to download Visual Genome images, and put them to /data/images/

Test

You can test our entire module on your image (generate a question for an unknown object in an image), run

python test.py -image path/to/image

Train

First, preprocess the data by

python src/preprocess.py

Next, extract features (you can download extracted features here)

python src/feature_extract.py

Then, for training visual question generation module, run

python src/train.py

Before run this code, you need to put data to /data folder correctly.

If you want to test our question generation module (generate questions from image, target region, and target word), run

python q_test.py

then you can get questions (q_result.json) for each target in images.