[150303]time_measure_overall

This is a manual log, composed on 2015 Mar. 3rd, to compare and contract time usage between parser and feature generator, either 1st-order or 3rd-order

The motivation of this testing log is that, after we have implemented a 3rd-order Eisner parser and related feature generator, we found the overall performance degrades dramatically, such that it is nearly impossible for us to measure accuracy even on a single training section. Therefore, it is urgent for us to figure out which part of the system slows down overall performance, and that the way to improve existing algorithms/procedures.

We did experiment based on several features of our program. The first feature is that the program is capable of training only a specified number of sentences before it terminates. The number is specified using command line argument --debug-run-number=[int]. This feature ensures we could compare performance on the same scale.

Another important feature is that it is possible to measure time usage for each sentence's training process. This is achived using built-in time library, and is controlled by -a command line argument.

The experiment consists of four independent parts, in which we managed to combine (1st-order parser, 3rd-order parser) and (1st-order feature, 3rd-order feature), yielding four combinations. Although 2 of them are meaningless because of mismatch of order between parser and feature generator, we cared mainly about time usage, instead of correctness and semantics.

We switch between parsers using command line argument --parser=[...]. However, switch between feature domains is more difficult. We introduced temporary methods for doing this. In order to enforce 1st-order feature for all parsers, we just return first order feature even if higher order features are requested from the caller of feature generator. This way, the time usage for any parser is only determined by parser itself. Similarly, in order to enforce 3rd-order features no matter which parser we were using, we force feature generator to compute higher order features even if only first-order is requested.

The result is posted below:

+==================+============+============+ | Parser | 1st-order | 3rd-order | | Feature |(50 sent's) | (10 sent's)| +------------------+------------+------------+ | 1st-order | 0.184303 | 1.713760 | +------------------+------------+------------+ | 3rd-order | 0.210180 | 2.115857 | +==================+============+============+

All values are average time for one sentence

From the above table, we could observe that, switcing from 1st-order parser to higher order nearly introduces a 10-times slow down, while switch between features are less observable, though still having a 0.85-times slow down.