QA systems provide users with precise answers to their questions rather than returning a set of related documents and results. And although the area of QA system and evaluation is very vast in itself, effort was made to summarize findings from a number of different publications in the area. The paper covers 3 main parts to describe the aspects of QA systems, their evaluation and comparison. The first part describes the field of QA systems, current trends in the field and overall framework for these systems. The second part of the paper discusses the different evaluation matrices which have been defined to evaluate QA systems. These matrices range from the more traditional ones to contemporary proposed methods. A study of evaluation matrices was necessary and was done to compare the performance of various QA systems which is discussed in the last section. Finally descriptions of five identified QA systems is given with their process of answer generation and performance measures based on the evaluation matrix followed by TREC (These QA systems were chosen as they had maximum data available about their performance and architecture). Although the performance data is very limited and usually do not come from an independent source but effort was made to obtain it and quantize them to predict the ideal QA system among the identified ones.
Full Paper: QA Systems