BL R&D Report 5481

S T A T I S T I C A L 0 F R E L E V A N C E FOR THE

B A S E S

A S S E S S M E N T

'IDEAL' I N F O R M A T I O N T E S T R E T R I E V A L

C O L L E C T I O N

H. K.

G i l b e r t S p a r c k J o n e s

Computer Laboratory University of Cambridge Corn Exchange Street Cambridge, England

0
March 1979

Acknowledgement

This work was supported by British Library Research and Development Department under Grant SI/G/267. We are grateful to Dr. P.M.E. Altham of the University Statistical Laboratory, and to Dr. C.J. van Rijsbergen and Dr. S.E. Robertson for their assistance.

Contents page SI Al Al A2 A3 A4

Summary Section A : The 'ideal1 test collection and obtaining relevance assessments for it Al A2 A3 A4 Introduction: the 'ideal1 information retrieval test collection The 'ideal' collection specification Relevance data Constraints on statistical methods

Section B : Statistical methods of determining relevance assessment requirements Bl 1. L B2 2. 1 2. 2 2. 2. 2.2. 2 2 2 B3 3 3 3. B4 4. 4. B5 B6 Section C CI C2 2. 1 2, 2 2.3 References Appendix 1 Tables and f:igures Table 0 Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Figures 1 and Preface: the research proposed and its conduct General requirement The Pool method Non-statistical summary Statistical presentation Scrutiny of design study argument Improved sampling rationale: the modified Pool method Accuracy of estimators Weakening the assumptions Conclusion on the Pool method Other methods: dead ends Logistic model Wilcoxon's signed ranks test Likelihood ratio test The Squares method Non-statistical summary Statistical presentation Comparison between the Pool and Squares method Multi-strategy comparisons

B] Bl B3
B4

B4 BIO BIO B15 BIG B18 B18 B19 B20 B23 B25 B26 B26 B32 B37 B41

Practical implications of the statistical methods for building and using the 'ideal' collection Summary of the properties of the statistical methods Application of the methods Testing to obtain statistical parameter values Practical implications of the methods Assessment of the methods

CI C] C3 C3 C3 C7 RJ Al A2 A3 A4 A6 A8 A12 A13 A14 A19 A23

Appendix 2 : Computing