Al Appendix 1: Tables and Figures Table O cf p.B8 The Pool method: percentage of pool requiring assessment for specified evaluation samples; approximate figures taken from the design study report (Sparck Jones and Bates (1977)). The Pool method: percentage of pool requiring assessment for specified evaluation samples; accurate comprehensive figures. The Pool approach: ideal evaluation conditions: (a) number of requests required for specified numbers of documents of known relevance status, assuming exhaustive assessments (b) number of documents required for specified numbers of requests. The Pool method ,modified: numbers of documents to be assessed for specified pool sizes, illustrated for recall. The Pool method: requirements for accuracy of estimators. Table 1 cf p.Bll Table 2 cf p.B14 Table 3 cf p.B15 Table 4 cf p.B17 Table 5 cf p.B18 The Pool method, weakened assumptions: percentage of pool requiring assessment for specified evaluation samples, pool not exhaustive. Wilcoxon test. Table 6 cf p.B24 Table 7 cf p.B36 The Squares method: number of documents to be retrieved by strategy A alone to support statement that A is better than B. Figure 1 Figure 2 The Pool method: illustration of Table 2. Illustrations of test collection relevant document distributions. A dot over a digit in the tables signifies recurring. A2 Table 0 d tha H -p O O O dP rH C rH D J-i II o rH LO rH 0) O v£> ro 00 CN kO O 0 ft tween nown -P C O H o •P nee LO * C D rH c u CN o\c M II o 0 ft o \D ** CN] CD CD PQ CD U C D rH X: M4 - M-l H ft — 00 0 0 ume ft 5M O 4-> o (D r H M II H 0 o\o 0 ^ > < H •H (0 C 4-> D U S^ CD c ro c fd i < >i -H r II LO CN dp etr C UIS fd W rd T5 0) ft rH Cn C D •P rd T3 0 Hi C O u 0 0 C D u tP C D -P fd -p CO -P d C M-i 3 D ses 3 P ^ 0 0 04 00 CN CN 5M II m r> II iJ CO •H C O C D H C C +J M 3 C T5 fd C D • H LO M-l C D ro rH C D 5M XI 0 X "1 0 Cn CO -H C CD to H 0 dp 0 o CN rH a> C ft rH M 3 CN rH -p C O C D -P rH C LO oV D o 0 0 o m O VO 00 rH -H M M II rH LO ft rH -P 4-> C O > m fd c C D £ -P C D 5M C D C O 'D •H rC 1 5-1 -iTl 0 M-i a ^ C D N -H •H & fl X 3 c •H O C O C r-} D ft C X O O U U O MM •P fd Cq CO — & 0) 5M 0 0 C CN o\° D U II H O C rH D 5M ft rH KD 00 00 CN ^ CN m 0 rH C O C 4J D -P M-l ti Q J C (3 D C •p O fd [fl C D C D 3 «• M D • -H e C D M 3 C O fd- B -p fd rC rH 0 o ft M-l o 0\° XI II LO 0 H 0 ft 0 * O cn. O r- O vo 0 0\° 8 3 rH C C D rH C •H C Cn D D U C > 5 & A -P SJ w V C D 5H cr C O 3 C D C rH rfj D 5 ft fi M C fd D P Q c e ^ CO O C -P D C O ft i •P >1 rd C D < D M II 0\° ft 0 •H M § r-", cd B MH w < -p .d 4J 3 -P •H rH 0 C -H D c 0 w a.( u i/) 'D -H P Q P 1< <: M u M M -H •H C D C •M X> Cn -H •H M p C O u C <*f LT XI > O 4-> fd .i:r> lH H H 0 u ti > -P (D c r: >.) G 5M •H C O 5 H •H C Cn 3 C -u u 3 Cn tf-H CD CO U CD rH 0 ft 0 a) w II C^ A ^o ft ii 2 II dp 1! II !! < C o x: ft -P B C D MM 5M C D U o\o 0 LO MH C > T s 0 in A ft CM o 00 LO VQ fa ^ 5 in LO CN fa CN vo roo o oo ^ ima C D -P x: MM 4-) CD CD A CO o ON o CN o CN O CN X 0 5M ft ft fd rH rH fd '-* B Cn H 0) o\o C O x: fd ft EM •• o C D w -p 5M Cn •H 3 C D D 0 C G M • & o O 00 m o o r- o o CV T o o fa Table w precentage es taken f 5M -H W C O Cn 0 u 0 .. 3 e 4J fd O -H O X CM O 5 M C D ft A3 Table 1 y A will exceed that tjn CD 4J O H O rH O C rH D •p LD CD CM II C D rH u II u o o\° 0 0 ft rH O LO rH O cD D rH rH rH o LO o rH O cDD O CM rH O CO o 00 O rrH CO o O VD O CO O LO r- o o o LO VD o CD O o v0 vD rLO rH pV O O o ^r o o vD 00 o o CM 00 o o o O 00 VD 00 CM O O o o CM 00 O o o 00 CM ^r CM ^r CM CM ft rH ^ rH rH VD vr CM rH 00 amp II o ft rH O rH r- 0^ VD 00 LO r^ LD vD 53 o -P 0 LO 0\° CO o M CO H 4J M II M -P LO CM C CD ft rH 0 O o rn o 00 00 C M Ox] O 00 CD rH ^r CM vD O rH CM rH < r0H ^ 0 VD 00 CM rH vD rH O rH ^r rH O rH CM rH > i £ U -p rtj vD kO r^ .00 a) f ses the 0 c o £ fH 0 C D o\° 0 H II ft rH ^r ^r •V0 vD O iX> VD 00 00 ^r CM 00 rH •p U) .el T3 C D o LO 0 o\° o 00 M •H 3 C D H rH II LD CM ft rH 0 CO 00 O 00 CO rH ^ CM O CM ^r rH 00 rH CM rH VD rH O rH O rH CM rH d) -p Ip rtf 4-1 4-' > G +J 0 CD 5H CD CO a) & M C D 69 u rH II O rH o\o 0 0 rH ft 0 0 rH O vD vD r- ^r a) CD • H 5 & ro H rd V< ^ c ^5 o w x: 0 ) O CD a) n • C D CD ^ T D (J) . -H CO -P N •H p td to •H CJ 1 CD CD ^ CD in PQ U fi 4-1 JH 4D C D 0 o\° u > g M II ft o * * * * * * * * rH o o * o o rH * <5 T -H 1+-I •H c u U H rd CJ -H •p U tp (]) m CD 4-' e 4-1 O V4 CD > i 4-1 •H rH •rH x: +^ 0^ - H •H VJ PQ I o\° CM COO f^CTi vD 00 m h LOvD CO c II X) p U c II II H •H O CJ^ C C Id D ^J rd 2 -p > rH id CD Q rc^ P 0 id U rH -P C D u r, D ft 0 C U) H 4-» ^ a) > 0 o -H K II ft II £ o\° tn u }H O\° 0 LO 4-1 CT> CD vD ^r o O CM VD rH 0D LO vD O vD rH 00 LO LO 0O LO ^r rr- oo LO CO vD LO LO O 00 LO ^r ^r v0 h LO LO O O o rH V0 h LO LO O G r*- rLO vD LO LO O O -H (7) co PM< PH 0 A ft 04 O O O O O O O O O O CD i 4J -P r - CM vO h rH rH O T rrCM CM ^r CM 00 CM vD UA < CM CM CM CM oo oo r00 ^ 00 oo 00 CM VD 0^ 00 ^r oo u H m > 0 1 -4-1 CO CD c CD id P rH id ^i C D CD 0 CD LO CO T5 H •s CO id C D rH e rH 0 0 P, dj -p t7> aJ Q) -H •H C 5^ ft CO C D ft c e •H P CD n c C) TJ CD U) m U M • H o\° LO rH ^ ft H ^ ft & ^ c e O CD 0 -H 0 !H LM 4-1 CO •P CO CD O O o0 0 o ^r O O LO O O vD O O CD r^ o 00 c o a> o O o difference between documents of known of A and of B o CO d 4-) 4-n 3 CD 0 rQ £ C D 0 ft 4J H 3 ft 3 ,C C 4J C £ D .C -H M -P nD ro CO C T3 C O Q 0 rd 4D id 1 1 a) Q) !H C D •rM > (/J ,a C D u A4 Table 2 (a) Exhaustive judgement RECALL Number of relevant documents per request 5 10 25 50 75 100 PRECISION Number of retrieved documents per request 5 10 25 50 75 100 250 500 1000 Number of requests required for 5% significance 830 424 172 89 60 46 19 12 8 1% significance 1135 581 235 121 82 63 26 17 12 Number of requests required for 5% significance 830 424 172 89 60 46 1% significance 1135 581 235 121 82 63 Table 2 : The Pool approach: ideal evaluation conditions (a) number of requests required for specified numbers of documents of known relevance status, assuming exhaustive assessments A5 Table 2(b) Exhaustive judgement Number of requests Sign ificance RECALL Number of relevant documents 445 785 85 117 40 57 21 28 15 19 11 15 9 12 8 10 7 9 6 8 5 7 5 6 PRECISION Number of retrieved documents 445 785 85 117 40 57 21 28 15 19 11 15 9 12 8 10 7 9 6 8 5 7 5 6 10 50 100 200 300 400 500 600 700 800 900 1000 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 Table 2 : The Pool approach: ideal evaluation conditions (b) number of documents required for specified numbers of requests A6 Table 3 Recall illustrated 100 documents in the pool Number of relevant documents required to be assessed 5 6 36 18 11 7 40 20 12 8 44 22 14 9 49 24 15 10 53 27 17 11 57 29 18 12 61 31 20 13 64 33 21 14 68 35 23 Number of relevant documents in pool 25 50 75 32 15 9 Number of relevant documents required to be assesed 15 16 75 40 25 17 79 42 27 18 82 44 28 19 85 46 30 20 88 48 31 21 91 22 94 23 96 24 99 25 100 58 38 Number of relevant documents in pool 25 50 75 72 38 24 Number of relevant documents required to be assessed 30 Number of relevant documents in pool 25 50 75 * 67 45 35 * 76 51 40 * 85 58 45 * 94 64 50 * 100 71 55 * • 60 * * 83 77 500 documents in the pool Number of relevant documents required to be assessed 5 6 186 97 65 49 18 7 208 109 74 55 21 8 230 121 82 61 23 9 250 133 90 67 25 10 270 144 97 73 28 11 290 155 105 79 30 12 309 166 113 85 33 13 328 177 120 91 35 14 346 188 128 96 37 Number of relevant documents in pool 25 50 75 100 250 163 85 57 43 16 Number of relevant documents required to be assessed 15 16 381 210 143 108 42 17 398 220 150 113 44 18 414 230 157 119 46 19 429 241 164 124 48 20 444 251 171 130 51 Number of relevant documents in pool 25 50 75 100 250 364 199 135 102 39 A7 Table 3 contd (500 documents in the pool) Number of relevant documents required to be assessed 25 Number of relevant documents in pool 25 50 75 100 250 500 300 206 156 61 30 * 347 240 183 72 35 * 392 273 209 83 40 • 45 • 50 * 500 369 284 115 55 * • 60 * * 429 332 136 434 306 234 94 472 338 259 104 400 308 125 1000 documents in the pool Number of relevant documents required to be assesed 5 Number of relevant documents in pool 25 50 75 100 250 500 328 172 116 87 34 16 6 374 197 133 100 39 18 7 418 221 150 113 44 21 8 461 245 166 125 49 23 9 502 268 182 137 54 26 10 542 290 197 149 59 28 11 582 313 213 161 64 12 620 335 228 173 69 13 657 357 243 184 73 14 694 379 258 195 78 - Number of relevant documents required to be assessed 15 Number of relevant documents in pool 25 50 75 100 250 500 729 400 273 207 83 16 763 422 288 218 87 17 797 443 303 230 92 18 829 464 317 241 96 19 860 484 331 252 101 20 889 504 346 263 105 21 917 22 943 23 966 24 986 25 998 602 415 316 128 Number of relevant documents required to be assessed 30 Number of relevant documents in pool 25 * 35 * 786 550 421 172 40 * 870 616 472 193 45 * 946 680 522 215 50 • 55 • 60 • 50 ! 696 75 483 1000 742 572 236 * 802 621 257 * 859 669 278 100 ; 369 250 149 not computed, not necessary Table 3 : The Pool method, modified: numbers of documents to be assessed for specified pool sizes, illustrated for recall A8 Table 4 (a) Assuming 1 - — ^ 1. a = upper bound to probability x 100 10% d = lower bound to divergence 0.05 0.04 0.03 0.02 0.01 0.05 0.04 0.03 0.02 0.01 0.05 0.04 0.03 0.02 0.01 > number of relevant 1 or retrieved in pool 272 425 756 1701 6806 384 600 1067 2401 9604 665 1040 1849 4160 16641 5% 1% A 1 (b) 'Exact1 form (— > — , say) a = 10% N 500 1000 1500 2000 2500 10% N 500 1000 1500 2000 2500 3000 3500 4000 d =• 0-04 n> 229 298 331 350 363 372 379 384 d = 0.05 n> 176 214 230 239 245 N = number of documents in the pool n> = lower bound for n A9 Table 4 contd a = 10% N 1000 2000 3000 4000 5000 6000 7000 a = 10% N 2000 5000 10000 15000 20000 a = 10% N 10000 20000 30000 40000 50000 60000 70000 d = 0.03 = n> 430 548 603 635 656 671 682 n> 919 1269 1453 1527 1567 d = 0.01 = n^ 4049 5078 5547 5816 5990 6112 6202 a = 5% N 500 1000 1500 2000 2500 3000 3500 d = 0.05 n> 217 277 305 322 332 340 346 d = 0.04 N n> 375 428 461 484 500 512 521 529 535 541 545 a = 5% 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000 A10 Table 4 contd = 5% N 2000 3000 4000 5000 6000 7000 8000 9000 10000 d = 0.03 n> 696 787 842 879 906 925 941 954 964 d = 0.02 n> 1333 1622 1936 2069 2143 2190 d = 0.01 n> 4899 6488 7275 7744 8056 8278 8445 8574 8678 = 5% N 3000 5000 10000 15000 20000 25000 a = 5% N 10000 20000 30000 40000 50000 60000 70000 80000 90000 = 1% N 1000 2000 3000 4000 5000 6000 7000 d = 0.05 n> 399 499 544 570 587 598 607 All Table 4 contd a = 1% N 2000 4000 6000 8000 10000 12000 d = 0.04 n^ 684 825 886 920 942 957 d = 0.03 a = 1% N 2000 5000 10000 15000 20000 n> 961 1350 1560 1646 1692 d = 0.02 a = 1% N 5000 10000 20000 30000 40000 n i 2270 2938 3443 3653 3768 d = 0.01 a = 19 N 25000 50000 75000 100000 125000 150000 175000 n> 9990 12485 13619 14266 14685 14979 15196 Table 4 : The Pool method: requirements for accuracy of estimators A12 Table 5 Suppose that only 90% of the relevant documents are contained in the pool. RECALL no. requests sig. % N> % of pool required as assessment sample rel rel rel rel rel = 5 =10 =25 =50 =75 % % % % % 66.66 33.33 22.25 84.44 42.22 28.15 48.8§ 24.44 16.30 66.66 33.33 22.22 100 40.00 20.00 13.33 53.33 26.66 17.7? 300 400 500 600 700 800 900 1000 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 15 19 11 15 9 12 8 10 7 9 6 8 5 7 5 6 88.88 35.55 17.7? 11.85 * 44.44 22.22 14.81 77.7? 31.11 15.55 10.37 100 40.00 20.00 13.33 66.66 26.66 13.33 8.88 88.8S 35.55 17.77 11.85 55.55 22.22 11.li 7.41 77.7? 31.ll 15.55 10.37 55.55 22.22 11.ll 66.66 26.66 13.33 7.41 8.88 Suppose that only 90% of the output of future strategy searches is in the pool. PRECISION retr = 25 300 400 500 600 700 800 900 1000 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 15 19 11 15 9 12 8 10 7 9 6 8 5 7 5 6 retr = 50 retr =100 retr =250 6.66 8.44 4.88 6.66 4.00 5.33 3.55 4.44 3.ll 4.00 2.66 3.55 2.22 3.11 2.22 2.66 retr = 1000 1.66 2.11 1.22 1.66 1.00 1.33 0.88 1.11 0.7? 1.00 0.66 0.88 0.55 0.7? 0.55 0.66 66.66 33.33 16.66 84.44 42.22 21.11 48.88 24.44 22.22 66.66 33.33 16.66 40.00 20.00 10.00 53.33 26.66 13.33 35.55 17.7? 8.88 44.44 22.22 11.ll 31.11 15.55 7.7? 40.00 20.00 10.00 26.66 13.33 35.55 11.11 22.22 11.11 31.11 15.55 22.22 11.ll 31.11 15.55 6.66 8.83 5.55 7.7? 5.55 6.66 Table 5 : The Pool method, weakened assumptions: percentage of pool requiring assessment for specified evaluation samples, pool not exhaustive A13 Table 6 no. requests sig. S> T > w For 95% power P> N 300 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 1.96 2.58 1.96 2.58 1.96 2.58 1.96 2.58 1.96 2.58 1.96 2.58 1.96 2.58 1.96 2.58 5894.698 7759.347 9069.82 5 11938.850 12670.721 16678.807 16651.938 21919.389 20980.099 27616.661 25629.336 33736.575 30578.836 40251.733 35811.377 47139.466 0.619 0.639 0.604 0.621 0.593 0.609 0.585 0.599 0.579 0.592 0.574 0.586 0.570 0.581 0.566 0.577 19 25 400 14 19 11 . 15 10 12 B 11 7 9 6 8 5 7 500 600 700 800 900 1000 w measure of difference between strategies A and B; other columns as Table 1« Table 6 : Wilcoxon test A14 Table 7 95% power assumed sig. n P,+P b c i.e. I. I R b+c b> P P b b c i.e. A > +P 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 50 0.25 6 5 0.935 12.5 10 11 14 15 13 14 17 19 22 23 21 23 25 27 29 31 13 14 17 19 23 25 26 28 32 34 38 41 42 44 46 49 51 54 0.923 0.966 0.866 0.902 0.858 0.897 0.810 0.871 0.804 0.853 0.797 0.848 0.778 0.823 0.751 0.789 0.890 0.928 0.810 0.871 0.792 0.839 0.761 0.804 0.743 0.778 0.718 0.762 0.718 0.745 0.701 0.738 0.691 0.724 19 50 0.50 18 25 32 50 0.75 31 37,5 45 100 0.25 17 25 34 100 0.50 40 50 61 100 0.75 67 75 84 A15 Table 7 contd sig. 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 n 500 P b+Pc b+C b> _A_> b e 0.670 0.697 0.655 0.685 0.649 0.669 0.619 0.636 0.611 0.631 0.608 0.626 0.594 0.610 0.592 0.607 0.589 0.607 0.619 0.641 0.611 0.631 0.608 0.626 0.584 0.597 0.581 0.594 0.578 0.591 0.567 0.579 0.566 0.577 0.565 0.576 0.25 106 63 66 73 77 84 87 129 133 140 145 152 157 196 202 206 212 216 223 126 131 140 145 155 160 256 262 272 279 288 295 388 397 402 410 415 424 125 144 500 0.50 228 250 272 500 0.75 356 375 394 1000 0.25 223 250 277 1000 0.50 469 500 531 1000 0.75 724 750 776 A16 Table 7 contd b+c b> p sig. b 2000 0.25 c t> c 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 462 252 259 272 279 292 299 499 509 531 541 563 573 768 780 788 800 807 820 378 386 402 410 426 434 760 772 788 800 816 828 1149 1163 1171 1186 1194 1209 0. 584 0. 599 0. 581 0. 594 0. 578 0. 591 0. 559 0. 570 0. 557 0. 567 0. ,556 0. ,565 0. ,547 0. ,555 ,547 0, .555 0. 0, .546 0, .554 0, .568 0, .579 .566 0, 0, .577 0, .565 0, .575 0 .548 0 .556 0 .547 0 .555 0 .546 0 .554 0 .539 0 .545 0 .538 0 .545 0 .538 0 .544 500 538 2000 0.50 938 1000 1062 2000 0.75 1462 1500 1538 3000 0.25 704 750 796 3000 0.50 1446 1500 1554 3000 0.75 2205 2250 2295 A17 Table 7 contd sig. VPc 4000 0.25 b+c b> be 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5000 0.75 5000 0.50 5000 0.25 4000 0.75 4000 0.50 946 503 513 531 541 559 569 1021 1035 1044 1058 1066 1080 1526 1543 1554 1571 1581 1598 629 640 660 671 690 701 1264 1279 1299 1315 1334 1350 1905 1923 1935 1954 1965 1985 0.559 0.569 0.557 0.567 0.556 0.565 0.541 0.548 0.541 0.548 0.540 0.547 0.534 0.539 0.533 0.539 0.533 0.539 0.553 0.562 0.552 0.560 0.550 0.558 0.537 0.543 0.536 0.543 0.536 0.542 0.530 0.535 0.530 0.535 0.530 0.535 1000 1054 1956 2000 2044 2946 3000 3054 1190 1250 1310 2431 2500 2569 3690 3750 3810 A18 Table 7 contd sig. ^b 10000 c b+c b> VPc/ 2415 1256 1271 1299 1315 1342 1358 2520 2541 2569 2591 2619 2641 3792 3819 3835 3862 3878 3905 0.537 0.543 0.536 0.543 0.536 0.542 0.526 0.531 0.526 0.530 0.526 0.530 0.521 0.525 0.521 0.525 0.521 0.525 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 5 1 0.25 2500 2585 10000 0.50 4902 5000 5098 10000 0.75 7415 7500 7585 * The three values given for b+c for each value of n represent the lower bound, expected value, and upper bound, for 95% confidence. n R b+c b n contingency table total probability that a document is retrieved by one strategy only number of documents retrieved by one strategy only number of documents retrieved by strategy A alone probability that a document is retrieved by strategy A given that it is retrieved by one strategy R Table 7 : The Squares method: number of documents to be retrieved by strategy A alone to support statement that A is better than B A19 Q 4** ^ *X" U. o o ro o '; .% 3 ' o • J £ 41 6 ;**§ Figure 1: The Pool method: illustration of Table 2 A20 Figure 2 m In d k Is I** Is •* 3 T- z 1T S v p Jvi Id fe 3 S 8 * o in KjoMfltnr*- OF- /K 1«T i°2 !O "tsT -! * \^ y cr cr y a 2 I* jc-> 1 3 O b F 1 « J* F UT U \2 .9 <£) wo si- o^ < N A22 Figure 2 contd UKCIS 182 REQUESTS 10715 RELEVANCE POSTINGS no0 rel0 no0 req. no. rel. no. req. 1 2 3 4 5 6 7 8 6 6 7 7 3 2 9 10 11 12 4 7 3 3 3 2 60 61 64 67 7/! 75 76 82 85 89 3 93 94 98 13 14 15 16 18 19 20 21 22 23 24 25 26 27 28 29 7 >0 3 6 4 3 2 1 2 2 99 100 103 105 108 109 112 4 2 3 4 3 2 2 1 3 2 1 31 34 35 36 37 40 42 3 2 1 2 2 2 1 1 1 2 1 1 113 123 125 127 130 136 140 149 150 151 154 159 177 180 183 198 231 233 266 319 408 43 44 45 46 47 48 49 52 53 54 56 57 50 3 2 1 2 2 1 413 510 511 554 Figure 2: Illustrations of test collection relevant document distributions A23 Appendix 2: Computing For convenience we repeat the statement of the problem which the program is trying to solve. Suppose that (considering recall) one has K relevant documents altogether in the pool and that at least n of these are required to be assessed. Suppose also that there are N documents in total in the pool. Then if a simple random sample of size S is taken for assessment, the probability that n of the relevant documents have been assessed is min(S,K) _ En <*> (g _ *> — (N) n (hypergeometrie distribution). Unfortunately the hypergeometrie distribution is quite messy to calculate, so quite often it is approximated to by the binomial distribution, this SK approximation holding whenever — > 25. Similarly it can be approximated to by the normal distribution, whenever the normal distribution is a suitable approximation to the biS nomial (that is, — < 0.1). N Unfortunately, neither condition is expected to hold in general, and so the approximations were not considered to be feasible. Now the magnitude of the numbers with which we are dealing is very large (N takes values from 100 - 10,000) and the IBM computer (a 370/165) cannot cope with N!: so we have to approximate N!. To do this Stirling's theorem is used, which says that N! ^ /(2TTN) ( ) - / N N e Taking logs one finds that logN! ^ 0.39909 + (n + ^)log 1Q N - 0.4342945N . (1) This expression is only accurate to 0(—) and so was only used for N > 30. Once this has been done the antilog is taken and the probability of ob- taining at least n relevant documents is calculated. The value of S is then varied until the largest value of S is obtained which has probability less than 0.95, and this value (incremented by 1) is the value tabulated in Table 3. A24 The fact that (1) is only accurate to 0(—•) is the reason for the N error mentioned in chapter B2; but as was said then, the probability of achieving the required number of relevant documents should still be greater than 0.93. Note that the order the data was read into the computer was such that K would remain unchanged while X ran through all its possibilities; N and. K would remain unchanged while X ran through e then K would change , and eventually N would change. (In the program n was denoted by X.) If N and K are the same and X has increased by 5, say, then one would only expect S to increase by a small amount. The 'IF1 statements at the beginning of the program (concerning A and B) are making use of this fact in order to speed up the search for the required sample size S. This improvement in technique is very important since to calculate the hypergeometric distribution is very costly in resources. Another way of speeding up the program would be to make use of the recursive property of the distribution. However the way the value of s jumps about makes this very awkward (although it could be used at the stage when S is always being decreased by 1) . . A lot of time was spent during the project on debugging and improving the program. Since it was proving to be so expensive it was decided to try and fit the normal distribution anyway. This was tried with data for which s had already been calculated by the hypergeometric distribution. Since there is some degree of inaccuracy already in the calculations (see earlier) it was decided that the normal approximation would be acceptable if the value of s it resulted in was within 5 of the previous value of S. The following table shows the results obtained: A2 5 N K X S N 0.320 0.880 0.270 0.670 0.090 0.310 SK N 8.00 22.00 13.50 33.50 6.75 23.25 S hyper normal 100 100 100 100 100 100 25 25 50 50 75 75 5 20 10 30 5 20 32 88 27 67 9 31 34 89 28 08 10 32 500 500 500 500 500 500 500 500 25 25 50 50 100 100 250 250 5 20 10 30 5 50 10 30 0.320 0.880 0.280 0.690 0.086 0.568 0.056 0.144 8.15 22.20 14.40 34.70 8.60 56.80 14.00 36.00 163 444 144 347 43 284 28 72 177 449 151 351 47 287 29 74 1000 1000 1000 1000 1000 1000 1000 1000 25 25 50 50 100 100 250 250 5 20 10 30 5 50 10 30 0.328 0.889 0.290 0.696 0.870 0.572 0.590 0.149 8.20 22.20 14.50 34.80 8.70 57.20 14.75 37.25 328 889 290 696 87 572 59 149 357 899 305 704 96 578 62 153 There are quite a few occasions when the difference is greater than 5 (indeed it can be as large as 28). Also there seems to be no pattern as to C Q.V when the fit is good and when it is not. Many times when — > 0.1 and —— < 25 there is only a difference of 1 between the two estimates (see for example the first row) . So no pattern could be found and the idea of the normal approximation was abandoned. This program could be used for any particular values of N, K and X (X is the required number of assessed relevant documents) by filing them in 16 format and putting a delimiter of 0 at the end. A listing of the program follows. A26 F O R T X C L G (PROGRAIM = %H? F T 0 5 F 0 0 1 = % H # ) I M P L I C I T REALMS < A ~ H ) , R E A L * 8 INTEGER NfKfX,S,TfV,AfD,G A= 0 5ECC=57 IF ( DABS(P-0o95f30 3oLTol oOE-6) GOTO 6 IF IF (PoGEo0o95D0) (SoNEoT) GOTO GOTO I 2 T=S 5=S+(N-5)/4 s=s+i 1 GOTO 2 3=S-1 P=PROB{N,K,X,S) IF (DABS(P-0o95O0)oLTolo0E-6) IF (PoLEo0o95D0) GOTO 6 GOTO 1 WRITE ( 6 , 9 ) N,KfX,S,P G=S A-N B= K GOTO 2 5 STOP GOTO 6 6 20 8 9 FORMAT (3 16) FORMAT { 1 9 H # OF DOCS I N P Q G L = f 1 6 , 1 3 H # R E T R I E V E D = , 1 5 , S 1 8 H H TO B E A S S E S S E O = , 1 3 , 1 3 H SAMPLE S I 2 E = , I 6 f 4 H P= , F 8 0 6 ) END DOUBLE P R E C I S I O N F U N C T I O N O R O B ( N , K f I , J ) I M P L I C I T REALMS ( A - H ) , REALMS { G - 2 ) INTEGER F PROB=0o0D0 W= DFLOAT(N) Y=DFi_OAT(K) Z=OFLOAT{ J ) I F ( J o G T o K ) GOTO 2 5 0 M= J GOTO 2 5 1 2 50 251 M=K D=Z-W+Y F=DINT(D) FF I ( F o G T o I ) I= GOTO 150 A27 150 7 300 DO 7 L = F , M C=DFLOAT(L) R = R I N L G G ( Y f C ) + H I N L O G ( W - Y , Z - C ) - B INLOG< W , 2 ) H=OASS{R) I F ( H o G r . o 5 0 o 0 D D ) GCTC 7 PR03=10o0D0**R+PRQ8 CONTINUE RETURN END DOUBLE PRECISION F U N C T I O N F L O G ( A ) IMPLICIT RtAL*H ( A - H ) f REAL*8 (0-2) • IF ( A o L T o 3 0 o 0 D 0 ) GOTO 75 Q=0o4342945D0 F L O G - O © 3 9 9 Q 9 D 0 + ( A + 0 * 5 C 0 )#DLOG!0(A >-G*A GOTO 76 75 IF ( A e G T o 2 o 0 0 0 ) GOTO 77 FLGG=DLGG10(A) GOTO 76 77 FLOG-FACTL(A) 76 RETURN END DOUBLE P R E C I S I O N FUNCTION D I N L O G f A , B ) IMPLICIT REALMS ( A - H ) , R E A L * 8 (0-2) IF (AoEQoB) G O T O 31 IF (BoNEoOoOOO) GOTO 33 31 DINL0G=0a000 GOTO 3 2 33 8INL0G=FL0G(A)-FLOG(A-B>-FLOG{3) 32 RETURN END DOUBLE P R E C I S I O N F U N C T I O N F A C T L ( A ) IMPLICIT REAL*8 (A-H) F=loODO L=DINT(A) 00 11 J-2 t L F=F*DFLOAT(J) 11 CONTINUE FACTL=DLOG10{F) RETURN 1 0C0 500 15 20 1 0 0 0 END 5 0 0 1 0 00 1 000 1000 1 000 1 000 1 000 1 000 1 0 00 2000 2000 2000 2000 200C 2 0 00 2000 2000 2000 20 0 0 2000 0 500 500 5 00 500 500 500 5 00 500 25 25 50 50 50 50 50 50 50 50 50 0 25 30 35 40 45 50 55 60 15 20 5 10 15 20 25 30 35 40 45 0