CHAPTER 5

Programs for setting up document test collections

The following files are used in the system (and are not necessarily disc resident): MFP1.S.PR0G MFP1.S.RUN MFP1.S.LM0D MFP1.S.TEXTS a a a a pds of program texts. pds of phoenix commands to run each program. load module library of programs. pds of data files used as defaults by the programs.

To invoke the members of MFP1.S.RUN as a private library, type LIBRARY MFP1.S.LMOD:COM0 Commands in the library work from a FROM to a TO file, and the defaults for FROM and TO are %C and %0 respectively. TO is set up as /LARGE. (This is phoenix jargon, phoenix being the command language in use on the IBM 370/165 at Cambridge.) STORE should be adjusted upwards from the usual default of 120 for online work when substantial sorting or data storage is required by the programs. When the programs do run out of store, the messages give a good indication of how much was really needed.

-68-

e.g.

DECHAR FROM .VASWANI.SRCE1 DECHAR FROM &S TO &T

This takes the original text of a standard test collection (FROM), and replaces all non-letters in the document abstracts by space. Lower case letters are forced to upper case. The syntax of the input is validated, in that document numbers must be 1,2,3f ... in turn, documents must be terminated with /<newline> and the whole collection with /<newlines><endoftext>

With the FROM file: 1 Compact memories have flexible capacities. A digital data storage system with capacity up to 24000 bits and random and or sequential access is described. / 2 An electronic analogue computer for solving systems of linear equations. Mathematical derivation of the operating principle and stability conditions for a computer consisting of amplifiers. /

00

-69-

Satellite observations of electrons artificially injected into the geomagnetic field. The geomagnetically trapped electrons resulting from the high altitude nuclear detonations of the ARGUS experiment have been observed on four radiation detectors in satellite explorer. The measurements for several satellite passes through the ARGUS shells are described and the significance of the results is summarized. / / DECHAR produces the TO file: 1 COMPACT MEMORIES HAVE FLEXIBLE CAPACITIES A DIGITAL DATA STORAGE SYSTEM WITH CAPACITY UP TO BITS AND RANDOM AND OR SEQUENTIAL ACCESS IS DESCRIBED / 2 AN ELECTRONIC ANALOGUE COMPUTER FOR SOLVING SYSTEMS OF LINEAR EQUATIONS MATHEMATICAL DERIVATION OF THE OPERATING PRINCIPLE AND STABILITY CONDITIONS FOR A COMPUTER CONSISTING OF AMPLIFIERS /

100 SATELLITE OBSERVATIONS OF ELECTRONS ARTIFICIALLY INJECTED INTO THE GEOMAGNETIC FIELD THE GEOMAGNETICALLY TRAPPED ELECTRONS RESULTING FROM THE HIGH ALTITUDE NUCLEAR DETONATIONS OF THE ARGUS EXPERIMENT HAVE BEEN OBSERVED ON FOUR RADIATION DETECTORS IN SATELLITE EXPLORER THE MEASUREMENTS FOR SEVERAL SATELLITE PASSES THROUGH THE ARGUS SHELLS ARE DESCRIBED AND THE SIGNIFICANCE OF THE RESULTS IS SUMMARIZED / /

-70-

e.g.

DESTOP WITH .OWN:STOPLIST DESTOP FROM &T TO .VAS.AO

This removes from the text of FROM the words supplied in a given stop list (WITH), together with words consisting of only one or two letters. The stop list may contain one or two letter words, but these are redundant. (Here and below words are defined as upper case letter sequences bounded by non-letters.) With the FROM file: 1 COMPACT MEMORIES HAVE FLEXIBLE CAPACITIES A DIGITAL DATA STORAGE SYSTEM WITH CAPACITY UP TO BITS AND RANDOM AND OR SEQUENTIAL ACCESS IS DESCRIBED / 2 AN ELECTRONIC ANALOGUE COMPUTER FOR SOLVING SYSTEMS OF LINEAR EQUATIONS MATHEMATICAL DERIVATION OF THE OPERATING PRINCIPLE AND STABILITY CONDITIONS FOR A COMPUTER CONSISTING OF AMPLIFIERS /

100 SATELLITE OBSERVATIONS OF ELECTRONS ARTIFICIALLY INJECTED INTO THE GEOMAGNETIC FIELD THE GEOMAGNETICALLY TRAPPED ELECTRONS RESULTING FROM THE HIGH ALTITUDE NUCLEAR DETONATIONS OF THE ARGUS EXPERIMENT HAVE BEEN OBSERVED ON FOUR RADIATION DETECTORS IN SATELLITE EXPLORER THE MEASUREMENTS FOR SEVERAL SATELLITE PASSES THROUGH THE ARGUS SHELLS ARE DESCRIBED AND THE SIGNIFICANCE OF THE RESULTS IS SUMMARIZED / / and the WITH file:

A, ABOUT, ABOVE, ACROSS, AFTER, AFTERWARDS, AGAIN AGAINST, ALL, ALMOST, ALONE, ALONG, ALREADY, ALSO ALTHOUGH, ALWAYS, AMONG, AMONGST, AN, AND, ANOTHER ANY, ANYHOW, ANYONE, ANYTHING, ANYWHERE, ARE, AROUND AS, AT, BE, BECAME, BECAUSE, BECOME, BECOMES BECOMING, BEEN, BEFORE, BEFOREHAND, BEHIND, BEING, BELOW BESIDE, BESIDES, BETWEEN, BEYOND, BOTH, BUT, BY CAN, CANNOT, CO, COULD, DOWN, DURING, EACH EG, EITHER, ELSE, ELSEWHERE, ENOUGH, ETC, EVEN EVER, EVERY, EVERYONE, EVERYTHING, EVERYWHERE, EXCEPT, FEW FIRST, FOR, FORMER, FORMERLY, FROM, FURTHER, HAD HAS, HAVE, HE, HENCE, HER, HERE, HEREAFTER HEREBY, HEREIN, HEREUPON, HERS, HERSELF, HIM, HIMSELF

DESTOP produces as TO file: 1 COMPACT MEMORIES FLEXIBLE CAPACITIES DIGITAL DATA STORAGE SYSTEM CAPACITY BITS RANDOM SEQUENTIAL ACCESS DESCRIBED / 2 ELECTRONIC ANALOGUE COMPUTER SOLVING SYSTEMS LINEAR EQUATIONS MATHEMATICAL DERIVATION OPERATING PRINCIPLE STABILITY CONDITIONS COMPUTER CONSISTING AMPLIFIERS /

100 SATELLITE OBSERVATIONS ELECTRONS ARTIFICIALLY INJECTED GEOMAGNETIC FIELD GEOMAGNETICALLY TRAPPED ELECTRONS RESULTING HIGH ALTITUDE NUCLEAR DETONATIONS ARGUS EXPERIMENT OBSERVED FOUR RADIATION DETECTORS SATELLITE EXPLORER MEASUREMENTS SATELLITE PASSES ARGUS SHELLS

-72-

DESCRIBED SIGNIFICANCE RESULTS SUMMARIZED / / The default value of WITH is MFP1.S.TEXTS:STOPLIST

VOCAB

e.g.

VOCAB FROM .VAS.AO TO .VAS.VOCAB STORE MOO VOCAB TO &VOC

This sorts the words in FROM and sends to TO a vocabulary list in the form: ABSORPTION ACCELERATION ACCESS ACCOMPANIED ACCOUNT ACCURACY ACHIEVED ACOUSTIC ADDER ADJUSTABLE ADVERSELY AFFECT AFFECTING AGREEMENT AIR ALLEN ALTITUDE AMERICAN

-73-

AMMONIA AMPLIFICATION

STEM

e.g.

STEM FROM &VOC TO &SVOC

STEM is a simple suffix stripping program. The output has the form: ABSORPT ACCELER ACCESS ACCOMPAN ACCOUNT ACCURAC ACHIEV ACOUST ADDER ADJUST ADVERS AFFECT AFFECT AGREEMENT ABSORPTION ACCELERATION ACCESS ACCOMPANIED ACCOUNT ACCURACY ACHIEVED ACOUSTIC ADDER ADJUSTABLE ADVERSELY AFFECT AFFECTING AGREEMENT

AIR
ALLEN ALTITUD AMERICAN AMMONIA AMPLIF AMPLIF AMPLIF

AIR
ALLEN ALTITUDE AMERICAN AMMONIA AMPLIFICATION AMPLIFIER AMPLIFIERS

-74-

AMPLITUD ANALOGU ANALYS ANALYS ANALYS

AMPLITUDE ANALOGUE ANALYSED ANALYSER ANALYSIS

Each word in the FROM file is output on a separate line preceded by a character string derived from the word, which is used to bring words together into conflation groups. The character string is printed in a field of width 24, and has a maximum size of 22 characters. (If necessary, characters are chopped out of the middle to bring it down to size.) The TO file should be sorted to bring the words into their conflation groups, e.g. STEM FROM &VOC SM

DICMAT

e.g.

DESLASH FROM .VASVOC.TERMS TO &U DICMAT FROM .VAS.AO TO .VAS.BO WITH &U DICMAT WITH .VTERMS STORE 250

The WITH file (the TO output from TERMNOS) consists of every word in the text of the FROM file, arranged in alphabetical order, and followed by an integer. DICMAT replaces each word in the FROM file by its corresponding integer, and sends the result to TO. Words not in WITH but in FROM are printed out as not being in the dictionary, but are otherwise ignored. With the FROM file:

-75-

1 COMPACT MEMORIES FLEXIBLE CAPACITIES DIGITAL DATA STORAGE SYSTEM CAPACITY BITS RANDOM SEQUENTIAL ACCESS DESCRIBED / 2 ELECTRONIC ANALOGUE COMPUTER SOLVING SYSTEMS LINEAR EQUATIONS MATHEMATICAL DERIVATION OPERATING PRINCIPLE STABILITY CONDITIONS COMPUTER CONSISTING AMPLIFIERS / •••• 100 SATELLITE OBSERVATIONS ELECTRONS ARTIFICIALLY INJECTED GEOMAGNETIC FIELD GEOMAGNETICALLY TRAPPED ELECTRONS RESULTING HIGH ALTITUDE NUCLEAR DETONATIONS ARGUS EXPERIMENT OBSERVED FOUR RADIATION DETECTORS SATELLITE EXPLORER MEASUREMENTS SATELLITE PASSES ARGUS SHELLS DESCRIBED SIGNIFICANCE RESULTS SUMMARIZED / / and the WITH file: ABSORPTION 1 ACCELERATION 2 ACCESS 3 ACCOMPANIED 4 ACCOUNT 5 ACCURACY 6 ACHIEVED 7 ACOUSTIC 8 ADDER 9 ADJUSTABLE 10 ADVERSELY 11

-76-

AFFECT 12 AFFECTING 12 AGREEMENT 13 AIR 14 ALLEN 15 ALTITUDE 16 AMERICAN 17 AMMONIA 18 AMPLIFICATION ....

19

DICMAT produces as TO file: 1 105 424 268 82 189 161 687 708 82 63 566 630 3 174

220 20 113 664 708 388 230 415 173 479 538 683 117 113 128 19 /

100 613 294 317 471 422 174 /

471 220 37 340 263 294 737 220 600 16 468 180 36 243 280 561 178 613 245 613 500 36 637 646 600 699

e.g.

BSORT OPT SF BSORT FROM .VAS.BO TO .VAS.B1 OPT SN

This takes a file in ab-form (FROM), and adjusts the b's for each a. 'ab-form' means a b a b • a b /

b ... b / b ... b / •• b ... b /

where the a f s and b's are integers. The OPT parameter may contain S, Ff B and N. S N F B causes causes causes in the causes single the b's to be sorted in ascending order. duplicate b's (i.e. a b equal to the previous b) to be discarded. a list of b's to be replaced by a single b with frequency count output. the output to be "brief", that is, multiple spaces are reduced to a space. This is recommended for very large data collections.

The N and F options take effect after sorting. N and F together cause all the frequency counts to be 1. So if the FROM file is:

-78-

1 105 424 268 82 189 161 687 708 82 63 566 630 3 174
/

2 220 20 113 664 708 388 230 415 173 479 538 683 117 113 128 19
/

100 613 294 317 471 422 174
/

471 220 37 340 263 294 737 220 600 16 468 180 36 243 280 561 178 613 245 613 500 36 637 646 600 699
/

OPT=S produces the output: 3 566 19 415 63 630 20 479 82 687 113 538 82 708 / 113 664

105
117 683

161
128 708

100

16 245 471

36 263 500

36 280 561

37 294 600

174 294 600

178 317 613

-79-

699

737 /

OPT=SF produces the output: 3 174 630 19 173 479 1 1 1 1 1 1 63 189 687 20 220 538 82 268 708 113 230 664 2 1 105 424 117 388 683 161 566

1/
2 1 1 128 415 708 1 1 1 /

100

16 180 280 468 613

<AJ CM

1 1 1 1 3

36 220 294 471 637

2 2 1

37 243 317 500 646

1 ! l 1 I

174 245 340 561 699

1 I

178 263

422 600
I
737

1 1 1 2
1 /

/ and 0PT=SN produces the output 1
3 630 19 479 63 687 20 538 82 708 /
113 664

105

161

174

189 220

268 230

424

566

117 683

128 708 /

173

388

415

100

16

36

37

174

178

180

220

243

245

263

-80-

280 613

294 637

317 646

340 699

422 737 /

468

471

500

561

600

RANK RANK FROM .VAS.B1 TO .VAS.RANK RANK STORE 250

e.g,

This takes a file in ab-form (FROM) and sorts the different b's into descending order of frequency, producing on TO the list of b's. TO has the typical form: 752 600 645 460 427 175 238 686 230 499 720 173 404 550 562 212 294 688 244 519 283 243 497 597 591 311 412 726 267 528 19 422 567 708 683 613 453 746 275 538 21 296 663 55 734 649 484 37 285 541 95 471 127 87 771 759 507 192 289 571 263 473 179 194 93 29 525 195 317 595 174 488 216 197 106 78 539 203 403 602 220 586 356 388 117 118 575 217 451 607 293 642 426 398 161 218 666 222 479 619

-81-

e.g.

BMAP FROM .VAS.BO TO .VAS.CO WITH .VAS.RANK BMAP TO &U WITH &RANK STORE 200

This takes a file in ab-form (FROM) and a separate ranking list of the b's (WITH) and produces on TO a mapped version of FROM in which each b is replaced by f(b), where b occurs as the f(b)-th number in the WITH file. The purpose of this operation is to renumber the b's so that 1 is the most common term, 2 the next most common, and so on. If FROM has the form: 1 105 424 268 82 189 161 687 708 82 63 566 630 3 174
/

2 220 20 113 664 708 388 230 415 173 479 538 683 117 113 128 19
/

100 613 294 317 471 422 174
/

471 220 37 340 263 294 737 220 600 16 468 180 36 243 280 561 178 613 245 613 500 36 637 646 600 699

/

-82-

and WITH t h e form: ' 752 600 645 460 427 175 238 686 230 499 .... 720 173 404 550 562 212 294 688 244 519 283 243 497 597 591 311 412 726 267 528 19 422 567 708 683 613 453 746 275 538 21 296 663 55 734 649 484 37 285 541 95 471 127 87 771 759 507 192 289 571 263 473 179 194 93 29 525 195 317 595 174 488 216 197 106 78 539 203 403 602 220 586 356 388 117 118 575 217 451 607 293 642 426 398 161 218 666 222 479 619

TO h. the form : as 1 396 291 9 94 574 694 344 44 477 333 116 49 222 388 8 / 169 116 34 205 50 728 81 34 388 12 372

2

39 4 /

147

90

•••• 100 54 11 436 11 16 9 341 87 54 465 735 / 75 601 14 522 437 54 62 353 278 7 13 353 62 16 699 319 137 8 9 163 702

/

DEFREQ

e.g.

DEFREQ FROM &A TO &B

This takes a file in abb-form, i.e. in ab-form but with the b's coming in pairs, and strips out the second of each pair of b's. So with the FROM file: 3 174 630 19 173 479 63 189 687 20 220 538 82 268 708 113 230 664 2 1 1 / 2 1 1 105 424 161 566

117 388 683

128 415 708

1 1 1 /

C\J CM

100

16 180 280 468 613

1 1 1 1 3

36 220 294 471 637

2
CM

1

37 243 317 500 646

I l I I l

174 245 340 561 699

1 1 1 1 1

178 263 422 600 737

1 1 1 2
1 /

DEFREQ produces as TO file: 3 630 19 479 63 687 20 538 82 708 / 113 664

105

161

174

189 220

268 230

424

566

117 683

128 708 /

173

388

415

-84-

00

16 280 613

36 294 637

37 317 646

174 340 699

178 422 737 /

180 468

220 471

243 500

245 561

263 600

INVERT

e.g.

INVERT FROM .VAS.CO TO .VAS.XO STORE 350 INVERT STORE 400 OPT F200T300

This takes a file in ab-form (FROM) and produces an inverted file in ab-form (TO) in which each afb pair of FROM corresponds to a bfa pair in TO. So w 1th the FROM file: 1 2 396 291 9 94 574 694 344 44 477 333 116 49 388 222 8 / 169 116 34 205 50 728 34

388
12

372 90

39 4 /

81

147

100

54 11 36 11

16 87 54 735 /

9 341 465

75 601 14

522 437 54

62 353 278

7 13 353

62 16 699

319 137 8

9 163 702

-85-

INVERT gives a TO file:

5 46 8 45
94 /

6 47
11 53

7 50
13 55

9 51
16 56

11 61 18 60

18 68 23 63

20 72 24 63

29 89 28 63

35 93 33 65

38
99 /

42 67

783 /

84

84 /

A limitation of INVERT is that it can only cope with one storeful of material at a time. To get over this, the range of values of b may be restricted by settings in the OPT parameter. If the OPT string is FxTy (F and T stand for 'from1 and f t o f ) , the TO file only contains those inverted b f s for which x <= b < y. If y is set, the TO file will not contain the terminating solidus character, which means that a valid inverted file can be set up by concatenations. e.g. DELETE &INV CURRENT .VAS.CO INVERT TO &INV/M0D STORE 400 OPT T300 INVERT TO &INV/M0D STORE 400 OPT F300T600 INVERT TO &INV/M0D STORE 400 OPT F600

-86-

INVERT2

e.g.

INVERT2 FROM .VAS.CO TO .VAS.XO STORE 350

This has the same spec as INVERT, but does not have the limitation of only one storeful at a time, and so is to be preferred for large data collections. It uses the IBM sort-merge utility, and involves a number of passes over the data. INVERT2 has no OPT parameter.

IMAP

e.g.

IMAP FROM &A TO &B WITH &RANK

This is like BMAP (q.v.) but is intended for use with output AND of TERMNOS. Each number b at the beginning of the line in FROM is mapped to f(b), where b occurs as the f(b)-th number in the WITH file. The numbers in TO are right justified in a field of width 6, e.g. 331 332 333 334 335 181 182 336 337 338 339 183 184 340 ABSORPTION ACCELERATION ACCESS ACCOMPANIED ACCOUNT ACCURACY ACHIEVED ACOUSTIC ADDER ADJUSTABLE ADVERSELY AFFECT AGREEMENT AIR

-87-

185 341 342 343 4 344

ALLEN ALTITUDE AMERICAN AMMONIA AMPLIFICATION ANALOGUE

SM
e.g. SM FROM &B OPT 1,6

This runs the IBM sort-merge utility and sorts the records of FROM to the file TO, ordering them by the EBCDIC collating order of the characters in columns m to n inclusive, where m,n is the value of the OPT parameter. (By default m=1 and n=79.) So with 0PT=1,6 and FROM in the form: 331 332 333 334 335 181 182 336 337 338 339 183 184 340 ABSORPTION ACCELERATION ACCESS ACCOMPANIED ACCOUNT ACCURACY ACHIEVED ACOUSTIC ADDER ADJUSTABLE ADVERSELY AFFECT AGREEMENT AIR

-88-

185 311 342 343 4 344

ALLEN ALTITUDE AMERICAN AMMONIA AMPLIFICATION ANALOGUE

TO would have the form:

1 USE 2 THEORETICAL 3 FREQUENCIES 4 AMPLIFICATION 5 ANALYSED 6 CIRCUIT

7 FIELD
8 DESCRIBED 9 ELECTRON 10 GENERAL 1 RESULT 1 12 DERIVATION 13 EXPERIMENT 14 MEASURED 15 GIVEN 16 OBSERVATION 17 OBTAIN 18 OSCILLATION 19 RELATED 20 SHOWN
Before sorting, FROM is converted to a file with FB format and LRECL=80. This means that the records to be sorted must not exceed 80 characters in length. The output from sort-merge is then filed to TO, so that TO does not have to be FB.

-89-

TERMNOS

e.g.

TERMNOS FROM &S TO &T TERMNOS TO &T AND &U

The FROM file should be the sorted output from STEM. The TO file consists of the words of vocabulary followed by an integer which gives a count of the conflation class. The AND file, if present, consists of the first word out of each conflation class preceded by its conflation class number. By default AND goes to %DUMMY. So with the FROM file in the form: ABSORPT ACCELER ACCESS ACCOMPAN ACCOUNT ACCURAC ACHIEV ACOUST ADDER ADJUST ADVERS AFFECT AFFECT AGREEMENT ABSORPTION ACCELERATION ACCESS ACCOMPANIED ACCOUNT ACCURACY ACHIEVED ACOUSTIC ADDER ADJUSTABLE ADVERSELY AFFECT AFFECTING AGREEMENT

AIR
ALLEN ALTITUD AMERICAN AMMONIA AMPLIF AMPLIF

AIR
ALLEN ALTITUDE AMERICAN AMMONIA AMPLIFICATION AMPLIFIER

-90-

AMPLIF AMPLITUD ANALOGU ANALYS ANALYS ANALYS

AMPLIFIERS AMPLITUDE ANALOGUE ANALYSED ANALYSER ANALYSIS

TO would have the form: ABSORPTION 1 ACCELERATION 2 ACCESS 3 ACCOMPANIED 4 ACCOUNT 5 ACCURACY 6 ACHIEVED 7 ACOUSTIC 8 ADDER 9 ADJUSTABLE 10 ADVERSELY 11 AFFECT 12 AFFECTING 12 AGREEMENT 13 AIR 14 ALLEN 15 ALTITUDE 16 AMERICAN 17 AMMONIA 18 AMPLIFICATION 19 AMPLIFIER 19 AMPLIFIERS 19 AMPLITUDE 20 ANALOGUE 21 ANALYSED 22 ANALYSER 22 ANALYSIS 22

and AND would have the form: 1 ABSORPTION 2 ACCELERATION 3 ACCESS 4 ACCOMPANIED 5 ACCOUNT 6 ACCURACY 7 ACHIEVED 8 ACOUSTIC 9 ADDER 10 ADJUSTABLE 11 ADVERSELY 12 AFFECT 13 AGREEMENT 14 AIR 15 ALLEN 16 ALTITUDE 17 AMERICAN 18 AMMONIA 19 AMPLIFICATION 20 AMPLITUDE 21 ANALOGUE 22 ANALYSED

Illustrative example - setting up the VASWANI test collection

Setting up the collection itself was done as follows. The numbers in brackets indicate the track sizes of the created output files. The store size is 120K unless otherwise indicated.

DECHAR FROM .TEXT.N6 TO .TEXT.NORMED (290) DESTOP FROM .TEXT.NORMED TO .TEXT.STOPPED (229) 479163 words read 99115 stopwords removed 104358 short words removed VOCAB FROM .TEXT.STOPPED TO .VOCAB (13) No of records read 73524 No of tokens 275690 No of types 11712 STEM FROM .VOCAB 11712 words read SM TERMNOS TO .TERMNOS (18) AND .TEMP.TERMDICT (11) 11712 different words read 7491 different terms read

58 sees 25 sees

2 min 19 sees, 400K

9 sees 9 sees, 150K 4 sees

DICMAT FROM .TEXT.STOPPED WITH .TERMNOS TO .TEMPDOCS.AB (143) 1 min 42 sees, 300K 275690 words read 440 (machine) words of unused workspace BSORT FROM .TEMPDOCS.AB OPT SN 11429 units read RANK TO .TERMRANK (5) Max term = 7491 Max frequency = 2511 1 min 2 sees 23 sees, 200K

-93-

11429 docs read 7491 terms output IMAP FROM .TEMP.TERMDICT WITH .TERMRANK 7491 lines read SM TO .TERMDICT (11) BMAP FROM .TEMPDOCS.AB WITH .TERMRANK TO .DOCS.AB (197) 7491 terms read from WITH file Maximum term was 7491 11429 docs read BSORT FROM .DOCS.AB TO .DOCS.ABS (109) OPT BS 11429 units read BSORT FROM .DOCS.ABS TO .DOCS.ABF (141) OPT BF 11429 units read DEFREQ FROM .DOCS.ABF TO .DOCS.ABN (165) 11429 docs read INVERT2 FROM .DOCS.AB TO .TERMS.ABS (137) BSORT FROM .TERMS.ABS TO .TERMS.ABF (163) OPT BF 7491 units read DEFREQ FROM .TERMS.ABF TO .TERMS.ABN (163) 7491 docs read 4 sees, 200K 6 sees, 200K 48 sees, 130K

57 sees

42 sees 40 sees

3 min 24 sees, 400K 53 sees 51 sees

Setting up the queries was done as follows. (The resources consumed here were small.) DESTOP FROM .QUERIES.TEXT 1086 words read 146 stopwords removed 208 short words removed

-94-

DICMAT WITH .TERMNOS FONT not in diet TRANSISTORISED not in diet SEND not in diet SEND not in diet DISCS not in diet INTERESTED not in diet NUMERIC not in diet OPTIMISING not in diet PRETREATMENT not in diet WISH not in diet WISH not in diet TRANSISTORISED not in diet RESTING not in diet 732 words read 440 (machine) words of unused workspace (.QUERIES.TEXT was then edited so that DISCS INTERESTED NUMERIC RESTING -> -> -> -> DISC* INTERESTED NUMERICAL REST*** .TERMNOS, and

300K

the words DISC, INTEREST, NUMERI
DICMAT was rerun.) DICMAT WITH .TERMNOS FONT not in diet TRANSISTORISED not in diet SEND not in diet SEND not in diet OPTIMISING not in diet PRETREATMENT not in diet WISH not in diet WISH not in diet TRANSISTORISED not in diet

300K

-95-

732 words read 440 (machine) words of unused workspace BMAP WITH .TERMRANK 7491 terms read from WITH file Maximum term was 7491 100 docs read BSORT TO .QUERIES.AB OPT SN 100 units read 130K

Appendix A. Other useful programs.

SCALE

e.g. SCALE TO &B AND &SCALE This file file This takes a file in ab-form (FROM) and produces on TO a version of the FROM with the a ! s forming a simple ascending sequence 1,2,3, ... The AND (which must be present) contains a simple list of the original a f s. file can be used for mapping b f s with the BMAP program, e.g.

SCALE FROM .TERMS.A1 TO .TERMS.B1 AND &SCALE BMAP FROM .D0CS.A1 TO .D0CS.B1 WITH &SCALE STORE 190 BMAP FROM .QS.A1 TO .QS.B1 WITH &SCALE STORE 190

-96-

RELS

e.g. RELS FROM .A TO .B This puts the text form of a test collection (the output of DECHAR) into 3-tuple relation form for input to the CODD database. The columns are: )CNO WORDNO WORD where WORD is the WORDNOth word in the document with number DOCNO.

-97-