A Study of Information Seeking and Retrieving - C.pdf

of 20
30 views
PDF
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Document Description
A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* Tefko Saracevic School of Communication, Information and Library Studies, Rutgers, The State University of New Jersey, 4 Huntington St. New Brunswick, N. J. 08903 Paul Kantor Tantalus Inc. and Department of Operations Research, Weatherhead School of Management, Case Western Reserve University, Cleveland, Ohio 44706 The objectives of the study were to conduct a series of observations and experiments under as r
Document Share
Document Tags
Document Transcript
  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* Tefko Saracevic School of Communication Information and Library Studies Rutgers The State University of New Jersey 4 Huntington St. New Brunswick N. J. 08903 Paul Kantor Tantalus Inc. and Department of Operations Research Weatherhead School of Management Case Western Reserve University Cleveland Ohio 44706 The objectives of the study were to conduct a series of observations and experiments under as real-life situ- ation as possible related o: (1) user context of questions in information retrieval; (2) the structure and classi- fication of questions; (3) cognitive traits and decision making of searchers; and (4) different searches of the same question. The study is presented in three parts: Part I presents the background of the study and de- scribes the models, measures, methods, procedures and statistical analyses used. Part II is devoted to results related to users, questions and effectiveness measures, and Part III to results related to searchers, searches and overlap studies. A concluding summary of all results is presented in Part III. Introduction This is a third and concluding article on a study whose aim was to contribute to a formal, scientific characterization of the elements involved in information searching and re- trieving, particularly in relation to the cognitive aspects and human decisions and interactions involved. The first part [l] presents the models, methods, measures, and procedures involved, together with a review of related works and all the background references. The second part [2] presents results on variables related to users, questions, and effectiveness measures. This third part is devoted to variables related to searchers and searching, and to overlap studies; it also con- tains conclusions for the study as a whole. The second part contains as an introduction a summary of the objectives and procedures, so that a reader interested in results alone can proceed reading Parts II and III on their Work done under the NSF grant IST85-05411 and a DIALOG grant for search time. Received July 9, 1987; accepted December 11, 1987. 0 1988 by John Wiley & Sons, Inc. own. In addition, Part II includes an Appendix listing the questions used in the study together with data on items retrieved and evaluated. A Final Report [3] deposited with NTIS and ERIC describes all aspects of the study in great detail and includes a series of Appendices containing full question statements, “raw” data, forms used, and flowcharts of procedures. Since Part II contains a summary of objectives and pro- cedures we shall proceed directly with a presentation of the results. Part II includes Tables 1 to 18 and the table number- ing continues here starting with Table 19. Searchers What Were the Results on Cognitive Tests? The 39 searchers (36 outside and 3 project searchers) were tested on three cognitive tests (for references on these tests see Part I): (1) Remote Associates Test RAT): claims to be a test of semantic or word association. The scores can vary from 0 to 30. 2) Symbolic Reasoning Test SRT): claims to be a test of the ability to make deductive inference based on symbols. The scores can vary from 0 to 30. 3) Learning Style Inventory LSI): claims to determine an individual’s preference for one of the postulated learning styles representing a characteristic method for acquiring and using information. The LSI yields six f the four learning modes (Con- ciete Expekenbe (CE , Reflective Observation (RO), sdores: gne foq eaph s Abstract Conceptualization (AC) and Active Experi- mentation (AE)) and two for combination scores deter- mining learning styles: AC - CE (the extent to which an individual emphasizes abstractness over con- creteness in learning) and AE - RO (emphasizes ac- tion over reflection). The four learning mode scores JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 39(3):197-216, 1988 CCC 0002-8231/881030197-20 04.00  TABLE 19. Searchers’ scores for cognitive tests (N searchers = 39; for explanation of abbreviations see text). Test Mean Standard Deviation Min. Max. posed to not relevant) in searches done by searchers who scored high (above mean) on a given cognitive test as op- posed to searchers who scored low (below mean)? RAT 13.03 5.2 5 28 SRT 10.61 5.0 3 21 LSI: CE 24.70 7.3 13 42 RO 27.90 7.4 12 44 AC 33.55 9.4 16 48 AE 33.68 7.8 15 46 AC-CE 8.60 15.1 -20 35 AE-RO 5.73 13.3 -29 31 can vary from 12 to 48 and the two combination scores from-36 to +36. Table 20 provides the answers; it summarizes the re- lationship between relevance odds and a number of charac- teristics of searchers. The structure of this and all other tables on relevance odds was described at length with the presentation of Table 5 in Part II. Briefly, the columns start- ing from the left show the following: the cut point or mean that divides the high and low scores for the variable; the cross product ratio calculated from the 2 X 2 contingency table; the corresponding logarithm; standard error ( + or - ); t-value used for test of significance; and the indication if the relation was significant at 95 (i.e., yes, when the mag- nitude of the t-value was above 2). In addition, searchers answered a question on their fre- quency of online searching on DIALOG (the service used in the study). Table 19 provides the means, standard deviations, and ranges for the three cognitive tests taken by searchers. The distributions (not shown) were not normal: for RAT there was a peak at 10 and another at 18, for SRT a peak at 5 and another at 14; for LSI talking about peaks is not appropriate but by plotting the AC - CE scores against AE - RO scores in a graph the respondents are placed in categories as to learning styles: 16 (41 ) of searchers were placed in the category called “converger,” 2 (5 ) in category “diverger,” 10 (26 ) in category “assimilator,” 9 (23 ) in category “accommodator,” and 2 (5 ) were “indeterminate.” Scores from the 36 outside searchers only were used in this analysis, as well as the next one on precision and recall odds, because they did one search each, while the project searchers did more than one, thus with project searchers a learning factor may have been present. The results indicate that items retrieved in searches done by searchers who: As to the frequency of searching DIALOG, 12 (3 1 ) of searchers reported searching DIALOG daily; 13 (33 ) twice a week; 3 (8 ) once a week; 2 (5 ) twice a month; and 9 (23 ) less than twice a month. Thus, 72 of search- ers used DIALOG at least once a week. scored high (above mean) on a word association test (RAT) were 60 (or by a factor of 1.60) more likely to be relevant or partially relevant as opposed to not relevant; indicated that they preferred a Concrete Experience mode of learning were 29 (I-0.71) less likely to be relevant or partially relevant; indicated that they preferred an Abstract Concep- tualization mode of learning style were 41 (or 1.41 times) more likely to be relevant or partially relevant; indicated that they emphasized abstractness over concrete- ness (AC-CE) as their learning style were 41 (or 1.41 times) more likely to be relevant or partially relevant. What Was the Relationship between Searchers Character- istics and Relevance Odds? In general, how a searcher scores on Remote Associates Test and what preference on learning he or she indicated affected relevance odds. Scores on Symbolic Reasoning Test and frequency of DIALOG searching did not have a significant effect. The following question was asked: What were the odds The question of what these (and other) cognitive tests that retrieved items be relevant or partially relevant (as op- really represent, as well as the validity of their claims are TABLE 20. Summary of the relation between searchers’ characteristics and the odds that a retrieved item be relevant or partially relevant (Done for outside searchers only) (N outside searchers = 36; N outside searches = 200; N items retrieved = 4841; statistical significance at 95 ; for abbreviations see text). Searchers Characteristics cut Stand. Point Odds Log Error t- Stat. (Mean) Ratio Odds +/- Value Signif. Frequency of searching RAT (0 to 30) SRT (0 to 30) LSI: CE (12 to 48) RO (12 to 48) AC (12 to 48) AE (12 to 48) AC-CE (-36 to +36) AE-RO (-36 to +36) 3.4 1.10 0.10 0.07 1.48 No 13.03 1.60 0.47 0.06 7.84 Yes 10.61 0.97 -0.03 0.06 -0.53 No 24.70 0.71 -0.34 0.06 -5.82 Yes 27.90 0.96 -0.04 0.06 -0.69 No 33.55 1.29 0.25 0.06 4.36 Yes 33.68 1.11 0.10 0.06 1.73 No 8.60 1.41 0.34 0.06 5.88 Yes 5.73 1.01 0.01 0.06 0.24 No 198 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-May 1988  issues debated in psychology and other fields interested in human testing. We can express our doubts, but we cannot assert or question the validity of tests used here either way. Thus, taking the test claims strictly on their face value, it seems that searchers who show higher abilities or skills in language expressions, particularly word association, and/or searchers who lean toward abstractness in learning are more successful as to retrieval of relevant items. Surprisingly, searchers who score high on symbolic or logical reasoning had no significant effect on relevance odds; however, if we took all the searchers together (outside plus project searchers as we did in the analyses presented in the Final Report [3] and not only the outside searchers as we did here) then we observed a small negative correlation between high scores on Symbolic Reasoning Test and relevance odds. While frequency of DIALOG searching had no signifi- cant effect on relevance odds, we have to underscore the fact that all searchers in the study were experienced and that no inexperienced searchers were included, thus this finding is not unexpected. In practical terms, the findings on cognitive tests suggest that sharpening the semantic competencies in general and in a subject in particular is one of the more useful activities leading to possible improvement in searching, be that by human intermediaries or intelligent interfaces. What Was the Relationship between Searchers Character- istics and Precision and Recall Odds? The following question was asked: What were the odds that searches produced by searchers with above mean score on given characteristics had above mean precision or recall? Table 21 provides the summary of the relationship be- tween searchers’ characteristics and precision and recall odds. (Abbreviations are explained at the beginning of this section.) The detailed explanation provided with Table 6 in Part II about the structure and contents of the tables on precision and recall odds are appropriate here as well. Briefly, column 1 provides the cut points or mean scores for given characteristic which divides the high and low values; column 2 gives the cross odds ratio representing the odds of moving to the high value of precision or recall due to a high value of a given characteristic; column 3 provides the asso- ciated log odds; column 4, the standard error; column 5, the t-value; and column 6, the indication of significance at 95 . As can be seen, precision odds were significantly af- fected by only one characteristic (LSI-CE) and recall by three-all associated with the Learning Style Inventory. The results indicate that searches produced by searchers who indicated Concrete Experience as preferred learning mode were 50 (l-0.50) less likely to have high precision and 49 less likely to have high recall, or in opposite terms, they were two times (l/O.5 = 2) more likely to have low (below mean) precision and also 2 times as likely to have low (below mean) recall. Searchers who indicated Abstract Conceptualization as preferred mode of learning were 3.27 times more likely to have high (above mean) recall. No other characteristic, including frequency of searching had a significant effect on precision and recall. In general, searchers indicating concrete experience as their preferred learning mode had significantly lower odds for both, precision and recall. Searchers preferring abstract- ness over concreteness in learning had improved recall odds. This suggests that preference toward concreteness in learning diminishes both, precision and recall, while prefer- ence toward abstractness in learning enhances recall. Searchers who prefer abstract learning may have a better chance at higher precision and recall. These conclusions emphasize and amplify those made on the relation between searchers characteristics and relevance odds. Searches Results of searches are presented in this and the follow- ing two sections. In this section we treat the measures that reflect tactics and efficiency of searches. In the next section we concentrate on project searches and in the third section involving searches we deal with overlap among items re- trieved by different searches for the same question, together with overlap in search terms. Although, search variables were involved in all three sections, we divided them to underscore the point that quite different research questions have been asked in each. What Were the Figures for Tactics and Efficiency Measures? Table 22 presents the measures related to tactics and ef- ficiency for all searches (comparison between outside and project searches is given in Table 25). All measures are self explanatory with possible exception of command cycles: a cycle is a sequence of commands between beginning of a search and viewing of retrieved results (e.g. typing, print- ing) or between two viewings. Use of more than one cycle may indicate testing of output and possible adjustment of search strategy. The first three measures (number of com- mands, cycles, and search terms) relate to search tactics, while the last three (preparation, online, and total time) relate to efficiency or costs. As can be seen, a typical search (if there is such a thing) had about 15 commands, 3 cycles and 10 search terms, took about 13 minutes of preparation time, 14 minutes of online time, for a total of 27 minutes from start to end. However, these mean figures, as all others, have to be interpreted with caution, because the ranges were wide and the distributions were not normal, they were all skewed toward the low end of the scales. What Was the Relationship between Tactics and Efsi- ciency Measures and Relevance Odds? The following question was asked: What were the odds that searches with high (above mean) scores on tactics and efficiency measures retrieved items that are relevant or par- tially relevant as opposed to not relevant? The answers can JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-May 1988 199  TABLE 21. Summary of the relation between searchers’ characteristics and the odds that precision and recall be above average (Done for outside searchers only) (N outside searchers = 36; N searches = 200; statistical significance at 95 ; mean precision for outside searches = 0.54; mean recall for outside searches = 0.20; for abbreviations see text). cut Point W-1 Stand. Error +I- Searchers Characteristic Odds stat. Signif. t- 3 Odds Precision Frequency of searching RAT (0 to 30) SRT (0 to 30) LSI: CE (12 to 48) RO (12 to 48) AC (12 to 48) AE (12 to 48) AC-CE (-36 to +36) AE-RO (-36 to +36) Recall Frequency of searching RAT SRT LSI: CE RO AC AE AC-CE AE-RO 2.93 1.54 0.43 0.33 1.30 No 13.03 1.44 0.36 0.28 1.27 No 10.60 1.39 0.33 0.28 1.14 No 24.70 0.50 -0.69 0.29 -2.39 Yes 27.90 1.13 0.13 0.29 0.43 No 33.55 1.60 0.47 0.29 1.59 No 33.68 1.09 0.08 0.29 0.29 No 8.60 1.95 0.66 0.29 2.29 Yes 5.13 0.89 -0.12 0.28 -0.42 No 2.93 0.70 -0.36 0.33 -1.09 No 13.03 1.10 0.10 0.29 0.33 No 10.60 1.09 0.09 0.29 0.29 No 24.70 0.51 -0.68 0.29 -2.29 Yes 27.90 1.39 0.33 0.30 1.08 No 33.55 3.27 1.19 0.30 3.87 Yes 33.68 0.62 -0.48 0.30 -1.62 No 8.60 2.47 0.90 0.30 3.01 Yes 5.73 0.55 -0.59 0.29 -2.m No command cycles increased relevance odds by a factor of 1.18. A high (above mean) number of search terms de- creased relevance odds by 22 (l-0.78); or in opposite terms: items retrieved in searches with a low number of search terms were 28 (l/0.78) more likely to be relevant. High preparation time and high total time used for a search decreased relevance odds by 13 and 19 respectively; or in opposite terms: items retrieved in searches with low prep- aration time and total time were 15 (l/0.87) and 23 (l/O.8 1) respectively more likely to be relevant. It may be of interest to comment on two seemingly con- tradictory findings on relevance odds. We report here that a high number of search terms in a search decreased relevance odds, while earlier (Part II, Table 10) we reported that a high degree of complexity in a question (high number of search concepts) increased relevance odds. The two TABLE 22. Tactics and efficiency measures for all searches (N searches = 360). For a Search: Mean Standard Deviation Min. Mm. No. of Commands 14.5 7.7 2 50 No. of Command Cycles 3.4 2.0 1 14 No. of Search Terms 10.3 7.0 1 61 Preparation Time Ws.) 0.22 0.14 0.02 0.83 Online Connect Time 0.24 0.16 0.01 1.24 Total Time Used 0.46 0.25 0.10 1.96 be found in Table 23; this is a summary of relevance odds as related to the three tactics and three efficiency measures. As can be seen, the number of commands in a search had no significant effect, but a high (above mean) number of TABLE 23. Summary of the relation between tactics and efficiency measures for searches and odds that a retrieved item be relevant or partially relevant (N all searches = 360; N items retrieved = 8956; statistical significance at 95 ). Tactics or Efficiency Measure cut Point Odds Log (Mean) Ratio Odds Stand. Error r- Stat. +/- Value Signif. No. of Commands 14.51 No. of Command cycles 3.40 No. of Search Terms 10.33 Preparation Time (hrs) 0.22 Online Connect Time 0.24 Total Time Used 0.46 0.94 -0.07 0.04 -1.51 No 1.18 0.17 0.04 3.84 Yes 0.78 -0.25 0.05 -5.42 Yes 0.87 -0.14 0.04 -3.31 Yes 1.08 0.08 0.04 1.71 No 0.81 -0.21 0.04 -4.73 Yes 200 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-May 1988
Search Related
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x