Can user and task characteristics be used as predictors of success in health information retrieval sessions?

Abstract

Introduction. The concept and study of relevance has been a central subject in information science. Although research in information retrieval has been focused on topical relevance, other kinds of relevance are also important and justify further study. Motivational relevance is typically inferred by criteria such as user satisfaction and success. Method. Using an existing dataset composed by an annotated set of health Web documents assessed for relevance and comprehension by a group of users, we build a multivariate prediction model for the motivational relevance of search sessions. Analysis. The analysis was based on lasso variable selection, followed by model selection using multiple logistic regression. Results. We have built two regression models; the full model, which considers all variables of the dataset, has a lower estimated prediction error than the reduced model, which contains the statistically-significant variables from the full model. The higher values of evaluation metrics, including accuracy, specificity and sensitivity in the full model support this finding. The full model has an accuracy of 91.94%, and is better at predicting motivational relevance. Conclusions. Our findings suggest features that can be considered by search engines to estimate motivational relevance, to be used in addition to topical relevance. Among these features, a high level of success in Web search and in health information search on social networks and chats are some of the most influencing user features. This shows that users with higher computer literacy might feel more satisfied and successful after completing the search tasks. In terms of task features, the results suggest that users with clearer goals feel more successful. Moreover, results show that users would benefit from the help of the system in clarifying the retrieved documents.