Table 1 Views about and experiences with key Health Systems Evidence (HSE) features
HSE feature | Key findings | Illustrative quotes |
|---|---|---|
Registration and sign-in | • Four participants reported feeling that the registration and sign-in decreased their willingness to use the database, while others said that they did not consider it a nuisance, especially since many sites now require sign-in • Five participants reported that having to remember a password could discourage use, but some noted that this challenge could be overcome by the option to save passwords on a computer and by frequent use of the site • Three participants expressed confusion over the purpose and value of registration and sign-in and two stated they would prefer a one-time only registration for mailing purposes, but without the repeated need to sign-in | “I don’t know why there’s a need to even sign in. That’s what I’m saying. I don’t know why, unless you’re trying to track the people who are using it somehow” “It does not limit my willingness to use it but I find it a bit annoying because I thought when I signed up the first time, it says a one-time registration so I assumed that I would sign in one time and then from then on, it would recognize my computer” |
Open (basic) search page | • Four participants stated that they found the open search page intuitive • Participants generally felt it was simple to use and user friendly, and liked being presented with it before deciding to move to the advanced search page • In discussing the open search page, participants also highlighted their preference for searching with keywords instead of having to do a very specific search immediately | “For the most part I think keywords are just the way we search. It’s just part of the way you do it normal life, so I find normal searching, keyword searching, to be the easiest for me and the most intuitive” |
Advanced search page | • Participants noted that they liked the ability to search by health system topic and combine those searches with an open search (i.e. using keywords) and limits • Three participants thought it was organized intuitively, indicating that the page did not look ‘too busy’ given that different sections of the page can be expanded to display more functionality • Many expressed concerns over the layout, with seven stating they found some of the functions and categories difficult to find and three saying they were unsure where to enter search terms • The most frequently requested changes were to add an option to limit a search to specific countries and to high-income countries (and not just low- and middle-income countries) | “if you’re starting from that main page where there’s just the one search field and then you click on advanced searching, I feel like it should keep that main search field there at the top of the page and then when you click on expand, it should expand below that and have the four options, the tick boxes, below that” “if I had the option of saying you know tick, tick, tick, I want these countries, that would be amazing” and “because [the ministry] does a lot of comparative work with the Commonwealth countries” |
Search results overview page | • Eight participants said that they liked the search results overview page, which appears after completing a search but before the detailed search results page is provided • Participants like the opportunity to select the types of documents they would like to have appear in the detailed search results • One participant noted they would prefer if this page could be selected as an optional intermediate step and two participants thought the organization was confusing • Six of the participants had never noticed the option to re-run the search in EvidenceUpdates or in PubMed (using the search hedges for health services research), or at least felt it was too hard to find and needed to be made more prominent • Upon learning that these features were available, participants said that having the option to re-run their search in EvidenceUpdates and PubMed was helpful | “…really loved the way that it categorized the results in all those little squares so I knew immediately what type of evidence I was going to get” “Nobody’s going to see that stuff [option to rerun searches in EvidenceUpdates and PubMed]. You’ve got a lot of little notes. What people do is they ignore the little notes at the bottom” |
Detailed search results page | • Four participants felt that the results page looked ‘too busy’ and had too much information • Many participants highlighted information that they felt was most important and therefore needed to be made more prominent by moving it to the left columns (to facilitate screening of information across rows) and ensuring the most important information has contrasting font (to make it more noticeable) • Most participants suggested that the title should be made most prominent and to have it link directly to either the one-page summary or to the full-text • Feedback about the priority that should be accorded to the other columns was mixed, with each of the remaining columns (countries in which studies were conducted or that are the focus of the document, the last year the literature was searched or year published, quality rating, type of question, and type of document) being suggested by at least one participant as important to be emphasized more in the results table | “The page is too much information. You’re just on information overload on the page. You need three or four columns, something like that” “When I click on the title, I expect that it will take me to your one-page summary. But I think that if you had ‘full text report’ itself, if you could just have the title linked to that directly so that rather than having to click on Show Links and then Click on Full Text Report, title itself just hyperlinked directly to the Full Text Report, that would just eliminate an extra step” |
Links to one-page summaries | • Three participants indicated that they found the one-page summaries to be helpful with the main reasons being that it offered a pre-digested form of information from the paper and could easily be saved or printed to be viewed later • Three other participants felt that the material in the summaries was too similar to the results page and some also felt that, due to time pressures they are under, they would not usually take the time to read a summary as they were more concerned with finding the full-text version of the document | “just having adjusted information; I just find it to be great. It’s pre-digested information which is usually helpful in cases where I’m trying to summarize” “as a policy person with one hour time limit, to get something to senior management, I wouldn’t look at it” “have the starting at the last year literature search, quality rating, all of that stuff down to the citation up closer to the top and then put the type of document, type of question, health system topics, domain, etc. lower down” |
Monthly evidence service | • Only two participants reported receiving the monthly evidence service, however, five stated that they thought they would be valuable as it is convenient to not have to go look for research and to remind them to use HSE | “if people knew about this type of website where you could have information sent directly to you without having to keep going in and do searches, I think probably would be very helpful. That’s one of the main ways that I get information” |
Types of documents | • Participants said they liked the diversity of documents in HSE, particularly the economic evaluations and documents related to health care renewal. • Two participants also said they liked that ongoing systematic reviews (i.e. systematic review protocols) and systematic reviews being planned (i.e. registered systematic review titles) were indexed in HSE | “Knowing that someone is going to be doing a systematic review in cases where we can’t find information […] because we consider our finding to be there hasn’t been much research, so it’s nice to be able to say that someone is looking at this” |
Supplementary material/portals | • Six participants stated that they found the Evidence-Informed Healthcare Renewal (EIHR) Portal helpful and relevant to their work because it makes it easy to examine what other jurisdictions are doing, as well as being helpful to give them access to grey literature given that academic/peer-reviewed literature often does not address topics that meet their needs and interests • Despite finding it useful, three participants stated they had never noticed the EIHR Portal before | “because there are topics where the academic literature isn’t as pertinent and documents from the grey literature are more useful but it’s harder to find those documents” “…might be an easier term to call it rather than evidence informed healthcare renewal. It just sounds really jargony and doesn’t really apply to what you just told me, in my opinion” |