Table 3 Users’ experiences using Evidence Aid^a

Findability

• Easy to find the site with a Google search or URL

• Easy to locate desired articles using a search engine, tabs, closed dictionary or a combination

• Easy to locate desired information about/from articles using initial display and/or supplementary webpages on site or on other sites

• None

• Home page does not have a search engine *

• Difficulty identifying correct terms to enter into the search engine

• Difficulty locating the search bar *

• Difficulty locating user-friendly summaries

• Lack of search categories to narrow down the search results

• Latest content banner on home page is heavily focused on internal Evidence Aid activities and less on finding and retrieving resources to support evidence-informed decision-making in crisis zones

• Difficulty locating supplemental information on other sites (e.g., dead links)

• Four cluster areas under “Resources” are helpful in finding research evidence *

• Results ‘tags’ further narrows the search results *

• Website is easy to find with a Google search or URL

• Clearly marked tabs on home page to arrive at Resources, Events, etc.,

• Search bar under Resources is helpful

• Rapid display of search results

• Add an advanced search filter (e.g., date of last search, specific contexts, language preference) for more targeted search results *

• Bring forward to the Home page the following:

 ▪ search engine

 ▪ four cluster areas

 ▪ feature systematic reviews and best practice health guidelines related to a current crisis

 ▪ organize search results according to target user (e.g., researcher, decision-maker, etc.)

Usability

• Purpose and scope of the site clearly described

• Basic tasks easily accomplished on first use

• Search goals achieved with effectiveness, efficiency and satisfaction

• None

• None

• Basic tasks require undertaking multiple steps to arrive at results on first use *

• Not clear what the purpose of the website is, target audience, or the type of evidence it provides

• Not clear how Evidence Collection is created and how it is different from the available research evidence on website

• Create a clearer statement on purpose of site and type of evidence it provides *

• Promote and clarify the purpose of Evidence Collection

Usefulness

• Nature of information retrieved provides value (e.g., addresses question without jargon and in an understood language)

• None

• None

• Little focus on current and ongoing crises – most evidence presented is post crisis

• User-friendly summaries do not contain enough relevant details to make an informed decision on whether to read the full article

• Lack of systematic reviews and guidelines related to:

 ❖ participants’ particular areas of professional interests or field of work *

 ❖ context-specific research evidence

 ❖ all answers to questions relevant to humanitarian action

 ❖ site, search results, user-friendly summaries, and full text in different languages

• Lack of single studies (e.g., “real-time” data and evaluations) in addition to systematic reviews and guidelines

• Useful in providing an independent evidence website for curated evidence on crisis zones for decision-makers working in the field *

• Once located:

 ❖ user-friendly summary is concise, easy to understand, and practical in deciding whether to read full text

 ❖ the evidence-based guidelines are good in providing take-home messages

• Contains evidence related to both man-made and natural disasters

• Provision of some content on the site in other languages (e.g., Spanish) is useful

• Contains more systematic reviews than other websites

• Fills the gap between academic research (i.e., systematic reviews) and action in humanitarian aid settings

• Resources tab regarding recent crises is important

• Better linkage of evidence into action – turning the evidence available into explicit actionable points for decision makers in crisis zones *

• Include research evidence that addresses:

 ▪ strengthening health systems

 ▪ implementation strategies for interventions in humanitarian crisis

 ▪ building capacity towards use of research evidence

• Add more content in other languages

• Survey end-users about their needs

Desirability

• Images reflect the purpose and scope

• Design conveys a unique and appropriate identity (e.g., name, logo, font type & size, colours, and sophistication of features)

• None

• None

• Photos are ordinary (i.e., academic looking) and repetitive *

• Amount of white background on screen is problematic for people who work on big screens & high definition

• Font size is smaller than some respondents would prefer

• Logo is similar to the ONE and MSF website

• Basic design: simple, not a lot of pop ups *

• Home page is clean and organized

• Good choice of colors on the website

• “Evidence Matters” is catchy

• Title: “Evidence Aid” is appealing

• Use compelling photos relevant to humanitarian contexts *

• Use infographics to breakdown key findings from the evidence

Accessibility

• Accommodates diverse user contexts (e.g., inability to pay article-access fees or avoid them through affiliations with universities with paid subscriptions, inability to use high bandwidth features)

• Accommodates diverse physical functioning in the user (e.g., colour choices accommodate color blindness, font size is changeable)

• None

• Some of the evidence available is not open-access (i.e., requires a payment) *

• Widespread use of the color red creates challenges for those with color blindness *

• Concerns over whether documents can be read online or have to be downloaded first - a problem in a low bandwidth internet setting *

• Accessible to a broad spectrum of people working in the humanitarian sector who have access to the internet *

• Useful during a current large-scale humanitarian crisis or for select other topics in providing time-limited free access to full-text articles that are normally behind a pay wall

• Create a mobile friendly app or use responsive design *

Credibility^a

• Easy to verify accuracy of information on site (e.g., clear indication of inclusion criteria of research evidence, objective assessment of available research evidence, links to credible external sources (e.g., Cochrane), provides complete exhaustive summary of evidence (e.g., systematic reviews))

• Clear illustration that honest and trustworthy people stand behind the site (e.g., profile description of people)

• Site is updated frequently (i.e., content been reviewed recently)

• Restraint in any promotional content (e.g., ads) or direct link to a particular funding source

• None

• None

• Not clear what inclusion criteria is used to include the best available evidence *

• Concerns over the frequency of updating latest available research evidence

• Direct and clear link to Cochrane Library increased level of trust by respondents *

• Seems to be continuously updated with the newest content upfront for easier access

• Team and advisory committee behind the site are credible individuals

• Objective assessment of research evidence

• Availability of systematic reviews

• Available research evidence does not seem to be directly linked to a particular funding source

• Give greater visibility to major contributors & funders *

Value for the user

• Intended users are aware of the site

• The site advances the mission of the organization: “to alleviate suffering and save lives by providing the best available evidence on the effectiveness of humanitarian action and enabling its use”

• None

• None

• Lack of awareness among humanitarian aid workers about the existence of or value added by Evidence Aid *

• A solid attempt to putting together multiple sources of information in one spot *

• Substantial efforts in partnering with other organizations to fill gaps with new systematic reviews

• Specificity to humanitarian action is helpful

• Emphasize why evidence matters in humanitarian action *

• Continue collaborating with other organizations to fill gaps with new systematic reviews *

• Choose from a menu of additional ways to enable the use of the site and its contents (e.g., personalized briefing notes, rapid evidence synthesis, webinars, etc.)