Skip to main content

Table 2 Research engagement actions, subactions, subaction levels, raw utilities, standard errors, and rescaled utility coefficients

From: Using conjoint analysis to develop a system to score research engagement actions by health decision makers

Research engagement action

Subaction

Levels of each subaction

Raw utility coefficient

Standard error

Rescaled utility coefficient†

1. Searching for research

a. Policymaker searched academic literature databases and/or physical libraries

i) No

−3.59***

0.27

0

ii) Yes

0

–

2.83

b. Policymaker searched grey literature sources

i) No

−1.81***

0.16

0

ii) Yes

0

–

1.42

c. Policymaker obtained research by chance, research that was on-hand, or provided by colleagues

i) No

−1.37***

0.11

0

ii) Yes

0

–

1.08

d. Policymaker requested experts (researchers, working groups, librarians, or other research experts) to identify research

i) No

−1.98***

0.16

0

ii) Yes

0

–

1.56

e. Policymaker searched generic databases or search engines

i) No

−1.12***

0.11

0

ii) Yes

0

–

0.88

f. Policymaker examined reference lists, citation indices, or databases of references

i) No

−1.55***

0.14

0

ii) Yes

0

–

1.22

2. Research obtained and used

a. Policymaker found systematic reviews and/or meta-analyses

i) No

−1.94***

0.24

0

ii) Yes

0

–

3.29

b. Policymaker found books and/or technical monographs

i) No

−0.34***

0.12

0

ii) Yes

0

–

0.58

c. Policymaker found primary research and/or theoretical research

i) No

−0.99***

0.19

0

ii) Yes

0

–

1.67

d. Policymaker found unpublished research and/or conference resources

i) No

−0.49***

0.13

0

ii) Yes

0

–

0.82

e. Policymaker found internal policies, evaluations, or data

i) No

−0.23*

0.10

0

ii) Yes

0

–

0.39

f. Policymaker found policies, evaluations, or data from external organisations or registries

i) No

−0.60***

0.11

0

ii) Yes

0

–

1.01

g. Policymaker obtained recent (up-to-date) research from the above categories

i) No (Older research)

−0.73***

0.15

0

ii) Yes (Recent)

0

–

1.24

3. Appraising relevance

a. Policymaker assessed whether the research was applicable to the policy context or policy issue

i) No

−1.70***

0.17

0

ii) Yes

0

–

2.06

b. Policymaker assessed whether research recommendations were actionable and/or feasible?

i) No

−1.53***

0.18

0

ii) Yes

0

–

1.84

c. Policymaker assessed whether the research was consistent with previous research on the issue

i) No

−1.02***

0.15

0

ii) Yes

0

–

1.23

d. Policymaker assessed if research was compatible with his/her OR the organisation’s values, knowledge, or experience

i) No

−0.97***

0.14

0

ii) Yes

0

–

1.17

e. Policymaker consulted experts to assess the relevance of research

i) No

−1.07***

0.12

0

ii) Yes

0

–

1.29

f. Policymaker undertook these actions as part of a pre-specified strategy

i) No (ad-hoc, unplanned)

−1.16***

0.18

0

ii) Yes

0

–

1.40

4. Appraising quality

a. Policymaker assessed whether the design or conclusions of the research were valid

i) No

−1.16***

0.22

0

ii)Yes

0

–

2.00

b. Policymaker evaluated whether the design or conclusions of the research were described clearly and comprehensively

i) No

−0.68***

0.16

0

ii) Yes

0

–

1.17

c. Policymaker assessed whether the source of the research was credible

i) No

−0.64***

0.12

0

ii) Yes

0

–

1.10

d. Checked if the research cited, or was referenced in other high-quality research or policy documents

i) No

−0.45**

0.18

0

ii) Yes

0

–

0.77

e. Policymaker consulted experts to assess quality

i) No

−0.76***

0.19

0

ii) Yes

0

–

1.31

f. Policymaker assessed the level of evidence of the research

i) No

−0.88***

0.17

0

ii) Yes

0

–

1.51

g. Policymaker undertook these actions as part of a pre-specified strategy

i) No (ad-hoc, unplanned)

−0.67***

0.22

0

ii) Yes

0

–

1.15

5. Generating new researchers

a. Policymaker expressed explicit intentions to generate or commission new research (to follow-up the current policy) OR stated that he/she had already undertaken this research

i) No (no intentions to generate new research)

−2.08***

0.28

0

ii) No (uncertain intentions only)

−1.97***

0.27

0.18

iii) Yes

0

–

3.42

b. Policymaker mentioned thorough research generation activities

i) No

−1.72***

0.19

0

ii) Yes

0

–

2.84

c. Policymaker mentioned less intensive research activities

i) No

−0.96***

0.14

0

ii) Yes

0

–

1.58

d. Policymaker advocated for future research to be undertaken

i) No

−0.60***

0.15

0

ii) Yes

0

–

0.99

6. Interacting with researchers

a. Policymaker engaged in thorough collaborative activities with researchers

i) No

−2.56***

0.25

0

ii) Yes

0

–

3.75

a. Policymaker engaged in less intensive interactions with (other) researchers

i) No

−0.91***

0.11

0

ii) Yes

0

–

1.33

b. Policymaker engaged in sporadic contact with (other) researchers?

i) No

−0.67***

0.11

0

ii) Yes

0

–

0.98

c. Policymaker actively initiated these interaction activities

i) No

−2.01***

0.22

0

ii) Yes

0

–

2.94

  1. †Utility coefficients were rescaled so that they became positive, with the lowest level of each subaction having a zero-coefficient, and adding up to 9.
  2. ***p < 0.001; **p < 0.01; *p < 0.05.