Table 1 Areas of policy implementation and outcome evaluation assessment [11, 19,20,21]

Adoption

The number, percentage and representativeness of jurisdictions that pass or enact the policy, including allocation of resources for implementation and enforcement

What is the policy mechanism employed (e.g., executive order, legislation, regulation, contracts)? At what level of government was the policy addressed? Is the policy based on evidence-based scientific or clinical recommendations? Was the policy formally approved or passed? Was the policy adopted? If so, what was the adoption date? What is the scope or elements of the policy and at what level (i.e. federal, state, local) is it being adopted/implemented. Were implementation process steps outlined in statute and regulation? If so, capture language. Was implementation evaluation written into the law? Was the policy implemented? If so, was there an implementation date? What resources and funding were available to ensure implementation of the policy? Are resources or training available to support lower capacity implementors to ensure equity? What existing social, political or economic realities were occurring at the time of implementation of this policy?

Is there equity in where the policy was passed? Is there equity in where it was implemented? Is there equity in the funding resources? Is there equity in the training resources? Is there over-enforcement or under-enforcement in some locations or with some community groups?

Penetration/reach

The number, percentage and representativeness of individuals affected by the policy and the systems changed or processes improved with policy implementation

Scope/reach of the policy (actual population reach, systems change, process improvement). Does the policy address any social determinants of health? Was there a system change with this policy? What was the actual population reach for this policy (especially noting across race/ethnicity, geography, income, education? Did this policy improve or worsen any existing processes?

Is there consideration for how the policies reach those at the greatest risk or in the greatest need? Is there equity in the reach and penetration for the benefits to all locations and all community groups? Is there equity in the burdens to all locations and all community groups?

Acceptability

The perception among relevant audiences, including decision makers, implementers and intended beneficiaries, that the policy is agreeable, acceptable or adequate

Who were the collaborators involved in adoption and implementation? Were people who would be most impacted by the policy involved in implementing? What supporting policies, resources (e.g., technical assistance, training) and/or processes were put in place to enforce the implementation of the policy? Was the policy well received across all priority populations? If so, by whom? If not, by whom? Was the policy change adequately communicated? If so, by whom/by what agency or stakeholder? How did the communication happen? What was the level of engagement with enactment? Amongst implementers, how was the policy received across impacted populations? Did the population of focus know about the policy and support it?

Assure that those in the community can provide feedback and co-create policy solutions and prioritization. Who was consulted in the development of the policy approach? Who decides the acceptability of the policy approach? Is the policy approach culturally relevant and affirming to all groups? Is the policy approach a priority to all groups? Is feedback from all locations and all community groups considered? Which locations and community groups were engaged in how the policy would be implemented? How is the policy approach communicated and who are the spokespeople?

Feasibility

The extent to which the policy can be successfully implemented in a particular community or setting

Were implementers adequately trained to implement all aspects of the policy? Were there barriers to implementation? If so, what were they? Were they overcome? If so, how? What/who were the facilitators for implementation?

Is the policy approach feasible for all locations and all community groups? Are there resources to address unequal barriers and burdens?

Fidelity

The extent to which the policy was implemented as intended

Was the policy implemented as intended?

Were differences by locations and community groups considered so that the policy could be consistently implemented as intended (i.e. socio-cultural, geographic and other areas of diversity)

Implementation cost

The value of all resources required to implement the policy, including time, facilities and materials

What was the cost of implementation? Are there any annual appropriations in place associated with implementation?

Was the unequal burden of costs considered for locations and community groups that are under-invested? For locations and community groups that are under-invested, will the implementation costs take resources from other areas?

Effectiveness

Change in intended outcomes, appropriate to the amount of time since the policy was adopted. Successful policy implementation may require years or even decades to change health outcomes; changes in environments or behaviours may be appropriate short-term outcomes

Was the policy implemented as intended? Did it have its expected outcomes? Was it implemented equitably across different communities, populations or geographies?

Is the policy approach effective for all groups and all community groups? Will the policy approach close absolute and relative gaps between groups?

Unintended consequences

Any outcomes (positive or negative) that were not intended by those who developed or adopted the policy. [21]

Were there any unintended consequences experienced by the priority population or other population groups? If so, describe. Were there any unintended consequences experienced during implementation at the systems level? If so, please describe. Were disparities increased or decreased with implementation?

Are there disproportionate unanticipated consequences for under-invested locations and community groups? Are unintended consequences equitably addressed for all groups?

Sustainability

Maintenance of the policy over time. Includes continued implementation, enforcement, and resulting outcomes. [19]

Was the policy change sustained over time? How long? Did outcomes vary over time? If so, how? Was the implementation process sustainable over time?

Maintenance of the policy over time. Includes continued implementation, enforcement and resulting outcomes. Assessment should focus on two levels: communities reached by the policy, and jurisdictions that enact the policy. [19] Are there differences for locations and community groups in how policy approaches are maintained over time?

Monitoring

Assessing the impact of the law through surveys and other data monitoring systems

Is there capacity for monitoring? If so, capture the language (roles and responsibilities; who is responsible for monitoring?). Is there a role for a federal, state and or local health agency and any dedicated appropriations/funding for evaluation?

Are there differences in the burden or risks of monitoring for some locations and community groups?