From insight network to open policy practice: practical experiences

Background Evidence-informed decision-making and better use of scientific information in societal decisions has been an area of development for decades but is still topical. Decision support work can be viewed from the perspective of information collection, synthesis and flow between decision-makers, experts and stakeholders. Open policy practice is a coherent set of methods for such work. It has been developed and utilised mostly in Finnish and European contexts. Methods An overview of open policy practice is given, and theoretical and practical properties are evaluated based on properties of good policy support. The evaluation is based on information from several assessments and research projects developing and applying open policy practice and the authors’ practical experiences. The methods are evaluated against their capability of producing quality of content, applicability and efficiency in policy support as well as how well they support close interaction among participants and understanding of each other’s views. Results The evaluation revealed that methods and online tools work as expected, as demonstrated by the assessments and policy support processes conducted. The approach improves the availability of information and especially of relevant details. Experts are ambivalent about the acceptability of openness – it is an important scientific principle, but it goes against many current research and decision-making practices. However, co-creation and openness are megatrends that are changing science, decision-making and the society at large. Against many experts’ fears, open participation has not caused problems in performing high-quality assessments. On the contrary, a key challenge is to motivate and help more experts, decision-makers and citizens to participate and share their views. Many methods within open policy practice have also been widely used in other contexts. Conclusions Open policy practice proved to be a useful and coherent set of methods. It guided policy processes toward a more collaborative approach, whose purpose was wider understanding rather than winning a debate. There is potential for merging open policy practice with other open science and open decision process tools. Active facilitation, community building and improving the user-friendliness of the tools were identified as key solutions for improving the usability of the method in the future.

The original discussion was held in Finnish and can be found from http://fi.opasnet.org/fi/Keskustelu:Jaettu_ymm %C3%A4rrys. Trapezoids are statements. Light blue is an opening fact statement ('Openness causes serious problems to the quality of policy making.'), and blue is closing fact statement that is updated based on the discussion ('Openness causes serious problems to the quality of policy making, if it is too tightly connected to impulsive thinking in social media.'). Orange arguments are true and gray are false. Red arrow is an attack, green is a defense, and gray is irrelevant (Accessed 1 Feb 2020).

13/40
Appendix S3: Open policy ontology Shared understanding aims at producing a description of different views, opinions, and facts related to a specific topic such as a decision process. The open policy ontology describes the information structures that are needed to document shared understanding of a complex decision situation. The purpose of the structure is to help people identify hidden premises, beliefs, and values and explicate possible discrepancies. This is expected to produce better understanding among participants.
The basic structure of a shared understanding is a network of items and relations between them. This network uses Resource description framework, which is an ontology standard used to describe many Internet contents. Items and relations (aka properties) are collectively called resources. Each item is typically of one of the types mentioned below. This information is documented using property instance of (e.g. Goherr assessment is instance of assessment).
Items are written descriptions of the actual things (people, tasks, publications, or phenomena), and on this page these descriptions rather than the actual things are discussed. Different item types have different levels of standardisation and internal structure. For example, knowledge crystals are web pages that always have headings question, answer and rationale, and the information is organised under those headings. Some other items describe e.g. statements that are free-text descriptions about how a particular thing is or should be (according to a participant), and yet some others are metadata about publications. A common feature is that all items contain information that is relevant for a decision.
In the open policy ontology, each item may have lengthy texts, graphs, analyses or even models inside them. However, the focus here is on how the items are related to each other. The actual content is often referred to as one key sentence only (description). Each item also has a unique identifier URI that is used for automatic handling of data.
The most important items are knowledge crystals and they are described here.
• Assessment describes a particular decision situation and focuses on estimating impacts of different options. Its purpose is to support the making of that decision. Unlike other knowledge crystals, assessments typically have a defined start and end dates and they are closed after the decision is made. They also have contextually and situationally defined goals`to be able to better serve the needs of the decision makers of the decision.
• Variable answers a particular factual or ethical question that is typically needed in one or more assessments. The answer of a variable is continually updated as new information arises, but its question remains constant in time. Variable is the basic building block of assessments. In R, variables are typically implemented using ovariable objects from OpasnetUtils package.
• Method tells how to systematically implement a particular information task. Method is the basic building block for describing the assessment work (not reality, like variables). In practice, methods are "how-to-do" descriptions about how information should be produced, collected, analysed, or synthesised in an assessment. Typically, methods contain a software code or another algorithm to actually perform the method easily. In R, methods are typically ovariables 14/40 that require some context-specific upstream information about dependencies before it can be calculated.
There are also other important classes of items: • Publication is any documentation that contains useful information related to a decision.
Publications that are commonly used at Opasnet include encyclopedia article, lecture, nugget, and study. Other publications at Opasnet are typically uploaded as files.
• Encyclopedia article is an object that describes a topic like in Wikipedia rather than answers a specific research question. They do not have a predefined attribute structure.
• Lecture : Lecture contains a piece of information that is to be mediated to a defined audience and with a defined learning objective. It can also be description of a process during which the audience learns, instead of being a passive recipient of information.
• Nugget is an object that is not editable by other people than a dedicated author (group) and is not expected to be updated once finalised. They do not have a predefined attribute structure.
• Study describes a research study and its answers, i.e. observational or other data obtained in the study. The research questions are described as the question of the information object, and the study methods are described as the rationale of the object. Unlike in an article, introduction or discussion may be missing, and unlike in a variable, the answer and rationale of the study are more or less fixed after the work is done; this is because the interpretations of the results typically happen elsewhere, e.g. in variables for which a study contains useful information.
• Discussion is a hierarchically structured documentation of a discussion about a defined statement or statements.
• Stakeholder page is used to describe a person or group that is relevant for a decision or decision process; they may be an actor that has an active role in decision making or is a target of impacts. Contributors of Opasnet are described on their own user pages; other stakeholders may have their page on the main namespace.
• Process describes elements of a decision process.
• Action describes what, who and when should act to e.g. perform an assessment, make a decision, or implement policies.
Relations show different kinds of connections between items.
• Causal link tells that the subject may change the object (e.g. affects, increases, decreases, prevents).
• Participatory link describes a stakeholder's particular role related to the object (participates, negotiates, decides).
• Operational link tells that the subject has some kind of practical relation to the object (executes, offers, tells).
• Evaluative link tells that the subject shows preference or relevance about the object (has truthlikeness, value, popularity, finds important).
• Referential link tells that the object is used as a reference of a kind for the subject (makes relevant; associates to; has reference, has tag, has category).
• Argumentative link occurs between statements that defend or attack each other (attack, defend, comment).
• Property link connects an evaluative (acceptability, usability), a logical (opposite, inverse) or set theory (has subclass, has part) property to the subject.

Item types
This ontology is specifically about decision making, and therefore actions (and decisions to act) are handled explicitly. However, any natural, social, ethical or other phenomena may relate to a decision and therefore the vocabulary has to be very generic.

Relation types
Relations are edges between items (or nodes). A relation I is said to be an inverse of relation R, iff, for all items subject and object, claim "subject R object" is always equal to claim "object I subject". Ovariable is an object class that is used in R to operationalise knowledge crystals. In essence, impact assessment models are built using ovariables as the main tool to organise, analyse, and synthesise data and causal relations between items. The purpose of ovariables is to offer a standardised, generalised, and modular solution to modelling. Standardised means that all ovariables have the same overall structure, and this makes it possible to develop generalised functions and processes to manipulate them. Modular structure of a model makes it possible to change pieces within the model without braking the overall structure of functionality. For example, it is possible to take an existing health impact model, replace the ovariable that estimates the exposure of the target population with a new one, and produce results that are otherwise comparable to the previous results but differ based on exposure.
What is the structure of an ovariable such that • it complies with the requirements of variable and • it is able to implement probabilistic descriptions of multidimensional variables and • it is able to implement different scenarios?
An ovariable contains the current best answer in a machine-readable format (including uncertainties when relevant) to the question asked by the respective knowledge crystal. In addition, it contains the information needed to derive the current best answer. The respective knowledge crystal typically has an own page at Opasnet, and the code to produce the ovariable is located on that page under subheading Calculations.
It is useful to clarify terms here. Answer is the overall answer to the question asked (including an evaluated ovariable), and it is the reason for producing the knowledge crystal page in the first place. Answer is typically located near the top of the page to emphasise its importance. An answer may contain text, tables, or graphs on the web page. It typically also contains an R code for evaluating the respective ovariable. Output is the key part (technically a slot) of the answer within an ovariable and contains the details of what the reader wants to know about the answer. All other parts of the ovariable are needed to produce the output or understand its meaning. Finally, Result is the key column of the Output table (technically a data frame) and contains the actual numerical values for the answer.

Slots
The ovariable is a class S4 object defined by OpasnetUtils in R software system. An ovariable has the following separate slots that can be accessed using X@slot (where X is the name of the ovariable): @name 32/40 • Name of <self> (the ovariable object) is useful since R's S4 classes doesn't support self reference. It is used to identify relevant data structures as well as to set up hooks for modifiers such as scenario adjustments.

@output
• The current best answer to the question asked.
• A single data frame (a 2D table type in R) • Not defined until <self> is evaluated.
• Possible types of columns: • Result is the column that contains the actual values of the answer to the question of the respective knowledge crystal. There is always a result column, but its name may vary; it is of type <self>Result.
• Indices are columns that define or restrict the Result in some way. For example, the Result can be given separately for males and females, and this is expressed by an index column Sex, which contains locations Male and Female. So, the Result contains (at least) one row for males and one for females. If there are several indices, the number of rows is typically the product of numbers of locations in each index. Consequently, the output may become very large with several indices.
• Iter is a special kind of index used in Monte Carlo simulations. Iter contains the number of the iteration. In Monte Carlo, the model is typically run 1000 or 10000 times.
• Unit contains the unit of the Result. It may be the same for all rows, but it may also vary from one row to another. Unit is not an index.
• Other, non-index columns can exist. Typically, they are information that were used for some purpose during the evolution of the ovariable, but they may be unimportant in the current ovariable if they have been inherited from parent ovariables. Due to these other columns, the output may sometimes be rather wide. @data • A single data frame that defines <self> as such.
• data slot contains data about direct measurements or estimates of the output. Typically, when data is used, the output can be directly derived from the information given, with possibly some manipulations such as dropping out unnecessary rows or interpreting given ranges or textual expressions as probability distributions.

@marginal
• A logical vector that indicates full marginal indices (and not parts of joint distributions, result columns, or units or other row-specific descriptions) of output. @formula • A function that defines <self> using objects from dependencies as inputs.
• Returns either a data frame or an ovariable, which is then used as the output of the ovariable.
• Formula and dependencies slots are always used together. They estimate the answer indirectly in cases when there is knowledge about how this variable depends on the results of other variables (called parents). The slot dependencies is a table of parent variables and their identifiers, and formula is a function that takes the outputs of those parents, applies the defined code to them, and in this way produces the output for this variable.

@dependencies
• A data frame that contains names and tokens or identifiers for model runs of variables required for <self> evaluation (list of causal parents). The following columns may be used: • Name: name of an ovariable or a constant found in the global environment (.GlobalEnv).
• Key: the run key (typically a 16-character alphanumeric string) of a model run that is stored to Opasnet server. Key to be used in objects.get() function to fetch the dependent object.
• Ident: Page identifier and rcode name to be used in objects.latest() function where the newest run contains the dependent object. Syntax: "Op_en6007/answer".
• Also other columns are allowed (e.g. Description), and they may contain additional information about parents.
• Dependencies is a way of enabling references in ovariables by using function OpasnetUtils/ComputeDependencies. It creates variables in .GlobalEnv environment so that they are available to expressions in formula.
• This identifier is used to download data from the Opasnet database for the data slot (by default, only if empty) upon <self> evaluation.
• By default, the data defined by ddata is downloaded when an ovariable is created. However, it is also possible to create and save an ovariable in such a way that the data is downloaded only when the ovariable is evaluated. @meta • A list of descriptive information of the object. Typical information include date created, username of the creator, page identifier for the Opasnet page with the ovariable code, and identifier of the model run where the object was created.
• Other meta information can be added manually.

OpasnetUtils and operations with ovariables
OpasnetUtils is an R package found in CRAN repository (cran.r-project.org). It contains tools for open assessment and modelling at Opasnet, especially for utilising ovariables as modelled representations of knowledge crystals. Typically, ovariables are defined at Opasnet pages, and their data and evaluated output are stored to Opasnet server. There are also special user interface tools to enable user inputs before an R code is run on an Opasnet page; for further instructions, see http://en.opasnet.org/w/Rtools. However, ovariables can be used independently for building modular assessment models without any connection to Opasnet.
The example code shows some of the most important functionalities. Each operation is followed by an explanatory comment after # character.
install.packages("OpasnetUtils") # Install the package OpasnetUtils. This is done only once per computer.
library(OpasnetUtils) # Open the package. This is done once per R session.
An ovariable is well defined when there is enough data, code or links to evaluate the output. Ovariables often have upstream dependencies whose output affect the output of the ovariable at hand. Therefore, ovariables are usually stored in a well defined but unevaluated format (i.e. without output). This makes it possible to use the same ovariable in different contexts, and the output varies depending on the upstream dependencies. On the other hand, it is possible to store all evaluated ovariables of a whole assessment model. This makes it possible to archive all details of a certain model version for future scrutiny.
Ovariables have an efficient index handling, which makes it possible to do arithmetic operations such as sums and products in a very simple way with ovariables. The basic idea is that if the outputs of two ovariables have two columns by the same name, they are automatically merged (or joined, using the SQL vocabulary) so that rows are merged iff they have the same location values in those two columns. The same principle applies to all pairs of columns by the same name. After the merge, the arithmetic operation is performed, row by row, to the Result columns of each ovariable. This results in an intuitive handling of outputs using a short and straightforward code.
Recursion is another important property of ovariables. When an ovariable is evaluated, a code checks whether it has upstream dependencies. If it does, those ovariables are fetched and evaluated first, and recursively the dependencies of those ovariables are fetched also, until all dependencies have been evaluated. Case-specific adjustments can be done to this recursion by fetching some upstream ovariables before the first ovariable is evaluated; if an upstream ovariable exists already in the global environment, the existing object is used and the respective stored object is not fetched (dependencies are only fetched if they do not already exist; this is to avoid unnecessary computation).

Decisions and other upstream commands
The general idea of ovariables is such that their code should not be modified to match a specific model but rather define the knowledge crystal in question as extensively as possible under it's scope. In other words, it should answer its question in a reusable way so that the question and answer would be useful in many different situations. (Of course, this should be kept in mind already when the question is defined.) To match the scope of specific models, ovariables can be modified without changing the ovariable code by supplying commands upstream. A typical decision command is to make a new decision index with two scenarios, "business as usual" and "policy" and use the original ovariable result for business as usual and adjust the result for the policy e.g. by adding or multiplying it by a constant reflecting the impact of the policy on the ovariable. Such adjustments can be done on the assessment level without a need to change the ovariable definition in any way.
Evaluating a latent ovariable triggers first the evaluation of its unevaluated parent ovariables (listed in dependencies) since their results are needed to evaluate the child. This chain of evaluation calls forms a recursion tree in which each upstream variable is evaluated exactly once (cyclical dependencies are not allowed). Decision commands about upstream variables are checked and applied upon their evaluation and then propagated downstream to the first variable being evaluated. For example, decisions in decision analysis can be supplied this way: 1. pick an endpoint ovariable 2. make decision variables for any upstream ovariables (this means that you create new scenarios with particular deviations from the actual or business-as-usual answer of that ovariable) 3. evaluate endpoint ovariable 4. optimize between options defined in decisions.
Other commands include: collapse of marginal columns by sums, means or sampling to reduce data size; and passing input from model level without redefining a whole ovariable.

Opasnet Base
Opasnet Base is a storage database for all kinds of data needed in open assessments. It may contain parameter values for models, which are typically shown as small tables on knowledge crystal pages, from which they are automatically stored to the database. It may also contain large dataset such as research datasets or population datasets of thousands or even millions of rows, and they are uploaded to the database using an importer interface. Each table has its own structure and may or may not share 36/40 column names with other tables; however, if a table is directly used as data slot for an ovariable, it must have a Result column.
Technically, Opasnet Base is a noSQL database using MongoDB software. Metadata of the tables is stored in a MySQL database. This structure offers the speed, searchability, and structural flexibility that a large amount of non-standard data requires. The database also offers version control, as old versions of a data table are kept in the database when new data is uploaded.
The database also contains data about model runs that have been performed at Opasnet, if objects were stored during that model run. This makes it possible to fetch objects produced by a particular code on a particular knowledge crystal page. Typically the newest version is fetched, but information about the old versions are kept as well. The objects stored are not located in MongoDB but on server files that can be accessed with a key. It is also possible to save objects in a non-public way so that the key is not stored in the database and is only given to the person who ran the code. Due to disc storage reasons, Opasnet does not guarantee that stored objects will be kept permanently; therefore, it is a good practice to store final assessment runs with all objects to another location for permanent archival.
There are several ways to access database content.
• If the data is on an Opasnet page, simply go to that page, e.g. http://en.opasnet.org/w/Mercury_concentrations_in_fish_in_Finland#Data • Use a link to the Opasnet Base interface, e.g. http://en.opasnet.org/w/Special:Opasnet_Base? id=op_en4004.mercury_in_baltic_herring • Use a function in R: dat <-opbase.data("Op_en4004", subset="Mercury in Baltic herring") • Use a function in R for stored objects: objects.latest("Op_en4004", code_name="conc_mehg") For further instructions, see http://en.opasnet.org/w/Opasnet_Base_UI for user interface and http://en.opasnet.org/w/Table2Base for the wiki interface of small tables.  Opasnet for performing Open assessments and impact assessments. Knowledge crystals as integral parts of models and assessments. Simantics System Dynamics in semantic models. Jupyter notebooks for collaborative model development. Wikidata, Wikipedia as storages of structured data and information.

Laws and regulations
Semantic Finlex contains the whole Finnish legislation and e.g. the decisions by the Supreme Court in a semantic structure.

Methods
Preparation of documents, cocreation, real-time co-editing Several co-editing tools, e.g. Hackpad, MS Office365, Google Docs, Etherpad, Dropbox Paper, MediaWiki and Git. These tools enable the opening of the planning and writing phase of a decision. E.g. the climate strategy of Helsinki was co-created online with Google Docs and Sheets in 2018.

Development and spreading good practices
InnoVillage helps to develop practices faster, when everyone's guidance is available online and can be commented.

Organising systems for information and discussions
Decentralised social networking protocol Activitypub

Stakeholders
Expert services ResearchGate Solved and other expert networks.