How to best use STI policy experimentation to support transitions?

How to achieve this experimentation in practice has been the subject of much discussion within policy, industry, and academic communities. This policy brief highlights examples of what policy experimentation can mean in practice and presents considerations for policymakers moving forward. It is part of the OECD Agenda for Transformative Science, Technology and Innovation Policies, that has highlighted the critical importance of policy experimentation as governments look to scale up new science, technology and innovation approaches that can accelerate sustainability transitions and discontinue those that prove ineffective.

Experimentation in STI policy refers to the deliberate implementation of small-scale and/or temporary policy interventions designed to test the outcomes of new approaches. The goal is to assess whether these interventions should be scaled up if successful or phased out if they do not achieve desired results. Both the experimentation phase and the decision to scale or discontinue are crucial for innovation policy making.

Several STI policy practices exemplify experimentation. They include regulatory sandboxes that create controlled environments where businesses can test innovative products and services under less stringent regulations. These allow for adjustments based on real-world feedback without immediate widespread application. Another common form of experimentation is randomised control trials (RCTs), where participants are randomly assigned to either receive the intervention or be part of a control group, enabling a rigorous assessment of the intervention's causal impact. Both examples aim to foster innovation while mitigating the risks of premature large-scale deployment.

The need for experimentation has grown in view of increasing demands on STI policy. Effectively, the objectives set for STI policies extend to a growing range of socio-economic impacts, including equitable green and digital transition goals. Operationalising this wider set of policy goals calls for experiments with new approaches that have so far been tested only moderately.

Important rationales for experimentation include the following:

• The need to deal with disruptive (digital) technological change: Rapid advancements in digital technologies, illustrated by disruptive AI large language model (LLM) innovations, such as ChatGPT, require policies to protect consumers without blocking innovations that benefit the achievement of socio-economic goals. Testing and experimentation are consequently important.

• The urgency to accelerate green innovation: Addressing pressing climate and environmental challenges requires massive progress in innovations, with little past experience of acceleration to rely on. Gathering experimental evidence on different ways to optimise limited public resources to support green innovation is consequently important.

The following four important principles for STI policy experimentation as illustrated by concrete policy examples are important:

Institutionalising Experimentation: Mainstreaming experimentation in STI policy requires governments to foster a culture where experimentation is both accepted and encouraged. Finland, for example, established the Experimental Finland initiative, which encouraged and supported line ministries to undertake policy experiments; one pilot that was undertaken through initiative, Elements of AI, saw 7,500 people educated in the basics of machine learning within one year of its launch. Another example is the Canadian government’s “Experimentation Direction for Deputy Heads”, which requires public servants allocate a portion of programme funds for experimentation and create clear processes for evaluating and integrating lessons from experiments into new programmes. Horizon 2020 – the EU’s 86 billion USD (EUR 80 billion) 2014-2020 research and innovation funding programme, focused on breakthrough research around key societal challenges - illustrates how modular and adaptable experimentation models can be institutionalised.

 Embedding Flexibility in Experimentation Design: Governments are exploring how to embed iterative learning into their policy implementation and undertake more regular assessments of how and whether STI policy is accelerating sustainability transitions. For example, as part of the Impact Canada Initiative, Canadian policymakers have drawn on behavioural research through the Program for Applied Research and Climate Action to improve the impact of and social reaction to climate policy interventions, which in turn can be leveraged to improve STI policy on critical technology and infrastructure needs for the low-carbon transition such as transport. Not all experiments are successful, and policymakers also need to be able to articulate clear exit strategies and criteria for winding down or redirecting efforts when experiments fail.

 Building Public Sector Capacities and Skills for Experimentation: Policy experimentation for STI can be challenging for the public sector, and officials may need new training and capacities for the public sector to play a role as an incubator and accelerator of new experimental approaches to policy. Curiosity about new ways to design and deliver public services, combined with a user-centric focus on how industry and consumers benefit from them, are important skills for innovation and experimentation in STI policy development. The urgent need for these skills has even led governments to experiment on how to develop them, see for example the US Civic Digital Fellowship, which started in 2017 to recruit the next generation of technologists to work with public agencies.

 Ensuring Robust Monitoring and Evaluation: Achieving systemic STI policy objectives requires more complex monitoring, helping policymakers determine what works, what does not, and why. This is demonstrated by France's "Towards a Chemical Pesticide-Free Agriculture" initiative, which supports breakthrough research into alternative agricultural solutions. This initiative funded ten research projects and employed a real-time evaluation approach, allowing project and programme managers to collaboratively shape and regularly revise their approach based on emerging results. Workshops facilitated by the programme enabled researchers and managers to anticipate the impacts of the research, address potential bottlenecks — such as regulatory hurdles or technical challenges — and adjust the programme as it progressed.

When referring to specific experiments, scaling up often refers to the ways in which an experiment becomes larger in terms of content and remit. This may involve spatial scaling (geographical expansion), content scaling (extending across domains and practices), and/or actor scaling (extending towards different partnerships and actors involved), all of which often involve resource scaling (expansion of funding). In some cases, experiments are more suited to be horizontally diffused – which refers to their replication and reproduction elsewhere. This often involves a different spatial context (e.g. an experiment replicated in a different city) but could also involve a different organisational or institutional context (e.g. replicated by different actors in the same location), or the repetition and transfer of its design and approach within and across sectors (e.g. energy, water, mobility).

The reality of most STI policy experimentation is that, where implemented, experiments are often not scaled when they succeed, while experiments that fail are not necessarily discontinued. This is because STI experiments operate within dense policy environments involving diverse stakeholders with different perspectives and interests. Incentives may not allow for acknowledging failures. Successes may not be fully realised because of limited financial resources or due to complex legal and regulatory processes that come into play as soon as the focus moves to the wider deployment of innovative solutions.

Certain measures can help experimentation to succeed, such as ensuring structures created during experimentation processes are reversible. To the extent possible, experiments should be designed in ways that do not create vested interests, nor imply the necessity of permanent organisational changes, while at the same time leaving such a possibility open should an experiment prove to be successful and replicable. Policymakers should be able to duplicate, iterate and build on experimentation in ways that have organisational implications, but the introduction of a policy experiment should not imply that organisational changes are inevitably part of the process. This is essential to facilitate the phase out of unsuccessful experiments. Examples range from ensuring human resources allocated to operating an initiative can be re-deployed to other initiatives in the case of failure, to creating flexible ways of allocating financial resources where experiments suggest the planned format is not succeeding.

Moreover, deciding on which experiments to upscale and which to discontinue is difficult in a context where a wide range of objectives have been set for STI policies, relating not only to achieving socio-economic impacts but also to realising longer-term green transition, resilience, and inclusivity goals. These are difficult to measure and attribute to specific policy initiatives. What is more, with many objectives set, it is unclear if failure to contribute to any one of them would be sufficient grounds to discontinue, such as, for instance, when initiatives turn out to be economically inefficient but resilience-enhancing. The challenge raises the importance of updating how policymakers assess the successes and failures of a given policy approach, finding ways to embed additional evaluation metrics that can allow a broader range of impacts – both positive and negative – to be studies when considering the continuation, adaptation or cancellation of a policy experiment. Carefully designing and articulating evaluation metrics is critical.

In addition, sharing examples of experimentation – such as the Swedish Challenge Driven Innovation (CDI) programme and Vinnova’s Strategic Innovation Programme (SIP) – can provide important policy lessons for the STI community as it explores the rationale and practical implications of expanding policy experimentation to achieve sustainability transitions. For example, the CDI, through its collaborative, multi-stage funding approach, supports projects that address societal challenges such as sustainability and public health, while the SIP fosters innovation across strategic sectors such as digital transformation and sustainable energy. Peer learning exercises to exchange on those experiences can be a useful way to ensure such learning takes place. Organisations also have to provide the incentives and opportunities to absorb and build on those insights in future policy initiatives.

Finally, STI policymakers should be aware of the risk of “projectification”, or excessive focus on small-scale pilots and projects. This can lead to duplication of efforts, consultation fatigue, and reduce policy coordination capacities. Moreover, many small-scale pilots may prevent scaling as resources and capacities are too thinly spread.

Caroline PAUNOV (✉ caroline.paunov@oecd.org)