The tools of policy formulation : new perspectives and new challenges

It is generally accepted that policy tools and instruments exist at all stages of the policy process (Howlett 2011, p. 22). But as was pointed out in Chapter 1, only some tools and instruments, operating at certain policy stages, have garnered the sustained analytical attention of policy researchers. Policy formulation – a very important but imperfectly understood stage – has certainly been targeted by developers of new tools, ranging from foresight and scenario tools that seek to open up problem framings and conceptualizations, through to tools like cost–benefit analysis (CBA) that seek to recommend preferred policy solutions. Tool developers and policy analysts have also made many normative recommendations on how these and other policy formulation tools should be used (Vining and Weimer 2010; Dunn 2004). But as was made clear in Chapter 1, mainstream policy researchers have largely ignored policy formulation tools, meaning that a lot less is known about how they have actually been utilized in practice. As Howlett et al. (Chapter 8) suggest, policy researchers have long suspected that they probably play some role in structuring policymaking activity, but what that function is remains a largely unexplored research topic. The general aim of this book is to investigate – for the first time – what can be gained by bringing the study of policy formulation tools back into the mainstream of public policy research. We say ‘back into’ because having been a central concern of policy analysis in the 1950s and 1960s, it gradually fell out of fashion and, as Chapter 1 explained, policy researchers turned their attention to the fine detail of a small subset of the policy implementation instruments, namely regulation and taxation. The aim of our final chapter is to draw upon the findings of the empirical chapters to identify some initial conclusions and pinpoint a number of promising new avenues for research on policy formulation tools. Conscious that this


INTRODUCTION
It is generally accepted that policy tools and instruments exist at all stages of the policy process (Howlett 2011, p. 22). But as was pointed out in Chapter 1, only some tools and instruments, operating at certain policy stages, have garnered the sustained analytical attention of policy researchers. Policy formulation -a very important but imperfectly understood stage -has certainly been targeted by developers of new tools, ranging from foresight and scenario tools that seek to open up problem framings and conceptualizations, through to tools like cost-benefit analysis (CBA) that seek to recommend preferred policy solutions. Tool developers and policy analysts have also made many normative recommendations on how these and other policy formulation tools should be used (Vining and Weimer 2010;Dunn 2004). But as was made clear in Chapter 1, mainstream policy researchers have largely ignored policy formulation tools, meaning that a lot less is known about how they have actually been utilized in practice. As Howlett et al. (Chapter 8) suggest, policy researchers have long suspected that they probably play some role in structuring policymaking activity, but what that function is remains a largely unexplored research topic.
The general aim of this book is to investigate -for the first time -what can be gained by bringing the study of policy formulation tools back into the mainstream of public policy research. We say 'back into' because having been a central concern of policy analysis in the 1950s and 1960s, it gradually fell out of fashion and, as Chapter 1 explained, policy researchers turned their attention to the fine detail of a small sub-set of the policy implementation instruments, namely regulation and taxation. The aim of our final chapter is to draw upon the findings of the empirical chapters to identify some initial conclusions and pinpoint a number of promising new avenues for research on policy formulation tools. Conscious that this has the look and feel of a sub-field 'in the making', in the second section we begin by critically reflecting on the typology and definition of tools proposed in Chapter 1. Given the current state of knowledge, we believe it is especially important to engage in basic, foundational activities such as these, otherwise the sub-field will not consolidate quickly enough to support future endeavours. We then analyse all the chapters (2-12) from the perspective of the analytical framework for understanding policy formulation tools, covering actors, venues, capacities and effects.
In the third section, we seek to make sense of this rich empirical detail by drawing on relevant policy theories. In our view, it would be a mistake to develop a dedicated theory of policy formulation tools as this would perpetuate the isolation of the sub-field. A more productive strategy is, as many scholars of policy instruments have finally come to recognize , to build upon and where possible enrich more general policy theoretical frameworks. Unlike tool theories that mostly operate at the micro level, these frameworks allow analysts to move beyond definitions and typologies, towards more conditional explanations of tool choices, capacities and uses. To that end, the third section explains why and how three particular bodies of theory are especially well suited to this task. We show that potentially one of the most valuable functions performed by the theories is to problematize the underlying motive for using the tools in the first place (and hence task(s) to be accomplished). Recall from Chapter 1 that when the tools first began to emerge in the 1950s, they were mainly perceived as a means to harvest information to help decision makers address the substantive aspects of policy problems (Radin 2013, p. 23). Consequently, we start with theories which broadly correspond to this fairly rationalistic and linear conceptualization of policy formulation, before moving onto other, rather different motives and/or tasks. Finally, the last section reflects on what a more systematic approach to examining the tools may add to our collective understanding of -in turn -the tools themselves, policy formulation and policymaking more generally, politics, and finally, the field of policy analysis. Throughout, we pinpoint some critical challenges that are likely to emerge as a new sub-field of policy research of tools coalesces and matures.

Definitions and Typologies
In Chapter 1, we argued that policy formulation tools constitute a particular category of policy tools, which is analytically distinct from the implementing instruments exhaustively catalogued by Salamon (2002) and the procedural instruments identified by Howlett (2000). We defined a policy formulation tool as: a technique, scheme, device or operation (including -but not limited to -those developed in the fields of economics, mathematics, statistics, computing, operations research and systems dynamics), which can be used to collect, condense and make sense of different kinds of policy relevant knowledge to perform some or all of the various inter-linked tasks of policy formulation.
On reflection, we believe that this definition is sufficiently broad to capture all the relevant tools, including, crucially, those developed within both positivist and post-positivist traditions. Below, we dwell a little more on what is meant by the tasks of policy formulation. But for now, it is sufficient to note that a broad definition allows the full range of tools explored in this book to be brought out of the 'back room' and studied in a more politically attuned and comparable fashion.
A broad definition also allowed us to propose a comprehensive typology of the main tool types (see Table 1.1), which maps onto -to quote our definition -'the interlinked tasks of policy formulation'. Crucially, it relates the tool functions as they are often presented -in other words, according to idealized, 'textbook' functions -to the policy formulation tasks that they have potential to be harnessed to in practice. The typology does this by deliberately not, as has often been done in the past, drawing on the 'idealized' policy appraisal steps or the internal specifications of particular tools, both of which assume that the tools are centre-stage. Rather, it attempts to situate tools within an appreciation of what actually goes on in policy formulation.
At the broadest level (and drawing on Chapters 2-7), the various tools do seek to address different policy formulation tasks. For example, scenarios were originally created to explore different visions and objectives, as opposed to recommending a particular policy response, a task for which CBA was designed and appears much better suited. In addition, to the extent that their main task is to collect, condense and make sense of policyrelevant information, there appears to be no significant overlap between policy formulation tools and the main implementing instruments. In fact, they are different entities: policy formulation tools can and are used to assess the impacts of different implementing instruments.
However, when confronted by the rich empirical detail contained in Chapters 2-7, we can appreciate that Table 1.1 misses some important nuances. First, many of the main tool types contain many more subtypes than we originally expected. For example, there are prospective, explorative and descriptive types of scenarios; descriptive, performance and composite types of indicators; and multi-attribute, outranking and interactive forms of MCA. And one of the striking findings of Chapter 2 was that participatory tools are in fact an agglomeration of many different tools and methods. Nonetheless, speaking in favour of Table 1.1, there does not appear to be a significant degree of overlap between the main subtypes.
Second, in spite of this variety, many tools do not simply stand alone as separate and clearly specified entities. Some appear to defy the assumption that their application is necessarily an exclusive, expert-led affair; for instance, scenario tools, CBA and MCA can all be applied in a more or a less participatory fashion. Third, and relating to how tools may -in theory -be applied in practice, some of the more technical, substantive and content-related tools (such as CBA) seem to have relatively 'hard' boundaries, which in turn encourage score cards and other measures of the quality of application. By contrast, the more process-based tools such as scenarios and participatory tools have relatively fuzzy boundaries, with much less agreement on purposes and methods of quality evaluation. For these, the quality of application is even more value-laden a judgement. This could be why some chapters (for example, Chapter 2) have the word 'tools' in the title whereas others (for example, Chapter 7) refer to 'a tool'.
Finally, tools do not necessarily map neatly onto policy formulation tasks; they may be appropriate for different tasks in different ways. To take two examples, the same tools may be used for options assessment and to assist with selecting a policy design, and scenarios can be used to characterize problems as well as clarify objectives. This should not be too surprising: in Chapter 1 we noted that the policy formulation tasks are often interlinked in practice and do not necessarily follow a linear progression. Expecting anything different would be to conflate policy formulation with an idealized conception of policy assessment.
Therefore, on closer inspection, creating a usable typology of formulation tools is not as straightforward as one might imagine. In fact, this difficulty might explain why so many tool developers and users have invested so much (perhaps far too much?) time and effort in debating typologies and toolkits (Chapter 1) of decision support tools. Simply listing the policy formulation tools (as is done in Table 8.2, for example) is not a typology; similarly the distinction between simple, formal and advanced tools (see Chapter 1) does not appear to suffice either (for example, depending on the venue of use, CBA can be practised in all three forms). If used flexibly, therefore -an assumption which we open up a little more belowwe believe that Table 1.1 offers a sufficiently sharp analytical device for organizing and making sense of the main (sub )types, and flagging how they are intended to work in principle. It provides a better way to organize the formulation tools than the broader typologies that have been created to encompass all tools and instruments (such as Hood (1983)). And, crucially, when used alongside the more finely grained typologies that have been developed for the implementing (Salamon 2002) and procedural instruments (Howlett 2000), it draws the observer's attention to some significant differences that have not attracted sufficient discussion in the instruments literature until now.
So far the discussion of Table 1.1 has been about policy formulation tools as they are designed and could theoretically be deployed. In the following subsections we explore -via our analytical framework -how these tasks (or uses) work out in practice.

Actors
The first element of the analytical framework concerns the actors who develop and/or promote particular policy formulation tools. As well as highlighting the critical importance of agency in tool selection and deployment, this element speaks to a broader debate, raised in Chapter 1, about the status and behaviour of the various policy formulators. Across the 11 empirical chapters, three main types of actor appear to have actively promoted and/or developed policy formulation tools: decision makers; knowledge producers and/or providers; and knowledge brokers (Howlett 2011, pp. 31-33).
Decision makers at state and international levels have been assiduous promoters of policy formulation tools, almost since the dawn of policy analysis (Dunn 2004, p. 40). Chapter 7 confirms that states were an early and influential promoter of CBA as the 'cornerstone of modern policy analysis' (Mintrom and Williams 2013, p. 5). CBA was initially developed in the 1930s to take the political heat (and conflict) out of state-planned and funded infrastructure projects such as dams -a role, incidentally, now being reprised in the developing world (Chapter 10). Nowadays, national finance ministries and core executives continue to support the application of indicators and CBA through the publication of rules, statutes and best practice guides (Chapters 7 and 9), under different rhetorical banners including better regulation, administrative modernization and evidence-based policymaking. Governmental actors also work within international organizations such as the OECD and scientific bodies like the Intergovernmental Panel on Climate Change to share best practices on many tools, including scenarios (Chapter 3), indicators (Chapter 4) and CBA (Chapter 7). The research arm of the European Commission has directly funded many complex computer models (Nilsson et al. 2008) and taken active steps to ensure they are more heavily utilized in formalized systems of policy-level appraisal (Chapter 9). Chapter 9 identifies the policy officials in line ministries that undertake such appraisal as both potential users and promoters of the tools. We explore their motives for doing these things below.
Under the category of knowledge producers and/or providers, the chapters identify a myriad of actors, in state and non-state settings, who variously: • • Invent tools and numerous variants thereof (for example, academics and technical officials in state bureaucracies); • • Refine and update them (for example, scenario developers); • • Provide the policy-relevant knowledge that is fed into policy formulation activities (for example, statisticians, policy specialists and special advisers).
Academics have constituted a notable source of support for tools. Initially it was economists with strong technical skills (Mintrom and Williams 2013, p. 4) who were in the vanguard, but then other disciplines fed a growing supply of tools such as indicators, MCA and computer modelling. Participatory approaches have emerged, very much out of the postpositivist critique of the policy sciences (Chapter 2). Tools, therefore, have both pragmatic (how to formulate policy) and normative (how policy should be formulated) underpinnings. Industry too has made notable contributions to the development of forecasting, simulation gaming (Chapter 2) and scenario tools (Chapter 3). Consultants and think tanks have also created complex modelling tools such as the influential MARKAL energy model (Chapter 12) as well as scenarios (Chapter 3), and been active disseminators of other tools across government (Chapter 8).
Finally, in some of the chapters, knowledge (or policy) brokers are identified as playing critically important roles. In theory, knowledge brokers are supposed to adopt a more or less neutral role between science and policy. In practice, there are many different subtypes and some chapters emphasize the potentially important role they play in matching tools to policy problems (for example, models to scenarios in processes of integrated assessment -see Chapter 5).
Crucially, all these actors are analytically distinct from the suppliers of policy-relevant knowledge (Radaelli 1995). The tools provide a means to turn knowledge to different policy purposes, that is, a translation function. The growth in policy formulation tools is a tangible manifestation of the broadening and deepening of the policy analysis and advisory community from one dominated by generalist bureaucrats and 'econocrats' (Mintrom and Williams 2013, p. 9), to one comprising a multitude of actors within a more open and plural policy advisory system. Instead of 'speaking truth to power' as Wildavsky (1979) would have it, putting policy formulation tools alongside the actors that utilize them provides a sharper picture of how modern policy analysts seek to 'share the truth with many actors of influence' (Craft and Howlett 2012, p. 85). Adopting a tools perspective on policy formulation -that is, following a particular tool as it is picked up and deployed in different policy formulation venues -arguably offers a new and potentially fruitful way to 'open up the black box' of policy formulation, supplementing the standard methods of following issues or focusing on policy advisory systems.

Venues
The second element of our analytical framework relates to the suggestion that policymakers apply tools in policy formulation venues, defined on the basis of their location (internal and external to government) and the sources of knowledge that they draw upon (official versus unofficial). In Chapter 1, we sought to open up two lines of potentially productive inquiry. First, by whom, for what purposes and in what form are tools used in particular policy formulation venues? By 'use' we mean that a particular tool has been specifically deployed to inform the formulation of policy, or its contribution has somehow been referenced or otherwise credited in a particular set of policy formulation activities. Second, what factors shape the selection and deployment of particular policy formulation tools?

Venues of use: by whom, for what purposes and in what form are tools used?
By whom have different policy formulation tools been used? In the past, the standard assumption in policy analysis was that it was the state and its constituent organizations that mainly selected and deployed the tools, with a particularly strong preference (according to Meltsner (1976) at least) for the more substantive-technical variants such as models and CBA (see also Chapter 1). In other words, tool use was mainly clustered in the internalofficial quadrant of Figure 1.1. Much later Radin (2013) and others (Nilsson et al. 2008) argued that even in this quadrant, the use of such tools was greatly exaggerated; process-related tools such as checklists and participatory tools were at least as common (see Chapter 8), and in the other three quadrants of Figure 1.1 were likely to be relatively more common.
Chapters 2-12 show that these standard assumptions should indeed now be questioned. Evidently, there are many different actors involved in the policy formulation process, drawing upon and deploying a broad range of tools (in other words, tools are much more widely spread across the four quadrants in Figure 1.1). Nonetheless, the pattern of use across the venues is even more uneven (or 'lumpy') than Meltsner (1976) and Radin (2013) suggested. Chapters 8 and 9 offer a much more detailed insight into the differentiated patterns of uptake. Chapter 8 suggests that in Canada, more substantive-technical tools are more likely to be used in the governmental (as opposed to the NGO) sector, and in the more economically (as opposed to socially and environmentally) focused sectors. That said, even amongst government officials, Radin's suspicion does seem to hold true: government officials are more likely to use tools such as brainstorming, consultation exercises and checklists than more formalized tools such as CBA (see Table 8.2). Chapter 9 examines tool use in the relatively new and formalized venue of impact assessment (in other words, squarely in the top right quadrant of Figure 1.1) and finds a strong variation between countries where tools are hardly used at all, and others where their use is much more the norm. In other words, specific tools do not completely dominate specific venues.
A more general point emerges from many chapters: in practice it can be difficult to determine when a tool has been 'used' because it may not necessarily appear in its 'textbook' form, or be formally documented in a way that researchers can study empirically. The distinction between textbook and 'actual' forms stands out for tools such as CBA, which prescribe clear steps and procedures which are often not followed in practice (Chapter 7). For the less standardized tools, variable use is not simply difficult to measure but is often seen as a virtue -think of the 'contextualization' of modelling tools for example (Chapter 5) or the more exploratory types of participatory tool.
The chapters suggest too that the purposes to which the tools are put in the various policy formulation venues also exhibit a great deal of variation. Purposes can be thought of in at least two distinct senses: vis-à-vis the well-known stages or steps of policy formulation (as in Chapter 1); and in relation to the pre-existing 'design space' (Howlett 2011, p. 141), that is, does it seek a radical or a more incremental departure from the status quo? As regards the former, certain tools appear to be far better suited (and be more heavily used in relation) to certain policy formulation tasks than others. In Chapter 1, the first step was presented as being one of problem characterization (in other words, what is the nature of 'the problem'?). For this, scenarios and public participatory techniques seem to be uniquely well suited. Nevertheless, the more projective forms of modelling and even indicators can be used to -and, according to the chapter authors do -shape problem perceptions. The second step (problem evaluation) is something that scenarios and indicators appear to be better suited to. By contrast the final step (policy design -recommending a mix of policy interventions) is something that CBA and MCA were specifically designed to address, although participatory tools may also play a part in ensuring that the design process remains transparent and/or legitimate. Indicators may be less likely to recommend one single option, but they can be (and indeed are -Chapter 10) used to justify the option that is selected and help to monitor performance over time.
The other way to consider the purposes to which a tool is put is relate it to the pre-existing 'design space'. In other words, does it seek to implement the existing policy regime (comprising an internally consistent set of policy objectives, goals and instruments) (Howlett 2011, p. 142), in a more efficient or cost-effective fashion, or does it seek to stretch the existing design space by incorporating new problem formulations or radically different policy approaches? In many tool-related literatures this is directly comparable to the distinction between policy analysis that 'opens up' debate and that which 'closes it down' (Stirling 2008). Here we come across the normative divide between tool developers whose goal is to 'open up' (see for example the debate in the participatory tools literature -Chapter 2) and those for whom 'selecting the best option' is the overriding priority (economists in particular seem to be the obvious exemplar). In Chapters 2-7, this fundamental difference was repeatedly stated; indeed in the chapter on participation (Chapter 2), the difference between so-called 'differentiation' and 'unification' divides the literature in two. Similarly, politicians may initially be attracted to tools such as indicators to 'open up' debate, but by adopting them may unwittingly end up 'closing down' political debate in a way that 'locks in' extant policy designs (Chapter 11).

Venues of use: what factors shape the selection and deployment of particular tools?
Originally, in the policy instruments literature the choice between tools was regarded as mainly determined by ideological factors (Doern and Phidd 1992). However, this assumption was quickly dropped and researchers set about exploring more specific/conditional factors. These are generally divided into the characteristics of the instruments themselves (whether they open up or close down; whether they match the steps in formulation -see above) and various external factors (actor constellation; situational/contextual conditions such as prevailing institutions; and international factors) (Bähr 2010, p. 3;Peters 2002;Eliadis et al. 2007, p. 40).
The literature on policy formulation tools is still too immature to test these explanations, although the authors of the chapters in Part III were asked to select different tool-venue relationships and explore them from their preferred theoretical vantage points. Nevertheless, taken together the 11 chapters hint at some possible explanations which could, in future, be more systematically tested. A number of attributes characteristic of the tools are cited in several of the chapters. For example, is a tool capable of (or salient to) the main policy formulation tasks to be addressed? A computer model, for example, must be capable of manipulating certain key variables to be deemed worthy of consideration. Similarly, indicators that are measurable, simple and adaptable appear more likely to be taken up than others. The idea, commonplace in the policy instruments literature, that policy tools are in principle substitutable (Hill 2009, p. 178), does not seem as applicable to policy formulation tools.
Regarding factors external to the tools, international factors are noted in several of the chapters, including the perceived need to follow EU requirements (Chapter 9) or align to OECD best practices (Chapter 7)or, in the case of participatory tools (Chapter 2), the relatively weak compulsion to apply them expressed in some international legal agreements. Legalization as a potential driver of tool use is also noted in a number of chapters (including 6 and 7). In the UK and Canada, Chapters 7 and 8 respectively suggest that pressure from ministries of finance lies behind the relative popularity of CBA. By contrast, the use of MCA, indicators and most participatory tools is less likely to be mandatory (Chapters 2 and 6). Consequently, there is a live debate on what can be gained (and also conceded -see Chapter 2) by legislating to force tool use. Finally, the fit between a tool and its external environment (including the policy design space) appears to be a critical determinant of the extent to which they are used in policy formulation. The fit can, of course, be manipulated by any of the actors discussed above.
To conclude, there do appear to be clear and discernible patterns in the way that policy formulation tools are used. Whether one starts with the tools and looks across to the venues (in other words, Chapters 2-7) or explores different combinations of tools in and across particular venues (in other words, Chapters 8-12), the patterns seem to recur and hence in principle seem worthy of further exploration. Indeed, one especially intriguing possibility is that the most significant differentiating factor may eventually be policy type, not venue, something which was not fully captured in Figure 1.1. A number of chapters (including 4, 6, 8 and 9) reveal that certain types of tools are more commonly deployed in relation to particular policy areas and problems (for example, the correlation between modelling/scenarios and areas of scientific uncertainty such as climate change), but there may be others, as the authors of Chapter 9 imply.
The two questions posed at the beginning of this section on 'venues' may appear rather straightforward. They are of course basically congruent with the two questions that Salamon (1989, p. 265) originally posed, namely: what influences the choice of tools? And what policy consequences ('or effects') does this choice have? Indeed, the first of these -the selection of tools -is to a large extent the issue of policy formulation in a nutshell (Howlett -in Hill 2009, p. 176). But they are unlikely to be easy questions to answer; after all, Salamon's intervention has pretty much defined the research agenda in the instruments sub-field for the last 25 years. In a later section, we suggest that the most preferable way to relate these questions to policy formulation tools is to start from a set of sound theoretical bases.

Capacities
Chapter 1 conceived of the relationship between policy capacity and policy formulation tools in three main ways. First, there are the policy-analytic capacities that inhere within each tool; capacities that have already been partially discussed under the subheading of 'venues' above. Thinking more broadly about the main tasks of policy analysis in government -analysing problems, recommending responses, clarifying value choices and underlying assumptions, democratizing and legitimizing (Mayer et al. 2004, section 7) -it is obvious that each one is associated with different policy formulation tools. The more tools that a policymaker can draw upon, then ipso facto the greater her potential policy capacity. In principle, therefore, the presence and availability of policy formulation tools help to expand policy capacities, although we should not automatically assume that the relationship is immediate or unidirectional, as the previous sections have revealed.
Second, the chapters also raise the question of what policy capacities are in turn required by policymakers in order to employ -and perhaps more fundamentally to select -certain policy formulation tools. For example, the more rigidly procedural tools such as MCA and CBA are associated with demands for specialist staff, systems of training and oversight. Where these associated capacities are weak or not present, the utilization of the tool may be less effective than expected (see Chapters 8 and 9 for example). Chapter 7 suggests that one -and only one -of the reasons why benefits are more likely to be omitted in CBA calculations is because of the technical difficulty of accounting for them in situations of concentrated costs and dispersed benefits (a typical situation in many regulatory design situations) (Lowi 1972). Less overtly procedural tools such as scenarios and foresight exercises seem to require the presence of somewhat different capacities. For example, in many countries the application of such tools was institutionalized in central planning bureaus from the 1960s and 1970s (Chapter 3). Similarly, one of the prime movers in the dissemination of indicators has been the very national statistical offices that subsequently produce and report on their implementation. Finally (and as noted above), it may be important that national finance ministries and international organizations such as the World Bank and the OECD appear to have been the most enthusiastic adopters and advocates of CBA. Chapter 1 hints at the presence of a self-replicating logic: these ministries first push for the application of such tools and then use evidence of their patchy performance to justify the need for new capacities, such as training, more staff, and/or more oversight functions. The presence of strong associations between certain existing capacities and the selection of new tool capacities may not, therefore, be necessarily unidirectional (in other words, actors select tools) or open (in other words, there may be some inherent bias towards certain types of tools). We return to this point below. Staying for a moment longer at the more generic level, what a focus on these associated capacities may eventually provide is, for example, a means to understand the effects of deploying particular tools, how they might fit into or seek to stretch the existing 'design space' and so on.
Finally, several chapters open up the potentially very broad (but nonetheless important) question of what factors might conceivably enable or constrain the availability of the capacities associated with particular policy formulation tasks. The fact that critical supporting capacities may not automatically be available in all policy systems is raised in several chapters, but especially 8 and 9. For example, the authors of Chapter 8 on policy capacities in Canada demonstrate that the toolkit used is much larger than that summarized in Chapters 2-7. Moreover, they identify a pattern of increasing sophistication in policy analysis as one moves from the non-governmental sector to the governmental one, and from the less 'economic' units of government to the more economically oriented ones. Chapter 9 paints a similar picture of differentiated use across the EU.
To conclude (and as noted in Chapter 1), the term 'policy capacity' has been in good currency in public administration and institutional analysis for many decades (for a summary, see Weaver and Rockman 1993), but is now enjoying renewed interest in the context of the re-discovery of the state as a powerful agent of governing and a site of policy formulation (Howlett et al. 2014, p. 4;Matthews 2012;Jenkins and Patashnik 2012). What the chapters of this book offer is a different way to think about policy capacities, as well as a source of fresh insights into how patterns of capacity availability affect, and are affected by, the availability and use of certain tools. These relationships appear in a rather different form in developing countries and in complex, multi-level governance situations such as the EU, where capacities are inchoate and/or in a particularly strong state of flux (Jordan and Schout 2006;Hertin et al. 2009). In developing countries with weakly developed policy spheres, policy formulation tools such as CBA are promoted as a means of overcoming long legacies of political clientalism. However, those seeking to transplant policy formulation models unmodified from the OECD to such settings should be aware of the need for them to be underpinned by sufficiently strong capacities. Chapter 10 revealed that these tools were much more likely to be available when independent agencies are given control than when this task is allocated to government.

Effects
Finally, Chapters 2-12 examined what effects, both intended and actual, the policy formulation tools produce when they are employed. The policy instruments literature has been struggling to answer this question, at least for implementation instruments, ever since Salamon (2002, p. 2) noted that each instrument imparts its own unique spin or 'twist' on policy. Not surprisingly, the less mature sub-field of policy formulation tools has much work to do in relation to 'effects'. Indeed, one of the striking findings from the tool-focused literatures summarized in Chapters 2-7 is how few of them have even identified it as a priority research topic. Some literatures (around CBA and computer models for example) have made more progress than others, but in general, the level of critical engagement has been low. More often than not, certain effects have simply been presumed to flow from the selection of particular tools (for example, that using CBA results in the identification of the pareto optimal policy solution).
As noted in Chapter 1, this collective failure probably has much to do with the disciplinary background of the contributors, but it also reflects an entirely understandable desire to stay anchored in the relatively clearcut world of textbooks and typologies. Nonetheless, the chapters do suggest some potentially useful categorizations that could form the basis of future work. For example (and drawing on Turnpenny et al. (2009, p. 648)), a broad distinction can be drawn between 'substantive' effects (the extent to which tools generate change -or work to ensure continuity -in a given policy field) and 'process-based' effects (in other words, system-wide effects which arise from the use of particular tools). A wide array of substantive effects are flagged up in the chapters, ranging from learning around new means to achieve policy goals (predominant amongst tools such as CBA, but also computer modelling) to heuristic-conceptual effects on problem understandings (see for example Chapters 2 and 5). Large-scale, system-wide energy models may play an important role in facilitating adjustments to new 'policy images', through the development of new policy paradigms and policy objectives (Chapter 12). More fundamentally, some tools (for example, participatory backcasting) have been developed with the avowed aim of facilitating 'out of the box thinking', that restructures actor preferences in a profound way. Meanwhile, the procedural effects are potentially also very wide ranging. For example, Chapter 11 argues that indicators help to channel political attentionespecially among overloaded oversight bodies -such that a 'broader critique' of the policy status quo becomes less and less likely. In addition, some participatory tools such as the devil's advocate technique and participatory backcasting have the aim of generating new understandings and uncovering extant political power relationships.
A second important distinction relates to the difference between intended and unintended effects. We have already noted the difference between the 'imagined' effects that the advocates of tools aspire to provide (to use the terminology employed by Atkinson in Chapter 7) and their 'actual use'. In some of the chapters, the unintended effects are presented positively (as new problem framings -see Chapters 2 and 4 for example) whereas in others, they are presented much more negatively (for example 'gaming the system', 'closing down' debate, and nurturing 'reductionist' thinking are all noted in Chapter 4). To a large extent, the difference is one of prior expectations, purposes and ultimately values. Thus, by their very nature, the more procedurally inflexible tools such as CBA appear more prone to performance deficits. But more open, participatory tools can also produce unexpected effects; for example, Chapter 2 recounts how backcasting approaches all too easily entrench political differences and forms of participation. Consequently, the new sub-field of policy formulation research should be careful to pose more probing questions (for example, unexpected by whom and why?) rather than assume that everything which is unexpected is necessarily bad (or the opposite!). Finally, some effects may be extremely difficult to categorize. For example, Chapter 11 tells the story of how, paradoxically, in the case of indicators, 'a set of tools designed to shift the political focus onto outcomes was deployed in a way that resulted in a preoccupation with process'.
To conclude, understanding effects arguably constitutes the biggest analytical challenge of all, but one which the nascent sub-field of policy formulation is beginning to engage with. Chapters 2-12 already suggest that it will require very careful and patient diachronic forms of analysis (cf. Owens et al. 2004), sensitive to the multiple rationalities that motivate actors to use particular tools in the first place. At present, there remains a definitive 'pro-use' bias in the tools literatures (indeed Chapters 5 and 6 explicitly focus on known examples of use). Indeed, the authors of Chapter 2 argue that political elites may be reluctant to explore the potential of more open participatory tools and methods that typically aim at opening up current problem framing and thus imperil their control. Yet experts in policy formulation tools may also be unwittingly sustaining this blind spot, especially if (as seems to be the case for participatory tools and to a lesser extent for indicators) they cannot agree on what their purpose should be, hence the prevalence of very open evaluation criteria that are extremely difficult to apply.

THEORIZING POLICY FORMULATION TOOLS: RE-ASSEMBLING THE PIECES
In this section we explore some of the issues raised above through three different theoretical lenses, with the aim of both grounding the findings within established theoretical traditions, and using the findings to highlight particular gaps in each theory. Both approaches lead to a range of promising new research questions.
If all tools embody an 'implicit political theory' which provides both a raison d'être for policy analysis and a causal account of how it should proceed (Weale 2001, p. 378), then the theory informing many studies and practices of policy formulation is a rational-instrumental one: '[t]he idea is one of a linear process in which a problem exists, information is lacking, [tools] produce information, and the decision maker can eventually decide' (Radaelli 2004, p. 743). Rational theories have constituted such a significant theme of policy analysis since the 1950s that they represent an obvious stepping off point for those wishing to think afresh about policy formulation (Howlett et al. 2014) and other aspects of the 'new' policy design (Howlett and Lejano 2013). In what follows, we therefore start with this theoretical lens before moving onto two very different ones.

Policy Formulation as Rationality
Although policy analysts long ago dispensed with the notion that policymakers are rational in the sense of having very clear and stable views of means and ends, a sense of rationality lives on in many contemporary frameworks such as policy learning, knowledge utilization, evidence-based policymaking and policy instrument selection and use. Selecting tools in a more or less rational fashion to achieve policy formulation goals and tasks clearly corresponds to what Weiss (1979, p. 427) identified as a 'problemsolving model' of knowledge utilization.
Scholars who wish to study policy formulation tools from a more or less rational standpoint will find a number of reasons to do so. First of all, it offers an intuitively straightforward basis on which to typologize tools. Our proposed typology is, after all, inherently rationalistic in its conception of the intersection of means and ends. Although it offers an incomplete guide to tool choices in practice, it may be regarded as a good starting point all the same.
Second (and related to the above), more rational theories offer a means to engage in normative policy analysis (in other words, 'analysis for' policy). To varying degrees of explicitness, rational assumptions pervade the thoughts of those who produce tool handbooks, supply (in other words, develop) tools and/or 'compliance test' their performance (for example, de Ridder et al. 2007, pp. 430-431). In fact, the original purpose of tools was to base decisions on rational arguments and evidence, instead of bargaining and political interests (Chapter 1; see also Turnpenny et al. (2009, p. 644)). Even if this normative ideal is rarely observed, it behoves of policy analysts to explain the divergence. And there is also the question of why rationalism appears to appeal so strongly to politicians and (perhaps in a somewhat different way) to scholars of policy design, on which more below.
Third, the rationality or otherwise of policy formulation also encourages scholars to think about tool choices in and across different venues. Were rationality dominant, we would expect policy formulators to select tools to match problem types and policy formulation tasks, rather than repeatedly rely upon the same tool (Linder and Peters 1989, p. 37). Different actors may prioritize different kinds of knowledge depending on, among other things, their core preferences and whether they are in the public, private or third sectors. Furthermore, actors operating in venues at different levels of governance might be expected to seek out different types of knowledge. For example, actors at EU and UK levels of decision making appear to seek out a more strategic overview of drivers and impacts (Turnpenny et al. 2014), whereas those charged with implementing policy at the 'street' level tend to be more heavily influenced by their client groups (Haines-Young and Potschin 2014).
Of course politics repeatedly intrudes into the operation of all toolseven the most explicitly 'rational' ones such as CBA. One of the most active debates in the CBA literature (see Chapters 7 and 10), is around the apparent asymmetry between cost and benefit predictions ex ante versus ex post. Is this genuinely the product of 'appraisal optimism' amongst policy analysts (Chapter 7), or because of special interests crying wolf over cost estimates? Although specific examples of policy formulation may not follow a rational-instrumental form, many of the chapters in Part II nonetheless reveal a surprisingly strong element of purposiveness in the selection and use of policy tools in general. Many detect a rather limited use of more sophisticated modelling-based tools, which suggests that some actors may be following the (rational) principle of proportionality, that is, using tools only where and when significant impacts and/or high levels of uncertainty are expected.
Fourth, if the main purpose of policy appraisal is to 'make institutions think differently' (Radaelli 2007, p. 3), then policy formulation tools are an obvious means to extend their collective 'regulatory imagination' (Dunlop 2014, p. 215). Rationalism thus encourages analysts to consider what types of tools generate what types of learning in particular venues (Turnpenny et al. 2009, p. 648). There are many types and degrees of policy learning (Dunlop 2014, pp. 210-211), and potentially many different research designs that could be employed to probe them. One approach is to follow Chapter 12 and trace out the use of a single tool across different policy venues. Chapter 12 found that the MARKAL energy model has provided opportunities for (conceptual) learning and helped to rationalize pre-existing policy decisions (in other words, more political uses of knowledge). Causality is, of course, very difficult to pin down in such studies, particularly as regards the more conceptual forms of use which regularly extend over long periods of time. Policy formulators may themselves also unwittingly compound this problem by refusing to reveal sources, especially in relation to the more symbolic and political categories of use. Nonetheless, these analytical challenges -which are well known to scholars of learning and knowledge utilization -will have to be confronted if scholars of policy formulation tools are to move beyond broad brush explanations of selection and adoption couched in Cash et al.'s (2002) terms of 'credibility, salience and legitimacy' (see Chapters 3 and 4).

Policy Formulation as Control
A rather different lens through which to study the interaction of actors, venues, capacities and effects is that of executive oversight and/or political control over non-majoritarian agencies (Turnpenny et al. 2009, p. 645). This idea was originally elaborated with the USA in mind, and has since been tested on the UK by Froud et al. (1998) and on the EU by Radaelli and Meuwese (2010). According to an extensive literature (for a summary, see Thatcher and Stone Sweet (2002)), elected politicians actively promote the use of certain policy formulation tools in order to: • • Provide information on whether departments and agencies, or supranational bodies such as the European Commission, are operating in venues and in ways which damage important political constituencies; • • Prevent these bodies from being captured by vested interests, engaging in overzealous implementation and/or presenting their political masters with a policy fait accompli; • • Build delay into the policy system, thereby permitting greater oversight and ensuring political legitimacy. (Radaelli and de Francesco 2010, p. 284) Several chapters in this book draw on the terminology of political control to highlight certain tool-related tasks and thus explain the development and utilization of certain kinds of tool. The classic example is of course CBA, the use of which is legally prescribed in many countries. Other than indicators (Chapters 4 and 11) and certain types of risk analysis, no other policy formulation tool consistently enjoys such high-level political backing. A focus on political control can, however, also help to explain the emergence of more process-based tools such as forms of participatory assessment. According to Chapter 2, some were originally developed (and are now widely deployed) to deal with 'an angry public'. Indeed focus groups, consensus conferences and forms of brainstorming are used much later in the policy process too, chiefly to secure sufficient public support for the policy option which is eventually selected at the end of the formulation stage.
There are, however, many important questions still to be addressed by those seeking to move the new sub-field in this theoretical direction. First of all, while political control may be the means through which certain tools are imposed on agencies, theories of control do not fully account for why politicians learn about, and over time become committed to, them in the first place. Both the chapters on indicators (Chapter 4) and CBA (Chapter 7) imply that diffusion partly occurs via softer channels of influence, such as guidelines, best practice examples and academic networks (Benson and Jordan 2011). Related to that (and building on the findings of Chapter 9), it is important to explain why only certain countries, policy sectors and/or policy venues are so heavily populated with tools of control such as CBA, whereas in others their use is virtually absent. It should therefore be possible to start with theories of comparative politics such as political control of bureaucracies, and use policy formulation tools as a case study (Turnpenny et al. 2009, p. 647). It will be especially intriguing to try to explain how far the adoption of tools that 'open up' debate and challenge the status quo can be explained using theories of political control.
Second, just because agencies and departments are required to employ formulation tools does not necessarily mean that they will faithfully use them. Chapter 9 suggests that tools constitute 'an incomplete contract' between principals and agents that can be actively shaped by the latter. The notion of 'perfunctory' forms of usage invites further work, perhaps linked to the idea that some agents might be following rituals of verification (Radaelli and de Francesco 2010, p. 282), as manifest in the tendency for some tools to be used in a manner that departs significantly from the guidance in the handbooks and textbooks. One possible explanation is that bureaucrats in the agencies actively resist political control. Another is that they might like to use tools, but lack the policy capacities to do so ). Finally, it might be that all policy formulation tools are prone to suffer some unintended consequences, no matter how much political backing or force they enjoy. This observation is certainly one of the explanations offered by Boswell et al. to account for the use of indicators in the UK (Chapter 11).
Finally, there are opportunities to build links between the rationality and control perspectives in order to explain how policy formulators resist the imposition of political control. Do they, for example, employ the tool (for example, CBA) as required, but in a manner that utilizes knowledge in a politically advantageous way? One of the emerging debates within the CBA community (Chapter 7) is how and why policy knowledge is fed into assessments in a form that suits particular actors (for example, target groups seeking laxer regulation, or eligible regions bidding for greater state spending).

Policy Formulation Tools as Institutions
A third perspective views policy formulation tools as institutions in themselves that over time generate enduring policy feedback effects. In comparison with theories emphasizing control and rationality, this perspective challenges the sense of linearity apparent in many tool literatures and of course our own tool typology. From this perspective, as they are used, tools gradually take on a life of their own. Tools do, as noted above, seem to incorporate a particular logic or view of the world. Those employing them will, therefore, tend to conceive of problems in a way that perpetuates their use. Over time, tools tend to develop 'tool constituencies' that have invested time and resources in furthering their use; a pattern that only becomes fully apparent when their long-term 'careers' are studied over time (Lascoumes and Le Galés 2007, p. 17). To the extent that tools are not politically neutral, this body of theory suggests that they deserve to be treated as causal factors in their own right (Kassim and Le Galés 2010, p. 5;Radaelli and Meuwese 2010). For example, in terms of the choice between tools, technical effectiveness considerations will not necessarily be the dominant criteria; sometimes instruments may determine preferences (not the other way around).
In some respects, this approach corresponds to the self-sustaining logic that appears to have been at work in the way that certain tools have created a need for more policy specialists in government -think, for example, of how the need for skills in CBA has grown (at least relatively) in the last 10 years, as government in general has shrunk (Mintrom and Williams 2013, p. 7). Indeed in several chapters, references are explicitly made to tool constituencies (Voss and Simons 2014) (for example, the 'indicator industry' -Chapter 4), which have a stake in the development of a particular policy formulation approach, as distinct to their commitment to a particular policy objective or level of governance. In Chapter 2 the claim was made that certain participatory tools evolve slowly over time, pushed by particular advocates. Schick (1977, p. 261) was one of the first to raise this point when he argued that the policy analysis community had fragmented into different tool-focused sections that engage in 'tireless tinkering' (Schick 1977, p. 261) with 'their' preferred tools and methods. At the time, he claimed that their main effect was to bewilder policymakers. In fact, the effects may be more complex; they may, for example, open the door to policy influence. Dunlop (2014, p. 212), for example, has noted how certain tools confer legitimacy on (or 'certify') particular knowledge claims made by particular actors. CBA, for example, is well known amongst environmentalists for having a much greater ability to 'clinch' policy debates than other tools (Owens and Cowell 2002). This may explain why some environmentalists actively seek out opportunities to employ such tools to ensure their own knowledge claims are equally valid and hence usable (Dunlop 2014, p. 213).
In Chapter 2, however, a slightly different set of claims was made in relation to tool-specific constituencies. For example, over time participatory conferences and conflict avoidance tools, as well as certain computerized models (see Chapter 12), might develop such a strong set of political backers that they gradually morph into new policy venues. Or advocates of different tools compete for political attention and funding, or even engage in a much deeper ideological battle with one another (see Chapter 2). The manner in which newer tools such as MCA and scenarios have gradually emerged as a reaction to the more mainstream tools such as CBA and models, could conceivably be explained in much the same way.
In comparison to the other two, this perspective has rather mixed theoretical roots, drawing on political sociology (Lascoumes and Le Galés 2007;Voss and Simons 2014), systems thinking (Jordan and Matt 2014), historical institutionalism ) and social constructivism (Hajer 1995). Future work might therefore profitably explore the relationship between actors, venues, capacities and effects in a more precise and systematic fashion. For example, in some situations politicians are assumed to select certain tools to conceal their true motives, whereas in others they appear to do so in order to reveal them (Kassim and Le Galés 2010, p. 10). This suggests that actors may have different and to an extent unique tool preferences -a matter which we considered in section 2 above.
Following the careers of particular tools is unlikely to uncover the specific tool choices at work; but analysing the choice between tools may do (an approach, for example, adopted in Chapters 8 and 9).
Second, and related to that, there do appear to be discernible patterns in the selection and use of tools that seem a lot more functional than this theoretical perspective seems able fully to account for. Indeed, it struggles to account for the appearance of entirely new tools; if self-replication were entirely dominant, the scope for tool innovation would be minuscule. The impression given, however, is that new tools emerge in the wake of crisis events.

CONCLUSIONS, NEW PERSPECTIVES AND NEW CHALLENGES
Over thirty years after Hood (1983) published his landmark book on policy tools, political and academic interest in them remains as high as ever. Many definitions, taxonomies and explanatory theories have been developed. However, public policy researchers have somehow managed collectively to overlook an entire class of policy-relevant tools. To policy instruments and procedural tools, we should now add the 'new' sub-category of policy formulation tools. In the 1970s and 1980s, certain types of policy formulation tool fell out of academic and political fashion and many observers assumed -understandably -that they were no longer relevant or could even be quietly forgotten.
Having looked -as we have done for the first time in this book -across the main types of policy formulation tool, we can confidently conclude that they are not in decline and nor have they been consigned to the dusty shelves of Self's (1981) backroom. On the contrary, they have expanded in number and their use has multiplied across many different venues. Recalling Salamon's argument that there has been a 'massive proliferation' in the tools of government, policy researchers should appreciate that the revolution in tool use was actually even more 'remarkable' than he claimed (Salamon 2002, p. 609). Why? Because he neglected to add the tools of policy formulation to his stock take.
Nevertheless, the existing literatures on policy formulation tools remain fragmented, not only across the main tool types but also different disciplines. For policy analysts, the divide between those tool experts seeking to pursue research 'on policy' and those preferring to undertake analyses 'for policy' seems even more pronounced than in other comparable subareas of policy analysis such as policy instruments. Indeed, the chapters of this book have more fully revealed that the debate amongst the policy analysts about tools can, on occasions, be as heated as that relating to policy goals and objectives (Mintrom and Williams 2013, p. 13), relating to both technical matters such as definitions and typologies, but also extending to more fundamental ontological and normative matters.
The tools summarized in the chapters of this book are very different to the ones that emerged in earlier eras, reinforcing the need for a fresh look. Indeed many have emerged out of, and been actively informed by, the critique emerging from the democratic theorists and the post-positivists. This book seeks to reinvigorate our understanding by drawing them back into the mainstream. For this, analysts require common concepts, parsimonious definitions and usable taxonomies. In this and the opening chapter we have sought to supply and then critically reflect on all three. We now invite readers to apply, test and critique them, perhaps using the theoretical perspectives outlined above; perspectives that we feel should, in time, be more fully integrated into broader theories of the policy process. Of course at the level of specific tools, debate about definitions, typologies and purposes will doubtless continue. We see that as a healthy sign, but believe that agreement at the broader level is now needed to generate a common and hopefully more fruitful research agenda, perhaps organized around our framework of actors, venues, capacities and effects.
What stands to be gained by embarking on a more systematic approach to the study of policy formulation tools? In Chapter 1 we suggested that there is potentially much more to add to our collective understanding of the tools themselves which, as repeatedly noted throughout this book, have often been studied in a rather isolated, static and descriptive manner. At the time of this writing it is very difficult to answer questions about tool choices and effects that Salamon challenged scholars of policy instruments to address many decades ago. It is also very difficult to work out how policy formulation tools interact with other tools and instruments (Howlett 2011, p. 27). Thinking more generally about forms of analyses for policy, the policy formulation tools literature has much ground to make up in relation to prescriptive advice on the selection and mixing of tools. At present there are no maxims (Howlett et al. 2014) of the type found in the policy instruments literature (for example, escalate slowly up the pyramid of intervention) or meta-tools to inform the design of tool packages. Clearly, inconsistencies between some tool pairings are more obvious than between others. MCA and participatory approaches do seem to mix more freely with one another than, for example, CBA and scenarios. But there is plenty of fresh work to be done on whether and indeed why this might be the case.
Second, in Chapter 1 we argued that a renewed focus on policy formulation tools can add to our collective understanding not only of policy formulation but public policy more generally. Of all the stages of the policy processes, policy formulation is arguably the one we know the least about. It is often complex, fluid and usually much less accessible to public scrutiny than other stages. Looking through the prism of tools is methodologically advantageous in the sense that, drawing on Hood (1983, pp. 115-131), it reduces complexity and permits comparisons to be made more easily across time, and between different policy areas and political systems. The chapters in this book have, we think, shown the potential of a 'tools approach' to shed new light on these issues. They confirm that the tools play a significant role in structuring policymaking activity and in determining the content of policy outputs and thus policy outcomes. The chapters also suggest that the tools are vital aspects not only of policy design, but also the nascent debate about policy capacities (Howlett 2011, p. 146).
Third, we have suggested that studying policy formulation tools more intensively may -paradoxically -add something to our collective understanding of politics; 'paradoxically' because the tools were originally conceived as a means to take the political heat out of policymaking. The chapters of this book have confirmed that the politics around policy formulation tools are, by their nature, often more subtle than those emerging around policy instruments, but they are no less important for it. Moreover, the chapters have shown that even if tool choices seldom make political headlines, over time they can have profound effects on the way problems are conceptualized and policy recommendations made to decision makers. They have also more fully revealed how they are used to control line agencies and depoliticize areas of policymaking. Policy formulation is the point in the policy process when the political commitment to 'do something' expressed during agenda setting runs into the constraints and the opportunities of the status quo. Long ago, Dahl and Lindblom (1953, pp. 16-18) argued that instruments lie at the very heart of the public policy process. Rather than debating in terms of grand ideologies such as capitalism and socialism, policy actors communicate in the more technical language of regulations, taxes and so on. A tools perspective offers insights into governing beyond formal rules, administrative systems and constitutions. Academics, scientists, policy consultants and think tanks were shown to play a determinative role. Matters of policy formulation are often not publicly debated, but the tools used and the effects they eventually generate undeniably involve questions of political power and the distribution of social values, and as such deserve to be a subject of analysis in their own right.
Finally, the chapters have suggested that bringing the tools into the mainstream of policy research may also help us to learn more about ourselves and our multidisciplinary field of policy analysis. To tell the story of policy formulation tools is to tell of the emergence and professionalization of policy analysis. From their origins in the 1940s, the popularity of the tools waxed and waned. They were originally developed by economists, statisticians and systems analysts to 'speak the truth to power' (Goodin et al. 2006, p. 7). As we pointed out in Chapter 1, their designers and advocates fell short in delivering upon their undoubted promise and they were conveniently forgotten about by many public policy scholars. When and why this happened is a story that deserves to be told as part of the broader 'turn back' to policy formulation tools.