Summaries of AI Policy Resources

The resources below include papers, reports, and articles relevant to AI policy debates.

They are organized chronologically in the following seven (interrelated) categories:

You can find more information about national and international AI strategies and policies here, and an overview of AI policy challenges and recommendations here.

This is intended to be an educational resource. The Future of Life Institute led the development of the Asilomar AI Principles, but does not necessarily endorse the other policy principles or recommendations listed below.

This page is a work in progress and will continue to be updated. If you have feedback or information about reports that should be included here, please send a short description to jessica@futureoflife.org.

AI Policy Overviews and Recommendations

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Miles Brundage et al.; arXiv; April 2020

“This report suggests various steps that different stakeholders in AI development can take to make it easier to verify claims about AI development, with a focus on providing evidence about the safety, security, fairness, and privacy protection of AI systems. Implementation of such mechanisms can help make progress on the multifaceted problem of ensuring that AI development is conducted in a trustworthy fashion.”

Toward AI Security: Global Aspirations for a More Resilient Future
Jessica Cussins Newman; Center for Long-Term Cybersecurity; February 2019

“This report uses the lens of global AI security to investigate the robustness and resiliency of AI systems, as well as the social, political, and economic systems with which AI interacts. The report introduces a framework for navigating the complex landscape of AI security, visualized in the AI Security Map. This is followed by an analysis of AI strategies and policies from ten countries, using the AI Security Map to identify areas of convergence and divergence. This comparative exercise highlights significant policy gaps, but also opportunities for coordination and cooperation among all surveyed nations. Five recommendations are provided for policymakers around the world who are hoping to advance global AI security.”

AI Governance: A Research Agenda
Allan Dafoe; Governance of AI Program; Future of Humanity Institute; University of Oxford; August 2018

“Artificial intelligence (AI) is a potent general purpose technology. Future progress could be rapid, and experts expect that superhuman capabilities in strategic domains will be achieved in the coming decades. The opportunities are tremendous, including advances in medicine and health, transportation, energy, education, science, economic growth, and environmental sustainability. The risks, however, are also substantial and plausibly pose extreme governance challenges. These include labor displacement, inequality, an oligopolistic global market structure, reinforced totalitarianism, shifts and volatility in national power, strategic instability, and an AI race that sacrifices safety and other values. The consequences are plausibly of a magnitude and on a timescale to dwarf other global concerns. Leaders of governments and firms are asking for policy guidance, and yet scholarly attention to the AI revolution remains negligible. Research is thus urgently needed on the AI governance problem: the problem of devising global norms, policies, and institutions to best ensure the beneficial development and use of advanced AI.”

Artificial Intelligence and International Affairs: Disruption Anticipated
M. L. Cummings, Heather M. Roff, Kenneth Cukier, Jacob Parakilas and Hannah Bryce; Chatham House; June 2018

“This Chatham House report examines some of the challenges for policymakers, in the short to medium term, that may arise from the advancement and increasing application of AI. It is beyond the scope of the report to offer a fully comprehensive set of predictions for every possible ramification of AI for the world. Significant areas not addressed here – including medicine, public health and law – might be fundamentally transformed in the next decades by AI, with considerable impacts on the processes of the international system. Furthermore, towards the end of the process of compiling the report, public attention has increasingly turned to the possibility of AI being used to support disinformation campaigns or interfere in democratic processes. We intend to focus on this area in follow-up work.”

The AI Shift: Implications For Policymakers
Sarah Villeneuve and Nisa Malli; The Brookfield Institute for Innovation + Entrepreneurship, Ryerson University; May 2018

“On March 23rd, 2018, the Brookfield Institute for Innovation + Entrepreneurship (BII+E), with the Ontario government’s Policy Innovation Hub, hosted a one-day conference, AI + Public Policy: Understanding the shift. This event was among the first of its kind in Canada, designed to develop a shared understanding of the core technical concepts and historical context of AI among multi-sectoral participants, and to encourage deliberation on the cross-cutting challenges and public policy implications of this evolving technology. This report 1) Provides a summary of the insights from speakers and participants throughout the event. 2) Explores the challenges and opportunities that AI poses to society. 3) Dives deeper into the public policy implications of AI and the role of government in this evolving technical landscape.”

Regulating Artificial Intelligence Proposal for a Global Solution
Olivia J. Erdelyi and Judy Goldsmith; Association for the Advancement of Artificial Intelligence; 2018

“Given the ubiquity of artificial intelligence (AI) in modern societies, it is clear that individuals, corporations, and countries will be grappling with the legal and ethical issues of its use. As global problems require global solutions, we propose the establishment of an international AI regulatory agency that — drawing on interdisciplinary expertise — could create a unified framework for the regulation of AI technologies and inform the development of AI policies around the world. We urge that such an organization be developed with all deliberate haste, as issues such as cryptocurrencies, personalized political ad hacking, autonomous vehicles and autonomous weaponized agents, are already a reality, affecting international trade, politics, and war.”

AI In The UK: Ready, Willing And Able?
Select Committee on Artificial Intelligence; UK Parliament – House of Lords; April 2018

“Our inquiry has concluded that the UK is in a strong position to be among the world leaders in the development of artificial intelligence during the twentyfirst century. Britain contains leading AI companies, a dynamic academic research culture, a vigorous start-up ecosystem and a constellation of legal, ethical, financial and linguistic strengths located in close proximity to each other. Artificial intelligence, handled carefully, could be a great opportunity for the British economy. In addition, AI presents a significant opportunity to solve complex problems and potentially improve productivity, which the UK is right to embrace. Our recommendations are designed to support the Government and the UK in realising the potential of AI for our society and our economy, and to protect society from potential threats and risks.”

European AI Landscape
Charlotte Stix; European Commission; April 2018

“Europe has a leading edge in artificial intelligence (AI) and robotics. This workshop report describes activities being carried out in the field of AI in different Member States and in some of the countries associated to Horizon 2020. Learn about the academic, industrial and funding ecosystems, and find out more about the various governmental initiatives and strategies related to AI.”

Here’s how the US needs to prepare for the age of artificial intelligence
Will Knight; MIT Technology Review; April 2018

“Government indifference toward AI could let the US lose ground to rival countries. But what would a good AI plan actually look like?”

Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability
Dillon Reisman, Jason Schultz, Kate Crawford, Meredith Whittaker; AI Now; April 2018

“Public agencies urgently need a practical framework to assess automated decision systems and to ensure public accountability”

A National Machine Intelligence Strategy for the United States
William A. Carter, Emma Kinnucan, and Josh Elliot; CSIS and Booz Allen Hamilton; March 2018

“This strategy should have two overarching goals. The first is to promote the safe and responsible development of MI technology by funding long-term R&D where the private sector is not incentivized to invest, developing a workforce for the MI age, creating dynamic commercial markets for MI technologies to capture the innovation of the private sector, and proactively managing the risks and disruptions that MI will bring. The second goal is to maintain U.S. leadership in MI by reinforcing our innovation base and establishing strategic partnerships to leverage the comparative advantages of our allies and lead the development of global MI governance.”

For A Meaningful Artificial Intelligence: Towards A French And European Strategy
Cédric Villani; French Parliament; March 2018

This report focuses on how Europe can accomplish six interconnected objectives: “1) Building a Data-Focused Economic Policy 2) Promoting Agile and Enabling Research and 3) Assessing the Effects of AI on the Future of Work and the Labor Market, and Experiment Adequate Policy Responses 4) Artificial intelligence Working for a More Ecological Economy 5) Ethical Considerations of AI 6) Inclusive and Diverse AI”

Artificial Intelligence: The Race Is On The Global Policy Response To AI
FTI Consulting; February 2018

“As this snapshot shows, governments across the globe are now shifting gear, taking a more active role. Until recently, the predominant approach by governments on AI policy issues focused on providing funding opportunities for research. Now they are setting regulatory boundaries (e.g. data protection in the EU) and providing incentives (e.g. “Made in China 2025” industrial plan) in an effort to protect the socio-economic fabric of their societies while also creating certainty for businesses. As with all things new, regulatory attention and intervention can be a blessing or a curse.”

Artificial Intelligence and Foreign Policy
Ben Scott, Stefan Heumann and Philippe Lorenz; Stiftung Neue Verantwortung; January 2018

“This paper seeks to provide a foundation for planning a foreign policy strategy that responds effectively to the emerging power of AI in international affairs. The developments in AI are so dynamic and the implications so wide-ranging that ministries need to begin engaging immediately. That means starting with the assets and resources at hand while planning for more significant changes in the future. Many of the tools of traditional diplomacy can be adapted to this new field. While the existing toolkit can get us started, this pragmatic approach does not preclude thinking about more drastic changes that the technological changes might require for our foreign policy institutions and instruments.”

Government AI Readiness Index
Richard Stirling, Hannah Miller, Emma Martinho-Truswell; Oxford Insights; January 2018

“Artificial intelligence (AI) will revolutionise public service delivery. Governments around the world are starting to see its enormous potential: for their economies, their societies, and their own public services. Until now, most research has focused on the technical implementation and likely impacts of AI. We asked a different question: how well-placed are the national governments in the OECD to take advantage of the benefits of automation in their operations? We have created a world-first Government AI Readiness Index (full ranking below) to capture the current capacity of OECD governments to absorb – and exploit – the innovative potential of AI.?”

Ethically Aligned Design, v2
IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems; December 2017

“The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems) is launching the second version of Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, (EADv2) the most comprehensive, crowd-sourced global treatise regarding the Ethics of Autonomous and Intelligent Systems available today.”

AI NOW 2017 Report
Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, Kate Crawford; New York University; December 2017

“Building on the inaugural 2016 report, The AI Now 2017 Report addresses the most recent scholarly literature in order to raise critical social questions that will shape our present and near future. A year is a long time in AI research, and this report focuses on new developments in four areas: labor and automation, bias and inclusion, rights and liberties, and ethics and governance. We identify emerging challenges in each of these areas and make recommendations to ensure that the benefits of AI will be shared broadly, and that risks can be identified and mitigated.”

Artificial Intelligence: A Policy-Oriented Introduction
Anne Bowser, Michael Sloan, Pietro Michelucci, Eleonore Pauwels; The Wilson Center; November 2017

“Recognizing the increasing integration of technology in society, this policy brief grounds the present excitement around AI in an objective analysis of capability trends before summarizing perceived beneﬁts and risks. It also introduces an emerging sub-ﬁeld of AI known as Human Computation, which can help achieve future AI capabilities by strategically inserting humans in the loop where pure AI still falls short. Policy recommendations suggest how to maximize the beneﬁts and minimize the risks for science and society, particularly by incorporating human participation into complex socio-technical systems to ensure the safe and equitable development of automated intelligence.”

Accountability of AI Under the Law: The Role of Explanation
Finale Doshi-Velez, Mason Kortz; Berkman Klein Center at Harvard university; November 2017

“[We] briefly review current societal, moral, and legal norms around explanation, and then focus of the different contexts under which explanation is currently required under the law. We find that there exists great variation around when explanation is demanded, but there also exist important consistencies: when demanding explanation from humans, what we typically want to know is whether and how certain input factors affected the final decision or outcome.”

Artificial Intelligence Policy: A Primer and Roadmap
Ryan Calo; Social Science Research Network; October 2017

“Talk of artificial intelligence is everywhere. People marvel at the capacity of machines to translate any language and master any game. Others condemn the use of secret algorithms to sentence criminal defendants or recoil at the prospect of machines gunning for blue, pink, and white-collar jobs. Some worry aloud that artificial intelligence will be humankind’s “final invention.” “This essay, prepared in connection with UC Davis Law Review’s 50th anniversary symposium, explains why AI is suddenly on everyone’s mind and provides a roadmap to the major policy questions AI raises. The essay is designed to help policymakers, investors, technologists, scholars, and students understand the contemporary policy environment around AI at least well enough to initiate their own exploration.”

Destination unknown: Exploring the impact of Artificial Intelligence on Government September 2017 Working Paper
Joel Tito; Center for Public Impact; September 2017

“This paper is intended to help government officials navigate the unfamiliar terrain of this new set of technologies. It is broken down into six sections. Section One defines and introduces AI and machine learning algorithms (MLAs). Section Two articulates how AI can be deployed to help existing government functions. Section Three examines how AI can be effective in changing policymaking processes. Section Four outlines the challenges for governments that may prevent them from capturing the benefits of AI. Section Five describes the risk for governments of failing to act and of getting things wrong. Section Six proposes a set of recommendations to place governments in a position to capture the benefits of AI.”

A Next Generation Artificial Intelligence Development Plan
China State Council; July 2017

“The rapid development of artificial intelligence (AI) will profoundly change human society and life and change the world. To seize the major strategic opportunity for the development of AI, to build China’s first-mover advantage in the development of AI, to accelerate the construction of an innovative nation and global power in science and technology, in accordance with the requirements of the CCP Central Committee and the State Council, this plan has been formulated.”

Data management and use: Governance in the 21st century
British Academy, The Royal Society; June 2017

“A set of high-level principles is needed to visibly shape all forms of data governance and ensure trustworthiness and trust in the management and use of data as a whole. The promotion of human flourishing is the overarching principle that should guide the development of systems of data governance. The four principles that follow provide practical support for this overarching principle across the varied ways data is managed and used: 1) protect individual and collective rights and interests 2) ensure that trade-offs affected by data management and data use are made transparently, accountably and inclusively 3) seek out good practices and learn from success and failure 4) enhance existing democratic governance.”

Artificial Intelligence and Machine Learning: Policy Paper
Internet Society; April 2017

“In this paper, we seek to provide an introduction to AI to policymakers and other stakeholders in the wider Internet ecosystem. The paper explains the basics of the technology behind AI, identifies the key considerations and challenges surrounding the technology, and provides several high-level principles and recommendations to follow when dealing with the technology.”

Policy Desiderata in the Development of Superintelligent AI
Nick Bostrom, Allan Dafoe, Carrick Flynn; Future of Humanity Institute, Oxford University; Yale University; 2017

“Machine superintelligence could plausibly be developed in the coming decades or century. The prospect of this transformative development presents a host of political challenges and opportunities. This paper seeks to initiate discussion of these by identifying a set of distinctive features of the transition to a machine intelligence era. From these distinctive features, we derive a correlative set of policy desiderata—considerations that should be given extra weight in long-term AI policy compared to other policy contexts. We argue that these desiderata are relevant for a wide range of actors (including states, AI technology firms, investors, and NGOs) with an interest in AI policy. However, developing concrete policy options that satisfy these desiderata will require additional work.”

The AI Policy Landscape
Matt Chessen, Brent M. Eastwood, Tyler Prochazka; Medium; March 2017

“This is an evolving collection of information and links about who is doing what in the realm of AI policy, laws and ethics. This list is perpetually under construction. I am furiously going through my research notes to extract topics, people, organizations and resources, but this is an ongoing and time consuming effort. Rather than let the perfect be the enemy of the good, I wanted to post this unfinished resource now and update it continuously.”

Brief Initial Thoughts on Artificial Intelligence Policy
Nick Bostrom; University of Oxford; 2016

“Artificial intelligence (AI) will likely act across a wide range of political, social, and economic dimensions, making it inappropriate to think of AI only as a single issue, but also as a collection of issues. Given the possibilities for path-dependence or lock-in, governments should ensure that short- and medium-term AI policies remain compatible longer-term considerations. We identify five distinct policy areas related to AI:

Data & privacy
Autonomous systems & liability
Automation & unemployment
Military, security, and geopolitical coordination:
Implications of superintelligent AI”

A Roadmap for US Robotics From Internet to Robotics 2016 Edition
University of California San Diego, Carnegie Mellon University, et al; November 2016

“The present document is a summary of the main societal opportunities identified, the associated challenges to deliver desired solutions and a presentation of efforts to be undertaken to ensure that US will continue to be a leader in robotics both in terms of research innovation, adoption of the latest technology and adoption of appropriate policy frameworks that ensure that the technology is utilized in a responsible fashion.”

Preparing For The Future Of Artificial Intelligence
John P. Holdren, Alan Bruce, Ed Felton, Michael Garris, Terah Lyons; Executive Office of the President National Science and Technology Council Committee on Technology; October 2016

“As a contribution toward preparing the United States for a future in which Artificial Intelligence (AI) plays a growing role, we survey the current state of AI, its existing and potential applications, and the questions that are raised for society and public policy by progress in AI. We also make recommendations for specific further actions by Federal agencies and other actors. A companion document called the National Artificial Intelligence Research and Development Strategic Plan lays out a strategic plan for Federally-funded research and development in AI.”

The National Artificial Intelligence Research And Development Strategic Plan
John P. Holdren, Alan Bruce, Ed Felton, Michael Garris, Terah Lyons; Executive Office of the President National Science and Technology Council Committee on Technology; October 2016

“Artificial intelligence (AI) is a transformative technology that holds promise for tremendous societal and economic benefit. AI has the potential to revolutionize how we live, work, learn, discover, and communicate. AI research can further our national priorities, including increased economic prosperity, improved educational opportunities and quality of life, and enhanced national and homeland security. Because of these potential benefits, the U.S. government has invested in AI research for many years. Yet, as with any significant technology in which the Federal government has interest, there are not only tremendous opportunities but also a number of considerations that must be taken into account in guiding the overall direction of Federally-funded R&D in AI. This National Artificial Intelligence R&D Strategic Plan establishes a set of objectives for Federally-funded AI research, both research occurring within the government as well as Federally-funded research occurring outside of government, such as in academia. The ultimate goal of this research is to produce new AI knowledge and technologies that provide a range of positive benefits to society, while minimizing the negative impacts.”

Federal Automated Vehicles Policy: Accelerating the Next Revolution In Roadway Safety
US Department of Transportation, National Highway Traffic Safety Administration; September 2016

“The remarkable speed with which increasingly complex highly automated vehicles (HAVs) are evolving challenges DOT to take new approaches that ensure these technologies are safely introduced (i.e., do not introduce significant new safety risks), provide safety benefits today, and achieve their full safety potential in the future. To meet this challenge, we must rapidly build our expertise and knowledge to keep pace with developments, expand our regulatory capability, and increase our speed of execution. This Policy is an important early step in that effort. We are issuing this Policy as agency guidance rather than in a rulemaking in order to speed the delivery of an initial regulatory framework and best practices to guide manufacturers and other entities in the safe design, development, testing, and deployment of HAVs. In the following pages, we divide the task of facilitating the safe introduction and deployment of HAVs into four sections: 1) Vehicle Performance Guidance for Automated Vehicles 2) Model State Policy 3) NHTSA’s Current Regulatory Tools 4) New Tools and Authorities.”

European Union regulations on algorithmic decision-making and a “right to explanation”
Bryce Goodman, Seth Flaxman; Oxford University; August 2016

“We summarize the potential impact that the European Union’s new General Data Protection Regulation will have on the routine use of machine learning algorithms. Slated to take effect as law across the EU in 2018, it will restrict automated individual decision-making (that is, algorithms that make decisions based on userlevel predictors) which “significantly affect” users. The law will also effectively create a “right to explanation,” whereby a user can ask for an explanation of an algorithmic decision that was made about them. We argue that while this law will pose large challenges for industry, it highlights opportunities for computer scientists to take the lead in designing algorithms and evaluation frameworks which avoid discrimination and enable explanation.”

Smart Policies for Artificial Intelligence
Miles Brundage, Joanna Bryson; August 2016

“We have written this article hoping to inform next steps by governments in formalizing, integrating, and improving AI policy, as well as to inform business leaders and ordinary citizens. As we mentioned, there already exists what we call de facto AI policy – a patchwork of policies impacting the field’s development in myriad ways, though only a few of which are already explicitly oriented toward AI. The key question is not whether AI will be governed, but how it is currently being governed, and how that governance might become more informed, integrated, effective, and anticipatory. As a society, we already know enough about some critical issues, and we could take proactive steps to learn more. As authors, we make recommendations concerning these below. But first, we start by explaining what AI is and reviewing the key components of its de facto policy regime. Then we show how lessons from the governance from other technologies, and early lessons from de facto AI policy, can inform a smart approach to AI policy that continues to foster innovation while safeguarding society as citizens and consumers.”

Draft Report with recommendations to the Commission on Civil Law Rules on Robotics
Mady Delvaux; European Parliament; May 2016

“Whereas now that humankind stands on the threshold of an era when ever more sophisticated robots, bots, androids and other manifestations of artificial intelligence (“AI”) seem poised to unleash a new industrial revolution, which is likely to leave no stratum of society untouched, it is vitally important for the legislature to consider all its implications…Calls for the creation of a European Agency for robotics and artificial intelligence in order to provide the technical, ethical and regulatory expertise needed to support the relevant public actors, at both EU and Member State level, in their efforts to ensure a timely and well-informed response to the new opportunities and challenges arising from the technological development of robotics.”

Robots and robotic devices: Guide to the ethical design and application of robots and robotic systems
The British Standards Institution; April 2016

“This British Standard gives guidance on the identification of potential ethical harm and provides guidelines on safe design, protective measures and information for the design and application of robots. It builds on existing safety requirements for different types of robots; industrial, personal care and medical. This British Standard describes ethical hazards associated with the use of robots and provides guidance to eliminate or reduce the risks associated with these ethical hazards. Significant ethical hazards are presented and guidance given on how they are to be dealt with for various robot applications. Ethical hazards are broader than physical hazards. Most physical hazards have associated psychological hazards due to fear and stress. Thus, physical hazards imply ethical hazards and safety design features are part of ethical design. Safety elements are covered by safety standards; this British Standard is concerned with ethical elements. This British Standard is intended for use by designers and managers, amongst others.”

Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies
Matthew U. Scherer; Harvard Journal of Law & Technology; Spring 2016

“This article will advance the discussion regarding the feasibility and pitfalls of government regulation of AI by examining these issues and explaining why there are, nevertheless, some potential paths to effective AI regulation. Part II will examine the characteristics of AI that present regulatory challenges. Some of these challenges are conceptual, such as how to define artificial intelligence and how to assign moral and legal responsibility when AI systems cause harm. Other challenges are practical, including the inherent difficulties in controlling the actions of autonomous machines, which may render ex post regulation ineffective; the related risk that AI systems will perform actions that are unforeseeable to their designers and operators; and the potential for AI to be developed so clandestinely or diffusely as to render effective ex ante regulation impracticable. Despite these challenges, the legal system’s deep regulatory toolkit and the already large and ever-increasing role of large corporations in AI development mean that effective AI regulation should nevertheless be possible.”

AI Principles

Principled Artificial Intelligence
Jessica Fjeld and Adam Nagy; Berkman Klein Center; January 2020

“The rapid spread of artificial intelligence (AI) systems has precipitated a rise in ethical and human rights-based frameworks intended to guide the development and use of these technologies. Despite the proliferation of these “AI principles,” there has been little scholarly focus on understanding these efforts either individually or as contextualized within an expanding universe of principles with discernible trends. To that end, this white paper and its associated data visualization compare the contents of thirty-six prominent AI principles documents side-by-side. This effort uncovered a growing consensus around eight key thematic trends: privacy, accountability, safety and security, transparency and explainability, fairness and non-discrimination, human control of technology, professional responsibility, and promotion of human values.”

Principles for Accountable Algorithms and a Social Impact Statement for Algorithms
Nicholas Diakopoulos et al.; Fairness, Accountability, and Transparency in Machine Learning (FAT/ML); 2018

“Automated decision making algorithms are now used throughout industry and government, underpinning many processes from dynamic pricing to employment practices to criminal sentencing. Given that such algorithmically informed decisions have the potential for significant societal impact, the goal of this document is to help developers and product managers design and implement algorithmic systems in publicly accountable ways. Accountability in this context includes an obligation to report, explain, or justify algorithmic decision-making as well as mitigate any negative social impacts or potential harms. We begin by outlining five equally important guiding principles.”

The Toronto Declaration: Protecting The Rights To Equality And Non-Discrimination In Machine Learning Systems
Access Now; May 2018

“As machine learning systems advance in capability and increase in use, we must examine the positive and negative implications of these technologies. We acknowledge the potential for these technologies to be used for good and to promote human rights but also the potential to intentionally or inadvertently discriminate against individuals or groups of people. We must keep our focus on how these technologies will affect individual human beings and human rights. In a world of machine learning systems, who will bear accountability for harming human rights? Whilst this Declaration is focused on machine learning technologies, many of the norms and principles included are equally applicable to artificial intelligence more widely, as well as to related data systems. The declaration focuses on the rights to equality and non-discrimination. Machine learning, and artificial intelligence more broadly, impact a wider array of human rights, such as the right to privacy, the right to freedom of expression, participation in cultural life, the right to remedy, and the right to life.”

Asilomar AI Principles
Max Tegmark et al.; Future of Life Institute; January 2017

“These principles were developed in conjunction with the 2017 Asilomar conference. To date, the Principles have been signed by 1274 AI/Robotics researchers and 2541 others. Artificial intelligence has already provided beneficial tools that are used every day by people around the world. Its continued development, guided by the following principles, will offer amazing opportunities to help and empower people in the decades and centuries ahead.”

Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness
Mike Ananny; Science, Technology, & Human Values; 2015

“Part of understanding the meaning and power of algorithms means asking what new demands they might make of ethical frameworks, and how they might be held accountable to ethical standards. I develop a definition of networked information algorithms (NIAs) as assemblages of institutionally situated code, practices, and norms with the power to create, sustain, and signify relationships among people and data through minimally observable, semi autonomous action. Starting from Merrill’s prompt to see ethics as the study of ‘‘what we ought to do,’’ I examine ethical dimensions of contemporary NIAs. Specifically, in an effort to sketch an empirically grounded, pragmatic ethics of algorithms, I trace an algorithmic assemblage’s power to convene constituents, suggest actions based on perceived similarity and probability, and govern the timing and timeframes of ethical action.”

Principles of robotics
Vivienne Parry et al.; UK Engineering and Physical Sciences Research Council; September 2010

“In September 2010, experts drawn from the worlds of technology, industry, the arts, law and social sciences met at the joint EPSRC and AHRC Robotics Retreat to discuss robotics, its applications in the real world and the huge amount of promise it offers to benefit society. As we consider the ethical implications of having robots in our society, it becomes obvious that robots themselves are not where responsibility lies. Accordingly, rules for real robots, in real life, must be transformed into rules advising those who design, sell and use robots about how they should act. The meeting delegates devised such a set of “rules” with the aim of provoking a wider, more open discussion of the issues.”

Framing the AI Debate

Resisting Reduction: A Manifesto: Designing our Complex Future with Machines, Version 1.0
Joi Ito; MIT Press; November 2017

“Nature’s ecosystem provides us with an elegant example of a complex adaptive system where myriad “currencies” interact and respond to feedback systems that enable both flourishing and regulation. This collaborative model–rather than a model of exponential financial growth or the Singularity, which promises the transcendence of our current human condition through advances in technology—should provide the paradigm for our approach to artificial intelligence. More than 60 years ago, MIT mathematician and philosopher Norbert Wiener warned us that “when human atoms are knit into an organization in which they are used, not in their full right as responsible human beings, but as cogs and levers and rods, it matters little that their raw material is flesh and blood.” We should heed Wiener’s warning.”

Report from the AI Race Avoidance Workshop
Marek Rosa, Olga Afanasjeva, Will Millership; GoodAI, AI Roadmap Institute; October 2017

“It is important to address the potential pitfalls of a race for transformative AI, where: 1) Key stakeholders, including the developers, may ignore or underestimate safety procedures, or agreements, in favor of faster utilization 2) The fruits of the technology won’t be shared by the majority of people to benefit humanity, but only by a selected few

Race dynamics may develop regardless of the motivations of the actors. For example, actors may be aiming to develop a transformative AI as fast as possible to help humanity, to achieve economic dominance, or even to reduce costs of development. There is already an interest in mitigating potential risks. We are trying to engage more stakeholders and foster cross-disciplinary global discussion. We held a workshop in Tokyo where we discussed many questions and came up with new ones which will help facilitate further work.”

Reconciliation between Factions Focused on Near-Term and Long-Term Artificial Intelligence
Seth Baum; Social Science Research Network; May 2017

“This paper argues that the presentist-futurist dispute is not the best focus of attention. Instead, the paper proposes a reconciliation between the two factions based on a mutual interest in AI. The paper further proposes a realignment to two new factions: an “intellectualist” faction that seeks to develop AI for intellectual reasons (as found in the traditional norms of computer science) and a “societalist faction” that seeks to develop AI for the benefit of society. The paper argues in favor of societalism and offers three means of concurrently addressing societal impacts from near-term and long-term AI: (1) advancing societalist social norms, thereby increasing the portion of AI researchers who seek to benefit society; (2) technical research on how to make any AI more beneficial to society; and (3) policy to improve the societal benefits of all AI. In practice, it will often be advantageous to emphasize near-term AI due to the greater interest in near-term AI among AI and policy communities alike. However, presentist and futurist societalists alike can benefit from each others’ advocacy for attention to the societal impacts of AI. A reconciliation between the presentist and futurist factions can improve both near-term and long-term societal impacts of AI.”

Making the AI revolution work for everyone
Nicolas Miailhe and Cyrus Hodes; The Future Society, AI Initiative; 2017

“If the definitional boundaries of Artificial Intelligence (AI) remains contested, experts agree that we are witnessing a revolution. “Is this time different?” is the question that they worryingly argue over when they analyze the socio-economic impact of the AI revolution as compared with the other industrial revolutions of the 19th and 20th centuries. This Schumpeterian wave may prove to be a creative destruction raising incomes, enhancing quality of life for all and generating previously unimagined jobs to replace those that get automatized. Or it may turn out to be a destructive creation leading to mass unemployment abuses, or loss of control over decision-making processes. This depends on the velocity and magnitude of the development and diffusion of AI technologies, a point over which experts diverge widely. Policy-makers need to invest more resources to develop a finer understanding of the very notion and dynamics of the AI revolution. Moreover, societies’ abilities to shape the AI revolution into a “creative destruction” and diffuse its benefits to all will mostly depend on how societies react, both individually and collectively. The solutions are firmly enmeshed in politics.”

A Survey on AI Risk Communication Strategies
Ben Garfinkel, Allan Dafoe, Owen Cotton-Barratt; August 2016

“[The authors] conducted a survey to study the effectiveness of different presentations of the risks posed by artificial intelligence. 900 participants, were presented with randomly chosen descriptions of AI risk and asked for their opinions on the field of AI Safety and on the likely social impacts of AI. Although the sample size was too small to draw firm conclusions, it is clear that some particular presentations of AI risk are more effective than others. In particular, we believe the results suggest that guided thinking exercises, appeals to authority, and analogies to other transformative technologies are likely to be effective, while science-fiction references and vivid or grave disaster scenarios are not.”

Promoting Beneficial AI

On the Promotion of Safe and Socially Beneficial Artificial Intelligence
Seth D. Baum; AI & Society; October 2017

“This paper discusses means for promoting artificial intelligence (AI) that is designed to be beneficial for society. The AI field is focused mainly on building AIs that are more capable, with little regard to social impacts. Two types of measures exist for encouraging a shift towards building beneficial AI. Extrinsic measures impose constraints or incentives on AI researchers to induce them to pursue beneficial AI even if they do not want to. Intrinsic measures encourage AI researchers to want to pursue beneficial AI. Prior research focuses on extrinsic measures. Efforts to promote beneficial AI must consider intrinsic factors by studying the social psychology of AI research communities.”

Life 3.0: Being Human in the Age of Artificial Intelligence
Max Tegmark; Knopf; August 2017

“How will Artificial Intelligence affect crime, war, justice, jobs, society and our very sense of being human? The rise of AI has the potential to transform our future more than any other technology—and there’s nobody better qualified or situated to explore that future than Max Tegmark, an MIT professor who’s helped mainstream research on how to keep AI beneficial.”

Guide to Working in AI Policy and Strategy
Miles Brundage; 80000 Hours; August 2017

“The last few years have seen dramatic growth in the number of people doing technical research to figure out how we can safely program an artificial general intelligence. There is another topic that is just as important and has become relatively more neglected: improving AI policy and strategy. This includes questions like how can we avoid a dangerous arms race to develop powerful AI systems; how can the benefits of advanced AI systems be widely distributed; and how open should AI research be? If we handle these issues badly, it could lead to disaster, even if we can solve the technical challenges associated with controlling a machine intelligence.

We need answers to AI policy and strategy questions urgently because i) implementing solutions could take a long time, ii) some questions are better addressed while AI is less advanced and fewer views/interests on the topic are locked-in, and iii) we don’t know when particular AI capabilities will be developed, and can’t rule out the possibility of surprisingly sudden advances.”

Transparent, explainable, and accountable AI for robotics
Sandra Wachter, Brent Mittelstadt, and Luciano Floridi; Science Robotics; May 2017.

“To create fair and accountable AI and robotics, we need precise regulation and better methods to certify, explain, and audit inscrutable systems.Systems can make unfair and discriminatory decisions, replicate or develop biases, and behave in inscrutable and unexpected ways in highly sensitive environments that put human interests and safety at risk.”

Positively shaping the development of artificial intelligence
Robert Wiblin; 80,000 hours; March 2017

“Rapid progress in machine learning has raised the prospect that algorithms will one day be able to do most or all of the mental tasks currently performed by humans. This could ultimately lead to machines that are much better at these tasks than humans.. Humanity’s superior intelligence is pretty much the sole reason that it is the dominant species on the planet. If machines surpass humans in intelligence, then just as the fate of gorillas currently depends on the actions of humans, the fate of humanity may come to depend more on the actions of machines than our own. Working on a newly recognized problem means that you risk throwing yourself at an issue that never materializes or is solved easily – but it also means that you may have a bigger impact by pioneering an area others have yet to properly appreciate, just like many of the highest impact people in history have done. In what follows, we will cover the arguments for working on this area, and look at the best ways you can contribute.”

Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability
Mike Ananny and Kate Crawford; News media & society; 2016

“Models for understanding and holding systems accountable have long rested upon ideals and logics of transparency. Being able to see a system is sometimes equated with being able to know how it works and govern it—a pattern that recurs in recent work about transparency and computational systems. But can “black boxes’ ever be opened, and if so, would that ever be sufficient? In this article, we critically interrogate the ideal of transparency, trace some of its roots in scientific and sociotechnical epistemological cultures, and present 10 limitations to its application. We specifically focus on the inadequacy of transparency for understanding and governing algorithmic systems and sketch an alternative typology of algorithmic accountability grounded in constructive engagements with the limitations of transparency ideals.”

Ethically Aligned Design: A Vision for Prioritizing Human Wellbeing with Artificial Intelligence and Autonomous Systems (AI/AS): Version 1
IEEE; December 2016

“To fully benefit from the potential of Artificial Intelligence and Autonomous Systems (AI/AS), we need to go beyond perception and beyond the search for more computational power or solving capabilities AI/AS have to behave in a way that is beneficial to people beyond reaching functional goals and addressing technical problems. This will allow for an elevated level of trust between humans and our technology that is needed for a fruitful pervasive use of AI/AS in our daily lives. By aligning the creation of AI/AS with the values of its users and society we can prioritize the increase of human wellbeing as our metric for progress in the algorithmic age.”

Social Impacts of AI

AI Now 2019 Report
Kate Crawford, et al.; AI Now; December 2019

“This year we saw a wave of pushback, as community groups, researchers, policymakers, and workers demanded a halt to risky and dangerous AI. AI Now’s 2019 report spotlights these growing movements, examining the coalitions involved and the research, arguments, and tactics used. We also examine the specific harms these coalitions are resisting, from AI-enabled management of workers, to algorithmic determinations of benefits and social services, to surveillance and tracking of immigrants and underrepresented communities. What becomes clear is that across diverse domains and contexts, AI is widening inequality, placing information and control in the hands of those who already have power and further disempowering those who don’t. The way in which AI is increasing existing power asymmetries forms the core of our analysis, and from this perspective we examine what researchers, advocates, and policymakers can do to meaningfully address this imbalance.”

Researchers Combat Gender and Racial Bias in Artificial Intelligence
Dina Bass, Ellen Huet; Bloomberg; December 2017

“Companies use AI to predict everything from the credit worthiness to preferred cancer treatment. The technology has blind spots that particularly affect women and minorities.”

The MADCOM future: how artificial intelligence will enhance computational propaganda, reprogram human culture, and threaten democracy… And what can be done about it
Matt Chessen; Atlantic Council; September 2017

“Part I of this paper describes MADCOMs and future risks from their enhanced capabilities; Part II outlines three scenarios exploring the implications for individuals, organizations, and governments; Part III provides recommendations on how the US government, industry, and society should respond to the threats and opportunities posed by foreign actors armed with these new technologies. The three scenarios do not paint a rosy picture, ranging from anarchy in the information environment as MADCOMs dominate online conversations and reality is entirely obscured, to the outbreak of a MADCOMs arms race, to the creation of cognitive security states that preserve global order via a new Internet 2.0.”

AI NOW 2017 Report
Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, Kate Crawford; New York University; December 2017

Physiognomy’s New Clothes
Blaise Agüera y Arcas, Margaret Mitchell and Alexander Todorov; Medium; 2017

“Rapid developments in artificial intelligence and machine learning have enabled scientific racism to enter a new era, in which machine-learned models embed biases present in the human behavior used for model development. Whether intentional or not, this “laundering” of human prejudice through computer algorithms can make those biases appear to be justified objectively.”

An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence
Osonde A. Osoba, William Welser IV; Rand; 2017

“Machine learning algorithms and artificial intelligence systems influence many aspects of people’s lives: news articles, movies to watch, people to spend time with, access to credit, and even the investment of capital. Algorithms have been empowered to make such decisions and take actions for the sake of efficiency and speed. Despite these gains, there are concerns about the rapid automation of jobs (even such jobs as journalism and radiology). A better understanding of attitudes toward and interactions with algorithms is essential precisely because of the aura of objectivity and infallibility cultures tend to ascribe to them. This report illustrates some of the shortcomings of algorithmic decisionmaking, identifies key themes around the problem of algorithmic errors and bias, and examines some approaches for combating these problems. This report highlights the added risks and complexities inherent in the use of algorithmic decisionmaking in public policy. The report ends with a survey of approaches for combating these problems.”

The Risks of Artificial Intelligence to Security and the Future of Work
Osonde A. Osoba, William Wheeler IV; Rand; January 2017

“This Perspective explores the policy implications of the rise of algorithms and artificial intelligence (AI) in two domains of significant importance and public interest: security and employment. These domains are only a subselection of larger set of affected domains identified by a panel of experts. We drill down on the near-to-medium term trends and implications of AI proliferation in these domains. In brief, we highlight the potential for significant disruption due to AI proliferation on issues of cybersecurity, justice (criminal and civil), and labor market patterns. Our discussion of the future of work also presents a novel framework for thinking about the susceptibility of occupations to automation. The Perspective ends with a set of AI policy recommendations informed by the trends we highlight.”

Artificial Intelligence and Life in 2030
Barbara J. Grosz, Russ Altman, Eric Horvitz, et al.; Stanford University, AI100; September 2016

“The One Hundred Year Study on Artificial Intelligence, launched in the fall of 2014, is a long-term investigation of the field of Artificial Intelligence (AI) and its influences on people, their communities, and society. It considers the science, engineering, and deployment of AI-enabled computing systems. As its core activity, the Standing Committee that oversees the One Hundred Year Study forms a Study Panel every five years to assess the current state of AI. The Study Panel reviews AI’s progress in the years following the immediately prior report, envisions the potential advances that lie ahead, and describes the technical and societal challenges and opportunities these advances raise, including in such arenas as ethics, economics, and the design of systems compatible with human cognition. The overarching purpose of the One Hundred Year Study’s periodic expert review is to provide a collected and connected set of reflections about AI and its influences as the field advances. The studies are expected to develop syntheses and assessments that provide expert-informed guidance for directions in AI research, development, and systems design, as well as programs and policies to help ensure that these systems broadly benefit individuals and society.”

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
Cathy O’Neil; 2016

“We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we get a car loan, how much we pay for health insurance—are being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: Everyone is judged according to the same rules, and bias is eliminated. But as Cathy O’Neil reveals in this urgent and necessary book, the opposite is true. The models being used today are opaque, unregulated, and uncontestable, even when they’re wrong. Most troubling, they reinforce discrimination: If a poor student can’t get a loan because a lending model deems him too risky (by virtue of his zip code), he’s then cut off from the kind of education that could pull him out of poverty, and a vicious spiral ensues. Models are propping up the lucky and punishing the downtrodden, creating a “toxic cocktail for democracy.” Welcome to the dark side of Big Data.”

Stuck in a Pattern: Early Evidence on “Predictive Policing” and Civil Rights
David Robinson and Logan Koepke; Upturn; August 2017

“The term “predictive policing” refers to computer systems that use data to forecast where crime will happen or who will be involved. Some tools produce maps of anticipated crime “hot spots,” while others score and flag people deemed most likely to be involved in crime or violence. Though these systems are rolling out in police departments nationwide, our research found pervasive, fundamental gaps in what’s publicly known about them. How these tools work and make predictions, how they define and measure their performance and how police departments actually use these systems day-to-day, are all unclear. Further, vendors routinely claim that the inner working of their technology is proprietary, keeping their methods a closely-held trade secret, even from the departments themselves. And early research findings suggest that these systems may not actually make people safer — and that they may lead to even more aggressive enforcement in communities that are already heavily policed.”

The ethics of algorithms: Mapping the debate
Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi; Big Data & Society; 2016

“In information societies, operations, decisions and choices previously left to humans are increasingly delegated to algorithms, which may advise, if not decide, about how data should be interpreted and what actions should be taken as a result. More and more often, algorithms mediate social processes, business transactions, governmental decisions, and how we perceive, understand, and interact among ourselves and with the environment. Gaps between the design and operation of algorithms and our understanding of their ethical implications can have severe consequences affecting individuals as well as groups and whole societies. This paper makes three contributions to clarify the ethical importance of algorithmic mediation. It provides a prescriptive map to organise the debate. It reviews the current discussion of ethical aspects of algorithms. And it assesses the available literature in order to identify areas requiring further work to develop the ethics of algorithms.”

The Threat of Algocracy: Reality, Resistance and Accommodation
J. Danaher; Philosophy and Technology, January 2016

“One of the most noticeable trends in recent years has been the increasing reliance of public decision-making processes (bureaucratic, legislative and legal) on algorithms, i.e. computer programmed step-by-step instructions for taking a given set of inputs and producing an output. The question raised by this article is whether the rise of such algorithmic governance creates problems for the moral or political legitimacy of our public decision-making processes. Ignoring common concerns with data protection and privacy, it is argued that algorithm-driven decisionmaking does pose a significant threat to the legitimacy of such processes. Modeling my argument on Estlund’s threat of epistocracy, I call this the ‘threat of algocracy’. The article clarifies the nature of this threat, and addresses two possible solutions (named, respectively, “resistance” and “accommodation”). It is argued that neither solution is likely to be successful, at least not without risking many other things we value about social decision-making. The result is a somewhat pessimistic conclusion in which we confront the possibility that we are creating decision-making processes that constrain and limit opportunities for human participation.”

The Future of Employment: how susceptible are jobs to computerisation?
Carl Benedikt Frey, Michael A. Osborne; September 2013

“We examine how susceptible jobs are to computerisation. To assess this, we begin by implementing a novel methodology to estimate the probability of computerisation for 702 detailed occupations, using a Gaussian process classifier. Based on these estimates, we examine expected impacts of future computerisation on US labour market outcomes, with the primary objective of analysing the number of jobs at risk and the relationship between an occupation’s probability of computerisation, wages and educational attainment. According to our estimates, about 47 percent of total US employment is at risk. We further provide evidence that wages and educational attainment exhibit a strong negative relationship with an occupation’s probability of computerisation.”

Safety, Security, and Existential Risks of AI

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Miles Brundage et al.; arXiv; April 2020

Toward AI Security: Global Aspirations for a More Resilient Future
Jessica Cussins Newman; Center for Long-Term Cybersecurity; February 2019

Emerging Disruptive Technologies and Their Potential Threat to Strategic Stability and National Security
Christopher A. Bidwell and Bruce W. MacDonald; Federation of American Scientists; September 2018

“For the national security community, AI has important implications, not only for the impact of the AI technology itself, but also for the combination of AI with other technological developments related to offensive military operations (such as underwater drones, aerial drones, mobile missile launcher locations, antisubmarine warfare, counter-C3I, and the development of swarm tactics). On the offensive side, U.S. military leaders’ and policymakers’ assumptions about the stealthiest of platforms may have to be re-evaluated in light of the technological advances. Adjustments to the nuclear triad may be in order.”

How Might Artificial Intelligence Affect the Risk of Nuclear War?
Edward Geist and Andrew J. Lohn; Security 2040; RAND Corporation; 2018

“Today’s nuclear balance relies on several conditions that may not hold. Progress in computing and data availability are making it possible for machines to accomplish many tasks that once required human effort or were considered altogether impossible. This artificial intelligence (AI) might portend new capabilities that could spur arms races or increase the likelihood of states escalating to nuclear use—either intentionally or accidentally—during a crisis.1 The RAND Corporation convened a series of workshops that brought together experts in AI and nuclear security to explore ways that AI might be a stabilizing—or destabilizing—force by the year 2040.”

Technology Roulette: Managing Loss of Control as Many Militaries Pursue Technological Superiority
Richard Danzig; Center for a New American Security; May 2018

“This report recognizes the imperatives that inspire the U.S. military’s pursuit of technological superiority over all potential adversaries. These pages emphasize, however, that superiority is not synonymous with security. Experience with nuclear weapons, aviation, and digital information systems should inform discussion about current efforts to control artificial intelligence (AI), synthetic biology, and autonomous systems. In this light, the most reasonable expectation is that the introduction of complex, opaque, novel, and interactive technologies will produce accidents, emergent effects, and sabotage. In sum, on a number of occasions and in a number of ways, the American national security establishment will lose control of what it creates.”

Lethal Artificial Intelligence and Change: The Future of International Peace and Security
Denise Garcia; International Studies Review; May 2018

“The development of artificial intelligence and its uses for lethal purposes in war will fundamentally change the nature of warfare as well as law-enforcement and thus pose fundamental problems for the stability of the international system. To cope with such changes, states should adopt preventive security governance frameworks based upon the precautionary principle of international law, and upon previous cases where prevention brought stability to all countries. Such new global governance frameworks should be innovative as current models will not suffice.”

The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Miles Brundage et al.; Future of Humanity Institute, Oxford University; Centre for the Study of Existential Risk, University of Cambridge; Center for a New American Security; Electronic Frontier Foundation; OpenAI; February 2018

“This report surveys the landscape of potential security threats from malicious uses of artificial intelligence technologies, and proposes ways to better forecast, prevent, and mitigate these threats. We analyze, but do not conclusively resolve, the question of what the long-term equilibrium between attackers and defenders will be. We focus instead on what sorts of attacks we are likely to see soon if adequate defenses are not developed.”

Specifying AI safety problems in simple environments
Victoria Krakovna, Jan Leike, Laurent Orseau; Google DeepMind; November 2017

“We present a suite of reinforcement learning environments illustrating various safety properties of intelligent agents. These problems include safe interruptibility, avoiding side effects, absent supervisor, reward gaming, safe exploration, as well as robustness to self-modification, distributional shift, and adversaries. To measure compliance with the intended safe behavior, we equip each environment with a performance function that is hidden from the agent. This allows us to categorize AI safety problems into robustness and specification problems, depending on whether the performance function corresponds to the observed reward function. We evaluate A2C and Rainbow, two recent deep reinforcement learning agents, on our environments and show that they are not able to solve them satisfactorily.”

Artificial Intelligence and National Security
Greg Allen, Taniel Chan; The Belfer Center at Harvard University; July 2017

“Future progress in AI has the potential to be a transformative national security technology, on a par with nuclear weapons, aircraft, computers, and biotech. Each of these technologies led to significant changes in the strategy, organization, priorities, and allocated resources of the U.S. national security community. We argue future progress in AI will be at least equally impactful. Advances in AI will affect national security by driving change in three areas: military superiority, information superiority, and economic superiority. Taking a “whole of government” frame, we provide three goals for U.S. national security policy toward AI technology and provide 11 recommendations.”

The map of x-risk-preventing organizations, people and internet resources
Alexei Turchin; Infographic

Existential Risk Diplomacy and Governance
Sebastian Farquhar, John Halstead, Owen Cotton-Barratt, Stefan Schubert, Haydn Belfield, Andrew, Snyder-Beattie; Global Priorities Project; January 2017

“The first half of this report offers an overview of existential risks. The second half presents three opportunities for humanity to reduce these risks. These were chosen with the help of over 50 researchers and policy-makers out of more than 100 proposals emerged from three workshops at the University of Oxford and the Ministry of Foreign Affairs in Helsinki.”

The Landscape of AI Safety and Beneficence Research: Input for Brainstorming at Beneficial AI 2017
Richard Mallah; Future of Life Institute; January 2017

“Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to research how to maximize these benefits while avoiding potential pitfalls. This document gives numerous examples of research topics aimed at ensuring that AI remains robust and beneficial.”

Strategic Implications of Openness in AI Development
Nick Bostrom; Oxford University, Future of Humanity Institute; 2017

“This paper attempts a preliminary analysis of the global desirability of different forms of openness in AI development (including openness about source code, science, data, safety techniques, capabilities, and goals). Short-term impacts of increased openness appear mostly socially beneficial in expectation. The strategic implications of medium and long-term impacts are complex. The evaluation of long-term impacts, in particular, may depend on whether the objective is to benefit the present generation or to promote a time-neutral aggregate of well-being of future generations. Some forms of openness are plausibly positive on both counts (openness about safety measures, openness about goals). Others (openness about source code, science, and possibly capability) could lead to a tightening of the competitive situation around the time of the introduction of advanced AI, increasing the probability that winning the AI race is incompatible with using any safety method that incurs a delay or limits performance. We identify several key factors that must be taken into account by any well-founded opinion on the matter.”

Clopen AI: Openness in different aspects of AI development
Victoria Krakovna; August 2016

“There has been a lot of discussion about the appropriate level of openness in AI research in the past year – the OpenAI announcement, the blog post Should AI Be Open?, a response to the latter, and Nick Bostrom’s thorough paper Strategic Implications of Openness in AI development.

There is disagreement on this question within the AI safety community as well as outside it. Many people are justifiably afraid of concentrating power to create AGI and determine its values in the hands of one company or organization. Many others are concerned about the information hazards of open-sourcing AGI and the resulting potential for misuse. In this post, I argue that some sort of compromise between openness and secrecy will be necessary, as both extremes of complete secrecy and complete openness seem really bad. The good news is that there isn’t a single axis of openness vs secrecy – we can make separate judgment calls for different aspects of AGI development, and develop a set of guidelines.”

Global Catastrophic Risks 2016
Owen Cotton-Barratt, Sebastian Farquhar, John Halstead, Stefan Schubert, Andrew Snyder-Beattie; Global Challenges Foundation; 2016

“This report addresses one of the most important issues of our age – global catastrophic risk. Over the last decades, behavioural psychology has taught us that, as a species, we are bad at assessing scope. Issues that affect ten people do not intuitively seem ten times more important than those that affect one person. Global catastrophic risks are one area where our scope insensitivity might prove the most dangerous.”

Superintelligence
Nick Bostrom; Oxford University Press; May 2015

“Superintelligence asks the questions: what happens when machines surpass humans in general intelligence? Will artificial agents save or destroy us? Nick Bostrom lays the foundation for understanding the future of humanity and intelligent life. The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. If machine brains surpassed human brains in general intelligence, then this new superintelligence could become extremely powerful—possibly beyond our control. As the fate of the gorillas now depends more on humans than on the species itself, so would the fate of humankind depend on the actions of the machine superintelligence.”

Unprecedented Technological Risks
Nick Beckstead, Nick Bostrom, Niel Bowerman, Owen Cotton-Barrat, William MacAskill, Seán Ó hÉigeartaigh, Toby Ord; Future of Humanity Institute, Oxford University; Centre for the Study of Existential Risk, University of Cambridge; September 2014

“Policy to control [unprecedented technological] risks should aim at:

Decreasing the chance of bad outcomes: For example, a member country could propose to the UN that there should be guidance ensuring intergovernmental transparency and accountability on new potentially dangerous technological development.

Improving our ability to respond if bad outcomes do occur: For example, investment in early-detection monitoring for new pathogens and general-purpose vaccine, antiviral, and antibiotic development.

Improving our current state of knowledge: For example, commissioning a review to provide a detailed assessment of the risks from new technologies and to recommended policies.”

International Cooperation vs. AI Arms Race
Brian Tomasik; Foundational Research institute; December 2013

“There’s a decent chance that governments will be the first to build artificial general intelligence (AI). International hostility, especially an AI arms race, could exacerbate risk-taking, hostile motivations, and errors of judgment when creating AI. If so, then international cooperation could be an important factor to consider when evaluating the flow-through effects of charities. That said, we may not want to popularize the arms-race consideration too openly lest we accelerate the race.”

Racing To The Precipice: A Model Of Artificial Intelligence Development
Stuart Armstrong, Nick Bostrom, Carl Shulman; Future of Humanity Institute; December 2013

“This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first – by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases.”

Our Final Invention: Artificial Intelligence and the End of the Human Era
James Barrat; 2013

“Artificial Intelligence helps choose what books you buy, what movies you see, and even who you date. It puts the “smart” in your smartphone and soon it will drive your car. It makes most of the trades on Wall Street, and controls vital energy, water, and transportation infrastructure. But Artificial Intelligence can also threaten our existence. In as little as a decade, AI could match and then surpass human intelligence. Corporations and government agencies are pouring billions into achieving AI’s Holy Grail―human-level intelligence. Once AI has attained it, scientists argue, it will have survival drives much like our own. We may be forced to compete with a rival more cunning, more powerful, and more alien than we can imagine. Through profiles of tech visionaries, industry watchdogs, and groundbreaking AI systems, Our Final Invention explores the perils of the heedless pursuit of advanced AI. Until now, human intelligence has had no rival. Can we coexist with beings whose intelligence dwarfs our own? And will they allow us to?”

Minimizing Global Catastrophic and Existential Risks from Emerging Technologies Through International Law
Grant S. Wilson; ResearchGate; November 2012

“This Article argues that bioengineering already presents a global catastrophic or existential risk, like from the ability of scientists to engineer an extremely deadly virus, while nanotechnology and artificial intelligence will likely present global catastrophic or existential risks in the near future, such as through the development of nanotechnology super weapons or through misuse of artificial greater-than-human intelligence. Emerging technologies should be regulated at the international level because global catastrophic risks and existential risks have a global effect, so a failure to regulate dangerous emerging technologies in one country might be hugely detrimental to the entire world. However, current international instruments only regulate bioengineering to a small extent and nanotechnology and artificial intelligence essentially not at all.”

Artificial Intelligence as a Positive and Negative Factor in Global Risk
Eliezer Yudkowsky; Machine Intelligence Research Institute, Oxford University; 2008

“I have perforce analyzed the matter from my own perspective; given my own conclusions and done my best to support them in limited space. It is not that I have neglected to cite the existing major works on this topic, but that, to the best of my ability to discern, there are no existing major works to cite (as of January 2006). It may be tempting to ignore Artificial Intelligence because, of all the global risks discussed in this book, AI is hardest to discuss. We cannot consult actuarial statistics to assign small annual probabilities of catastrophe, as with asteroid strikes. We cannot use calculations from a precise, precisely confirmed model to rule out events or place infinitesimal upper bounds on their probability, as with proposed physics disasters. Bu this makes AI catastrophes more worrisome, not less.”

AI Technological Landscape

Artificial Intelligence Index: 2017 Annual Report
Yoav Shoham, Raymond Perrault, Erik Brynjolfsson, Jack Clark, Calvin LeGassick; November 2017

“The first half of the report showcases data aggregated by the AI Index team. This is followed by a discussion of key areas the report does not address, expert commentary on the trends displayed in the report, and a call to action to support our data collection efforts and join the conversation about measuring and communicating progress in AI technology.”

A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy
Seth D. Baum; Global Catastrophic Risk Institute; November 2017

“Artificial general intelligence (AGI) is AI that can reason across a wide range of domains. It has long been considered the “grand dream” or “holy grail” of AI. It also poses major issues of ethics, risk, and policy due to its potential to transform society: if AGI is built, it could either help solve the world’s problems or cause major catastrophe, possibly even human extinction. This paper presents the first-ever survey of active AGI R&D projects in terms of ethics, risk, and policy. A thorough search identifies 45 projects of diverse sizes, nationalities, ethical goals, and other attributes. Most projects are either academic or corporate. The academic projects tend to express goals of advancing knowledge and are less likely to be active on AGI safety issues. The corporate projects tend to express goals of benefiting humanity and are more likely to be active on safety. Most projects are based in the US, and almost all are in either the US or a US ally, including all of the larger projects. This geographic concentration could simplify policymaking, though most projects publish open-source code, enabling contributions from anywhere in the world. These and other findings of the survey offer an empirical basis for the study of AGI R&D and a guide for policy and other action.”

Artificial General Intelligence: Timeframes & Policy White Paper
Allison Duettmann; Foresight Institute; November 2017

“This meeting was initiated by the observation that some researchers’ timelines for AGI arrival were shortening and the perceived increased urgency for drafting potential policy responses to the related arrival scenarios. This report outlines participants’ timeline estimates for the achievement of Artificial General Intelligence and problems associated with arriving at timeline estimates.Rather than focusing on investigating exact timelines in more detail, it is more instructive to consider different high risk scenarios caused by Artificial Intelligence. The main part of the report focuses on three high-risk scenarios, (1) cyber security, (2) near-term AI concerns, and (3) cooperation leading up to Artificial General Intelligence. While some immediate recommendations for further investigation of potential policy responses were made, the meeting’s main intention was not to reach consensus on specific topics but to open up much-needed dialogue and avenues for cooperation on topics of high importance for policy considerations pertaining to Artificial Intelligence.”

Summaries of AI Policy Resources

AI Policy Overviews and Recommendations

AI Principles

Framing the AI Debate

Promoting Beneficial AI

Social Impacts of AI

Safety, Security, and Existential Risks of AI

AI Technological Landscape

Artificial Intelligence

Nuclear

Biotechnology

Climate

Follow Us