How well can we predict the future? In this podcast, Ariel speaks with Anthony Aguirre and Andrew Critch about the art of predicting the future, what constitutes a good prediction, and how we can better predict the advancement of artificial intelligence. They also touch on the difference between predicting a solar eclipse and predicting the weather, what it takes to make money on the stock market, and the bystander effect regarding existential risks.

Anthony is a professor of physics at the University of California at Santa Cruz. He’s one of the founders of the Future of Life Institute, of the Foundational Questions Institute, and most recently of metaculus.com, which is an online effort to crowdsource predictions about the future of science and technology. Andrew is on a two-year leave of absence from MIRI to work with UC Berkeley’s Center for Human Compatible AI. He cofounded the Center for Applied Rationality, and previously worked as an algorithmic stock trader at Jane Street Capital.

The following interview has been heavily edited for brevity, but you can listen to it in its entirety above or read the full transcript here.

Ariel: To start, what are predictions? What are the hallmarks of a good prediction? How does that differ from just guessing?

Anthony: I would say there are four aspects to a good prediction. One, it should be specific, well-defined and unambiguous. If you predict something’s going to happen, everyone should agree on whether that thing has happened or not. This can be surprisingly difficult to do.

Second, it should be probabilistic. A really good prediction is a probability for something happening.

Third, a prediction should be precise. If you give everything a 50% chance, you’ll never be terribly wrong, but you’ll also never be terribly right. Predictions are really interesting to the extent that they say something is either very likely or very unlikely. Precision is what we would aim for.

Fourth, you want to be well-calibrated. If there are 100 things that you predict with 90% confidence, around 90% of those things should come true.

The precision and the calibration kind of play off against each other, but it’s very difficult to be both about the future.

Andrew: Of the properties Anthony said, being specific, meaning it’s clear what the prediction is saying and when it will be settled — I think people really don’t appreciate how psychologically valuable that is.

People really undervalue the extent to which the specificity property of prediction is also part of your own training as a predictor. The last property that Anthony said, being calibration, is not just a property of a prediction. It’s a property of a predictor.

A good predictor is somebody who strives for calibration while also trying to be precise and get their probabilities as close to zero and one as they can.

Ariel: What is the difference between prediction versus just guessing or intuition? For example, knowing that the eclipse will happen in August versus not knowing what the weather will be like yet.

Andrew: The problem is that weather data is very unpredictable, and the locations of planets and moons and stars are predictable. I would say that it’s lack of a reliable model for making the prediction or a reliable method.

Anthony: There is an incredibly accurate prediction of the eclipse this coming August, but there is some tiny bit of uncertainty that you don’t see because we know so precisely where the planets are.

When you look at weather, there’s lots of uncertainty because we don’t have some measurement device at every position measuring every temperature and density of the atmosphere and the water at every point on earth. There’s uncertainty in the initial conditions, and then the physics amplifies those initial uncertainties into bigger uncertainties later on. That’s the hallmark of a chaotic physical system, which the atmosphere happens to be.

It’s an interesting thing that the different physical systems are so different in their predictability.

Andrew: That’s a really important thing for people to realize about predicting the future. They see the stock market, how unpredictable it is, and they know the stock market has something to do with the news and with what’s going on in the world. That must mean that the world itself is extremely hard to predict, but I think that’s an error. The reason the stock market is hard to predict is because it is a prediction.

If you’ve already made a prediction, predicting what is wrong about your prediction is really hard — if you knew that, you would have just made that part of your prediction to begin with. That’s something to meditate on. The world is not always as hard to predict as the stock market. I can predict that there’s going to be a traffic jam tomorrow on the commute from the East Bay to San Francisco, between the hours of 6:00 a.m. and 10:00 a.m.

I think some aspects of social systems are actually very easy to predict. An individual human driver, might be very hard to predict. But if you see 10,000 people driving down the highway, you get a strong sense of whether there’s going to be a traffic jam. Sometimes unpredictable phenomena can add up to predictable phenomena, and I think that’s a really important feature of making good long-term predictions with complicated systems.

Anthony: It’s often said that climate is more predictable than weather. Although the individual fluctuations day-to-day are difficult to predict, it’s very easy to predict that, in general, winter in the Northern Hemisphere is going to be colder than the summer. There are lots of statistical regularities that emerge, when you average over large numbers.

Ariel: As we’re trying to understand what the impact of artificial intelligence will be on humanity how do we consider what would be a complex prediction? What’s a simple prediction? What sort of information do we need to do this?

Anthony: Well, that’s a tricky one. One of the best methods of prediction for lots of things is just simple extrapolation. There are many physical systems that, once you can discern if they have a trend, you can fit a pretty simple function to.

When you’re talking about artificial intelligence, there are some hard aspects to predict, but also some relatively easy aspects to predict, like looking at the amount of funding that’s being given to artificial intelligence research or the computing power and computing speed and efficiency, following Moore’s Law and variants of it.

Andrew: People often think of mathematics as a source of certainty, but sometimes you can be certain that you are uncertain or you can be certain that you can’t be certain about something else.

A simple trend, like Moore’s Law, is a summary of what you see from a very complicated system, namely a bunch of companies and a bunch of people working to build smaller and faster and cheaper and more energy efficient hardware. That’s a very complicated system that somehow adds up to fairly simple behavior.

A hallmark of good prediction is, when you find a trend, the first question you should ask yourself is what is giving rise to this trend, and can I expect that to continue? That’s a bit of an art. It’s kind of more art than science, but it’s a critical art, because otherwise we end up blindly following trends that are bound to fail.

Ariel: I want to ask about who is making the prediction. With AI, for example, we see smart people in the field who predict AI will make life great and others are worried. With existential risks we see surveys and efforts in which experts in the field try to predict the odds of human extinction. How much can we rely on “experts in the field”?

Andrew: I can certainly tell you that thinking for 30 consecutive minutes about what could cause human extinction is much more productive than thinking for one consecutive minute. There are hard-to-notice mistakes about human extinction predictions that you probably can’t figure out from 30 seconds of reasoning.

Not everyone who’s an expert, say, in nuclear engineering or artificial intelligence is an expert in reasoning about human extinction. You have to be careful who you call an expert.

Anthony: I also feel that something similar is true about prediction. In general, making predictions is greatly aided if you have domain knowledge and expertise in the thing that you’re making a prediction about, but far from sufficient to make accurate predictions.

One of the experiences I’ve seen running Metaculus, is that there are people that know a tremendous amount about a subject and just are terrible at making predictions about it. Other people, who, even if their actual domain knowledge is lower, the fact that they are comfortable with statistics, that they’ve had practice making predictions are just much, much better at it.

Ariel: Anthony, with Metaculus, one of the things that you’re trying to do is get more people involved in predicting. What is the benefit of more people?

Anthony: There are a few benefits. One is that lots of people get the benefit of practice. Thinking about things that you tend to be more wrong on and what they might correlate with — that’s incredibly useful and makes you more effective.

In terms of actually creating accurate predictions, you’ll have more people who are really good at it. You can figure out who is good at predicting, and who is good at predicting a particular type of thing. One of the interesting things is that it isn’t just luck. There is a skill that people can develop and obtain, and then can be relied upon in the future.

Then, the third, and maybe this is the most important, is just statistics. Aggregating lots of people’s predictions tends to make a more accurate aggregate.

Andrew: I would also just like to say that I think the existence of systems like Metaculus are going to be really important for society improving its ability to understand the world.

Whose job is it to think for a solid hour about a human extinction risk? The answer is almost nobody. So we ought not to expect that just averaging the wisdom of the crowds is going to do super well on answering a question like that.

Ariel: Back to artificial intelligence and the question of timelines. How helpful is it for us to try to make predictions about when things will happen with AI? And who should make those predictions?

Andrew: I have made a career shift to coming up with trying to design control mechanisms for highly intelligent AI. I made that career shift, based on my own personal forecast of the future and what I think will be important, but I don’t reevaluate that forecast every day, just as I don’t reevaluate what neighborhood I should live in every day. You, at some point, need to commit to a path and follow that path for a little while to get anything done.

I think most AI researchers should, at some point, do the mental exercise of mapping out timelines and seeing what needs to happen, but they should do it deeply once every few years in collaboration with a few other people, and then stick to something that they think is going to help steer AI in a positive direction. I see a tendency to too frequently reevaluate timeline analyses of what’s going to happen in AI.

My answer to you is kind of everyone, but not everyone at once.

Anthony: I think there’s one other interesting question, which is the degree to which we want there to be accurate predictions and lots of people know what those accurate predictions are.

In general, I think more information is better, but it’s not necessarily the case that more information is better all the time. Suppose, that I became totally convinced, using Metaculus, that there was a high probability that artificial superintelligence was happening in the next 10 years. That would be a pretty big deal. I’d really want to think through what effect that information would have on various actors, national governments, companies, and so on. It could instigate a lot of issues. Those are things that I think we have to really carefully consider.

Andrew: Yeah, Anthony, I think that’s a great important issue. I don’t think there are enough scientific norms in circulation for what to do with a potentially dangerous discovery. Honestly, I feel like the discourse in most of science is a little bit head in the sand about the feasibility of creating existential risks from technology.

You might think it would be so silly and dumb to have some humans produce some technology that accidentally destroyed life, but just because it’s silly doesn’t mean it won’t happen. It’s the bystander effect. It’s very easy for us to fall into the trap of: “I don’t need to worry about developing dangerous technology, because if I was close to something dangerous, surely someone would have thought that through.”

You have to ask: whose job is it to be worried? If no one in the artificial intelligence community is point on noticing existential threats, maybe no one will notice the existential threats and that will be bad. The same goes for the technology that could be used by bad actors to produce dangerous synthetic viruses.

If you’ve got something that you think is 1% likely to pose an extinction threat, that seems like a small probability. Nonetheless, if 100 people have a 1% chance of causing human extinction, well someone probably has a good chance of doing it.

Ariel: Is there something hopeful that you want to add?

Anthony: Pretty much every decision that we make is implicitly built on a prediction. I think that if we can get better at predicting, individually, as a group, as a society, that should really help us choose a more wise path into the future, and hopefully that can happen.

Andrew: Hear, hear.

Visit metaculus.com to try your hand at the art of predicting.

The rise of artificial intelligence presents not only technical challenges, but important legal and ethical challenges for society, especially regarding machines like autonomous weapons and self-driving cars. To discuss these issues, I interviewed Matt Scherer and Ryan Jenkins. Matt is an attorney and legal scholar whose scholarship focuses on the intersection between law and artificial intelligence. Ryan is an assistant professor of philosophy and a senior fellow at the Ethics and Emerging Sciences group at California Polytechnic State, where he studies the ethics of technology.

In this podcast, we discuss accountability and transparency with autonomous systems, government regulation vs. self-regulation, fake news, and the future of autonomous systems.

The following interview has been heavily edited for brevity, but you can listen to it in its entirety above or read the full transcript here.

Ariel: I typically think of ethics as the driving force behind law. As such, Ryan, I was hoping you could talk about the ethical issues facing us today when it comes to artificial intelligence.

Ryan: Broadly speaking, the mission of both ethics and law might be to discover how to best structure life within a community and to see to it that that community does flourish once we know certain truths. Ethics does some of the investigation about what kinds of things matter morally, what kinds of lives are valuable, how should we treat other people. Law does an excellent job of codifying those things and enforcing those things.

One of the easiest ways of telling whether a decision is a moral decision is whether it stands to make some people better off and some people worse off. And we’re seeing that take place right now with artificial intelligence. That adds new wrinkles to these decisions because oftentimes the decisions of AI are opaque to us, they’re difficult to understand, they might be totally mysterious. And while we’re fascinated by what AI can do, I think the developers of AI have implemented these technologies before we fully understand what they’re capable of and how they’re making decisions.

Ariel: Can you give some examples of that?

Ryan: There was an excellent piece by ProPublica about bias in the criminal justice system, where they use risk assessment algorithms to judge, for example, a person’s probability of re-committing a crime after they’re released from prison.

ProPublica did an audit of this software, and they found that not only does it make mistakes about half the time, but it was systematically underestimating the threat from white defendants and systematically overestimating the threat from black defendants. White defendants were being given leaner sentences, black defendants as a group were being given harsher sentences.

When the company that produced the algorithm was asked about this, they said look, it takes in something like 137 factors, but race is not one of them’. So it was making mistakes that were systematically biased in a way that was race-based, and it was difficult to explain why. This is the kind of opaque decision making that’s taking place by artificial intelligence.

Ariel: As AI advances, what are some of the ethical issues that you anticipate cropping up?

Ryan: There’s been a lot of ink spilled about the threat that automation poses to unemployment. Some of the numbers coming out of places like Oxford are quite alarming. They say as many of 50% of American jobs could be eliminated by automation in the next couple decades.

Besides the obvious fact that having unemployed people is bad for society, it raises more foundational questions about the way that we think about work, the way that we think about people having to “earn a living” or “contribute to society.” The idea that someone needs to work in order to be kept alive. And most of us walk around with some kind of moral claim like this in our back pocket without fully considering the implications.

Ariel: And Matt, what are some of the big legal issues facing us today when it comes to artificial intelligence?

Matt: The way that legal systems across the world work is by assigning legal rights and responsibilities to people. The assumption is that any decision that has an impact on the life of another person is going to be made by a person. So when you have a machine making the decisions rather than humans, one of the fundamental assumptions of our legal system goes away. Eventually that’s going to become very difficult because there seems to be the promise of AI displacing human decisionmakers out of a wide variety of sectors. As that happens, it’s going to be much more complicated to come up with lines of legal responsibility.

I don’t think we can comprehend what society is going to be like 50 years from now if a huge number of industries ranging from medicine to law to financial services are in large part being run by the decisions of machines. At some point, the question is how much control can humans really say that they still have.

Ariel: You were talking earlier about decision making with autonomous technologies, and one of the areas where we see this is with self driving cars and autonomous weapons. I was hoping you could both talk about the ethical and legal implications in those spheres.

Matt: Part of the problem with relying on law to set standards of behavior is that law does not move as fast as technology does. It’s going to be a long time before the really critical changes in our legal systems are changed in a way that allows for the widespread deployment of autonomous vehicles.

One thing that I could envision happening in the next 10 years is that pretty much all new vehicles while they’re on an expressway are controlled by an autonomous system, and it’s only when they get off an expressway and onto a surface street that they switch to having the human driver in control of the vehicle. So, little by little, we’re going to see this sector of our economy get changed radically.

Ryan: One of my favorite philosophers of technology [is] Langdon Winner. His famous view is that we are sleepwalking into the future of technology. We’re continually rewriting and recreating these structures that affect how we’ll live, how we’ll interact with each other, what we’re able to do, what we’re encouraged to do, what we’re discouraged from doing. We continually recreate these constraints on our world, and we do it oftentimes without thinking very carefully about it. To steal a line from Winston Churchill, technology seems to get halfway around the world before moral philosophy can put its pants on. And we’re seeing that happening with autonomous vehicles.

Tens of thousands of people die on US roads every year. Oftentimes those crashes involve choices about who is going to be harmed and who’s not, even if that’s a trade-off between someone outside the car and a passenger or a driver inside the car.

These are clearly morally important decisions, and it seems that manufacturers are still trying to brush these aside. They’re either saying that these are not morally important decisions, or they’re saying that the answers to them are obvious. They’re certainly not always questions with obvious answers. Or if the manufacturers admit that they’re difficult answers, then they think, ‘well the decisions are rare enough that to agonize over them might postpone other advancements in the technology’. That’s a legitimate concern, if it were true that these decisions were rare, but there are tens of thousands of people killed on US roads and hundreds of thousands who are injured every year.

Ariel: I’d like to also look at autonomous weapons. Ryan, what’s your take on some of the ethical issues?

Ryan: There could very well be something that’s uniquely troubling, uniquely morally problematic about delegating the task of who should live and who should die to a machine. But once we dig into these arguments, it’s extremely difficult to pinpoint exactly what’s problematic about killer robots. We’d be right to think, today, that machines probably aren’t reliable enough to make discernments in the heat of battle about which people are legitimate targets and which people are not. But if we imagine a future where robots are actually pretty good at making those kinds of decisions, where they’re perhaps even better behaved than human soldiers, where they don’t get confused, they don’t see their comrade killed and go on a killing spree or go into some berserker rage, and they’re not racist, or they don’t have the kinds of biases that humans are vulnerable to…

If we imagine a scenario where we can greatly reduce the number of innocent people killed in war, this starts to exert a lot of pressure on that widely held public intuition that autonomous weapons are bad in themselves, because it puts us in the position then of insisting that we continue to use human war fighters to wage war even when we know that will contribute to many more people dying from collateral damage. That’s an uncomfortable position to defend.

Ariel: Matt, how do we deal with accountability?

Matt: Autonomous weapons are going to inherently be capable of reacting on time scales that are shorter than humans’ time scales in which they can react. I can easily imagine it reaching the point very quickly where the only way that you can counteract an attack by an autonomous weapon is with another autonomous weapon. Eventually, having humans involved in the military conflict will be the equivalent of bringing bows and arrows to a battle in World War II.

At that point, you start to wonder where human decision makers can enter into the military decision making process. Right now there’s very clear, well-established laws in place about who is responsible for specific military decisions, under what circumstances a soldier is held accountable, under what circumstances their commander is held accountable, on what circumstances the nation is held accountable. That’s going to become much blurrier when the decisions are not being made by human soldiers, but rather by autonomous systems. It’s going to become even more complicated as machine learning technology is incorporated into these systems, where they learn from their observations and experiences in the field on the best way to react to different military situations.

Ariel: Matt, in recent talks you mentioned that you’re less concerned about regulations for corporations because it seems like corporations are making an effort to essentially self-regulate. I’m interested in how that compares to concerns about government misusing AI and whether self-regulation is possible with government.

Matt: We are living in an age, with the advent of the internet, that is an inherently decentralizing force. In a decentralizing world, we’re going to have to think of new paradigms of how to regulate and govern the behavior of economic actors. It might make sense to reexamine some of those decentralized forms of regulation and one of those is industry standards and self-regulation.

One reason why I am particularly hopeful in the sphere of AI is that there really does seem to be a broad interest among the largest players in AI to proactively come up with rules of ethics and transparency in many ways that we generally just haven’t seen in the age since the Industrial Revolution.

One macro trend unfortunately in the world stage today is increasingly nationalist tendencies. That leads me to be more concerned than I would have been 10 years ago that these technologies are going to be co-opted by governments, and ironically that it’s going to be governments rather than companies that are the greatest obstacle to transparency because they will want to establish some sort of national monopoly on the technologies within their borders.

Ryan: I think that international norms of cooperation can be valuable. The United States is not a signatory to the Ottawa Treaty that banned anti-personnel landmines, but because so many other countries are, there exists the informal stigma that’s attached to it, that if we used anti-personnel landmines in battle, we’d face backlash that’s probably equivalent to if we had been signatories of that treaty.

So international norms of cooperation, they’re good for something, but they’re also fragile. For example, in much of the western world, there has existed an informal agreement that we’re not going to experiment by modifying the genetics of human embryos. So it was a shock a year or two ago when some Chinese scientists announced that they were doing just that. I think it was a wake up call to the West to realize those norms aren’t universal, and it was a valuable reminder that when it comes to things that are as significant as modifying the human genome or autonomous weapons and artificial intelligence more generally, they have such profound possibilities for reshaping human life that we should be working very stridently to try to arrive at some international agreements that are not just toothless and informal.

Ariel: I want to go in a different direction and ask about fake news. I was really interested in what you both think of this from a legal and ethical standpoint.

Matt: Because there are now so many different sources for news, it becomes increasingly difficult to decide what is real. And there is a loss that we are starting to see in our society of that shared knowledge of facts. There are literally different sets of not just worldviews, but of worlds, that people see around them.

A lot of fake news websites aren’t intentionally trying to make large amounts of money, so even if a fake news story does monumental damage, you’re not going to be able to recoup the damages to your reputation from that person or that entity. It’s an area where it’s difficult for me to envision how the law can manage that, at least unless we come up with new regulatory paradigms that reflect the fact that our world is going to be increasingly less centralized than it has been during the industrial age.

Ariel: Is there anything else that you think is important for people to know?

Ryan: There is still a great value in appreciating when we’re running roughshod over questions that we didn’t even know existed. That is one of the valuable contributions that [moral philosophers] can make here, is to think carefully about the way that we behave, the way that we design our machines to interact with one another and the kinds of effects that they’ll have on society.

It’s reassuring that people are taking these questions very seriously when it comes to artificial intelligence, and I think that the advances we’ve seen in artificial intelligence in the last couple of years have been the impetus for this turn towards the ethical implications of the things we create.

Matt: I’m glad that I got to hear Ryan’s point of view. The law is becoming a less effective tool for managing the societal changes that are happening. And I don’t think that that will change unless we think through the ethical questions and the moral dilemmas that are going to be presented by a world in which decisions and actions are increasingly undertaken by machines rather than people.

This podcast and transcript were edited by Tucker Davey.

Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we interview researchers and thought leaders who we believe will help spur discussion within our community. The interviews do not necessarily represent FLI’s opinions or views.

2016 saw some significant AI developments. To talk about the AI progress of the last year, we turned to Richard Mallah and Ian Goodfellow. Richard is the director of AI projects at FLI, he’s the Senior Advisor to multiple AI companies, and he created the highest-rated enterprise text analytics platform. Ian is a research scientist at OpenAI, he’s the lead author of the Deep Learning textbook, and he’s a lead inventor of Generative Adversarial Networks.

The following interview has been heavily edited for brevity, but you can listen to it in its entirety above or read the full transcript here.

Ariel: Two events stood out to me in 2016. The first was AlphaGo, which beat the world’s top Go champion, Lee Sedol last March. What is AlphaGo, and why was this such an incredible achievement?

Ian: AlphaGo was DeepMind’s system for playing the game of Go. It’s a game where you place stones on a board with two players, the object being to capture as much territory as possible. But there are hundreds of different positions where we can place a stone on each turn. It’s not even remotely possible to use a computer to simulate many different Go games and figure out how the game will progress in the future. The computer needs to rely on intuition the same way that human Go players can look at a board and get kind of a sixth sense that tells them whether the game is going well or poorly for them, and where they ought to put the next stone. It’s computationally infeasible to explicitly calculate what each player should do next.

Richard: The DeepMind team has one network for what’s called value learning and another deep network for policy learning. The policy is, basically, which places should I evaluate for the next piece. The value network is how good that state is, in terms of the probability that the agent will be winning. And then they do a Monte Carlo tree search, which means it has some randomness and many different paths — on the order of thousands of evaluations. So it’s much more like a human considering a handful of different moves and trying to determine how good those moves would be.

Ian: From 2012 to 2015 we saw a lot of breakthroughs where the exciting thing was that AI was able to copy a human ability. In 2016, we started to see breakthroughs that were all about exceeding human performance. Part of what was so exciting about AlphaGo was that AlphaGo did not only learn how to predict what a human expert Go player would do, AlphaGo also improved beyond that by practicing playing games against itself and learning how to be better than the best human player. So we’re starting to see AI move beyond what humans can tell the computer to do.

Ariel: So how will this be applied to applications that we’ll interact with on a regular basis? How will we start to see these technologies and techniques in action ourselves?

Richard: With these techniques, a lot of them are research systems. It’s not necessarily that they’re going to directly go down the pipeline towards productization, but they are helping the models that are implicitly learned inside of AI systems and machine learning systems to get much better.

Ian: There are other strategies for generating new experiences that resemble previously seen experiences. One of them is called WaveNet. It’s a model produced by DeepMind in 2016 for generating speech. If you provide a sentence, just written down, and you’d like to hear that sentence spoken aloud, WaveNet can create an audio waveform that sounds very realistically like a human pronouncing that sentence written down. The main drawback to WaveNet right now is that it’s fairly slow. It has to generate the audio waveform one piece at a time. I believe it takes WaveNet two minutes to produce one second of audio, so it’s not able to make the audio fast enough to hold an interactive conversation.

Richard: And similarly, we’ve seen applications to colorizing black and white photos, or turning sketches into somewhat photo-realistic images, being able to turn text into images.

Ian: Yeah one thing that really highlights how far we’ve come is that in 2014, one of the big breakthroughs was the ability to take a photo and produce a sentence summarizing what was in the photo. In 2016, we saw different methods for taking a sentence and producing a photo that contains the imagery described by the sentence. It’s much more complicated to go from a few words to a very realistic image containing thousands or millions of pixels than it is to go from the image to the words.

Another thing that was very exciting in 2016 was the use of generative models for drug discovery. Instead of imagining new images, the model could actually imagine new molecules that are intended to have specific medicinal effects.

Richard: And this is pretty exciting because this is being applied towards cancer research, developing potential new cancer treatments.

Ariel: And then there was Google’s language translation program, Google Neural Machine Translation. Can you talk about what that did and why it was a big deal?

Ian: It’s a big deal for two different reasons. First, Google Neural Machine Translation is a lot better than previous approaches to machine translation. Google Neural Machine Translation removes a lot of the human design elements, and just has a neural network figure out what to do.

The other thing that’s really exciting about Google Neural Machine Translation is that the machine translation models have developed what we call an “Interlingua.” It used to be that if you wanted to translate from Japanese to Korean, you had to find a lot of sentences that had been translated from Japanese to Korean before, and then you could train a machine learning model to copy that translation procedure. But now, if you already know how to translate from English to Korean, and you know how to translate from English to Japanese, in the middle, you have Interlingua. So you translate from English to Interlingua and then to Japanese, English to Interlingua and then to Korean. You can also just translate Japanese to Interlingua and Korean to Interlingua and then Interlingua to Japanese or Korean, and you never actually have to get translated sentences from every pair of languages.

Ariel: How can the techniques that are used for language apply elsewhere? How do you anticipate seeing this developed in 2017 and onward?

Richard: So I think what we’ve learned from the approach is that deep learning systems are able to create extremely rich models of the world that can actually express what we can think, which is a pretty exciting milestone. Being able to combine that Interlingua with more structured information about the world is something that a variety of teams are working on — it is a big, open area for the coming years.

Ian: At OpenAI one of our largest projects, Universe, allows a reinforcement learning agent to play many different computer games, and it interacts with these games in the same way that a human does, by sending key presses or mouse strokes to the actual game engine. The same reinforcement learning agent is able to interact with basically anything that a human can interact with on a computer. By having one agent that can do all of these different things we will really exercise our ability to create general artificial intelligence instead of application-specific artificial intelligence. And projects like Google’s Interlingua have shown us that there’s a lot of reason to believe that this will work.

Ariel: What else happened this year that you guys think is important to mention?

Richard: One-shot [learning] is when you see just a little bit of data, potentially just one data point, regarding some new task or some new category, and you’re then able to deduce what that class should look like or what that function should look like in general. So being able to train systems on very little data from just general background knowledge, will be pretty exciting.

Ian: One thing that I’m very excited about is this new area called machine learning security where an attacker can trick a machine learning system into taking the wrong action. For example, we’ve seen that it’s very easy to fool an object-recognition system. We can show it an image that looks a lot like a panda and it gets recognized as being a school bus, or vice versa. It’s actually possible to fool machine learning systems with physical objects. There was a paper called Accessorize to a Crime, that showed that by wearing unusually-colored glasses it’s possible to thwart a face recognition system. And my own collaborators at GoogleBrain and I wrote a paper called Adversarial Examples in the Physical World, where we showed that we can make images that look kind of grainy and noisy, but when viewed through a camera we can control how an object-recognition system will respond to those images.

Ariel: Is there anything else that you thought was either important for 2016 or looking forward to 2017?

Richard: Yeah, looking forward to 2017 I think there will be more focus on unsupervised learning. Most of the world is not annotated by humans. There aren’t little sticky notes on things around the house saying what they are. Being able to process [the world] in a more unsupervised way will unlock a plethora of new applications.

Ian: It will also make AI more democratic. Right now, if you want to use really advanced AI you need to have not only a lot of computers but also a lot of data. That’s part of why it’s mostly very large companies that are competitive in the AI space. If you want to get really good at a task you basically become good at that task by showing the computer a million different examples. In the future, we’ll have AI that can learn much more like a human learns, where just showing it a few examples is enough. Once we have machine learning systems that are able to get the general idea of what’s going on very quickly, in the way that humans do, it won’t be necessary to build these gigantic data sets anymore.

Richard: One application area I think will be important this coming year is automatic detection of fake news, fake audio and fake images and fake video. Some of the applications this past year have actually focused on generating additional frames of video. As those get better, as the photo generation that we talked about earlier gets better, and also as audio templating gets better… I think it was Adobe that demoed what they called PhotoShop for Voice, where you can type something in and select a person, and it will sound like that person saying whatever it is that you typed. So we’ll need ways of detecting that, since this whole concept of fake news is quite at the fore these days.

Ian: It’s worth mentioning that there are other ways of addressing the spread of fake news. Email spam uses a lot of different clues that it can statistically associate with whether people mark the email as spam or not. We can do a lot without needing to advance the underlying AI systems at all.

Ariel: Is there anything that you’re worried about, based on advances that you’ve seen in the last year?

Ian: The employment issue. As we’re able to automate our tasks in the future, how will we make sure that everyone benefits from that automation? And the way that society is structured, right now increasing automation seems to lead to increasing concentration of wealth, and there are winners and losers to every advance. My concern is that automating jobs that are done by millions of people will create very many losers and a small number of winners who really win big.

Richard: I’m also slightly concerned with the speed at which we’re approaching additional generality. It’s extremely cool to see systems be able to do lots of different things, and being able to do tasks that they’ve either seen very little of or none of before. But it raises questions as to when we implement different types of safety techniques. I don’t think that we’re at that point yet, but it raises the issue.

Ariel: To end on a positive note: looking back on what you saw last year, what has you most hopeful for our future?

Ian: I think it’s really great that AI is starting to be used for things like medicine. In the last year we’ve seen a lot of different machine learning algorithms that could exceed human abilities at some tasks, and we’ve also started to see the application of AI to life-saving application areas like designing new medicines. And this makes me very hopeful that we’re going to start seeing superhuman drug design, and other kinds of applications of AI to just really make life better for a lot of people in ways that we would not have been able to do without it.

Richard: Various kinds of tasks that people find to be drudgery within their jobs will be automatable. That will lead them to be open to working on more value-added things with more creativity, and potentially be able to work in more interesting areas of their field or across different fields. I think the future is wide open and it’s really what we make of it, which is exciting in itself.

Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we interview researchers and thought leaders who we believe will help spur discussion within our community. The interviews do not necessarily represent FLI’s opinions or views.

FLI’s Ariel Conn recently spoke with Heather Roff and Peter Asaro about autonomous weapons. Roff, a research scientist at The Global Security Initiative at Arizona State University and a senior research fellow at the University of Oxford, recently compiled an international database of weapons systems that exhibit some level of autonomous capabilities. Asaro is a philosopher of science, technology, and media at The New School in New York City. He looks at fundamental questions of responsibility and liability with all autonomous systems, but he’s also the Co-Founder and Vice-Chair of the International Committee for Robot Arms Control and a he’s a Spokesperson for the Campaign to Stop Killer Robots.

The following interview has been edited for brevity, but you can read it in its entirety here or listen to it above.

ARIEL: Dr. Roff, I’d like to start with you. With regard to the database, what prompted you to create it, what information does it provide, how can we use it?

ROFF: The main impetus behind the creation of the database [was] a feeling that the same autonomous or automated weapons systems were brought out in discussions over and over and over again. It made it seem like there wasn’t anything else to worry about. So I created a database of about 250 autonomous systems that are currently deployed [from] Russia, China, the United States, France, and Germany. I code them along a series of about 20 different variables: from automatic target recognition [to] the ability to navigate [to] acquisition capabilities [etc.].

It’s allowing everyone to understand that autonomy isn’t just binary. It’s not a yes or a no. Not many people in the world have a good understanding of what modern militaries fight with, and how they fight.

ARIEL: And Dr. Asaro, your research is about liability. How is it different for autonomous weapons versus a human overseeing a drone that just accidentally fires on the wrong target.

ASARO: My work looks at autonomous weapons and other kinds of autonomous systems and the interface of the ethical and legal aspects. Specifically, questions about the ethics of killing, and the legal requirements under international law for killing in armed conflict. These kind of autonomous systems are not really legal and moral agents in the way that humans are, and so delegating the authority to kill to them is unjustifiable.

One aspect of accountability is, if a mistake is made, holding people to account for that mistake. There’s a feedback mechanism to prevent that error occurring in the future. There’s also a justice element, which could be attributive justice, in which you try to make up for loss. Other forms of accountability look at punishment itself. When you have autonomous systems, you can’t really punish the system. More importantly, if nobody really intended the effect that the system brought about then it becomes very difficult to hold anybody accountable for the actions of the system. The debate — it’s really kind of framed around this question of the accountability gap.

ARIEL: One of the things we hear a lot in the news is about always keeping a human in the loop. How does that play into the idea of liability? And realistically, what does it mean?

ROFF: I actually think this is just a really unhelpful heuristic. It’s hindering our ability to think about what’s potentially risky or dangerous or might produce unintended consequences. So here’s an example: the UK’s Ministry of Defense calls this the Empty Hangar Problem. It’s very unlikely that they’re going to walk down to an airplane hangar, look in, and be like, “Hey! Where’s the airplane? Oh, it’s decided to go to war today.” That’s just not going to happen.

These systems are always going to be used by humans, and humans are going to decide to use them. A better way to think about this is in terms of task allocation. What is the scope of the task, and how much information and control does the human have before deploying that system to execute? If there is a lot of time, space, and distance between the time the decision is made to field it and then the application of force, there’s more time for things to change on the ground, and there’s more time for the human to basically [say] they didn’t intend for this to happen.

ASARO: If self-driving cars start running people over, people will sue the manufacturer. But there’s no mechanisms in international law for the victims of bombs and missiles and potentially autonomous weapons to sue the manufacturers of those systems. That just doesn’t happen. So there’s no incentives for companies that manufacture those [weapons] to improve safety and performance.

ARIEL: Dr. Asaro, we’ve briefly mentioned definitional problems of autonomous weapons — how does the liability play in there?

ASARO: The law of international armed conflict is pretty clear that humans are the ones that make the decisions, especially about a targeting decision or the taking of a human life in armed conflict. This question of having a system that could range over many miles and many days and select targets on its own is where things are problematic. Part of the definition is: how do you figure out exactly what constitutes a targeting decision, and how do you ensure that a human is making that decision? That’s the direction the discussion at the UN is going as well. Instead of trying to define what’s an autonomous system, what we focus on is the targeting decision and firing decisions of weapons for individual attacks. What we want to acquire is meaningful human control over those decisions.

ARIEL: Dr. Roff, you were working on the idea of meaningful human control, as well. Can you talk about that?

ROFF: If [a commander] fields a weapon that can go from attack to attack without checking back with her, then the weapon is making the proportionality calculation, and she [has] delegated her authority and her obligation to a machine. That is prohibited under IHL, and I would say is also morally prohibited. You can’t offload your moral obligation to a nonmoral agent. So that’s where our work on meaningful human control is: a human commander has a moral obligation to undertake precaution and proportionality in each attack.

ARIEL: Is there anything else you think is important to add?

ROFF: We still have limitations AI. We have really great applications of AI, and we have blind. It would be really incumbent on the AI community to be vocal about where they think there are capacities and capabilities that could be reliably and predictably deployed on such systems. If they don’t think that those technologies or applications could be reliably and predictably deployed, then they need to stand up and say as much.

ASARO: We’re not trying to prohibit autonomous operations of different kind of systems or the development and application of artificial intelligence for a wide range of civilian and military applications. But there are certain applications, specifically the lethal ones, that have higher standards of moral and legal requirements that need to be met.

Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we interview researchers and thought leaders who we believe will help spur discussion within our community. The interviews do not necessarily represent FLI’s opinions or views.

Interview with Robin Hanson
A few weeks ago, I had the good fortune to interview Robin Hanson about his new book, The Age of Em. We discussed his book, the future and evolution of humanity, and the research he’s doing for his next book. You can listen to all of that here. And read on for my review of Hanson’s book…

Age of Em Review

As I’ve interviewed more and more people who focus on and worry about the future, an interesting theme keeps popping up: choice. Over and over, scholars and researchers desperately try to remind us that we have a say in our future. The choices we make will impact whether or not our future goes the way we want it to.

But choosing a path for our future isn’t like picking a breakfast cereal. Our options aren’t all in front of us, with detailed information on the side telling us how we may or may not benefit from each choice. Few of us can predict how our decisions will shape our own lives, let alone the future of humanity. That’s where Robin Hanson comes in.

Hanson, a professor of economics at George Mason University, advocates that the activities that will shape our future can be much better informed by what is likely to happen than most of us realize.  He recently wrote the book, The Age of Em: Work, Love and Life when Robots Rule the Earth, which describes one possible path that may lay before us based on current economic and technologic trends.

What Is the Age of Em?

Em is short for emulation — in this case, a brain emulation. In this version of the future, people will choose to have their brains scanned, uploaded and possibly copied, creating a new race of robots and other types of machine intelligence. Because they’re human emulations, Hanson expects ems will think and feel just as we would. However, without biological needs or aging processes, they’ll be much cheaper to run and maintain. And because they can do anything we can – just at a much lower cost – it will be more practical to switch from human to em. Humans who resist this switch or who are too old to be useful as ems will end up on the sidelines of society. When (if) ems take over the world, it will be because humans chose to make the transition.

Interestingly, the timeline for how long ems will rule the world will depend on human versus em perspective. Because ems are essentially machines, they can run at different speeds. Hanson anticipates that over the course of two regular “human” years, most ems will have experienced a thousand years – along with all of the societal changes that come with a thousand years of development. Hanson’s book tells the story of their world: their subsistence lifestyles made glamorous by virtual reality; the em clans comprised of the best and the brightest human minds; and literally, how the ems will work, love, and live.

It’s a very detailed work, and it’s easy to get caught up in the details of which aspects of em life are likely, which details seem unrealistic, and even if ems are more likely than artificial intelligence to take over the world next. And there have been excellent discussions and reviews of the details of the book, like this one at Slate Star Codex. But I’m writing this review almost as much in response to commentary I’ve read about the book as I am about the book itself because there’s another question that’s important to ask as well: Is this the future that we want?

What do we want?

For a book without a plot or characters, it offers a surprisingly engaging and compelling storyline. Perhaps that’s because this is the story of us. It’s the story of humanity — the story of how we progress and evolve. And it’s also the story of how we, as we know ourselves, end.

It’s easy to look at this new world with fear. We’re so focused on production and the bottom line that, in the future, we’ve literally pushed humanity to the sidelines and possibly to extinction. Valuing productivity is fine, but do we really want to take it to this level? Can we stop this from happening, and if so, how?

Do we even want to stop it from happening? Early on Hanson encourages us to remember that people in the past would have been equally horrified by our own current lifestyle. He argues that this future may be different from what we’re used to, but it’s reasonable to expect that humans will prefer transitioning to an em lifestyle in the future. And from that perspective, we can look on this new world with hope.

As I read The Age of Em, I was often reminded of A Brave New World by Aldous Huxley. Huxley described his book as a “negative utopia,” but much of what he wrote has become commonplace and trivial today — mass consumerism, drugs to make us happy, a freer attitude about sex, a preference for mindless entertainment to deep thought. Though many of us may not necessarily consider these attributes of modern society to be a utopia, most people today would choose our current lifestyle over that of the 1930s, and we typically consider our lives better now than at any point in history. Even among people today, we see sharp divides between older generations who are horrified by how much privacy is lost thanks to the Internet and younger generations who see the benefits of increased information and connectivity outweighing any potential risks. Most likely, a similar attitude shift will take place as (if) we move toward a world of ems.

Yet while it’s reasonable to accept that in the future we would likely consider ems to be a positive step for humanity, the questions still remain: Is this what we want, or are we just following along on a path, unable to stop or change directions? Can we really choose our future?

Studying the future

In the book, Hanson says, “If we first look carefully at what is likely to happen if we do nothing, such a no-action baseline can help us to analyze what we might do to change those outcomes. This book, however, only offers the nearest beginnings of such policy analysis.” Hanson looks at where we’ve been, he looks at where we are now, and then he draws lines out into the future to figure out the direction we’ll go. And he does this for every aspect of life. In fact, given that this book is about the future, it also provides considerable insight into who we are now. But it represents only one possible vision for the future.

There are many more people who study history than the future, primarily because we already have information and writings and artifacts about historic events. But though we can’t change the past, we can impact the future. As the famous quote (or paraphrase) by George Santayana goes, “Those who fail to learn history are doomed to repeat it.” So perhaps learning history is only half the story. Perhaps it’s time to reevaluate the prevailing notion that the future is something that can’t be studied.

Only as we work through different possible scenarios for our future, can we better understand how decisions today will impact humanity later on. And with that new information, maybe we can start to make choices that will guide us toward a future we’re all excited about.

Final thoughts

Love it or hate it, agree with it or not, Hanson’s book and his approach to thinking about the future are extremely important for anyone who wants to have a say in the future of humanity. It’s easy to argue over whether or not ems represent the most likely future. It’s just as easy to get lost in the minutia of the em world and debate whether x, y, or z will happen. And these discussions are necessary if we’re to understand what could happen in the future. But to do only that is to miss an important point: something will happen, and we have to decide if we want a role in creating the future or if we want to stand idly by.

I highly recommend The Age of Em, I look forward to Hanson’s next book, and I hope others will answer his call to action and begin studying the future.

Note from FLI: Among our objectives is to inspire discussion and a sharing of ideas. As such, we post op-eds that we believe will help spur discussion within our community. Op-eds do not necessarily represent FLI’s opinions or views.