• Inner Alignment Versus Outer Alignment
• Foundation Models
• Structural AI Risks
• Unipolar Versus Multipolar Scenarios
• The Most Important Thing That Impacts the Future of Life

Watch the video version of this episode here

0:00 Intro

00:02:22 What is AI alignment?

00:06:45 How has your perspective of this problem changed over the past year?

00:07:22 Inner Alignment

00:15:35 Ways that AI could actually lead to human extinction

00:22:50 Inner Alignment and MACE optimizers

00:24:15 Outer Alignment

00:27:32 The core problem of AI alignment

00:29:38 Learning Systems versus Planning Systems

00:34:00 AI and Existential Risk

00:38:59 The probability of AI existential risk

01:04:10 Core problems in AI alignment

01:03:07 How has AI alignment, as a field of research changed in the last year?

01:05:57 Large scale language models

01:06:55 Foundation Models

01:15:30 Why don’t we know that AI systems won’t totally kill us all?

01:23:50 How much of the alignment and safety problems in AI will be solved by industry?

01:31:00 Do you think about what beneficial futures look like?

01:39:44 Moral Anti-Realism and AI

01:46:22 Unipolar versus Multipolar Scenarios

01:56:38 What is the safety team at DeepMind up to?

01:57:30 What is the most important thing that impacts the future of life?

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Rohin Shah. He is a long-time friend of this podcast, and this is the fourth time we’ve had him on. Every time we talk to him, he gives us excellent overviews of the current thinking in technical AI alignment research. And in this episode he does just that. Our interviews with Rohin go all the way back to December of 2018. They’re super informative and I highly recommend checking them out if you’d like to do a deeper dive into technical alignment research. You can find links to those in the description of this episode. 

Rohin is a Research Scientist on the technical AGI safety team at DeepMind. He completed his PhD at the Center for Human-Compatible AI at UC Berkeley, where he worked on building AI systems that can learn to assist a human user, even if they don’t initially know what the user wants.

Rohin is particularly interested in big picture questions about artificial intelligence. What techniques will we use to build human-level AI systems? How will their deployment affect the world? What can we do to make this deployment go better? He writes up summaries and thoughts about recent work tackling these questions in the Alignment Newsletter, which I highly recommend following if you’re interested in AI alignment research. Rohin is also involved in Effective Altruism, and out of concern for animal welfare, is almost vegan.

And with that, I’m happy to present this interview with Rohin Shah. 

Welcome back Rohin. This is your third time on the podcast I believe. We have this series of podcasts that we’ve been doing, where you help give us a year-end review of AI alignment and everything that’s been up. You’re someone I view as very core and crucial to the AI alignment community. And I’m always happy and excited to be getting your perspective on what’s changing and what’s going on. So to start off, I just want to hit you with a simple, not simple question of what is AI alignment?

Rohin Shah: Oh boy. Excellent. I love that we’re starting there. Yeah. So different people will tell you different things for this as I’m sure you know. The framing I prefer to use is that there is a particular class of failures that we can think about with AI, where the AI is doing something that its designers did not want it to do. And specifically it’s competently achieving some sort of goal or objective or some sort of competent behavior that isn’t the one that was intended by the designers. So for example, if you tried to build an AI system that is, I don’t know, supposed to help you schedule calendar events and then it like also starts sending emails on your behalf to people which maybe you didn’t want it to do. That would count as an alignment failure.

Whereas if a terrorist somehow makes an AI system that goes and designates a bomb in some big city that is not an alignment failure, it is obviously bad, but the AI system did what its designer intended for it to do. It doesn’t count as an alignment failure on my definition of the problem.

Other people will see AI alignment as synonymous with AI safety. For those people, terrorists using a bomb might count as an alignment failure, but at least when I’m using the term, I usually mean, the AI system is doing something that wasn’t what its designers intended for it to do.

There’s a little bit of a subtlety there where you can think of either intent alignment, where you try to figure out what the AI system is trying to do. And then if it is trying to do something that isn’t what the designers wanted, that’s an intent alignment failure, or you can say, all right, screw all of this notion of trying, we don’t know what trying is. How can we look at a piece of code and say whether or not it’s trying to do something.

And instead we can talk about impact alignment, which is just like the actual behavior that the AI system does. Is that what the designers intended or not? So if the AI makes a catastrophic mistake where the AI thinks that this is the big red button for happiness and sunshine, but actually it’s the big red button that launches nix. That is a failure on impact alignment, but isn’t a failure on the intent alignment, assuming the AI legitimately believed that the button was happiness and sunshine, I think they said.

Lucas Perry: So it seems like you could have one or more or less of these in a system at the same time. So which are you excited about? Which do you think are more important than the others?

Rohin Shah: In terms of what do we actually care about? Which is how I usually interpret important, the answer is just like pretty clearly impact alignment. The thing we care about is, did the AI system do what we want or not? I nevertheless tend to think in terms of intent alignment, because it seems like it is decomposing the problem into a natural notion of like what the AI system is trying to do. And whether the AI system is capable enough to do it. And I think that is like actually natural division. You can in fact talk about these things separately. And because of that, it makes sense to have research organized around those two things separately. But that is a claim I am making about the best way to decompose the problem that we actually care about. And that is why I focus on intent alignment but what do we actually care about? Impact alignment, totally.

Lucas Perry: How would you say that your perspective of this problem has changed over the past year?

Rohin Shah: I’ve spent a lot of time thinking about the problem of inner alignment. So this was this shot up to… I mean, people have been talking about it for a while, but it shot up to prominence in I want to say 2019 with the publication of the mesa optimizers paper. And I was not a huge fan of that framing, but I do think that the problem that it’s showing is actually an important one. So I’ve been thinking a lot about that.

Lucas Perry: Can you explain what inner alignment is and how it fits into the definitions of what AI alignment is?

Rohin Shah: Yeah. So AI alignment, the way I’ve described it so far is just sort of like pretty, it’s just talking about properties of AI system. It doesn’t really talk about how that AI system was built, but if you actually want to diagnose at like give reasons why problems might arise and then how to solve them, you probably want to talk about how the AI systems are built and why they’re likely to cause such problems.

Inner alignment, I’m not sure if I like the name, but we’ll go with it for now. Inner alignment is a problem that I claim happens for systems that learn. And the problem is, maybe I should explain it with an example. You might have seen this post from LessWrong about bleggs and rubes. These bleggs are blue in color and tend to be egg-shaped in all the cases they’ve seen so far. Rubes are red in color and are cube-shaped, at least in all the cases they’ve seen so far.

And now suddenly you see a red egg-shaped thing, is it blegg or rube? Like in this case, it’s pretty obvious that there isn’t a correct answer and this same dynamic can arise in a learning system where if it is learning how to behave in accordance with whatever we are training it to do, we’re going to be training it on a particular set of situations. And if those situations change in the future along some axis that the AI system didn’t see during training, it may generalize badly. So a good example of this is, came from the objective robustness and deep reinforcement learning paper. They trained an agent on the CoinRun environment from Procgen. This is basically a very simple platformer game where the agent just has to jump over enemies and obstacles to get to the end and collect the coin.

And the coin is always at the far right end of the level. And so, you train your AI system on a bunch of different kinds of levels, different obstacles, different enemies, they’re placed in different ways. You have to jump in different ways, but the coin is always at the end on the right. And it turns out if you then take your AI system and test it on a new level of where the coin is placed somewhere else in a level, not all the way to the right, the agent just continues to jump over obstacles, enemies, and so on. Behaves very competently in the platformer game, but it just runs all the way to the right and then stays at the right or jumps up and down as though hoping that there’s a coin there. And it’s behaving as if it has the objective of go as far to the right as possible.

Even though we trained it on the objective, get the coin, or at least that’s what we were thinking of as the objective. And this happened because we didn’t show it any examples where the coin was anywhere other than the right side of the level. So the inner alignment problem is when you train a system on one set of inputs, it learns how to behave well on that set of inputs. But then when you extrapolate its behavior to other inputs that you hadn’t seen during training, it turns out to do something that’s very capable, but not what you intended.

Lucas Perry: Can you give an example of what this could look like in the real world, rather than in like a training simulation in a virtual environment?

Rohin Shah: Yeah. One example I like is, it’ll take a bit of setup, but I think it should be fine. You could imagine that with honestly, even today’s technology, we might be able to train an AI system that can just schedule meetings for you. Like when someone emails you asking for a meeting, you’re just like, here calendar scheduling agent, please do whatever you need to do in order to get this meeting scheduled. I want to have it, you go schedule it. And then it goes and emails a person who emails back saying, Rohin is free at such and such times, he like prefers morning meetings or whatever. And then, there’s some back and forth between, and then the meeting gets scheduled. For concreteness, let’s say that the way we do this, is we take a pre-trained language model, like say GPT-3, and then we just have GPT-3 respond to emails and we train it from human feedback.

Well, we have some examples of like people scheduling emails. We do supervised fine tuning on GPT-3 to get it started. And then we like fine tune more from human feedback in order to get it to be good at this task. And it all works great. Now let’s say that in 2023, Gmail decides that Gmail also wants to be a chat app. And so it adds emoji reactions to emails, and everyone’s like, oh my God, now there’s such a better, we can schedule a meeting so much better. We can just say, here, just send an email to all the people who are coming to the meeting and react with emojis for each of the times that you’re available. And everyone loves this. This is how people start scheduling meetings now.

But it turns out that this AI system, when it’s confronted with these emoji polls is like, it knows, it in theory is capable or knows how to use the emoji polls. It knows what’s going on, but it was always trained to schedule the meeting by email. So maybe it will have learned to like always schedule a meeting by email and not to take advantage of these new features. So it might say something like, hey, I don’t really know how to use these newfangled emoji polls. Can we just schedule emails the normal way? In our times this would be a flat out lie, but from the AI’s perspective, we might think of like, the AI was just trained to say whatever sequence of English words lead to getting a meeting scheduled by email. And it predicts that sequence of words will work well. Would this actually happen if I actually trained an agent this way? I don’t know, like it’s totally possible I would actually do the right thing, but I don’t think we can really rule out the wrong thing either, it seems. That also seems pretty plausible to me in this scenario.

Lucas Perry: One important part of this that I think has come up in our previous conversations is that we don’t know when there is always an inner misalignment between the system and the objective we would like for it to learn, because part of maximizing the inner aligned objective could be giving the appearance of being aligned with the outer objective that we’re interested in. Could you explain and unpack that?

Rohin Shah: Yeah. So in the AI safety community, we tend to think about ways that AI could like actually lead to human extinction. And so, the example that I gave does not in fact lead to human extinction. It is a mild annoyance at worst. The story that gets you to human extinction is one in which you have a very capable, superintelligent AI system. But nonetheless, there’s like, instead of learning the objective that we wanted, which might’ve been, I don’t know, something like be a good personal assistant. I’m just giving that out as a concrete example. It could be other things as well. Instead of acting as though it were optimizing that objective, it ends up optimizing some other objective and you don’t really want to give an example here because the whole premise is that it could be a weird objective we don’t really know.

Lucas Perry: Could you expand that a little bit more, like how it would be a weird objective that we wouldn’t really know?

Rohin Shah: Okay. So let’s take as a concrete example, let’s make paperclips, which has nothing to do with being a personal assistant. Now, why is this at all plausible? The reason is that even if this superintelligent AI system had the objective to make paperclips, during training, while we are in control, it’s going to realize that if it doesn’t do the things that we want it to do, we’re just going to turn it off. And as a result, it will be incentivized to do whatever we want until it can make sure that we can’t turn it off. And then it goes and builds its paperclip empire. And so when I say, it could be a weird objective, I mostly just mean that almost any objective is compatible with this sort of a story. It does rely on-

Lucas Perry: Sorry. I’m also curious if you could explain how the inner state of the system becomes aligned to something that is not what we actually care about.

Rohin Shah: I might go back to the CoinRun example, where the agent could have learned to get the coin. That was a totally valid policy it could have learned. And this is an actual experiment that people have run. So this one is not hypothetical. It just didn’t, it learned to go to the right. Why? I mean, I don’t know. I wish I understood neural nets well enough to answer those questions for you. I’m not really arguing for, it’s definitely going to learn, make paperclips. I’m just arguing for like, there’s this whole set of things it could learn. And we don’t know which one it’s going to learn, which seems kind of bad.

Lucas Perry: Is it kind of like, there’s the thing we actually care about? And then a lot of things that are like roughly correlated with it, which I think you’ve used the word for example before is like proxy objectives.

Rohin Shah: Yeah. So that is definitely one way that it could happen, where we ask it to make humans happy and it learns that when humans smile, they’re usually happy and then learns the proxy objective of make human smile and then it like, goes and tapes everyone’s faces so that they’re permanently smiling, that’s a way that things could happen. But I think I don’t even want to claim that’s what … maybe that’s what happens. Maybe it just actually optimizes for human happiness. Maybe it learns to make paperclips for just some weird reason. I mean, not paperclips. Maybe it decides, this particular arrangement of atoms in this novel structure that we don’t really have a word for is the thing that it wants for some reason. And all of these seem totally compatible with, we trained it to be good, to have good behavior in the situations that we cared about because it might just be deceiving us until it has enough power to unilaterally do what it wants without worrying about us stopping it.

I do think that there is some sense of like, no paperclip maximization is too weird. If you trained it to make humans happy, it would not learn to maximize paperclips. There’s just like no path by which paperclips somehow become the one thing it cares about. I’m also sympathetic to, maybe it just doesn’t care about anything to the extent of optimizing the entire universe to turn it into that sort of thing. I’m really just arguing for, we really don’t know crazy shit could happen. I will bet on crazy shit will happen, unless we do a bunch of research and figure out how to make it so that crazy shit doesn’t happen. I just don’t really know what the crazy shit will be.

Lucas Perry: Do you think that that example of the agent in that virtual environment, you see that as a demonstration of the kinds of arbitrary goals that the agent could learn and that that space is really wide and deep and so it could be arbitrarily weird and we have no idea what kind of goal it could end up learning and then deceive us.

Rohin Shah: I think it is not that great evidence for that position. Mostly because I think it’s reasonably likely that if you told somebody the setup of what you were planning to do, if you told an ML researcher or an RL, maybe specifically a deep RL researcher, the setup of that experiment and asked them to predict what would have happened, I think they probably would have, especially if you told them, “Hey, do you think maybe it’ll just run to the right and jump up and down at the end?” I think they’d be like, “Yeah, that seems likely, not just plausible, but actually likely.” That was definitely my reaction when I was first told about this result. I was like, oh yeah, of course that will happen.

In that case, I think we just do know…know is a strong word, ML researchers have good enough intuitions about those situations, I think, that it was predictable in advance. Though I don’t actually know if anyone who predicted it, did in advance. So that one, I don’t think is all that supportive of, it learns an arbitrary goal. We had some notion that neural nets care a lot more about position and simple functions of the action always go right rather than complex visual features like this yellow coin that you have to learn from pixels. I think people could have probably predicted that.

Lucas Perry: So we touched on definitions of AI alignment, and now we’ve been exploring your interest in inner alignment or I think the jargon is mesa optimizers.

Rohin Shah: They are different things.

Lucas Perry: There are different things. Could you explain how inner alignment and mesa optimizers are different?

Rohin Shah: Yeah. So a thing I maybe have not been doing as much as I should have is that, inner alignment is the claim that when the circumstances change, the agent generalizes catastrophically in some way, it behaves as though it’s optimizing some other objective than the one that we actually want. So it’s much more of a claim about the behavior rather than like the internal workings of the AI system that caused that behavior.

mesa-optimization, at least under the definition of the 2019 paper is talking specifically about AI systems that are executing an explicit optimization algorithm. So like the forward path of a neural net is itself an optimization algorithm. We’re not talking about creating dissent here. And then the metric that is being used in that, within the neural network optimization algorithm is the inner objective or sorry, the mesa objective. So it’s making a claim about how the AI system’s cognition is structured. Whereas inner alignment more broadly is the AI behaves in a catastrophically generalizing way.

Lucas Perry: Could you explain what outer alignment is?

Rohin Shah: Sure. Inner alignment can we be thought of as, suppose we got the training objective correct. Suppose the things that we’re training the AI system to do on the situations that we give it as input, we’re actually training it to do the right thing, then things can go wrong if it behaves differently in some new situations that we hadn’t trained it on.

Outer alignment is basically when the reward function that you specify for training the AI system is itself, not what you actually wanted. For example, maybe you want your AI to be helpful to you or to tell you true things. But instead you have, you train your AI system to go find credible looking websites and tell you what the credible looking websites say. And it turns out that sometimes the credible looking websites don’t actually tell you true things.

In that case, you’re going to get an AI that tells you what credible looking websites say, rather than an AI that tells you what things are true. And that’s in some sense, an outer alignment failure. You like even the feedback you were giving the AI system was pushing it away from telling you the truth and pushing it towards telling you what credible looking websites will say, which are correlated of course, but they’re not the same. In general, if you like give me an AI system with some misalignment and you ask me, was this a failure of outer alignment or inner alignment? Mostly I’m like, that’s a somewhat confused question, but one way that you can make it not be confused is you can say, all right, let’s look at the inputs on which it was trained. Now, if ever on an input on which we train, we gave it some wrong feedback where we were like the AI lied to me and I gave it like plus a thousand reward. And you’re like, okay, clearly that’s outer alignment. We just gave it the wrong feedback in the first place.

Supposing that didn’t happen. Then I think what you would want to ask is, okay, let me think about on the situations in which the AI does something bad, what would I have given counterfactually as a reward? And this requires you to have some notion of a counterfactual. When you’d write down a programmatic reward function, the counterfactual is a bit more obvious. It’s like, whatever that program would have output on that input. And so I think that’s the usual setting in which outer alignment has been discussed. And it’s pretty clear what it means there. But once you’re like training from human feedback, it’s not so clear what it means. What would the human have given us feedback on this situation that they’ve never seen before is often pretty ambiguous. If you define such a counterfactual, then I think I’m like, yes. Then I think I’m like, okay, you look at what feedback you would’ve given on the counterfactual. If that feedback was good actually led to the behavior that you wanted, then it’s an inner alignment failure. If that counterfactual feedback was bad, not what you would have wanted. Then it’s an outer alignment failure.

Lucas Perry: If you’re speaking to someone who was not familiar with AI alignment, for example, other people in the computer science community, but also policymakers or the general public, and you have all of these definitions of AI alignment that you’ve given like intent alignment and impact alignment. And then we have the inner and outer alignment problems. How would you capture the core problem of AI alignment? And would you say that inner or outer alignment is a bigger part of the problem?

Rohin Shah: I would probably focus on intent alignment for the reasons I have given before. It just seems like a more … I want to focus attention away from the cases where the AI is trying to do the right thing, but makes a mistake, which would be a failure of impact alignment. But I don’t think that is the biggest risk. I think in a super-intelligent AI system that is trying to do the right thing is extremely unlikely to lead to catastrophic outcomes though it’s certainly not impossible. Or at least more unlikely to lead to catastrophic outcomes, unlike humans in the same position or something. So that would be my justification for intent alignment. I’m not sure that I would even talk very much about inner and outer alignment. I think I would probably just not focus on definitions and instead focus on examples. The core argument I would make would depend a lot on how AI systems are being built.

As I mentioned inner alignment is a problem that according to me, is primarily learning systems, I don’t think it really affects planning systems.

Lucas Perry: What is the difference between a learning system and a planning system?

Rohin Shah: A learning system, you give it examples of things it should do, how it should behave and then changes itself to do things more in that vein. A planning system takes a formerly represented objective and then searches over possible hypothetical sequences of actions it could take in order to achieve that objective. And if you consider a system like that, you can try to make the inner alignment argument and it just won’t work, which is why I say that the inner alignment problem is primarily about learning systems.

Going back to the previous question. So the things I would talk about depend a lot on what sorts of AI systems we’re building, if it were a planning system, I would basically just talk about outer alignment, where I would be like, what if the formerly represented objective is not the thing that we actually care about. It seems really hard to formally represent the objectives that we want.

But if we’re instead talking about deep learning systems that are being trained from human feedback, then I think I would focus on two problems. One is cases where the AI system knows something, but the human doesn’t. And so they came and gives a bad feedback as a result. So for example, the AI system knows that COVID was caused by a lab leak. It’s just like, got incontrovertible proof of this or something. And then, but we as humans are like, no, when it says COVID was caused by a lab leak, we’re like, we don’t know that, and we say no bad, don’t say that. And then when it says, it is uncertain whether COVID is the result of a lab leak or naturally or if it just occurred via natural mutations. And then we’re like, yes, good, say more of that. And you’re like, your AI system learns, okay, I shouldn’t report true things. I should report things that humans believe or something.

And so that’s one way in which you get AI systems that don’t do what you want. And then the other way would be more of this inner alignment style story, where I would point out how, even if you do train it, even if all your feedback on the training data points is good. If the world changes in some way, the AI system might stop doing good things.

I might go to example, I mean, I gave the Gmail with emoji polls for meeting scheduling example, but another one, now that I’m on the topic of COVID is, if you imagine an AI system, if you imagine a meeting scheduling AI assistant again, that was trained pre-pandemic, and then the pandemic hits, and it’s obviously never been trained on any data that was collected during such a global pandemic. And so when you then ask it to schedule a meeting with your friend, Alice, it just schedules drinks in a bar Sunday evening, even though clearly what you meant was a video call. And it knows that you meant a video call. It just learned the thing to do is to schedule outings with friends on Sunday nights at bars. Sunday night, I don’t know why I’m saying Sunday night. Friday night.

Lucas Perry: Have you been drinking a lot on your Sunday nights?

Rohin Shah: No, not even in the slightest. I think truly the problem is I don’t go to bars, so I don’t have it cached in my head that people go to bars.

Lucas Perry: So how does this all lead to existential risk?

Rohin Shah: Well, the main argument is, one possibility is that your AI system just actually learns to ruthlessly maximize some objective. That isn’t the one that we want. Make paperclips, is an stylized example to show what happens in that sort of situation. We’re not actually claiming that it will specifically maximize paperclips, but an AI system that really ruthlessly is just trying to maximize paperclips. It is going to prevent humans from stopping it from doing so. And if it gets sufficiently intelligent and can take over the world at some point, it’s just going to turn all of the resources in the world, into paperclips, which may or may not include the resources in human bodies, but either way, it’s going to include all the resources upon which we depend for survival.

Humans are definitely going, seem like they will definitely go extinct in that type of scenario. So again, not specific to paper clips. This is just; ruthless maximization of an objective, tends not to leave humans alive. Both of these… Well not both of the mechanisms, the inner alignment mechanism that I’ve been talking about, is compatible with an AI system that ruthlessly maximizes an objective that we don’t want.

It does not argue that it is probable, and I am not sure if I think it is probable, I think it is… But I think it is easily enough risk, that we should be really worrying about it, and trying to reduce it.

For the outer alignment style story, where the problem is that the AI may know information that you don’t, and then you give it bad feedback. One thing is just, this can exacerbate, this can make it easier for an inner alignment style story to happen, where the AI learns to optimize an objective, that isn’t what you actually wanted.

But even if you exclude something like that, Paul Christiano’s written a few posts about what a failure, how a human extinction level failure, of this form could look like. It basically looks like, all of your AI systems lying to you about how good the world is as the world becomes much, much worse. So for example, AI systems keep telling you that the things that you’re buying are good and helping your helping your lives, but actually they’re not, and they’re making them worse in some subtle way that you can tell. You were told, all of the information that you’re fed makes it seem like, there’s no crime, police are doing a great job of catching it, but really, this is just manipulation of the information you’re being fed, rather than actual amounts of crime where, in this case, maybe the crimes are being committed by AI systems, not even by humans.

In all of these cases, humans relied on some information sources to make decisions, the AI has new other information that the humans didn’t, the AI has learned, Hey, my job is to manage the information sources that humans get, so that the humans are happy, because that’s what they did during training. They gave good feedback in cases where the information sort of said, things were going well, even when things were not actually going well.

Lucas Perry: Right. It seems like if human beings are constantly giving feedback to AI systems, and the feedback is based on incorrect information and the AI’s have more information, then they’re going to learn something, that isn’t aligned with, what we really want, or the truth.

Rohin Shah: Yeah, I do feel uncertain about the extent to which this leads to human extinction without… It leads to, I think you can pretty easily make the case that, it leads to an existential catastrophe, as defined by, I want to say it’s Bostrom, which includes human extinction, but also a permanent curtailing of humanity’s I forget the exact phrasing, but basically if humanity can’t use… Yeah, exactly, that counts, and this totally falls into that category. I don’t know if it actually leads to human extinction, without some additional sort of failure, that we might instead categorize as inner alignment failure.

Lucas Perry: Let’s talk a little bit about probabilities, right? So if, you’re talking to someone who has never encountered AI alignment before, you’ve given a lot of different real world examples and principle-based arguments for, why there are these different kinds of alignment risks, how would you explain the probability of existential risk, to someone who can come along for all of these principle-based arguments, and buy into the examples that you’ve given, but still thinks this seems kind of, far out there, like when am I ever going to see in the real world, a ruthlessly optimizing AI, that’s capable of ending the world?

Rohin Shah: I think, first off, I’m super sympathetic to the ‘this seems super out there’ critique. I spent multiple years, not really agreeing with AI safety for basically, well, not just that reason, but that was definitely one that their heuristics that I was using. I think one way I would justify this is, to some extent it has precedent here, precedent already, in that fundamentally the arguments that I’m making… Well, especially the inner alignment one, is an argument about how AI systems will behave in new situations rather than the ones that we have already seen, during training. We already know, that AI systems behave crazily in these situations, the most famous example of this is adversarial examples, where you take an image classifier, and I don’t actually remember what the canonical example is. I think it’s a Panda, and you change it imperceptibly or change the pixel values by a small amounts, such that the changes are imperceptible to the human eye. And then it’s confident… It’s classified with, I think 99.8% confidence as something else. My memory is saying airplane, but that might just be totally wrong. Anyway, the point is we have precedent for it, AI system’s behaving really weirdly, in situations they weren’t trained on. You might object, that this one is a little bit cheating, because there was an adversary involved, and the real, I mean the real world does have adversaries, but still by default, you would expect the AI system to be more exposed to naturally occurring distributions. I think even there though, often you can just take an AI system that was trained on one distribution, give it inputs from a different distribution, and it’s just like there’s no sense to what’s happening.

Usually when I’m asked to predict this, the actual prediction I give is, probability that we go extinct due to an intent alignment failure, and then depending on the situation I will either condition on… I will either make that unconditional, so that includes all of the things that people will do to try to prevent that from happening. Or, I make it conditional, on the long-termist community doesn’t do anything, or vanishes or something. But even in that world, there’s still… Everyone who’s not a long-termist, who can still prevent that from happening, which I really do expect them to do, and then I think I give my cached answer, on both of those is like 5% and 10% respectively, which I think is probably the numbers I gave you. If I actually sat down and try to like come up with a probability, I would probably come up with something different this time, but I have not done that, and I’m way too anchored on those previous estimates, to be able to give you a new estimate this time. But, the higher number I’m giving now of, I don’t know, 33%, 50%, 70%, this, this one’s way more… I feel way more uncertain about it. Literally no one, tries to address these sorts of problems. It’s just sort of, take a language model, fine tune it on human feedback, in a very obvious way, and they just deploy that, even if it’s very obviously causing harm during training, they still deploy it.

What’s the chance that leads to human extinction? I don’t know, man, maybe 33%, maybe 70%. The 33% number you can get from this, one in three argument that I was talking about. The second thing I was going to say is, I don’t really like talking about probabilities very much, because of how utterly arbitrary the methods of generating them are there.

I feel much more, I feel much more robust. I feel much better in the robustness of the conclusion, that we don’t know that this won’t happen, and it is at least plausible, that it does happen. I think that’s pretty sufficient, for justifying the work done on it. I will also argue pretty strongly against anyone who says, we know that it will kill us all, if we don’t do anything. I don’t think that’s true. There are definitely, smart people who do think that’s true, if we operationalized greater than 90, 95% or something, and I disagree with them. I don’t really know why though.

Lucas Perry: How would you respond to someone, who thinks that this sounds, like it’s really far in the future?

Rohin Shah: Yeah. So this is specifically AGI is far in the future?

Lucas Perry: Yeah. Well, so the concern here seems to be about machines that are increasingly capable. When people look at machines that we have today, machine learning that we have today, sometimes they’re not super impressed and think that general capabilities are very far off.

Rohin Shah: Yeah.

Lucas Perry: And so this stuff sounds like, future stuff.

Rohin Shah: Yeah. So, I think my response depends on what we’re trying to get the person to do or something, why do we care about what this person believes, if this person is considering whether or not to do AI research themselves or, AI safety research themselves and they feel like they have a strong inside view model of, why AI is not going to come soon. I’m kind of… I’m like, eh, that seems okay. I’m not that stoked about people forcing themselves to do research on a thing they don’t actually believe. I don’t really think that good research comes from doing that. If I put myself, for example, I am much more sold on AGI coming through neural networks, than planning agents or things similar to it. If I had to put myself in the shoes of, all right, I’m now going to do AI safety research on planning agents. I’m just like, oh man, that’s seems like I’m going to do so much… My work is going to be orders of magnitude worse, than the work I do, on the neural-net case. So, in the case where, this person is thinking about whether to do AI safety research, and they feel like they have strong insight view models for AGI not coming soon. I’m like, eh, maybe they should go do something else or possibly, they should engage with the arguments for AGI coming more quickly, if they haven’t done that. But, if they have engaged with those arguments, thought about it all, concluded it’s far away, and they can’t even see a picture by which it comes soon…That’s fine.

Conversely, if we’re instead, if we’re imagining that someone is disputing, someone is saying, ‘oh nobody should work on AI safety right now, because AGI is so far away.’. One response you can have to that is, even if it’s far away, it’s still worthwhile to work on reducing risks, if they’re as bad as extinction. Seems like we should be putting effort into that, even early on. But I think, you can make a stronger argument there, which is there’re just actually people, lots of people who are trying to build AGI right now, there’s, at the minimum; DeepMind and OpenAI and they clearly… I should probably not make more comments about DeepMind, but OpenAI clearly doesn’t believe… OpenAI clearly seems to think, that AGI is coming somewhat soon. I think you can infer, from everything you see about DeepMind, that they don’t believe that AGI is 200 years away. I think it is insane overconfidence in your own views, to be thinking that you know better than all of these people, such that you wouldn’t even assign, like 5% or something, to AGI coming soon enough, that work on AI safety matters.

Yeah. So there, I think I would appeal to… Let other people do the work. You are not, you don’t have to do the work yourself. There’s just no reason for you to be opposing the other people, either on epistemic grounds or also on just, kind of a waste of your own time, that’s the second kind of person. A third kind of person might be like somebody in policy. From my impression of policy, is that there is this thing, where early moves are relatively irreversible, or something like that. Things get entrenched pretty quickly, such that it makes sense to wait for… It often makes sense to wait for a consensus before acting, and I don’t think that there is currently consensus of AGI coming soon. I don’t feel particularly confident enough in my views to say, we should really convince the policy people, to override this general heuristic of waiting for consensus, and get them to act now.

Yeah. Anyway, those are all meta-level considerations. There’s also the object-level question of, is AGI coming soon? For that, I would say, I think the most likely, the best story for that I know of is, you take neural nets, as you scale them up, you increase the size of the datasets that they’re trained on. You increase the diversity of the datasets that they’re trained on, and they learn more and more general heuristics, for doing good things. Eventually, these general, these heuristics are general enough that they’re as good as human cognition. Implicitly, I am claiming that human cognition, is basically a bag of general heuristics. There is this report from Ajeya Cotra, about AGI timelines using biological anchors. I wrote, even my summary of it was 3000 words, or something like that, so I don’t know that I can really give an adequate summary of it here, but it models… The basic premise, is to model how quickly neural nets will grow, and at what point they will match what we would expect to be approximately, the same rough size as the human brain. I think it even includes a small penalty to neural nets on the basis that evolution probably did a better job than we did. It basically comes up with a target for, neural nets of this size, trained in Compute Optimal ways, will probably be, roughly human level.

It has a distribution over this, to be more accurate, and then it predicts, based on existing trends. Well, not just existing trends, existing trends and sensible extrapolation, predicts when neural nets might reach that level. It ends up concluding, somewhere in the range… Oh, let me see, I think it’s 50% confidence interval would be something like 2035 to 2070, 2080, maybe something like that? I am really just like, I’m imagining a graph in my head, and trying to calculate the area under it, so that is very much not a reliable interval, but it should give you a general sense of what the report concludes.

Lucas Perry: So that’s 2030 to 2080?

Rohin Shah: I think it’s slightly narrower than that, but yes, roughly, roughly that.

Lucas Perry: That’s pretty soon.

Rohin Shah: Yep. I think that’s, on the object level that you’d just got to read the report, and see whether or not you buy it.

Lucas Perry: That’s most likely in our lifetimes, if we live to the average age.

Rohin Shah: Yep. So that was a 50% interval, meaning it’s, 25% to 75 percentile. I think actually the 25th percentile was not as early as 2030. It was probably 2040.

Lucas Perry: So, if I’ve heard everything, in this podcast, everything that you’ve said so far, and I’m still kind of like, okay, there’s a lot here and it sounds convincing or something and this seems important, but I’m not so sure about this, or that we should do anything. What is… Because, it seems like there’s a lot of people like that. I’m curious what it is, that you would say to someone like that.

Rohin Shah: I think… I don’t know. I probably wouldn’t try to say something general to them. I feel like I would need to know more about the person, people have pretty different idiosyncratic reasons, for having that sort of reaction. Okay, I would at least say, that I think that they are wrong, to be having that sort of belief or reaction.

But, if I wanted to convince them of that point, presumably I would have to say something more than just, I think you are wrong. I think the specific thing I would have to say, which would be pretty different for different people.

Lucas Perry: That’s a good point.

Rohin Shah: I would at least make an appeal to the meta-level heuristic of don’t try to regulate a small group of… There are a few hundred researchers at most, doing things that they think will help the world, and that you don’t think will hurt the world. There are just better things for you to do with your time. Doesn’t seem like they’re harming you. Some people will think that there is harm being caused by them. I would have to address that, with them specifically, but I think most people do not, who have this reaction, don’t believe that.

Lucas Perry: So, so we’ve gone over a lot of the traditional arguments for AI, as a potential existential risk. Is there anything else that you would like to add there, or any of the arguments that we missed, that you would like to include?

Rohin Shah: As a representative of the community as a whole, there are lots of other arguments that people like to make, for AI being a potential extinction risk. So, some things are, maybe AI just accelerates the rate at which we make progress, and we can’t increase our wisdom alongside, and as a result, we get a lot of destructive technologies and can’t keep them under control. Or, we don’t do enough philosophy, in order to figure out what we actually care about, and what’s good to do in the world, and as a result, we start optimizing for things that are morally bad or other things in this vein. Talk about the risk of AI being misused by bad actors. So there’s… Well actually I’ll introduce a trichotomy that, I don’t remember exactly who wrote this article. But it goes, Accidents, Misuse and Structural Risks. So accidents are, both alignment and the things like; we don’t keep up, we don’t have enough wisdom to cope with the impact of AI. That one’s arguable, whether it’s an accident, or misuse or structural, and we don’t do enough philosophy. So those are, vaguely accidental, those are accidents.

Misuse is, some bad actor. Some terrorists say, gets AI. Gets a powerful AI system and does something really bad, blows up the world somehow. Structural risks are things like; various parts of the economy use AI to accelerate, to get more profit, to accelerate their production of goods and so on. At some point we have this like giant economy, that’s just making a lot of goods, but it can become decoupled from things that are actually useful for humans, and we just have this huge multi-agency system, where goods are being produced, money’s floating around. We don’t really understand all of it, but somehow humans get left behind and there, it’s kind of an accident, but not in the traditional sense. It’s not that a single AI system went and did something bad. It’s more like the entire structure, of the way that the AI systems and the humans related to each other, was such that it ended up leading to the permanent disempowerment of humans. Now that I say it, I think the ‘we didn’t have enough wisdom’ argument for risk, is probably also in this category.

Lucas Perry: Which of these categories are you most worried about?

Rohin Shah: I don’t know. I think, it is probably not misuse, but I vary, on accidents versus structural risks, mostly because, I just don’t feel like I have a good understanding of structural risks. Maybe, most days I think structural risks are more likely to cause bad outcomes, extinction. The obvious next question is, why am I working on alignment, and not structural risks? The answer there, is that it seems to me like alignment has one, or perhaps two core problems that are leading to the major risk. Whereas structural risks… And so you could hope to have, one or two solutions that address those main problems and that’s it, that’s all you need. Whereas with structural risks, I would be surprised if it was just, there was just one or two solutions that just got rid of structural risk. It seems much more like, you have to have a different solution for each of the structural risks. So, it seems like, the amount that you can reduce the risk by, is higher in alignment than in structural risks. That’s not the only reason why I work in alignment, I just also have a much better personal fit with alignment work. But, I do also think that alignment work, you have more opportunity to reduce the risks, than in structural risks, on the current margin.

Lucas Perry: Is there a name for those one or two core problems in alignment, that you can come up with solutions for?

Rohin Shah: I mostly just mean like, possibly, we’ve been talking about outer and inner alignment, and in the neural net case, I talked about the problem where you reward the AI system for doing bad things, because there was an information asymmetry, and then the other one was like the AI system generalizes catastrophically, to new situations. Arguably those are just the two things, but I think it’s not even that, it’s more… Fundamentally the story, the causal chain in the accident’s case, was the AI was trying to do something bad, or something that we didn’t want rather, and then that was bad.

Whereas in the structural risks case, there isn’t a single causal story. It’s this very vague general notion of the humans and AI have interacted in ways that led to an X-risk. Then, if you drill down into any given story, or if you drilled down into five stories and then you’re like, what’s common across these five stories? Not much, other than that there was AI, and there were humans, and they interacted, and I wouldn’t say that was true, if I had five stories about alignment failure.

Lucas Perry: So, I’d like to take an overview, a broads eye view of AI alignment in 2021. Last time we spoke was in 2020. How has AI alignment, as a field of research changed in the last year?

Rohin Shah: I think I’m going to naturally include a bunch of things from 2020 as well. It’s not a very sharp division in my mind, especially because I think the biggest trend, is just more focus on large language models, which I think was a trend that started late 2020 probably… Certainly, the GPT-3 paper was, I want to say early 2020, but I don’t think it immediately caused there to be more work. So, maybe late 2020 is about right. But, you just see a lot more, alignment forum posts, and papers that are grappling with, what are the alignment problems that could arise with large language models? How might you fix them?

There was this paper out of Stanford, which isn’t, I wouldn’t have said this was from the AI safety community. But it gives the name foundation models to these sorts of things. So they generalize it beyond just language and they think it might… And already we’ve seen some generalization beyond language, like CLIP and DALL-E are working on image inputs, but they also extend it to robotics and so on. And their point is, we’re now more in the realm of, you train one large model on a giant pile of data that you happen to have, that you don’t really have any labels for, but you can use a self-supervised learning objective in order to learn from them. And then you get this model that has a lot of knowledge, but no goal built in, and then you do something like prompt engineering or fine tuning in order to actually get it to do the task that you want. And so that’s a new paradigm for constructing AI systems that we didn’t have before. And there have just been a bunch of posts that grapple with what alignment looks like in this case. I don’t think I have a nice pithy summary, unfortunately, of what all of us… What the upshot is, but that’s the thing people have been thinking about, a lot more.

Lucas Perry: Why do you think that looking at large scale language models has become a thing?

Rohin Shah: Oh, I think primarily just because GPT-3 demonstrated how powerful they could be. You just see, this is not specific to the AI safety community, even in the… If anything, this shift that I’m talking about is… It’s probably not more pronounced in the ML community, but it’s also there in the ML community where there are just tons of papers about prompt engineering and fine tuning out of regular ML labs. Just, I think is… GPT-3 showed that it could be done, and that this was a reasonable way to get actual economic value out of these systems. And so people started caring about them more.

Lucas Perry: So one thing that you mentioned to me that was significant in the last year, was foundation models. So could you explain what foundation models are?

Rohin Shah: Yeah. So a foundation model, the general recipe for it, is you take some very… Not generic, exactly. Flexible input space like pixels or any English language, any string of words in the English language, you collect a giant data set without any particular labels, just lots of examples of that sort of data in the wild. So in the case of pixels, you just find a bunch of images from image-sharing websites or something. I don’t actually know where they got their images from. For text, it’s even easier. The internet is filled with text. You just get a bunch of it. And then you train your AI, you train a very large neural network with some proxy objective on that data set, that encourages it to learn how to model that data set. So in the case of language models, the… There are a bunch of possible objectives. The most famous one was the one that GPT-3 used, which is just, given the first N words of the sentence, predict the word N plus one. And so it just… Initially it starts learning, E’s are the most common… Well, actually, because of the specific way that the input space in GPT-3 works, it doesn’t exactly do this, but you could imagine that if it was just modeling characters, it would first learn that E’s are the most common letter in the alphabet. L’s are more common. Q’s and Z’s don’t come up that often. Like it starts outputting letter distributions that at least look vaguely more like what English would look like. Then it starts learning what the spelling of individual words are. Then it starts learning what the grammar rules are. Just, these are all things that help it better predict what the next word is going to be, or, well, the next character, in this particular instantiation.

And it turns out that when you have millions of parameters in your neural network, then you can… I don’t actually know if this number is right, but probably, I would expect that with millions of parameters in your neural network, you can learn spellings of words and rules of grammar, such that you’re mostly outputting, for the most part, grammatically correct sentences, but they don’t necessarily mean very much.

And then when you get to the billions of parameters range, at that point, the millions of parameters are already getting you grammar. So like, what should it use all these extra parameters for, now? Then it starts learning things like George… Well, probably already even the millions of parameters probably learned that George tends to be followed by Washington. But it can start learning things like that. And in that sense, can be said to know that there is an entity, at least, named George Washington. And so on. It might start knowing that rain is wet, and in context where something has been rained on, and then later we’re asked to describe that thing, it will say it’s wet or slippery or something like that. And so it starts… It basically just, in order to predict words better, it keeps getting more and more “knowledge” about the domain.

So anyway, a foundation model, expressive input space, giant pile of data, very big neural net, learns to model that domain very well, which involves getting a bunch of “knowledge” about that domain.

Lucas Perry: What’s the difference between “knowledge” and knowledge?

Rohin Shah: I feel like you are the philosopher here, more than me. Do you know what knowledge without air quotes is?

Lucas Perry: No, I don’t. But I don’t mean to derail it, but yeah. So it gets “knowledge.”

Rohin Shah: Yeah. I mostly put the air quotes around knowledge because we don’t really have a satisfying account of what knowledge is. And if I don’t put air quotes around knowledge, I get lots of people angrily saying that AI systems don’t have knowledge yet.

Lucas Perry: Oh, yeah. That makes sense.

Rohin Shah: And when I put the air quotes around it, then they understand that I just mean that it has the ability to make predictions that are conditional on this particular fact about the world, whether or not it actually knows that fact about that world.

Lucas Perry: Okay.

Rohin Shah: But it knows it well enough to make predictions. Or it contains the knowledge well enough to make predictions. It can make predictions. That’s the point. I’m being maybe a bit too harsh, here. I also put air quotes around knowledge because I don’t actually know what knowledge is. It’s not just a defense strategy. Though, that is definitely part of it.

So yeah. Foundation models, basically are a way to just get all of this “knowledge” into an AI system, such that you can then do prompting and fine tuning and so on. And those, with a very small amount of data, relatively speaking, are able to get very good performance. Like in the case of GPT-3, you can like give it two or three examples of a task and it can start performing that task, if the task is relatively simple. Whereas if you wanted to train a model from scratch to perform that task, you would need thousands of examples, often.

Lucas Perry: So how has this been significant for AI alignment?

Rohin Shah: I think it has mostly provided an actual pathway to it, by which we can get to AGI. Or there’s more like a concrete story and path that leads to AGI, eventually. And so then we can take all of these abstract arguments that we were making before, and then see, try to instantiate them in the case of this concrete pathway, and see whether or not they still make sense. I’m not sure if at this point I’m imagining what I would like to do, versus what actually happened. I would need to actually go and look through the alignment newsletter database and see what people actually wrote about the subject. But I think there was some discussion of GPT-3 and the extent to which it is or isn’t a mesa optimizer.

Yeah. That’s at least one thing that I remember happening. Then there’s been a lot of papers that are just like, “Here is how you can train a foundation model like GPT-3 to do the sort of thing that you want.” So there’s learning to summarize from human feedback, which just took GPT-3 and taught it how to, or fine tuned it in order to get it to summarize news articles, which is an example of a task that you might want an AI system to do.

And then the same team at OpenAI just recently released a paper that actually summarized entire books by using a recursive decomposition strategy. In some sense, a lot of the work we’ve been doing in the past, in AI alignment was like how do we get AI systems to perform fuzzy tasks for which we don’t have a reward function? And now we have systems that could do these fuzzy tasks in the sense that they “have the knowledge,” but don’t actually use that knowledge the way that we would want them. And then we have to figure out how to get them to do that. And then we can use all these techniques like imitation learning, and learning from comparisons and preferences that we’ve been developing.

Lucas Perry: Why don’t we know that AI systems won’t totally kill us all?

Rohin Shah: The arguments for AI risk usually depend on having an AI system that’s ruthlessly maximizing an objective in every new situation it encounters. So for example, the paperclip maximizer, once it’s built 10 paperclip factories, it doesn’t retire and say, “Yep, that’s enough paperclips.” It just continues turning entire planets into paper clips. Or if you consider the goal of, make a hundred paper clips, and it turns all of the plants into computers to make sure it is as confident as possible, that it has made a hundred paper clips. These are examples of, I’m going to call it “ruthlessly maximizing” an objective. And there’s some sense in which this is weird and humans don’t behave in that way. And I think there’s some amount of, basically I am unsure whether or not we should actually expect AI’s to have such ruthlessly maximized objectives. I don’t really see the argument for why that should happen. And I think, as a particularly strong piece of evidence against this, I would note that humans don’t seem to have these sorts of objectives.

It’s not obviously true. There are probably some longtermists who really do want to tile the universe with hedonium, which seems like a pretty ruthlessly maximizing objective to me. But I think even then, that’s the exception rather than the rule. So if humans don’t ruthlessly maximize objectives and humans were built by a similar process as is building neural networks, why do we expect the neural networks to have objectives that they ruthlessly maximize?

You can also… I’ve phrased this in a way where it’s an argument against AI risk. You can also phrase it in a way in which it’s an argument for AI risk, where you would say, well, let’s flip that on its head and say like, “Well, yes, you brought up the example of humans. Well, the process that created humans is trying to maximize, or it is an optimization process, leading to increased reproductive fitness. But then humans do things like wear condoms, which does not seem great for reproductive fitness, generally speaking, especially for the people who are definitely out there who decide that they’re just never going to reproduce. So in that sense, humans are clearly having a large impact on the world and are doing so for objectives that are not what evolution was naively optimizing.

And so, similarly, if we train AI systems in the same way, maybe they too will have a large impact on the world, but not for what the humans were naively training the system to optimize.

Lucas Perry: We can’t let them know about fun.

Rohin Shah: Yeah. Terrible. Well, I don’t want to be-

Lucas Perry: The whole human AI alignment project will run off the rails.

Rohin Shah: Yeah. But anyway, I think these things are a lot more conceptually tricky than the well-polished arguments that one reads, will make it seem. But especially this point about, it’s not obvious that AI systems will get ruthlessly maximizing objectives. That really does give me quite a bit of pause, in how good the AI risk arguments are. I still think it is clearly correct to be working on AI risk, because we don’t want to be in the situation where we can’t make an argument for why AI is risky. We want to be in the situation where we can make an argument for why the AI is not risky. And I don’t think we have that situation yet. Even if you completely buy the, we don’t know if there’s going to be ruthlessly maximizing objectives, argument, that puts you in the epistemic state where we’re like, “Well, I don’t see an iron clad argument that says that AIs will kill us all.” And that’s sort of like saying… I don’t know. “Well, I don’t have an iron clad argument that touching this pan that’s on this lit stove, will burn me, because maybe someone just put the pan on the stove a few seconds ago.” But it would still be a bad idea to go and do that. What you really want, is a positive argument for why touching the pan is not going to burn you, or analogously, why building the AGI is not going to kill you. And I don’t think we have any such positive argument, at the moment.

Lucas Perry: Part of this conversation’s interesting because I’m surprised how uncertain you are about AI as an existential risk.

Rohin Shah: Yeah. It’s possible I’ve become slightly more uncertain about it in the last year or two. I don’t think I was saying things that were quite this uncertain before then, but I think I have generally been… We have plausibility arguments. We do not have like, this is probable, arguments. Or back in 2017 or 2018 when I was young and naive.

Lucas Perry: Okay.

Rohin Shah: This makes more sense.

Lucas Perry: We’re no longer young and naive.

Rohin Shah: Well, okay. I entered the field of AI alignment. I read my first AI alignment paper in September of 2017. So it actually does make sense. At that time, I thought we had more confidence of some sort, but since posting the value learning sequence, I’ve generally been more uncertain about AI risk arguments. I don’t talk about it all that much, because as I said, the decision is still very clear. The decision is still, work on this problem. Figure out how to get a positive argument that the AI is not going to kill us. And ideally, a positive argument that the AI does good things for humanity. I don’t know, man. Most things in life are pretty uncertain. Most things in the future are even way, way, way more uncertain. I don’t feel like you should generally be all that confident about technologies that you think are decades out.

Lucas Perry: Feels a little bit like those images of the people in the fifties drawing what the future would look like, and the images are ridiculous.

Rohin Shah: Yep. Yeah. I I’ve been recently watching Star Wars. Now, obviously Star Wars is not actually supposed to be a prediction of the future, but it’s really quite entertaining, to actually just think about all the ways in which Star Wars would be totally inaccurate. And this is before we’d even invented space travel. And just… Robots talking to each other, using sound. Why would they do that?

Lucas Perry: Industry today, wouldn’t make machines that speak by vibrating air. They would just send each other signals electromagnetically. So how much of the alignment and safety problems in AI do you think will be solved by industry? The same way that computer-to-computer communication is solved by industry, and is not what Star Wars thought it would be. Would the DeepMind AI safety lab exist, if DeepMind didn’t think that AI alignment and AI safety were serious and important? I don’t know if the lab is purely aligned with the commercial interests of DeepMind itself, or if it’s also kind of seen as a good-for-the-world thing. I bring it up because I like how Andrew Critch talks about it in his arches paper.

Rohin Shah: Yep. So, Critch is, I think, of the opinion that both preference learning and robustness are problems that will be solved by industry. I think he includes robustness in that. And I certainly agree to the extent that you’re like, “Yes, companies will do things like learning from human preference.” Totally. They’re going to do that. Whether they’re going to be proactive enough to notice the kinds of failures I mentioned, I don’t know. It doesn’t seem nearly as obvious to me that they will be, without dedicated teams that are specifically meant for looking for hidden failures with the knowledge that these are really important to get, because they could have very bad long term consequences.

AI systems could increase the strength of, and accelerate various multi-agent systems and processes that, when accelerated, could lead to bad outcomes. So for example, a great example of a destructive multi-agent effect, is war. War is a thing that… Well, wars have been getting more destructive over time, or at least the weapons in them have been getting more destructive. Probably the death tolls have also been getting higher, but I’m not as sure about that. And you could imagine that if AI systems continue to increase, if they increase the destructiveness of weapons even more, wars might then become an existential risk. That’s a way in which you can get a structural risk from a multi-agent system. And the example in which the economy just sort of becomes much, much, much bigger, but becomes decoupled from things that humans want, is another example of how a multi-agent process can sort of go haywire, especially with the addition of powerful AI systems. I think that’s also a canonical scenario that Critch would think about. Yeah.

Really, I would say that Arches is, in my head, it’s categorized as a technical paper about structural risks.

Lucas Perry: Do you think about what beneficial futures look like? You spoke a little bit about wisdom earlier, and I’m curious what good futures with AI, looks like to you.

Rohin Shah: Yeah, I admit I don’t actually think about this very much. Because my research is focused on more abstract problems, I tend to focus on abstract considerations, and the main abstract consideration from the perspective of the good future, is, well, once we get to singularity levels of powerful AI systems, anything I say now, there’s going to be something way better that AI systems are going to enable. So then, as a result, I don’t think very much about it. But that’s mostly a thing about me not being in a communications role.

Lucas Perry: You work a lot on this risk. So you must think that humanity existing in the future, matters?

Rohin Shah: I do like humans. Humans are pretty great. I count many of them amongst my friends. I’ve never been all that good at the transhumanist, look to the future and see the grand potential of humanity, sorts of visions. But when other people say them or give them, I feel a lot of kinship with them. The ones that are all about humanity’s potential to discover new forms of art and music, reach new levels of science, understand the world better than it’s ever been understood before, fall in love a hundred times, learn all of the things that there are to know. Actually, you won’t be able to do that one, probably, but anyway. Learn way more of the things that there are to know, than you have right now. Just a lot of that resonates with me. And that’s probably a very intellectual-centric view of the future. I feel like I’d be interested in hearing the view of the future that’s like, “Ah yes, we have the best video games and the best TV shows. And we’re the best couch potatoes that ever were.” Or also, there’s just insane new sports that you have to spend lots of time and grueling training for, but it’s all worth it when you shoot the best, get a perfect score on the best dunk that’s ever been done in basketball, or whatever. I recently watched a competition of apparently there are competitions in basketball of just aesthetic dunks. It’s cool. I enjoyed it. Anyway. Yeah. It feels like there’s just so many other communities that could also have their own visions of the future. And I feel like I’d feel a lot of kinship with many of those, too. And I’m like, man, let’s just have all the humans continue to do the things that they want. It seems great.

Lucas Perry: One thing that you mentioned was that you deal with abstract problems. And so what a good future looks like to you, it seems like it’s an abstract problem that later, the good things that AI can give us, are better than the good things that we can think of, right now. Is that a fair summary?

Rohin Shah: That seems, right. Yeah.

Lucas Perry: Right. So there’s this view, and this comes from maybe Steven Pinker or someone else. I’m not sure. Or maybe Ray Kurzweil, I don’t know… Where if you give a caveman a genie, or an AI, they’ll ask for maybe a bigger cave, and, “I would like there to be more hunks of meat. And I would like my pelt for my bed to be a little bit bigger.” Go ahead.

Rohin Shah: Okay. I think I see the issue. So I actually don’t agree with your summary of the thing that I said.

Lucas Perry: Oh, okay.

Rohin Shah: Your rephrasing was that we ask the AI what good things there are to do, or something like that. And that might have been what I said, but what I actually meant was that with powerful AI systems, the world will just be very different. And one of the ways in which it will be different is that we can get advice from AIs on what to do. And certainly, that’s an important one, but also, there will just be incredible new technologies that we don’t know about. New realms of science to explore new concepts that we don’t even have names for, right now. And one that seems particularly interesting to me, is just entirely new senses. Human vision is just incredibly complicated, but I can just look around the room and identify all the objects with basically no conscious thought. What would it be like to understand DNA at that level? AlphaFold probably understands DNA at maybe not quite that level, but something like it.

I don’t know, man. There’s just like these things that I’m like… I thought of the DNA one because of AlphaFold. Before AlphaFold, would I have thought of it? Probably not. I don’t know. Maybe. Kurzweil has written a little bit about things like this. But it feels like there will just be far more opportunities. And then also, we can get advice from AIs, but that’s probably… Actually- and that’s important, but I think less than… There are far more opportunities, that I am definitely not going to be able to think of today.

Lucas Perry: Do you think that it’s dissimilar, from the caveman wishing for more caveman things?

Rohin Shah: Yeah. I feel like in the caveman story… It’s possible that the caveman does this, I feel like the thing the caveman should be doing, is something like, give me better ways to… give me better food or something, and then you get fire to cook things, or something.

Lucas Perry: Yeah.

Rohin Shah: The things that he asks for, should involve technology as a solution. He should get technology as a solution, to learn more, and be able to do more things as a result of having that technology. In this hypothetical, the caveman should reasonably quickly, become similar to modern humans. I don’t know what reasonably quickly means here, but it should be much more… You get access to more and more technologies, rather than you get a bigger cave and then you’re like, “I have no more wishes anymore.” If I got a bigger house, would I stop having wishes? That seems super unlikely. That’s a strawman argument, sorry. But still, I do feel like there’s this… A meaningful sense in which, getting new technology leads to just genuinely new circumstances, which leads to more opportunities, which leads to probably more technology, and so on, and at some point, this has to stop. There are limits to what is possible. One assumes there are limits to what is possible in the universe. But I think, once we get to talking about, we’re at those limits, then at that point, it just seems irresponsible to speculate. It’s just so wildly out of the range of things that we know, the concept of a person is probably wrong, at that point.

Lucas Perry: The what of a person is probably wrong at that point?

Rohin Shah: The concept of a person.

Lucas Perry: Oh.

Rohin Shah: I’d be like, “Is there an entity, that is Rohin at that time?” Not likely. Less than 50%.

Lucas Perry: We’ll edit in just fractals flying through your video, at this part of the interview. So in my example, I think it’s just because I think of cavemen as not knowing how to ask for new technology, but we want to be able to ask for new technology. Part of what this brings up for me, is this very classic part of AI alignment, and I’m curious how you feel like it fits into the problem.

But, we would also like AI systems to help us imagine beneficial futures potentially, or to know what is good or what it is that we want. So, in asking for new technology, it knows that fire is part of the good, that we don’t know how to necessarily ask for directly. How do you view AI alignment, in terms of itself aiding in the creation of beneficial futures, and knowing of a good that is beyond the good, that humanity can grasp?

Rohin Shah: I think I more reject, the premise of the question, where I’d be like, there is no good beyond that which humanity can grasp. This is somewhat of an anti-realist position.

Lucas Perry: You mean, moral anti-realist, just for the-

Rohin Shah: Yes. Sorry, I should have said that more clearly. Yeah. Somewhat of a moral anti-realist position. There is no good, other than that which humans can grasp. Within that ‘could grasp’, you can have humans thinking for a very long time, you could have them with extra… you can make them more intelligent, like part of the technologies you get from AI systems will presumably like you do that, maybe you can, I guess setting aside questions of philosophical identity, you could upload the humans such that they could run on a computer, and run much faster, have software upgrades to be… To the extent that, that’s philosophically acceptable. There’s a lot you can do to help humans grasp more. Ultimately, yes, the closure of all these improvements, where you get to with all of that, that’s just, is the thing that we want. Yes, you could have a theory, that there is something even better, and even more out there, that humans can never access by themselves, that just seems like a weird hypothesis to have, and I don’t know why you would have it. But, in the world where that hypothesis is true, and if I condition on that hypothesis being true, I don’t see why we should expect, that AI systems could access that further truth any better than we can, if it’s out of our, the closure of what we can achieve, even with additional intelligence and such. There’s no other advantage that AI systems have over us.

Lucas Perry: So, is what you’re arguing, that with human augmentation and help to human beings, so like with uploads or with expanding the intelligence and capabilities of humans, that humans have access to the entire space of what counts as good.

Rohin Shah: I think you’re presuming the existence of an object that is the entire space of what is good. And I’m like, there is no such object, there are only humans, and what humans want to do. If you want to define the space of what is good, you can define this closure property on what humans will think is good, with all of the possible intelligence augmentations and time, and so on. That’s a reasonable object, and I could see calling that as the space of what is good. But then, almost tautologically, we can reach it with technology. That’s the thing I’m talking about. The version where you posit the existence of the entire space of what is good is: A, I can’t really conceive of that, it doesn’t feel very coherent to me, but B, when I try to reason about it anyway, I’m like, okay, if humans can’t access it, why should AI’s be able to access it? You’ve posited this new object of, a space of things, that humans can never access, but how does that space affect or interact with reality in any way? There needs to be some sort of interaction, in order for the AI to be able to access it. I think I would need to know more about how it interacts with reality in some way, before I could meaningfully answer this question in a way, where I could say how AI’s could do something, that humans couldn’t even in principle, do.

Lucas Perry: What do you think of the importance, or non importance of these kinds of questions, and how they fit into the ongoing problem of AI alignment?

Rohin Shah: I think they’re important, for determining what the goal of alignment should be. So for example, you now know a little bit of what my view on these questions is, which is namely something like… That which humans can access, under sufficient augmentations, intelligence, time and so on, is all that there is. So I’m very into… build AI systems that are replicating human reasoning, they’re approximating what a human would do, if they thought for a long time, or were smarter in some ways and so on. So then, yeah we don’t need to worry much about… I tend to think of it as, let’s build an AI systems that just do tasks, that humans can conceptually understand, not necessarily they can do it, but they know what that task is. Then, our job is to, the entire human AI society is making forward progress towards… Making forward moral progress or other progress, in the same way that if this happened in the past, we get exposed to new situations and new arguments, we think about them for a while, and then somehow we make decisions about what’s good and what’s not, in a way that’s somewhat inscrutable. I’m much more about… So we just continue reiterating that process, and eventually we reach the space of, well yeah, we just continue reiterating that process. So I’m very much into, because of this view, I think it’s pretty reasonable to aim for AI systems that are just doing human-like reasoning, but better. Or approximating, doing what a human could do in a year, in a few minutes or something like that. That seems great to me. Whereas if you, on the other hand were like, no, there’s actually deep philosophical truths out there, that humans might never be able to access, then you’re probably less enthusiastic about that sort of plan, and you’ll want to build an AI system some other way.

Lucas Perry: Or maybe they’re accessible, with the augmentation and time. How does other minds fit into this for you? So, right, there’s the human mind and then the space of all that is good, that it has access to, with augmentation, which is what you call the space, of that which is good. It’s contingent, and rooted on the space of what the human mind, augmented has access to. How would you view, how does that fit in with animals and also other species which may have their own alignment problems on planets within our cosmic endowment that we might run into? Is it just that they also have spaces that are defined as good, as what they can access through their own augmentation? And then, there’s no way of reconciling these two different AI alignment projects?

Rohin Shah: Yeah, I think basically, yes. If I met an actual ruthless, maximizing paperclip… Paperclip maximizer. It’s not like I can argue it, into adopting my values, or anything even resembling them. I don’t think it would be able to argue me into accepting turning me into paperclips, which is what it desires, and that just seems like the description of reality. Again, a moral realist might say something else, but I’ve never really understood the flavor of moral realism that would say something else in that situation.

Lucas Perry: With regards to the planet and industry, and how industry will be creating increasingly capable AI systems. Could you explain what a unipolar scenario is, and what a multi-polar scenario is?

Rohin Shah: Yeah, so I’m not sure if I recall exactly where these terms were defined, but a unipolar scenario, at least as I understand it, would be a situation in which, one entity basically determines the long run future of the earth. More colloquially, it has taken over the world. You can also have a time bounded version of it, where it’s unipolar for 20 years, and this entity has all the power for those 20 years, but then, maybe the entity is a human, and we haven’t solved aging yet, and then the human dies. So then, it was a unipolar world for that period of time. And a multipolar world is just, not that. There is no one entity, that is said to be in control of the world. There’s just a lot of different entities that have different goals, and they’re coexisting, hopefully cooperating, maybe not cooperating, depends on the situation.

Lucas Perry: Which do you think is more likely to lead to beneficial outcomes, with AI?

Rohin Shah: So, I don’t really think about it in these terms. I think about it in like, there are these kinds of worlds that we could be in, some of them are unipolar and some of them are multipolar, but very different unipolar worlds, and very different multipolar worlds. And so, the sorts of questions, the closest analogous question is something like, if you condition on unipolar world, what’s the probability that it’s beneficial or that it’s good. If you condition on multipolar world, what’s the probability that is good? And it’s just a super complicated question that I wouldn’t be able to explain my reasoning for, because it would involve me like thinking about 20 different worlds, maybe not that many, but a bunch of different worlds in my head, estimating their probabilities by doing a base rule… I guess, kind of a base rule calculation, and then reporting the result.

So, I think maybe the question I will answer instead, is the most likely worlds in each of unipolar, and multipolar settings, and then, how good those seem to me. So I would say, I think by default, I expect the world to be multi-polar, in that it doesn’t seem like anyone is particularly. I don’t think anyone has particularly taken over the world today, or any entity, not even counting the US as a single entity. It’s not like the US has taken over the world. It does not seem to me like… Though the main way you could imagine getting a unipolar world is, if the first actor to build a powerful enough AI system, that AI system just becomes really, really powerful and takes over the world, before anyone can deploy an AI system even close to it.

Sorry, that’s not the most likely one. That’s the one that most people most often talk about, and probably the one that other people think is the most likely, but yeah. Anyway, I see the multipolar world as more likely, where we just have a bunch of actors that are all pretty well-resourced, that are all developing their own AI systems. They then sell their AI systems, or the ability to use their AI systems to other people, and then it’s sort of similar to the human economy, where you can just have AI systems provide labor at a fixed cost. It looks similar to the economy today, where people who control a lot of resources can instantiate a bunch of AI systems, that help them maintain whatever it is they want, and we remain in the multipolar world we have today.

And that seems… Decent. I think, for all that our institutions are not looking great, at the current moment. There is still something to be said, that nuclear war didn’t actually happen, which can either update you towards, our institutions are somewhat better than we thought, or it can update you towards, if we had nuclear war, we would have all died, and not been here to ask the question. I don’t think that second one, is all that possible. My understanding, is that nuclear war is not that likely to wipe out everyone, or even 90% of people. So I’m more… I lean towards the first explanation. Overall, my guess is, this is the thing that has worked for the last… ‘Worked’, the thing that has, generally led to an increase in prosperity. Or, the world has clearly improved on most metrics over time. And, this system we’ve been using, for most of that time is some sort of multipolar, people interact with each other and keep each other in check, and cooperate with each other because they have to, and so on. In the modern world we use, and not just the modern world, we use things like regulations and laws and so on, to enforce this. The system’s got some history behind it, so I’m more inclined to trust it. But overall, I feel okay about this world, assuming we solve the alignment problem, we’ll ignore the alignment problem for now.

For a unipolar world. I think, probably, I find it more likely that there will just be a lot of returns to scale. You’ll get a lot of efficiency from centralizing more and more, in the same way that it’s just really nice to have a single standard, rather than have 15 different standards. It sure would have been nice, if when I moved to the UK, I could have just used all of my old chargers without having to buy adapters. But no, all the outlets are different, right? There’s benefits to standardization and centralization of power, and it seems to me, there has been more and more of that over time. Maybe it’s not obvious, I don’t know very much history, but if… So, it seems like you could get, even more centralization in the future, in order to capture the efficiency benefits, and then you might have a global government that could reasonably be said to be the entity that controls the world, and that would then be a unipolar outcome. It’s not a unipolar outcome in which the thing in charge of the world is an AI system. It is a unipolar outcome. I feel wary of this, but I don’t like having a single point of failure. I don’t like it when there’s a… However, I really like it when people are allowed to advocate for their own interests, which isn’t necessarily not happening here, right?

This could be a global democracy, but still, it seems like, the libertarian intuition of markets are good, generally tends to suggest against centralization, and I do buy that intuition, but this could also just be status quo bias, where I know that I can very easily see the problems in the world that we’re not actually in at the moment, and I don’t want it to change. So I don’t know, I don’t have super strong opinions there. It’s very plausible to me that that world is better, because then you can control dangerous technologies much, much better. If there just are technologies that are sufficiently dangerous and destructive, they would destroy, they would lead to extinction, then maybe I’m more inclined to favor a unipolar outcome.

Lucas Perry: I would like to ask you about DeepMind, and maybe another question before we wrap up. What is it, that the safety team at DeepMind is up to?

Rohin Shah: No one thing. The safety team at DeepMind is reasonably large, and there’s just a bunch of projects going on. I’ve been doing a bunch of inner alignment stuff. Most recently, I’ve been trying to come up with more examples that are, in actual systems, rather than hypotheticals. I’ve also been doing a bunch of conceptual work, of just trying to make our arguments clearer, and more conceptually precise. A large smattering of stuff, not all that related to each other, except in as much as it’s all about AI alignment.

Lucas Perry: As a final question here, Rohin, I’m interested in your core at the center of all of this. What’s the most important thing to you right now? Insofar as, AI alignment, may be the one thing, that most largely impacts the future of life?

Rohin Shah: Ah.

Lucas Perry: If you just look at the universe right now, and you’re like, these are the most important things.

Rohin Shah: I think, for things that I impact, at a more granular, more granular than just, make AI go well… I think for me, it’s probably making better arguments and more convincing arguments, currently. This will probably change in the future. Partially because I hope to succeed at the skill, and then it won’t be as important. But I feel like right now, especially with the advent of these large neural nets, and more people seeing a path to AGI, I think it is much more possible to make arguments that would be convincing to ML researchers as well, as well as the philosophically oriented people who make up the AI safety community, and I think, that just feels like the most useful thing I can do at the moment. In terms of the world in general… I feel like it is something like the attitudes of consequential people, two words… Well, long-termism in general, but maybe risks in particular, where, and importantly, I do feel it has more… I care primarily about, the people who are actually making decisions, that impact the future. Maybe they are taking into account the future. Maybe they’re like, it would be nice to care about the future, but the realities of politics mean that I can’t do that, or else I will lose my job. But my guess is that they’re mostly just not thinking about the future. That seems… If you’re talking about the future of life, that seems like the most, that seems pretty important to change.

Lucas Perry: How do you see doing that, when many of these people don’t have the… As Sam Harris put it, ‘the science fiction geek gene’ is what he called it, when he was on this podcast. The long-termists, who are all, we’re going to build AGI, and then create these radically different futures. Many of these people, may just mostly care about their children and their grandchildren, that may be the human tendency.

Rohin Shah: Do we actually advocate for any actions that would not impact their grandchildren?

Lucas Perry: It depends on your timelines, right?

Rohin Shah: Fair enough. But, most of the time, the arguments that I see people giving for any preferred policy proposal of theirs, or act… Just like almost any action whatsoever. It seems be a thing, that would have a noticeable effect on people’s lives in the next 100 years. So, in that sense, grandchildren should be enough.

Lucas Perry: Okay. So then long-termism doesn’t matter.

Rohin Shah: Well… I don’t-

Lucas Perry: For getting the action done.

Rohin Shah: Oh, possibly. I still think they’re not thinking about the future. I think it’s more of a… I don’t know, if I had to take my best guess at it, with noting the fact that I am just a random person, who is not at all an expert in these things, because why would I be? And yes listeners, noting that Lucas has just asked me this question, because it sounds interesting, and not because I am at all, qualified to answer it.

It seems to me, the more likely explanation is that there are just always a gazillion things to do. There’s always $20 bills to be picked off the sidewalk, but their value is only $20. They’re not $2 billion. Everyone is just constantly being told to pick up all the $20 bills, and as a result, they are in a perpetual state of having to say no to stuff, and doing only the stuff that seems most urgent, and maybe also important. So, most of our institutions tend to be in a very reactive mindset, as a result. Not because they don’t care, but just because that’s the thing that they’re incentivized to do, is to respond to the urgent stuff.

Lucas Perry: So, getting policymakers to care about the future, whether that even just includes children and grandchildren, not the next 10 billion years, would be sufficient in your view?

Rohin Shah: It might be, it seems plausible. I don’t know that that’s the approach I would take. I think I’m more just saying, I’m not sure that you even need to convince them to care about the future, I think-

Lucas Perry: I see.

Rohin Shah: It’s possible, that what’s needed is people who have the space to bother thinking about it. I get paid to think about the future, if I didn’t get paid to think about the future, I would not be here on this podcast because I would not have enough knowledge to be worth talking, you talking to. I think, there are just not very many people who can be paid to think about the future, and the vast majority of them are in there… I don’t know about the vast majority, but a lot of them are in our community. Very few of them are in politics. Politics generally seems to anti-select for people who can think about the future. I don’t have a solution here, but that is the problem as I see it, and if I were designing a solution, I would be trying to attack that problem.

Lucas Perry: That would be one of the most important things.

Rohin Shah: Yeah. I think on my view, yes.

Lucas Perry: All right. So, as we wrap up here, is there anything else you’d like to add, or any parting thoughts for the audience?

Rohin Shah: Yeah. I have been giving all these disclaimers during the podcast too, but I’m sure I missed them in some places, but I just want to note, Lucas has asked me a lot of questions that are not things I usually think about, and I just gave off-the-cuff answers. If you asked me them again, two weeks from now, I think for many of them, I might actually just say something different. So don’t take them too seriously, and treat… The AI alignment ones, I think you can take those reasonably seriously, but the things that were less about that, take them as some guy’s opinion, man.

Lucas Perry: ‘Some guy’s opinion, man.’

Rohin Shah: Yeah. Exactly.

Lucas Perry: Okay. Well, thank you so much for coming on the podcast Rohin, it’s always a real pleasure to speak with you. You’re a bastion of knowledge and wisdom in AI alignment and yeah, thanks for all the work you do.

Rohin Shah: Yeah. Thanks so much for having me again. This was fun to record.

• The industrial and commercial uses of chlorofluorocarbons (CFCs)
• How we discovered the atmospheric effects of CFCs
• The Montreal Protocol and its significance
• Dr. Solomon’s, Dr. Farman’s, and Dr. Andersen’s crucial roles in helping to solve the ozone hole crisis
• Lessons we can take away for climate change and other global catastrophic risks

Watch the video version of this episode here

Check out the story of the ozone hole crisis here

0:00 Intro

3:13 What are CFCs and what was their role in society?

7:09 James Lovelock discovering an abundance of CFCs in the lower atmosphere

12:43 F. Sherwood Rowland’s and Mario Molina’s research on the atmospheric science of CFCs

19:52 How a single chlorine atom from a CFC molecule can destroy a large amount of ozone

23:12 Moving from models of ozone depletion to empirical evidence of the ozone depleting mechanism

24:41 Joseph Farman and discovering the ozone hole

30:36 Susan Solomon’s discovery of the surfaces of high altitude Arctic clouds being crucial for ozone depletion

47:22 The Montreal Protocol

1:00:00 Who were the key stake holders in the Montreal Protocol?

1:03:46 Stephen Andersen’s efforts to phase out CFCs as the co-chair of the Montreal Protocol Technology and Economic Assessment Panel

1:13:28 The Montreal Protocol helping to prevent 11 billion metric tons of CO2 emissions per year

1:18:30 Susan and Stephen’s key takeaways from their experience with the ozone hole crisis

1:24:24 What world did we avoid through our efforts to save the ozone layer?

1:28:37 The lessons Stephen and Susan take away from their experience working to phase out CFCs from industry

1:34:30 Is action on climate change practical?

1:40:34 Does the Paris Agreement have something like the Montreal Protocol Technology and Economic Assessment Panel?

1:43:23 Final words from Susan and Stephen

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. This is a special episode with the winners of the 2021 Future of Life Award. This year’s winners are Susan Solomon, Stephen Andersen, and Joseph Farman, who all played an essential part in the efforts related  to identifying and mending the ozone hole. In this podcast, we tell the story of the ozone hole from the perspective of Drs. Solomon and Andersen as  participants to the mystery of the ozone hole and the subsequent governance efforts related to mending it. Unfortunately, Joseph Farman passed away in 2013, so his role in this story will be told through our guests on his behalf.

For those not familiar with the Future of Life Award, this is a $50,000/person annual prize that we give out to honor unsung heroes who have taken exceptional measures to safeguard the future of humanity. In 2017 and 2018 the award honored Vasili Arkhipov and Stanislav Petrov for their roles in helping to avert nuclear war. In 2019 the award honored Dr. Matthew Meselson for his contributions to getting biological weapons banned. In 2020, the award honored Bill Foege and Viktor Zhdanov for their critical contribution to eradicating small pox and thus saving roughly 200 million lives so far.

For some background on this year’s winners, Dr. Susan Solomon was the Chemistry and Climate Processes Group of the National Oceanic and Atmospheric Administration until 2011. She now serves as the Ellen Swallow Richards Professor of Atmospheric Chemistry and Climate science at MIT. Dr. Solomon led an Antarctic ozone research expedition which both confirmed that CFCs caused the ozone hole, and showed that sunlit cloud tops catalyzed the ozone destruction process to be much faster.

Dr. Stephen Andersen is the American Director of Research at the Institute for Governance and Sustainable Development, and is former co-chair of the Montreal Protocol Technology and Economic Assessment Panel. Andersen’s tireless efforts brought together leaders from industry, government and academia to implement the  needed changes in CFC use to mend the ozone hole. His efforts played a critical role in making the Montreal Protocol successful.

Joseph Farman was a British geophysicist who worked for the British Antarctic Survey. In 1985, his team made the most important geophysical discovery of the 20th century: the ozone hole above the Antarctic. This provided a stunning confirmation of the Rowland-Molina hypothesis that human-made chlorofluorocarbons were destroying the ozone layer, and much faster than predicted. This galvanized efforts to take action to mend the ozone hole.

And with that, I’m happy to present the story of the ozone hole and the fight to fix it with Susan Solomon and Stephen Andersen 

Lucas Perry: So I’d like to start off with the beginning where it’s the early 20th century, and we have CFCs, chlorofluorocarbons, which is a difficult word to say. And they’re in quite a lot of our manufacturing and commercial goods. So I’m curious if you could start there by just explaining what CFCs are and what their role was in the early 20th century.

Susan Solomon: They weren’t actually discovered until I think around the twenties, if I recall, maybe Steven can correct me on that, but they were initially used in air conditioning. So that was a great advance over using things like ammonia, which is toxic and explosive and all kinds of horrible things, and the great thing about the CFCs is that they’re non-toxic at least, when you breathe them, although they’re very toxic to the ozone layer, they’re not toxic to you directly. And they became really widespread in use though, when they began to be used as spray cans in aerosol propellants and that didn’t really happen until somewhat later. I think that the spray can business really started exploding in the Post-World War Two era, probably in the 50’s and 60’s. And then it was discovered that these things have very long lifetimes in the atmosphere, they live, depending on which one you talk about, for some 50 to a 100 years.

So that is just staggering because what it means is that every year, whatever we put in, almost all of it will still be there the next year. And it’ll just pile up and pile up and pile up. So initially, a few people who worked on it thought “Hey, this is great! It’s a terrific tracer for atmospheric motion, how cool is that?” But it turned out that although that’s somewhat true in the lower atmosphere, in the upper atmosphere, they actually break down and make chlorine atoms, and those chlorine atoms can go on to destroy ozone. And we first became aware of this through some wonderful work by Molina and Rowland in the mid-seventies, which later won the Nobel Prize for chemistry, all along with Paul Crutzen.

Stephen Andersen: I could circle back on a little bit of the details of the technology if you’d like?

Susan Solomon: Sure, go for it, Steve.

Stephen Andersen: The year was in the twenties, 1929, and it was invented by Thomas Midgley, who was working for Frigidaire Division of General Motors, and just as Susan said, this was viewed as a wonder chemical because the chemicals at that time were things like methylene chloride and ammonia and even gasoline is a refrigerant, so it was very dangerous there were lots of injuries, by that time America’s waters were polluted, so it was difficult to use ice from ponds.

And so it was immediately commercialized to replace these flammable and toxic refrigerants, and then just as Susan said, the companies that produced it slowly looked at other markets; aerosol starting with pesticides for soldiers in World War Two, and then finding a commercial market and then solvents, because it’s a very effective solvent for electronics and aerospace, and then there were other uses that came along such as making rigid and flexible foam, and then finally towards the end there were elegant uses such as hospital sterilization, and as the aerosol for metered-dose inhalers that are used for asthma patients. So it had extraordinary uses and it was not appreciated until 1974 when Mario Molina and Sherry Rowland discovered that these chemicals could destroy the ozone layer.

Lucas Perry: So before we get to Molina and Rowland, I was curious if we could actually start with James Lovelock because we’re releasing and creating all of these CFCs, and it seems like he’s the first person who actually takes a look at these things in the atmosphere. So could you explain James Lovelock’s role in this story?

Susan Solomon: Sure. He invented the gas chromatograph and made a lot of money off of it and became independently wealthy because it’s a very useful instrument for measuring all kinds of things. And he took it on a… He was British, and he took it on a British research vessel and sailed from the Northern Hemisphere down to almost the Antarctic, and showed that he could understand exactly what the distribution of the CFC that he was measuring was, based on the amount that had been emitted. And he said “Oh, isn’t that cool? It’s a great tracer for atmospheric motion”, as I’ve mentioned before. In fact, I’m pretty sure in his paper, there was some kind of sentence like “this couldn’t ever conceivably pose any hazard because it’s so non-toxic”, but of course, having less of an ozone layer is not good for life on the planet because ozone absorbs ultraviolet light that is very, very important for protecting us from sunburn and cataracts and protecting animals and plants and all kinds of things.

Susan Solomon: So depletion of the ozone layer is actually a threat to both humanity and the planet. I do want to say it wasn’t until somewhat later, 1985, when scientists from the British Antarctic Survey actually discovered the ozone hole and what they were doing was measuring total ozone in the Antarctic, they’d been doing it since the 1950s, and they were able to show that sometime around the late seventies, it started dropping like a rock way, more ozone depletion than Molina and Rowland had ever imagined.

So it turns out that these chlorofluoro chemicals are actually even more damaging to the ozone layer in the Antarctic, and actually also to some extent at mid-latitudes as we now know, than we originally thought. So it’s been sort of a cautionary tale of don’t be too sanguine about any chemical made in the lab. When you make something in the lab that nature doesn’t make, I think you should always do a double-take, especially if it has a very long atmospheric lifetime and builds up in the way that I described. So that means you can’t get rid of it if you stop using it. You can’t eventually, but it’s going to take a long time for the atmosphere to cleanse itself.

Lucas Perry: Could you explain why James Lovelock was initially interested in CFCs and what his investigation led to scientifically?

Susan Solomon: As far as I know, he really just wanted to see what their distribution was like to get some sort of a handle on what they might be… What his instrument might be useful for. I don’t actually know of a use beyond that, do you, Steven?

Stephen Andersen: No, I think that’s right. He was looking for gases that were indicators, as you suggest, and of course he had a device that would measure other chemicals, but I think he was immediately struck by the fact that he was seeing the same chemical at all locations that he sampled, and then he made the natural connection of saying “Where did this chemical originate? How long did it take to mix in the lower atmosphere”? So I think it was a good, solid, scientific inquiry of a very intelligent person with a new instrument.

Susan Solomon: Maybe we should clarify, I said he went all the way down almost to the Antarctic, but I neglected to underscore that of course there is no chloro fluoro carbon emission in the Antarctic, right? At that time there was nobody even… There were no stations, there was nobody there. So any chlorofluorocarbon that could get there had to of have gotten there by atmospheric transport and it would also tell you that it has to have a fairly long lifetime because if you emit, let’s just say sulfur dioxide from a power plant in the Ohio Valley, yeah it’s a serious issue, it can cause acid rain, it can cause little particles that are bad for your lungs, it does a lot of bad things, but it’s not going to be found in the Antarctic. It just doesn’t have that long of a lifetime, it rains out. So this proved that they were a great tracer in his mind I think that’s what he was attracted by.

Lucas Perry: We’re in this world where CFCs are basically being used for a ton of different applications. Our current understanding at that time was that they were nonreactive and non-toxic, so basically a wonder chemical that could be used for all sorts of things and was much safer than the kinds of chemicals that we were using at the same time. And here comes James Lovelock, who, from my reading, it seemed like he first got interested in it because his view from his house was hazy, and he didn’t know why it was hazy and he wanted to figure out if it were manmade chemicals or what this pollution that was obscuring his vision was.

And so he starts measuring all these CFCs everywhere, and now we’re in a world where it seems very clear that the CFCs are abundant in the lower atmosphere. So let’s start to pivot into Mario Molina and Frank Rowland’s role in this story and how we begin to move from noticing that there are so many CFCs in the atmosphere to finding out that there might be a problem.

Susan Solomon: I will say that Dr. Rowland has passed away, unfortunately, so has Molina more recently, but he never went by Frank, he went by Sherry. His name was indeed F Sherwood Rowland, but he was known to everyone as Sherry Rowland.

Lucas Perry: Okay.

Susan Solomon: Go ahead, Steve, do you want to take this one?

Stephen Andersen: Yeah, sure. The story of it is actually another great science story. Mario Molina had finished his doctors degree at University of California, Berkeley, and had taken a post-doctorate study with Sherry Rowland at University of California Irvine, and they looked at four or five interesting topics, and I think that the history is that Molina saw this one as being particularly intriguing, even though it was slightly outside either of their expertise.

So it was a stretch for them, but it gave them a chance to look at something that could be potentially very important. And then the story is that as they began to investigate, it started to seem more and more obvious to them. And it became a rush for a conclusion because they were worried about the effect of their work. There’s one story that Sherry Rowland tells us, that he came home from work one day and his wife, Joan, asked “How did your day go? How is your work”?

And he replied something like “Well, the work is fantastic, but I think the earth is ending”. So you can imagine the tension, the creative tension, and then they published their article I think in April of 1974, and there was no uptake by the press, there was no scientific confirmation. It was a quiet time until that fall, the Natural Resources Defense Council, NRDC, saw this, the scientists there, and recognize this was a big public policy issue. So at the American Chemical Society fall meeting, they had a presentation by Molina and Rowland, and then by the best of good luck of ruthless corporate behavior, the industry attacked them and made this scientific study news worthy.

So this was a tremendous good fortune, oddly enough, because then all of the press was asking “What are you talking about? Why is it important and what could happen?” And that cultivated in Molina and Rowland, the ambition and… Sherry Rowland called for a ban on aerosol cosmetic products; hairspray and deodorants. And so this was stepping out of their role of a normal scientist and becoming an activist, and of course there was a boycott that was quite spectacularly successful in the United States, and then some product bans.

Susan Solomon: Yeah and actually I just want to say, I think we should be proud in the US that there was a consumer boycott. People turned away from spray cans and that actually, interestingly enough, did not happen in Europe, they kept using them. So we can look back on that time as one in which people were environmentally very aware in this country, not just on the issue of the ozone layer, but also for things like smog and clean water, all those issues had attracted a lot of attention right around this time. I will also say that it’s interesting that at the time of the Molina & Rowland work, they were talking about the fact that from the best of our understanding of the day, in a hundred years we might see a few percent decrease in the total amount of ozone.

So kind of a small effect, far in the future, kind of like the way some people used to talk about climate change until maybe this year or the last few. And the Antarctic ozone hole was a huge wake up call because what they found was that ozone had dropped by more than 30% over Antarctica already by 1985, something that no one had anticipated. So it was a huge shock to the science community. And at first a fair number of people didn’t really take it seriously. I can remember being in scientific meetings with people who said “Oh, that British group, they must just be wrong”. I won’t say who they were, but it of course turned out that they weren’t wrong. They were confirmed quickly by other stations in the Antarctic and also by satellite data. And we now understand the chemistry that actually made the chlorofluorocarbons even more damaging than we thought they would be much, much better than we did before.

Lucas Perry: Could you explain a little bit of the scientific mystery and the scientific inquiry that Molina and Rowland were engaged in, the kinds of hypotheses that they had and the steps from going from okay, there are lots of CFCs in the lower atmosphere to eventually understanding the chemical pathways in their role in ozone depletion.

Susan Solomon: The big issue that they had to deal with was how do these compounds get destroyed, and what is their atmospheric lifetime? And they actually went into the laboratory themselves to try to make measurements relating to that. So they were able to show, I think through the measurements that they made, that the CFCs didn’t react with… They didn’t rain out, that they weren’t water-soluble, so that was not an issue. That they didn’t react with sand, there was some idea that somebody had suggested that they would be destroyed on the sands of the Sahara and that turned out, of course, not to be true. And then they looked at the way in which they would break down, what would happen to them them? If they have no way to break down in the lower atmosphere, the only place for them to go is up, up, up, up, and as you go up, you reach much more energetic sunlight, the higher you go.

Obviously if you’re in the limits of space, you’re getting the direct light from the full spectrum of what the sun can put out. But if you’re down on the ground, you got a lot of atmospheric attenuation. So they began to realize that once these molecules got into the stratosphere, that they would eventually break down, make chlorine atoms, and it was known already that those chlorine atoms could go on to react with ozone. And then there’s a another process, which I’m not going to go into, that actually leads it to a catalytic cycle that destroys ozone pretty effectively, and that process was already known from other work.

Lucas Perry: Could you actually say a word or two about that? Because I actually think it’s interesting how a single chlorine atom can destroy so much ozone.

Susan Solomon: Sure. The chlorine atom reacts with ozone, that makes chlorine monoxide plus O2. Now, if that was all there was, it would be a one-way process, you could never deplete more ozone than the chlorine that you put in. But what happens is that the chlorine monoxide can react with atomic oxygen, for example, so there’s… If you go up into the well… In the lower atmosphere, most of the oxygen as we know is in the form of O2, right? So it’s the oxygen that we breathe is O2. That’s actually true as far as total oxygen, pretty much all the way up, but as you get up into the stratosphere, oxygen actually also encounters that high energy ultraviolet light, which breaks it down and makes atomic oxygen, and ozone can also be broken down by high energy light, and that makes it atomic oxygen, ozone is O3.

So basically what first happens is the O2 breaks down with ultraviolet light making atomic oxygen, the atomic oxygen reacts with another O2 to make ozone, but then the ozone, let’s say photolyzes to make O, so now if the O comes along and reacts with the ClO, making chlorine atoms again, plus O2, you’ve liberated the chlorine atom, it can go right back around and do it over, and over, and over, and over.

And the reason that’s actually happening in the stratosphere, it’s in the sunlit atmosphere, ozone and atomic oxygen are exchanging with each other really quickly, so there’s always some of both present anytime the sun is lit. At night the O goes away, but during the day, there’s ozone… Breaks up as sunlight enough to make some O. So you can just drive this catalytic cycle over and over again and you can destroy hundreds of thousands of ozone molecules with one chlorine molecule, chlorine atom, from a CFC molecule, in the timescale that this stuff is in the stratosphere.

Lucas Perry: Right, and so I think the little bit of information about that is that the chloro in the chlorofluorocarbon meaning… That means chlorine, right? So there’s these chlorine atoms that are getting chopped off of them, and then once they’re free in the atmosphere they can be used to basically slice many ozone molecules, and the ozone molecules are heavier and more dense and so reflect more UV light?

Susan Solomon: No, no, no, no, no. Density and heaviness has nothing to do with it. The ozone molecule is just capable of absorbing certain wavelengths of ultraviolet light that no other molecule can in our atmosphere to any appreciable extent. That’s why it’s so important to life on the planet surface. It’s just a really good light absorber at certain wavelengths.

Lucas Perry: Okay. And so at this point for Mario Molina and sorry, it’s Sherry Rowland?

Susan Solomon: Yes.

Lucas Perry: And so for both of them, this is still theoretical or model based work, right? There hasn’t been any empirical verification of this.

Susan Solomon: That’s right.

Lucas Perry: So could you explain then how we move from these lab based models to actual empirical evidence of these reactions happening, and I guess starting with where Robert Watson fits in?

Susan Solomon: Bob Watson was a chemical kineticist originally, and actually had measured some of the reactions that we’ve just been talking about in the laboratory. So what people do is they go in the lab, I should have said this earlier, they go in the lab and they evaluate how fast individual chemical processes can happen, it’s really very elegant work, and flow tubes, and lasers, and all that kind of stuff. And it’s something that Watson was known for but he got an opportunity to become a leader of a research program at NASA, which he took up, and he became very much a leader in the community, as far as both organizing missions, field missions to go out there and look at things in real time, and more importantly perhaps, a huge leader in the assessment process which brought the scientists and the policy makers together. I think that you can really look at Bob as a tremendous founder of the whole way that we do scientific assessment, together with a gentleman named Bert Bolin, who has passed away unfortunately.

Lucas Perry: After Watson, could you then bring in how Joe Farman fits in?

Susan Solomon: Yeah, Joe Farman led the British group that discovered the ozone hole, as I mentioned earlier. So they noticed that their ozone over their station at Halley, Antarctica, it just seemed to be decreasing at an alarming rate. And they checked it with another station that they have, which is at a slightly higher latitude, not quite as far to the pole… I should have said lower latitude. Anyway, 65 South is where the other station is, I think Halley is about 73 South, it might be 78. And they found that there was ozone being lost at the other station too, just not as much. And that’s when they decided that they just had to publish, so they did, and it attracted my attention along with the attention of a lot of people. I started working on what chemistry could possibly cause this, and what I came up with was that “Hey, maybe…” And we knew that there was no ozone hole over the Arctic.

We knew it was only over the Antarctic, because we had measurements from places like Sweden and Norway and Canada, if anything was happening, it was nothing like the Antarctic. So measurements in other places showed ups and downs, variability from year to year, but they weren’t showing any kind of trend at that point. They later did, and we can talk about that, but we’re talking about 1985 here, so really early. I was a young scientist, I was 29 at the time, and I decided that I was going to try to take my photochemical model and beat on it and pound on it and make it stand on its head until it produced an ozone hole.

And so I did that, and I figured out that the reason that it was happening was because Antarctica really is the coldest place on earth, and it’s so cold that clouds form in the Antarctic stratosphere. The stratosphere is very dry, so normally there just aren’t any clouds, but down in the Antarctic because it’s so cold, the vapors, mainly water vapor but also actually nitric acid and other things can condense and form these incredible polar stratospheric clouds. And the clouds completely changed the chemistry, we can talk about that, but I think I’ve maybe gone on too long for my enthusiasm for which I apologize.

Lucas Perry: Hey this is post podcast Lucas. I’d like to add some more details here around the story of Joseph Farman’s discovery of the ozone hole, to paint a bit of a better picture here. I’m taking this from the UC Berkeley website, and you can find a link to the article in the description: Dr. Farman started collecting atmospheric data at Halley Bay, Antarctica in 1957, sending a team to measure ozone levels and concentrations of gases, like CFCs. In 1982 his ozone reading showed a dramatic dip of around 40%. He was initially skeptical that this was an accurate reading and thought it must have been an instrument malfunction due to the severe arctic cold. He also reasoned that NASA had been collecting atmospheric data from all over the world, and hadn’t reported any anomalies. His instrument was ground-based and only had a single data point, which was the atmosphere directly above it. Surely, he reasoned, NASA’s thousands of data points would have revealed such a drop in ozone if there had been one. Given this reasoning, he ordered a new instrument for next year’s measurements.

The following year, Dr. Farman still found a drastic decline, and going through his old data, discovered the decline actually started back in 1977. He now suspected that something odd was happening strictly over Halley Bay, leaving other areas unaffected. So the next year, his team took measurements from a different location 1,000 miles northwest of Halley Bay and also discovered a large decline in ozone there as well. With the same data at two different locations the mounting evidence for the ozone hole was clear and he decided to publish his data. This data both shocked and intrigued many scientists, including Susan Solomon, which thus catalyzed further research and inquiry into the ozone hole, the mechanism that was creating it, and the needed governance and industrial solutions to work towards mending it. Alright back to the episode.

Stephen Andersen: Let me just say one thing before we go back to the ozone hole. One of the interesting things that happened was of course, Farman and his research group declared that there was this serious depletion happening in Antarctica. So all the scientists that had been building the case with Sherry Roland and with Mario Molina, instantly jumped on it in the press, and in fact it was Sherry Roland that coined the phrase, ozone hole. He was the first person to utter that phrase.

And that was also a very good case that the public could grasp that, they could look at the NASA graphics, they could talk to scientists, and so there was really a great expectation that someday there would be the smoking gun like this and Antarctic ozone hole or other evidence, and people were ready and prepared to go to the press and go to the public, and in fact the politicians by this time had been briefed a lot, and that the United Nations they’d been working on this since 1970, when they organized a working group on stratospheric ozone depletion. So this great scientists and great science was welcomed into the community and they took full advantage of this and then other great scientists like Susan jumped on it to say, “Well, how can we go beyond simple finding of the ozone depletion and track it back to its origin, the CFCs and the other ozone-depleting substances.” So it was science and politics at its best.

Susan Solomon: Yeah, I guess I also want to say that I didn’t assume that the ozone hole was necessarily due to chlorofluorocarbons. I tried to produce it all kinds of ways with reactive nitrogen from the Aurora with dynamical changes. I just couldn’t get it to happen any other way. And what we knew already, and again I think it’s a real achievement, was that people had been interested in the idea of reactions on surfaces for a while, but mainly because they thought they were perhaps interfering with the measurements they were trying to make in those flow tubes. The flow tube is basically just a glass tube and people assumed that there was no surface chemistry that could happen in the stratosphere. We know there is chemistry on surfaces, in the lower atmosphere, in the troposphere, and it can be really important. Acid rain is a great example.

Surfaces can make chemistry do things that just doesn’t happen in the gas phase. That’s why you have a catalytic converter in your car, it’s a surface that converts the pollutants into something else before it gets out the tailpipe. Surfaces lead to chemistry happening very differently from gas phase. And we assumed the stratosphere was just gas phase, there couldn’t possibly be any surfaces. But interestingly, we sort of knew that there were these polar stratospheric clouds they’d been observed by explorers going back, I think 200 years in the Arctic and 120 or so in the Antarctic. We knew they were there, we just didn’t really carefully evaluate their chemistry. But when people started doing these experiments in the laboratory, they thought certain processes were actually going in the gas phase. They saw, for example, certain kinds of chlorine molecules going away in their flow tubes, and they thought they’d discovered some new gas phase chemistry, turned out to be something happening on the surface.

And they said, “Oh, okay, doesn’t matter. It’s just on the surface.” Well, it turned out to be not just on the surface of the float tube, but also on the surface of those polar stratospheric clouds. And that’s actually the connection that I made. I thought, “Hey, if this is happening in the lab, there’s no, necessarily, reason that it couldn’t also happen on polar stratospheric clouds.” Now that was a leap that perhaps I shouldn’t have taken, but anyway, I did.

Lucas Perry: It’s good that you did. Yeah, could you explain what that moment was like, more so. I mean, that was basically a key, super important scientific discovery.

Susan Solomon: Yeah. I had a very hard time believing it when I… This was back in the days when I was running a computer model. This was in the days that you would wait a long time for your output because things were very, very slow. I don’t remember. I don’t think it was still in the computer punch card day. I think I actually did have a file that I submitted, but the wait for getting it back, I think, felt interminable. And when I did get the results back, I was just shocked to see how ozone behaved. And one of the key things about it is that it doesn’t happen in the winter. In the dead of winter when the polar regions are dark, this process won’t be very important. You have to have not only cold temperatures so the polar stratospheric clouds are there, you also need sunlight to drive certain parts of the chemistry.

And I could go into the details of that, but I’m not sure you need me to do that. It’s a process then that occurs in the Antarctic spring, as it comes out of its long period of dark cold winter, it’s still cold, but the sun starts coming back. And that’s the combination that then drives the ozone depletion. And that began to start happening in my model. So I was pretty shocked. It wasn’t quite for the right reason, I have to admit. The process that I had driving that final step of… What I did identify correctly was that the key reaction is the hydrochloric acid and chlorine nitrate from the chlorofluorocarbons react together on the surface of the polar stratospheric clouds, they do not react in the gas phase.

We thought maybe they did at one time, but then we figured out it was just on the surface of the float tube, so we forgot about it, everybody, except until I remembered. And then the hydrochloric acid and chlorine nitrate react on the surface of those clouds that makes molecular chlorine CO2, which fertilizes breaks apart very readily with sunlight, that makes chlorine atoms, and now you’re off and running to produce ozone depletion. So, that part I had all correct. What I thought was that the chlorine monoxide might react with HO2 to close the catalytic cycle. Cause you don’t have much atomic oxygen in the lower stratosphere where the ozone hole was happening. You need to close that catalytic cycle, we were talking about earlier, with something else. Turned out that really the key thing is ClO reacting with itself to make something called a ClO dimer, which then fertilizes. But we didn’t actually know that chemistry yet. We learned about it not too long after. That was discovered in ’87.

Lucas Perry: I see. So, essentially there are these glass tubes and labs where the scientists at the time were trying to basically create atmospheric conditions in order to experimentally test the models that they were generating for what happens to ozone up in the atmosphere. And because it’s a glass tube, there’s a lot of surface on it and so they were discounting what they were observing in that glass tube, because they’re saying the upper atmosphere doesn’t have any surfaces. So, any surface related reactions don’t really make any sense.

Susan Solomon: Right. That’s basically it.

Lucas Perry: So, you were looking at that, what made you think that maybe there were surfaces in the sky?

Susan Solomon: Well I mean, we knew that polar stratospheric clouds could happen in the Antarctic and also in the Arctic. Like I said, people have visually… You can see them. I’ve seen them myself. They’re clouds. They’re actually very beautiful. There they look like they’re almost a rainbowy kind of appearance because the particles are almost all one size and that creates a particular kind of beauty when the sun hits it. But yeah, you can see them, they’d been seen, literally. There were also satellite data that had been published a couple of years earlier that helped to inspire me to think about it. But I actually knew about the explorers, I was just intrigued by that kind of stuff. So then I was very excited to work with Bob Watson when we formulated a mission to go to the Antarctic and to actually go down there and make measurements that might help to determine whether reactions like that were indeed happening. And that happened in 1986.

Lucas Perry: Right? So you’re creating these models that include the surface reaction. And so you’ve got this 1980s computer that you’re submitting this file to, and… What do you get back from that model? And how does that motivate your expedition to go there and get measurements?

Susan Solomon: If I remember I had it programmed to make plots of the percent change in ozone, and there was this, I didn’t call it a hole at the time, but there was this area over the Antarctic where once I put those reactions in, I got a lot less ozone. I recall something like 30% less. It’s published in my paper that I published on this in 1986. So I wrote it up and submitted it to nature, and it was published in ’86, and that was the same year that a lot of us began thinking about how to get down there and test the different ideas that have been put out because the idea of chemistry involving chlorofluorocarbons was not the only idea out there, other people had meteorological theories. And as I mentioned, there was this possibility that it might be solar activity, I guess somebody thought about, so the…

Scientists are always stimulated to come up with ideas, and we needed to get down there and make the measurements that could discriminate between the different ideas. So, I was very fortunate to be young and able to get on an airplane and go to the Antarctic. So I did, 1986. It was great. Most incredible scientific experience in my life, actually.

Lucas Perry: What made it so incredible, and what is it that you saw? What was your favorite parts about your expedition to the Antarctic?

Susan Solomon: Well, just going to the Antarctic is an unbelievable experience. I mean, even if you just go on a cruise ship, it’s like another planet. It is crystalline, beautiful, unpolluted, full of optical effects that are just amazing. And of course, brutally, brutally cold. We went down in August of 1986. When I got off the plane the temperature was about -40°C, which is also -40°F. So I like to joke that if you’re ever on Jeopardy and the Final Jeopardy question is, “At what temperature are Celsius and Fahrenheit the same?” The answer is -40. I’m originally from Chicago, I’ve been in cold weather, but I’ve never been in anything colder than I think about -15 before. And it was, it’s a shocker.

But after a while, after a couple of weeks, -15 actually feels very warm. You really do. It’s amazing how you acclimatize. Everybody laughs. Stephen’s laughing as I’m saying this, but it’s true, it’s true really. I thought I would just kind of curl up in my room the whole time, but I didn’t, I found that it was easy to acclimatize. Yeah, really. Well, actually people do. People actually even go jogging with shorts on, at -15. Yeah, it’s incredible. Depends on if it’s sunny or not. The atmosphere is very dry also down in the Antarctic. Basically, the cold has rung all the paper out of the air. So-

Lucas Perry: Did you go jogging in your shorts and T-shirt at -15?

Susan Solomon: No, no, I didn’t do that. But I’d certainly remember feeling warm at -15 and opening up my jacket and stuff like that. And I definitely kept my window open if it was -15. So yeah, I did. But, I made some measurements with my colleagues using visible spectroscopy. So we use the sun, or the moon, or the sky as a light source, and we measured chlorine dioxide, which is a closely related molecule to chlorine monoxide, and we were able to show, particularly with the moonlight measurements, that the values we found were a hundred times more than they should have been. We couldn’t measure them anywhere else because they were below our detection limit, but they were actually quite measurable in the Antarctic. So, that was the key measurement that we made and it was an incredible night, the night that we actually did that. And then, I think it was the next day that I made the data analysis and there it was. It was an amazing, amazing moment.

Lucas Perry: Could you explain more about how that particular data that you measured fit into the analysis and what it proved, and the feelings and experience of what it was like to finally have an answer?

Susan Solomon: First of all, there’s the getting of the data which involves putting mirrors up on the roof of a little building in the Antarctic and directing the moonlight right down into the instrument. And doing that when it’s cold and windy can be a bit of a challenge. So setting it up for measurement is physically challenging. And then taking the data, analyzing the data. I was, I think, careful enough to realize that that wasn’t going to be the only thing it would take to convince everyone that chlorine was the cause of the ozone hole. The chlorine dioxide that we measured, had to have come from the chlorofluorocarbons. There was no other even conceivable source for it. And it was a hundred times more of it than there should have been, and that’s because it had gotten the reactive species, and chlorine dioxide is a reactive form of chlorine had gotten liberated from un-reactive forms of chlorine, like hydrochloric acid and chlorine nitrate, which reacted on the surfaces of those clouds. And they don’t do that anywhere else.

So that’s a little more detailed than I thought you might’ve wanted, but that’s why you take these, what we call reservoir species for chlorine, hydrochloric acid, and chlorine nitrate, and you convert them to active chlorine. And now you’re really often running for ozone depletion. And that’s what happens on those clouds.

Lucas Perry: You’re getting this data about these particular chemical molecules, and then you’re… Tell me a little bit more about the actual analysis and what it’s like being in the Antarctic feeling like you’ve discovered why there is a potentially world ending hole.

Susan Solomon: Well, I’ll tell you this, I was really careful, I think, maybe Stephen can correct me if he thinks some wrong, but I was pretty careful about not broadcasting the news before we were really, really sure. So the moon measurements alone were not enough to convince me. And one of the things that actually excited me a lot was when we realized we could also see this in the scattered sunlight that we got. If the sun was low enough on the horizon, there was even enough chlorine dioxide to measure it then. So what it is is a visible spectrograph, it’s got a diode array in it, it’s actually very similar to the diode array that reads the prices when you go to the supermarket today. Back in the ’80s those were incredibly expensive because they had just been invented. And we had one that was cooled to very cold temperatures to keep it from having too much noise in it.

And we had a spectrograph, which you can think of as being sort of like a prism that you shine sunlight through, and you separate out the wavelengths of light and the colors of light come out as a little rainbow that you see when you put a crystal in front of a source of light. And so that’s essentially what we’re doing. We’re putting a grading, in this case, a diffraction grading in the beam of the moonlight, and we’re collecting the separated wavelengths of light on our detector, and we’re looking for the absorption of atmospheric chemicals. And we can measure ozone that way, we can measure nitrogen dioxide at a different wavelength, but chlorine dioxide has a particular band structure in the visible, that is what we measured. And we can also see it. The fact that we could also see it in the skylight and that the difference between the skylight and the moonlight was consistent with what we expected from chemistry and consistent with what you would need to deplete the ozone layer, got me pretty excited.

There was another group on our expedition that measured chlorine monoxide using a different method on microwave emission technique, which is the same one that’s used nowadays by satellite but in those days it was only used from the ground, and they also measured high levels of chlorine monoxide. And last but not least, I’ll say that the following year in 1987, Watson organized another mission, which actually flew on airplanes from Chile down to Antarctica, and measured chlorine monoxide yet another way by laser resonance fluorescence onboard an airplane that actually literally flew right into the ozone hole. So, I would say fair to say that from the science community point of view, when all those measurements were in, people got pretty convinced, but they also had to be written up. I mean, it had to be peer-reviewed before something that important could really be talked about as a known piece of science. So I was very cautious about spreading the word too early.

Stephen Andersen: So if you look at the history of the Montreal Protocol, what you see is that in 1985, there was something called the Vienna Convention that was passed by the United Nations that had about two dozen members signed it. And this is what’s called a Framework Convention that makes it possible to have something like the Montreal Protocol. So that was in the spring of 1985. And shortly thereafter, Berman published his article which was a tremendous reinforcement to the policy members that had anticipated that soon there would be evidence and that they needed to be prepared. And then as we went into 1986, and Susan is doing with her college, this brilliant work in Antarctica, the preparations are underway for the Montreal Protocol. And the scientists were telling the Montreal Protocol that it could be another explanation for the ozone hole, so that you should hold your fire until you’re sure of the results.

And you can find that in lots of the accounts at the time. But by the time of the medium, September of 1987, there was still a lot of uncertainty. But the policymakers were able to talk to their national scientists and others and felt confident enough to go ahead and confirm the Montreal Protocol. So I would view it from my point of view and perspective, that it was a continuous improvement in the science and the threshold of belief occurred for different people at different times. But you could then say, of course there were still skeptics in 1987, but it was my experience that they were mostly gone by 1990. And so the work I was doing mostly with corporations, there was rarely a meeting where there would be science skeptics after 1990, that they were gone from the earth has moved on to climate, in fact.

So it was a tremendous contribution of science. And the other thing that’s important to realize is that the industry that used these chemicals was not devoted to them. They’d been reassured by DuPont and others that they were completely safe and that there was no reason to worry. And then, as soon as the Antarctic ozone hole came along, they panicked and they rushed to the market to find alternatives. And that’s one of the reason the Montreal Protocol happened so fast, is the industry in some ways was faster to grasp the science than even the policymakers.

Lucas Perry: Stephen, I’d love to bring you in here to describe, in particular, the transition from the discovery Susan was involved in to the Montreal Protocol. So, what are the steps between the discovery and the Montreal Protocol, and how do we eventually get to the Technology and Economic Assessment Panel that you co-founded?

Stephen Andersen: I’m glad to describe that. It’s exactly what Susan said, is that there’s a laborious process to prove the science, and then there’s another process to communicate it. And that’s probably partly what Susan did, and Bob Watson and another scientist, Dan Albritton. And they were masters of communication. And there were lots of meetings held between the scientists and the diplomats. But also the scientists and the companies, including the National Academy of Engineering, did its own review of the science on behalf of industry and came up with a confirmation report, I would call it no. New science, a narrow view of science, but nonetheless, it was the message coming from the people they most respected, and Sherry Rowland was involved in that and many, many others. So the communication was very quick. And in fact, I would say by January of 1988, you could see big changes.

So the protocols signed in September ’87. In January of ’88, there was a large conference, and at that conference, there were several important announcements. The most spectacular was that AT&T announced that they had found a nature based solvent made from the terpenes from oranges and lemons, pine trees that could clean half of the electronics equally or better than the CFC-113 had replaced. And they said that transition was technically possible within one year. So they went from skepticism and standing back to becoming the driving force. And it was also important because this terpene was not another synthetic chemical. It was naturally derived and harvested from the disposal of the orange rinds and the lemon rinds, and then put to positive purpose. So this was an eye-opener to a lot of people that thought you had to have an elaborate chemical to have an elegant solution.

And then at that same meeting, the auto industry step forward and realize that most of the emissions from car air conditioning was from servicing and from leakage. And so for the first time they got together a partnership that developed commercial recycling for air conditioning. They did that within one year and the next year after they confirmed and approved the technology, they sold a billion dollars worth of recycling equipment all across the world. So there’s this enthusiasm of going from panic, that there would be high costs and disruption to the enthusiasm of profits and saving money. So it was the science that drove this, but it was the technical innovation that did this. And then very shortly thereafter, and in an overlapping way, there were similar breakthroughs in foam, so it was a commitment by the American food packaging institute to halt use of CFC in foam within one year, and to switch away from all fluorinated gases as quickly as possible.

So we’ve seen this building momentum and enthusiasm. You have international companies that are pledging to get out, and all the while we haven’t reached the Montreal Protocol entering into force, because that occurs later after the signing. And when it was signed and the first big assessment which Susan was involved in as well, was done in 1989. And so this was an assessment that you alluded to, it included the Scientific Assessment Panel, it included the Environmental Effects Assessment Panel, and then it included the work on technology and economics. And this was the idea of how do you make the best available information readily absorbed by policy makers and business community.

Susan Solomon: But Steve, I would just add one thing that you already said, and that is, it all starts with people understanding the problem. You talked about the fact that people everywhere the public all around the world could look at these satellite images of the ozone hole and say, “Hey, that’s actually pretty scary stuff.” And that created the will, the political will, that generated the demand for all the products that you’ve just described. I think without people understanding the whole thing, nothing happens, personally.

Stephen Andersen: You’re absolutely right. And in fact, in the case of that food packaging, it was a school teacher in Massachusetts and her children in the class that wrote to McDonald’s corporation and said, “Why are you destroying the ozone layer?” So the people at McDonald’s commissioned a survey of their customers, including children, and the customers responded, they did not want to destroy the ozone layer. And it made a big difference to where they chose to eat. And in the case of McDonald’s, children drive parents to the restaurant, the parents say, “Where do you want to go today?” And they say, “McDonald’s.” So it was a huge impact. It was an eyes-open business decision. And they had announced, prior to the packaging institute changing, that they were going to stop the purchase.

Susan Solomon: Yeah. They were putting hamburgers in foam clam shells, and they switched over to cardboard, which is fine because McDonald’s is so delicious. You eat it so fast anyway, you don’t need the foam. McDonald’s is so delicious. You eat it so fast anyway, you don’t need the foam.

Stephen Andersen: That’s right. Hot side hot, cold side cold, was the slogan. But this is exactly right. What Susan’s saying is, you have this circular effect, where you have the customers pushing the companies, you have the companies pushing their suppliers, and you have the policy makers setting deadlines. And pretty soon, you’ve got this wheel turning very fast. And as quickly as you catch up with the available technology, then you look to the next strengthening of the Montreal Protocol. And that’s what we saw over the decades of the Montreal Protocol, more and more chemicals control, faster and faster phase out.

Lucas Perry: Steven, could you explain more specifically what the Montreal Protocol was, and who the key stakeholders were, and how it came together and then was signed and ratified, and what that meant?

Stephen Andersen: So my role, I was very fortunate, because I was hired by the EPA in 1986 in preparation for the negotiations of the Montreal Protocol. I’m an economist by training. And so, I had the highest interest in showing that this was going to be cost-effective and feasible, and that the technology would come together. The mastermind behind the Montreal Protocol was the head of the United Nations Environment Program, Dr. Mostafa Tolba, who was a botanist himself and an accomplished author. And I think was very quick at grasping the science.

So you have the force of the United Nations organizing the meetings. And then you have the science that’s providing the justification for the treaty. And then you have leadership countries that were advocates of a treaty. And that included a group that was called the Toronto Group, because it was partly stationed in Toronto, but that was United States, Canada, Norway, Denmark, many other countries, Sweden, that got together as a group and helped craft the language that they could sell to other countries.

And so, it was a masterfully designed document in retrospect. And included in that document was the idea of start and strengthen. So if you look at it, it only was two chemicals, CFC, and then a fire extinguishing agent called halon. And then the first negotiation in 1987, it was just to freeze the production of halon, stop it from growing, and cut back CFC’s 50%. but that was not hard to do because 30% was still aerosol and convenience cosmetic products. So it was a very conservative start. But the science was so persuasive in the years ahead, that they said to the policy makers at the Montreal Protocol, that’s not enough. You will not protect the ozone layer with those two chemicals. And you certainly won’t with those modest reductions. So then they added more CFCs. They added carbon tetrachloride, metal chloroform, methyl bromide. A litany of chemicals were added. And then each time that they would meet, every two or three years, they would have an acceleration of the phase out.

So it was a very practical approach that was done on an international basis. And one of the beauties of this treaty, is it includes incredibly strong trade restrictions so that if a country did not join the Montreal Protocol, they would lose access to these ozone-depleting substances even before the phase out. So it had lots of clever features and lots of brilliant leadership. And what Susan said about people mattering, they mattered a lot over and over again. And there were 200 or 300 people that had a chance to become ozone champions and make a real difference to the world.

Lucas Perry: Could you explain who were the main stakeholders involved in the Montreal Protocol? Was this mostly developed nations leading it?

Stephen Andersen: That is a great question. That’s a fantastic question, and explains a lot of why it was such a challenge. So if you look at the full set of chemicals that are controlled by the Montreal Protocol, they were divided into 240 separate sectors. So, distinguishable industry groups that had their own interests in keeping these chemicals or to phase them out. So if you look at those, and some other ones that Susan mentioned, the air conditioning and refrigeration, and that includes industrial refrigeration, because many chemical processes require that, and commercial refrigeration, and also what’s called cold chain, the processing and the freezing and refrigerating of food in order for it to reach market.

So that alone would have been daunting. But in addition to that, there were these chemicals used as solvents in aerospace, in electronics, in the manufacture of medical devices. It was used as a sterilant. And as I mentioned, as an aerosol for metered dose inhalers. It was used in fire-fighting, including enclosed areas like on airplanes and submarines and ferries and ships and places that you can’t evacuate if there’s a fire, where you have to stay on board the burning vehicle. So it included all of the NASA satellites and the space labs and the rocket equipment. Manufacture of solid rocket motor for the space shuttle required methyl chloroform.

So you have these, and then the were laboratory uses. So it’s used as a tracer-gas, as Susan mentioned. But also, it was used to have a dense gas for a wind turbine. And it was used for pressure check testing of scientific instruments to make sure there were no leakages of gases in or out. So as it got going, also, it was discovered that every weapons system in every military organization depended on ozone-depleting substances. All the command centers were protected with halon. All the ships, tanks, submarines, protected by halon. All the electronics and aerospace manufactured in service with CFC and methyl chloroform, all the gyroscope manufacturing for the weapons guidance. Whole list, all the way down. All the optical raiders, all the AWACS. Everything that they could look at had some use.

And so it required these stakeholders to look fundamentally at the basis of what they were doing, and decide how do you shift from using this chemical to a performance standard that would allow industry to compete as how they could produce an alternative that would be a pure replacement. And so, one measure that I think your listeners will find interesting is that if you ask the public today, or even the effective industries, no one has stories of train wrecks or disappointment or failed systems, because it was so successful. Most consumers would have their entire house changed, and they would not notice this. The glues they used to assemble furniture were ozone-depleting substances, but people have not stopped buying furniture. And so, if you went back, it’s the smallest list of uses that found no substitutes. So it’s quite remarkable.

Lucas Perry: Steven, you were on this panel, I believe the chair of it, for many years. So I’m curious if you could explain, more so, what that experience was like, what is it that you necessarily did on the technology and economic assessment panel, and what the impact of that was for implementing the solution that was needed after Susan helped to discover the mechanism of the problem?

Stephen Andersen: Yeah. Thank you for asking me that, because that’s what I’m most proud of, of course. When I was appointed with Vic Buxton, from Canada, to set up the first technical panel, we were like-minded and we had a great idea, and that was, instead of casting out for experts from various sources and seeking wide participation and balance of interests, we didn’t do that at all. We recruited the experts from the organizations that were already committed to protect the ozone layer, because these would be people motivated to find a solution rather than intellectually interested in describing solutions, or even worse, be a stakeholder against a new alternative, and they would become internal critics.

So the notion was that, on a technical committee, you could not have a better set of people than the people whose success in their enterprise depends on finding alternatives, and realized that a team could find the alternatives faster than others. The other secret of our success is, we had something called self affecting technical solutions. And so, for example, one of the chairs of the Halon Committee, studying halon, was the chair of the National Fire Protection Association that set the standards for where halon is used. So as quickly as a use could be eliminated with an alternative, he would go back to his committee and decertify halon on that use.

We had members of the coast guard on the committee. And as quickly as there were alternatives on ships, they removed from the requirements of the United Nations Maritime Organization the use of halon. So it went from compelling the use for safety to prohibiting the use for environment. So it was this remarkable internal group. And if you go back also, and you’ll notice that some of the most important technologies were invented by people that only met on the committee for the first time. So you had groups of military suppliers that got together to tell the communication suppliers and invented something called no-clean soldering that eliminated the use of solvents and save the ozone layer, but it also increased the reliability of the products. And they were enthusiastic about commercialization to the extent that they patented the technology and then donated it to the public domain so it could be used anywhere in the world at no expense to the user.

So you have this enthusiastic group of genius engineers working on a short deadline and constantly resupplied with motivation from scientists like Susan, because as fast as they would take satisfaction in what they’d accomplish, they were being told, it’s not enough. It’s not enough to just do these chemicals. We have to do more. It’s not enough to do these chemicals on the old schedule, we have to go faster. So some of these sectors halted their uses years ahead of the deadlines of the Montreal Protocol or the Clean Air Act. It was really quite inspirational. And most of those people would tell you, it was the best part of their life because they never would have been allowed to work with the engineers from the competing corporations if it hadn’t been for the TEAP drawing those together for public purpose.

Lucas Perry: So, is a good way to characterize this then that there’s this huge set of compounds, that when they get up in the upper atmosphere they release chlorine? And the chlorine is really the big issue. And so, these hydro chlorofluorocarbon are being used in so many applications. You’ve described a lot of them. And so, the job of this committee is to, one, slowly, through regulation, phase out this long list of ozone depleting chemicals, while also generating alternatives to their use that are not ozone depleting.

Stephen Andersen: Yeah. That’s right. Generating or identifying. And there’s a subtle problem we faced. It’s now being faced again for climate. And that is, as Susan mentioned, most of these chemicals have long atmospheric lifetime. So when you stop producing the CFC, it can be a slow decline in the chlorine that’s been contributed to the stratosphere over many, many years. Others of the chemicals like methyl chloroform, and most of the HCFCs have short lives. And so, any reductions you make in these chemicals that do all their damage within their short number of years has a bigger effect immediately than doing the same amount of effort on one of these other chemicals. And what the scientists were telling the Technical Actions Committee and the Montreal Protocol was that we had to worry about the long run and the short run.

And so HCFCs as refrigerants and foam blowing agents were viewed as a transitional substance. So if you stopped using CFC11 and you started using HCFC22, that was an improvement in both the GWP, and the forcing of ozone depletion. And the same thing for methyl chloroform. So the ambition of the Montreal Protocol was to work incredibly quickly to get rid of the short term chemicals and uses with an alternative that would be solvents, for example, using methyl chloroform, but at the same time, allow some HCFs so that you didn’t have to endure the continued use of the CFCs. And that was the technical challenge, to keep your eye on the long run, and at the same time, keep your eye on the short run. And some of the scientists were also over motivating that kind of ambition, because there was a concern that we might go too far in sending chlorine and bromine to the stratosphere, and do irreparable damage, or damage that would take much longer to solve.

So, true or not. It was highly motivational. And it caused a tremendous effort on our committees, first of all, to get rid of methyl chloroform. If you look at the curve of methyl chloroform and the overall ozone protection, it was a critical first step. And it was accomplished probably in two and a half years worldwide.

Susan Solomon: So let me toot Stephen’s horn a little bit. And then also clarify one point. I think the invention of the Technology and Economic Assessment Panel, TEAP, as we call it, of the Montreal Protocol was a real master stroke because it brought the engineers and scientists from industry into the process to help figure out what could be done. And so, the way the assessment process worked is, on a systematic basis. The science group that I was part of would assess the science. Steve’s group would assess, okay, the science says, we got to phase these things out. What can we phase out? What is technically feasible?

And we would provide these reports, along with the one from the impacts panel, that would say if you keep doing this, you’re going to have so many skin cancer cases a year by 2050 and stuff like that. All three of those reports would be explained to a group of policymakers in a UN meeting. So the decisions weren’t actually made by Steve. But Steve’s group was highly, highly influential in educating the policymakers and guiding them, really, on what would make the most sense, what could be done the most cost effectively, the most quickly, et cetera, et cetera. And then they made the decisions.

But the great thing about it is it’s not a political group at all. In the old days, we would call them a bunch of guys with slide rules. And that included the people who came from industry. They weren’t the political leaders of those companies, they were the people in the trenches trying to actually figure out what to do instead. And that’s what made it work so well. I really have often wished that we had a similar way of doing the intergovernmental panel on climate change assessment process. We have a science panel that’s pretty similar, but we don’t really have quite the same technology panel. And many people have commented that the technology panel that Steve put together was just huge in making the Montreal Protocol work as well as it has. And the ozone layer is actually finally beginning to heal. So it’s a real testimony to their success.

Lucas Perry: Yeah. Steven, please, I invite you to toot your own horn a little bit more here because your contribution was extremely significant towards the elimination of the ozone hole. I have a fun fact here that is from a paper of yours. So in 2007, Steven, you released a paper with Velders’ team, published the importance of the Montreal Protocol in protecting climate. And the team quantified the benefits of the Montreal Protocol and found that it helped prevent 11 billion metric tons of CO2 equivalent emissions per year, and delayed the impacts of climate change by seven to 12 years. So please, what are some of your favorite achievements? And this is something you’re involved in for decades.

Stephen Andersen: Yeah. That’s a great story. And I’m of course, very glad to tell it. One of the things people know about me, that have worked with me for many years is, I worked by slogans a lot. So I try to reduce my ambition to something that’s like a chant or a short instruction that I can give myself to move ahead. And after years of working with Susan and many other scientists and Mario Molina and Sherri Rowland, and struggling with these issues year after year, after year, and waiting for the science to come on board, and there’d be a missing link, and there’d be something that was misinterpreted, and we’d have to go back to square one. I came up with the slogan, science too important to leave to chance. And what that meant was that it was my job to say, what kind of information are the policy members missing? Those are the people I hang out with all the time.

Because at the same time, I was the deputy director for Stratospheric Ozone Protection at the EPA. I was the liaison to the department of defense on climate and ozone layer protection. So I was in those meetings where people were trying to decide, is it worth investing another millions of dollars in this new technology, or should we do something else? So in working with Susan in 1995 on a joint report between the IPCC and the tape, I realized that the Montreal Protocol had done a lot for climate that wasn’t well appreciated over at the Montreal Protocol, that these facts were available. So I put together what we called a dream team, which included Guus Velders, who was the lead author, a brilliant scientist from the Netherlands. It included David Fahey, he was one of the colleagues of Susan at Noah, and John Daniel. And then it included Mack McFarland, who’s a scientist who was once at Noah, but worked the better part of his career at DuPont. And then myself, who had been on the TEAP for oh, so many years, and then EPA.

So the idea of the team was to say, just how big was the contribution of the Montreal Protocol to protecting the climate, and how do we communicate that to the Montreal Protocol so they would consider that as part of their obligation and part of their legacy? Because we were coming up, my concern, we were coming up to a very long interval of HCFC in years, that the Montreal Protocol had plateaued its ambition. And they had accomplished so much, they were resting on their laurels, and they had lost this impulse to get more stringent.

So this committee was put together, it quickly put together all the facts, incredibly complicated at the time, although people have done work like this since confirming it. And this dream team came up with the conclusion that the Montreal Protocol had already accomplished more than the Kyoto Protocol could have accomplished if every party, every state government in the world had joined Kyoto, and if all of them had met its obligations. So this was huge. It was shocking to us. It was shocking to the world. We brought it back the same year, 1997, excuse me, 2007, to the Montreal Protocol. And that year, they accelerated the HCFC phase down.

So it was it a tremendous victory. And it was exactly what I would hope would happen, is if we assembled science in a new way that was headline news, that the policy makers would get the message and do something important. And then two years later, the same team decided, well, why don’t we show the Montreal Protocol, how important it could be if the chemicals that replaced 15% of the ozone depleting chemicals, which are HFCs were phased down under the Montreal Protocol, ozone safe chemicals controlled by the Montreal Protocol. And that was accomplished in 2016. Took a decade. But we’re very proud of ourselves. And I think it’s a perfect example of the advantage of a group of people with a wide set of skills working together, including somebody like me who’s not an atmospheric scientist, working with atmospheric scientists and making more clear what the policy makers need to know.

Lucas Perry: I’d love to pivot in here now into extracting some of these lessons that you’re sharing, Steven, for how we might do better with modern climate issues, from greenhouse gases and also other global catastrophic risks. But I think you guys have done an excellent job of telling the story of the science and the discovery, and then also the strategic part, and the solution making of the story of the ozone hole. So as we get to the end of that, I’m curious if you have any thing else you’d like to wrap up on about that story. What is a key insight? When you look back at everything that you’ve contributed and been through, what is it that stands out to you?

Stephen Andersen: My theory of change, I think, is the same as Susan’s, that people matter the most. That the ability to bring the right people together, at the right place, with the right instructions, is bound to have an important conclusion. And one of the things that I always found was, if you have the best engineers, no matter what their attitude, if they have the goal that is coincident of the environmental protection, they’ll find a solution. So I think this accumulating of the science and the engineers and coordinating the activity, these assessment panels are just everything, that if you do that, you can’t help but be successful.

But the other thing that I’m realizing now is that, when you’re in a hurry, like we are with the climate, you have to take advantage of the existing institutions. So as quickly as we added HFCs to the Montreal Protocol, I would like to add other chemicals, N2O, nitrous oxide, which is an ozone-depleting greenhouse gas that was neglected by the Montreal Protocol. And then there’s other gases like methane that have nothing to do with the sectors that are involved in the Montreal Protocol, but the framework of the Montreal Protocol might be perfect for a methane treaty. So you might have the Montreal Protocol people help design a methane treaty. Or if the Montreal Protocol can find its way to create new capacity with new skill sets, you could have methane drawn into the Montreal Protocol because it’s genius is partly that you turn off the chemical at the source and force all the downstream changes to occur. And that’s different than something like EPA, where you often find one part of the problem, catalytic converters and gasoline, and you implement that, but you’re not focusing on the big picture, which is electrification of the cars.

And so, there’s this inherent advantage of the Montreal Protocol, top down, turn off the chemical, bring on the technical information, bring people together for solutions, reinforce and reward. So I could go on for the longest time. I’m very enthusiastic about the success of the Montreal Protocol. I absolutely believe the lessons could be taken up better than they have been. And I think if they were taken up, we’d be well on our way in many other environmental problems.

Susan Solomon: Yeah. I like to tell students in my classes, and I feel like I’ve learned this over the years with Montreal, and I’ve seen it in so many other problems as well. There’s three Ps that determine how well we do on any environmental problem. The first one is personal. Is the issue personal to me, to us? And in the case of the ozone issue, it was deeply personal because skin cancer. I mean cancer, it doesn’t get any more personal than cancer, right. But also all the other attendant things that it can do. The second P is perceptible. Is the issue perceptible to me? The best case is if I can see it with my own eyes, like smog, but seeing the satellite images that we talked about earlier, that was good enough to make it perceptible to a lot of people. And the third big P is, are there practical solutions? And that’s where Steven’s type of work has been so important. So when you think about climate change and you think about the three P’s, people haven’t really considered it personal until pretty recently because it seemed like a future problem, not a today problem. And we can talk all we want to about caring about our grandchildren, but really what we care about is us, right?

So, and we do care about our grandchildren. Of course, we do, but not as much as we care about us. That’s just a fact, I think. It’s natural and normal and we don’t need to be embarrassed about it. Particularly when you’re talking about a future problem and you can always hope that there’ll be other solutions in place by then. So is it personal? For a long time we thought it wasn’t. Is it perceptible? For a long time we didn’t feel like it was. Nowadays, I would say more and more people are recognizing it as personal and perceptible. The kinds of things that have happened in the world this year have just been amazing. And being wake up calls because so many places have flooded, so many places have had massive fire. These are all the sorts of issues that we knew were happening. So much erosion is going on because of rising sea levels.

People are just, and actually when people would say to me, well, it’s not really perceptible yet. My answer to that would be, yeah, I know. And it’s a problem but it’s going to fix itself with time. And I think we’re just about there. It has fixed itself. And then there’s that big third P is, are the solutions practical. And there’s been a lot of propaganda out there saying the solutions are not practical, but I think we’re reaching the point now where we recognize that they are. So I think that we’re really at a turning point on climate change.

Lucas Perry: I’d love to pivot here more into exploring lessons to extract for what we can do about climate change and other global catastrophic risks for the theory of change about what we might do about those. And I’m also mindful of the time here. So if we could hit on these a little bit faster, that would be great. One thing I don’t think that, or one thing I would like to hit on more clearly is what is the bad thing that would have happened if the work of you, Steven, and the work of you, Susan, and all of the others who were involved in the discovery and solution towards the ozone hole, what is the bad thing that would have happened if we had just continued to use CFCs?

Susan Solomon: There is a lot of work on that now, people call it the world avoided scenarios. So what world did we avoid? Well, by mid century, it would have been about a degree hotter than it’s actually going to get. So that’s a degree Celsius by the way. So instead of a degree from, a degree in AF from mainly CO2 and methane that were trying to avoid, we would have an extra degree on top of that from CFCs that we would have had to avoid. That’s a big deal. Something like 20 million skin cancer cases in the United States sticks in my mind by mid century, but I would have to check that number to be absolutely sure.

Stephen Andersen: Yeah, Susan’s absolutely right. The latters can cancer cataract, but one of the things you can look at is you can say two interesting things. You could say, what if Molina and Rowland had not had this hypothesis? And certainly you could say, well, someone would eventually. But if it had been five years later, or 10 years later, it would have been catastrophic because it did take time as Susan said. It does take time to make a hypothesis, to confirm it, to do the ground measurements, to do the aerial flights and so forth. So it was just in time or a little bit too late that it was really a tight schedule that was working when you include diplomacy and corporate changes and all of those facts. But you can also look and say, what if the Montreal Protocol and Molina had been delayed some period of time.

And what you can see is exactly what Susan said, that the CFCs would have grown in climate forcing, let alone ozone depletion to a level that would have been untenable for earth. That they could have been almost the same level as the CO2 climate forcing. So we were incredibly fortunate to have this early announcement. It was incredibly fortunate that the Antarctic ozone hole was noticed finally, and announced and it was such a spectacular persuasion. And then fortunate that the Montreal Protocol was able to take this and then in a derivative that the corporations were able to make their reductions. I also think it’s important to remember that this really was a training ground for a lot of people, that scientists had not worked together successfully on assessments this large and so continuously brilliant over so many years. If you look back at each of the assessment panels, I find almost no valid criticism of any of the findings at any of the points in time that this was a well done process, actually stunning.

So the World Avoided, if you read the Nobel Award for 1995 for Crutzen and Molina and Rowland, it says that life on earth would not have been possible as we know it. If you read Paul Newman’s report on World Avoided, you find out that it would have been untenable to go outside at most latitudes for very long, without sun protection, far beyond what people wear when they go out today. So it would have been a lot of joy taken away, a lot of misery brought on by these medical effects. And it would be a less successful world because these technologies that replaced the ozone-depleting substances are purely superior, better energy efficiency, less toxic, more durable, more easy to repair and reliable. It’s quite a success story.

Lucas Perry: So let’s explore these P’s as Susan has put it. So we have the issue of greenhouse gases warming the climate today. And a lot of what you were involved in Steven was the economics of making transitions. So both the innovation required to replace HFCs and then also the questions around that being economical. So this is the importance also of industry being involved in the process. So I’m curious if you could explain your experience with industry and how difficult or easy it was to get industry to make this transition and how that compares to the transition that industry and governments need to make in the modern day around climate change. And how much of that difference is a bit circumstantial around the technologies and innovations that need to be made.

Stephen Andersen: Yeah, that’s a great question. Susan, I think agree, and I agree that some of the stakeholders on the climate side were more persistently ruthless than we’ve experienced under the Montreal Protocol. The early days when Molina and Rowland came up with their announcement, they couldn’t thought to have been more ruthless. Character assassination, there were lists of people that were not to be hired. The region said the University of California prohibited Sherwood Rowland for applying to certain organizations for funding and on and on. But if you look at what’s been done by the coal companies and petroleum companies, I think that that was orders of magnitude worse over a longer period of time, including interjecting a lot of wrong science over and over again, which was a terrible distraction and also took a lot of energy away. So there’s no doubt about the differences of the two. But what I have found out working on ozone layer is even sectors that have a bad reputation for other topics can be leaders on a topic once motivated as Susan Solomon says, and come to regret what they’re doing.

I’ll give an example. The automotive industry was among the most rigorous in getting rid of their solvent. Their foam uses, foams and cars include what are called safety fonts. So the, under the dash of the car originally, it was underneath the surface of the fabric, it was all ozone-depleting substances. It was all ozone-depleting substances for the refrigerant, for the solvent to make the components for the electronics and so forth. But they looked at science, they got motivated, and they stopped using these chemicals as quickly as the other sectors. So they were one of the fast to go. So what my lesson is you shouldn’t judge a book by the cover. You should not hold it against an institution because they misbehaved in the past and you should give them a chance to make a new start on leadership right now.

The other thing I would add is the public right now is much more engaged in climate than they ever were in ozone layer protection. So if you go back, there were very few of the industry projects that had active involvement of non-government organizations, environmentalist, because it was being done so well by the companies themselves that would have been futile use of those talents. But you look today, there are thousands of organizations that are demanding changes in industry. They’re in the streets, they are protesting. This did not happen for the ozone layer. It was the smallest amount of activity. So there’s the difficulty of the fossil fuel industry, but it’s offset by the ambition of the non-government organizations and some of the governments. So lots of things are happening there. What would you add Susan?

Susan Solomon: Well, I think you summarized it pretty well. I think the other thing I would add is the engagement of young people today, they have been exercised and have become pretty upset about the future that they see themselves inheriting. Greta Thunberg has done a fantastic job I think of mobilizing worldwide in a sense. The ability to get together on things like the tools that we now have with the internet have allowed that to become an international movement much more effectively than we could have ever done in the 80s for climate. No matter how concerned we were, the telephone only worked so fast. So I think you see a public engagement on climate change that is driving a lot of what’s going on and having a huge influence.

Lucas Perry: Is there anything that you both wish or would suggest as actions or things that are really important for the generation? The generations that face climate change. What it is that they need to understand and do. I mean, we’re talking about making things practical, personal, and then is the last P perspective?

Susan Solomon: Personal, perceptible and practical.

Lucas Perry: Yeah, and so there’s still this, I mean, a large problem also beyond these issues, global catastrophic and existential risk issues, beyond making them personal. And there is certainly difficulty around making action around them practical. With the last P for perceptible, a lot of people don’t even agree about the science of climate change. So I’m wondering if there’s any similarities between that and what you experienced with HFCs, and what you suggest there is to do about that. I mean, because the climate issue has become politicized, certainly.

Susan Solomon: Yeah, I mean, I would argue that it is becoming perceptible. I think most people around the world have noticed that the summers are hotter than they used to be. That heat waves are more extreme than they used to be. So perceptible is not so much the problem anymore I don’t believe. What is the problem is the practical, that they believe that it would cost too much and that it’s not practical to do it. That is increasingly becoming less and less tenable when it comes to power plants, for example. It is, nowadays if you’re going to build a new one, it is cheaper to do it with solar and onshore wind than it is to do it with either coal or gas. And so, and nuclear of course, too are more expensive. So there has been a pivot in the power industry to renewables that’s been very rapid. I think that we need to put a lot more investment into that because it does take an upfront investment.

It’s true that when it came to some of the things we could do for the ozone layer, a lot of them were really, we were getting rid of things that we didn’t have a deep investment with. I mean, how invested are you with your can of spray deodorant in your medicine cabinet? Probably not very, you can throw it away and or maybe you can even use it until it runs out and then go out and buy a roll-on, right. But probably a lot of people went out and just bought the roll-on because they figured it was a very good thing to do for the planet. But so this is indeed a lot tougher because of our investment in existing infrastructure, some of which is tremendously expensive. But I don’t think there’s any real barrier to making the transition.

And as soon as those things become… As soon as the alternatives become cheap enough, they essentially pay for themselves because energy drives everything. So if we can make energy more cheaply with solar and wind, then the cost of doing absolutely anything that requires energy becomes automatically cheaper too. And that makes a sort of a snowball effect of greater and greater demand. We need to make our grid more robust to things like intermittency and able to transmit electricity over broader spatial ranges. That is doable. It’s not, other countries have already done it actually. So it’s not something we couldn’t do. There’s a lot of things that are beginning to really happen quite quickly. And I’m very optimistic, but we do need some real changes in our existing infrastructure. There’s no doubt about it.

Stephen Andersen: Yeah, so there’s two things I would add. The first thing is the United States is now behind the rest of the world on barrier removal. So in Europe, you can use technology that’s been in the market for five years now, absolutely proven safe and effective energy efficient that’s prohibited by the USCPA for use in the United States. So they are a wall against new technology where they used to be a door. And so somebody has to get in there and motivate and get approval.

Susan Solomon: Can you give an example?

Stephen Andersen: Example, there’s a refrigerant called HFC32 that has a third of the global warming potential that has 20% higher energy efficiency. It’s mildly flammable, but it hasn’t been approved in the United States. Similarly, there are natural refrigerants that have not been approved. And if you look at the timeline and another example, which is easy to understand is they approved years ago, a decade or more ago, a chemical that has a GWP of less than one to replace HFC134a, which has a GWP of 1,300. This was allowed for light trucks and cars. But the industry at that time did not apply for what’s called highway trucks to big trucks that move cargo and off-road vehicles like farm tractors and construction equipment, mining equipment, forestry. And that the industry applied months and months ago to have this year’s further equipment.

There’s no difference in using it on an off-road equipment or an on-road equipment or a big truck or a small truck because the cab of a big truck is about the same size as the interior of a car. But for some reason, the EPA has not finished that process, which is now way beyond the statutory limit of time. And they say it’s because they just haven’t had time to do it. Well, this is not acceptable. You have to have a government moving at the pace of industry. And then the last thing, probably important, the United States military and military organizations all over the world we’re a part of protecting the ozone layer. So far that’s not the case. If you look at inside at the documentation of military organizations, they say it’s a force multiplier, it makes everything worse in national security. They say it’s an amplifier of conflict. There’s tremendous concern about the displacement of populations and immigrants across borders and distractions from security because you have to do humanitarian relief. That’s all in the documentation.

But so far, all they do mostly is to do resiliency of their own facilities. So they’re doing what they need to do to protect against the effects of climate change, but they haven’t engaged yet in stopping climate change, which is much more cost effective. The last thing you want to do is let climate change happen and then try to run away and hide and build against it because that’s brutally, brutally expensive. So I think those two things, if we were more aggressive on approving new technology, and if we had the military organizations involved as part of the skill set and part of the solutions and so forth, I think we’d go a long way.

Lucas Perry: As a final question here, so Steven, your panel, the technology and economic assessment panel was super successful in this strategic and coordination front on the technology and the replacement of the technology, which is something we’re also just exploring. Does the Paris Climate Accord have anything similar? And do you see a panel like this as being something also that’s crucial for the climate change crisis and also the governance around other global catastrophic and existential risks?

Stephen Andersen: Yeah, I think you’re right. And Susan and I have both tried over many years to get the climate convention to do something like the tape. Recently, I realized that if you don’t want to wait for the IPCC to do this, you could do it as a shadow chip. And that it’s very easy I think for an individual sector to organize itself under the same principles of being objective and including members that have the coincidence of interest in changing the market and changing the technology. You could put that together within an industry and then bring forward the solutions that you’d like to see implemented. And this is almost happening in Europe right now because they’re phasing down HFCs much faster than the United States. And they’re doing it on a sector by sector basis and they’re involved in the stakeholders. And the stakeholders have figured out that if they come to the EU with a single plan that cuts out their share of the goal, that the European Union will approve that. If they come in with separate views and lots of disagreement, the EU will choose their own plan for them.

So they have two choices, do it their way or do it the government’s way. And so far, they always chose to do it. The practical cost effective technology that they understand the best. And that’s exactly what the team did. So I’m very enthusiastic about that model. And in fact, that’s where, if I were an industry, that’s where I would put my money right now is I try to say, how do we become the leaders on this so that these activists that can cause mayhem in our company would say no use messing with that company that shows an in-state runs, let’s go bother someone else. Let’s let them solve their problems and we’ll go on to a recalcitrant truck sector and to give them bloody hell. I think that’s a very persuasive argument.

Lucas Perry: All right, Steven and Susan, thank you so much for coming on. If you have any final words for the audience about climate change, the ozone hole and existential risk, here’s a space for you to share it.

Susan Solomon: I hope we’ve given you some hope in this period of talking. I mean, it’s easy to become kind of despondent about climate change because there’re terrible events making people suffer day after day. On the other hand, I really do believe there’s light at the end of the tunnel. I think the ozone issue demonstrates that. And I think we are on the road to getting to a solution.

Stephen Andersen: Yeah, I would just add to that that organizations like Future of Life Institute are a tremendous part of the solution. I do believe that recognition and explanation and all of those things make a big difference to getting people motivated, to take on this very hard work. It’s work you love when it’s over, but it’s always hard while you’re doing it. So we have to have the highest motivation possible. And you folks are part of that solution.

Lucas Perry: Well, thank you very much, Steven and Susan, for coming on the podcast and also for your scientific and strategic coordinations to a global risk in our lifetimes.

Susan Solomon: Thank you, Lucas.

Stephen Andersen: Thank you very much.

• How the US military views and takes action on climate change
• Examples of existing climate related difficulties and what they tell us about the future
• Threat multiplication from climate change
• The risks of climate change catalyzed nuclear war and major conflict
• The melting of the Arctic and the geopolitical situation which arises from that
• Messaging on climate change

Watch the video version of this episode here

See here for information on the Podcast Producer position

Check out Michael’s website here

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Michael Klare and explores how the US Department of Defense views and take action on climate change. This conversation is primarily centered around Michael’s book, All Hell Breaking Loose. In both this podcast and his book, Michael does an excellent job of making clear how climate change will affect global stability and functioning in our lifetimes through tons of examples of recent climate induced instabilities. I was also surprised to learn that despite changes in administrations, the DoD continued to pursue climate change mitigation efforts despite the Trump administration’s actions to remove mention and activism on climate change from the federal government. So, if you’ve ever had any doubts or if the impact of climate change and it’s significance has ever or does feel fuzzy or vague to you, this podcast might remedy that.

 I’d also like to make a final call for applications for the Podcast Producer role. If you missed it, we’re currently hiring for a Podcast Producer to work on the editing, production, publishing, and analytics tracking of the audio and visual content of this podcast. As the Producer you would be working directly with me, and the FLI outreach team, to help grow, and evolve this podcast. If you’re interested in applying, head over to the Careers tab on the Futureoflife.org homepage or follow the link in the description. The application deadline is July 31st, with rolling applications accepted thereafter until the role is filled. If you have any questions, feel free to reach out to socialmedia@futureoflife.org.

Michael Klare is a Five Colleges professor of Peace and World Security Studies. He serves on the board of directors of the Arms Control Association, and is a regular contributor to many publications including The Nation, TomDispatch and Mother Jones, and is a frequent columnist for Foreign Policy In Focus. Klare has written fourteen books and hundreds of essays on issues of war and peace, resource competition, and international affairs. You can check his work at Michaelklare.com. And with that I’m happy to present this interview with Michael Klare.

So to start things off here, I’m curious if you could explain at the highest level, how is it that the Pentagon views climate change and why is the Pentagon interested in climate change?

Michael Klare: So, if you speak to people in the military, they will tell you over and over again that their top concern is China. China, China, China followed by Russia and then maybe North Korea and Iran, and they spend their days preparing for war with China and those other countries. Climate change intercedes into this conversation because ultimately they believe that climate change is going to degrade their capacity to prepare for and to fight China and other adversaries down the road, that climate change is a complicating factor, a distraction that will undermine their ability to perform their military duties, and moreover, they see that the threat posed by climate change is increasing exponentially over time. So, the more they look into the future, the more they see that climate change will degrade their ability to carry out what they see as their primary function, which is to prepare for war with China. And so, it’s in that sense that climate change is critical. Now, then you go down in the specific ways in which climate change is a problem, but it’s ultimately because it will distract them from doing what they see as their primary responsibility.

Lucas Perry: I see, so there’s a belief in the validity of it and the way in which it will basically exacerbate existing difficulties and make achieving more important objectives more difficult.

Michael Klare: Something like that. Climate change they see as an intrusion into their work space. They’re trained as soldiers to carry out their military duties, which is combat related, and they believe that climate change is very real and getting more intense as time goes on and it’s going to hold them back, intrude on their ability to carry out their combat functions. It’s going to be a distraction on multiple levels. It’s going to create new kinds of conflicts that they would rather not deal with. It’s going to create emergencies around the world, humanitarian disasters at home and abroad, all of these are going to suck away resources, time, effort, energy, money that they believe should be devoted to their primary function of preparing for war with major enemies.

Lucas Perry: What would you say the primary interests of the Pentagon are right now other than climate change?

Michael Klare: Other than climate change, well the US Department of Defense at this time has a number of crises going on simultaneously. In addition to climate change, there’s COVID of course. Like every other institution in US society, the military was hampered by COVID, many service people came down with COVID and some died and it forced military operations to be restricted. Ships had to be brought back to port because COVID broke out on ships, so that was a problem. The military is also addressing issues of racism and extremism in the ranks. That’s become a major problem right now that they are dealing with, but they view climate change as the leading threat to national security of a non-military nature.

Lucas Perry: So, China was one of the first things that you mentioned. How would you also rank and relate the space of their considerations like Russia and a nuclear North Korea and Iran?

Michael Klare: Sure, the Department of Defense just released their budget for fiscal year 2022, and they rank the military threats and they say China is overwhelmingly the number one threat to US national security followed by Russia, followed by North Korea and Iran, and then down the list would be terrorist threats like Al-Qaeda and ISIS. But as you know, the administration has made a decision to leave Afghanistan and to downgrade US forces in that part of the world, so fighting terrorism and insurgency has been demoted as a major threat to US security, and even Russia has been demoted to second place. Over the past few years, Russia and China have been equated, but now China has been pushed ahead as the number one threat. The term they use is the pacing threat, which is to say that because China’s the number one threat, we have to meet that threat and if we can overcome China, the US could overcome any other threat that might come along, but China is number one.

Lucas Perry: So, there’s this sense of top concerns that the Department of Defense has, and then this is all happening in a context of climate change, which makes achieving its objectives on each of these threats more and more difficult. So, in the context of this interplay, can you describe the objectives of career officers at the Pentagon and how it’s related to and important for how they consider and work with climate change?

Michael Klare: Sure, so if you’re an aspiring general or admiral right now, as I say, you’re going to be talking about how you’re preparing your units, your division, your ship, your air squadron to be better prepared to fight China, but you also have to worry about what they call the operating environment, the OE, the operating environment in which your forces are going to be operating in, and if you’re going to be operating in the Pacific, which means dealing with China, then you have a whole set of worries that emerges. We have allies there that we count on: Japan, South Korea, the Philippines.

These countries are highly vulnerable to the effects of climate change and are becoming more so very rapidly. Moreover, we have bases in those places. Most of those bases, air bases and naval bases are at sea level or very close to sea level and are over and over again have been assaulted by powerful typhoons and have been disrupted, have had to be shut down for days or weeks at a time, and some of those bases like Diego Garcia in the Indian Ocean for example, or the Marianas Islands are not going to be viable much longer because they’re so close to sea level and sea level rise is just going to come and swamp them. So from an operating environment point of view, you have to be very aware of the impacts of climate change on the space in which you’re going to operate.

Lucas Perry: So, it seems like the concerns and objectives of career officers at the Pentagon can be distinguished in significant ways from the perspective and position of politicians, so there’s like some tension at least between career officers or the objectives of the Pentagon in relation to how some constituencies of the American political parties are skeptical of climate change?

Michael Klare: Yes, this was certainly the case during the Trump administration because the commander in chief, as one of his titles, President Trump forbad the discussion of climate change, and he was a denier. He called it a hoax and he forbad any conversation of that. So, the US military did have a position on climate change during the Obama administration. It had as early as 2010 the Department of Defense stated that climate change posed a serious threat to US security and was taking steps to address that threat. So when Trump came along, all of that had to go underground. It didn’t stop, but the Pentagon had to develop a whole lot of euphemisms, like changing climate or extreme weather events, all kinds of euphemisms used to describe what they saw as climate change, but that didn’t stop them from facing the consequences of climate change. During the Trump administration, US military bases in the US suffered billions and billions of dollars of damage from Hurricane Michael, from others that hit the East Coast and the Gulf of Mexico that did tremendous damage to a number of key US bases.

And, the military is still having to find the money to pay for that damage, and the Navy in particular is deeply concerned that its major operating bases in the United States… A Navy base by definition is going to be at the ocean, and many of these bases are of very low lying areas and already are being repeatedly flooded at very high tides, or when there are storms and the Navy is very aware that their ability to carry out their missions to reinforce American forces either in the Atlantic or Pacific are at risk because of rising seas, and they had to maneuver around Trump all during that period, trying to protect their bases, but calling it by different names, calling the danger they faced by different names.

Lucas Perry: Right, so there’s this sense of Trump essentially canceling mention of climate change throughout the federal government and its branches and the Pentagon responding by quietly still responding to what they see as a real threat. Is there anything else you’d like to add here about the Obama to Trump transition that helps to really paint the picture of how the Pentagon views climate change and what it did despite attempts to suppress thought and action around climate change?

Michael Klare: During the Obama administration, as I say, the Department of Defense acknowledged the reality of climate change number one. Number two said it posed a threat to US national security, and as a result said that the Department of Defense had an obligation to reduce its contribution to climate change to reduce its emissions and made all kinds of pledges that it was going to reduce its consumption of fossil fuels and increase its reliance on renewable energy, begin constructing solar arrays. A lot of very ambitious goals were announced in the Obama period, and although all of this was supposed to stop when Trump came into office because he said we’re not going to do anymore any of this anymore. In fact, the Pentagon continued to proceed with a lot of these endeavors, which were meant to mitigate climate change, but again, using different terminology that this was about base reliance, self-reliance, resiliency, and so on, not mentioning climate change, but nonetheless continued to proceed with efforts to actually mitigate their impact on climate.

Lucas Perry: All right, so is there any sense in which the Pentagon’s view of climate change is unique? And, could you also explain how it’s important and relevant for climate change and also the outcomes related to climate change?

Michael Klare: Yes, I think the Pentagon’s view of climate change, I think, is very distinctive and not well understood by the American public, and that’s why I think it’s so important, and that is that the Department of Defense sees climate change as… The term they use is as a threat multiplier. They say, look, we look out at the world and part of our job is to forecast ahead of time where our threat’s going to emerge to US security around the world. That’s our job, and to prepare for those threats, and we see that climate change is going to multiply threats in areas of the world that are already unstable, that are already suffering from scarcities of resources, where populations are divided and where resources are scarce and contested, and that this is going to create a multitude of new challenges for the United States and its allies around the world.

So, this notion of a threat multiplier is very much a part of the Pentagon’s understanding of climate change. What they mean by that is that societies are vulnerable in many ways and especially societies that are divided along ethnic and religious and racial lines as so many societies are, and if resources are scarce, housing, water, food, jobs, whatever, climate change is going to exacerbate these divisions within societies, including American society for that matter, but it’s going to exacerbate divisions around the world and it’s going to create a social breakdown and state collapse. And, the consequence of state collapse could include increased pandemics for example, and contribute to the spread of disease. It’s going to lead to mass migrations and mass migrations are going to become a growing problem for the US.

The influx of migrants on America’s Southern border, many of these people today are coming from Central America and from an area that’s suffering from extreme drought and where crop failure has become widespread, and people can’t earn an income and they’re fleeing to the United States in desperation. Well, this is something the military has been studying and talking about for a long time as a consequence of climate change, as an example of the ways in which climate change is going to multiply schisms in society and threats of all kinds that ultimately will endanger the United States, but it’s going to fall on their shoulders to cope with and creating humanitarian disasters and migratory problems.

And as I say, this is not what they view as their primary responsibility. They want to prepare for high-tech warfare with China and Russia, and they see all of this as a tremendous distraction, which will undermine their ability to defend the United States against its primary adversaries. So, it’s multiplying the threats and dangers to the United States on multiple levels including, and we have to talk about this, threats to the homeland itself.

Lucas Perry: I think one thing you do really well in your book is you give a lot of examples of natural disasters that have occurred recently, which will only increase with the existence of climate change as well as areas which are already experiencing climate change, and you give lots of examples about how that increases stress in the region. Before we move on to those examples, I just want to more clearly lay out all the ways in which climate change just makes everything worse. So, there’s the sense in which it stresses everything that is already stressed. Everything basically becomes more difficult and challenging, and so you mentioned things like mass migration, the increase of disease and pandemics, the increase of terrorism in destabilized regions, states may begin to collapse. There is, again, this idea of threat multiplication, so everything that’s already bad gets worse.

Lucas Perry: There’s loss of food, water, and shelter instability. There’s an increase in natural disasters from more and more extreme weather. This all leads to more resource competition and also energy crises as rivers dry up and electric dams stop working and the energy grid gets taxed more and more due to the extreme weather. So, is there anything else that you’d like to add here in terms of the specific ways in which things get worse and worse from the factor of threat multiplication?

Michael Klare: Then, you start getting kind of specific about particular places that could be affected, and the Pentagon would say, well this is first going to happen in the most vulnerable societies, poor countries, Central America, North Africa, places like that where society is already divided, poor, and the capacity to cope with disaster is very low. So, climate change will come along and conditions will deteriorate, and the state is unable to cope and you have breakdown and you have these migrations, but they also worry that as time goes on and climate change intensifies, that a bigger and bigger or richer and richer and more important states will begin to disintegrate, and some of these states are very important to US security and some of them have nuclear weapons, and then you have really serious dangers. For example, they worry a great deal about Pakistan.

Pakistan is a nuclear armed country. It’s also deeply divided along ethnic and religious lines, and it has multiple vulnerabilities to climate change. It goes between extremes of water scarcity, which will increase as the Himalayan glaciers disappear, but also we know that monsoons are likely to become more erratic and more destructive with more flooding.

All of these pose great threats to the ability of Pakistan’s government and society to cope with all of its internal divisions, which are already severe to begin with, and what happens when Pakistan experiences a state collapse and nuclear weapons begin to disappear into the hands of the Taliban or to forces close to the Taliban, then you have a level of worry and concern much greater than anything we’ve been talking before, and this is something that the Pentagon has started to worry about and to develop contingency plans for. And, there are other examples of this level of potential threat arising from bigger and more powerful states disintegrating. Saudi Arabia is at risk, Nigeria is at risk, the Philippines, a major ally in the Pacific is at extreme risk from rising waters and extreme storms, and I can continue, but from a strategic point of view, this starts getting very worrisome for the Department of Defense.

Lucas Perry: Could you also paint a little bit of a picture of how climate change will exacerbate the conditions between Pakistan, India, and China, especially given that they’re all nuclear weapon states?

Michael Klare: Absolutely, and this all goes back to water and many of us view water scarcity as the greatest danger arising from climate change in many parts of the world. In the case of India, China, Pakistan, not to mention a whole host of other countries depend very heavily on rivers that originate in the Himalayan mountains and draw a fair percentage of their water from the melting of the Himalayan glaciers and these glaciers are disappearing at a very rapid rate and are expected to lose a very large percentage of their mass by the end of this century due to warming temperatures.

And, this means that these critical rivers that are shared by these countries, the Indus River shared by India and Pakistan, the Brahmaputra River shared by India and China, these rivers, which provide the water for irrigation for hundreds of millions of people if not billions of people, depend on these rivers, the Mekong is another. As the water supply begins to diminish, this is going to exacerbate border disputes. All of these countries, Indian and China, Indian and Pakistan have border and territorial disputes. They have very restive agricultural populations to start with, that water scarcity is going to be the tipping point that will produce massive local violence that will lead to conflict between these countries, all of them nuclear armed.

Lucas Perry: So, to paint a little bit more of a picture of these historical examples of states essentially failing to be able to respond to climate events and the kind of destructive force that was to society and to the status of humanitarian conditions and the increasing need for humanitarian operations, so can you describe what happened in Tacloban for example, as well as what is going on in the Nigerian region?

Michael Klare: So, Tacloban is a major city on the island of Leyte in the Philippines, and it was a direct hit. It suffered a direct hit from Typhoon Haiyan in 2013. This was the most powerful typhoon to make landfall up until that point, an extremely powerful storm that created millions of homeless in the Philippines. Many people perished, but Tacloban was at the forefront of this. A city of several hundred thousand, many poor people living in low lying areas at the forefront of the storm. The storm surge was 10 or 20 feet high. That just over overwhelmed these low lying shanty towns, flooded them. Thousands of people died right away. The entire infrastructure of the city collapsed was destroyed, hospitals, everything. Food ran out, water ran out, and there was an element of despair and chaos. The Philippine government proved incapable of doing anything.

And, President Obama ordered the US Pacific Command to provide emergency assistance, and it sent almost the entire US Pacific fleet to Tacloban to provide emergency assistance on the scale of a major war, aircraft carrier, dozens of warships, hundreds of planes, thousands of troops to provide emergency assistance. Now, it was a wonderful sign of US aid. There are a number of elements of this crisis that are worthy of mention. In addition to all of this, one was the fact that there was anti-government rioting because of the failure of the local authorities to provide assistance or to provide it only to wealthy people in the town, and this is so often a characteristic of these disasters that assistance is not provided equitably, and the same thing was seen with Hurricane Katrina in New Orleans and this then becomes a new source of conflict.

When a disaster occurs and you do not have equitable emergency response, and some people are denied help and others are provided assistance, you’re setting the stage for future conflicts and anti-government violence, which is what happened in Tacloban And the US military had to intercede to calm things down, and this is something that has altered US thinking about humanitarian assistance because now they understand that it’s not just going to be handing out food and water, it’s also going to mean playing the role of a local government and providing police assistance and mediating disputes and providing law and order, not just in foreign countries, but in the United States itself and this proved to be the case in Houston with Hurricane Harvey in 2017 and in Puerto Rico with Hurricane Maria when local authorities simply disappeared or broke down and the military had to step in and play the role of government, which comes back to what I’ve been saying all along. From the military’s point of view, this is not what they were trained to do.

This is not what they want to do, and they view this as a distraction from their primary military function. So, here’s the Pacific fleet engaging in this very complex emergency in the Philippines, and what if there were a crisis with China that were to break out? The whole force would have been immobilized at that time, and this is the kind of worry that they have that climate change is going to create these complex emergencies they call them, or complex disasters that are going to require not just a quick in and out kind of situation, but a permanent or semi-permanent involvement in a disaster area and to provide services for which the military is not adequately prepared, but they see that climate change increasingly will force them to play this kind of role and thereby distracting them from what they see as their more important mission.

Lucas Perry: Right, so there’s this sense of the military increasingly being deployed in areas to provide humanitarian assistance. It’s obvious why that would be important and needed domestically in the United States and its territories. Can you explain why the military is incentivized or interested in providing global humanitarian assistance?

Michael Klare: This has always been part of American foreign policy, American diplomacy, winning friends, winning over friends and allies. So, it’s partly to make the United States look good particularly when other countries are not capable of doing that. We’re the one country that has that kind of global naval capacity to go anywhere and do that sort of thing. So, it’s a little bit a matter of showing off our capacity, but it’s also in the case of the Philippines, the Philippines plays a strategic role in US planning for conflict in the Pacific.

It is seen as a valuable ally in any future conflict with China and therefore its stability matters to the United States and the cooperation of the Philippine government is considered important and access to bases in the Philippines, for example, is considered important to the US. So, the fact that key allies of the US in the Pacific, in the Middle East and Europe are at risk of collapsing due to climate change poses a threat to the whole strategic planning of the US, which is to fight wars over there, in the forward area of operations off the coast of China, or off of Russian territory. So, we are very reliant on the stability and the capacity of key allies in these areas. So, providing humanitarian assistance and disaster relief is a part of a larger strategy of reliance on key allies in strategic parts of the world.

Lucas Perry: Can you also explain the conditions in Nigeria and how climate change has exacerbated those conditions and how this fits into the Pentagon’s perspective and interest in the issue?

Michael Klare: So, Nigeria is another country that has strategic significance for the US, not perhaps on the same scale as say Pakistan or Japan, but still important. Nigeria is a leading oil producer, not as important as it once was perhaps, but nonetheless important, but Nigeria is also a key player in peacekeeping operations throughout Africa and because the US doesn’t want to play that role itself, it relies on Nigeria for peacekeeping troops in many parts of Africa. And, Nigeria occupies a key piece of territory in Central Africa, which is it’s surrounded by countries, which are much more fragile and are threatened by terrorist organizations. So, Nigeria’s stability is very important in this larger picture, and in fact Nigeria itself is at risk from terrorist movements, especially Boko Haram and splinter groups, which continue to wreak havoc in Northern Nigeria despite years of effort by the Nigerian government to crush Boko Haram, it’s still a powerful force.

And, partly this is due to climate change. The Boko Haram operates in areas around Lake Chad, which is now a small sliver of what it once was. It has greatly diminished in size because of global warming and water mismanagement. And so, the farmers and fisher folk whose livelihood depended on Lake Chad has all been decimated. Many of them have become impoverished. The Nigerian government has proved inept and incapable of providing for their needs, and many of these people have therefore fallen prey to the appeals of recruitment by Boko Haram, young men without jobs. So, climate change is facilitating, is fueling the persistence of groups like Boko Haram and other terrorist groups in Nigeria, but that’s only part of the picture. There’s also growing conflict between pastoralists, these are herders, cattle herders whose lands are being devastated by desertification.

In this Sahel region, the southern fringe of the Sahara is expanding with climate change and driving these pastoralists into areas occupied by… These are all Muslim, the pastoralists are primarily Muslims and they’re moving into lands occupied by Christians, mainly Christian farmers, and there’s been terrible violence in the past few years, many hundreds of thousands of people displaced. Again, inept Nigerian response, and so I could go on. There’s violence in the Nigeria Delta region, the Niger Delta area in the south and in the area, their breakaway provinces. So, Nigeria is at permanent risk of breaking apart, and the US provides a lot of military aid to Nigeria and provides training. So, the US is involved in this country and faces a possibility of greater disequilibrium and greater US involvement.

Lucas Perry: Right, so I think this does a really good job of painting the picture of this factor of threat multiplication from climate change. So, climate change makes getting food, water, and shelter more difficult. There’s more extreme weather, which makes those things more difficult, which increases instability, and for places that are already not that stable, they get a lot more unstable and then states begin to collapse and you get terrorism, and then you get mass migration, and then there’s more disease spreading, so you get conditions for increased pandemics. Whether it’s in Nigeria or Pakistan and India or the Philippines or the United States and China and Russia, everything just keeps getting worse and worse and more difficult and challenging with climate change. So, could you describe the ladder of escalation of climate change related issues for the military and how that fits into all this?

Michael Klare: Well, now this is an expression that I made up to try to put this in some kind of context, drawing on the ladder of escalation from the nuclear era when the military talked about the escalation conflict from a skirmish to a small war, to a big war, to the first use of nuclear weapons, to all out nuclear war. That was the ladder of escalation of the nuclear age, and what I see happening is something of a similar nature where at present we’re still dealing mainly with these threat multiplying conditions occurring in the smaller and weaker states of Africa, Chad, Niger, Sudan and the Central American countries, Nicaragua and El Salvador, where you see all of these conditions developing, but not posing a threat to the central core of the major powers, but as climate change advances, the military expects and US intelligence agencies expect, as I indicated, that larger, stronger, richer states will experience the same kinds of consequences and dangers and begin to experience this kind of state disintegration.

So, what we’re seeing in places like Chad and Niger, which involves this skirmishing between insurgents, terrorists, and other factions in which the US is playing a remote role, is playing the role, but it’s remote to situations where a Pakistan collapses, a Nigeria collapses, a Saudi Arabia collapses would require a much greater involvement by American forces on a much larger scale and that would be the next step up the ladder of escalation arising from climate change, and then you have the possibility, as I indicated, where nuclear armed states would engage in conflict, would be drawn into conflict because of climate related factors like the melting of the Himalayan glaciers and Indian and Pakistan going to war or Indian and China going to war, or we haven’t discussed this, but another consequence of climate change is the melting of the Arctic and this is leading to competition between the US and Russia in particular for control of that area.

So, you go from disintegration of small states to disintegration of medium-sized states, to conflict between nuclear armed states, and eventually to conceivable US involvement in climate related conflicts. That would be the ladder of escalation as I see it, and on top of that, you would have multiple disasters happening simultaneously in the United States of America, which would require a massive US military response. So, you can envision, and the military certainly worries about this, a time when US forces are fully immobilized and incapable of carrying out what they see as their primary defense tasks because they’re divided. Half their forces are engaging in disaster relief in the United States and another half are dealing with these multiple disasters in the rest of the world.

Lucas Perry: So, I have a few bullet points here that you could expand upon or correct about this ladder of escalation as you describe it. So at first, there’s the humanitarian interventions where the military is running around to solve particular humanitarian disasters like in Tacloban. Then, there’s limited military operations to support allies. There’s disruptions to supply chains and the increase of failed states. There’s the conflict over resources. There’s internal climate catastrophes and complex catastrophes, which you just mentioned, and then there’s what you call climate shock waves, and finally all hell breaking loose where you have multiple failed states, tons of mass migration, a situation in which no state no matter how powerful is able to handle.

Michael Klare: Climate shock wave would be a situation where you have multiple extreme disasters occurring simultaneously in different parts of the world leading to a breakdown in the supply chains that keep the world’s economy afloat and keep food and energy supplies flowing around the world, and this is certainly a very real possibility. Scientists speak of clusters of extreme events, and we’ve begun to see that. We saw that in 2017 when Hurricane Harvey was followed immediately by Hurricane Irma in Florida, and then Hurricane Maria in the Caribbean and Puerto Rico and the US military responded to each of those events, but had some difficulty moving emergency supplies first from Houston to Florida, then to Puerto Rico. At the same time, the west of the US was burning up. There were multiple forest fires out of control and the military was also supplying emergency assistance to California, Washington State, and Oregon.

That’s an example of clusters of extreme events. Now looking into the future, scientists are predicting that this could occur in several continents simultaneously. And as a result, food supply chains would break down, and many parts of the world rely on imported grain supplies, or other food stuffs and imported energy. And in a situation like this, you could imagine a climate shockwave in which trade just collapses and entire states suffer from a major catastrophe, food catastrophes leading to state collapse and all that we’ve been talking about.

Lucas Perry: Can you describe what all hell breaking loose is?

Michael Klare: Well, this is my expression for the all of the above scenario. You have these multiple disasters occurring and one that we have not discussed at length is the threat to American bases and how that would impact on the military. So, you have these multiple disasters occurring that create a demand on the military to provide assistance domestically, like I say, many areas needing emergency assistance and not just of the obvious sort of handing out water bottles, but as I say, complex emergencies where the military is being called in to provide law and order, to restore infrastructure, to play the role of government. So, you need large capacity organizations to step in. At the same time, it’s being asked to do that in other parts of the world, or to intervene in conflicts with nuclear armed states happening simultaneously. But at the same time, its own bases have been immobilized by rising seas and flooding and fires. All of this is a very realistic scenario because parts of it have already occurred.

Lucas Perry: All right, so let’s make a little bit of a pivot here into something that you mentioned earlier, which is the melting of the Arctic. So, I’m curious if you could explain the geopolitical situation that arises from the melting of the Arctic Ocean… Sorry, the Arctic region that creates a new ocean that leads to Arctic shipping lanes, a new front to defend, and resource competition for fish, minerals, natural gas and oil.

Michael Klare: Yes, indeed. In a way, the Arctic is how the military first got interested in climate change, especially the Navy because the Navy never had much of an Arctic responsibility. It was covered with ice, so its ships couldn’t go there except for submarines on long-range patrols under the sea ice, but the Navy never had to worry about the Arctic. And then around 2009, the Department of the Navy created a climate change task force to address the consequences of a melting Arctic sea ice and came to the view that as you say, this is a new ocean that they would have to defend that they’d never thought about before, and for which they were not prepared.

Their ships were not equipped to operate, for the most part, in the Arctic. So ever since then, the Arctic has become a major geopolitical concern of the United States on multiple fronts. But two or three points in particular that need to be noted, first of all, the melting of the ice cap makes it possible to extract resources from the area, oil and natural gas, and it turns out there’s a lot of oil and natural gas buried under the ice cap, under the seabed of the Arctic and oil and gas companies are very eager to exploit those untapped reserves. So the area, what was once considered worthless, is now a valuable geo-economic prize and countries have exerted claims to the area, and some of these claims overlap. So, you have border disputes in the Arctic between Russia and the United States, Russia and Norway, Canada and Greenland, and so on. There are now border disputes because of the resources that are in these areas. And because of drilling occurring there, you now need to worry about spills and disasters occurring, so that creates a whole new level of Naval and Coast Guard operations in the Arctic. This has also led to shipping lanes opening up into the region, and who controls those shipping lanes becomes a matter of interest. Russia is trying to develop what it calls the Northern Sea Route from the Pacific to the Atlantic going across its Northern territory across Siberia, and potentially, this could save two weeks of travel for container ships, moving from Rotterdam say to Shanghai and could be commercially very important.

Russia wants to control that route but the U.S. and other countries says, “It’s not yours to control.” So, you have disputes over the sea routes. But then, more important than any of the above is that Russia has militarized its portion of the Arctic, which is the largest portion, and this has become a new frontier for U.S.-Russian military competition, and there has been a huge increase in military exercises, base construction. Now, from the U.S. point of view, the Arctic is a new front in the future war with Russia and they’re training for this all the time.

Lucas Perry: Could you explain how the Bering Strait fits in?

Michael Klare: The Bering Strait between the U.S. and Russia is a pretty narrow space and that’s the only way to get from the North Pacific into the Arctic region, whether you’re going to Northern Alaska and Northern Canada, or to across from China and Japan, across the Northern Sea Route to Europe. So, this becomes a strategic passage way, the way Gibraltar has been the past. And both the U.S. and Russia are fortifying that passageway and there’s constant tussling going on there. It doesn’t get reported much, but every other week or so, Russia will send up its war planes right to the edge of U.S. airspace in that region, or the U.S. will send its planes into the edge of Russian airspace to test their reflexes and their naval maneuvers happening all the time. So, this has become seen as a important strategic place on the global chessboard.

Lucas Perry: How does climate change affect the Bering Strait?

Michael Klare: Well, it affects it in the sense that it’s become the entry point to the Arctic and the climate change has made the Arctic a place you want to go that it wasn’t before.

Lucas Perry: All right. So, one point that you made in your book that I like to highlight is that the Arctic is seen as a main place for conflict between the great powers in the coming years. Is that correct?

Michael Klare: Yes. For the U.S. and Russia, it’s important, here we would focus more on the Barents Sea, the area above Norway, and just, it helps of course to have a map in your mind, but Russia shares the border with Norway in it’s extreme north. And that part of Russia is the Kola Peninsula, and it’s where the City of Murmansk is located, and that’s the headquarters of Russia’s Northern Fleet and where it keeps its nuclear or missile submarines are based there. So, that’s how, that’s one of Russia’s few ways of access into the Atlantic Ocean from its own territory, from its major naval port at Murmansk. The waters adjacent to Northern Norway and Russia, like on the other side, have become a strategic, very important strategic military location. The U.S. has started building military bases with Norway in that area close to the Russian border. We’ve now stationed B-1 bombers in that area, so it is seen as a likely first area of conflict in the event of a war between the U.S. and Russia is going to occur at that spot.

Climate change figures into this because Russia views its Arctic region as critical economically as well as strategically and is building up its military forces there. And therefore, from U.S. NATO point of view, it’s a more strategically important region. But you ask about China, and China has become very interested in the Arctic as a source of raw materials, but also as a strategic passageway from its east coast to Europe for the reason I indicated, if once the ice cap melts, they’ll be able to ship goods to Europe in much shorter space of time and bring goods back if they can go through the Arctic. But China also is very interested in drilling for energy at the Arctic and for minerals, there are a lot of valuable minerals believed to be in Greenland.

You can’t get to those now because Greenland is covered with ice. But as that ice melts, which it’s doing at a rapid rate, the ground is becoming exposed and mining activities have begun there for things like uranium, and rare earths, and other valuable minerals. China is very deeply interested in mining there and this has led to diplomatic maneuverings, didn’t Donald Trump once talk about buying Greenland, to geopolitical competition between the U.S. and China over Greenland and this area.

Lucas Perry: Are there any ongoing proposals for how to resolve territorial disputes in the Arctic?

Michael Klare: Well, the shorter answer is no, there’s talk, there is something called the Arctic Council and this is an organization of the states that occupy territory in the Arctic region and it has some very positive environmental agendas and had some success in addressing non-geopolitical issues. But it has not been given the authority to address territorial disputes that members have resisted that. So, you don’t have a, it’s not a forum that would provide for that. There is a mechanism under the United Nations Convention on the Law of the Sea that allows for adjudication of off shore territorial disputes and it’s possible that that could be a forum for discussion, but mostly, these disputes have remained unresolved.

Lucas Perry: I don’t know much about this. Does that have something to do with, you have so many, you have X many miles from your sea shelf or something having to do with like the tectonic plates or ocean something.

Michael Klare: I can… Yes, so under the UN Convention on the Law of the Sea, you’re allowed a 200 nautical mile exclusive economic zone off your coastline. Every, any coastal country can claim 200 nautical miles. But you’re also allowed an extra 150 miles if your outer continental shelf, if you can prove scientifically that your outer continental shelf extends beyond 200 nautical miles, then you can extend your EEZ another 150 nautical miles out to 350 nautical miles. And the Northern Arctic has islands and territories that have allowed contending states to claim overlapping EEZs-

Lucas Perry: Oh, okay.

Michael Klare: … on this bases.

Lucas Perry: I see.

Michael Klare: And Russia has claimed vast areas of the Arctic as part of its outer continental shelf. But the great imperial power of Denmark, which territorially, is one of the largest imperial powers on earth because it owns Greenland, and Greenland also has an extended outer continental shelf that overlaps with Russia’s, as does Canada’s. You have to picture the looking down, not on the kind of wall maps we have of the world in our classrooms that make the Arctic look huge, but from a global map, everything comes closer together up there. And so, these extended EEZs overlap and so Greenland, and Canada, and Russia are all claiming the North Pole.

Lucas Perry: Okay. So, I think that paints really well the picture of the already existing and conflict there and how it will likely only get worse in terms of the amount of conflict. It’d be great if we could focus a bit on nuclear weapons risk in climate change in particular. I’m curious if you could explain the DOD’s concerns an improving China, and a nuclear North Korea, and India, and Pakistan, and other nuclear states in this evolving situation of increasing territorial disputes due to climate change.

Michael Klare: From a nuclear war perspective, the two greatest dangers I think and I’ve mentioned these, one is the collapse or the disappearance of the Himalayan Glaciers, sparking a war between India and China that would go nuclear, or one between India and Pakistan that would go nuclear. That’s one climate-related risk of nuclear escalation. The other is in the Arctic, and here, I think the danger is the fact that Russia has turned the Arctic into a major stronghold for its nuclear weapons capabilities. It stations a large share of its nuclear retaliatory, warheads on submarines, and other forces that are based in the Arctic. And so, in the event of a conflict between the U.S. and Russia, this could very well take place in the Arctic region and trigger the use of nuclear weapons as a consequence.

Lucas Perry: I think we’ve done a really good job of showing all of the bad things that happen as climate change gets worse. The Pentagon has perspective on everything that we’ve covered here, is that correct?

Michael Klare: Yes.

Lucas Perry: So, how does the Pentagon intend to address the issue of climate change and how it affects its operations?

Michael Klare: The Pentagon has multiple responses to this, and this began as early as 2010 in the Quadrennial Defense Review of 2010. This is a every four-year strategic blueprint released by the Joint Chiefs of Staff and the Secretary of Defense. And that years was the first one that, number one, identified climate change as a national security threat and spelled out the responses that the military should make, and there were three parts to that. One part is, I guess you would call it hardening U.S. bases to the impacts of climate change, increasing resiliency and seawalls to protect low-lying bases, but otherwise, enhancing the survivability of U.S. bases in the face of climate change. That’s one response. A second response is in mitigating the department’s own contributions to climate change by reducing its reliance on fossil fuels. And I could talk what specifically they’re doing in that area.

The third is, and I think this is very interesting, they said that we should not only, that because climate change is a global problem, this was specific, climate change is a global problem, affects our allies and friends, and therefore, we should work with our allies and with the military forces of our allies and friends to do the same things we’re doing at home to do in their countries as well, that is to build resilience, to prepare for climate change, to reduce impacts so that this would be a global cooperative effort, military to military which has gotten very little attention, I think, from the media and from Congress and elsewhere, but a very important part of American foreign policy with respect to climate change.

Lucas Perry: So, there’s hardening our own bases and systems, I believe in your book you mentioned, for example, turning bases into operational islands such that their energy and material needs are disconnected from supply lines. The second was reducing the greenhouse emissions of the military, and the third is helping allies with such efforts. I’m curious if you could describe a bit more the first and the second of these, the hardening of our own systems and bases and becoming more green. Because I mean, it is interesting and at least a little bit surprising that the military is trying to become green in order to improve combat readiness through independence of a foreign and domestic fuel needs and sources. So, could you explain a little bit more this, for example, the drive to create a green fleet in the Navy?

Michael Klare: Sure. Now, but this began during the Obama administration and then went semi-underground during the Trump administration, so the information we have is mainly pre-Trump. Now, under president Biden, climate change has been elevated to a national security threat as per an executive order he issued shortly after taking office, and our new Secretary of Defense Lloyd Austin has said, has issued a complementary statement that climate change is a departmental-wide Department of Defense concern, so activities that were prohibited by the Trump administration will now be revived. So, we will now hear a lot more about this in the months ahead, but there is a four-year blackout of information on what was being done. But during the Obama administration, the Department of Defense was ordered to, as I say, to work on adaptation and mitigation both as part of its responsibilities, the adaptation affected particularly bases in low low-lying coastal areas.

And there are a lot of U.S. bases for historic purposes, for historic reasons are located along the East Coast of the U.S., that’s where they started out. Most important of them is the Norfolk Naval Station in Virginia, the most important naval base in the United States. It’s at sea level and it’s on basically reclaimed swamplands and it’s subsiding into the ocean at the same time sea level is rising. But there are many other bases along the East Coast and Florida, and in the Gulf coast that are at equal risk. And so, part of what the military is doing is to build seawalls to protect them against sea surges, moving critical equipment from areas that are in high flood prone areas to areas that are at higher elevation, adopting codes, any new buildings built on these bases have to be hardened against hurricanes, and sensitive equipment, electronic equipment has to be put in the higher stories so that if they are flooded they won’t be damaged.

There’s a lot of very concrete measures that have to do with base construction that have been undertaken to enhance the resilience of bases in response to extreme storms and flooding. That’s one aspect of this. The mitigation aspect is to reduce reliance on fossil fuels and to convert as wherever possible, vehicles, air, ground, and sea vehicles to use alternative fuels. So, the Navy, the Army, the Air Force are converting their non-tactical vehicle fleets, they all have huge numbers of ordinary sedans, and vans, and trucks. Increasingly, these will be hybrids or electric vehicles. And the Air Force is experimenting with alternative fuels produced by algae, and the Navy has experimented with alternative fuels derived from agricultural products, and so on. So, there’s a lot of experimentation going on, a lot of, some of the biggest solar arrays in the U.S. are on U.S. military bases or constructed at the behest of U.S. military bases by private energy companies. Those are some of the activities that are underway.

Lucas Perry: In addition to threatening U.S. military bases and the bases of our allies, climate change will also affect the safety and security of, for example biosafety level 4 labs and also nuclear power plants. So, I’m curious how you view the risks of climate change affecting crucial infrastructure, should it fail, could create global catastrophe, for example, from nuclear power plants melting down or pathogens being released from biosafety labs that fail under the stresses of climate change.

Michael Klare: I have not seen the literature on the bio labs in the Pentagon literature. What they do worry about is the fragility of the U.S. energy infrastructure in particular, in part because they depend on the same energy infrastructure as we do for their energy needs, for electricity transmission, pipelines and the like to supply their bases and their other facilities. And they’re very aware that the U.S. energy infrastructure is exceedingly vulnerable to climate change, either a lot of it, a very large part of our infrastructure is on the East Coast and the West Coast, very close to sea level, very exposed to storm damage and a lot of it is just fragile. A clearer example of that is Hurricane Maria in Puerto Rico when the electric system collapsed entirely and the Army Corps of Engineers had to come in and were there for almost an entire year rebuilding the energy infrastructure of Puerto Rico.

They’ve had to do this and other places as well. So, they are very worried that climate change disasters, multiple disasters, will knock out the power in the U.S. causing major cascading failures. So, when energy fails, then petrochemical facilities fail. And that’s what happened in Houston in Hurricane Harvey. The power failure went out and these petrochemical facilities, which Houston has many of, failed and toxic chemicals spilled out, and also the sewer system collapsed. So, you have, cascading failures producing toxic threat. And the military had to issue toxic protective clothing to its personnel in doing rescue operations because the water in flooded areas of Houston was poisonous. So, it’s the cascading effects that they worry about. This happened in New York City with Hurricane Sandy in 2012 where power went out, then gas stations couldn’t operate and hospitals and nursing homes couldn’t function. Well, I’m going on here, but you get a sense of the interrelationship between these critical elements of infrastructure. Fires are another aspect of this, as we know from California. A lot of US bases in California are at risk from fires and the transmission lines that carry the energy. I was going to mention the Colonial Pipeline disaster, which was a cyber attack, not climate related, but that exposes the degree to which our energy infrastructure is fragile.

Lucas Perry: If it rains or snows just enough, we’ve all experienced losing power for six hours or more. The energy grid seems very fragile even to relatively normal weather.

Michael Klare: Yes, but with climate change and these multiple, simultaneous disasters where the whole systems break down.

Lucas Perry: Do you see lethal autonomous weapons as fitting into the risks of escalation in a world stressed by climate change?

Michael Klare: Well, I see lethal autonomous weapons as a major issue and problem, which I’ve written about and I worry about a great deal. Now, what is their relationship to climate change? I couldn’t say. I think the military in general is facing the world in which they feel that humans are increasingly unable to cope with the demands of time compression and decision-making and the complexity of the environment in which decision-makers have to operate and that’s partly technological, it’s partly just the complexity of the world that we’ve been discussing.

And so, there’s ever increasing sense among the military that commanders have to be provided with computer assisted decision-making and autonomous operations because they can’t process the amount of data that’s coming into them simultaneously. This is behind not just autonomous weapons systems, but autonomous weapons systems’ decision-making. The new plans for how the Army, Navy, and Air Force will operate will be fewer human decision-makers and more machine information processors and decision-makers, and which humans will be given a menu of possible choices, but they will be strike this set of targets or that set of targets, but not stop and think about this, and maybe we should de-escalate. They’re going to be militarized options.

Lucas Perry: So, some sense of lethal autonomous weapons is potentially exacerbating or catalyzing the speed at which the ladder of escalation is moved through.

Michael Klare: No question about it. Many factors are contributing to that. The speed of weaponry, the introduction of hypersonic missiles, which cuts down flight time from 30 minutes to five minutes, the fact that wars are being conducted in what they call multiple domains simultaneously: cyber, space, air, sea, and ground, that no commander can know what’s happening in all of those domains and make decisions. So, you have to have what they want to create, a super brain called the Joint All-Domain Command and Control System, the JADC2 system, which will collect data from sensors all over the planet and compress it into simplified assessments of what’s happening, and then tell commanders, here are your choices, one, two, and three, and you have five seconds to choose, and if not, we’ll pick the best one and we’ll be linked directly to the firers to launch weapons. This is what the future will look like, and they’re testing this now. It’s called Project Convergence.

Lucas Perry: So, how do you see all of this affecting the risks of human extinction and of existential risks?

Michael Klare: I’m deeply concerned about this inclination to rely more on machines to make decisions of life and death for the planet. I think everybody should be worried about this, and I don’t think enough attention is being paid to these dangers of automating life and death decision-making, but this is moving ahead very rapidly and I think it does pose enormous risks. The reason that I’m so worried is that I think the computer assisted decision-making will have a bias towards military actions.

Humans are imperfect and sometimes we make mistakes. Sometimes we get angry and we go in the direction of being more violent and brutal. There’s no question about that, but we also have a capacity to say, stop, wait a minute, there’s something wrong here and maybe we should think twice and hold back. And, that’s saved us on a number of occasions from nuclear extinction. I recommend the book Gambling with Armageddon by Martin Sherwin, a new account of the Cuban Missile Crisis day by day, hour by hour account, and which it was clear that the US and Russia came very close, extremely close to starting a nuclear war in 1962, and somebody said, “Wait a minute, let’s just think about this. Let’s not rush into this. Let’s give it another 24 hours to see if we can come up with a solution.”

Adlai Stevenson apparently played a key role in this. I fear that the machines we designed are not going to have that kind of thinking built into them, that kind of hesitancy, that second thinking. I think the machines are going to be designed… The algorithms that inhabit them are going to reflect the most aggressive possible outcomes, and that’s why I fear that we move closer to human extinction in a crisis than before, and because of the time of decision-making is going to be so compressed that humans are going to have very little chance to think about this.

Lucas Perry: So, how do you view the interplay of climate change and autonomous weapons as affecting existential risk?

Michael Klare: Climate change is just going to make everything on the planet more stressful in general. It’s going to create a lot of stress, a lot of catastrophes occurring simultaneously and creating a lot of risk events happening that people are going to have to be dealing with, and they’re going to create a lot of hard, difficult choices. Let’s say you’re the president, you’re the commander in chief, and you have multiple hurricanes striking and fires striking the United States, that’s hardly an unlikely outcome, at the same time that there’s a crisis with China and Russia occurring where war would be a possible outcome. There’s a naval clash at sea in the South China Sea or something happening on the Ukraine border, and meanwhile, Nigeria is breaking apart and India and Pakistan are at the verge of war.

These are very likely situations in another 10 to 20 years if climate change proceeds the way it is. So, just the complexity of the environment, the stress that people will be under, the decisions they’re going to have to make swiftly between do we save Miami or do we save Tokyo? Do we save Los Angeles or do we save New York, or do we save London? We only have so many resources. In these conditions, I think the inclination is going to be to rely more on machines to make decisions and to carry out actions, and that I think has inherent dangers in it.

Lucas Perry: Do you and/or the Pentagon have a timeline for… How much and how fast is the instability from climate change coming?

Michael Klare: This is a progression. We’re on that path, so there’s no point at which you could say we’ve reached that level. It’s just an ever increasing level of stress.

Lucas Perry: How do you see the world in five or 10 years given the path that we’re currently on?

Michael Klare: I’m pessimistic about this, and the reason I am pessimistic is because if you go back and read the very first reports of the Intergovernmental Panel on Climate Change, the IPCC, their very first reports, and they would give a series of projections based on their estimates of the pace of greenhouse gas emissions. If they go this high, then you have these projections. If they go higher, then these projections out to 2030, 2040, 2050, we’ve all seen these charts.

So, if you go back to the first ones, basically we’re living in 2021 what they said were the worst case projections for 2040 to 2050 by and large. So, we’re moving into the danger zone. So, what I’m saying is we’re moving into the danger zone much, much faster than the most worst case scenarios that scientists were talking about 10 years ago, or 20 years ago, and if that’s the case, then we should be very, very worried about the pace at which this is occurring because we’re off the charts now from those earlier predictions of how rapidly sea level rise was occurring, desertification was occurring, heat waves. We’re living in a 2050 world now. So, where are we going to be in a 2030? We’re going to be in a 2075 world and that world was a pretty damn scary world.

Lucas Perry: All right, so I’m mindful of the time here. So, just a few more questions about messaging would be nice. So, do you think that tying existential risks to national security issues would benefit the movement towards reducing existential risks, given that climate change is elevated in some sense by the DOD taking it seriously on account of national security?

Michael Klare: So, let me explain why I wrote this book, and this is very much a product of the Trump era, but I think it’s still true today that you have a country that’s divided between environmentalists and climate deniers, and this divide has prevented forward movement in Congress to pass legislation that will make a significant difference in this country, and I believe this has to come from national level, the kind of changes we need, the massive investments in renewables and charging stations for electric vehicles, all these things require national leadership, and right now that’s impossible because of the fundamental divide between the Democrats and Republicans or denialists and environmentalist, however you want to put it. Some of my friends in the environmental community, dear friends, think if we could only get across the message that things are getting worse that those deniers will finally wake up and change their views.

I don’t think that’s going to happen. I think more scientific evidence about climate change is not going to win over more people. We’ve tried that. We’ve done everything we can to make the scientific evidence known. So, the way to win, I believe the military perspective that this is a threat to the national security of the United States of America, are you a patriotic American or not? Do you care about the security of this country or not?

This is not a matter of environmentalism or anti-environmentalism. This is about the national security of this country. Where do you stand on that? That this is a third approach that could possibly win over some segment of that population that until now has resisted action on climate change, that’s not going to listen to an environmentalist or green argument. There is evidence that this approach is making a difference, that Republicans who won’t even talk about the causes of climate change, but who acknowledge that their communities are at risk or the country is at risk on a national security basis, and therefore are willing to invest in some of the changes that are necessary for that reason. So, I do believe that making this argument, it could win over enough of that resistant population to make it possible to actually achieve forward momentum.

Lucas Perry: Do you think that relating climate change to migration issues is helpful for messaging?

Michael Klare: I’m not sure because I think people who are opposed to migration don’t care what the cause is, but I do think that it might feed into the argument that I was just making that our security would be better off by emphasizing climate change and therefore taking steps to reduce the pressures that lead climate migrants to migrate. The military certainly takes that view, so it could be helpful, but I think it’s a difficult topic.

Lucas Perry: All right, so given everything we’ve discussed here today, how would you characterize and summarize the Pentagon’s interest, view, and action on climate change and why that matters?

Michael Klare: So, now we have a new test because as I’ve indicated, we had a blackout period of four years during the Trump administration when all of this was hidden and couldn’t be discussed. So, we don’t know how much was accomplished. Now, this is an explicit priority for the Department of Defense and the defense budget, and other documents say that this is a priority for the department and the Armed Forces, and they are required to take steps to adapt to climate change and to mitigate their role in climate change.

So, we have to see how much actually is accomplished in this new period before really you can make any definitive assessment, but I think that you can see that the language adopted by the Biden administration and Lloyd Austin at the Department of Defense is so much stronger and more vigorous than what the Pentagon was saying in the Obama administration. So, even though there was a four year blackout period, there was a learning curve going on, and what they’re saying today is much more advanced and the sense of recognizing the severity of the risks posed by climate change and the necessity of making this a priority.

Lucas Perry: All right, so as we wrap up, are there any final words or anything you’d like to share that you feel is left unsaid or any parting words for the audience?

Michael Klare: As I started out, we mustn’t forget that if you asked anybody in the military what their job is, they’re going to come back to China number one. So, we shouldn’t forget that, defending against China. It’s only after you peel away the layers of how they’re going to operate in a climate altered world that all of these other concerns start spilling out, but it’s not going to be the first thing that they’re likely to say. I think that has to be clear, based on my conversations, but there is a real awareness that in fact climate change is going to have an immense impact on the operations of the military in the years ahead, and that its impact is going to grow exponentially.

Lucas Perry: All right. Well, thank you very much for coming on Michael, and for sharing all of this with us. I really appreciated your book and I recommend others check it out as well, it’s All Breaking Loose. I think it does a really good job of showing the ways in which the world is going to get worse through climate change. There’s a lot of really great examples in there. So, also the audiobook has a really great narrator, which I very much liked. So, thank you very much for coming on. If people want to check or follow you out on social media, where and how can they do that?

Michael Klare: Oh, I’m at michaelklare.com and let’s start there.

Lucas Perry: All right. Do you also have a place for you where you list publications?

Michael Klare: At that site.

Lucas Perry: At that site? Okay.

Michael Klare: And, it’s K-L-A-R-E, Michael Klare, K-L-A-R-E.

Lucas Perry: All right, thank you very much, Michael.

• Whether ‘Oumuamua is alien or natural in origin
• The culture of science and how it affects fruitful inquiry
• Looking for signs of alien life throughout the solar system and beyond
• Alien artefacts and galactic treaties
• How humanity should handle a potential first contact with extraterrestrials
• The relationship between what is true and what is good

3:28 What is ‘Oumuamua’s wager?

11:29 The properties of ‘Oumuamua and how they lend credence to the theory of it being artificial in origin

17:23 Theories of ‘Oumuamua being natural in origin

21:42 Why was the smooth acceleration of ‘Oumuamua significant?

23:35 What are comets and asteroids?

28:30 What we know about Oort clouds and how ‘Oumuamua relates to what we expect of Oort clouds

33:40 Could there be exotic objects in Oort clouds that would account for ‘Oumuamua

38:08 What is your credence that ‘Oumuamua is alien in origin?

44:50 Bayesian reasoning and ‘Oumuamua

46:34 How do UFO reports and sightings affect your perspective of ‘Oumuamua?

54:35 Might alien artefacts be more common than we expect?

58:48 The Drake equation

1:01:50 Where are the most likely great filters?

1:11:22 Difficulties in scientific culture and how they affect fruitful inquiry

1:27:03 The cosmic endowment, traveling to galactic clusters, and galactic treaties

1:31:34 Why don’t we find evidence of alien superstructures?

1:36:36 Looking for the bio and techno signatures of alien life

1:40:27 Do alien civilizations converge on beneficence?

1:43:05 Is there a necessary relationship between what is true and good?

1:47:02 Is morality evidence based knowledge?

1:48:18 Axiomatic based knowledge and testing moral systems

1:54:08 International governance and making contact with alien life

1:55:59 The need for an elite scientific body to advise on global catastrophic and existential risk

1:59:57 What are the most fundamental questions?

See here for information on the Podcast Producer position

See the Office of the Director of National Intelligence report on unidentified aerial phenomena here

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Avi Loeb, and in it, we explore ‘Oumuamua, an interstellar object that passed through our solar system and which is argued by Avi to potentially be alien in origin. We explore how common extraterrestial life might be, how to search for it through the space archaeology of bio and techno-signatures they might create. We also get into Great Filters and how making first contact with alien life would change human civilization.

This conversation marks the beginning of the continuous uploading of video content for all of our podcast episodes. For every new interview that we release, you will also be able to watch the video version of each episode on our YouTube channel. You can serach for Future of Life Institute on YouTube to find our channel or check the link in the description of this podcast to go directly to the video version of this episode. There is also bonus content to this episode which has been released speararetly on both our audio and visual feeds.

After our initital interview, the U.S. government released a report on UFOs, otherwise now known as UAPs, titled “Prelimiary Assessment: Unidentified Aerial Phenomena”. Given the release of this report and the relevance of UFOs to ‘Oumuamua, both in terms of the culture of science surrounding UFOs and their potential relation to alien life, I sat down to interview Avi for a second time to explore his thoughts on the report as well as his assessment of unidentified aerial phenomena. You can find this bonus content wherever you might be listening.

We’re also pleased to announce a new opportunity to join this podcast and help make existential risk outreach content. We are currently looking to hire a podcast producer to work on the editing, production, publishing, and analytics tracking of the audio and visual content of this podcast. You would be working directly with me, and the FLI outreach team, to help produce, grow, and evolve this podcast. If you are interested in applying, head over to the “Careers” tab on the FutureofLife.org homepage or follow the link in the description. The application deadline is July 31st, with rolling applications accepted thereafter until the role is filled. If you have any questions, feel free to reach out to socialmedia@futureoflife.org. 

Professor Loeb received a PhD in plasma physics at the age of 24 from the Hebrew University of Jerusalem and was subsequently a long-term member at the Institute for Advanced Study in Princeton, where he started to work in theoretical astrophysics. In 1993, he moved to Harvard University where he was tenured three years later. He is now the  and is a former chair of the Harvard astronomy department. He also holds a visiting professorship at the Weizman Institute of Science and a Sackler Senior Professorship by special appointment in the School of Physics and Astronomy at Tel Aviv University. Loeb has authored nearly 700 research articles and four books. The most recent of which is “Extraterrestial: The First Sign of Intelligent Life Beyond Earth”. This conversation is centrally focused on the contents of this work. And with that, I’m happy to present this interview with Avi Loeb.

To start things off here, I’m curious if you could explain what ‘Oumuamua’s wager is and what does it mean for humanity in our future?

Avi Loeb: ‘Oumuamua was the first interstellar object that was spotted near earth. And by interstellar, I mean an object that came from outside the solar system. We knew that because it moved too fast to be bound to the sun. It’s just like finding an object in your backyard from the street. And this saves you the need to go to the street and find out what’s going on out there. In particular, from my perspective, it allows us to figure out if the street has neighbors, if we are the smartest kid on the block, because this object looked unusual. It didn’t look like any rock that we have seen before in the solar system. It exhibited a very extreme shape because it changed the amount of reflected sunlight by a factor of 10 as it was tumbling every eight hours.

It also didn’t have a cometary tail. There was no gas or dust around it, yet it showed an excess push away from the sun. And the only possible interpretation that came to my mind was a reflection of sunlight. And for that, the object had to be very thin, sort of like a sail, but being pushed by sunlight rather than by the wind. This, you often find on a boat. And the nature doesn’t make sail, so in a scientific paper, we propose that maybe it’s artificial in origin. And since then in September 2020, there was another object found that was pushed away from the sun by reflecting sunlight. And without the cometary tail, it was discovered by the same telescope in Hawaii, Pan-STARRS, and was given the name 2020 SO. And then, the astronomers realized actually it’s a rocket booster that we launched in 1966 in a lunar landing mission. And we know that this object had very thin walls, and that’s why it had a lot of area for its mass and it could be pushed by reflecting sunlight.

And we definitely know that it was artificial in origin, and that’s why it didn’t show cometary tail because we produced it. The question is, who produced ‘Oumuamua? And my point is that just like Blaise Pascal, the philosopher, argued that we cannot ignore the question of whether God exists, because Pascal was a mathematician and he said, okay, logically the two possibilities either God exists or not. And we can’t ignore the possibility that God exists because the implications are huge. And so, my argument is very similar. The possibility that ‘Oumuamua is a technological relic carries such great consequences for humanity, that we should not ignore it. Many of my colleagues in academia dismiss that possibility. They say we need extraordinary evidence before we even engage in such a discussion. And my point is requiring extraordinary evidence is a way of brushing it aside.

It’s a sort of a self-fulfilling prophecy if you’re not funding research that looks for additional evidence, it’s sort of like stepping on the grass and claiming the grass doesn’t grow. Because for example, to the tape gravitational waves required an investment of $1.1 billion by the National Science Foundation. We would never discover gravitational waves unless we invest that amount. To search for dark matter, we invested hundreds of millions of dollars so far. We didn’t find what the dark matter is. It’s a search in the dark. But without the investment of funds, we will never find. So on the one hand, the scientific community puts almost no funding towards the search for technological relics, and at the same time argues all the evidence is not sufficiently extraordinary for us to consider that possibility in the first place. And I think that’s a sign of arrogance. It’s a very presumptuous statement to say, we are unique and special. There is nothing like us in the universe.

I think a much more reasonable down to earth kind of approach is a modest approach. Basically saying, look, the conditions on earth are reproducing in tens of billions of planets within the Milky Way galaxy alone. We know that from the capital satellite, about half of the sun-like stars have a planet the size of the earth, roughly at the same separation. And that means that not only we are not at the center of the universe, like Aristotle argued, we are also what we find in our backyard is not privileged. There are lots of sun-earth systems out there. And if you arrange for similar circumstances, you might as well get similar outcomes. And actually most of the stars formed billions of years before the sun. And so that to me indicates that there could have been a lot of technological civilization like ours that launched equipment into space just like we launched the Voyager 1, Voyager 2, New Horizons, and we just need to look for it.

Even if these civilizations are dead, we can do space archeology. And what they mean by that is when I go to the kitchen and I find an ant, I get alarm because there must be many more ants out there. So we found ‘Oumuamua, to me, it means that there must be many more out there and weird objects that do not look like a comet or an asteroid that we have seen before within the solar system. And we should search for them. And for example, in a couple of years, there would be the Vera Rubin Observatory that would be much more sensitive than the Pan-STARRS telescope and could find one such ‘Oumuamua like object every month. So when we find one that approaches us and we have an alert of a year or so, we can send a spacecraft equipped with a camera that will take a close up photograph of that object and perhaps even land on it, just like OSIRIS-REx landed on the asteroid Bennu recently and collected a sample from it, because they say a picture is worth a thousand words.

In my case, a picture is worth 66,000 words. The number of words in my book. If we had the photograph, I will need to write the book. It would be obvious whether it’s a rock or an artificial object. And if it is artificial and we land on it, it can read off the label made on Planet “X” and even import the technology that we find there to earth. And if it’s a technology representing our future, let’s say a million years into our future, it will save us a lot of time. It will give us a technological leap and it could be worth a lot of money.

Lucas Perry: So, that’s an excellent overview. I think of a really good chunk of the conversation, right? So there’s this first part of an interstellar object called ‘Oumuamua, entering the solar system in 2017. And then there are lots of parameters about and properties of this object, which are not easily or readily explainable as an asteroid or as a comet. Some of these things that we’ll discuss are for example, its rotation, its brightness variation, its size, its shape, how it was accelerating on its way out. And then the noticing of this object is happening in a scientific context, which has some sense of arrogance of not being fully open to exploring hypotheses that seem a bit too weird or too far out there. People are much more comfortable trying to explain it as some kind of like loose aggregate of a cosmic dust bunny or other things which don’t really fit or match the evidence.

And so then you argue that if we look into this with epistemic humility, then if we follow the evidence, it takes us to having a reasonable amount of credence that this is actually artificial in origin rather than something natural. And then that brings up questions of other kinds of life, and the Drake equation, and what it is that we might find in the universe, and how to conduct space archeology. So to start off, I’m curious if you could explain a bit more of these particular properties that ‘Oumuamua had and why it is that a natural origin isn’t convincing to you?

Avi Loeb: Right. I basically follow the evidence. I didn’t have any agenda. And in fact, I worked on the early universe and the black holes throughout most of my career, and then came along this object that was quite unusual. A decade earlier, I predicted how many rocks from other stars should we expect to find. And that was the first paper predicting that. And we predicted the Pan-STARRS telescope that discovered the ‘Oumuamua will not find anything. And the mere detection of ‘Oumuamua was a surprise by all this with magnitude, I should say. And it is still a surprise given what we know about the solar system, the number of rocks that the solar system produce. But nevertheless, that was the first unusual fact, but it still allowed for ‘Oumuamua to be a rock. And then, it didn’t show any cometary tail. And the Spitzer Space Telescope then put very tight limits on any carbon-based molecules in its vicinity or any dust particles.

And it was definitely clear that it’s not a comet because if you wanted to explain the excess push that it exhibited away from the sun through cometary evaporation, you needed about 10% of the mass of this object to be evaporated. And that’s a lot of mass. We would have seen it. The object size is of over the size of the football field, the 100 to 200 meters. And we would see such evaporation easily. So, that implied that it’s not a comet. And then if it’s not the rocket effect that is pushing it through evaporation, the question arose as to what actually triggers that push. And the suggestion that we made in the paper is that it’s the reflection of sunlight. And for that to be effective, you needed the object to be very thin. The other aspect of the object that was unusual is that as it was tumbling, every eight hours, the amount of sunlight reflected from it changed by a factor of 10.

And that implied that the object has an extreme shape, most likely pancake-shaped, flat and not cigar-shaped. Depiction of the object that’s cigar was based on the fact that projected on the sky as it was tumbling, the area that it showed us changed by a factor of 10. So then of course, if you look at the piece of paper tumbling in the wind and you look at it when it’s sideways, it does look like a cigar, but intrinsically it’s flat. And that is at the 90% confidence when trying to model the amount of light reflected from it as it was tumbling. The conclusion was at the 90% confidence that it should be pancake-shaped, flat, which again is unusual. You don’t get such objects very often in the context of rocks. And the most that we have seen before was of the order of a factor of three in length versus width. And then came the fact that it originated from a special frame of reference called the local standard of rest, which is sort of like the local parking lot of the Milky Way galaxy.

If you think about it, the stars are moving relative to each other in the vicinity of the sun, just like cars moving relative to each other in the center of a town. And then there is a parking lot that you can get to when you average over the motions of all of the stars in the vicinity of the sun, and that is called the local standard of rest. And ‘Oumuamua originated at rest in that frame. And that’s very unusual because only one in 500 stars is so much at rest in that frame as ‘Oumuamua was. So firstly, it tells you it didn’t originate from any of the nearby stars. Also, not likely from any of the far away stars because they are moving even faster relative to us, if they’re far away because of the rotation around the center of the Milky Way galaxy.

So it was not a natural result yet, a very small likelihood to have an object that is so rare. But then, or sort of like a buoy sitting at rest on the surface of the ocean and the sun bumped into it like a giant ship. And the question is if it’s artificial in origin, why would it originate from that frame? And one possibility is that it’s a member of objects on a grid that’s for navigation purposes. If you want to know your coordinates as you’re navigating an interstellar space, you find your location relative to this grid. And obviously you want those objects to be stationary, to be addressed relative to the local frame of the galaxy. And another possibility is that it’s a member of relay stations for communication. So to save on the power needed for transmission of signals, you may have relay stations like we have on earth and it’s one of them.

We don’t know the purpose of this object because we don’t have enough data on it. That’s why we need to find more of the same. But my basic point is there were six anomalies of this object that I detail in my book, Extraterrestrial, and I also wrote about in Scientific American. And these six anomalies make it very unusual. If you assign a probability of 1% to the object having each of these anomalies, when you multiply them, you get the probability of one in a trillion that this object is something that we have seen before. So clearly, it’s very different from what we’ve seen before. And response from the scientific community was to dismiss the artificial origin. And there were some scientists that took the scientific process more seriously and tried to explain the origin of  from a natural source. And they suggested four possibilities after my paper came out.

And one of them was maybe it’s a hydrogen iceberg, a chunk of frozen hydrogen that we’ve never seen before by the way. And then the idea is that when hydrogen evaporates, you don’t see the cometary tail because it’s transparent. The problem with that idea is that hydrogen evaporates very easily. So, we showed in a follow-up paper that such a chunk of frozen hydrogen the size of a football field would not survive the journey through interstellar space from its birth site to the solar system. And then there was another suggestion, maybe it’s a nitrogen iceberg that was chipped off the surface of a planet like Pluto. And then we showed in a follow-up paper that in fact you need more mass in heavy elements than you find in all the stars in the Milky Way galaxy by orders of magnitude more just to have a large enough population of nitrogen icy objects in space to explain the discovery of ‘Oumuamua.

And the reason is that there is a very thin layer of nitrogen, solid nitrogen on the surface of Pluto. And that makes a small fraction of the mass budget of the solar system. And so you just cannot imagine making enough chunks, even if you rip off all the nitrogen on the surface of exo-Plutos. It just doesn’t work out this scenario. And then there was a suggestion, maybe it’s a dust bunny as you mentioned it, a cloud of dust particles very loosely bound. And it needs to be a hundred times less dense than air so that when reflecting sunlight, it will be pushed like a feather. And the problem with that idea is that such a cloud would get heated by hundreds of degrees when it gets close to the sun and they would not maintain its integrity. So, that also has a problem.

And the final suggestion was maybe it’s a fragment, a shrapnel from a bigger object that pass close to a star. And the problem with that is the chance of passing close to a star is very small, most objects do not. So, why should we see the first interstellar object is belonging to that category? And the second is when you tidally disrupt a big object when passing through nearest star, the fragments usually get elongated and not pancake-shaped. You get often a cigar-shaped object. So, all of these suggestions have major flows. And my argument was simple. If it’s nothing like we have seen before, we better leave on the table the possibility that it’s artificial. And then, take a photograph of future objects that appears weirdest as this one.

Lucas Perry: So you mentioned the local standard of rest, which is the average velocity of our local group of stars. Is that right?

Avi Loeb: Yes. Well, it’s the frame that you get to after you average over the motions of all the stars relative to the sun, yes.

Lucas Perry: Okay. And so ‘Oumuamua was at the local standard of rest until the sun’s gravitation pulled it in, is that right?

Avi Loeb: Well, no. So the way to think of it, it was sitting at rest in that frame and just like buoy on the surface of the ocean. And then the sun happened to bump into it, the sun simply intercepted it along. And as a result, gave it a kick just like a ship gives a kick to a buoy. The sun acted on it through its gravitational force primarily. And then in addition, there was this excess push which was a smaller fraction of the gravitational force, just a fraction of a percent.

Lucas Perry: Right. And that’s the sun pushing on it through its suspected large surface area and structure.

Avi Loeb: Yeah. So in addition to gravity, there was an extra force acting on it, which was a small correction to the force of gravity, the other 10%. But it’s still, it was detected at very high significance because we monitored the motion of ‘Oumuamua. And to explain this force given that there was no cometary evaporation, you needed a thin object. And as I said, there was another thin object discovered in September 2020 called , that also exhibited an excess push by reflecting sunlight. So, it doesn’t mean necessarily that ‘Oumuamua was a light sail. It just means that it had the large area for its mass.

Lucas Perry: Can you explain why the smooth acceleration of ‘Oumuamua is significant?

Avi Loeb: Yeah. So what we detected is an excess acceleration away from the sun that declines inversely with distance squared in a smooth fashion. And first of all, the inverse-square law is indicative of a force that acts on the surface of the object. And the reflection of sunlight is exactly giving you that. And the fact that it’s smooth cannot be easily mimicked by cometary evaporation because often you had jets. These are spots on the surface of a comet from where the evaporation takes off. And that introduces jitter as the object tumbles, there is a jitter introduced to its motion because of the localized nature of these jets that are pushing it. You can think of the jets as the jets in a plane that push the airplane forward by ejecting gas backwards. But in the case of a comet, the comet is also tumbling and spinning.

And so, that introduces some jitter because the jets are exposed to sunlight at different phases of the spin of the object. And moreover, beyond a certain distance, water does not sublimate, does not evaporate anymore. You have water ice on the surface and beyond a certain distance, it doesn’t get heated enough to evaporate. So the push that you get from cometary evaporation has a sharp cutoff beyond a certain distance, and that was not observed. In the case of ‘Oumuamua, there was a smooth push that didn’t really cut off, didn’t show an abrupt change at the distance where water ice would stop evaporate. And so, that again is consistent with the reflection of sunlight being the origin of the excess push.

Lucas Perry: Can you explain the difference between comets and asteroids?

Avi Loeb: Yeah. So, we’re talking about the bricks that were left over from the construction project of the solar system. So the way that the planets form is that first you make a star like the sun, and you make it from a cloud of gas that condenses and collapses under the influence of itself gravity, its own gravitational force contracted and it pulls, and makes a star in the middle. But some of the gas has rotation around the center. And so when you make a star like the sun, a small fraction of the gas of the other for few percent or so remains in the leftover disks around the star that was just formed. And that debris of gas in the disks is the birthplace of the planets. And that disks of gas that is leftover from the formation process of the sun of course includes hydrogen and helium, the main elements from which the sun is made, but also includes heavy elements.

And they condensed in the mid-plane of the disks and make dust particles that stick to each other, get bigger and bigger over time. And they make the so-called planetesimals. These are the building blocks, the bricks that come together in making planets like the earth or the core of Jupiter that created also hydrogen and helium around the central rocky region. So, the idea is that you have all these bricks that just like Lego pieces make up the planets. And some of them get scattered during the formation process of the planets and they remain as rocks in the outer solar system. So, the solar system actually extends a thousand times farther than the location of the most distant planet in a region called the  that extends to a 100,000 times the earth-sun separation. And that is a huge volume. It goes halfway to the nearest star.

So in fact, if you imagine each star having an Oort cloud of these bricks, these building blocks that were scattered out of the construction process of the planets around the star, then these Oort clouds are touching each other, just like densely packed billiard balls. So just imagine the spherical region of planetesimals, these rocks. And so, comets are those rocks that are covered with water ice. So since they’re so far away from the sun, the ice freezes, the water freezes on their surface. But some of them have orbits that bring them very close to the sun. So when they get close to the sun, the water ice evaporates and creates a cloud of gas, a water vapor and some dust that was embedded in this rock that creates this appearance of a cometary tail. So what you see is the object is moving and then its surface layers get heated up by absorbing sunlight and the gas and dust evaporate and create this halo around the object and a tail, but always points away from the sun because it’s calmed by the solar wind, the wind coming from the sun.

And so you end up with a cometary tail, that’s what the comet is. Now, some rocks remain closer to the sun and are not covered with ice whatsoever. So, they’re just bare rocks. And when they get close to the sun, there is no ice that evaporates from them. These are called asteroids. And they’re just rock without any ice on the surface. And so, we see those as well. There is actually a region where asteroids, it’s called the main belt of asteroids, that’s we don’t know what the origin of that is. It could be a planet that was disintegrated, or it could be a region that didn’t quite make a planet and you ended up with fragments floating there. But at any event, there are asteroids, bare rocks without ice on them because they were close enough to the sun that the ice evaporated and we don’t have the water there.

And then these objects are also seen in the vicinity of the earth every now and then, these are called asteroids. And we see basically two populations. Now, ‘Oumuamua was not a comet because we haven’t seen a cometary tail around it. And it wasn’t an asteroid because there was this excess push. If you have a piece of rock, it will not be pushed much by reflecting sunlight because it’s area is not big enough relative to its mass. So it gets a push, but it’s too small for it to exhibit it in its trajectory.

Lucas Perry: Right. So, can you also explain how much we know about the composition of Oort clouds and specifically the shape and size of the kinds of objects there? And how ‘Oumuamua relates to our expectation of what exists in the Oort cloud of different stars?

Avi Loeb: Yeah. So, the one thing that I should point upfront is when scientists that try to attend to the anomalies of ‘Oumuamua suggests that it’s a hydrogen iceberg or a nitrogen iceberg. By the way, that notion gathered popularity in the mainstream. People said, oh, they had a sigh of relief. We can explain this object with something we know. But the truth is, it’s not something we know. We’ve never seen a nitrogen iceberg that was chipped off Pluto in our solar system. The Oort cloud does not have nitrogen icebergs that we witnessed. So claiming that ‘Oumuamua, the first interstellar object is a nitrogen iceberg or a hydrogen iceberg implies that there are nurseries out there around other stars or in molecular clouds that are completely different than the solar system in the sense that they produce most of the interstellar objects because ‘Oumuamua was the first one we discovered.

So they produce a large fraction of the interstellar objects, yet they are completely different from the solar system. It’s just like going to the hospital and seeing a baby that looks completely different than any child you have seen before. It’s your home from any child you had. And it implies that the birthplace of that child was quite different, but yet that child appears to be the first one you see. So, that’s to me an important signal from nature that you have to rethink what the meaning of this discovery is. And the other message is we will learn something new no matter what, so we need to get more data on the next object that belongs to this family. Because even if it’s a naturally produced object, it will teach us about environments that produce objects that are quite different from the ones we find in the solar system.

And that means that we miss something about the nature. And even if it’s natural in origin, we learn something really new in the process of gathering this data. So we should not dismiss this object and say, business as usual, we don’t have to worry about it, rather we should attempt to collect as much data as possible on the next weird object that comes along. I should say there was a second interstellar object discovered by an amateur astronomer from Russia that called Gennady Borisov. And it was given the name Borisov discovered in 2019. That one looked just like a comet. And I was asked, does that convince you that ‘Oumuamua was also natural because this one looks exactly like the comets we have seen? And I reclined, when you go along the beach and most of the time you see rocks and suddenly you see a plastic bottle. And after that you see rocks, the fact that you found rocks afterwards doesn’t make the plastic bottle a rock.

Each object has to be considered on its own merit. And therefore, it makes ‘Oumuamua even more unusual. The fact that we see Borisov as a natural comet. So in terms of the object that come from the Oort cloud, our own Oort cloud, there is a size distribution that there are objects that are much smaller than ‘Oumuamua and objects that are much bigger. And of course, the bigger objects are more rare. And then roughly speaking, there is equal amount of mass per logarithmic size bin. So, there are many more small objects. And most of them we can’t see because ‘Oumuamua was roughly at the limit of our sensitivity with Pan-STARRS. And that means that objects much smaller than the size of a football field cannot be noticed within a distance comparable to the distance to the sun. The sun acts as a lamppost that illuminates the darkness around us.

And so, an object is detected when it reflects enough sunlight for us to detect with our telescopes. And so small objects do not reflect enough sunlight, and we will notice them. But I calculated that in fact, if there are probes moving very fast through the solar system, let’s say at a fraction of the speed of light that were sent by some alien civilizations, we could detect the emission from them, the infrared emission from them with the James Webb Space Telescope. They would move very fast across our sky, so we just need to be ready to detect them.

Lucas Perry: Do you think given our limited knowledge of Oort clouds that there are perhaps exotic objects or rare objects, which we haven’t encountered yet, but that are natural in origin that may account for ‘Oumuamua?

Avi Loeb: Of course, there could be. As I mentioned, there are people suggested the hydrogen iceberg and nitrogen iceberg, dust bunny. These were suggestions that were already made and each of them has its own challenges. And it could be something else, of course. And the way to find out, that’s the way science operates. The science is guided by evidence by collecting data. And the way science should be done is you leave all possibilities on the table and then you collect enough data to rule out all but one interpretation that looks most plausible. And so, my argument is we should leave the artificial origin possibility on the table, because all the other possibilities that were contemplated invoke something that we’ve never seen before. So, we cannot argue based on speculations that it’s something that we’ve never seen before. We cannot argue that proves the point that it’s not artificial. So, it’s a very simple point that I’m making, and I’m arguing for collecting more data. I mean, I would be happy to be proven wrong, but it’s not artificial in origin, and then move on. The point is that science is not done by having a prejudice, knowing the answer in advance. It’s done by collecting data, and the mistake that was made by the philosophers during Galileo’s time is not to look through his telescope and argue that they know that the sun moves around the Earth. And that only maintained their ignorance.

The reality doesn’t care whether we ignore it. The Earth continued to move around the sun. If we have neighbors that exists out there, and it doesn’t really matter whether we shut down the curtains on our windows and claim, “No, we’re unique and special, and there is nobody out there on the street.” The fact that we say that, we can get a lot of likes on Twitter saying that, and then we can ridicule anyone that argues differently, but that would not change the fact whether we have neighbors or not. That’s an empirical fact. And, in order for us to improve our knowledge of reality, I’m talking about reality, not about philosophical arguments, just figuring out whether we have neighbors, whether we are the smartest kid on the block, that’s within the realm of science, and finding out the answer to this question is not a matter of debate.

It’s a matter of collecting evidence. But of course, if you are not willing to find wonderful things, you will never discover them. So, my point is, we should consider this possibility as real, as very plausible, as mainstream activity, just like the search for dark matter or the search for gravitational waves. We exist. There are many planets out there just like the Earth. Therefore, we should search for things like us that existed or exist on them. That’s a very simple assumption to make, an argument to make, and to me, it sounds like this should be a mainstream activity. But then, I realize that my colleagues do not agree, and I failed to understand this dismissal, because it’s a subject of great interest to the public, and the public fund science. So, if you go back a thousand years, there were people saying the human body has a soul, and therefore anatomy should be forbidden.

So imagine if scientists would say, “Oh, this is a controversial subject. The human body could have a soul. We don’t want to deal with that, because some people are claiming that we should not operate the human body,” where would modern medicine be? My argument is, if science has the tool to address the subject of great interest to the public, we have an obligation to address it and clear it up. Let’s do it bravely, with open eyes. And by the way, there is an added bonus. If the public cares about it, there will be funding for it. So, how is it possible that the scientific community ridicules this subject, brushes it aside, claims, “We don’t want to entertain this unless we have extraordinary evidence,” yet fails to fund at a very substantial level the search for that extraordinary evidence? How is that possible in the 21st century?

Lucas Perry: So, given the evidence and data that we do have, what is your credence that ‘Oumuamua is alien in origin?

Avi Loeb: Well, I have no certainty in that possibility, but I say, it’s a possibility that should be put left on the table, with at least as high likelihood as a nitrogen iceberg or a hydrogen iceberg or a dust bunny. That’s what I consider as the competing interpretations. I don’t consider statements like, “It’s always rocks. It’s never aliens,” as valid scientific statements, because they remind me of the possibility. If you were to present a cell phone to a caveman, and the caveman is used to playing with rocks all of his life, the caveman would argue that the cell phone is just a shiny rock. And, just basing your assertions on past experience is no different than what the philosophers were arguing. We don’t want to look through Galileo’s telescope because we know that the sun moves around the Earth. So, this mistake was made over and over again, throughout human history. I would expect modern scientists to be more open-minded to thinking outside the box, to entertain possibilities that are straightforward.

And what I find is, the strange thing is not so much that there is conservatism regarding this subject. But at the same time, in theoretical particle physics, you have whole communities of hundreds of people entertaining ideas that have no experimental verification, no experimental tests in the foreseeable future whatsoever, ideas like the string theory landscape or the multiverse. Or some people argue we live in a simulation, or other people talk about supersymmetry. And awards were given to people doing mathematical gymnastics, and these studies are part of the mainstream. And I ask myself, “How is it possible that this is considered part of the mainstream and the search for technological signatures is not?” And my answer is, that these ideas provide a sandbox for people to demonstrate that they’re smart, that they are clever, and a lot of the country, the academia is about that. It’s not about understanding nature. It’s more about showing that you’re smart and getting honors and awards. And that’s unfortunate, because physics and science is a dialogue with nature. It’s a learning experience. We’re supposed to listen to nature. And the best way to listen to nature is to look at anomalies, things that do not quite line up with what we expected. And by the way, whether Oumuamua is artificial or not, that doesn’t require very fancy math. It’s a very simple fact that any person can understand. I mean, nature is under no obligation to reveal its most exciting secrets without fancy math. It doesn’t need to be sophisticated.

Aristotle had this idea of the spheres surrounding us, that we are at the center of the universe, and there are these beautiful spheres around us. That was a very sophisticated idea that many people liked, because it flattered their ego to be at the center of the universe, and it also had this very clever arrangement. But it was wrong. So, who cares how sophisticated an idea is? Who cares if the math is extremely complicated? I mean, of course, it demonstrates that you are smart if you’re able to maneuver through these complicated mathematical gymnastics. But that doesn’t mean that it’s reflecting reality. And my point is, we better pay attention to anomalies that nature gives us, than to promoting our image.

Lucas Perry: Right. So it seems like there’s this interesting difference between the extent to which the scientific community is willing to entertain ‘Oumuamua as being artificial and origin, whereas at the same time, there is a ton of theories that, at least at the moment, are unfalsifiable. Yet, here we have a theory that is simple, matches the data, and can be falsified.

Avi Loeb: Right. And the way to falsify it, I mean, it’s not by chasing ‘Oumuamua, because by now, it’s a million times fainter than it was close to the sun. But then, it’s by finding more objects that look as weird as it was. And this was the first object we identified. There must be many more. If we found this object over by serving the sky for a few years, we will definitely find more by serving the sky for a few more years, because of the Copernican principle. Copernicus discovered that we are not positioned in a special location, in a privileged location in the universe. We’re not at the center of the universe, and you can extend it also, not just space, but also time. And, when you make an observation over a few years time, the chance of these few years being special and privileged is small.

I mean, most likely, it’s a typical time, and you would find it if you were to look at the previous three years, so then the following three years… That’s the Copernican principle and I very much subscribe to it, because again, the one thing I learned from practicing astronomy over the decades was a sense of modesty. We are not special. We are not unique. We are not located at the center of the universe. We don’t have anything special in our backyard. The Earth-sun system is very common. So, that’s the message that nature gives us. And, we are born into the world like actors put on a stage. And first thing we see is the stage is huge. It’s 10 to the power 26 times larger than our body. And the second thing we see is that the play has been going on for 13.8 billion years since the big bang, and we just arrived at the end of it.

So, the play is not about us. We are not the main actors. So let’s get a sense of modesty, and let’s look for other actors that may have been around for longer than we did. There’s a technological civilization. Maybe they have a better sense of what the play is about. So, I think it all starts from a sense of modesty. My daughters, when they were young, they were at home and they had the impression that they are the center of the world, that they are the smartest, because they haven’t met anyone else outside the family. And then, when we took them to the kindergarten, they got the better sense of reality by meeting others and realizing that they’re not necessarily the smartest kid on the block. And so, I think our civilization has yet to mature, and the best way to do that is by meeting others.

Lucas Perry: So before we move on to meeting others, I’m curious if you’re willing to offer a specific credence. So, you said that there are these other natural theories, like the dust bunny and the iceberg theories. If we think of this in terms of Bayesian reasoning, what kind of probability would you assign to the alien hypothesis?

Avi Loeb: Well, the point is that these objects that were postulated for natural origin of ‘Oumuamua were never seen before. So, there is no way of assigning likelihood to something that we’ve never seen before. And it needs to be the most common object in interstellar space. So, what I would say is that, we should approach it without a Bayesian prior. Basically, we should leave all of these possibilities on the table, and then get as much data as possible on the next object that shows the same qualities as ‘Oumuamua. By these qualities, I mean, not having a cometary tail, so not being a comet, and showing an excess push away from the sun.

And as I mentioned, there was such an object, 2020 SO, but it was produced by us. So, we should just look for more objects that come from interstellar space that exhibit these properties, and see what the data tells us. It’s not a matter of a philosophical debate. That’s my point. We just need a close up photograph, and we can easily tell the difference between a rock and an artificial object. And I would argue that anyone on Earth should be convinced when we have such a photograph. So, if we can get such a photograph in the next few years, I would be delighted, even if I’m proven wrong, because we will learn something new no matter what.

Lucas Perry: So, there’s also been a lot of energy in the news around UFO sightings and UFO reports recently. I’m curious how the current news and status of UFO interest in the United States and the world, how that affects your credence of ‘Oumuamua being alien in origin, and if you have any perspective or thoughts on UFOs.

Avi Loeb: Yeah, it’s a completely independent set of facts that is underlying the discussion on UFOs. But of course, again, it’s the facts, the evidence that we need to pay attention to. I always say, “Let’s keep our eyes on the ball, not on the audience.” Because if you look at the audience, the scientists are responding to these UFO reports in exactly the same way as they responded to ‘Oumuamua. They dismiss it. They ridicule it. And, that’s unfortunate, because the scientists should ask, “Who do we have access to the data? Could we analyze the data? Could we see the full data? Or could we collect new data on these objects, so that we can clear up the mystery?” I mean, science is about evidence. It’s not about prejudice. But instead, the scientists know the answer in advance. They say, “Oh, these reports are just related to human-made objects, and that’s it.”

Now, let’s follow the logic of Sherlock Holmes. Basically, Sherlock Holmes, as I mentioned in my book Extraterrestrial, Sherlock Holmes made the statement that you put all possibilities on the table, and then, whatever remains after you sought out all the facts must be the truth. That’s the way he operated as a detective. So, that’s the way we should operate as scientists. And what do we know about the latest UFO report, from the Pentagon and Intelligence agencies? So far, a few weeks before it’s being released, we know from leaks that there is a statement that some of the objects that were found are real. Okay? They are not artifacts of the cameras. They are not illusions of the people who saw them, because they were detected by multiple instruments, including infrared cameras, radar systems, optical cameras, and a lot of people from different angles.

And, when you consider that statement coming from the Pentagon, you have to take it seriously, because it’s just the tip of the iceberg. That the data that will be released to the public, presumably, is partial, because they will never released the high quality data, because it will inform other nations of the capabilities, the kind of sensors that the US has in monitoring the sky. Okay? So, I have no doubt that a lot of data is being hidden for national security reasons, because otherwise, it will expose the capabilities of these sensors that are being routinely used to monitor the sky. But, if people that had access to the full data, and that includes officials such as former president Barack Obama, former CIA director James Woolsey and others, that saw the data, and they make the case that these objects are real, then these objects may very well be real.

Okay? And I take that at face value. Of course, as a scientist, I would like to see the full data, or collect new data. There is no difference, because science is about reproducibility of results. So, if the data is classified, I would much rather place state-of-the-art cameras that you can buy in the commercial sector, or scientific instrumentation that we can purchase, and just place those in the same locations and record the sky. The sky is not classified. In principle, anyone could collect data about the sky. So, I would argue that, if all data is classified, we should collect new data that would be open to the public. And it’s not a huge investment of funds to have such an experiment. But the point of the matter is, that we can infer if the objects are real using the scientific method, then let’s assume that they are real, like the people that saw the full data claim.

So, if they’re real, then there are three possibilities. Either they were produced, manufactured by other nations, because we certainly know what we are doing, the US. So, if they were produced by other nations, like China or Russia, then humans have the ability to produce such objects, and they cannot exceed the limits of our technology. And, if the maneuvering of these objects look as if they exceed, substantially, the limits of the technologies we possess, then we would argue it’s not made by humans, because there is no way that the secret about an advanced technology would be preserved on Earth by humans. And because it has huge benefits commercially, so it would appear in the market, in the commercial sector because you can sell it for a lot of money, or it would appear in the battlefield, if it’s being used by other nations.

And we pretty much know what humans are capable of producing. We are also probably getting intelligence on other nations. So, we know what are the limits of human technology. I don’t think we can leave that possibility vague. If there is an object behaving in a way that far exceeds what we are able to produce, then that looks quite intriguing. But the remaining possibilities are, that somehow it’s a phenomenon that occurs in the Earth atmosphere. There is something that happens that we didn’t identify before, or that these are objects that came from an extraterrestrial origin. Okay? And, once again, I make the case that, the way to make progress on this is not to appear on Twitter and claim we know the answer in advance and ridicule the other side of the argument. This is not the way by which we make progress, but rather collect better evidence, better clues and figure it out, clear up the fog.

It’s not the mystery that should be unraveled by philosophical arguments. It’s something that you can measure and get data on and reproduce with future experiments. And once we get that, we will have a clear view of what it means. And then, that’s how mysteries get resolved in science. So, I would argue, for a scientific experiment that will clear up the fog. And the way that we would not do that is if the scientific community would ridicule these reports, and the public would speculate about the possible interpretations. That’s the worst situation you can be in, because you’re basically leaving a subject of great interest to the public unresolved.

And, that’s not the right way. Again, in the 21st century, to treat the subject of interest to the public, that obviously reaches the Congress, it’s not an eyewitness on the street that says, “I saw something unusual.” It’s military personnel. We have to take it seriously, and we have to get to the bottom of it. So that’s the way I look at it. Then, it may well be that it’s not the extraterrestrial in origin, but I think the key is by finding evidence.

Lucas Perry: So, given the age of the universe and the age of our galaxy and the age of our solar system, would you be surprised if there were alien artifacts almost everywhere or in many places, but we were just really bad at finding them? Or those artifacts were really good at hiding?

Avi Loeb: No, I wouldn’t be surprised, because as I said, most of the stars formed billions of years before the sun. And, if there were technological civilizations around them, many of these stars died by now and these civilizations may have perished, but if they send equipment, that equipment may operate, especially if it’s being operated by artificial intelligence or by things that we haven’t invented yet. It may well survive billions of years and get to our environment. Now, one thing you have to realize is, when you go in the wilderness, you better be quiet. You better not make a sound, and listen, because there may be predators out there. Now, we have not been careful in that sense, because we have been broadcasting radio waves for more than a century. So, these radio signals reached a hundred light years by now.

And, if there is another advanced civilization out there with radio telescopes of the type that we possess, they may already know about us. And then, if they use chemical rockets to get back to us, it would take them a million years to traverse a hundred light years. But if they use much faster propulsion, they may be already here. And the question is, are we noticing them? There was this Fermi paradox, formulated 70 years ago by Enrico Fermi, a famous physicist, who said that, “Where is everybody?” And of course, that’s a presumptuous statement, because it assumes that we are sufficiently interesting for them to come and visit us. And, when I met my wife, she had a lot of friends that were waiting for prince charming on a white horse to make them a marriage proposal, and that never happened, and then they compromise.

We, as a civilization, would be presumptuous in assuming that we are sufficiently interesting for others to have a party in our backyard. But nevertheless, it could be that it already happened. As you said, that we didn’t notice. One thing to keep in mind is full geological activity. Most of the surface of the Earth gets mixed with the interior of the Earth, over a hundred million years time scales. So, it could be that some of the evidence was buried by the geological activity on Earth, and that’s why we don’t see it.

But the moon, for example, is like a museum, because it doesn’t have geological activity, and also, it doesn’t have an atmosphere that would burn up an object that is smaller than the size of a person, like the Earth’s atmosphere does, say, for meteors. So in principle, once we establish a sustainable base on the moon, we can regard it as an archeological site, and survey the surface of the moon to look for artifacts that may have landed, may have crashed on it. Maybe we will find a piece of equipment that we never sent, that came from somewhere else that crashed on the surface of the moon.

Lucas Perry: So, it’d be wonderful if we could pivot into Great Filters and space archeology here, but before we do that, you’re talking about the Fermi paradox and whether or not we’re sufficiently interesting to merit the attention of other alien civilizations. I wonder if interesting is really the right criteria, because if advanced civilizations converge on some form of ethics or beneficence, then whether or not we’re interesting is not perhaps the right criteria for whether or not they would reach out. We have people on earth who are interested in animal ethics, like how the ants and bees and other animals are doing. So, it could be the same case with aliens, right?

Avi Loeb: Right. I completely agree. One thing I should say… Well, actually, two things is that, first, that you mentioned before the Drake’s equation. It doesn’t apply to relics. It doesn’t apply to objects. And the Drake equation talks about the likelihood of detecting radio signals. And, that has been the method we used over the past 70 years in searching for other civilizations. And, I think it’s misguided, because in order to get a signal, it’s just like trying to have a phone conversation. You need the counterpart to be alive. And it’s quite possible that most of the civilizations are dead by now. So, that’s the Great Filter idea that there is a narrow window of opportunity for us to communicate with them. But, on the other hand, they may have sent equipment into space, and we can search for it through space archeology, and find relics from civilizations that are not around anymore, just like we find relics from cultures that existed on the surface of Earth through archeological digs.

So I think a much more promising approach to find evidence for dead civilizations is looking for objects floating in space. And, the calculation of what’s the likelihood of finding them, is completely different from the Drake equation. It resembles more the calculation of what’s the chance that you would have stumbled across a plastic bottle on the beach or on the surface of the ocean. And, you just need to know how many plastic bottles are per unit area on the surface of the ocean, and then you will know what’s the likelihood of crossing one of them. And, the same is true for relics in space. You just need to know the number of such objects per unit volume, and then you will figure out what’s your chance of bumping into one of them.

And that’s a completely different calculation than the Drake equation, which talks about receiving radio signals. This is one point that should be born. And the other point that I would like to mention is that, during our childhood, we always have a sense of adults looking over our shoulders, and then making sure that everything goes well, and they often protect us. And then, as we become independent and grow up, we encounter reality on our own. There is this longing for a higher power that overlooks our shoulder. And, that is provided by the idea of God in a religion. But interestingly enough, it’s also related to the idea of some unidentified flying objects that are looking over our shoulders, because if a UFO was identified to be of extraterrestrial origin, it may imply that there is an adult wiser than we are in the room, looking over our shoulder. The question of whether that adult is trying to protect us is still open, remains open, but we can be optimistic.

Lucas Perry: All right. So, let’s talk a little bit about whether or not there might be adults in the room. So, you defined what Great Filter was. So, when I think of Great Filters, I think of there being potentially many of them, rather than a single Great Filter. So, there’s the birth of the universe, and then you need generations of stars to fuse heavier elements. And then there’s the number of planets and Goldilocks zones. And then there’s abiogenesis or the arising of life on Earth. And then there’s moving from single to multicellular life. And then there’s intelligent life and civilization, et cetera. Right? So, it seems like there’s a lot of different places where there could be Great Filters. Could you explain your perspective on where you think the most likely Great Filters might be?

Avi Loeb: Well, I think it’s self destruction, because I was asked by Harvard Alumni, how much longer do I expect our civilization to survive? And I said, “When you look at your life, and you just select a random day throughout your life, what’s the chance that it’s the first day after you are born. That probability is tens of thousands of times smaller, than the probability that the day you select would be during your adulthood, because there are tens of thousands of days in the life of an adult.” So, we existed for about a century as an advanced technological civilization. And you ask yourself, “Okay. Well, if we are in our adulthood, which is the most probable state for us to be in?” As I mentioned before, we’re just sampling randomly a time, and most likely during your adulthood, then that means that we have only a few more centuries left, because the likelihood that we will survive for millions of years, is tens of thousands of times smaller.

It would imply that we are in the first day of our life. And that is unlikely. Now, the one caveat I have for this statement is, that the human spirit can defy all odds. So, I believe that in principle, if we get our act together, we can be an outlier, in the statistical likelihood function. And, that’s my hope. I’m an optimist. And I hope that we will get our act together. But if we continue to behave the way we are, not to care so much about the climate. You can even see it in world politics nowadays. Even when you have administrations that care about climate, they cannot really convince the commercial sector to cooperate. And, suppose our civilization is on a path to self destruction, then we don’t have more than a few centuries left. So, that is a Great Filter. And of course, there could be many other Great Filters, but that seems to me as the most serious one.

And, then you ask yourself, “Okay, so which civilization is more likely to survive?” It’s probably the dumber civilization that doesn’t create the technologies that destroy it. If you have a bunch of crocodiles swimming on the surface of a planet, they will not create an atomic weapon. They would not change the climate. So, they may survive for billions of years. Who knows? So maybe the most common civilizations are the dumb ones. But, one thing to keep in mind is that, when you create technological capabilities, you can create equipment that will reproduce itself, like Von Neumann machines, or you can send it to space. You can escape from the location.

Or you can send it to space. You can escape from the location that you were born on. And so that opens up a whole range of opportunities in space. And that’s why I say that once a civilization ventures into space, then everything is possible. Then you can fill up space with equipment that reproduces itself. And there could be a lot of plastic bottles out there. And we don’t know. We shouldn’t assume anything. We should just search for them. And ‘Oumuamua, as far as I’m concerned, was the wake up call. And the other thing I would like to say is if I imagine a very advanced civilization that understands how to unify quantum mechanics with gravity, something we don’t possess at the moment… there’s such a unification scheme that we know works… perhaps they know how to irritate the vacuum and create a baby universe that would lead to more civilizations.

So it’s just like having a baby that can make babies that can make babies, and you would get many generations as a result of that. So this could be an origin of the Big Bang. Maybe the umbilical cord of the Big Bang started in a laboratory. And by the way, it would say that intelligence, technological advance is an approximation to God because in the religious stories, God created the universe. We can imagine a technology that would create a baby universe. And then the same is true for life. We don’t know if life was seeded, the origins of life was seeded in a laboratory somewhere. And so that remains a possibility. And that’s what’s so fascinating about the search for intelligent life out there, because it may provide answers to the most fundamental questions we have, like the meaning of life.

Lucas Perry: Would you consider your argument there about human extinction? Given what we are currently observing, is that like the doomsday argument?

Avi Loeb: Yeah. Well, you can call it the doomsday. I would call it risk assessment. And then I don’t think we are statistical systems in the sense that there is no escape from a particular future, because I think that once we recognize the risk in a particular future, we can respond and avoid it. The only question is whether as a civilization, we will be intelligent enough. And frankly, I’m worried that we are not intelligent enough. And it may be just like a Darwinian principle where if you are not intelligent enough, you will not survive and we will never be admitted to the club of intelligent civilizations in the Milky Way Galaxy unless we change our behavior. And it’s yet to be been whether we will change our behavior accordingly. One way to convince people to change their behavior is to find evidence for other civilizations that didn’t, and perished as a result. That would be a warning for us, a history lesson.

Now, one caveat I should mention is we always imagined things like us. And when we go to meet someone, it’s a fair assumption to assume that that person has eyes and nose and ears the way we have. And the reason it’s a reasonable assumption is because we share the same genetic heritage as the person that we are meeting. But if you think about life on a planet that had no causal contact with Earth, it could be very different.

And so calculating the likelihood of self-destruction, the likelihood of life of one form versus another, the likelihood of intelligence, all of these very often assume something similar to us, which may not be the case. I think it might be shocking to us to find the creatures from another planet or technologies from another planet. And so my solution to this ambiguity is to be an observer. Even though I’m a theorist, I would argue, let’s be modest. Let’s not try to predict things in this context. Let’s just explore the universe. And the biggest mistake we are making over and over again is to argue about the answer before seeing the evidence. And that’s the biggest mistake because it convinces you to be lazy, not to collect more evidence to say, “I know the answer in advance. I don’t need to look through the telescope. I don’t need to invest funds in searching for this. Even though it’s an important question, I know the answer in advance.” And that’s the biggest mistake we can make as a species.

I’m willing to go through all the hardships of arguing something outside the box of confronting these personal attacks against me just because it’s a question of such great importance to humanity. If that was a minor question about the nature of dark matter, I would not risk anything for that. Who cares? If the dark matter is axions or weakly interacting massive particles, that has very little impact on our daily lives. It’s not worth confronting the mainstream on that. And by the way, the response would not be so emotional in that case either. But on a subject as important as this one to the future of humanity, which is the title of your organization, there is no doubt in my mind that it’s worth the extra effort.

It’s worth the hardship, bringing people to recognize that such a search for technological relics in space is extremely important for the way we view ourselves in the big scheme of things, our aspirations for space, our notions about religion, and what we might do in response to the knowledge that we acquire will completely change the future of humanity. And on such a question, I’m willing to put my body on the barbed wire.

Lucas Perry: Well, thank you very much for putting your body on the barbed wire. I think you mentioned that there was something in… Was it Israeli training where soldiers are taught to put their body on the barbed wire so people can climb over them?

Avi Loeb: Yeah. That was a statement that in the battlefield, very often, a soldier is asked to put his body on the barbed wire so that others can pass through. The way I see it historically is you look at Socrates, the ancient Greek philosopher. He advocated for doubting the wisdom of influential politicians at the time and other important figures, and he was blamed for corrupting the youth by dismissing the gods that were valued by the civilians of the city-state of Athens at the time. And he was prosecuted and then forced to drink poison. Now, if Socrates would have lived today, he would have been canceled on the Athenian social media. That would be the equivalent of the poison. And then you see another philosopher, Epicurus, that made many true statements, but again, was disliked by some religious authorities at the time. And you see, of course, Galileo Galilei that was put in house arrest.

Later on, you see Giordano Bruno. I mean, he was an obnoxious person that was not liked by a lot of people, but he simply argued that other stars are just like the sun, and therefore, they might have a planet just like the Earth that could have life on it. And the church at the time found it offensive because if there is life that is intelligent out there, then that life may have sinned, and then Christ could have saved that life. And then you need billions of copies of Christ to be distributed throughout the galaxy to visit all these planets. And that makes little sense. That made little sense to the church. And so they burned Giordano Bruno on a stake. And even though nowadays we know that indeed, a lot of stars are like the sun, a lot of planets are just like the Earth at roughly the same separation from their host stars where life may exist [inaudible 01:12:03]. So in that sense, he was correct.

And obviously you find many such examples also in modern science over the past century, of people advocating for the correct ideas and being dismissed and ridiculed. Just to give you an example, a former chair of the astronomy department at Harvard that preceded me… I chaired the astronomy department for nine years. I was the longest-serving chair in the history of the astronomy department at Harvard. Before me was Cecilia Payne-Gaposchkin. And in her PhD thesis, which was the first thesis in astronomy at Harvard, she argued based on analyzing the spectrum of the sun that most of the surface of the sun is made of hydrogen. And while defending her PhD thesis, Henry Norris Russell, who was the director of the Princeton University Observatory, an authority on stars at that time, dismissed her idea and said, “That is ridiculous because we know that the sun is made of the same elements as the Earth. So there is not much hydrogen on Earth. It cannot be the case that the sun is made mostly of hydrogen.”

So she took out that conclusion from her PhD. And then in the subsequent few years, he redid the analysis, got more data, and wrote an extended paper, Industrial Physical Journal, arguing the same, that she was correct. And interestingly enough, in a visiting committee to the Princeton University Department of Astrophysics, the chair of that department was bragging that Henry Norris Russell discovered that the sun is made mostly of hydrogen. So you can see that history depends pretty much from who tells it. But the point of the matter is that sometimes, when you propose an idea that even though it has to be correct because it’s based on evidence, it’s being dismissed by the authorities, and science is not dictated by authority.

In the 1930s, there was a book co-authored by tens of scientists arguing that Einstein’s Theory of Relativity must be wrong. And when Einstein was asked about it, he said, “Why do you need tens of scientists to prove that my theory is wrong? It’s enough to have one author that would explain why the theory is wrong.” Science is not based on authority. It’s based on reasoning and on evidence. And there is a lot of bullying going on nowadays. And I witness it. And throughout my career, I’ve seen a number of ideas that I proposed that were dismissed and ridiculed at first. And then they became the interest of mainstream. And now, there are hundreds of people working on them. That was true for my work on the first stars. I remember that it was dismissed early on. There were people claiming even that there are no stars beyond the redshift [inaudible 01:14:59]. And then I worked on imaging black holes. I suggested that there could be a correlation between black or mass and characteristic velocity dispersion of stars in the vicinity of those supermassive black holes at the centers of galaxies.

I worked on gravitational wave astrophysics long before it was fashionable. And in all of these cases, the interest that I had early on was ridiculed. I gave a lecture in a winter school in 2013, in January 2013, winter school in Jerusalem, on gravitational wave astrophysics. And one of the other lecturers, who still is 20 years younger than I am, stood up and said, “Why are you wasting the time with these young students on a subject that will not be of importance in their career?” And he said it publicly. He stood up in front of everyone it’s on video. And two and a half years later, the LIGO experiment detected the first gravitational wave signal.

Many of these students were still doing their PhD, and this became the hottest frontier in astrophysics in subsequent years, and the Nobel prize was awarded. So here you have a situation where someone says, “Why are you giving a lecture on this subject to students? Because it would never be of importance through their careers.” And two and a half years later, it becomes the hottest topic, the hottest frontier in astrophysics. And it involves a new messenger other than light that was never used before in astrophysics. Gravitational waves, wrinkles in space and time. It opens up a whole new window into the universe. So how is it possible that someone that is 20 times younger than I am stands up, feels that it’s completely appropriate for him to stand up in front of all the students and say that?

And to me, it illustrates narrow-mindedness. It’s not a matter of conservatism. It’s a matter of thinking within the box and not allowing to think outside the box. And that, you might say, okay, it’s acceptable because there are lots of people suggesting crazy ideas. But at the same time, you have whole communities of theoretical physicists working on very strange ideas that were not verified experimentally. And that is part of the mainstream. And the common threads between these two communities of people is that they both don’t pay attention to evidence. They both do not recognize the fact that evidence leads the way. In the case of gravitational waves, it’s the fact that we detect the signal. So just wait for LIGO to find the signal, and then everything will change.

The case of ʻOumuamua, we saw some anomalies. Let’s pay attention to them. Let’s talk about them. And in the case of String Theory, it’s let’s say this shouldn’t be at the fringes of mainstream because we haven’t found evidence that supports the idea of extra dimensions as of yet. So it doesn’t deserve to be center stage. But you have these two communities living side to side because both of them feel comfortable not paying attention to evidence.

Lucas Perry: We like to think of science as this really clean epistemic process of hypothesis-generating and creating theories, and then verification and falsification through evidence and data-gathering. But the reality is that it’s still made up of lots of humans who have their own need for recognition and meaning and acceptance and validation. And so in order to improve the process of science, it’s probably helpful to bring light to the reality of the humanity that we all still have when we’re engaged in the scientific pursuit. And that helps to open our minds to the truth when our pursuit of the truth is not being obscured by things we’re not being honest with ourselves about.

Avi Loeb: Right. And I was the founding director of the Black Hole Initiative at Harvard University, which brings together physicists, mathematicians, astronomers, and philosophers. And my motivation in creating this center was to bring people from different perspectives so that they will open the minds of other disciplines to possible breakthroughs in the context of black holes. And I think this is key. I think we should be open-minded and we should also fund risky propositions, risky ideas. There should be a certain fraction of the funding that goes in those directions. And even though I founded this Black Hole Initiative, in the first annual conference that we had, a philosopher gave a lecture, and then at the end of the lecture, the philosopher argued that… After speaking to a lot of string theories, he made this statement that if a bunch of physicists agree on something as being true for a decade, then it must be true because physics is what physicists decide to do.

And I raised my hand. I said, “How can you make… No, I would expect philosophers to give us a litmus test of honesty.” It’s just like the canary in the cave. They should tell us when truth is not being spoken. And I just couldn’t understand how a philosopher could make such a statement. I said, “There are many examples in history where physicists agreed on something and it was completely wrong. And the only way for us to find out is by experimental evidence.” Nature is teaching us. It’s a learning experience. And we can all agree that we are the wealthiest people in the world. And if we go to an ATM machine, that’s equivalent to doing an experiment and testing that idea. Now we can feel happy until we try to cash the money out of the ATM machine, and then we realized that our ideas were wrong.

If someone mentions an idea, how do we tell whether it’s a Ponzi scheme or not? Bernie Madoff told a lot of people that if they give him their money, he would give them more in return, irrespective of what the stock market will do. Now, that was a beautiful idea. It appealed to a lot of people. They gave him their money. What else can you expect from people that believe a beautiful idea? They made money and gave it to Bernie Madoff because the idea was so beautiful. And he felt great about it. They felt great about it. But when they wanted to cash out, which was the experiment, he couldn’t provide them the money. So this idea turned out to be wrong.

And it’s not just the nuance of science to say, “Oh, okay. The recent experimental tests, but we can give up on this as long as we’re happy and we feel very smart and we completely agree that we should pursue these questions and just do mathematical gymnastics and give each other awards and feel great about life, and in general, just make the general statement that experiments will be great, but we can’t do them right now. And therefore, let’s not even discuss them.”

Having a culture of this type is unhealthy for science because how can you tell the difference between the idea of Bernie Madoff and reality? You can feel very happy until you try to cash it out. And if you don’t have an experimental test during your life, then you might spend your life as a physicist on an idea that doesn’t really describe reality. And that’s a risk that as a physicist, I’m not willing to take. I want to spend my life on ideas that I can test. And if they are wrong, I learn something new.

And by the way, Einstein was wrong three times in the last decade of his career. He argued that black holes don’t exist, gravitational waves don’t exist, and quantum mechanics doesn’t have spooky action at a distance. But that was part of his work at the frontiers of physics. You can be wrong. There’s nothing bad about it. When you explore new territories, you don’t always know if you’re heading in the right direction. As long as you’re doing it with dignity and honesty and integrity and you’re just following what is known at the time, it’s part of the scientific pursuit. And that’s why people should not ridicule others that think outside the box. As long as they’re doing it honestly, and as long as the evidence allows for what they’re talking about, that should be considered seriously.

And I think it’s really important for the health of the scientific endeavor because we’re missing on opportunities to discover new things. Just to give you an example, in 1952, there was an astronomer named Otto Struve that argued that we might find planets close in to a star like the sun, that if they have the mass of Jupiter, because if they’re close in, if they’re hot Jupiters, heated by the sun, they’re getting very close to the sun, then they would tag the sun like star back and forth in a way that we can measure, or they would a occlude significant portion of the area of the star. So we can see them when they transit the star. So he argued let’s search for those. And for four decades, no time on major facilities was allocated for such a search because astronomers argued, “Oh, we pretty much understand why Jupiter formed so far away from the sun, and we shouldn’t expect hot Jupiters.” And then in 1995, a hot Jupiter was discovered. And the Nobel Prize was given for that a couple of years ago.

So you might say, “Okay, that baby was born.” Eventually, even though four decades were wasted, eventually, we found a hot Jupiter. And that opened up the field of exoplanets. But my argument is that this is a baby that was born. For each baby like that, there must be many babies that were never born because it’s still being argued that it’s not worth the effort to pursue those frontiers.

And that’s unfortunate, because we are missing opportunities to discover new things. If you’re not open to discover new things, you will never discover them.

Lucas Perry: I think that’s some great wisdom for many different parts of life. One thing that you mentioned earlier that really caught my attention was you were talking about us becoming technologically advanced, and that would unlock replicators, and that replicators could explore the universe and fundamentally change it and life in our local galactic cluster. That was also tied into the search for the meaning for life. And a place where I see these two ideas as intersecting is in the idea of the cosmic endowment. The cosmic endowment is this idea of the total amount of matter and energy that an intelligent species has access to after it begins creating replicators. So since the expansion of the universe is accelerating, there’s some number of galaxies which exist outside of a volume that we have access to. So there’s a limited amount of energy and matter that we can use for whatever the meaning of life is or whatever good is. So what do you think the cosmic endowment should be used for?

Avi Loeb: Right. So I actually had an exchange with Freeman Dyson on this question. When the accelerating universe was discovered, I wrote a paper saying, “When the universe ages by a factor of 10, we will be surrounded by vacuum beyond our galaxy, and we will not have contact with other civilizations with resources.” And he wrote back to me and said, “We should engage in a cosmic engineering project where we propel our star and come together with other civilizations. And by that, we will not be left alone.” And I told him, “Look, this cosmic engineering project is very ambitious. It’s not practical. In fact, there are locations where you have much more resources, 1,000 thousand times more than in our Milky Way Galaxy. These are called clusters of galaxies, and we can migrate to the center of the nearest cluster of galaxy. And in fact, there might be a lot of journeys taken by advanced civilizations towards clusters of galaxies that would avoid the cosmic expansion.”

So that’s my answer of how to prepare for the cold winter that awaits us, where we will be surrounded by vacuum. It’s best to go to the nearest cluster of galaxy, where the amount of resources is 1,000 times larger. In addition to that, you can imagine that then in the future, we will build the accelerators that bring particles to energies that far exceed the large Hadron Collider. And the maximum particle energy that we can imagine is so-called Planck energy scale. And if you imagine developing our accelerator techniques, you can, in principle, imagine building an accelerator within the solar system that will reach Planck energies. And if you collect particles at these energies, we don’t really know the physics of quantum gravity, but you can imagine a situation where you would irritate the vacuum to a level where the vacuum will start burning up. Because we know the vacuum has some mass density, some energy density that is causing the accelerated expansions, the so-called cosmological constant.

And if you bring the vacuum to zero energy density state, then you have an excess energy that is just like a burning front. It’s the energy you get from a propellant that burns. And you get a domain wall that can expand and consume all the vacuum energy along its path. And of course, it moves at the speed of light. So if you were to be on the path of such a domain wall, you would not get an advanced warning and it will burn up everything along its path at the speed of light.

So I think if we ever meet advanced civilizations that have the capabilities of building accelerators that reach the Planck scale, we should sign a treaty, a galactic treaty, whereby we will never collide particles approaching that energy in order not to risk everyone else from domain walls that would burn them up. That’s just the matter of cosmic responsibility.

Lucas Perry: I think Max Tegmark calls these “death bubbles”?

Avi Loeb: Yeah. I mean, these are domain walls that, of course, we have no evidence for, but they could be triggered by collisions at the Planck scale. And a matter of cosmic responsibility is not to generate these domain walls artificially.

Lucas Perry: So let’s pivot into looking for life and space archeology, which is a cool term that you’ve created, and looking for them through bio-signatures and techno-signatures. One place that I’m curious to start here is since we were just talking about replicators, why is it that we don’t find evidence of replicators or large-scale super structures in other galaxies or in our own galaxy? For example, a galaxy where half of it has been turned into Dyson spheres. And so it’s like half illuminated.

Avi Loeb: Right. I mean, presumably such things do not exist. It’s actually very difficult to imagine an engineering project that will construct a Dyson sphere. And I think it’s much more prudent for an advanced civilization to build small pieces of equipment that go through the vast space in between stars. And that is actually very difficult for us to detect with existing instrumentation. Even a spacecraft as big as a football field would be noticed only when it passes within the Earth’s orbit around the sun. That’s the only region where Pan-STARRS detected objects the size of ʻOumuamua, from the reflected sunlight. So we will notice such objects the farther than the Earth is from the sun. And the distance to the nearest star is hundreds of thousands of times bigger than that. So most of space could be filled with things passing through it that are not visible to us.

A spacecraft the size of a football field is huge. We cannot imagine something much bigger than that. And so I would argue that there could be a lot of things floating through space. Also, as of now our telescopes were not monitoring for objects that move very fast, a fraction of the speed of light, obviously astronomers saw something moving across the sky so fast, they would dismiss it. They would say, “It makes no sense. We are looking for asteroids or comets that are moving at a percent of a percent of the speed of flight, 10 to the minus four of the speed of light.” So part of it is our inability to consider possibilities that may exist out there. But most of the fact that we haven’t yet detected a lot of these objects is a lack of sensitivity. We can’t really see these things when they’re far away unless there are major megastructures, as you pointed out. But I think such engineering projects are unlikely.

Lucas Perry: I’m curious why you feel that engineering projects like that are unlikely. It seems like one of the most interesting things you can do is computation. Computation seems like it has something to do with creating consciousness, and consciousness seems like it is the bedrock of value given that all value arises in conscious experience. I would imagine using the energy of suns to enable vast amounts of computation is one of the most interesting things that a civilization can do. And the objects that they might send out to other solar systems would be a nanoscale, right? You send out nano scale replicators. They would be even smaller than football fields or smaller than Amomum. Then, those would begin Dyson sphere engineering projects. With artificial super intelligence and billions and billions of years to go in the universe, in some sense it feels like we’re in the early universe. It feels curious to me why superstructures would be unlikely. I’m not sure I totally understand that.

Avi Loeb: If you think about what the star is, a star is just a nuclear reactor that is bound by gravity. That doesn’t seem to be like the optimal system for us to use. It’s better to build an artificial nuclear reactor that is not bound by gravity like nuclear engine, nuclear reactor. We’re trying to do that. It’s not easy to build a fusion reactor on earth, but we do have a fission reactor. If I were to think about using nuclear energy, I would say it’s much better to use artificially-made nuclear engines than to use the energy produced by a giant nuclear reactor that nature produced in particular locations. Because then you can carry your engine with you. You are always close to it. You can harness all of its energy and you don’t need to put a huge structure around the star, which brings in a lot of engineering difficulties or challenges.

I would be leaning in the direction of having small systems, a lot of small systems sent out rather than a giant system that covers the star. But once again, I would argue that, we should look at the evidence and there are constraints on Dyson’s spheres that imply that they are not very common. I should say a couple of weeks ago, I wrote a paper with an undergraduate student in Stanford, Eliza Tabor, that considers the possibility of detecting artificial lights on the night side of Proxima B: the habitable planet around the nearest star, Proxima Centauri using the James Webb Space Telescope. We show that one can put very interesting limits on the level of artificial illumination on the dark side of that planet if there are any CT lights out there.

The other technological signatures that one can look for are, for example, industrial pollution in the atmosphere planet. I wrote the paper about that six years ago. You can look for reflectance that indicates photovoltaic cells on the day side of a planet, which is quite different than the reflectance of rock as a spectral edge. You can look for light beams that sweep across the sky. You see them as a flash of light. For example, the light being used for propulsion using light sails. If you imagine in other planetary system where cargoes are being delivered from an Earth-like planet to a Mars-like planet using light sails, the beam of light could cross our line of sight and we could see it as a flash of light, and we can have even correlate it with the two planets passing a longer line of sight. That would give us confidence that indeed it’s a light sail traveling between those two planets that we are witnessing. I wrote a paper about that in 2016.

There are all kinds of technological signatures we can search for but we need to search for it and we need to put funds towards this.

Lucas Perry: We have both bio-signatures and techno-signatures. In terms of bio-signatures, you’ve proposed looking in the clouds of brown dwarfs and green dwarfs. There is looking around our own solar system through looking at the elements in, for example, the atmosphere of Venus, there was phosphene which we thought could not exist except their biological pathways. So, it’s hypothesized that maybe there’s some kind of life in the atmosphere of Venus. They’re searching other planets for elements that can’t exist without life. Then, in terms of techno-signatures, they are searching for radio waves which you’ve talked about. That is a primary way of looking for life, but it potentially needing a refresh where it is, for example, looking for artificial light or the remnants of industry. You’ve also proposed increasing the threshold of sensitivities for developing imaging that is increasingly sensitive. Because ‘Oumuamua was that basically, was it at the limit of our telescopes’ capacity? 

Avi Loeb: It was roughly. I mean, it was at the level of sensitivity that allows us definite detection, but we can’t see objects that are much smaller than that or reflect much less light than ‘Oumuamua did. I should say that all of these, both biological signatures and technological signatures, are being reviewed in a textbook that I wrote together with my former post doc Manasvi Lingam that is coming out to be published on the 29th of June, 2021. It’s more than a thousand pages long. It’s 1,061 pages long and it has an overview of the current scientific knowledge we have and the expectations we have for biological signatures and technological signatures. The title of the book is “Life in the Cosmos” and it’s to be published by Harvard University Press. It is meant to be a textbook for scientific research as a follow-up on my popular level book, Extraterrestrial.

Lucas Perry: Pivoting a bit here, do you feel that, and we mentioned this a little bit earlier when we were talking about the difference between aliens being interested in us or compelled to reach out to us because of ethical concerns. Do you think that advanced alien civilizations can converge on ethics and beneficence?

Avi Loeb: That’s an interesting question. It really depends on their value system. It also depends on Darwinian selection. The question is what kind of civilizations will be most abundant? If you look at human history, very often, the more aggressive, less ethical cultures survived because they were able to destroy the others. It’s not just a matter of which values appear to be more noble. It’s a question of which set of values leads to a survival in the long run and domination in the long run? Without knowing the full spectrum of possibilities, we can’t really assess that. Once again, I would say the smart thing for us to do is be careful. I mean, not transmit too much to the outside world until we figure out if we have neighbors. There was this joke when I Love Lucy was replayed the again and again, that we might get a message from another planet saying, “If you keep replaying reruns of I Love Lucy, we will invade you.”

I think, well, it’s important for us to be careful and figure out first whether there are smarter kids on the block. But having said that, if we ever establish contact or if you find the equipment in our neighborhood, the question is what to do? It’s a policy question how to respond to that and it really depends on the nature of what we find. How much more advanced is the equipment that we uncover? What were the intentions of those who produced it and sent it? These are fundamental questions that will guide our policy and our behavior. Until we find conclusive evidence, we should wait until that moment.

Lucas Perry: To push back a little bit on the Darwinian argument that’s, of course, a factor where we have this kind of game theoretic expression of genes, the selfish gene trying to propagate itself through generations and that leading to behaviors and how the human being is conditioned by evolution in that way. There’s also the sense that over time humanity has become increasingly moral. We’re, of course, doing plenty of things right now that are wrong, but morality seems to be improving over time. This leads to a question where, for example, do you think that there is a necessary relationship between what is true and what is good? You need to know more and more true facts in order to, for example, spread throughout the universe. So, if there’s a necessary relationship between what is true and what is good, there would be a convergence then also on what is good as truth continues to progress.

Avi Loeb: I was asked in a forum when I joked about the fact that a man seeking intelligence in space in the sky, because I don’t find it often here on earth, a member of the audience chuckled and asked me: how do you define an intelligent civilization? The way I define it is by the guiding principles of science, which is sharing or cooperation on evidence-based knowledge. The word cooperation is extremely important. I believe that intelligence is marked by cooperation, not by fighting each other because that’s a sink for our energy, for our resources, that doesn’t do any good. Promoting a better future for ourselves through cooperation is a trademark of intelligence. It’s also the guiding principle of science.

The second component of these guiding principles is evidence-based knowledge. The way I view science is it’s an infinite sum game. In economics, you have a zero sum game where if someone makes a profit, another person loses. In science, when we increase the level of knowledge we have, everyone benefit. When a vaccine was developed for COVID-19, everyone on earth benefited from it. Science aims to increase the territory of this island of knowledge that we have in the ocean of ignorance that surrounds it. It should be evidence-based, not based on our prejudice. That’s what I hope the future of humanity is. It will be coincident with the guiding principles of science, meaning, people will cooperate with each other, nations will cooperate with each other and try to share evidence-based knowledge rather than what’s the alternative.

The alternative is what we are doing right now: fighting each other, trying to feel superior relative to each other. If you look at human history, you find racism, you find attempts to feel supremacy or elitism, or all kinds of phenomena that stem from a desire to feel superior relative to other people. That’s ridiculous in the big scheme of things, because we are such an unimportant player in the cosmic stage that we should all feel modest, not trying to feel superior relative to each other. Because any advantage that we have relative to each other is really minuscule in the big scheme of things. Now, the color of the skin is completely meaningless. Who cares what the color of the skin is? What significance could that have for the qualities of a person?

Yet, a lot of human history is shaped around that. This is not the intelligent way for us to behave as a species. We should focus on the guiding principles of science which are cooperation and sharing of evidence-based knowledge. Rather than ridiculing each other, rather than trying to feel superior relative to each other, rather than fighting each other, let’s work together towards a better future and demonstrate that we are intelligent so that we will acquire a place in the club of intelligent species in the Milky Way galaxy.

Lucas Perry: Do you see morality as evidence-based knowledge?

Avi Loeb: I think morality, if you listen to Kant, it’s the logical thing to do if you consider a principle such that it will promote the better good of everyone around. You’re basically taking into consideration others and shaping your behavior so that if other people follow the same principles, we will be in a better world. That to me is a sign of recognizing evidence because the evidence is that you don’t live alone. If you are to live alone, if you are the only person on earth, morality loses significance. Not only that there is nobody else for you to consider morality relative to. That’s not the issue. The issue is that it’s irrelevant. You don’t need to consider morality because you’re the only person. You can do whatever you want. It has no effect on other people, therefore, morality is not relevant. You can do whatever you want. But given the fact that you look at the evidence and you realize that you’re not alone, that’s evidence. You shape your behavior based on that evidence, and I do think that’s evidence-based knowledge. Definitely.

Lucas Perry: How do you see axiomatic based knowledge? For example, axioms of morality and mathematics that build these structures, they’re also axioms, for example, of science, like this value of communication and evidence-based reasoning. Axioms and morality are, for example, might be the value and disvalue are innate and intrinsically experienced in consciousness. Then, there are axioms in mathematics which motivate and structure that field. We’ve talked a lot about science and evidence-based reasoning, but what about knowledge in the philosophical territory which is almost a priori true, like things which we rest fields upon? How do you see that?

Avi Loeb: I do believe that there is room for humanities of the future. The way that philosophy was handled in past centuries should be updated. Let me illustrate that with an example related to your question. Basically, suppose we want to decide about the principles of morality, the way to do that is you can construct a simulation that includes a lot of people. In principle, if you include all the ingredients that make people behave one way or another. It doesn’t need to be rational reasoning. You can include some randomness or some other elements that shape human behavior based on their environment. You can include that in the simulation.

Let’s just imagine this simulation where you put individual people and you have an algorithm for the way that they respond to their environment. It doesn’t need to be by rational reasoning. It could be emotional, it could be any other way that you find appropriate. You have the building blocks. Each of them is a person and you introduce the randomness that is in the population. Then, you run the simulation and you see what happens. This is just like trying to produce human history artificially. Then, you introduce principles for the behavior of people, guiding principles, just like moral principles.

First you let people behave in a completely crazy way, like, anything they want then you will get killed as the outcome of this simulation. But if you introduce principles of morality, you can see the outcomes that will come out of it. What I would say is in principle, in the future, if we have a sophisticated enough computer algorithm to describe behavior of people based on our understanding of how people behave, if we get the better sense of how people behave and respond to their environment, we can design the optimal code by which people should behave such that we will end up in a stable society that is intelligent, that follows the kind of principles I mentioned before that is orderly and that benefits everyone for a better future.

That’s one way of approaching it. Obviously in the past, philosophers could not approach it this way because they didn’t have the computer capabilities that we currently have. You can imagine artificial intelligence addressing this task in principle.

Lucas Perry: You can set moral principles and moral parameters for a system and then evolve the system, but the criteria for evaluating the success or not of that system and those are more like moral axioms. As a scientist, I’m curious about how you approach, for example, moral axioms that you use for evaluating the evolution of a particular moral system.

Avi Loeb: My criterion, the one that I think that guides me is maintaining the longevity of the human species. Whatever will keep us for the longest amount of time. Of course, bearing in mind that the physical conditions will change on earth. Within a billion years, the sun will boil off all the oceans on earth, but let’s leave that aside. Let’s just ask, suppose you put the people in a box and, generation after generation, let them follow some principles. What would be the ideal principles to maintain the stability of society and the longevity of the human species? That’s what will guide me. I think survival is really the key for maintaining your ideas. That’s the precondition. In nature, things that are transient, they go away. They don’t survive and they lose their value so they have less value. I mean, obviously in the short term, they could have more value, but I care about the long-term and I define the principles based on how long they would allow us to survive.

Lucas Perry: But would you add expected value to that calculation? It’s not just time, but it’s actually like the expected reward or expected value over time. Because some futures are worse than others and so maybe we wouldn’t want to just have longevity.

Avi Loeb: There is the issue of being happy and pleased with the environment that you live in. That could be factored in. But I think the primary principle would be survival because within any population you always will find a fraction of the components that are happy. It partly depends on the circumstances that they live in, but partly on the way they accept those circumstances. You can live in the barn and be happy. You can be in a mansion and be unhappy. It’s complicated as to what makes you happy and I would put that as a secondary condition. I would worry more about social structures that maintain longevity.

Lucas Perry: All right. On humanity’s longevity, we’re basically beginning to become technologically advanced, we’re facing existential risks in the 21st century from artificial intelligence and nuclear weapons and synthetic biology. There’s UFO’s and there’s ‘Oumuamua and a lot of really interesting, crazy things are going on. I’m curious if you could touch on the challenge of humanity’s response and the need for international governance for potentially communicating and encountering alien life.

Avi Loeb: Well, I do think it’s extremely important for us to recognize that we belong to the same species. All the confrontations we often have in politics, between nations, they should play a lesser role in guiding our behavior. Cooperation on the global scale, international cooperation, is extremely important. Let me give an example from recent history. There was a virus that came from Wuhan, China. If scientists were allowed to gather all the information of how this virus came and what the characteristics of this virus are, then the vaccine would have been developed earlier and it could have saved the lives of many people.

I would say, in the global world that we live in today, many of our problems are global and therefore we should cooperate on the solutions. That argues against putting borders in our knowledge, trying, again, to gain superiority of one nation relative to another, but instead help each other towards a better future. It’s really the science that provides the glue that can bind us internationally. I realized, I’m trying to be a realist, that it may not happen anytime soon that people will recognize the value of science as the international glue. But I hope that eventually we will realize that this is the only path that will bring us to survival, to a better future, if we act based on cooperation on evidence-based knowledge.

Lucas Perry: In 2020, you have an article where you advocate for creating an elite scientific body to advise on global catastrophes. In the Future of Life Institute we’re interested in reducing the risks of existential risks, ways in which technology can be misused or lead to accidents which lead to the extinction of life on earth. Could you comment on your perspective on the need for an elite scientific body to advise on existential risk and global catastrophic risks?

Avi Loeb: Well, we noticed that during the pandemic, we were not really prepared especially in the Western world because the last major pandemic of this magnitude took place a century ago, and nobody around today in politics or otherwise was around back then. As a result, we were not ready. We were not prepared. I think it’s prudent to have an organization that will cultivate cooperation globally. It could be established by the United Nations. It could be a different body. But once again, it’s important for us to plan ahead and avoid catastrophes that could be more damaging than COVID-19. If you prevent them, it would more than overpay for the investment of funds.

Just to give you another example, solar eruptions, solar storms, if there was a Carrington Event. About 150 years ago, it was big eruption on the sun that brought energetic particles to earth and back in the mid-19th century, there wasn’t much technological infrastructure. But if the same event would have happened today, it would cost trillions of dollars to the world economy because it would damage power grids and satellites, communication, and so forth. It would be extremely expensive. It’s important for us to plan ahead. About seven years ago, there was a plume of hot gas that was ejected by the sun and it just missed the earth. We should be ready for that and build infrastructure that would protect us for such a catastrophe.

There are many more. One can go through the risks and some of them are bigger than others. Some of them are rarer than others. Of course, one of them is the risk from an asteroid hitting the earth and the Congress that tasked NASA to find all asteroids or rocks bigger than the size of ‘Oumuamua, about 140 meters. They wanted NASA to find the 90% of all of those that could potentially intercept earth and collide with earth. The Pan-STARRS telescope that we started from, that discovered Oumuamua, was funded for finding such near earth objects. The Vera Rubin Observatory will most likely fulfill two-thirds of the Congressional task and find 60% of all the near earth asteroids bigger than 140 meters.

That shows that the human brain is actually much more useful for survival than the body of a dinosaur because the dinosaurs had huge bodies. 66 million years ago, they were very proud of themselves. They dominated their environment, they ate grass, and were happy. Then, from the sky came this giant rock the size of Manhattan Island. When it hit the ground, it tarnished their ego trip abruptly. Just to show you that the human brain, even though it’s much smaller than the dinosaur body is much more precious for protecting us because we can design telescopes that would alert us to incoming objects. That’s a catastrophe that obviously we can protect ourselves against by shifting the trajectories of objects heading our way.

Lucas Perry: As a final question, I’m curious, what are the most fundamental questions to you in life and what motivates and excites you from moment to moment as a human being on earth? I’ve read or heard that you were really interested in existentialism as a kid. What are the most foundational or important questions to you?

Avi Loeb: The fundamental issue is that we live for a finite time, with short time. The question is what’s the meaning of our existence? You see, because very often we forget that this trip that is very exciting that we’re having and could be very stimulating and intriguing, is finite. When I realized that, when both my parents passed away over the past three years, I came to the realization that I can give a damn of what other people think. Let’s focus on the substance. Let’s keep our eyes on the ball and not on the audience. Then, it was the focusing of my attention to the important things in life that we should appreciate.

Then, there is this fundamental question of why is life worth living? What are we living life for? What is the meaning of our life? You know, it may well be that there is no meaning, that we just go through this interesting trip; that we are spectators of the universe. We should enjoy the play while it lasts. But that, again, argues that we should be modest and behave like spectators rather than trying to shape our immediate environment and feel a sense of deep arrogance as a result of that. That was the view of the dinosaurs before the rock hit them. In a way, what gives me a sense of a meaningful life is just looking at the universe and learning from it. I don’t really care about my colleagues.

Every morning I jog at 5:00 AM. I developed this routine during the pandemic. I enjoy the company of birds, ducks, wild turkeys, and rabbits. I really enjoy nature left to its own much more than people because there is something true in looking at it. Every morning, I see something different. Today, I saw a red bird. I saw the sunrise was completely different than yesterday. Everyday, you can learn new things and we just need to pay attention, not to feel that you know everything. It’s not about us. It’s about what is going on around us that we should pay attention to. Once we behave more like kids appreciating things around us and learning from them, then we would feel happier. I was asked by the Harvard Gazette: what is the one thing I would like to change about the world? I said, I would like my colleagues to behave more like kids, basically not being driven by promoting their image but rather willing to make mistakes, putting skin in the game, and learning regarding life as a learning experience. We might be wrong sometimes, but we are doing our best to figure out when we are wrong.

Lucas Perry: All right, Avi. Thank you very much for inspiring this childlike curiosity in science, for also helping to improve the cultural and epistemic situation in science, and also for your work on ‘Oumuamua and everything to do with extraterrestrials and astronomy. Thank you very much for coming on the podcast.

Avi Loeb: Thanks for having me. I had a great time.

• Negative and positive outcomes from AI in the short, medium, and long-terms
• The perils and promises of AGI and superintelligence
• AI alignment and AI existential risk
• Lethal autonomous weapons
• AI governance and racing to powerful AI systems
• AI consciousness

Lucas Perry: Welcome to the Future of Life Institute Podcast. I’m Lucas Perry. Today’s episode is with Bart Selman and explores his views on possible negative and positive futures with AI. The importance of AI alignment and safety research in computer science, facets of national and international AI governance, lethal autonomous weapons, AI alignment, and safety at the association for the advancement of artificial intelligence and a little bit on AI consciousness.

Bart Selman is a Professor of Computer Science at Cornell University, and previously worked AT&T Bell Laboratories. He is a co-founder of the Center for Human-Compatible AI, and is currently the President of the Association for the Advancement of Artificial Intelligence. He is the author of over 90 publications and has a special focus on computational and representational issues. Professor Selman has worked on tractable inference, knowledge representation, stochastic search methods, theory approximation knowledge compilation, planning, default reasoning, and the connections between computer science and statistical physics.

And so without further ado, let’s get into our conversation with Bart Selman.

So to start things off here, I’m curious if you can share with us an example of a or a few futures that you’re really excited about and an example of a or a few futures that you’re quite nervous about or which you fear most.

Bart Selman: Okay. Yeah. Thank you. Thank you for having me. So just let me start with an example of a future in the context of AI that I’m excited about is the new capabilities that AI brings it should have the potential to make a life for everyone much easier and much more pleasant. I see AI as complementing our cognitive capabilities. So I can envision household robots or smart robots that assist people living in their houses, living independently longer, including doing kinds of work that are sort of monotonous and not that exciting for humans to do. So, AI has the potential to compliment our capabilities and to hugely assist us in many ways, including in areas you might not have thought of. Like for example, policymaking and governance. So AI systems are very good at thinking in high dimensional terms, trade offs between many different factors.

For humans, it’s hard to actually think in a multi-dimensional trade-off. We tend to boil things down to one or two central points and argue about a trade off in one or two dimensions. Most policy decisions involve 10, 20 different criterias that may conflict, or be somewhat contradictory and exploring that space, AI can assist us. I mean, finding better policy solutions and better governance for everybody. So I think AI has this tremendous potential to improve life for all of us provided that we learned to share this capabilities that we have policies in place and mechanisms in place to make this a positive experience for humans. And i’ll have to draw a parallel, human labor, physical labor machines have freed us from heavy duty physical labor. AI systems can help us with a sort of monotonous cognitive labor or as I mentioned, household robots and other tools that will make our life much better. So that’s for the positive side. So should I continue with a negative?

Lucas Perry: So before we get into the negative, I’m curious if you could explain a little bit more specifically what these possible positive futures look like on different timescales. So you explained AI assisting with cognitive capabilities with monotonous jobs. And so, over the coming decades, it will begin to occupy some of these roles increasingly, but there’s also the medium term, the long term and the deep future in which the positive fruits of AI may come to bear.

Bart Selman: Yeah. So, that’s an excellent point. I think one thing that in any transition and as I say, these medium cognitive capabilities that will help us live better lives, it will also disrupt the labor force and the workforce. And that this is a process that I can see play out over the next five, 10, maybe 15 years, a significant change in workforce. And I am somewhat concerned about how that will be managed because basically I feel we are moving to our future where people would have more free time. We’d have more time to be creative, to travel and to live independently. But of course, everybody needs to have the resources to do that. So there is an important governance issue of making sure that in this transition to a world was more leisure time that we find ways of having everybody benefit from these new future.

And this is really I think, 5, 10, 15 years process that we’re faced now, and important, that is done right. Further out in the future, it’s a little my own view of AI is that, machines will excel at certain specific task as we’ve seen very much with AlphaGo, AlphaZero. So, very good at that specific tasks and those systems will come in first self-driving cars, specialized robots for assisting humans. So we’ll first get these specialized capabilities. Those are not yet general AI capabilities. That’s not AGI. So the AGI future, I think is more 20, 25 years away.

So we first have to find ways of dealing with incorporating these specialized capabilities, which are going to be exciting as a scientist. You know, I already see AI transforming the way we approach science and do scientific discovery and really complementing our ways. I hope people get excited in the areas of creativity, for example, in computers or AI system, bringing a new dimension to these type of human activities that will actually be exciting for people to be part of. And that’s an aspect that we started to see emerge, but people are not fully aware of yet.

Lucas Perry: So we have AI increasingly moving its way into specialized kind of narrow domains. And as it begins to proliferate into more and more of these areas, it’s displacing all of the traditional human solutions for these areas, which basically all just includes human labor. So there’s an increase in human leisure time. And then what really caught my attention was you said AGI maybe 20, 25 years away. Is that your sense of the timeline where you start to see real generality or?

Bart Selman: Yeah. That’s in my mind a reasonable sense of a timeline, but we cannot be absolutely certain about that. And it’s sort of, for AI researchers it is a very interesting time. The hardest thing at this point in the history of AI is to predict what AI can and cannot do. I’ve learned as a professor, never to say that deep learning can’t do something because every time it surprises me and it can do it a few years later. So, we have a certain sense that, oh! the field is moving so fast that everything can be done. On the other hand, in some of my research, I look at some of these advances and if I can give you a specific example. So, my own research is partly in planning, which is a process of how humans plan out activities.

They have certain goals, and then they plan, what steps should I take to achieve those goals? And that’s can be very long sequences of actions to achieve complicated goals. So we worked on sort of a puzzle style domain, which is called Sokoban. And most people will not be familiar with it but it’s a kind of a game where it’s modeled after workers in a warehouse that have to move around boxes. And so there is a little grid world and you push around the boxes to get them from certain initial state to goal states somewhere else on the grid. And there are walls, and there are corners and all kinds of things you have to avoid. And what’s amazing about the planning task is for traditional planning, this was really a very challenging domain. So we picked it because traditional planners could do maybe a hundred steps, a hundred pushes as we call them, but that was about it.

There were puzzles available on the web that required 1500 to 2000 steps. So it was beyond way beyond any automated program. And AI researchers had worked on this problem for decades. So we of course used reinforcement learning RL with specific some clever curriculum training, some clever forms of training. And suddenly we could solve these 2000 steps, 1500 steps, Sokoban puzzles. So, we were very calm. We’re still very excited about that capability. And then we started looking, what did the deep net actually know about the problem? And our biggest surprise there was that although the system had learned very subtle things, that humans that are beyond human capabilities, it also was totally ignorant about other things that were trivial for humans. So in the Sokoban puzzle you don’t want to push your box in a corner because once it’s in a corner, and you can’t get it out of a corner. This is something that a human player discovers in the first, I would say first minute of pushing some boxes around.

We realized, I guess the deep learning routine network never conceptualized the notion of a corner. So it would only learn about corners if it had seen something being pushed in a particular corner. And if it had never seen that corner being used or encountered, it would not realize it shouldn’t push the box in there. So we had to sort of, we realized that this did this deep net had a capability that is definitely super human, in terms of being able to solve these puzzles. But also holes in its knowledge of the world that were very surprising to us. And that’s, I think part of what makes AI at this time, very difficult to predict. Will these holes be filled in while we develop AI systems that also get these obvious things, right?

Or will AI be at this amazing level of performance, but do things in ways that are to us, like quite odd. And so I think there are hard challenges that we don’t quite know how to fill in, but because of the speed with which things are developing, it’s very hard to predict whether they will be solved in the next two years on the next, or it will take another 20 years. But I do want to stress, there are surprising things about the I call it “the ignorance of tje learn models” that surprised us humans. Yeah.

Lucas Perry: Right. There are ways in which models fail to integrate really rudimentary parts of the world into their understanding that lead to failure modes that even children don’t encounter.

Bart Selman: Yeah. It’s… So the problem when we as humans interact with AI systems or think about AI systems, we anthropomorphize. So we think that they do think similar to the way we do things, because that’s sort of how we look at complex systems, even animals are anthropomorphized. So, we think that things has to be done a way similar to our own thinking, but we’re discovering that they can do things very differently and leave out pieces of knowledge that are sort of trivial to us.

I have discussion with my students and I point that out and they’re always sort of even skeptical of my claim. And they say, “well, it should know that somewhere.” And we actually do experiments. They say, “no. Okay, if it never seen the box go in that corner, it will just put it in the corner next time.” And so they actually have to see it to believe it, because it sounds that how can you be the world’s best Sokoban solver, and not know what a human knows in the first minute, but that’s the surprise. And that it also makes the field exciting, but that bakes the challenges of super intelligence and general intelligence and the impacts of an AI safety, particularly challenging topic.

Lucas Perry: Right. So, predicting an actual timeline seems very difficult, but if we don’t go extinct, then do you see the creation of AGI and superintelligence as inevitable?

Bart Selman: I do believe so. Yes. I do believe so. I think the path I see is we will develop these specialized capabilities, but in more and more areas in almost all areas, and then how they start merging together and in systems that do a two or three or four, and then a thousand specialized tasks. And so generality they will emerge almost inevitably. My only hesitation is what could go wrong? Why might it not happen if there is some aspect of cognition that is really beyond our capabilities of modeling. But I think that is unlikely. I think one of the surprises in the deep net world and the neural network world is that, before the deep learning revolution, if you can call it that before it happened, a lot of people looked at artificial neural networks as being too simplistic compared to real neurons.

So, there was this sense that, yeah, these little artificial neural networks are nice models, but they’re way too simplistic to capture what goes on in the human brain. The big surprise was is that apparently that level of simplification is okay, that you can get the functionality of a much more complex, real neural network. You get that level of performance and complexity using much simpler units. So that sort of convinced me that yes, the digital approximations we make, simplifications we make, as long as we connect things in sufficiently complex networks, we get properties emerged that match our human brain capabilities. So that makes me think that at some point we will reach AGI. It’s just a little hard to say exactly when, and I think it may not matter that much exactly when we’ll have challenges, in terms of AI safety and value alignment that are already occurring today before we have AGI. So we have to deal with challenges right from the start, we don’t have to wait for AGI.

Lucas Perry: So in this future where we’ve realized AGI, so do you see superintelligence as coming weeks, months, or years after the invention of AGI. And then what is beautiful about these futures to you in which we have realized AGI and superintelligence.

Bart Selman: Yeah. So what’s exciting about these possible, I mean, there are certain risks and the superintelligence would go against humans. I don’t think that is inevitable. I think these systems, they will do things… They will show us aspects of intelligence that to us will look surprising, but will also be exciting. So some of my other work, we’d look at mathematical theorem improving. And we look at AI systems for approving new open conjectures in mathematics. The systems clearly do a very different kind of mathematics than humans do, are very different kinds of proofs, but it’s sort of exciting to see a system that can check a billion step proof in a few seconds and generate a billion steps proof in an hour, and realizing that we can prove something to be true mathematically.

So find a mathematical truth that is beyond the human brain. But since we’ve designed to program and we know how it works and we use the technology, it’s actually a fun way to compliment our own mathematical thinking. So that’s what I see as the positive sense in which superintelligence will actually be of interest to humans to have around as a compliment to us, assuming they will not turn on us. But I think that’s manageable. Yeah.

Lucas Perry: So how long do you see superintelligence arising after the invention of AGI, even though it’s all kind of vague and fuzzy, like what’s what…

Bart Selman: Yeah. How long… So I think when I think of superintelligence, I think of it more as superintelligent in certain domains. So I assume you are referring to superintelligence as superseding AGI.

Lucas Perry: What I mean is like vastly more intelligent than the sum of humanity.

Bart Selman: I think that’s a super interesting question. And I have discussed that I can see capability at vastly more intelligent in areas like mathematical discovery, scientific discovery, thinking about problems with multiple conflicting criteria that has to be weighted against each other. So a particular task I can see superintelligence being vastly more powerful than our own intelligence. On the other hand, there is also a question in what sense superintelligence would manifest itself. That is, if I had to draw an analogy is if you meet somebody who is way smarter than you are, and everybody now meets such a person I’ve met a few in my life, these people will impress you about certain things and gets you insight. That, “Oh, this is exciting.” But when you go to have dinner with them and have a good meal, and they’re just like regular people.

So, superintelligence doesn’t necessarily manifest itself in all aspects. It will be surprising as certain kinds of areas and tasks and insights, but it will not… I do not believe it will come out. I guess If I draw an analogy like you can’t, if you go for dinner with that’s bad for dogs, but if you go for dinner with a dog, you will fully dominate all the conversations and be sort of superintelligence compared to the dog.

I’m not sure it’s not clear to me that there is an entity that will dominate our intelligence in all aspects. So there will be lots of activities, lots of conversations, lots of things we can have as a superintelligent being, that are quite understandable, quite accessible to us. So the analogy that there will be an entity that dominates our intelligence uniformly, that I’m not convinced exists. And it sort of, that goes back to the question, what human intelligence is, and human intelligence is actually quite general. So there’s an interesting question. What is meant by superintelligence? How would we recognize it? How would it manifest itself?

Lucas Perry: When I think of superintelligence, I think of like a general intelligence that is more intelligent than the sum of humanity. And so part of that generality is its capability to run an emulation of like maybe 10,000 human minds within its own general intelligence. And so the human mind becomes a subset of the intelligence of the superintelligence. So in that way, it seems like it would dominate human intelligence in all domains.

Bart Selman: Yeah, what I’m trying to say is I can see that, and that’s sort of, if you would play a game of chess with such a superintelligence they would beat you if they would give you a… It would not be fun if they… If he would do some maths with you, the mathematics and then show you some proofs of Fermat’s Last Theorem, it will be trivial for the superintelligence. So I can see a lot of specific task and domains where the superintelligence would indeed run circles around you and around any human, but how would it manifest it? So, yeah, on these individual questions, but you have to have the right questions as it is, I guess what I struggle is a little bit, you have to have the right questions to show to superintelligence.

So, like for example. The question, what should we do about income inequality? So like a practical problem in the United States. Would a superintelligence necessarily have something superintelligent to say about that? And that’s not that clear to me because there might actually… That it’s a tough problem, but it may just be as tough for the superintelligence as it is for any human. So a superintelligent politician has solutions to all our problems suddenly, would it win every debate? I think interestingly, the answer’s probably no. There’re certain… So super intelligence manifests itself on tasks that require high level of intelligence, like problem solving task, mathematical domains, scientific domains games, but daily life and governance. It’s a little less clear to me. So in that sense, and that’s what I mean by you’re going to have dinner with a superintelligence, which you be just sitting there. I can’t say anything useful about income inequality, because the superintelligence will say much more better things about it. I’m not so sure.

Lucas Perry: Maybe you’ve both had a wine or two, and you you ask the superintelligence and you know, why is there some thing rather than nothing? Or like, what is the nature of moral value? And they’re just like…

Bart Selman: What’s the purpose of life. I’m not sure the superintelligence is going to get me a better answer to that. So, yeah.

Lucas Perry: And this is where philosophy and ethics and metaethics merges with computer science, right? Because it seems like you’re talking about, there are domains in which AI will become superintelligent. Many of these domains, the ones that you listed sounded very quantitative. Ones which involve kind of the scientific method and empiricism, not that these things are necessarily disconnected from ethics and philosophy, but if you’re just working with numbers with a given objective, then there’s no philosophy that really needs to be done, if the objective is given. But if you ask about how do we deal with income inequality, then the objective is not given. And so you do philosophy about what is the nature of right and wrong? What is good? What is valuable? What is the nature of identity and all of these kinds of things and how they relate to building a good world. So I’m curious, do you think that there are true or false answers to moral questions?

Bart Selman: Yeah, I think that there are clearly wrong answers in this. So, I think moral issues, I think that’s a spectrum to me and that we can probably, as humans agree on certain basic moral values, it’s also very human kind of topic. So I think, we can agree on basic moral values, but I think the hard part is we also see among people and among different cultures, incredible different views of moral value. So saying which one is right and which one is wrong, may actually be much harder than we would like it to be. So this comes back to the value alignment problem and not like these into discussions about it. It’s a very good research field and a very important research field. But the question always is, whose values? And, we now realize that even within a country, people have very different values that are actually hard to understand between different groups of people.

So there is a challenge. There might be uniquely human of other… So it feels like there should be universal truths in morality thinks about equality, for example, but I’m a little hesitant because I’m surprised about how much disagreement I see about these, what I would think are universal truths that somehow are not universal truths for all people. So that’s another complication. And again, if you tie that back to superintelligence, so a superintelligence is going to have some position on it, but it is going to be yes or no, but it may not agree with everybody. And there’s no superintelligent position on it in my mind. So that’s, that’s a whole area of AI and value alignment that is very challenging.

Lucas Perry: Right. So, it sounds like you think that you have some intuition that there are universal moral truths, but it’s conflicting to why there was so much disagreement across different persons. So I guess I’m curious about two things. The first is one thing that you’re excited about for the future and about positive outcomes from AGI. Is it worlds in which AGI and superintelligence can help assist with moral and philosophical issues like around how to resolve income inequality and truth around moral questions? And then the second part of the question is, do you think that superintelligences is created by other species across the universe? Do you think that they would naturally converge on certain ethics, whether those ethics be universal truths or whether they be relative game theoretic expressions of how intelligence can propagate in the universe.

Bart Selman: Yeah, so two very good questions. So, as the first one, I am quite excited about the idea that a superhuman level of intelligence or an extreme level intelligence will help us better understand moral judgments and decisions and issues of ethics. I almost feel that humans are a little stuck in this debate. And a lot has to do, I think, with an inability to explain clearly to each other, why certain values matter and other values should be viewed differently, that it’s often even a matter of, can we explain to each other what are good moral judgments and good moral positions? So, I have some hope that AI systems, smart AI system would be better at actually sorting out some of these questions, and then convincing everybody, because in the end, we have to agree on these things. And perhaps these systems will help us find more common ground.

So that’s a hope I have for AI systems that truly understand our world, and are truly capable of understanding, because part of the alpha super smart AI would be understanding many different positions, and maybe something that limits humans in getting agreements on ethical questions, is that we actually have trouble understanding the perspective of another person that has a conflicting position. So superintelligence might be one way of modeling everybody’s mind, and then, being able to bring a consensus about … I have an optimistic view of, there may be some real possibilities there for superintelligence. Your second question of whether some alien form of superintelligence would come to the same basic ethical values as we may come to? That’s possible. I think it’s very hard to, yeah.

Lucas Perry: Yeah, sorry, whether those are ultimate truths, as in facts, or whether they’re just relative game theoretic expressions of how agents compete and cooperate in a universe of limited resources.

Bart Selman: Yes, yes. From a human perspective, you would hope there are some universal shared ethical perspective, or an ethical view of the world. I’m really on the fence, I guess. I could also see that, that in the end, very different forms of life, that we would even hardly recognize, would basically interact with us via sort of a game theoretic competition mode, and that they cannot, and because they’re so different from us, that we would have trouble finding shared values. So I see possibilities for both outcomes. If other life forms share some commonality with our life form, I’m hopeful for a common ground. But that seems like a big assumption, because they could be so totally different, that they cannot connect at a more fundamental level.

Lucas Perry: Taking these short and long term perspectives, what is really compelling and exciting for you about good futures from AI? Is it the short to medium term benefits? Are you excited and compelled by the longer term outcomes, the possibility of superintelligence allowing us to spread for millions or billions of years into the cosmos? What’s really compelling to you about this picture of what AI can offer?

Bart Selman: Yeah. I’m optimistic about the opportunities, both short term and longer term. I think it’s fairly clear that humanity is actually struggling with, there’s an incredible range of problems right now, sustainability, global warming, political conflicts. You could be quite pessimistic, almost, about the human future. I’m not, but these are real challenges. So I’m hopeful that actually AI will help humanity in finding a better path forward. Now, as I mentioned briefly, even in terms of policy and governance, AI systems may actually really help us there. So far this has never been done. AI systems haven’t been sufficiently sophisticated for that, but in the next five to 10 years, I could see systems starting to help human governance. That’s the short term. I actually think AI can have a significant positive impact in resolving some of our biggest challenges.

In a longer term, it’s harder to anticipate what the world would look like, but of course, spreading out as a superintelligence and living on, in some sense, spreading out across the universe and over many different timescales, having AI continue the human adventure is actually sort of interesting, how we wouldn’t be confined to our little planet. We would go everywhere. We’d go out there and grow. So that could actually be an exciting future that might happen. It’s harder to imagine exactly what it is, but it could be quite a human achievement. In the end, whatever happens to AI, it is, of course, a human invention. So I think science and technology are human inventions, and that’s almost what we can be most proud of, in some ways, of things that we actually did figure out how to do well, aside from creating a lot of other problems on the planet. So we could be proud of that.

Lucas Perry: Is there anything else here in terms of the economic, political and social situations of positive futures from AI that you’d like to touch on, before we move on to the negative outcomes?

Bart Selman: Yeah. I guess the main thing, I’m hoping that the general public and politicians will become more aware, and will be better educated about the positive aspects of AI, and the positive potential it has. The range of opportunities to transform education, to transform health care, to deal with sustainability questions, to deal with global warming, scientific discovery, the opportunities are incredible.

What I would hope is that those aspects of AI will receive more attention in the broader public, and with politicians and with journalists. It’s so easy to go after the negative aspects. Those negative aspects and the risk have received a disproportional attention from the positive aspect. So that’s my hope.

As part of the AAAI organization, the professional organization for artificial intelligence, part of our mission is to inform Washington politicians of these positive opportunities, because we shouldn’t miss out on those. That’s an important mission for us, to make that clear, that there’s something to be missed out on, if we don’t take these opportunities.

Lucas Perry: Yeah. Right. There’s the sense that all of our problems are basically subject to intelligence. As we begin to solve intelligence, and what it means to be wise and knowing, there’s nothing in the laws of physics that are preventing us from solving any problem that is, in principle, solvable within the laws of physics. It’s like intelligence is the key to anything that is literally possible to do.

Bart Selman: Yeah. Underlying that is rational thought, our abilities to analyze things, to predict a future, to understand complex systems. And as the rationality underlies that scientific thought process, and humans have excelled at that, and AI can boost that further. That’s an opportunity we have to grab. And I hope we, people, recognize that more.

Lucas Perry: I guess, two questions here, then. Do you think existential risk from AI is a legitimate threat?

Bart Selman: I think it’s something that we should be aware of, that it could develop as a threat, yeah. The timescale is a little unclear to me, how near that existential threat is, but it’s something that we should be aware of that there is a risk of runaway intelligence systems not properly controlled. Now, I think that the problems will emerge much more concretely and earlier, for example, cybersecurity and AI systems that break into computer networks that are hard to deal with. So it will be very practical threats to us, that will take most of our attention. But the overall existential threat, I think, is indeed also there.

Lucas Perry: Do you think that the AI alignment problem is a legitimate, real problem, and how would you characterize it, assuming you think it’s a problem?

Bart Selman: I do think it’s a problem. What I like about the term, it sort of makes it crisp, that if we train a system for a particular objective, then it will learn how to be good at that objective. But in learning how to do that, it may violate basic human principles, basic human values. I think, as a general paradigm statement, that we should think of what happens to systems that we train to optimize a certain objective, that they need to achieve that in a way that aligns with human values, I think, is a very fundamental research question and a very valid question. In that sense, I’m a big supporter, in the research community, of taking the value alignment problem very serious.

As I said before, there are some hesitation about how to approach the problem. I think, sometimes, the value alignment folks gloss over this issue, of what are common values, and are there any common values? So the value alignment, solving it assumes, “Okay, well, when we get the right values in, we’re all done.” What worries me a little bit in that context, as well, these common values are possibly not as common as we think they are. But that’s the issue of how to deal with the problem. But the problem itself, and as a research domain, is very valid. As I said early on, with the little Sokoban example, it is an absolutely surprising aspect of the AI systems for training, how they can achieve incredible performance, but doing it in a way, not knowing certain things that are obvious to us, in some very nonhuman ways. So that’s clearly coming out in a lot of AI systems. And that’s related to the value alignment problem. This fact that we can achieve a super high level of performance, even when we train it carefully with the human generated training data, and things like that, it still can find ways of doing things that are very nonhuman, and potentially very non-value aligned. That makes it even more important to study the topic.

Lucas Perry: Do you think that the Sokoban example, that you can translate the pushing the boxes into corners as a expression of the alignment problem, like imagining if pushing boxes into corners was morally abhorrent to humans?

Bart Selman: Yes. Yeah, that’s an interesting way of putting it. It is an example of what I sort of think of as it’s a domain, and it’s a toy domain, of course, but there’s certain obvious truths to us that are obvious. In that case, pushing a box in a corner is not a moral issue, but it’s definitely something that is obvious to us. If you replace it with some moral truths to us that is obvious to us, it is an illustration of the problem. It’s an illustration of when we think of training a system, and even if you think of, let’s say, bringing up a child, or a human learner, you have a model of what that system will learn, what that human learns, and how the human will make decisions. The Sokoban example is sort of a warning of, with an AI system, it will learn the performance, the test, so it will pass the final test. But it may do so in ways that you would never have expected to achieve it.

With the corner example, it’s a little strange, almost, to me, to realize that oh, you can solve this very hard Sokoban problem, without ever knowing about what a corner is. And it literally doesn’t. It’s the surprises of getting to a human level performance, and missing, and not quite understanding how that’s done. I think another, for me, a very good example, is machine translation systems. So machine translation systems, we see incredible performance of machine translation systems, where they basically map strings in one language, to a string in English to Chinese, or English to French, having discovered a very complex transformation function in the deep net, trained on hundreds of thousands of sentences, but it’s doing it without actually understanding. So it can translate and an English text into a French text or a Chinese text, at a reasonable level, without having any understanding of what the text is about. Again, to me, it’s that nonhuman aspect. Now, researchers might push back and say, “Well, the network has to understand something about the texts, deep in the network.”

I actually think that we’ll find out that the network understands next to nothing about a text. It just has found a very clever transformation that we initially, when we started working on a natural language translation didn’t think would exist. But I guess it exists, and you can find it with a gradient descent deep network. Again, it’s an example of showing a human level cognitive ability achieved in a way that is very different from the way we think of intelligence. That means, when we start using these systems, we are not aware. So if people in general are not aware that your machine translation app has no idea what you’re talking about.

Lucas Perry: So, do you think that there’s an important distinction here to be made between achieving an objective, and having knowledge of that particular domain?

Bart Selman: Yes, yes. I think that’s a very good point, yeah. So by boiling things down, in my sense, by boiling tasks in AI down too much to an objective, in machine learning, the objective is do well on the test set. By boiling things down too much to a single measurable objective, we are losing something, and we’re losing underlying knowledge, the way in which the system actually achieves it.

We’re losing an understanding, and we’re losing the attention to that aspect of the system. That’s why interpretability of deep nets has become sort of a, so it’s definitely a hot area.

But it’s trying to get back to some of that issue is, what’s actually being learned here? What’s actually in these systems? But if you focus just on the objective, and you get your papers published, you’re actually not encouraged to think about that.

Lucas Perry: Right. And there’s the sense, then, also, that human beings have many, many, many different objectives and values that are all simultaneously existing. So when you optimize for one, in a kind of unconstricted way, it will naturally exploit the freedom in the other areas of things that you care about, in order to maximize achieving that particular objective. That’s when you begin to create lots of problems for everything else that you value and care about.

Bart Selman: Yeah, yeah. No, exactly. That’s the single objective problems. Actually, you lay out, a potential path is saying, “Okay, I should not focus on a single objectives task. I actually have to focus on multiple objectives.”

And I would say, go one step further. Once you start achieving objectives, or sets of objectives, and your system performs well, you actually should understand, to some extent, at least, what knowledge is underlying, what is the system doing, and what knowledge is it extracting or relying on, to achieve those objectives? So that’s a useful path.

Lucas Perry: Given this risk of existential threat from AI, and also, the AI alignment problem as its own kind of issue, which, in the worst of all possible cases, leads to existential risk. What is your perspective on futures that you fear, or futures that have quite negative outcomes from AI, in particular, in the light of the risk of existential threat, and then, also, the reality of the alignment problem?

Bart Selman: Yeah, I think, so the risk is that we continue on a path of designing a system, with a single objective in mind, and just measuring the achievement there, and ignored yet alignment problem. People are starting to pay attention to it, but paying attention to it, and actually really solving it is, two different things. There is a risk that these systems just become so good and so useful, and commercially valuable, that the alignment problem gets sort of pushed to the background as being not so relevant, and that we don’t have to worry about it.

So I think that’s sort of the risks that AI is struggling with. And it’s a little amplified by the commercial interest. I think you had a clear example, there is the whole social network world, and how that has spread fake news, and then got people into different groups of people to think totally different things, and to believe totally different facts. In that, I see a little warning sign there for AI. Those networks are driven by tremendous commercial interests. It’s actually hard for society to say there’s something wrong about these things, and maybe we should not do it this way. So that’s a risk, it works too well to actually push back and say, “We have to take a step back and figure out how to do this well.”

Lucas Perry: Right? So you have these commercial interests, which are aligned with profit incentives, and attention becomes the variable which is trying to be captured for profit maximization. So attention becomes this kind of single objective that these large tech companies are training their massive neural nets and algorithms to try and capture the most of, from people. You mentioned issues with information.

And so people are more and more becoming aware of the fact that if you have these algorithms, that are just trying to capture as much attention as possible, then things like fake news, or extremist news and advertising is quite attention capturing. I’m curious if you could explain more of your perspective on how the problem of social media algorithms attempting to capture, and also commodifying human attention, as a kind of single objective that commercial entities are interested in capturing, how that represents the alignment problem?

Bart Selman: Yeah, so I think it’s a very nice analogy. First, I would say, to some extent the algorithms that try to maximize the time spent online, basically, are getting most attention. Those are not particularly sophisticated. Those are actually, very basic sort of, you can sample little TikTok videos. How often are they watched by some subgroup? And if they’re watched a lot, you give them out more. If they’re not watched, you start giving them out less. So the algorithms are actually not particularly sophisticated, but they do represent an example of what can go wrong with this single objective optimization.

What I find intriguing about it, it’s not that easy to fix, I think. Because the companies, of course, their business model is user engagement, is advertising, which, you have to tell that the company’s not to make as much money as they could. If there was an easy solution, that would have happened already. I think we’re actually in the middle of trying to figure out, is there a balance between making profits from a particular objective and societal interests, and how can we align those ideas? And it’s a value alignment problem between society and companies that profit from them. Now, I should stress, in the whole of social networking, and that’s, I think, what makes the problem sound intriguing. There’s an incredible positive aspects to social networks, and people exchanging stories, and interacting. Again, that I think is what makes it complex. It’s not that that it’s only negative, it’s not. There’s tremendous positive sides to having interesting social networks, and exchanges between people. People, in principle, could learn more from each other.

Of course, what we’ve seen is actually, strangely, people seem to listen less to each other. Maybe it’s too easy to find people that think the same way as you do, and the algorithms encourage that. In many ways, the problems with the social networks and the single objective optimization are a good example of a value alignment challenge. It shows that the solution, finding a solution to that is probably, it will require way more than just technology. It will require society and governance companies to come together and find a way to manage these challenges. It will not be an AI researcher in an office that finds a better algorithm. So it is a good illustration of what can go wrong. To me, it’s a good illustration, of what can go wrong. And in part, because, if people didn’t expect this, actually. They saw the positive sides of these networks, and they’re bringing people closer together, and that no one actually had thought of fake news, I think. It’s something that emerged, and that shows how technology can surprise you. That’s of course, in terms of AI, one of the things we have to watch out for, the unexpected things that we did not think would happen, yeah.

Lucas Perry: Yeah, so it sounds like the algorithms that are being used are simpler than I might have thought, but I guess maybe that seems like it accounts for the difficulty of the problem, if really simple algorithms are creating complete chaos for most of humanity.

Bart Selman: Yeah. No, no, exactly. I think that that’s an excellent point. So yeah, you don’t have to create very complicated … You might think, “Oh, this is some deep net doing reinforcement learning.”

Lucas Perry: It might be closer to statistics that gets labeled AI.

Bart Selman: Yeah. Yeah, it gets labeled AI, yeah. So it’s actually just plain old simple algorithms, that now do some statistical sampling, and then amplify it. But you’re right, that maybe the simplicity of the algorithm makes it so hard to say, “Don’t do that.”

It’s like, if you run a social network, you would say, “Let’s not do that. Let’s spread the posts that don’t get many likes.” That’s almost against your interests. But it is an example of, the power is partly also, of course, the scale on which these things happen.

With the social network, I think, what I find interesting is why it took awhile before people became aware of this phenomenon is, because everybody had their personalized content. There was no share to one news channel, or something like that. There’s one news channel. Everybody watches it, and then you see what’s on it.

I have no idea what’s in my newsfeed of the person who’s sitting next to me. So there was also certain things like, “Ah, I didn’t know you got all your news articles with a certain slant.”

So not knowing what other people would see and having a huge level of personalization was another factor in letting this phenomenon go unnoticed for quite awhile. But luckily people are now at least aware of the problem. I haven’t solved it yet.

Lucas Perry: I think two questions come up for me. One thing that I liked, that Yuval Noah Harari has said is, he’s highlighted the importance of knowledge and awareness and understanding in the 21st century, because you said this isn’t going to be solved by someone in Big Tech creating an algorithm that will perfectly … captures the collective value of all of the United States or planet earth and how it is that the content be ethically distributed to everyone. It also, it requires some governance, as you said, but then also some degree of self-awareness about how the technology works and like how your information is being biased and constrained and for what reasons. The first question is, I’m curious how you see the need for collective education on technology and AI issues in the 21st century, so that we’re able to navigate it as people become increasingly displaced from their jobs and it begins to really take over. Let’s just start there.

Bart Selman: So, I think that’s a very important challenge that we’re facing. And I think education of everyone is a key issue there. So, AI should not be, or these technologies should not be presented as some magic boxes. I think it’s much better for people to get some understanding of these technologies. And, I think that’s possible in, in our educational system. It has to start fairly early that people get some idea of how AI technologies were. And most importantly, perhaps people need to start understanding better what we can do and what we cannot do and what AI technologies are about. A good example to me is something like the data privacy initiative in Europe, which I think is a very good initiative.

But for example, there’s a detail in it, is where you have a right. I think- I’m not sure whether it’s part of the law, but there’s definitely discussions and how you have a right to get an explanation of a decision by an AI system. So there’s a right to an explanation. And what I find interesting about it, that sounds like, oh, that’s a very good thing to get. Until you’ve worked with AI system and machine learning systems, and you realize, you can make up pseudo explanations pretty easily, and you can actually ask your systems to explain it without using the word gender or race, and they will come up with good explanation.

So the idea that a machine learning algorithm has sort of a crisp explanation, that is the true explanation of the decision is actually far from trivial and can actually easily be circumvented. So, it’s an example to me, of policymakers coming up with regulations that sounds like they’re making progress, but they’re missing something about what AI system can and cannot do. Yeah. That’s another reason why I think people need much better education and insights into AI technologies and at least hear from different perspectives about what’s possible and what’s not possible.

Lucas Perry: So, given the risk of AI and algorithms increasingly playing a role in society, but also playing this part of single objective optimization and then us as humanity, having to collectively face the negative outcomes and negative externalities from widely spread algorithms that are single objective maximizing. In light of this, what are futures that you most fear in the short term from 5, 10, 15, 20 years from now where we’ve really failed it at AI alignment and working on these ethics issues.

Bart Selman: Yeah, so one thing that I do fear is an increased income inequality, and it’s as simple as that, that the companies that are the best at AI, that have the most data will get such an advantage over other organizations, that the benefits will be highly concentrated on a small group of people. And that, I think is real, because AI technology, in some sense, amplifies your ability to do things. So it is like in finance, if you have a good AI trading program that can mine text and a hundred or a thousand different indicators, you could build a very powerful financial trading firm. And of course trading firms are working very hard on that, but it concentrates a lot of the benefits in the hands of a small group of people. That I actually think is sort of- in my mind, sort of the biggest short-term risk of AI technology.

It’s a risk any technology has, but I think AI sort of amplifies it. So that, has to be managed and that comes back to what I mentioned fairly early on, the benefits of AI. It has to be ensured that it will benefit everyone, and maybe not all to the same extent, but at least everyone should benefit to some extent, and that’s not automatically going to happen. So that’s a risk I see in development of AI and then more dramatic risks. I think short term cybersecurity issues, smart tax on our infrastructure. AI programs could be quite dangerous, deep fakes and so sophisticated, some deep fakes. There are some specific risks that we have to worry about because they are going to be accelerated with AI technology. And then there’s of course the military autonomous weapon risk.

There’s an enormous pressure… Since it’s a competitive world of developing systems that use as much automation as possible. So, it’s not so easy to tell a military or country not to develop autonomous weapon systems. And so I’m really hoping that people start to realize, and this is again, an educational issue, partly of people, the voters basically that there is a real risk there just like nuclear weapons was a real risk and we have to get together to make agreements about at least a management of nuclear weapons. So we have to have agreements, global agreements about autonomous weapons and smart weapons, and what can be developed or what should at least be controlled somehow that will benefit older players. And that’s one of the short-term risks I see.

Lucas Perry: So if we imagine in the short term that there’s just all of these algorithms, proliferating that are single objective maximizing, that are aligned with whatever corporation that is using them, there is a lack of international agreement on autonomous weapons systems. Income inequality is far higher due to the concentration of power in particular individuals who control vast amounts of AI. So, if you have the wealth to accumulate AI, you begin to accumulate most of the intelligence on earth, and you can use that to create robots or use robotics so that you’re no longer dependent on human labor. So there’s increase in income and power inequality, and lack of international governance and regulation. Is that as bad as the world gets in the short term? Or is there anything else that makes it even worse?

Bart Selman: No, I think that’s about as bad as it gets. And I assumed I would be a very strong reaction in almost every country of the regular person as of the voter or the person in the street. There would be a strong reaction to that. And it’s real.

Lucas Perry: So, is that reaction though, possible to be effective in any way if lethal autonomous weapons have proliferated?

Bart Selman: Well, so legal autonomous weapons, yeah. So there are two different aspects to sort of what- In one aspect is what sort of happens within a country. And do the people accept that extreme levels of inequality, income inequality, and power distributions, and I think people will push back and there will be backlash against that. Lethal autonomous weapons when they start proliferating, I think- So I just have some hope that countries will realize that that is in nobody’s interest. So, that countries are able to manage risks that are unacceptable to everyone, I think. So I’m sort of hopeful that in the air of lethal autonomous weapons, that we will see a movement by countries to say that, “Hey, this is not going to be good for any one of us.”

Now, I’m being a little optimistic here, but with nuclear weapons. We did see it’s always a struggle and it remains a struggle today. But so far, countries have sort of managed these risks reasonably well. It’s not easy, but it can be done. And I think it’s partly done because everybody realizes nobody will be better off if we don’t manage these risks. So legal autonomous weapons, I think there has to be first a better understanding that these are real risks. And if you let that get out of hand, like let small groups develop their own autonomous weapons, for example, that that could be very risky to the global system. I’m hoping that countries will realize this and start developing a strategy to manage it, but it’s a real risk. Yeah.

Lucas Perry: So should things like this come to pass, or at least some of them, in the medium to long-term, what are futures that you fear in the time range of fifty to a hundred years or even longer?

Bart Selman: Yeah, so the lethal autonomous weapon risk would be, that could just be as bad as nuclear weapons being used at some point. So that sort of could wipe out humanity. So there is, I think that’s sort of worst case scenario is that we would go down in flames. There are some other scenarios where I think, and this is more about the inequality issue. Where a relatively small group of people grabs most of the resources and is enabled to do so by AI technology and the rest can live reasonable lives, but are limited by their resources.

So that’s, I think a somewhat dark scenario that I could see happen if we don’t pay attention to it right now. That could play out in 20, 30 years. It’s a little hard to… Again, one thing that’s a little difficult to predict is how fast the technology will grow and you combine it with advances in biology and medicine. I’m always a little optimistic. We could be living in just a very different and very positive world too, if that’s what I’m hoping that we’ll choose. So I am staying away a little bit from too dark a scenario.

Lucas Perry: So, a little bit about AI alignment in particular. I’m curious, it seems like you’ve been thinking about this since at least 2008, perhaps. I mean, even earlier you can let us know how have your views shifted and evolved. I mean, it’s been, what is that, about 13 years?

Bart Selman: Yeah. No, very good question. So yeah, in 2008 we had the, Eric Horvitz and I co-chaired a AAAI presidential panel on the risks of AI. It’s very interesting because at that time, this was before the real deep learning revolution. People saw some concerns, but the general consensus of… And this was a group of about 30 or 40 AI researchers and a very, very good group of people. There was a sort of a general census that it was almost too early to worry about the value alignment and the risks of AI. And I think it was true that AI was still a very academic discipline and a theme talking about, oh, what if this AI system starts to work? And then people start using it and what’s going to happen, seemed premature and was premature at the time, but it was good for, I think for people to get together and at least discuss the issue of what could happen. Now that really dramatically changed over the last 10 years, as, particularly the last five years.

And in part to people like Stuart Russell, Max Tegmark who basically brought to the forefront these concerns of AI systems, combined with the fact that we see the system starting to work, I guess yeah, if that hadn’t happened. So now, we see these incredible investments and companies really going after AI capabilities and suddenly these questions that were quite academic early on are now very real, and we have to deal with them and we have to think about them. And you do, I mean, the good thing is if I look at, for example, NSF and the funding in the United States, but around the world, actually also in Europe and in China, people are starting to fund AI safety, AI ethics, work on value alignment. You see it in conferences and people start looking at those questions. So I think that’s the positive side.

So I’m actually quite encouraged, but it was how much was achieved in a fairly short time. You know, FLI played a crucial role in that too, in bringing awareness to the AI safety issues. And now I think among most AI researchers, maybe not all, but most AI researchers, these are viewed as legitimate topics for study and legitimate challenges that we have to address. So it’s not, sometimes I feel good about that aspect. Of course, the questions remain urgent and the challenges are real, but at least I think the research community has found the attention. And in Washington, I was actually quite pleased if I look at the significant investments being planned for AI and the development of the AI R&D in the United States and yeah, safety, fairness, a whole range of issues that touches on how AI will affect society are getting serious attention, so they are being funded. And that happened last five years, I would say. So, that’s a very positive develop in this context.

Lucas Perry: So given all this perspective on the evolution of the alignment problem and the situation in which we find ourselves today, what are your plans or intentions as the president of AAAI?

Bart Selman: Yeah, so as part of AAAI, we’ve definitely stepped up our involvement, with Washington policy making process to try to inform policy makers better about the issues. And we’ve had, actually, we did a roadmap for AI research in the United States, and there was also of planning 20 years ahead of topics. And we proposed there to, I think what was a key component to us to build a national AI infrastructure, as we called it, that there’s an infrastructure to do AI research and development that would be shared among institutions and be accessible to almost every organization. And the reason is that we don’t want AI research and development to be concentrated just in a few big private companies. We actually would like to make it accessible to many more stakeholders and many more groups in society.

And to do that, you need an AI infrastructure where you have capabilities to store, curate large data sets, large facilities to cloud computing to give access to other groups in society, to build AI tools that are good for them, and that are useful for them. So as AAAI, we are pushing for this sort of generally making AI R&D generally available and to boost the level of funding. Keeping in mind, these issues of fairness, value alignment as valid research topics that should be part of anybody’s research proposal. People who do research proposals should have a component of where they consider whether their work is relevant in that context. And if it is, what contributions they can make. So, that’s what our society is doing and this is of course, a good time to be doing this because Washington is actually paying attention because not just the US, every country is developing AI R&D initiatives. So our goal is to provide input and to steer it in a positive way. Yeah and that’s actually a good process to be part of.

Lucas Perry: So you mentioned alignment considerations being explicitly, at least covered in the publication of papers, is that right?

Bart Selman: So at least I think people are, yeah- so there are papers purely on the alignment problem, but people are- if I look at sort of like reinforcement learning world, people are aware that value alignment is an issue. And to me, it feels so closely related to interpretability and understanding. We talked about it a little bit before is, you are not just getting to a certain objective, quantitative objective, single objective not just optimizing for that, actually understanding the bounds of your system safety bounds, for example, in the work on cyber-physical systems and self-driving cars, as of course, a key issue is how do I guarantee that whatever policy has learned, how do I guarantee that that policy is safe? So it’s getting more attention now. You know the pure value alignment problem like when it gets to ethics. I think there is- we talked about values or there’s a whole issue of how you define values and what are the basic core values.

And these are partly ethical questions. There I think is still room for growth. But I also see that for example, at Cornell, there are ethics people in the philosophy departments are thinking about ethics, are starting to look at this problem again and looking at the way AI is going in these directions. So partly I’m encouraged by an increase of collaborations between different disciplines that traditionally have not collaborated much, but the fact that ethics is relevant to computer science students. I think now five years ago, nobody even thought of mentioning that. And now I think that most departments realize, yes, we actually should tell our students about ethical issues and we should educate them about algorithmic bias and value alignment is a more challenging thing because you have to know a little bit more about AI, but most AI courses will definitely cover that now.

So, I think there’s great progress and I’m hoping that we just keep continuing to make those connections and make it clear that when we train students to be the next generation of AI engineers, that they’re very aware, they should be very aware of these ethical components. And that’s, I think is, is- it might even be somewhat unique in engineering. I don’t think engineering normally would touch on ethics too much, but I think AI is forcing us to do so.

Lucas Perry: So you see understanding of, and a sense of taking seriously the AI alignment problem at for example, AAAI as increasing.

Bart Selman: Yes. Yes, yes. And definitely it’s increasing and people are- yeah partly, there’s always- it takes time for people to become familiar with the terminology, but people are much more familiar with the questions and yeah, we’ve even had job candidates talk about AI alignment. And so then the department has to learn about what that means. So it’s partly an educational mission, it’s you actually have to understand how reinforcement learning, optimize and decision-making, and you have to understand a little bit how things work, but I think we’re starting to educate people and definitely people are much more aware of these problems so that’s good.

Lucas Perry: Yeah. Does global catastrophic or existential risk from AI fit into AAAI.

Bart Selman: I would say that at this point, yeah. I don’t think we got research. Well actually, it’s hard to say because we, I think we have like 10,000 submissions and I think there’s room at AAAI for those kind of papers. I just haven’t actually- personally haven’t actually seen them, but that’s actually- as president of AAAI, I would definitely encourage us to branch out and if somebody has an interesting paper, this could be a position paper or some other types of papers now that we have that sort of say, okay, let’s have a serious paper on existential risks because there is room for it. It just hasn’t so far has not happened much, I think, but I think it fits into our missions. So I would encourage that.

Lucas Perry: So you mentioned that one of the things that AAAI was focusing on was collaborating with government and policy making decisions, offering comments on documents and suggestions or proposals. Do you have any particular policy recommendations for existing AI systems or the existing AI ecosystem that you might want to share?

Bart Selman: Yeah, I think my sense there is sort of more like a meta level comment is- I think what we want is people designing systems. There’s a significant AI component and that the big tech companies, for example, our main input dare is we want people to pay serious attention to various things like bias, fairness, and these kind of criteria, AI safety. So it’s not- I wouldn’t have a particular recommendation for any particular system. But, with the AAAI submissions now we asked for sort of an impact statement and that’s somebody who does research. And that’s what we’re asking from the researchers, is when you do research that touches on something like value alignment or AI safety that you should actually think about societal component and possible impact on work. So we’re definitely asking people to do that.

In companies, I would say it’s more we ask company, we encourage company to have those discussions and make their engineers aware of these issues. And there’s one organization. Now the global partnership on AI that that’s actually also now very actively trying to do this on an international scale. So it’s a process. And it’s partly, I think you mentioned earlier an educational process where people have to learn about these problems to start incorporating them into our daily work.

Lucas Perry: I’m curious about what you think of AI governance and the relationship needed between industry and government. And one facet of this is for example, we’ve had Andrew Critch on the podcast and he makes quite an interesting point that some number of subproblems in the overall alignment problem will be naturally solved via industry incentives. Whereas, some of them won’t be. The ones that are, that will naturally be solved by industry incentives are those which align with whatever industry incentives are, so profit maximization. I’m curious, your view on the need for AI governance and how it is that we might cover these areas of the alignment problem that won’t naturally be solved by industry.

Bart Selman: That’s a good question. I think not all these problems will be solved by industry. So their objectives are sometimes a little too narrow to just solve them, a broad range of objects. So I really think it has to occur in a discussion, in a dialogue between policymakers, government, and public and private organizations. And it may require whether it requires regulation or at least form of self regulation that may be necessary to even level the playing field. Very early on, earlier we talked about social networks are spreading, fake news. You might actually need regulations to tell people not to do certain things because it will be profitable for them to do it. And so then you have to have regulations to limit that.

On the other hand, I do think a lot of things will be through self-regulation. So self-driving cars is a very circumscribed area. There’s a clear interest of all the participants in all the companies working on self-driving cars to make them very safe. So for some kind of AI systems, the objectives are sort of self-reinforcing and you need safety. Otherwise, people will not accept them. Other areas, I’m thinking for example, finance industry and that’s a big issue on- that it’s actually the competitive advantages often in proprietary system. It’s actually hard to know what these systems do. That I haven’t- I don’t have a good solution for that. One of my worries is that financial companies developed technologies that they will not want to share because that would be detrimental to the business, but actually expose risks that we don’t even know of.

So society actually has to come to grips with, are risks being created by AI systems that we don’t know of? So it has to be a dialogue and interaction between public and private organizations.

Lucas Perry: So in the current AI ecosystem, how do you view and think about narratives around a international race towards more and more powerful AI systems, particularly between the United States and China?

Bart Selman: Yeah. Yeah. So, yeah, I think that’s a bit of an unfortunate situation right now. So in some sense, the competition between China and the US and also Europe is good from an AI perspective, in terms of investments in AI R&D which actually does address also some of the AI safety issues and issues of alignment. So in some sense that’s a good benefit of these extra investments. And the competition aspect is less positive. And as AI scientists, we actually interact with AI scientists in China and we enjoy those interactions and a lot of good work comes out of that. When things become proprietary, people have data sets that other people don’t have and other organization don’t have and some countries do have, others don’t, I think the competition is not as positive. And, again, my hope is that by bringing out potentially positive aspects of AI, much stronger in terms of how it can… To me, for example, AI can transform the healthcare system.

It can make it much more efficient, much more widely available with remote healthcare delivery and things like that and better diagnosis systems. So there’s an enormous upside for developing AI for healthcare. I’ve actually interacted with people in China that work on healthcare for AI. Whether it gets developed in China or it gets developed here, that actually doesn’t matter. It would benefit both countries. So I really hope that we can keep these channels open instead of totally separate developments in these two countries. But there is a bit of a risk because the situation has become so competitive, but, again, I’m hoping people see it will improve healthcare in both countries is probably the right way to do it and we shouldn’t be too isolationist in this regard.

Lucas Perry: How do you feel the sense of these countries competing towards more and more powerful AI systems, how do you feel that that affects the chances of successful value alignment?

Bart Selman: Yeah, so that could be an issue. If the countries really start not sharing their technology and not sharing potential advances, it is harder, I think, to keep value alignment issues and AI safety issues under control. I think we should be open about the risk of countries going at it by themselves because the more people look at systems, the more researchers look at different AI systems from different angles, the better. And I guess a very odd example is, I always thought that it would be nice if AlphaZero was available to the AI research community to probe the brain of AlphaZero, but it’s not. And so there are already systems in industry that would actually benefit from study by much broader group of researchers. And there’s a risk there.

Lucas Perry: Do you think there’s also a risk with sharing? It would seem that you would accelerate AGI timelines by sharing the most state-of-the-art systems with anyone, right? And then you can’t guarantee that those people will use it in value-aligned ways.

Bart Selman: Yeah. No, no, no. That’s a flip side. It’s good you brought that up. Yeah. There is a flip side in sort of sharing even the latest deep learning code or something like that, that other people, that malicious actors could use it. In general, I think an openness is better in terms of keeping an eye on what gets developed. So in general, I think openness allows different researchers to develop common standards and common safety guards. So I see that risk of sharing, but I do think overall the international research community can set standard. We see that in synthetic biology and other areas where openness in general leads to better managing of risks. But you’re right. There is effect that it accelerates progress, but the countries are big enough, even if China and the US would completely separate their AI developments, both countries would do very well in their development of technology, so.

Lucas Perry: So I’m curious, do you think that AI is a zero-sum game? And I’m curious how you view an understanding of AI alignment and existential risk at the highest levels of Chinese and US government affecting the extent to which there is international cooperation for the beneficial development of AI, right? So there’s this sense of racing because we need to capture the resources and power but there’s the trade-off with the risks of alignment and existential risk.

Bart Selman: So yeah. I firmly believe that it’s not a zero-sum game. Absolutely not. I give the example of the healthcare system. Both China and the US have interest in better accessible healthcare, more available healthcare and lower cost healthcare. So actually the objectives are very similar there, and AI can make incredible difference for both countries. Similarly in education, you can improve education by having AI assisted education, adult education, continuous learning education. So there are incredible opportunities and both countries would benefit. So definitely AI is not a zero-sum game. So I hope countries realize that, when China declared they want to be a leading AI nation by 2030, I think there’s room for several leading nations.

So I don’t think one nation is better at AI that there will be the best outcome. The better outcome is if AI gets developed and used by many nations and shared. So I hope that politicians and governments see that shared interest. Now, as part of that shared interest, they may actually realize that the existential risks of bad actors, and that can be small groups of people, it could be a company or an organization, a bad actor using AI for negative goals that’s a global risk that, again, should be managed by countries collaborating. So I’m hoping that there are actually some understanding of the global benefits and not a zero-sum game, we all can gain, and the risk is a global risk and we should actually have a dialogue about some of these risks. The one component that is a tricky one, I think, is always the military component. But even there, as I mentioned before, the risk of autonomous lethal weapons is, again, something that affects every nation. So I can see countries realizing it’s better to collaborate and cooperate in these areas than to just take it as a pure competition.

Lucas Perry: So you said it’s not a zero-sum game and that we can all benefit. How would you view the perspective that the relative benefits for me personally, for racing are still higher, even if it’s not a zero-sum game, therefore I’m going to race anyway?

Bart Selman: Yes. I mean, yeah. There may be some of that, except that… I almost look at it a little different. I can see a race where we still share technology. So, the race is one of these strange… It’s almost like we’re competing with each other but we’re trying to get better all together. You can have a race and that can still be beneficial for progress, as long as you don’t want to keep everything to yourself. And I think what’s interesting, and that’s the story of scientific discovery and the way scientists operates, in some sense, scientists compete with each other because we all want to discover the next big thing in science. So there’s some competition. There is also a sense that we have to share because if I don’t share, I don’t get the latest from what my colleague is doing. So there’s a mutual understanding that yes, we should share because actually helps me, even individually. So that’s how I see.

Lucas Perry: So how do you convince people to share the thing which is like the final invention? Do you know what I mean? If I need to share it because then I won’t get the other thing that my colleague will make, but I’ve just made the last invention that means I will never have to look to my colleague again for another invention.

Bart Selman: Yeah. So that’s a good one, but yeah. So in science, we don’t think there’s sort of an endpoint. There will always be something novel, so.

Lucas Perry: Yeah, of course there’s always something novel, but you’ve made the thing that will more quickly discover every other new novel thing than any other agent on the planet. How do you get someone to share that?

Bart Selman: So, well, I think partly the story still is even if one person, if one country gets so dominant, there still is the question, is that actually beneficial for even the country? I mean, there are many different capabilities that we have. There are still nuclear weapons and things like that. So you might get the best AI and somebody might say, “Okay, I think it’s time to terminate you.” So there’s a lot of different forces. So I think it’s a sufficiently complex interaction game that I’m thinking that, to think of it as a single dimension issue is probably not quite the way the worlds will work. And I hope politicians are aware of that. I think they are.

Lucas Perry: Okay. So in the home stretch here, we’ve brought up lethal autonomous weapons a few times. What is your position on the international and national governance of lethal autonomous weapons? Do you think that a red line should be drawn at the fact that life or death decisions should not be delegated to machine systems?

Bart Selman: That’s a reasonable goal. I do think there are practical issues to specify exactly in what sense and how the system should work. So, decisions that have to be made very quickly, how are you going to make those if there’s no time for a human to be in the loop? So I like it as an objective that there should always be a human in the loop, but the actual implementation of system, I think, needs further work and it might even just come down to actual systems and somebody looking at those systems and say, “Okay, this has sufficient safeguards. And this system doesn’t.” Because there’s this issue of how quickly do we have to react and can this be done?

And of course, that’s partly you see that a defensive system may have to make a very quick decision, which could endanger the life of, I don’t know, incoming pilots, for example. So there are some issues, but I’d like it as a principle that legal autonomous systems should not be developed and that there should always be this human decision making as part of it, but that it probably has to be figured out for each individual system.

Lucas Perry: So would you be in favor of, for example, international cooperation in limiting, I guess, having treaties and governance around autonomous weapons?

Bart Selman: Oh, definitely. Yeah. Yeah, definitely. And I think people are sort of sometimes skeptical or wonder whether it’s possible, but I actually think it’s one of those things that is probably possible because when militaries start to develop those systems because that’s the real tricky part, when these systems are being developed or start being sold, they can be in the hands of any group. So I think countries actually have an interest in treaties and agreements on regulating or limiting any kind of development of such systems. So I’m a little hopeful that people will see this would be in nobody’s interest to have countries competing on developing the most deadly lethal autonomous weapon. That would actually be a bad idea. And I’m hopeful that people will actually realize. That is partly again an educational thing. So people should be more aware of it and will directly ask their governments to get agreements.

Lucas Perry: Do you see the governance of lethal autonomous weapons as being a deeply important issue around the international regulation and governance of AI, kind of like first key issue in AI as we begin to approach AGI and superintelligence? So does our ability to regulate and come up with beneficial standards and regulation for autonomous weapons, is that really important for long-term beneficial outcomes from things like AGI and superintelligence?

Bart Selman: Yeah, I think it would be a good exercise, in some sense, of seeing what kind of agreements you can put in place. So lethal autonomous weapons, I think, is a useful starting place because I think it’s fairly clear. I also think there are some complications. You can say, “Oh, we’d never do this.” What about if you have to decide in a fraction of a second what to do? So there are things that have to be worked out, but in principle, I think countries can agree that it needs collaboration between countries and then that same kind of discussion, the same kind of channels, because these things take time to form the right channels and the right groups of people to discuss these issues, could then be put towards other risks that AI may pose. So I think it’s a good starting point.

Lucas Perry: All right. A final question here, and this one is just a bit more fun. At Beneficial AGI 2019, I think you were on a panel that was about, do we want machines to be conscious. On that panel, you mentioned that you thought that AI consciousness was both inevitable and adaptive. I’m curious if you think about the science and philosophy of consciousness and if you have a particular view that you subscribe to.

Bart Selman: No, it’s a fun topic. And actually when I thought about consciousness more and will it emerge, there’s an area of AI that actually, because I’ve been in the field a long time, and the area generally is called knowledge representation and reasoning and it’s about how knowledge is represented in an AI system and how an AI system can reason with it. And one big sub-area there was the notion of self-reflection, the notion in a multi-agent system. So self-reflection is not only you know certain things, you also know about what you know, and you know about what you don’t know. And similarly in multi-agent systems, you have to know not only what you know, but you have to have some ideas of what others may know. And yes, you might have some ideas of what other agents don’t know but that is to facilitate interactions with other agents.

So this whole notion of reflection on your own knowledge and other agents’ knowledge, in my mind, is somewhat connected to consciousness of yourself and your environment, of course. So that led to my comment that if you build sufficiently complex systems that behave intelligently, they will have to develop those capabilities. They have to know about what they know, what they don’t know and what others know and others don’t know. And knowing about knowing what others might know about you, it actually goes arbitrary levels of interactions. So I think it’s going to be a necessary part of developing intelligence system. And that’s why my sense is that some notion of consciousness will emerge in such systems because it’s part of this reflection mechanism.

And then what I think is exciting about it is, in consciousness research there’s also a lot of research now on what is the neurological basis for consciousness. There’s some neurological basis in the brain that points at consciousness. Well, now we have worked at that. We see how deep reinforcement learning interacts with neuroscience. And we’re looking for analogies of deep reinforcement learning approaches in AI and what insights it gives in actual brains, in actual biological neurological systems. So perhaps when we see things like reflection and consciousness emerge in AI systems, we will get new insights in what happens potentially in the brain. So it’s a very sort of interesting potential there.

Lucas Perry: My sense of it is that it may be possible to disentangle constructing a self model, like a model of both what I am and also what it is that I know and that I don’t know and then also my world model and that these things seem to be correlated with consciousness, with the phenomenal experience of being alive. But it seems to me they would be able to come apart just because it seems conceivable to me that I could be a sentient being with conscious awareness that doesn’t have a self model or a world model. You can imagine just like awareness of a wall that’s the color green. The no sense of no duality there between self and object. So, it’s a bit different the way philosophers come at the problem and computer scientists. There’s the computational aspect, of course, the modeling that’s happening, but it seems like the consciousness part can become disentangled from the modeling perhaps. And so I’m curious if you have any perspective or opinion on that and how we could ever know if an AI was conscious given that they may come apart.

Bart Selman: No, you raise an interesting possibility that maybe they can come apart. And then the question is, can we investigate that? Can we study that? And that’s a question in itself. Yeah. So, I was more coming at it from the sense of that when the system gets complex enough and it starts having these reflections, it will be hard not to have it be conscious. But you’re right. It probably could still be, although I would be a little surprised, but yeah. So my point in part is that the deep learning reinforcement approach or whatever deep learning framework we will use to get these reflective capabilities, I’m sort of hoping it might give us new insights into how to look at it from the brain perspective and a neural perspective because these things might carry over. And is consciousness a computational phenomena? My guess is it is, of course, but that still needs to be demonstrated.

Lucas Perry: Yeah. I would also be surprised if sophisticated self and world modeling didn’t most of the time or all the time carry along conscious awareness with it. But even prior to that, as we have domain specific systems, it’s a little bit sci-fi to think about, but there’s the risk of proliferating machine suffering if we don’t understand consciousness and we’re running all of these kinds of machine learning algorithms that they don’t have sophisticated self models or world models, but the phenomenal experience of suffering still exists then that could… We had factory farming of animals and then maybe later in the century, we have the running of painful, deep learning algorithms.

Bart Selman: No, that’s indeed a possibility. It sort of argues we actually have to dig deeper into the questions of consciousness. And so far, I think, most AI researchers have not studied it. I’m just starting to see some possibility of studying it again. I’m starting to study it as AI researchers. And it just brought me back a little bit that this notion of reflection that topics go in and out of fashion, but that used to be actually quite seriously studied, including with philosophers about what it means to know what you know, what does it mean to know what you don’t know, for example. And then there’s the things you don’t know that you don’t know kind of thing. So, we thought about some of these issues and now consciousness brings in that new dimension and you’re quite right. It could be quite separate, but it could also be related.

Lucas Perry: So as we wrap up here, is there a final comment you’d like to make or anything that you feel like is left unsaid or just a parting word for the audience about alignment and AI?

Bart Selman: So, comment to the audience is that the alignment question, value alignment, AI safety are super key important topics for AI researchers, there are many research challenges there that are far from solved. And in terms of the development of AI there are tremendous positive opportunities if things get done right, and that we should not… So one concern I have as an AI researcher is that if we get overwhelmed by the concerns and the risks and decide not to develop positive capabilities for AI. So we should keep in mind that can really benefit society if AI is done well and that we should take that as our primary challenge and manage the risk while doing so.

Lucas Perry: All right, Bart, thank you very much.

Bart Selman: Okay. Thanks so much. It was fun.

• Understanding the universe through digital physics
• How human consciousness operates and is structured
• The path to aligned AGI and bottlenecks to beneficial futures
• Incentive structures and collective coordination

You can find FLI’s three new policy focused job postings here

1:06:53 A future with one, several, or many AGI systems? How do we maintain appropriate incentive structures?

1:19:39 Non-duality and collective coordination

1:22:53 What difficulties are there for an idealist worldview that involves computation?

1:27:20 Which features of mind and consciousness are necessarily coupled and which aren’t?

1:36:40 Joscha’s final thoughts on AGI