Theoretical Population Biology 12S (2019) 38-55

Theoretical Population Biology 12S (2019) 38-55 ITSEVIER Contents lists available at ScienceDirect Theoretical Population Biology journal home pa go: ..vww.elscvier.comflocateApb A two-player iterated survival game John Wakeley a.*, Martin Nowak a'b'c 'Department of Organismic and Evolutionary Biology. Harvard University. Cambridge. MA, 02138. USA b Program for Evolutionary Dynamics. Harvard University. Cambridge. MA 02138. USA ' Department of Mathematics, Harvard University. Cambridge, hfA 02138, USA ARTICLE INFO Ankle history: Received 22 May 2018 Available online 12 December 2018 Keywords: Prisoner's Dilemma Survival game Iterated game Replicator equation Moran model I. Introduction ABSTRACT We describe an iterated game between two players, in which the payoff is to survive a number of steps. Expected payoffs are probabilities of survival. A key feature of the game is that individuals have to survive on their own if their partner dies. We consider individuals with hardwired, unconditional behaviors or strategies. When both players are present. each step is a symmetric two-player game. The overall survival of the two individuals forms a Markov chain. As the number of iterations tends to infinity, all probabilities of survival decrease to Zero. We obtain general, analytical results for n-step payoffs and use these to describe how the game changes as it increases. In order to predict changes in the frequency of a cooperative strategy over time, we embed the survival game in three different models of a large, well- mixed population. Two of these models are deterministic and one is stochastic. Offspring receive their parent's type without modification and (finesses are determined by the game. Increasing the number of iterations changes the prospects for cooperation. All models become neutral in the limit (ri co). Further, if pairs of cooperative individuals survive together with high probability, specifically higher than for any other pair and for either type when it is alone, then cooperation becomes favored if the number of iterations is large enough. This holds regardless of the structure of pairwise interactions in a single step. Even if the single-step interaction is a Prisoner's Dilemma, the cooperative type becomes favored. Enhanced survival is crucial in these iterated evolutionary games: if players in pairs start the game with a fitness deficit relative to lone individuals, the prospects for cooperation can become even worse than in the case of a single-step game. 02018 Elsevier Inc. All rights reserved. More than a century ago, Kropotkin (1902) argued that what he called mutual aid should be ranked among the main factors of evolution. an even more important driver of evolution than within- species competition. The idea of mutual aid is similar to that of mu- tualism: partnerships may be beneficial to both partners and thus unconditionally favored. Kropotkin had been deeply impressed by the abilities of animals to endure harsh winters and perilous migrations in northern Eurasia by living together and supporting each other in situations where a lone animal had little chance of surviving. Thus, he inferred a connection between cooperative behaviors and survival under adverse conditions. But Kropotkin did not spell out the ways in which adversity might favor cooperative behaviors, nor did he consider that cooperative behaviors could be disadvantageous. He took the very fact that animals seem to do better in groups than alone as evidence for mutual aid, apparently unaware that any interaction presents the opportunity, or at least * Corresponding author. E-mail address: wakeleyefas.harvard.edu U. Wakeley). hups://doLorg/10.1016b.tpb.2018.12.001 0040-S809/0 2018 Elsevier Inc. All rights reserved. the temptation, to be the one who comes out ahead even if it is at the expense of other individuals. Indeed, game theory and evolutionary theory have uncovered numerous situations in which cooperative or otherwise helping behaviors are detrimental t0 the individual or selectively disad- vantageous (Hofbauer and Sigmund, 1998; Mesterton-Gibbons. 2000; Cressman, 2005; Schecter and Gintis, 2016). It has been shown using a variety of models that such behaviors can be disfa- vored even if there are obvious advantages to mutual cooperation. For example, in the classic two-player game called the Prisoner's Dilemma (Tucker, 1950; Rapoport and Chammah, 1965; Axelrod. 1984). cooperation results in a "reward payoff' if one's partner also cooperates but a "suckers payoff' if one's partner defects. Defection yields a "temptation payoff" if one's partner cooperates but a "punishment payoff" if one's partner also defects. The reward is of course better than the punishment, but it is further assumed that the temptation payoff is greater than the reward, and the sucker's payoff is even worse than the punishment This leads to a paradox: an individual who defects always receives a higher payoff but it is clearly illogical for everyone to defect. The Prisoner's Dilemma, in which the payoff for defection is higher than for cooperation regardless of one's partner behavior, EFTA00810742 J. Wakeley and M. Nowak/Theoretical Population Biology 125(2019)38-55 39 Table 1 The payoff, a(n). J(n), c(n) or d(n). that an individual receives in a symmetric two-player game depends both on the individual and on the individual's paltrier. In a survival game, these payoffs are probabilities of survival A and B denote two possible strategies or types of individuals (e.g. Dove and Hawk). Payoffs are simultaneously awarded to both players, i.e. each is considered the Individual (and the other the Partner) and awards a payoff according to the table. Partner A Individual A B o(n) c(n) b(n) d(n) represents one of four possible types of two-player games. It is the polar opposite of what Kropotkin (1902) imagined for mutual aid, that instead it would cooperation that would always yield the higher payoff. Between these extremes lie two other kinds of games. In the Stag Hunt (Skyrms, 2004) and related games, the payoff to an individual is higher if it matches the partner's behavior regardless of what that behavior is. Cooperators in the Stag Hunt go for the big game, a stag, which two such individuals can catch but one alone cannot. Non-cooperators opt out of the big-game hunt and accept a middling payoff, a hare, which a single individual can catch. In this case, cooperation yields the higher payoff only when one's partner also cooperates. The Hawk-Dove game (May- nard Smith and Price, 1973; Maynard Smith, 1978) represents the fourth kind, in which having a different behavior than one's partner produces a higher payoff. Doves cooperate by sharing resources but retreat when challenged. Hawks do not cooperate. They are ready to fight to avoid leaving an interaction empty-handed. A Hawk gets the entire resource when facing a Dove but suffers badly when fac- ing another Hawk Hawk and Dove, of course, refer to stereotypical personalities not animals. In this last case, cooperation yields the higher payoff only when one's partner does not cooperate. Besides Hawk-Dove, other well-studied games of this type are the game of chicken and the snowdrift game (Doebeli and liauert. 2005: Nowak, 2006a). We introduce a two-player survival game which can fall into any of these four classes. It may also change, for example from a Hawk-Dove game into a Prisoner's Dilemma, depending on one key feature, the length of the game. In this game, an initially sampled pair of individuals confronts a hazardous situation which is repeated n times. With reference to Kropotkin (1902). we might imagine that each day of a long journey presents a similar set of challenges which threaten survival. They might, for example, be trying to survive a number of very cold nights or attempting to defend themselves repeatedly against a predator. How they fare in each step depends on their behavior and their partner's behavior. The payoff for an individual is all or nothing: either survive to the end of the game or not. Crucially, an individual must face the perilous situation alone for the remainder of the game if its partner dies. Survival payoffs are thus meted out at each step, and this will be important for determining total payoffs in the game. Denoting survival as I and death as 0, the expected total payoff to an individual in a given situation (i.e. witha specified partner initially) is its probability of surviving to the end of the game. The n-step survival game is a symmetric two-player game with total payoffs, a(n), d(n), c(n) and d(n) as Table 1. Using Table 1 to represent an n-step game that is a Prisoner's Dilemma, with A for cooperation and B for defection, the reward would be a(n), the sucker's payoff b(n), the temptation payoff c(n) and the punishment d(n). More generally, depending on a(n), b(n), c(n) and d(n), any n-step survival game falls into one of the four classes described above. Ignoring the detail that some payoffs might be identical, these are defined as follows. If a(n) < c(n) and b(n) < d(n). cooper- ation has the lower payoff regardless of the partner. Then the game falls into the class exemplified by the Prisoner's Dilemma. which we will call PD for short. If a(n) > c(n)and b(n) a d(n), cooperation has the higher payoff only when one's partner cooperates. This is the Stag-Hunt class of games, or SH for short. If a(n) e c(n) and b(n) > d(n), cooperation has the higher payoff only when one's partner does not cooperate. This is the Hawk-Dove class, or HD for short. Finally, if a(n) > c(n) and b(n) > d(n), cooperation has the higher payoff regardless of the partner. We follow De Jaegher and Hoyer (2016) and call this a Harmony Game. or HG for short. We refer to the n-step survival game as a repeated or iterated game because the payoff structure in each of then steps is identical and equivalent to the single-step version of the game. The notion of repeated or iterated games is generally predicated on the idea that individuals can act differently in different steps and can react to their partner's behavior. When this is true, the relatively sim- ple conclusions about which strategy may be favorable based on Table 1 do not necessarily hold. For example, if two individuals play an iterated Prisoner's Dilemma under these conditions, then reactive strategies like Tit-for-Tat (Axelrod, 1984) or win-stay, lose-shift (Nowak and Sigmund, 1993) can be favored over the single-iteration strategy of always-defect. Such repeated games allow for the phenomena of direct and indirect reciprocity (Trivers. 1971; Nowak and Sigmund, 1998) and trigger strategies which punish non-cooperation (Osborne and Rubinstein, 1994). These are examples of a general phenomenon of behavioral responsive- ness (Van Cleve and Alccay, 2014) which is one of a small number of mechanisms known to promote the evolution of costly cooper- ation (reviewed in Nowak, 20066; Van Cleve and Alccay, 2014). We do not consider reactive strategies or behavioral respon- siveness in this work. Individuals have one of two possible simple strategies: A or B as in Table 1. When both individuals are present, which is always the case initially, their survival probabilities in each iteration are given by the single-step version of Table 1. i.e. with a( 1), b(1), c(1) and d( 1). We will refer to these single-step survival probabilities as a, b, c and d. The n-step survival proba- bilities, a(n), b(n), c(n) and d(n), will depend on these as well as on the probabilities of survival when an individual is alone. Although the game always begins with two individuals, if one dies the other must continue. In the remaining steps, a loner plays a game against Nature. Specifically, a loner survives a single step with probability as if it has strategy A and probability do if it has strategy B. We are especially interested in cases in which the single-step, two-player game defined by a, b. c and d is of a different type than the n-step game defined by a(n), b(n), c(n) and d(n). Several previous works have considered games in which the random survival of individuals is an important factor and envi- ronmental conditions may be harsh. Eshel and Weinshall (1988) introduced a model in which payoffs are probabilities of survival. Payoffs were drawn randomly from a distribution, and the game was repeated with a fixed probability. As in the model we pro- pose, Eshel and Weinshall (1988) allowed that the game may con- tinue even if one individual dies. They considered optimal strategy choice by individuals under the assumption that individuals have perfect knowledge of the game's structure, including the distribu- tion of the payoffs. Eshel and Shaked (2001)described a similar sur- viva I game but included a general probability that both members of a pair survive, thus allowing for arbitrary synergistic effects of partnership (Hauert et al.. 2006; Kun et al.. 2006) within each step of the game. Eshel and Weinshall (1988) had assumed that pairwise survival probabilities were the products of two individual survival probabilities. Garay (2009) combined the idea of a survival game with the "selfish herd" theory of Hamilton (1971) to produce a model of cooperation between pairs of individuals in defense against re- peated attacks by a predator. Individuals were of two possible types, characterized by a probability of helping their partner in defense against attack. Garay (2009) derived the total survival EFTA00810743 40 J. Wakefey and M. Nowak/ Theoretical Population Biology 125 (2019)38-55 probabilities of individuals in different kinds of partnerships given a faced number of attacks, and used these to describe evolutionarily stable strategies (Maynard Smith and Price, 1973) in an infinite population. Using the example contrast of one attack versus eight attacks, this revealed cases in which helping could not invade a completely selfish population given a one-attack game but could invade if the number of attacks was larger (Garay. 2009). Kropotkin's notion that high levels of adversity could favor cooperation has also been studied using simulations of structured populations. Harms (2001) found that in a Prisoner's Dilemma co- operators could gain an advantage at the inhospitable margins of a population by colonizing patches cleared by the local extinction of defectors. More recent simulations of another model by Smaldino et al. (2013) also found cases in which cooperation can be favored despite the Prisoner's Dilemma. Specifically, if there is a high cost of living for all individuals which, if unchecked, would lead to the extinction of the population and if occasional interactions with cooperators are required for survival, then cooperation can be favored. As in Harms (2001). the success of cooperators in this case relied on their ability to form clusters (Smaldino et al., 2013). De Jaegher and Hoyer (2016) considered two game-theoretic models of the behavior and ecology of a pair of individuals, in which higher levels of adversity can change the type of the game. Their Model I includes a degree of complementarity which rescales payoffs for mixed-strategy pairs compared to same-strategy pain. and which could represent an environmental challenge for mixed pairs. Their Model 2 includes a number of attacks by a predator on individuals protecting a common resource. Cooperators are immune to attacks while defectors can sustain at most one at- tack before the common resource is lost. As in Garay (2009). the number of attacks indicates the level of environmental challenge. Cooperation may be disfavored when the number of attacks is small and yet become favored when the number of attacks is large. Depending on the other parameters in the model, however, a range of other switches between types of games may occur as the number of attacks increases. De Jaegher (2017) extended these results to multi-player games. Our concerns here are similar to those of Garay (2009) and De Jaegher and Hoyer (2016). We study how the structure of the two-player iterated survival game changes as a function of its parameters, in particular the number of steps it. We compare the conclusions for single-step games with those for n-step games. Because the payoffs which are probabilities of survival in this game decrease as n increases. n is a measure of adversity. We present general results for any level of adversity, then focus on the possible structures of large-n games. We find conditions under which any type of single-step game, be it a Prisoner's Dilemma, a Stag Hunt or a Hawk-Dove game. will become a Harmony Game as n increases. We identify three other large-n results, one of which shows the opposite: single-step advantages of cooperation may disappear as n grows. To facilitate comparisons, we define A throughout as the more cooperative type based on the single-step game, so that a > d. Thus, AA pairs survive better than BB pairs. We consider both infinite and finite populations but in neither case is there spatial or any other kind of structure. We do not develop a detailed biological. ecological or behavioral model. Akin to what is done in models of diploid viability selection, we describe the game and its results directly in terms of survival probabilities. Our results are applicable to a wide range of specific scenarios in which payoffs affect viability (as opposed to fertility or fecundity) and in which partnerships may either enhance survival or detract from it. It Evolutionary dynamics Comparing a(n) to c(n) and b(n) to d(n) in Table I indicates whether the more cooperative strategy A yields the higher or the lower payoff given partners of type A and B, respectively. In game theory and much of evolutionary game theory which treats pop- ulations, these two comparisons are relevant because individuals choose or modify their strategies based on their knowledge of the game and its outcomes (Sandholm, 2010). When individuals cannot alter or choose their behaviors but payoffs affect fitness. the same sorts of evolutionary models are used to predict changes in the frequency of hardwired behaviors in a population. In this section, we briefly summarize the evolutionary game dynamics of infinite populations. The replicator equation, in which general payoffs are inter- preted as contributions to individual fitness (Taylor and Jonker, 1978; Hofbauer et al., 1979; Zeeman, 1980; Schuster and Sigmund, 1983; Hofbauer and Sigmund, 1988, 1998) provides an evolution- ary setting for the classification of two-player games. In a survival game payoffs are fitnesses, specifically viabilities. If x is the relative frequency of A in an infinite population, the replicator equation gives the instantaneous rate of change = — x)(la(n)— c(n)Ix lb(n)— d(n)j(1 — x)). (1) Eq. (I) illustrates the evolutionary consequences of partner- dependent payoffs when pain are formed at random in proportion to the frequencies of A and B. When x x 1. so thatA is very common and nearly all partners are A, it is the sign of a(n) — c(n) that determines whether A is favored in the population. On the other hand, when x x 0, so that nearly all partners are B, the sign of b(n) — d(n) is what matters. Beyond this. Eq. ( I ) shows that for any value of x, it is the sign of (a(n) — c(n)ix lb(n) — d(n)1(1 — x) that determines whether A is favored. Thus. in a population and in evolution the relative magnitudesof both a(n)—c(n) and b(n)—d(n) are important. such that one may dominate the other at a given value of x. This is not something that can be gleaned directly from Table 1. Without mutation, the fates of A and B in this infinite population are entirely determined by Eq. (1). From any starting point, x will move deterministically toward one of three possible equilibria which are the solutions of k = 0 and which may correspond to the evolutionarily stable strategies mentioned in Section 1. Two monomorphic equilibria always exist. z = 0 and 1 = 1. The polymorphic equilibrium b(n) — d(n) — (2) b(n) — d(n) — a(n) c(n) might also exist, depending on the payoffs. It exists, which is to say that Eq. (2) gives a biologically meaningful value, when a(n), b(n), c(n) and d(n) are such that 0 < it < 1. Eq. ( I ) defines the same four types of symmetric two-player games. In the PD case, a(n) — c(n) < 0 and b(n) — d(n) < 0. and Eq. (1) shows that z < 0 for all values of x E (0, 1). Starting at any value x < 1, the frequency of A will decrease to zero. The polymorphic equilibrium given by Eq. (2) does not exist. In the SH case, a(n)—c(n) > 0 and b(n)—d(n) < O. ThenA isdisfavored when x is below the polymorphic equilibrium Si in Eq. (2) and favored when x > 2. The polymorphic equilibrium exists and is unstable. In the HD case, a(n) — c(n) < 0 and b(n) — d(n) > 0. Then A is favored when x <2. and disfavored when x > z. The polymorphic equilibrium exists and is stable. In the HG case. a(n)— c(n) > 0 and b(n) — d(n) > O. Then k > 0 and A is favored for all x e (0. 1). The polymorphic equilibrium does not exist. Fig. I shows examples of a Prisoner's Dilemma, a Stag Hunt and a Hawk-Dove survival game. depicting payoffs in their contexts (upper panels) and associated shapes of k (lower panels). A Harmony Game is not shown but would be another case of directional selection, like Fig. ID but with 1 > 0 for all x e (0, 1). These four types of games defined by the shape of ic over the interval 10, 1] are identical to what is observed in classical models EFTA00810744 J. Wakeley and M. Nowak/Theoretical Population Biology 125(2019)38-55 41 A D 2 Prisoners Dilemma 11. c(n) 096 a(n) ..................., a(n) 0.94 098 M^> o. -0.CO1 X -0.002 -0.003 0 0.25 0.5 0.75 x B 7' ow n) = 1. v E 094 z a o sa n) -0 c • n) BB M BA Ark - BB MBA AA Individual-Partner Pair Individual-Partner Pair Stag Hunt E 0.001 0. A -0.001 -0.002 -0.003 Stag Hunt 0 0.25 0.5 0.75 x C 1. k 0.98 0.98 :E 0.94 F 0. A -0.001 -0.002 0 0.25 0.5 0.75 1 Hawk-Dove cm) U^) a(n) d(n) BB MBA AA Individual-Paitner Pair Hawk-Dove Flg. 1. A and D: Prisoner's Dilemma with payoffs a = 0.97. b = 0.94, c = 0.99, d = 0.95 (a linear transformation of the classic R = 3, 5 = 0, 7 = 5, P = 1 of Axelrod 0984e.g.a = 0.94 + R/100).B and E: Stag Hunt, with a = 0.99.0 = 0.94, c = d = 0.97 (corresponding to a stag value of 0.05 and a hare value of 0.03 added to a baseline survival probability of 0.94). C and F: Hawk-Dove, with a = 0.97, b = 0.95, c = 0.99, d = 0.94 (corresponding to a cost of fighting of 0.03 and a resource value of 0.04, with a baseline survival probability of 0.95). Line segments in A, B and C connect the survival probabilities for B (left) versus A (right) when each occurs with given type of partner. The curves in D. E and F show k from Eq. (1). In diploid population genetics. these three cases fora are called directional selection (D), underdominance (E) and overdominance (F). of diploid viability selection (e.g. see section 4.2 in Nagylaki, 1992). The difference is that, barring atypical phenomena such as meiotic drive (Dunn, 1953; Sandler and Novitsky, 1957) or segregation - distortion (Sandler and Hiraizumi, 1960; Hard. 1974), standard diploid models always have b(n) = c(n). It is by allowing b(n) # c(n) that two-player games introduce the paradox of the Prisoner's Dilemma, in which c(n) > a(n) and d(n) > b(n) but a(n) > d(n). In order to have c(n) > a(n) and d(n) > b(n) in a standard diploid model it would be necessary to have a(n) e d(n). Accordingly. the Prisoner's Dilemma is the most stringent form of a cooperative dilemma (Doebeli and Hauert, 2005; Hauert et al., 2006; Nowak, 2012). A cooperative dilemma exists when (i) mutual cooperation results in a higher payoff than mutual non-cooperation, so that a(n) > d(n) when A represents cooperation, but (ii) there is incen- tive to be non-cooperative in at least one of three ways: (iia) c(n) > a(n), (iib) d(n) > b(n) or (iic) c(n) > b(n) (Nowak, 2012). The Prisoner's Dilemma includes all three of these incentives. Games with fewer barriers to cooperation, such as the Stag Hunt and the Hawk-Dove game, represent relaxed cooperative dilemmas. The Harmony Game involves no cooperative dilemma and presents no barrier to cooperation. 2. An n-step survival game between two players Our survival game always includes n iterations and begins with a pair of individuals. In each step, both might survive or one, the other or both might die. These same outcomes hold for the complete game, only the probabilities of surviving will be smaller. We consider two unconditional strategies. A and B. When both players are present, their individual survival probabilities are given by the single-step version of Table I with a(1) • a, b(1) • b, c(1) • c and d(1) • d. However, the next iteration must be faced by whoever has survived so far, until all n iterations are done. Thus, an individual might have to play alone. Then the survival probabilities become ao forA without a partner and do for B without a partner. In all of what follows. A will be the more cooperative strategy, meaning that a > d. The n-step survival game is a stochastic process with six possi- ble states: the paired states AA, AB and BB, the loner states A and B, and the state in which both individuals have died which we denote 0. It is convenient to represent a single iteration using the matrix AA AB BB A IA a2 0 0 2a(1 - a) AB 0 be 0 b(1 — r) BB 0 0 d2 0 A 0 0 0 a0 B 0 0 0 0 O 0 0 0 0 B 0 O (1 — a)2 1 r(1 — b) (1 — b)( 1 — c) 2d(1 — d) (1 — d)2 O 1 — ao (3) do 1 — do O 1 with entries equal to the transition probabilities among he six states. For example, the transition from state AB to state B means that the B individual survives and the A individual dies. In a single step, this occurs with probability c(1— b). Note that the transitions in Eq. (3) include the fates of both individuals but the individuals are not labeled Individual and Partner as they are in Table I. The process described by Eq. (3) is depicted in Fig. 2. State O is an absorbing state. There is no possibility of transition be- tween the paired states M. AB, and BB. Instead each of these feeds either straight into state 0, from which there is no escape. or into one of the loner states. A or B. and from there into 0. Therefore, transitions from any of the starting, paired states to absorption in state 0 involve either one or two changes of state. Further, all of the transitions that cannot occur in a single iteration (the 0 entries in the matrix) cannot ever occur regardless of the number of iterations. Because of this simple structure, the n-step transition probabilities can be calculated directly by conditioning on the times these transitions take place, i.e. on their positions in the sequence of n iterations. It is also possible to compute the n-step transition probabilities using standard techniques for Markov chains, and this provides a useful framework for decom- posing the process and describing its behavior when n is large. Details of the matrix approach are given in the Appendix. but two key features of it inform our presentation. The first is the fact of the absorbing state (0) in which both individuals have died. It will be reached eventually, meaning in the limit n oo. For large n the game process will be just the approach to this state. Second, when n is large, the rate of approach to state 0 will be given by the largest non-unit eigenvalue of the matrix in Eq. (3). Because it is an upper triangular matrix, the eigenvalues are simply the entries on the diagonal. By convention, we call the largest of these 2. = land EFTA00810745 42 J. Wake f ey and M. Nowak/Theoretical Population Biology 12S (2019)38-55 0 ® 0 a0 rvR 0 .® Flg. 2. Flow diagram of the stochastic process given by the matrix of Eq. (3).Arrows show all possible transitions among the six states (A4. At BB. A. B. 0).States AA, AB, BB. A and B are transient. State 0 is absorbing. The process always begins in one of the three states on the left, M. AB or BB. After n iterations. it may be in any of the six possible states. note that this corresponds to the eventual absorption in state 0. In all, we have = 1, a2, bc, d2, ao, . (4) Following the discussion of Table 1 and Eq. ( I), we expect the fates of A and 8 in this iterated game to depend on the relative magnitudes of a versus c and b versus d. Eq. (4) suggests that their fates will also depend on the relative magnitudes of the pair survival probabilities. a2. be and d2. and on the loner survival probabilities. ao and 4. and that this dependence may be especially strong when n is large. We compute the n-step pair survival probabilities directly as Prob(AA —* AA) = a2" Prob(AB —a. AB) = (bcr Prob(BB BB) = dm. (5) (6) (7) Then, by considering the possibility that one of the individuals might die in step 1 < i < ri and the other individual survives to the end of the game, we have re Prob(AA -* A) = E (a2)1-1 2a(1 - a)arl = 02. 4) 22a(1 — a) — an Prob(AB -> A) = Prob(AB B) = E(bc)i- 1 b(1 - c)arl a-t ((kr d°)1_ c) It DbCr ICO — LOCIrj i-t CO — b) (ocr - dfrc doProb(BB 8) = E 2d(1 - (14)I = (d2n A„ ) 2d( 1 d) d2 The only other possibility is that neither individual survives, so we also have Prob(AA 0) = 1 —Prob(AA AA) — Prob(AA —* A) (16) Prob(AB —a. 0) = 1 — Prob(AB AB) —Prob(AB A) Prob(AB B) (17) Prob(BB —* 0) = 1 — Prob(BB —a. BB) — Prob(BB B). (18) In the Appendix we show how these probabilities are obtained using techniques for Markov chains. Eqs. (8) through (15) make it clear that this paired survival process is one in which the fortunes of individuals may change. perhaps drastically depending on the values of ao and do relative to a, b, c and d. Consider starting state AB. Eq. (12) shows how the probability of being in state Bat the end of n iterations is computed. In words, both individuals survive for some time (i -1 steps), then A dies, and B survives the rest of time (n - i steps). If A dies, B trades its individual survival probability of c. which B enjoys in the presence and A, fora loner survival probability of do. It could be that do c c, making B worse off after A dies. In fact, B's fate is closely tied with A's because this switch could occur quickly if A's survival probability, b, is small. Of course. while there is a cost to B when A dies (assuming 4 c c), the death of A in this partnership also represents a rather direct disadvantage toA. In order to understand whether A or B will prevail in evolution, it is necessary to account for the full dynamics of reproduction in a population, with fitnesses that are determined by this game. We will take this up in Section 3. In the n-step game, the differences a(n) - c(n) and b(n) — d(n) give the conditions under which A is favored. The n-step payoffs. or survival probabilities, are computed by accounting for the two ways an individual may survive the game. An individual survives if both it and its partner survive or if it survives but its partner dies. The total probabilities of individual survival in each kind of partnership are a(n) = Prob(AA AA) + Prob(AA -o A)/2 _a2" a—ao + a(1 — a) a2 — ao ao — a2 b(n) = Prob(AB AB) + Prob(AB A) b ao b(1 — c) = (kr be - ao + 4 ao — be c(n) = Prob(AB AB) Prob(AB B) = (bcr c — do +4 c(1 — b) be-do do — bc d(n) = Prob(BB BB) + Prob(BB B)/2 — d2" d dO + 4d — d) d2 — 4 4 - d2 The transitions AA A and BB B are adjusted by a factor of 1/2 because they are equally likely to happen by the death of the partner as by the death of the focal individual. Eqs. (19) through (22) are our general results, namely the n-step survival probabilities for individuals of types A and B given each kind of initial partnership. They are exact for any values of a, b, c, d, ao and do greater than zero and less than one, and for any number of iterations n 1. As expected, when n = 1, they reduce to the single-step survival probabilities a(1) = a, b( 1) = b, c(1) = c and d(1) = d. The two key differences in payoff are then a(n) - c(n)- am a ao Go — a2 + 4 — a)a ao (15) (19) (20) (21) (22) EFTA00810746 J. Waketey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 43 — _ (krc__ c(I — b) be—do ° do — bc b(n)— d(n)= (bon + aiot bt c) bc — a ao o aol — bc do_ d2nd_ d(1 — d) d2 — do ° do — d2 Each term in Eqs. (23) and (24) includes a factor A? for one of the non-unit eigenvalues (a2, bc, d2. a0. do). These factors are positive. The denominator of each term is the difference between two eigenvalues. If we know the eigenvalues, we know whether the de- nominators are positive or negative. In some cases the numerator might be negative, but by the definition of a, b, c, d, ao and do as probabilities, the numerator will be positive if the denominator is positive. Thus Eqs. (23) and (24) are written so that the sign of each term indicates whether it favors A (+ ) or disfavors A (—) when the denominator is positive. We do this to facilitate the analysis of large n, in which case a(n) — c(n) and b(n) — d(n) will come to be dominated by the terms involving the largest non-unit eigenvalue. Another way to write Eqs. (23) and (24) is on) — c(n) = (a — ao)°a2" — an (c do)(bcr - cag a2 — ao be—do + a; — (25) b(n)— d(n)= (b — ao)(kr _4 2* — 4 (d do) bc ao d2 — do + ao — d; (26) which brings attention to the possible survival value of having a partner and of being A versus B without a partner. Thus, a — ao and c — do in Eq. (25) are the increments to the survival probabilities of A and B when each has partner A compared to when they are alone. Likewise. b — ao and d — 4 in Eq. (26) are the corresponding increments for A and B with partner B versus alone. Note that the fractions multiplying these increments are always positive. All else being equal, a(n)—c(n) increases witha—ao > 0 but decreaseswith c — do > 0, and b(n)—d(n)increases with b — ao > 0 but decreases with d -4 > O. Negative increments have the opposite effects. For example, if B suffers by having a B partner compared to being alone, then d—do < 0. which in turn favors A via the difference b(n)—d(n). The last two terms of both Eqs. (25) and (26) quantify the n-step effects of differences in the loner survival probabilities, ao and do. If ao > do the contribution to both a(n) — c(n) and b(n) — d(n) is positive, and if 4 < do it is negative. These five possible increments to individual survival, namely a —ao, b—ao, c —do, d—do and ao—do. in Eqs. (25) and (26 ) are helpful for understanding some of our results and the structure of survival games generally. However, it is also clear from the additional presence of the pair-state eigenvalues in Eqs. (25) and (26) as well as in Eqs. (23) and (24). that these five increments do not entirely determine the prospects for cooperation. The relative magnitudes of the five eigenvalues will also be important. This may be seen in the denominators in Eqs. (25), (26), (23) and (24), which contrast the survival of pairs to the survival of loners: a2—ao, be —ao. bc —do, d2-4. The preceding expressions, Eqs.(23)and (24). are especially useful for predicting the structure of the game when n is large. (23) (24) 3. Four types of prolonged survival games Here, we present analytical results for large n and use our exact results to illustrate how the prospects for A depend on n for intermediate values. The analysis of large n allows us to answer questions such as: Is there a number of iterations beyond which A is unambiguously favored? More generally, we ask how the game might change for A, for better or for worse, as the number of iterations increases. We present results both for a(n) — c(n) and b(n) — d(n) and in terms of the unified predictions of Eq. ( I ). Eq. (1) is a standard replicator equation for a symmetric two- player game. In the Appendix, we show how it may be derived for the n-step survival game. This has two notable features: the game works by removing individuals from the population instead of affecting their rates of reproduction and the derivation follows pairs of individuals rather than single individuals. Also in the Appendix, we take a discrete-time, population -genetic approach to show that the change in frequency of A over one generation is Ax = x(1 — x)0a(n)— c(n)1x (b(n)— d(n)1(1 — x))/th, (27) which is identical in form to ic in Eq. ( I ) but scaled by the average fitness, tb. Eq. (27) is directly comparable to classical results for diploid viability selection. Importantly, tun 0. Therefore, all of the conclusions concerning the ultimate fate of the cooperative type A in the population, based on the sign of the change in frequency. will be the same whether one appeals to Eq. ( I ) or Eq. (27). Each of the n-step payoffs a(n), b(n), c(n) and d(n) depends on two of the non-unit eigenvalues, and each eigenvalue is raised to the power n. Thus a(n), b(n), c(n) and d(n) all decrease to zero as n tends to infinity. Surviving more steps is less likely than surviving fewer steps. As a consequence, the two key fitness differences. a(n) — c(n) and b(n) — d(n), and the instantaneous rate of change. also approach zero in the limit 17 -. co. We study the approach to this neutral limit in order to understand whether increasing n fundamentally alters the prospects for cooperation. For example. if ic < 0 when n = 1 but the neutral limit z = 0 is approached from above, then there exists a value of n beyond which k > 0. In this case. the single-step game favors the non-cooperative type B but large-n games favor the cooperative type A. In the following subsections, we describe four main categories of large-n. or prolonged, survival games. These are characterized both in the usual way by the relative values of a, b, c and d, that is by the structure of the single-step game. and by the rank order of non-unit eigenvalues a2, bc, d2 ao and do. We have defined the cooperative strategy A throughout by a > d, which guarantees that a2 > d2. Thus d2 will never be the largest non-unit eigenvalue. The first two types of prolonged survival games we describe are those for which a2 is the largest and those for which bc is the largest. The third and fourth types of prolonged survival games are ones in which lone individuals survive with high probability. Simplifi- cations emerge when 17 is large because the n-step payoffs become dominated by their largest terms. We present approximations for large n, showing just the leading terms of a(n)—c(n)and b(n)—d(n), and of it. 11. Games in which cooperation will prevail The first case we consider is when having a partner significantly enhances survival and it is theAA type not theAB type that survives best This is case in which a2 is the largest non-unit eigenvalue: a2 > bc, ao, do. Thus, the probability that both members of an AA pair survive a single iteration is greater than the corresponding probability for any other pair and for either type of lone individual. The condition a2 > bc may be true of any kind of two-player game, but the chance it is depends on the game. One way to quantify this is toconsider the total parameter space of two-player. single-step survival games. Because a, b. c and d are probabilities and here it is assumed that a > d, this space is equal to half of a four-dimensional hypercube. Further, since a > d implies a2 > d2, there are just three possible relations of the pair-state eigenvalues, a2, bc and d2. Either bc > a2, a2 > bc > d2 or d2 > bc. The latter two satisfy the condition a2 > bc. Table 2 lists the proportions of the total parameter space satisfying the criteria for each type of single-step game. In addition, Table 2 shows the percentages EFTA00810747 44 J. WaRefry and M. Nowak/ Theoretical Population Many 125 (2019)38-55 Table 2 Partitioning of the four-dimensional hypercube into fractions meeting the game conditions in the first column. The types of single-step games are abbreviated PD, SH. HD and HG for Prisoners Dilemma, Stag Hunt, Hawk-Dove game, and Harmony Game. Exact results were obtained by integration in Mathematica (Wolfram Research. Inc., 2018). Percentages are rounded to the nearest 0.1%. Note that in all cases a > el by definition, so it is always true that 02 > d2. Game type PD: a <c,b<d with a > (b + c)/2 with a > (b + c)12.d < (b+ c)/2 SH: a>c,b<d HD: a < b > d HG: a>c,b>d Proportion Order of pair-state eigenvalues be >a,a2 > bc > d2el? > bc 1/12 10% 40% SO% (3/4 of 1/12) 0% 40% 60% (1/2 of 1/12) 0% 60% 40% 1/4 0% 11.1% 88.9% 1/4 80% 20% 0% S/12 10% 663% 23.3% of the game-specific parameter regions corresponding to each of the three possible relations of the pair-state eigenvalues. So, for example, 90% of PD-type games have az > bc, compared to 20% of HD-type games. When the single-step game is a canonical Prisoner's Dilemma, it is always true that az > bc. This is due to the assumption that the reward payoff must be larger than the average of the temptation payoff and the suckers payoff (see, e.g. Rapoport and Chammah. 1965; Axelrod, 1984), in which case we have 02> b c )2- bc +(b c )2(- (- > bc. (28) 2 In what follows, it will be important to know which is the second- largest non-unit eigenvalue, especially in the consideration of fi- nite populations in Section 4. Thus, the two cases az > bc > dz and d2 > bc are distinguished in Table 2. We find that canonical Prisoner's Dilemmas are 40%az > bc > dz and 60%d2 > bc. If we also require that the punishment payoff must be less than the average of the temptation payoff and the sucker's payoff (e.g. see p. 219 of Cressman. 2005). this 40 : 60 split is reversed. It would be possible to expand Table 2 by including ao and do and integrating over the six-dimensional space of all parameters. There would be more than three relations among the five non-unit eigenvalues. We do not pursue this here. For the prolonged games considered in this section and in Section 3.2, it is taken for granted that neither ao nor do is the largest non-unit eigenvalue. In contrast. in Sections 3.3 and 3.4 it is taken for granted that one of ao or do is the largest non-unit eigenvalue or they both are and they are equal. If we were to include a0 and do in Table 2, we expect they would often be larger than az and bc due to the fact that these pair-state eigenvalues are products of probabilities. There is just one term in Eqs. (19) through (22) which includes the factor a2.. corresponding to what is assumed here to be the largest non-unit eigenvalue, az. It is in the expression for a(n). Thus. we have a(n) — c(n) r a2„ > 0 a2 ao when n is very large. Eq. (29) gives the approximate value of a(n)— c(n) as it approaches zero in the limit n co. This result shows that whatever value or sign a(1) — c(1) = a — c might have in a single iteration, there exists a value of n above which a(n) — c(n) will be greater than zero. The leading term in the other key difference. b(n) — d(n), will depend on which of bc, dz. 00 or do is the next-largest eigenvalue. Consider, for example, a game which strongly enhances survival, such that az, bc, d2 > ao, do. If bc is the next largest eigenvalue after az, then from Eq. (24) we have b(n)— d(n) e- (bcr bc — ao > 0. (30) b ao (29) lim 1?-.00 a(n) — c(n) Thus, when n is very large, the advantage in survival that A has over B when the partner is A will be much greater than whatever difference there is in survival between A and B when the partner is B. One may argue from an evolutionary standpoint that the large-n Stag Hunt game, with a(n) — c(n) in Eq. (29) and b(n) — d(n) in Eq. (31), will be a very relaxed cooperative dilemma. Specifically. owing to Eq. (32), a large-n Stag Hunt of this sort will have its unstable polymorphic equilibrium x in Eq. (2) close to zero, with the result that A will be favored over most of the range of x. Finally, when n is very large the replicator equation, Eq. (1), becomes In this case, the large-n game becomes a Harmony Game in whichA is unambiguously favored. On the other hand, if the second largest eigenvalue is dz, then from Eq. (24) we have b(n)— d(n) dead—d° e0. (31) crz — do In this case, the large-n game becomes or remains a Stag Hunt game, so A does not become unambiguously favored. The chances of being in either of these two sub-cases are given in Table 2. Similar arguments can be applied when ao or do is the second- largest non-unit eigenvalue. Then Eq. (24) shows that for large iv, b(n) — d(n) > 0 if ao is the second-largest, and b(n) — d(n) < 0 if do is the second-largest. Regardless of which is the second-largest non-unit eigenvalue, if az is the largest then a(n) — c(n) will be much greater than b(n) — d(n) when n is large. This may be expressed as b(n) — d(n) (32) a— a° k rr x2(1 — x)a2"— 0 >0 for all x e(0, 1). (33) We conclude that, if n is large enough in the case of this first type of prolonged survival game, the more cooperative type A will be uniformly favored in an infinite population. This result requires only that az is the largest non-unit eigenvalue. It is robust to differences in the loner survival probabilities 00 and do. As long as az is large enough, the non-cooperative type B cannot be rescued by an advantage in loner survivability. Table 2 shows that az > bc for the great majority of possible single-step survival games, the exception being games of the Hawk-Dove type, of which only 20% will have this property. Fig. 3 illustrates how the two key differences in payoff. a(n) — c(n) and b(n) — d(n), change as n increases for two examples based on the Prisoner's Dilemma. In Fig. 3A, the survival probabilities for individuals with partners are a = 0.8. b = 0.6, c = 0.9. d = 0.7. Thus. the cooperative type A incurs an 11% loss of fitness compared to B when the partner is A and a 14% loss when the partner is B. The survival probabilities for loners are much smaller and identical: ao = do = 0.3. The eigenvalues are az = 0.64. bc = 0.54. = 0.49, ao = 0.3 and do = 0.3. This game starts as a Prisoners EFTA00810748 J. Wakeley and M. Nowak/Theoretical Population Biology 125 (2019)38-55 45 A Payoff Difference Number of iterations. n B Number of Iterations, n fig. 3. Values of the two key differences. a(n) — c(n)and b(n) — d(n). as a function of the number of iterations. for two different survival games which are Prisoner's Dilemmas when n = 1. Panel A: a = 0.8. b =0.6,c =0.9.d= 0.7 and ao = do = 0.3. Panel 6: a= 0.9.6 =0.8,c =0.95. d = 0.85 and ao =do = 0.65. A 0.04 PO 8 ,go z 0. 0.02 0 -0.02 -0.04 -0.06 -0.08 HD HG 1 10 20 30 40 SO 60 70 80 • a(n)-c(n) • b(n)-d(n) Number of Iterations, n B 0.04 PD 8 0.02 0. -0.02 I -0.04 (5 -0.06 -0.08 SH HG • • a(n)-c(n) • b(n)-d(n) 1 10 20 30 40 60 60 70 80 Number of Iterations, n Fag. 0. Values °IG(n) — c(n) and b(n) — d(n) over n = 1...80 iterations for two games which begin as Prisoner's Dilemmas at n = I. In both cases, the loner survival probabilities are ck, = 4 = 0.9. PanelA: a = 0.97.6 = 0.94. c = 0.99,d = 0.95; these are the same as in Fig. IA and D. Panel It a = 0.9733.6 = 0.94.c = 0.99, d = 0.9567: this differs from panel A only in that a and d are spaced evenly between c and b. Dilemma (PD) when n = 1. For intermediate numbers of iterations. specifically n = 3 and n = 4, the game changes to one in which A is disfavored when rare (b(n) — d(n) < 0) but favored when common (a(n) — c(n) > 0), as in the Stag Hunt (SH). Then as n grows it becomes a game in which A is uniformly favored, as in the Harmony Game (HG). Fig. 3B displays results for a very similar game, but one in which the single-iteration probabilities of individual survival are shifted closer to 1 by a factor of 1/2. That is, a = 0.9, b = 0.8. c = 0.95. d = 0.85 and ao = do = 0.65. Now the eigenvalues are a2 = 0.81. be = 0.76, d2 = 0.7225. ao = 0.65 and do = 0.65. Overall, the picture is similar to Fig. 3A, with the three phases PD then SH then HG. But now in moving from n = I ton = 2. the selection against A becomes stronger rather than weaker. This is more in line with the usual notion of iterated games, in which payoffs are assumed to accrue additively. In survival games, however, this decrease cannot continue because all games become neutral as n approaches infinity. Fig. 3 may be compared to Fig. 4 (Model 2AIII) of De Jaegher and Hoyer (2016). with their number of attacks being analogous to n. A point of contrast between our model based on multiplying single-step survival probabilities and the behavioral and ecological scenario, Model 2 of De Jaegher and Hoyer (2016) is that in our model the payoff differences a(n) — c(n) and b(n) — d(n) may be non-monotonic. as they are in Fig. 3, whereas these are always monotonic in Model 2 of De Jaegher and Hoyer (2016). We will return to Model 2 of De Jaegher and Hoyer (2016) and make further comparisons in the Discussion. Because survival probabilities decrease with each iteration. n is a measure of the adversity faced by individuals, or the harshness of the environment Mother way to measure adversity is directly in terms of single-step survival probabilities. Adversity is greater when a, b, c, d, ao and do are smaller. Fig. 3 provides an example of how an increase in this sort of environmental challenge affects the outcome of the game. Single-step survival probabilities are smaller in