From Actions to Habits
Neuroadaptations Leading to Dependence
Henry H. Yin, Ph.D.
HENRY H. YIN, PH.D., is an assistant professor in the Department of Psychology and Neuroscience, Duke University, Durham, North Carolina.
Recent work on the role of overlapping cerebral networks in action selection and habit formation has important implications for alcohol addiction research. As reviewed below, (1) these networks, which all involve a group of deep-brain structures called the basal ganglia, are associated with distinct behavioral control processes, such as reward-guided Pavlovian conditional responses, goal-directed instrumental actions, and stimulus-driven habits; (2) different stages of action learning are associated with different networks, which have the ability to change (i.e., plasticity); and (3) exposure to alcohol and other addictive drugs can have profound effects on these networks by influencing the mechanisms underlying neural plasticity. Key words: Addiction; alcohol and other drug (AOD) dependence; AOD use behavior; brain; neuroadaptation; cerebral networks; neural pathways; basal ganglia; neural plasticity
Addiction is a series of misguided actions. Yet how the brain selects and generates actions has received surprisingly little attention in addiction research. In recent years, considerable progress has been made in identifying the neural circuits responsible for the control of goal-directed actions and habit formation. It is becoming increasingly clear that drugs of abuse can alter these neural pathways. This article discusses the mechanisms underlying reward-guided action selection and their implications for research on alcohol addiction.
The Organization of Cortico-Basal Ganglia Networks
Understanding how the brain generates actions must begin with a discussion of the cortico-basal ganglia networks.1 [1This and other technical terms can be found in the Glossary, pp. 345–347.] These networks form a hierarchy for motivated behavior (Swanson 2000; Yin and Knowlton 2005, 2006), which consists of variations on a basic motif, a prototypical network critical for behavioral selection. In this network, glutamatergic (excitatory) projection neurons from the cerebral cortex, a highly layered structure, send axons to the nuclei underneath, commonly known as the basal ganglia, which contain γ-aminobutyric acid (GABA)-ergic (inhibitory) projection neurons. The inhibitory outputs from the basal ganglia, in turn, are directed at downstream structures in the brainstem and in various thalamic nuclei whose projections reenter the cortex.
There is reason to believe that the basal ganglia circuits and their intrinsically generated oscillations are responsible for the generation and selection of behavioral programs; and the variations in patterns of connectivity and in the expression of key proteins like membrane receptors may be tailored for different types of global control processes, as described below (Gerdeman et al. 2003; Yin and Knowlton 2006). A striking feature of such control processes is that they can be measured behaviorally using specific tests.
As recent research has shown, normal mechanisms of learning and memory are usurped by exposure to addictive drugs, so that instead of serving normal biological needs they defect to the purpose of drug seeking (Hyman et al. 2006). There is no consensus, however, on precisely what type of learning process is usurped by addictive substances. Current hypotheses focus on the enhancement of craving, or incentive sensitization (Robinson and Berridge 2003), and on the avoidance of harmful consequences of withdrawal, or allostasis (Le Moal and Koob 2007). These hypotheses largely neglect the central issue of how actions are selected. One reason for this neglect is that the chief behavioral measures in the field (e.g., self-administration and conditioned place preference2 [2Conditioned place preference is a commonly used technique to evaluate preferences for environmental stimuli that have been associated with a reward. In general, this procedure involves several trials where the animal is presented with the reward (e.g., food or the effects of a drug of abuse) paired with placement in a distinct environment containing various cues (e.g., tactile, visual, and olfactory). When later tested in the normal state, approaches and the amount of time spent in the compartments previously associated with reward serve as an indicator of preference and a measure of reward learning.]) lack sufficient analytical power to isolate contributions of distinct neural networks. As discussed below, a major challenge in addiction research is to understand the mechanisms underlying these behavioral control processes and how they are affected by exposure to alcohol and other drugs.
Three Modes of Behavioral Control
What, then, are these control processes and why are they so important for understanding alcohol addiction? In the study of behavior guided by rewards (i.e., appetitive behavior), researchers are now able to distinguish three major modes of behavioral control with simple experimental tests. These three modes are Pavlovian approach,3 [3In Pavlovian conditioning, a previously neutral stimulus, such as a tone or light, becomes associated with an unconditional stimulus, such as food, to the extent that it will, by itself, evoke a response related to the unconditional response. This new response is called the conditional response ] goal-directed action, and habit. Although these are rather broad classes of behavioral control with simple operational definitions, they shed considerable light on the integrative functions of the cortico-basal ganglia networks.
Preparatory appetitive Pavlovian behaviors (e.g., approaching location of reward and stimuli that predict reward) and goal-directed instrumental actions are both controlled by the anticipation of the reward. For both, reducing the value of the reward (e.g., by selective satiety, in which the animal is sated on the particular reward offered but not other rewards) or taste aversion induction (in which a particular food is paired with an injection of lithium chloride that results in gastric discomfort) can reduce performance (Colwill and Rescorla 1985; Yin and Knowlton 2002). In both, too, performance is controlled by a predictor of reward and the reward itself. But for Pavlovian approach, the predictor of reward is a stimulus arranged by the experimenter and entirely independent of the animal’s behavior, whereas in instrumental behavior the predictor is the self-generated action by the animal. This distinction is revealed by direct manipulation of the postulated contingencies (e.g., increasing the probability of reward independent of the predictor, be it a particular action in the case of instrumental learning or a stimulus in the case of Pavlovian conditioning) (Hammond 1980; Schwartz and Gamzu 1977). Manipulating the relationship between stimulus and outcome specifically affects Pavlovian behavior, whereas manipulating the action–outcome relationship specifically affects instrumental behavior (Dickinson 1994, 1997; Schwartz and Gamzu 1977).
Habit, a third mode of behavioral control, is not affected by changes in outcome value. Habits persist even if the reward becomes less attractive or if the action is not necessary to earn the reward. Unlike appetitive Pavlovian conditional responses, which are controlled by the stimulus–outcome contingency, all instrumental behaviors initially are goal directed and controlled by the action–outcome contingency. The performance of such actions is exquisitely sensitive not only to its causal efficacy (i.e., by the extent to which the outcome depends on the action) but also to the value of the ensuing consequence (Dickinson 1985; Dickinson and Balleine 1993; Yin and Knowlton 2005, 2006). Under certain conditions, such as extensive training, however, such goal-directed actions are transformed into habits.
As shown by a number of studies in the last two decades, habitual control of instrumental behavior emerges gradually with repeated performance and is relatively unaffected by changes either in outcome value (e.g., devaluation) or in instrumental contingency (Adams 1982; Adams and Dickinson 1981). Thus, once lever pressing for a sucrose reward becomes habitual in this sense, induced taste aversion or unlimited exposure to sucrose prior to a probe test––conducted with the lever extended but without the presentation of a reward––will not reduce the rate of lever pressing compared with controls that did not receive the devaluation treatment.
This basic distinction is supported by a series of studies from Yin and colleagues (2004, 2005a,b, 2006), who established a functional dissociation between associative and sensorimotor striata in the control of instrumental actions. They showed that the associative or medial striatum (similar to most of the caudate nucleus in primates) is critical for the early, goal-directed stage of action learning, whereas the sensorimotor or lateral striatum (similar to the putamen in primates) is more critical for the later, more habitual stage (see figure 1). Together with studies of other structures in these networks (Corbit and Balleine 2003; Corbit et al. 2001, 2002, 2003), this line of research has established that control over instrumental behavior lies with the associative cortico-basal ganglia network in the early stages of learning but switches to the sensorimotor cortico-basal ganglia network in later stages (Yin and Knowlton 2005, 2006; Wickens et al. 2007a,b).
With respect to the neural adaptations that lead to alcohol dependence, then, the key question is, Which control processes are affected by alcohol as casual drinking becomes compulsive drinking? Drugs of abuse can enhance Pavlovian approach behavior (e.g., approaching environmental stimuli associated with reward), which is largely mediated by the ventral striatum (nucleus accumbens) and the associated cortico-basal ganglia circuit (Corbit et al. 2001; Day et al. 2007; Hyman et al. 2006; Parkinson et al. 2000). In fact, because of the inability to isolate Pavlovian from instrumental modes of behavioral control, current research on addiction has focused almost exclusively on the nucleus accumbens; but we now know that this is only part of the story. As reviewed above, the cortico-basal ganglia networks, which involve the medial (associative) and lateral (sensorimotor) striatal regions above the nucleus accumbens, are responsible for instrumental control processes (see figure 2). Thus, previous work has, by and large, neglected the contributions of the associative and sensorimotor networks in the study of addiction.
Implications for Alcohol Addiction
A trademark of habitual behavior is that the expected value of the outcome does not affect the behavior. It is as if the value of the outcome has become fixed, so that even if alcohol consumption is associated repeatedly with aversive consequences, such consequences do not alter the performance of the action itself. For this reason, habits have been viewed by some researchers as an intermediate stage before the development of compulsivity (Everitt and Robbins 2005). In the case of alcohol consumption, such a model would emphasize first a shift from casual drinking to habitual drinking, followed by a shift to compulsive drinking. Nonetheless, although the process of habit formation bears a certain resemblance to addiction, addictive behaviors are not the same as enhanced habits (Yin and Knowlton 2005). At first glance, both develop after repeated exposures, and both are insensitive to outcome devaluation. But there are important differences as well. For example, habitual behavior is easily extinguished when the reward is no longer delivered, whereas compulsive behavior is very resistant to extinction (Mowrer 1960). Thus, whereas decades of work has identified the distinct control processes outlined above, we still have little understanding of how these processes interact in producing normal behavior, which rarely is dominated by one process alone. Compulsive behavior, for example, is probably an amalgamation of Pavlovian and instrumental processes.
Figure 1. Schematic illustration showing cortico-basal ganglia networks in relation to serial adaptation. A shift from the associative to the sensorimotor cortico-basal ganglia network is observed during habit formation.
SOURCE: Yin and Knowlton 2006.
Appetitive Pavlovian instrumental interactions can take a number of forms. In all, stimuli with incentive value increase the likelihood of action for reward. Although conditioned reinforcement sometimes refers to action-contingent stimuli, Pavlovian instrumental transfer always measures the effect of action-independent stimuli. In conditioned reinforcement, cues produced by instrumental actions can form associations with the reward; and after repeated pairing they become viable reinforcers for the actions (Mowrer 1960). For compulsive drinking, conditioned reinforcement (the feel of the bottle, the taste of alcohol) can play an important role. In Pavlovian instrumental transfer, cues that independently predict reward can elicit central motivational states that enhance instrumental performance. For example, the environmental stimuli associated with drinking (e.g., the sight of a bar) can trigger craving for alcohol and, in turn, alcohol-seeking behavior. Much of the power of advertising, for example, probably derives from the ability of Pavlovian stimuli to trigger motivational states that enhance the selection of certain actions.
The nucleus accumbens is known to play a critical role in Pavlovian instrumental transfer; lesions of this area selectively abolish transfer (Corbit et al. 2001). Interestingly, recent work (Corbit and Janak 2007) has also implicated the dorsal striatum. The sensorimotor striatum in particular appears to play a critical role in the ability of reward-predicting cues to enhance instrumental lever pressing. Such results suggest the possibility of interactions between ventral and more dorsal striatal regions in Pavlovian instrumental interactions.
The Role of Plasticity
It is possible that all addictive drugs, including alcohol, can affect the capacity for change (i.e., plasticity) in the cortico-basal ganglia networks, thereby altering normal learning processes that are critical for selecting and controlling actions. Although plasticity at all parts of the cortico-basal ganglia network may be involved in addiction, the striatum appears to be the critical node where massive excitatory inputs are transformed into an inhibitory output that ultimately controls behavior (Lo and Wang 2006; Nauta 1989). The glutamatergic transmission can be altered, both presynaptically, in the amount of glutamate released from the axon terminal, and postsynaptically, in the trafficking and expression of various glutamate receptors
Figure 2.The cortico-basal ganglia networks. An illustration of the major corticostriatal projections and dopaminergic projections in terms of the four major cortico-basal ganglia networks and their corresponding behavioral functions. Emphasis is placed on the spiraling midbrain–striatum–midbrain projections, which allows information to be propagated forward in a hierarchical manner. Note that this is only one possible neural implementation; interactions via different thalamo–cortico–thalamic projections also are possible (Haber 2003).
SOURCE: Yin and Balleine 2008.
Recent studies (Jedynak et al. 2007; Nelson and Killcross 2006; Porrino et al. 2004) show that exposure to drugs like cocaine and amphetamine can result in significant plasticity in the striatum and potentially accelerate the initial shift from actions to habits. Alcohol may produce similar effects. Acute application of alcohol to brain slices can reverse the direction of plasticity in the associative striatum (Yin et al. 2007). Thus, a train of stimulation that normally leads to increased activity in a striatal region critical for goal-directed actions results in long-term depression instead. One interpretation of these results suggests that the reversal of striatal plasticity could promote habit formation by reducing the overall synaptic strength of the associative striatum, which is a critical component of the brain’s system for the control of goal-directed actions. Previous work (Corbit and Balleine 2003; Corbit et al. 2003; Yin et al. 2004, 2005a,b, 2006) showed that disrupting the network for goal-directed actions results in a switch to a habitual mode of behavioral control, and vice versa. It remains to be seen if alcohol is able to promote habit formation in vivo by targeting this mechanism.
The preliminary conceptual framework and the behavioral tests discussed here suggest a number of promising avenues for future study. Researchers can measure, for example, the effects of alcohol on each of these control processes, on their interactions, and on the underlying neural substrates at the cellular level as well as at the level of neural circuits. Further work also can investigate the effects of particular factors (e.g., stress) on susceptibility to addiction and to relapse using the same strategy. The extent of our ignorance in these areas is considerable. An exciting and challenging path lies ahead.
The author declares that he has no competing financial interests.
Adams, C.D. Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology 33b:109–122, 1982.
Adams, C.D., and Dickinson, A. Instrumental responding following reinforcer devaluation. Quarterly Journal of Experimental Psychology 33:109–122, 1981.
Colwill, R.M., and Rescorla, R.A. Postconditioning devaluation of a reinforcer affects instrumental responding. Journal of Experimental Psychology: Animal Behavior Processes 11:120–132, 1985.
Corbit, L.H., and Balleine, B.W. The role of prelimbic cortex in instrumental conditioning. Behavioural Brain Research 146:145–157, 2003. PMID: 14643467
Corbit, L.H., and Janak, P.H. Inactivation of the lateral but not medial dorsal striatum eliminates the excitatory impact of Pavlovian stimuli on instrumental responding. Journal of Neuroscience 27:13977– 13981, 2007. PMID: 18094235
Corbit, L.H.; Muir, J.L.; and Balleine, B.W. The role of the nucleus accumbens in instrumental conditioning: Evidence of a functional dissociation between accumbens core and shell. Journal of Neuroscience 21:3251–3260, 2001. PMID: 11312310
Corbit, L.H.; Muir, J.L.; and Balleine, B.W. Lesions of mediodorsal thalamus and anterior thalamic nuclei produce dissociable effects on instrumental conditioning in rats. European Journal of Neuroscience 18:1286–1294, 2003. PMID: 12956727
Corbit, L.H.; Ostlund, S.B.; and Balleine, B.W. Sensitivity to instrumental contingency degradation is mediated by the entorhinal cortex and its efferents via the dorsal hippocampus. Journal of Neuroscience 22:10976–10984, 2002. PMID: 12486193
Day, J.J.; Roitman, M.F.; Wightman, R.M.; and Carelli, R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nature Neuroscience 10:1020–1028, 2007. PMID: 17603481
Dickinson, A. Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society B308:67–78, 1985.
Dickinson, A. Instrumental conditioning. In: Mackintosh, N.J., Ed. Animal Learning and Cognition. Orlando, FL: Academic, 1994, pp. 45–79.
Dickinson, A. Bolles’s psychological syllogism. In: Bouton, M.E., and Fanselow, M.S., Eds. Learning, Motivation, and Cognition. Washington, DC: American Psychological Association, 1997.
Dickinson, A., and Balleine, B. Actions and responses: The dual psychology of behaviour. In: Eilan, N.; McCarthy, R.A.; Brewer, B., Eds. Spatial Representation: Problems in Philosophy and Psychology. Malden, MA: Blackwell Publishers, 1993, pp. 277–293.
Everitt, B.J., and Robbins, T.W. Neural systems of reinforcement for drug addiction: From actions to habits to compulsion. Nature Neuroscience 8:1481–1489, 2005. PMID: 16251991
Gerdeman, G.L.; Partridge, J.G.; Lupica, C.R.; and Lovinger, D.M. It could be habit forming: Drugs of abuse and striatal synaptic plasticity. Trends in Neuroscience 26:184–192, 2003. PMID: 12689769
Haber, S.N. The primate basal ganglia: Paraller and integrative networks. Journal of Chemical Neuroanatomy 26(4):317–330, 2003. PMID: 14729134
Hammond, L.J. The effect of contingency upon the appetitive conditioning of free-operant behavior. Journal of the Experimental Analysis of Behavior 34:297–304, 1980. PMID: 16812191
Hyman, S.E.; Malenka, R.C.; and Nestler, E.J. Neural mechanisms of addiction: The role of reward-related learning and memory. Annual Review of Neuroscience 29:565–598, 2006. PMID: 16776597
Jedynak, J.P.; Uslaner, J.M.; Esteban, J.A.; and Robinson, T.E. Methamphetamine-induced structural plasticity in the dorsal striatum. European Journal of Neuroscience 25:847–853, 2007. PMID: 17328779
Le Moal, M., and Koob, G.F. Drug addiction: Pathways to the disease and pathophysiological perspectives. European Neuropsychopharmacology 17:377–393, 2007. PMID: 17169534
Lo, C.C., and Wang, X.J. Cortico-basal ganglia circuit mechanism for a decision threshold in reaction time tasks. Nature Neuroscience 9:956–963, 2006. PMID: 16767089
Mowrer, O. Learning Theory and Behavior. New York: John Wiley & Sons, 1960.
Nauta, W.J.H. Reciprocal links of the corpus striatum with the cerebral cortex and limbic system: A common substrate for movement and thought? In: Mueller, J., Ed. Neurology and Psychiatry: A Meeting of Minds. Basel, Switzerland: Karger, 1989, pp. 43–63.
Nelson, A., and Killcross, S. Amphetamine exposure enhances habit formation. Journal of Neuroscience 26:3805–3812, 2006. PMID: 16597734
Parkinson, J.A.; Willoughby, P.J.; Robbins, T.W.; and Everitt, B.J. Disconnection of the anterior cingulate cortex and nucleus accumbens core impairs Pavlovian approach behavior: Further evidence for limbic cortical-ventral striatopallidal systems. Behavioural Neuroscience 114:42–63, 2000. PMID: 10718261
Porrino, L.J.; Lyons, D.; Smith, H.R.; et al. Cocaine self-administration produces a progressive involvement of limbic, association, and sensorimotor striatal domains. Journal of Neuroscience 24:3554– 3562, 2004. PMID: 15071103
Robinson, T.E., and Berridge, K.C. Addiction. Annual Review of Psychology 54:25–53, 2003. PMID: 12185211
Schwartz, B., and Gamzu, E. Pavlovian control of operant behavior. In: Honig, W., and Staddon, J.E.R., Eds. Handbook of Operant Behavior. Old Tappan, NJ: Prentice Hall, 1977, pp. 53–97.
Swanson, L.W. Cerebral hemisphere regulation of motivated behavior. Brain Research 886:113–164, 2000.
Wickens, J.R.; Budd, C.S.; Hyland, B.I.; and Arbuthnott, G.W. Striatal contributions to reward and decision making: Making sense of regional variations in a reiterated processing matrix. Annals of the New York Academy of Sciences 1104:192–212, 2007a. PMID: 17416920
Wickens, J.R.; Horvitz, J.C.; Costa, R.M.; and Killcross, S. Dopaminergic mechanisms in actions and habits. Journal of Neuroscience 27:8181–8183, 2007b. PMID: 17670964
Yin, H.H., and Knowlton, B.J. Reinforcer devaluation abolishes conditioned cue preference: Evidence for stimulus-stimulus associations. Behavioural Neuroscience 116:174–177, 2002. PMID: 11895179
Yin, H.H., and Knowlton, B.J. Addiction and learning. In: Wiers, R.W., and Stacy, A.W., Eds., Handbook of Implicit Cognition and Addiction. Thousand Oaks, CA: Sage, 2005, pp. 167–183.
Yin, H.H., and Knowlton, B.J. The role of the basal ganglia in habit formation. Nature Reviews. Neuroscience 7:464–476, 2006. PMID: 16715055
Yin, H.H.; Knowlton, B.J.; and Balleine, B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience 19:181–189, 2004. PMID: 14750976
Yin, H.H.; Knowlton, B.J.; and Balleine, B.W. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. European Journal of Neuroscience 22:505–512, 2005a. PMID: 16045503
Yin, H.H.; Knowlton, B.J.; and Balleine, B.W. Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behavioural Brain Research 166:189–196, 2006. PMID: 16153716
Yin, H.H.; Ostund, S.B.; and Ballentine, B.W. Reward-guided learning beyond dopamine in the nucleus accumbeans: The integrative functions of cortico-basal ganglia networks. European Journal of Neuroscience 28(8):1437–1448, 2008. PMID: 18793321
Yin, H.H.; Ostlund, S.B.; Knowlton, B.J.; and Balleine, B.W. The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience 22:513–523, 2005b. PMID: 16045504
Yin, H.H.; Park, B.S.; Adermark, L.; and Lovinger, D.M. Ethanol reverses the direction of long-term synaptic plasticity in the dorsomedial striatum. European Journal of Neuroscience 25:3226–3232, 2007. PMID: 17552991