(Note, this is a preprint of an article published with some editorial changes in 1998 in Calow, P. (ed.) Encyclopaedia of Ecology and Environmental Management. Blackwell Press.  pp. 379-380)
isolating mechanisms The reproductive characteristics which prevent species from fusing. Isolating mechanisms are particularly important in the biological species concept, in which species of sexual organisms are defined by reproductive isolation, i.e. a lack of gene mixture. Two broad kinds of isolating mechanisms between species are typically distinguished, together with a number of sub-types (modified from Mayr 1970):

1) Pre-mating isolating mechanisms. Factors which cause species to mate with their own kind (assortative mating).

a) Temporal isolation. Individuals of different species do not mate because they are active at different times of day or in different seasons.
b) Ecological isolation. Individuals mate in their preferred habitat, and therefore do not meet individuals of other species with different ecological preferences.
c) Behavioral isolation. Potential mates meet, but choose members of their own species.
d) Mechanical isolation. Copulation is attempted, but transfer of sperm does not take place.
2) Post-mating isolating mechanisms. Genomic incompatibility, hybrid inviability or sterility.
a) Gametic incompatibility. Sperm transfer takes place, but egg is not fertilized.
b) Zygotic mortality. Egg is fertilized, but zygote does not develop.
c) Hybrid inviability. Hybrid embryo forms, but of reduced viability.
d) Hybrid sterility. Hybrid is viable, but resulting adult is sterile.
e) Hybrid breakdown. First generation (F1) hybrids are viable and fertile, but further hybrid generations (F2 and backcrosses) may be inviable or sterile.
An alternative classification of isolating mechanisms contrasts pre-zygotic isolation (items 1+ 2a above) with post-zygotic isolation (items 2b-e above). As an example of the application of isolating mechanisms, the apple-feeding host race of the tephritid fruit fly (Rhagoletis pomonella) differs from the hawthorn-feeding race in that the apple race emerges earlier in the year (1a), and each host race preferentially chooses to rest, lay eggs and mate on its own host plant (1b). On the other hand, laboratory experiments show that there is little behavioral, mechanical, or post-mating isolation (1c,d; 2a-e).

The term isolating mechanisms was introduced by T Dobzhansky in the 1930s, and has been popularized in a number of books by E Mayr. Both authors originally proposed that isolating mechanisms were group traits beneficial at the level of the species; today, this is generally disbelieved. Recent authors have pointed out that the word "mechanism" is particularly misleading as pre-mating and post-mating isolation are likely to evolve as a by-product of natural selection or genetic drift within species, rather than as a direct result of their utility as barriers to fertilization and gene mixing between species (a process known as reinforcement). A leading critic of the biological species concept and of the term isolating mechanisms is HEH Paterson, who argues that species are cohesive wholes as a result of pre-zygotic sexual signalling within species, rather than due to isolating mechanisms between species. Paterson therefore introduced a competing idea of species, the recognition concept of species, in which isolating mechanisms were replaced by specific mate recognition systems as an alternative. Unfortunately, the word "system" has as many group-benefit connotations as "mechanism", and the recognition concept of species has not gained universal acceptance.

There is also the terminological problem that reproductive isolation combines traits that reduce gene flow, such as mate choice or fertilization barriers, with traits that select against genes that have flowed, such as hybrid incompatibility. Lumping these two antagonistic features is confusing, since they are unrelated and evolve in very different ways. For instance, whereas it is conceivable that reinforcement might evolve to reduce an individual's tendency to mate with another species and produce inviable offspring, it is almost impossible to imagine that hybrid inviability itself would evolve as an adaptation. This reproductive isolation terminology leads also to a muddled use of the term gene flow as the opposite of reproductive isolation; in other words, gene flow comes to include not only the flow of genes, but also the effects of any natural selection on the frequency of such genes within each population.

Perhaps the most fundamental problem with isolating mechanisms (and specific mate recognition systems) is that species are implied to be qualitatively different from subspecies, races, or forms by their possession of these traits. Races cannot, in theory, differ in either type of trait because only species are defined by their possession. Arguably, by making species seem qualitatively different from races, these terms have spawned a number of special models of speciation where geographic isolation, also known as allopatry, or sudden bursts of evolution in small founder populations (founder events or punctuated equilibria) play important roles. Only such unusual conditions were thought to be able to give rise to new species that differ in isolating mechanisms (or specific mate recognition systems). In reality, there is little to distinguish mate choice and disruptive natural selection commonly observed within species from pre-mating and post-mating isolation between species; and, indeed, it is hard to distinguish species from races in many actual organisms (see species concepts).

Most serious research on speciation now avoids using the term isolating mechanisms because of these unwanted connotations. Instead, researchers clearly distinguish between mate choice, hybrid incompatibility and other forms of reproductive isolation. It is probable that the term isolating mechanisms is becoming obsolete, and even today is confined mostly to undergraduate textbooks.

J.L.B.M. (JLB Mallet)

Mayr E. (1970) Populations, Species, and Evolution. Harvard University Press, Cambridge, Mass.