The LOCOS process utilizes the different rates of oxidation of silicon and silicon nitride to locally mask the wafer surface.
A silicon nitride layer is used to mask the areas where no oxide should grow; an oxide layer forms only on the regions free of nitride. Since silicon and silicon nitride have different coefficients of thermal expansion, a thin oxide layer – the pad oxide – is deposited between the nitride mask and the substrate to prevent strain caused by temperature changes.
For lateral isolation of transistors, a field oxide (FOX) is then grown on the bare silicon surface. While a silicon dioxide layer forms on the exposed silicon during field oxidation, the pad oxide causes a lateral diffusion of oxygen underneath the nitride mask, resulting in slight oxide growth at the edge of the mask. This oxide extension has the shape of a bird's beak, whose length depends on the oxidation process as well as on the thickness of the nitride and the pad oxide.
(Source: Consiglio Nazionale delle Ricerche)
Besides this effect, which can consume up to 1 µm of the area intended for devices, wet oxidation also causes the so-called white ribbon or Kooi effect. Here, nitride from the masking layer reacts with hydrogen to form ammonia NH3, which diffuses to the silicon surface and causes nitridation there. This nitride must be removed prior to gate oxidation, since it would otherwise act as a mask.
Despite these negative effects, the LOCOS process is a suitable method for enabling high packing density. Because of the reduced unevenness, without the formation of edges and steps, the resolving power in photolithography is improved. The field oxide can be etched back slightly; this slightly reduces the grown oxide, but the length of the bird's beak decreases, and the surface is again flattened somewhat more. This is referred to as fully recessed LOCOS.
Nevertheless, the bird's beak is the point at which the process ultimately fails. Its length depends on the thickness of the pad oxide and the nitride, not on the size of the structures. It therefore does not shrink as devices become smaller. As long as the isolation regions were several micrometers wide, an extension of a few tenths of a micrometer hardly mattered; at structures below half a micrometer, it consumes most of the area it is meant to isolate. A thinner pad oxide would shorten it, but would then transfer the stress from the nitride mask to the silicon unabated, creating crystal defects.