To start, teeth are not in concrete. Every tooth has a root, and around that root is a ligament that allows the tooth some small movement. It is this little bit of play that helps us to prevent fracturing teeth due to the heavy forces of chewing (such as a building or bridge has movement structurally built into them to prevent fracture/damage over years of environmental forces).
When braces are removed, it takes about 2 years for the ligament around the root of the tooth to reattach as well as for the bone around the ligament to solidify properly. That being said, the ligament is still present, and therefore teeth always have continual play (though slight). Over years of enduring the heavy forces of chewing, and without the use of any retainer wear during this period, teeth can play and end up visually moving. As a result, it is suggested (for life) to at least wear your retainer(s) on and off after the first two years of wearing them every night to bed. Wearing your retainers on and off during the month (to bed…for life) can help preserve the beautiful smile that the orthodontist achieved and created.