Classical phonon scattering calculations predict the drag force acting on defects and dislocations rises linearly with temperature, in direct contradiction with molecular dynamics simulations that find the drag force to be independent of temperature for nanoscale defects. Using the Mori-Zwanzig projection technique,with no recourse to elasticity or scattering theories, we derive a general Langevin equation for highly mobile crystal defects and dislocations, with full treatment of discreteness and nonlinearity in the defect core. We obtain an analytical expression for the drag force that is evaluated in molecular statics and molecular dynamics, extracting the force on a defect directly from the interatomic forces. Our results show that a temperature-independent drag force arises because vibrations in a discrete crystal are never independent of the defect motion, an implicit assumption in any phonon-based approach. This effect remains even when the Peierls barrier is effectively zero, invalidating qualitative explanations involving the radiation of phonons. We apply our methods to an interstitial defect in tungsten and solitons in the Frenkel-Kontorova model, finding very good agreement with trajectory-based estimations of the thermal drag force.