We study by light microscopy a soft colloidal glass consisting of a compact arrangement of polydisperse elastic spheres. We show that its slow and non-stationary dynamics results from the unavoidable small fluctuations of temperature, which induce intermittent local shear deformations in the sample, because of thermal expansion and contraction. Temperature-induced shear provokes both reversible and irreversible rearrangements whose amplitude decreases with time, leading to an exponential slowing down of the dynamics with sample age. Moreover, we find that the motion of the irreversible rearrangements is close to ballistic, in agreement with our previous light scattering results. Our observations provide the first direct experimental evidence of ballistic motion associated with the slow dynamics of a variety of soft glassy materials, as inferred from scattering techniques, and suggest a possibly general mechanism for these dynamics.