Spin transport in interacting Fermi gases has sparked a lot of interest in the ultracold atoms community. One of the most challenging aspects is how to extract the homogeneous gas properties from measurements performed in harmonic traps. The trapping potential creates a density inhomogeneity which can significantly alter the transport behaviour of the gas, because the local mean free path can vary strongly from point to point in the trap. For the same reason, the transverse velocity during the relaxation to equilibrium is not constant as a function of radius. We present a systematic study of the spin drag in an elongated harmonic trap based on the Boltzmann equation using a combination of analytical and numerical methods in the dilute limit. In this regime we are able to deal with the spatial density changes without any uncontrolled approximations. We can make definite predictions for the spin drag coefficient in the collisionless limit and obtain the experimentally inaccessible transverse velocity profile in both the collisionless and hydrodynamic regimes. Our model reproduces quantitatively recent measurements of the spin drag coefficient and can help guide future experimental studies.