Trailing at the break the Town hit back with Jemale McKenzie levelling the score on 65 minutes.
Five minutes the Hatters were 2-1 ahead when Zane Banton converted an Arel Amu cross.
Amu was then fouled in the penalty area on 75 minutes and Charlie Smith made it 3-1 from the spot.
Portsmouth pulled one back five minutes before the end but Luke Trotman secured the victory for the Town two minutes from time.
Town: Barnes, Trotman, Galliford, Rees, Tamplin (sub Hawkes 15), Chabata, Amu, Nwgala (sub Maguraushe 70), Smith, McKenzie-Lowe (sub Tyali 80), Banton. Subs not used: King, Allman.