Goals from Jim Stevenson, Zane Banton and Solomon Taiwo - from the spot - were not enough to beat the Championship side.
The Hatters' second string had won their previous eight fixtures this season.
The Town development squad lost their 100 per cent winning record following a 4-3 defeat at Reading on Tuesday.