The Hatters hopes of escaping the drop took a massive blow in their lead-up to Wembley as they suffered their first defeat in three matches.
Despite enjoying large periods of possession and creating numerous chances, the Town were unable to handle the pace of Rotherham on the counter-attack, who on the night were clinical in front of goal.
The Millers raced into a 2-0 goal lead inside 38 minutes against the run of play with goals from Reuben Reid and Drewe Broughton before a Chris Martin header three minutes before the break gave the Hatters hope.
But further goals in the second half from Danny Harrison and Mark Hudson for the visitors secured the points for the Yorkshire despite Asa Hall's goal sandwiched in between 20 minutes from the end again giving the Town hope.
The defeat leaves the Hatters 11 points from safety with just 18 points left to play for.
Manager Mick Harford kept the faith with the same side that won 2-1 at Morecambe on Saturday.
Defender Mark Bower was playing his last game of his loan spell from Bradford City at the heart of the Hatters defence along side George Pilkington.
Rotherham manager Mark Robins, meanwhile, included self-confessed Luton fan Drewe Broughton in attack and four changes to his side that drew 0-0 with Accrington Stanley at the weekend.
After swapping ends following the toss, the Hatters kicked towards the Kenilworth Road end with the Millers sporting their luminous all-yellow away strip.
The Town started on the front-foot looking for their fourth straight victory and fifth win in six games in their attempt to close the gap further on the sides outside of the League 2 relegation zone.
Buoyed by recent results the Kenilworth Road crowd were in good voice as the Hatters began in the ascendancy and they almost had something to shout about in the ninth minute when Chris Martin's downward header had to be clawed out by Andy Warrington in the Millers' goal following Kevin Nicholls' deep cross.
Seconds later Tom Craddock, who was looking to continue his hot streak in front of goal, poked a shot goalwards but his shot was so high it hit the Kenilworth Road roof.
However, despite their bright, start the Town found themselves behind on 11 minutes when Reuben Reid chased a long-ball to get the better of Bower and Pilkington to slot past Dean Brill with a well-taken cool finish. It was the striker's 17th goal of the season.
Back came the Hatters, and on 15 minutes Craddock showed great close control and technique to beat his marker on the right-hand byline to shoot towards Warrington's goal but his effort was blocked bravely by Rotherham captain Pablo Mills who was floored by the power of the striker's shot.
Then, a minute later another Nicholls cross was met by Martin but the referee adjudged to have fouled Warrington when challenging for the ball and, after an ensuing melee between both sets of players, the Town striker and Mills were cautioned by referee Deadman.
On 21 minutes, after Nicholls was upended by Ian Sharps, Kevin Gallen saw his well-struck daisy cutter of a free-kick fly inches wide of the target.
Then from Nicholls' resulting corner, Martin's near-post header was beaten away by Warrington.
The Town were enjoying another fine spell of possession and on the balance of play were unlucky to behind, with Gallen in particular shining for the hosts with his diligent hold-up and link-up play.
Nicholls' left-wing corners were flying in causing the visitors all sorts of the problems, and other vicious in-swinging flag-kick from the captain on 28 minutes had to be punched away by Warrington.
But Rotherham were always looking dangerous on the counter-attack and on 29 minutes they almost doubled their lead when Reid saw his shot bounce agonisingly wide of Brill's right-hand post after receiving a square pass from Broughton.
Back came the Hatters though, and on 31 minutes Nicholls tried his luck with a volley from outside the penalty box that was blocked by Marc Joseph after Rotherham failed to clear David Livermore's corner.
The game developing into a fascinating spectacle as the Town continued to surge forward looking for the equaliser, roared on by the home crowd but always wary of the pace the Millers possessed on the break.
Then, the inevitable happened and on 38 minutes the visitors did double their advantage when Broughton slid along the turf to turn the ball beyond Brill from close range after Reid had touched on a long throw.
It was a major kick in the teeth for the Hatters after they had dominated the half, but no sooner had they got two goals down they got the goal their endeavour deserved when reducing arrears in emphatic style three minutes before half time.
Craddock rescued a lost cause down the right flank to cut the ball back for Asa Hall to deliver over a dangerous cross to the back post which was met by the on-rushing Martin who thumped a powerful header past Warrington from six yards for his 12th goal of the campaign.
The goal clearly lifted the spirit of the players and crowd to give them cause for optimism for a crucial second period.
HALF TIME: LUTON TOWN 1-2 ROTHERHAM UNITED
There were no changes for either side at the break as the Town looked to peg back the visitors.
The Hatters were certainty on top at the start of the second half but without the intensity of the first half.
But once again then Town were caught cold by the Millers when the visitors further extended their lead eight minutes in the second half.
After Broughton saw his hooked shot cleared off the line by Bower, Danny Harrison - one of the four changes in the Rotherham side on the night - headed the ball straight back and the ball dropped into the top corner of the net.
The visitors looked like they got the measure of dealing with the Town's attack and a Martin shot on the hour that was comfortably gathered by Warrington was the closest the hosts came to making inroads in the Rotherham lead.
By this time, Harford had made a switch on the pitch, swapping Livermore with Emanuel down the left flank.
Craddock came close to pulling a goal back for the Hatters on 67 minutes when his curling shot was deflected behind by Jason Taylor.
Again the Millers showed why they were so dangerous on the break when they came close to adding a fourth on 68 minutes.
Jason Taylor fed Reid who's shot was parried by Brill who then had to block at the feet of Taylor at the second attempt.
Seconds later, this time at the other end, Emanuel hammered a shot wide of Warrington's right-hand post after Rotherham had failed to clear the danger caused by Gallen.
However, the Town hauled themselves back into proceedings to make for a frantic last 20 minutes when cutting the deficit on 70 minutes.
The ball fell kindly for Hall on the edge of the Rotherham penalty area and the midfielder was able to steer a low shot past Warrington for his 11th goal of the season.
But the Hatters were living dangerously and a minute after cutting the visitors' lead they almost fell further behind again when Rotherham right-back Dale Tonge hit the angle of the goal frame with Brill helpless.
With 15 minutes left, and as the Town began their assault on the Millers' half in search of an equaliser, Emanuel shot with his weaker right-foot wide of Warrington's goal after being found by Gallen.
Referee Deadman incurred the wrath of the Kenilworth Road crowd and a number of the Hatters players after failing to pull back the play when he decided to play an advantage in their favour.
Harford made his first substitution of the evening 10 minutes from time when withdrawing Emanuel in favour of Sam Parkin as the Hatters went in search of an equaliser.
But the Town were again caught on the break and were punished by the visitors three minutes from time when Reid rolled the ball into the path of substitute Mark Hudson who fired home confident past Brill for the Millers' fourth goal of the night.
But that was the end of any of the meaningful action as the Town suffered their seventh home defeat of the season leaving it all do to save their Football League lives.
FULL TIME: LUTON TOWN 2-4 ROTHERHAM UNITED
Luton Town: Brill, Keane, Pilkington, Nicholls, Emanuel (Parkin 80), Livermore, Hall, Martin, Bower, Craddock, Gallen.
Unused substitutes: Davis, Henderson, Asafu-Adjaye, Jarvis.
Booked: Martin (16)
Rotherham United: Warrington, Tonge, Harrison, Sharps, Fenton, Birchill (Hudson 74), Joseph, Broughton (Ryan Taylor 65), Reid, Mills, Jason Taylor.
Unused substitutes: Nicholas, Clarke, Cann.
Booked: Mills (16), Sharps (20), Broughton (62)
Referee: D Deadman
Attendance: 5,975 (including 196 from Rotherham)
