Syd Owen was replaced by ex-Charlton goalkeeper Sam Bartram, but the rot had set in and 33 players were used in the 1960/61 season but the Town could not manage to finish in the top half of the table. The board were blamed by the supporters for allowing the money earned from the 1959 Cup run to be handed over to the tax man rather than be spent on players, but it was all now too late.

There was a general feeling that Bartram was slowly turning the ship around after he finished the 1961/62 campaign in mid-table but he was not given a chance to regain past glories, being relieved of his duties 'by mutual consent'. His replacement was to be ex-Town coach Jack Crompton but he changed his mind after a week in the post on medical grounds.

Bristol City coach Bill Harvey then took over and presided over another relegation campaign as the Town sank dismally to Division Three at the end of 1962/63. Unfortunately, instead of being allowed to consolidate, the team plunged headlong towards Division Four at an alarming rate, only saved by an unexpected nine game unbeaten run as the season drew to a close.

The stay of execution was only temporary as the Town, only six years from their FA Cup Final appearance, slumped to the football basement in 1965 amidst apathy and crowds of less than 3,000. Harvey had resigned mid-season to be replaced by ex-Town player and manager George Martin and although he introduced some discipline behind the scenes, he was, frankly, too old.

Having said that, the Hatters made a good effort at escaping Division Four at the first attempt and were only let down by rows over player bonuses as the season entered its final couple of games. No money, all experienced players released as a legacy of the bonuses row and crowds at an all time low left a tired George Martin with no option but to resign, to be replaced by Allan Brown, who had starred for the Town in the 1959 Cup run.

Brown had some work to do! With the team in second bottom position, soon after he rejoined the club in November 1966, they crashed 1-8 to bottom club Lincoln, arguably the club's lowest spot in their history.