The foul to the left of goal had been conceded by Brown as a scrappy match was on the brink of falling apart.
For the final half hour plus five minutes more for stoppages, Colchester were under constant pressure, but keeper Aidan Davison saw them through to a draw.
These are edgy times for City, a hot tip for promotion, but now in the wrong end of the table and with Marcus Stewart their expensive signing from Sunderland out for a month with a broken finger.
Chairman and millionaire fundraiser Steve Lansdown had to appeal in the programme "this is not the time to throw in the towel".
Certainly there was no sign of this on the field although the lively U's striker Chris Iwelumo took a lot of taming.
His physical presence and the keen support of Mark Yeates on his left and Kevin Watson in central midfield often had City on the back foot.
With three players on international duty, City could have called off the game instead they preferred to take the £30,000 fee from Sky for live TV coverage.
Colchester, with three changes, where forced into a fourth when Sam Stockley limped out of the warm-up to be replaced in defence by Garry Richards.
In a troubled first half City could easily have slipped behind, Colchester were winning the corner count by seven to one and Steve Phillips rescued City, fisting away a rasping shot from Watson which looked destined for the top right-hand corner.
Then he had a tremendous let-off when Kem Izzet blasted a shot against the bar from close range.