Johnson took charge at Ashton Gate in 2005 and won promotion to the Championship two years later before taking them to within 90 minutes of a place in the Premier League. However, their dreams were ended by a 1-0 defeat to Hull at Wembley.
Johnson presided over 239 games, winning 96, drawing 69 and losing 74, the last of which came courtesy of their last-gasp 3-2 reverse at relegation-threatened Plymouth on Tuesday night.
Johnson's assistant Keith Millen will take charge of Saturday's visit of Newcastle.
Chairman Steve Lansdown admitted it was "the right time to make a change". He said: "I would like to place on record my thanks to Gary for the fantastic work he has done over four and a half years at the club.
"Winning promotion from League One and reaching the Championship play-off final will always live long in the memory for myself and I'm sure for our supporters as well.
"Results haven't gone as well as everyone would have liked this season and it's been mutually agreed now is the right time to make a change."
Johnson revealed he was pleased with what he achieved at the club.
The 54-year-old said: "I'd like to thank the chairman for his continual support throughout the last four and a half years and congratulate him on taking the club to the level it currently enjoys.
"I thank the majority of the club's fans for their tremendous support during my time with the club.
"I thank the players and staff for their continued efforts in getting this club from what was a position at the bottom of League One when I joined to promotion to the Championship, to the play-off final for a place in the Premier League and then to compete in the Championship thereafter.
"I leave Bristol City FC with fond memories but feel the time is right to move on to a new challenge."
Johnson was named TEAMtalk's Football League Manager of the Year in 2007.
Newcastle 5/6, Draw 12/5, Bristol City 7/2
Source: Team Talk
Source: Team Talk