The right-back completely deceived keeper Stephen Collis with a back post floater which ended in the net.
Only 13 minutes had gone and City manager Gary Johnson could not disguise his delight as his new club put one over his old one.
Worried by Yeovil's anticipated high-tempo game, he had reshaped his midfield, changing both wing-backs while the visitors were without the influence of centre-back Terry Skiverton who has damaged knee ligaments.
Skiverton might have kept City's first goal out and he would certainly have been there for the second eight minutes later. This came from a free-kick given away by Scott Guyett down the left flank.
Stephen Collis came to collect but, as he stretched, Mark McCammon powered in with a scoring header. It had been Orr's first goal for City but for McCammon it was his third in four games since his loan move from Brighton.
Two goals down and not even at the halfway point in the first period, this was crisis time for Yeovil who are hovering around the relegation zone.
They responded with character and determination, piling on the pressure as they swept forward. Arron Davies beat the home keeper as he cut in from the right only to miss the far post by a balls width.
Then only a great stop by Liam Fontaine, the young defender on loan from Fulham, came between Chris Cohen and a Yeovil goal.
Both sides made changes in the second half and Yeovil had to battle through 20 minutes of relentless pressure as City sought a killer third goal.
But the visitors then broke out to make it 2-1. Brazilian keeper Adriano Basso made a classy save from Paul Terry but was beaten by Davies' fierce follow-up which scorched past his left hand.
Yeovil went close to an equaliser when Phil Jevons, who had come on for Matt Harold, fired a 30-yard effort just wide.