The ruling for standing up for a fifth rak'a in a four rak'a prayer, whether intentionally or not
We pray at work daily with one of our colleagues acting as our imam. Once when he was leading us in the dhuhr (noon) prayer which is a four rak'a (cycle of prayer) prayer, he rose for a fifth rak'a after he had made the fourth. More than one person tried to alert him to his mistake by saying "subhan Allah" but he did not heed their gesture and proceeded to pray the fifth rak'a. He then made a prostration of forgetfulness after the fifth rak'a and closed the prayer. When asked why he did not respond to his colleagues' alert he said he was closer to standing than he was to sitting and for this reason he proceeded to pray the fifth rak'a.
We would like to know the ruling for this prayer and whether it is valid.
The imam's payer is valid if he adds a [surplus] rak'a unintentionally or due to an uncertainty and then proceeds to complete the prayer, basing his action on the lesser number of rak'as. The congregants' utterance of "subhan Allah" makes it obligatory upon him to sit if he remembers for certain that he is praying the fifth rak'a and not the fourth. Otherwise, if he does not remember and is certain or assumes that the congregants are mistaken, then it is not permissible for him to sit, but he is to proceed with the prayer, else he invalidates the prayer by sitting.
As for the congregants, whoever from among them is certain that the imam has made a mistake, then it is necessary for him not to follow him, but to make the intention to cease participation in the group prayer and finish alone. Alternatively, he may wait for the imam until he sits for the last testification of faith and finish the prayer with him. But whoever is uncertain that the imam has made a mistake, must continue to follow him.
Imam Ibn Hajar al-Haitami wrote in Tuhfat al-Muhtaj fi Sharh al-Manhaj (2/194), "If the imam rises absentmindedly to pray a fifth rak'a, it is not permissible for a congregant to follow him even if he is a latecomer or uncertain of the number of rak'as. He is not to heed the possibility that he may have omitted an integral of a rak'a. This is because the obligation regarding this matter is that he is either certain or near certain that the imam has prayed a fifth rak'a, in which case he is to either cease participation and close the prayer or wait for the imam to sit and finish the prayer with him based on the relied upon opinion of the school."
The above applies if the imam truly does not remember that he has prayed four rak'as and then becomes certain that he stood up for a fifth. But if he remembers due to the congregants' alert that he has indeed stood for a fifth rak'a, his attempt to rectify his mistake by arguing that he was closer to standing than to sitting is of no consequence. It was a duty upon him at the time to sit for the testification of faith. Failing to do this, invalidates his prayer because he did not transition from one of the sunnas of prayer to an obligatory action. Rather, he has made a deliberate addition that is not a part of the prayer.
Allah the Almighty knows best.