Carlton Cole put the Hammers in front on the night with a thunderbolt strike from outside the box and the Hammers went into the interval with a 3-1 advantage.
In the second half former West Ham midfielder Lee Bowyer got his customary goal against the Hammers before a Roger Johnson header took the game into extra time.
The Hammers were always on the back foot in the second half and it was no surprise when Craig Gardner found the winner for Birmingham in extra time.