The Hammers had to come from behind to take victory with Victor Obinna earning the plaudits with a hat-trick.
Obinna had put the Hammers one up after only four minutes deflecting a Mark Noble effort into the net.
Forest then settled quickly and equalised through veteran Dele Adebola after 18 minutes before David McGoldrick put them in front just before half-time.
Obinna levelled the game going into the break with a stunning long range strike from near the sidelines.
At the start of the second half Forest looked threatening but the decisive moment came when Obinna converted a 52nd minute penalty to grab his hat-trick.
West Ham will now play Burnley at the Boleyn Ground in the Fifth Round.