A second half dominated by determined Berwick attacking and superb goalkeeping by Myles Hogarth ended in drama.
Two minutes from time, Berwick's Mark Cowan lost possession in midfield and the ball broke for Robert Dunn, who hit a fierce shot against the bar. It rebounded 20 yards to Doug Wilson, who crashed home a tremendous drive. It was Stirling's only real chance in a half totally dominated by the home team.
But late on Rangers were awarded a penalty when Craig Ferguson handed a David Murie corner. Andy Seaton took the kick and it hit the underside of the bar before the referee ruled it had bounced over the line.
Rangers made a flying start after blunders by the Albion defence. Kevin Gordon shot just wide inside a minute, and in the third minute Gordon Connelly lobbed the ball home.
But after 13 minutes Stirling were level, Dunn scoring from the spot after Grant McNicoll brought down Martin Glancy.
Rangers bombarded the Stirling goal in the second half and Hogarth pulled off a wonderful double save from Ian Little and then made two more great stops from Murie and Paul Hampshire.