Serbia had dominated against a Costa Rica side who had rarely threatened after a bright start, but chances were thin on the ground until Kolarov made the breakthrough in the 56th minute.
They famously reached the quarter-finals four years ago, although Oviedo missed out due to a long-term injury, after topping a group that included England, Italy and Uruguay.
The win was only Serbia's second victory in the World Cup finals since they competed as an independent nation, their other was a 1-0 win over Germany in 2010.
He was a constant threat and the clear focal point of Costa Rica's defensive efforts.
The 23-year-old midfielder was the standout player in his side's 1-0 victory and he proved his worth on the field after a stunning debut performance in Russian Federation.
Bookmakers unanimously considered the favorite in this match, the Serbian team.
The goal also represented a new record in this World Cup.
"The match against Switzerland is going to be the most hard one", he said. "We should have capitalised on our opportunities like they did. I don't want to do the maths yet", Kolarov said.
"We didn't qualify for last World Cup and the last two Euros, so we didn't get a chance to enjoy these matches for a long time".
See how to watch and follow live updates from the game here.
Meanwhile, Costa Rica boss Óscar Ramírez spoke defiantly after his sides loss, claiming that the World Cup is far from over for his side despite the defeat.
Milinkovic-Savic gave Navas a workout in the first half, and while he didn't score, he is clearly a potent weapon for Serbia.