It was Rosberg's third win of the year and sixth overall as he extended his lead in the drivers' championship over Hamilton to 29 points.
Valtteri Bottas came third for his first career Formula One podium while Williams teammate Felipe Massa, who started from pole Position, took fourth.
Four-time Formula One champion Sebastian Vettel had an engine problem in the second lap and was doubled by the field before quitting the race in the 36th on Red Bull's home circuit.
Hamilton, who was ninth after qualifying, used a blistering start and earned four places from the start and won another place to work his way up to fourth in the opening lap.
Rosberg immediately overtook Bottas but lost that position again shortly after the first turn.
Massa dropped from pole to fourth after the top-four had their first pit stops.
Sergio Perez remained in the lead until the 27th, overtaken by Rosberg who accelerated for the fastest lap so far to create distance to Hamilton, who got stuck behind Perez.
From lap 30 on, the four leaders battled it out with clear advantages for both Mercedes drivers.
Rosberg remained in the lead for the rest of the race, not allowing a pushing Hamilton a chance to overtake him.
The race once more confirmed Mercedes has best adapted to the introduction this season of the 1.6-liter turbocharged V6 engines with build-in energy recovery system.
Hamilton won four races in a row while Rosberg never finished worse than second.
Mercedes' stranglehold on F1 victories was broken two weeks ago in Montreal by Daniel Ricciardo's maiden win, though Red Bull failed to threaten Mercedes again on its home circuit in Austria.
Mercedes dominated practice Friday with Rosberg and Hamilton posting the fastest times, but Williams hit back the next day when Bottas led the final practice before Massa grabbed pole position - his first in six years.
Mercedes' German driver Nico Rosberg makes a pit stop as he competes in the Austrian Formula One Grand Prix at the Red Bull Ring in Spielberg. (AFP photo)
The track was new to all but four of the 22 drivers as F1 returned to Austria for the first time since 2003.
At 4.3-kilometer, the Red Bull Ring is one of the shortest circuits on the calendar, though with just nine turns, it features long straights. Situated in Central Austria's countryside, the track includes inclines of up to 12 percent.
It was the 27th Austrian GP, which has been held on different circuits since 1964, with Alain Prost winning a record three times. Until 1987, the races took place in nearby Zeltweg before Austria first disappeared from the calendar for 10 years.
Spielberg also hosted the race from 1997-2003. Five years after the last race, Red Bull reportedly invested 200 million euros ($270 million) for upgrading the track and its surrounding infrastructure.
The Austrian GP has been one of two new races to this season's F1 calendar alongside the Russian GP, scheduled for Sochi in October.
Results from the Austrian Formula One Grand Prix, the eighth race of the season, in Spielberg on Sunday:
1. Nico Rosberg (GER/Mercedes) 1hr 27min 54.976sec, 2. Lewis Hamilton (GBR/Mercedes) at 1.932sec, 3. Valtteri Bottas (FIN/Williams) 8.172, 4. Felipe Massa (BRA/Williams) 17.358, 5. Fernando Alonso (ESP/Scuderia Ferrari) 18.553, 6. Sergio Perez (MEX/Force India) 28.546, 7. Kevin Magnussen (DEN/McLaren Mercedes) 32.031, 8. Daniel Ricciardo (AUS/Red Bull Racing) 43.522, 9. Nico H?lkenberg (GER/Force India) 44.137, 10. Kimi Raikkonen (FIN/Scuderia Ferrari) 47.777, 11. Jenson Button (GBR/McLaren Mercedes) 50.966, 12. Pastor Maldonado (VEN/Lotus) at 1 lap, 13. Adrian Sutil (GER/Sauber) 1 lap, 14. Romain Grosjean (FRA/Lotus) 1 lap, 15. Jules Bianchi (FRA/Marussia) 2 laps, 16. Kamui Kobayashi (JPN/Caterham) 2 laps, 17. Max Chilton (GBR/Marussia) 2 laps, 18. Marcus Ericsson (SWE/Caterham) 2 laps, 19. Esteban Gutierrez (MEX/Sauber) 2 laps
Retirements: Daniil Kvyat (RUS/Toro Rosso): 25th lap; Sebastian Vettel (GER/Red Bull Racing): 35th lap; Jean-Eric Vergne (FRA/Toro Rosso): 60th lap
1. Nico Rosberg (GER) 165.0pts, 2. Lewis Hamilton (GBR) 136.0, 3. Daniel Ricciardo (AUS) 83.0, 4. Fernando Alonso (ESP) 79.0, 5. Sebastian Vettel (GER) 60.0, 6. Nico H?lkenberg (GER) 59.0, 7. Valtteri Bottas (FIN) 55.0, 8. Jenson Button (GBR) 43.0, 9. Felipe Massa (BRA) 30.0, 10. Kevin Magnussen (DEN) 29.0, 11. Sergio Perez (MEX) 28.0, 12. Kimi Raikkonen (FIN) 19.0, 13. Romain Grosjean (FRA) 8.0, 14. Jean-Eric Vergne (FRA) 8.0, 15. Daniil Kvyat (RUS) 4.0, 16. Jules Bianchi (FRA) 2.0
1. Mercedes 301.0pts, 2. Red Bull Racing 143.0, 3. Scuderia Ferrari 98.0, 4. Force India 87.0, 5. Williams 85.0, 6. McLaren Mercedes 72.0, 7. Toro Rosso 12.0, 8. Lotus 8.0, 9. Marussia 2.0 (AFP)