After going down in a shock loss in the series opener on Friday, the hosts were determined to put in a much better performance and they delivered.
Chasing a low target of 156, Sri Lanka lost two early wickets as Danushka Gunathilaka (8) and Kusal Mendis (0) both fell to opening bowler Tendai Chatara (2/33).
However, senior batsman Upul Tharanga and Niroshan Dickwella (35) added 67 for the third wicket to ease some of the nerves in the changeroom.
Dickwella fell with the total on 77 when he was caught and bowled by Zimbabwe captain Graeme Cremer in the 19th over.
Tharanga was unbeaten on 75 as he and his skipper Angelo Mathews (28 not out) took their team over the line with more than 19 overs to spare.
Earlier, left-arm spinner Lakshan Sandakan took 4/52 from 10 overs as Sri Lanka bowled their visitors out for a paltry 155 in 33.4 overs.
Opener Hamilton Masakadza top-scored for the African side with 41 while Malcolm Waller hit 38 from 29 balls batting at number seven.
The highlight of the innings though was hat-trick for debutant Wanidu Hasaranga. The 19-year-old right-arm spinner captured the last three wickets of the innings in consecutive deliveries.
The third ODI of series takes place in Hambantota on Thursday.