Technically, a VoIP phone conversation only utilizes 100kbps of bandwidth in each direction. However, there are other concurrent needs for bandwidth during a phone call.
Seems if a minimum is needed then 5 Mbps (Megabits per second) would be a good bottom end to cover all data needs, however, call quality can be affected by surges of data transport caused by other devices demanding bandwidth on the same network.
Bottom line, there is no hard rule here to ensure no degradation of audio quality. Typically, the outbound bandwidth is subscribed at a lower rate than the inbound bandwidth, and should outbound traffic have audio quality issues, then a router could be added to prioritize the outbound audio traffic.