- Linear-response conductance of a two-terminal Aharonov-Bohm (AB) interferometer is an even function of magnetic field. This phase symmetry is not expected to hold beyond the linear-response regime and in simple AB rings the phase of the oscillations changes smoothly (almost linearly) with voltage bias. However, in an interferometer with a quantum dot in its arm, tuned to the Coulomb blockade regime, experiments indicate that phase symmetry seems to persist even in the nonlinear regime. In this paper we discuss the processes that break AB phase symmetry and show that breaking of phase symmetry in such an interferometer is possible only after the onset of inelastic cotunneling, i.e., when the voltage bias is larger than the excitation energy in the dot. The asymmetric component of AB oscillations is significant only when the contributions of different levels to the symmetric component nearly cancel out (e.g., due to different parity of these levels), which explains the sharp changes in the AB phase. We show that our theoretical results are consistent with experimental findings.