- A theoretical approach proposed by Elperin et al. [Phys. Rev. E 66, 066305 (2002)] is developed further to investigate formation of large-scale coherent structures in a nonrotating turbulent convection via excitation of a large-scale instability. In particular, the convective-wind instability that causes formation of large-scale coherent motions in the form of cells, can be excited in a shear-free regime. It was shown that the redistribution of the turbulent heat flux due to nonuniform large-scale motions plays a crucial role in the formation of the coherent large-scale structures in the turbulent convection. The modification of the turbulent heat flux results in strong reduction of the critical Rayleigh number (based on the eddy viscosity and turbulent temperature diffusivity) required for the excitation of the convective-wind instability. The large-scale convective-shear instability that results in the formation of the large-scale coherent motions in the form of rolls stretched along imposed large-scale velocity, can be excited in the sheared turbulent convection. This instability causes the generation of convective-shear waves propagating perpendicular to the convective rolls. The mean-field equations that describe the convective-wind and convective-shear instabilities, are solved numerically. We determine the key parameters that affect formation of the large-scale coherent structures in the turbulent convection. In particular, the degree of thermal anisotropy and the lateral background heat flux strongly modify the growth rates of the large-scale convective-shear instability, the frequencies of the generated convective-shear waves, and change the thresholds required for the excitation of the large-scale instabilities. This study elucidates the origin of the large-scale circulations and rolls observed in the atmospheric convective boundary layers.