Recently developed super nested array families have drawn much attention owing to their merits on keeping the benefits of the standard nested arrays while further mitigating coupling in dense subarray portions. In this communication, a new mutual coupling model for nested arrays is constructed. Analyzing the structure of the newly formed mutual coupling matrix, a transformation of the distorted steering vector to separate angular information from the mutual coupling coefficients is revealed. By this property, direction of arrival (DOA) estimates can be determined via a grid search for the minimum of a determinant function of DOA, which is induced by the rank reduction property. We also extend the robust DOA estimation method to accommodate the unknown mutual coupling and gain-phase mismatches in the nested array. Compared with the schemes of super nested array families on reducing the mutual coupling effects, the solutions presented in this paper has two advantages: (a) It is applicable to the standard nested arrays without rearranging the configuration to increase the inter-element spacing, alleviating the cross talk in dense uniform linear arrays (ULAs) as well as gain-phase errors in sparse ULA parts; (b) Perturbations in nested arrays are estimated in colored noise, which is significant but rarely discussed before. Simulations results corroborate the superiority of the proposed methods using fourth-order cumulants.