In the nano-fabrication field, high precision displacement tracing of the fabricating beam is extensively required. Due to the coherence noise and the sensitivity to environmental disturbances, the commonly used measuring methods base on the laser interferometry are unstable. In this paper, a high-precision measuring method for the three-dimensional displacements is developed based on the low coherence interferometry. The interferogram at a particular location is unique and distinctive, which can be applied as a benchmark for the absolute measurement of positions. Consequently, interferograms are continuously acquired during the movement of the nano-stage, then the quantitative relationship between the stage position/tilt and the interferograms is established by analytic calculation. Besides, the influence of random errors can be suppressed by the averaging effect of the least squares fitting, thereby enhancing the precision by more than an order of magnitude compared with traditional methods. The measuring uncertainty is derived and the impacts of the main influencing factors are investigated. Experiments demonstrate that the measuring repeatability can achieve 1.16 nm. As a result, the proposed method can reliably obtain the absolute position and three dimensional trajectory of the nano-stage, and it is of significance to improve the reliability of nano-measurement and fabrication.