Ultrasonic phased array (UPA) provides a powerful tool for nondestructive testing (NDT) of carbon fiber-reinforced plastic (CFRP). By the aid of full matrix capture (FMC) technique, the optimum resolution of anisotropic CFRP inspection could be achieved by the total focusing method (TFM). The directional dependence of ultrasonic velocity is one of the biggest challenges due to the anisotropy of CFRP. The objective of this research is to establish a joint method to estimate direction-dependent velocity and damage location of CFRP. To obtain group velocity without prior knowledge of neither theoretical calculation nor experimental determination, a limited angle range of the anisotropic velocity is first obtained by backwall reflection method (BRM), which is then extended by analyzing the relation between the time delay of backwall and side drilled hole (SDH) reflection. The effectiveness of the proposed method is experimentally demonstrated with UPA imaging of SDH in composite laminates.