Aluminum matrix composites reinforced with irregularly arranged fibers was fabricated using liquid metal infiltration technology. The cooling-induced residual stress and the subsequent tensile behavior were investigated by numerical and experimental method. The results show that the macroscopic thermal shrinkage curves obtained from the numerical simulation are consistent with the experimental ones. After the cooling process, the matrix and fiber are in the tensile and compressive stress states, respectively. The irregular fiber arrangement leads to an inhomogeneous residual stress distribution, which causes the plastic deformation and damage initiation of the matrix alloy within the smallest inter-fiber gaps. The numerical simulations involving the residual stress yield the tensile stress-strain curves that are in good agreement with the experimental ones. The cooling-induced residual stress and local damage promote the failure evolution behavior of the matrix and interface during the tensile process. As a result, the presence of residual stress resulted in reductions in axial strength and elastic modulus of 17.1% and 18.2%, and in transverse strength and fracture strain of 11.4% and 10.6%. The failure modes obtained from the numerical simulations are further validated by the fracture morphology of the tensile specimens.