From the massive fatigue test data analysis, we find there is a big mismatch between the fatigue life predictions done by the stress-based method (SN) and those by the strain-based method (εN) at the material’s yield stress, namely the moderate-cycle fatigue. As derived from the test data, the SN and εN methods are most widely used in engineering applications. Thus, it is necessary to address this discrepancy between the SN and the εN at material’s yield stress. For ductile metals, the moderate-cycle fatigue is a damage accumulation process in which the elastic strain and plastic strain are comparable. Yield stress is a critical point where the elastic deformation transits to the plastic deformation. Based on a normalized damage concept, we propose a data-driven approach, an elasto-plastic damage accumulation model, to address the fatigue life prediction at the yield stress. By differentiating the damages caused by the elastic and plastic behaviors, the fatigue damage of each loading cycle is formulated as a function of both stress and strain amplitudes. With introducing the strain energy density-based weighting factor for the elastic and plastic behaviors, the proposed model can accord well with the classical methods from low-cycle fatigue to high-cycle fatigue. When it comes to the material’s yield stress, the fatigue life predicted by the proposed model compares favorably with the test data of two different alloys. Therefore, beyond clarifying the mismatch between the classical approaches at the yield stress, the proposed model is expected to find extensive applications in fatigue design and damage evaluation of structures and materials at the yield stress.