Biological invasions have caused the loss of freshwater biodiversity worldwide. The interplay between adaptive responses and demographic characteristics is expected to be important for the resilience of populations to biological invasions, but the interaction between these factors is poorly understood. The native freshwater gastropod Amnicola limosus is distributed along spatial variation in impact from an invasive molluscivorous fish (Neogobius melanostomus), as well as in calcium concentration, which limits the distribution of this invader and thus provides refuges for the gastropods. We investigated if refuge populations could provide migrants to declining invaded gastropod populations through gene flow (i.e., demographic rescue), which could also help maintain genetic diversity (i.e., genetic rescue). We also tested for genetic adaptation of A. limosus to the invasive predator and the low calcium habitats. We conducted pooled whole-genome sequencing of twelve gastropod populations from the Upper St. Lawrence River, complemented with a laboratory reciprocal transplant of wild F0 A. limosus to measure survival and fecundity in treatments of water calcium concentration (low/high) and round goby cue (present/absent). We found that gene flow is restricted from the low-calcium uninvaded refugia towards high-calcium invaded populations, implying that the potential for demographic and genetic rescue is limited. We also detected signatures of divergent selection between habitat types and evidence of low fitness of individuals from refugia populations in both habitat types, which could be either a cause or consequence of the population structure between habitat types and highlights the potential conflict between demographic/genetic rescue and adaptation.