1. Introduction
In ecological system, predation and competition are often assumed to be the important factors that affect species coexistence [7, 8, 11, 25]. It is further investigated thoroughly in [1, 12, 13]. Gurevitch et al. [6] showed that predator can promote coexistence by lowering the strength of competition. It is a common phenomenon that predator can affect prey populations by direct killing. Recent field studies show that the indirect effect of predator species on prey species has a major impact than direct killing [2-5, 14]. Thus, it is reasonable to incorporate the fear effect in the model focussed on the role of predator regarding coexistence of competing species. This type of mechanism can slow down the competition in respect of resource competition. Thus avoidance behaviour developed by fear usually stimulates coexistence provided prey partition resources, but not predators, whereas it weaken coexistence if prey partition predators but not the resources. Zanette et al. [29] carried out experiments on song sparrows and observed 40% reduction in offspring production due to fear from the predator. With this fact in mind, Wang et al. [27] first developed the predator-prey model incorporating the cost of fear into prey reproduction. They found that the cost of fear has no impact in dynamical behavior when predation follows Holling type I response function whereas it can stabilize the system by discarding periodic orbits considering Holling type II response function. Since then several studies are found in predator-prey models by introducing fear component in prey reproduction. Wang and Zou [28] investigated a predator-prey model with the cost of fear and adaptive avoidance of predators and established that both strong adaption of adult prey and the large cost of fear induces destabilizing effect while large population of predators stabilize the system. Sasmal and Takeuchi [22] discussed the dynamics of a prey-predator model incorporating two facts: fear effect and group defense. Mondal et al. [16] analyzed the predator-prey model considering both the effects of fear and additional food and showed stability of equilibrium points and Hopf bifurcation. Zhang et al. [30] investigated the influence of anti- predator behavior due to fear of predators to a Holling type II prey-predator model allowing a prey refuge and demonstrated the global stability analysis of the equilibria of the model and showed Hopf bifurcation. Previous studies [16-19, 27, 28, 30] are mainly confined in two species that cannot properly explain the fear effect when multiple species are present. So present study attempts to investigate the predator fear which affects prey behavior when prey species are in competition. This study also address the question of species coexistence.
Takeuchi and Adachi [24] studied the following two competing prey and one predator model in Lotka-Volterra form:
\begin{equation} \frac{dx_{1}}{\text{dt}}=x_{1}\left(r_{1}-x_{1}-\alpha x_{2}-\varepsilon y\right),\nonumber \\ \end{equation}\begin{equation} \frac{dx_{2}}{\text{dt}}=x_{2}\left(r_{2}-\text{βx}_{1}-x_{2}-\mu y\right),\nonumber \\ \end{equation}
\(\frac{\text{dy}}{\text{dt}}=y\left(-d+c\varepsilon x_{1}+c\mu x_{2}\right).\)(1) Here the variables\(x_{1}\ \mathrm{\text{and\ }}x_{2}\)represent the densities of prey \(y\) that of predator.\(r_{1}\ \mathrm{\text{and\ }}r_{2}\) are the intrinsic growth rate of prey. \(\text{α\ }\mathrm{\text{and\ }}\beta\) are parameters representing the competitive effects between two prey.\(\text{ε\ }\mathrm{\text{and\ }}\mu\) are coefficients of decrease of prey species due to predation. \(c\) is the equal conversion rate of the predator. All the parameters are assumed to be positive. In [24], the authors showed stability and Hopf bifurcation. They also pointed out that the stable equilibrium bifurcates to a periodic motion with a small amplitude when the predation rate increases and chaotic motion appears when one of two prey is superior than the other. Finally, they remarked that predator mediated coexistence is possible by the close relationship between preferences of a predator and competitive capacities of two prey. However, studies in [24] only considers the effect of direct killing on prey populations and ignore the fear effect in the model equations. In the real world , the intraspecific competition among predator exists. Taking the cost of fear on reproduction term only and intraspecific competition and unequal conversion rate of predator, system (1) becomes
\begin{equation} \frac{dx_{1}}{\text{dt}}=x_{1}\left(\frac{r_{1}}{1+k_{1}y}-x_{1}-\alpha x_{2}-\varepsilon y\right),\nonumber \\ \end{equation}\begin{equation} \frac{dx_{2}}{\text{dt}}=x_{2}\left(\frac{r_{2}}{1+k_{2}y}-\text{βx}_{1}-x_{2}-\mu y\right),\nonumber \\ \end{equation}
\(\frac{\text{dy}}{\text{dt}}=y\left(-d+c_{1}\varepsilon x_{1}+c_{2}\mu x_{2}-hy\right)\)(2)
where \(k_{i},\ i=1,\ 2\) represents the level of fear and \(h\)denotes the intraspecific competition within the predator population.\(c_{i},\ i=1,\ 2\) is the conversion efficiency of the predator. Justification for considering the fear term can be found in [27].
The rest of the paper is organized as follows. In Sec. 3, we state results on positivity and boundedness of the solutions of the system. In Sec. 4, existence and stability of different equilibrium points are discussed. Furthermore, persistence criterion is developed in the same section. Hopf bifurcation around the positive equilibrium point and the nature of the limit cycle emerging through Hopf bifurcation are derived in Sec. 5. Numerical simulations are performed in Sec. 6. A brief discussion concludes in Sec. 7.