Consider $x^{4}-x^{3}-2x^{2}-x^{2}y^{2}+xy^{2}+2y^{2}$. Do the grouping $x^{4}-x^{3}-2x^{2}-x^{2}y^{2}+xy^{2}+2y^{2}=\left(x^{4}-x^{3}-2x^{2}\right)+\left(-x^{2}y^{2}+xy^{2}+2y^{2}\right)$, and factor out $x^{2}$ in the first and $-y^{2}$ in the second group.
Consider $x^{2}-x-2$. Factor the expression by grouping. First, the expression needs to be rewritten as $x^{2}+ax+bx-2$. To find $a$ and $b$, set up a system to be solved.
$$a+b=-1$$ $$ab=1\left(-2\right)=-2$$
Since $ab$ is negative, $a$ and $b$ have the opposite signs. Since $a+b$ is negative, the negative number has greater absolute value than the positive. The only such pair is the system solution.
$$a=-2$$ $$b=1$$
Rewrite $x^{2}-x-2$ as $\left(x^{2}-2x\right)+\left(x-2\right)$.
$$\left(x^{2}-2x\right)+\left(x-2\right)$$
Factor out $x$ in $x^{2}-2x$.
$$x\left(x-2\right)+x-2$$
Factor out common term $x-2$ by using distributive property.
$$\left(x-2\right)\left(x+1\right)$$
Consider $x^{2}-y^{2}$. The difference of squares can be factored using the rule: $a^{2}-b^{2}=\left(a-b\right)\left(a+b\right)$.
