4 Ways to Show a Group is Not Simple
You know the Sylow game. You're given a group of a certain order and are asked to show it's not simple. But where do you start? Here are four options that may be helpful when trying to produce a nontrivial normal subgroup.
Option 1: Show there is a unique Sylow p-subgroup.
This is the usually the first thing you want to try, especially if $|G|$ is pretty large. Here your goal is to show that $n_p$ - the number of Sylow $p$-subgroups of $G$ - is equal to 1. This automatically implies that the unique Sylow $p$-subgroup is normal, and so $G$ is not trivial. (Tip: it's often helpful to compute $n_p$ for the largest prime $p$ first.)
Example
Let $G$ be a group of order $520=2^3\cdot 5\cdot 13$. Letting $n_p$ denote the number of Sylow $p$-subgroups of $G$, we see $$n_2\in\{1,5,13,65\}, \qquad n_5\in\{1,26\}, \qquad n_{13}\in\{1,40\}.$$ Suppose neither $n_2,n_5$ nor $n_{13}$ equals 1 (else we're done). Then $n_{13}=40$ and there are $40(13-1)=480$ elements of order 13. Similarly, $n_5=26$ and there are $26(5-1)=104$ elements of order 5. This means $G$ contains at least $480+104=584$ elements, which is impossible. Hence at least one of $n_2,n_5$ or $n_{13}$ equals 1 and so $G$ is not simple.
Option 2: Show ker𝜑 is non-trivial for a homomorphism 𝜑 on G.
Suppose $H\leq G$ is a subgroup of index $n$ with $|G|=p^kn$ where $p\nmid n$ and $k\geq 1$. This trick works whenever $n!<|G|$: Let $G$ act on its collection of cosets $G/H$ by multiplication (so for any $g\in G$ and $xH\in G/H$ let $g\cdot xH:=gxH$). Then define a homomorphism $\varphi$ from $G$ to $S_{G/H}\cong S_n$ by $\varphi(g)=\sigma_g$ where $\sigma_g$ is the permutation given by $\sigma_g(xH)=g\cdot xH=gxH$.
Then $\ker\varphi=\cap_{x\in G}xHx^{-1}$ is a normal subgroup* of $G$ and is nontrivial since if $n!< |G|$ then
- $\ker\varphi\neq \{e\}$, otherwise $\varphi$ is injective which would imply $|G|\leq |S_n|=n!$.
- $\ker\varphi\neq G$, otherwise $|G|=|\cap_{x\in G}xHx^{-1}|\leq |H|$ which isn't possible
Example
Suppose $G$ is a group of order $36=2^2 \cdot3^2$. Let $H$ be a Sylow 3 subgroup of $G$ so that the index of $H$ in $G$ is 4. Let $\varphi:G\to S_4$ be the homomorphism defined above. Then $\ker\varphi= \cap_{x\in G}xHx^{-1}$ is a normal subgroup of $G$. If $\ker\varphi=\{e\}$, then $\varphi$ is injective which implies $36=|G|\leq |S_4|=24$ which is not possible. So $\ker\varphi\neq\{e\}$. Also, $\ker\varphi\neq G$, otherwise $36=|G|=|\cap_{x\in G}xHx^{-1}|\leq |H|=9$, which is also impossible. Thus $\ker\varphi$ is nontrivial and so $G$ is not simple.
Option 3: For Sylow p-subgroups H & K, show H∩K is normal in G.
One way to do this is to show the normalizer of $H\cap K$ is the entire group $G$, in other words $g(H\cap K)g^{-1}=H\cap K$ for all $g\in G$.
Example
Let $G$ be a group of order $108=2^2\cdot 3^3$. The number of Sylow 3-subgroups is either 1 or 4. Assume it's 4 (else we're done). So let $H$ and $K$ be distinct Sylow 3-subgroups, each having order 27.
Let $N=N_G(H\cap K)$ be the normalizer of $H\cap K$. Our goal is to show $N=G$ so that $H\cap K\triangleleft G$ is a nontrivial normal subgroup, proving that $G$ is not simple. First notice that $$|H\cap K|= \frac{|H||K|}{|HK|}\geq \frac{|H||K|}{|G|}=\frac{27\cdot 27}{108}\approx 6.8.$$ We also need $|H\cap K|$ to be a divisor of $27$ since $H\cap K\subset H$. This forces $|H\cap K|=9.$
Observe that $H\cap K\triangleleft H$ and $H\cap K\triangleleft K$ since the index of $H\cap K$ in each of $H$ and $K$ is $27/9=3$ and 3 is the smallest prime divisor of $27=|H|,|K|.$ Thus $H,K\subset N$. This implies $HK\subset N$ too.**
Finally, since $N\leq G$ is a subgroup we know $|N|$ must divide $108=|G|$. But $HK\subset N$ implies $81=|HK|\leq |N|$ as well, so we must have $|N|=108$. This allows us to conclude $N=G$.
Option 4: Find a subgroup of G with index p, where p is the smallest prime divisor of |G|.
Anytime $H\leq G$ is a subgroup with $[G:H]=p$ where $p$ is the smallest prime divisor of $|G|$, it follows that $H$ is normal in $G$. (So, for instance, this is how you know all subgroups of index 2 are normal.) Now you may or may not be able to immediately find such a subgroup given the order of $G$, but it's still something to keep in mind while looking to prove your group is not simple. In the previous example, we used this result to conclude that the intersection $H\cap K$ was not only a subgroup of $H$ and $K$, but was in fact normal. This was a key step in showing $H\cap K$ was normal in the whole group $G$.
Footnotes:
* Let's check that $\ker\varphi$ really is of that form: $\ker\varphi=\{g\in G:\varphi(g)= \text{id}\}$ where $\text{id}$ is the identity permutation in $S_n$. This set is equal to $\{g\in G:gxH=xH \text{ for all $x\in G$ }\}= \{g\in G:x^{-1}gxH=H \text{ for all $x\in G$ }\}$ which is the same as $\{g\in G:g\in xHx^{-1} \text{ for all $x\in G$ }\}$. The latter is precisely $\cap_{x\in G}xHx^{-1}.$ That $\ker\varphi$ is normal in $G$ is clear since $\varphi$ is a homomorphism.
** Proof: Since $H\subset N$ and $K\subset N$, we know $h(H\cap K)h^{-1}=H\cap K$ for all $h\in H$. Likewise $k(H\cap K)k^{-1}=H\cap K$ for all $k\in K$. So for any $hk\in HK$, we have $$hk(H\cap K)(hk)^{-1}=h(k(H\cap K)k^{-1})h^{-1}=h(H\cap K)h^{-1}=H\cap K.$$