Solutions

We want to find $a,b,c,d$ such that

$$\begin{align*} f(f(x))=x &\implies f\left(\frac{ax+b}{cx+d}\right)=x \\ &\implies \frac{af(x)+b}{cf(x)+d}=x \\ &\implies af(x)+b=xcf(x)+dx \\ &\implies af(x)-xcf(x)=dx-b \\ &\implies f(x)(a-cx)=dx-b \\ &\implies f(x)=\frac{dx-b}{a-cx}. \end{align*}$$

Thus,

$$\frac{ax+b}{cx+d}=\frac{dx-b}{a-cx}.$$

Then

$$\begin{align*} (ax+b)(a-cx)&=(dx-b)(cx+d) \\ a^2x-acx^2+ab-bcx&=cdx^2+d^2x-bcx-bd \\ a^2x-acx^2+ab&=cdx^2+d^2x-bd \\ cdx^2+acx^2+d^2x-a^2x-bd-ab&=0 \\ x^2(cd+ac)+x(d^2-a^2)+(-bd-ab)&=0. \end{align*}$$

For all $x$. Hence, we want $a,b,c,d$ such that the coefficients are all $0$. That is,

$$\begin{align*} cd+ac&=0, \tag{1}\\ (d-a)(d+a)&=0, \tag{2}\\ bd+ab&=0. \tag{3} \end{align*}$$

This gives the solutions:

1. $d=a$, $c=0$, $b=0$, $d\in\mathbb{R}$.
2. $d=-a$, $a,b,c\in\mathbb{R}$.