Solutions

a) Let

$$f(x)=\frac{\cos(x^2)}{x}.$$

Then

$$\lim_{x\to\infty} f(x)=0$$

and

$$f'(x)=\frac{-2x^2\sin(x^2)-\cos(x^2)}{x^2}=-2\sin(x^2)-\frac{\cos(x^2)}{x^2}.$$

Hence,

$$\lim_{x\to\infty} f'(x)$$

does not exist because

$$\lim_{x\to\infty} -2\sin(x^2)$$

does not exist while

$$\lim_{x\to\infty}\frac{-\cos(x^2)}{x^2}=0 \qquad \text{(see exercise 5-8c)}.$$

b) Suppose

$$\lim_{x\to\infty} f(x)$$

exists and that, for contradiction,

$$\lim_{x\to\infty} f'(x)=l>0$$

without loss of generality. Then by definition there exists $y$ such that

$$\forall x,\ x>y \implies |f'(x)-l|<\frac{l}{2}$$

so

$$\frac{l}{2}<f'(x)<\frac{3l}{2}.$$

But by the mean value theorem this means that for any $x>y$ we have

$$\frac{f(x)-f(y)}{x-y}=f'(z)>\frac{l}{2}$$

for some $z\in(x,y)$. Rearranging the terms we get

$$f(x)>f(y)+\frac{l}{2}(x-y),$$

which means that $f$ is unbounded and therefore contradicts the supposition that

$$\lim_{x\to\infty} f(x)$$

exists.

c) Without loss of generality, suppose for contradiction that

$$\lim_{x\to\infty} f''(x)=l>0.$$

By the same kind of argument put forth in b), we know that

$$\lim_{x\to\infty} f'(x)=\infty.$$

But this implies that $f$ is unbounded and ergo

$$\lim_{x\to\infty} f(x)$$

does not exist.