Yes (I am not expert in neural nets, but that appears to be exactly what this is saying). If you look at what goes into a neural net and compare it to what goes into a Fourier transform, it should be obvious that neural nets have even more than they actually need to do this task.

This statement doesn't make sense to me. A neural network literally can't produce anything besides a continuous function, and the universality theorem says that there is no continuous function they can't (approximately) produce.

So what could you possibly mean when you say neural networks have "more" than they need to do something which characterizes exactly what they can and can't do?

There are more coefficients than necessary. The FT has the minimum; it's bijective.