Distorion level in the FFT of the sine wave was to high!
Doing the measurement of the board we noticed, that harmonics level in the sine wave spectrum was to high.
As you can see, the second harmonics is only 50 dB below the first harmonic level. The sine wave source was the Rhode & Schwartz SML01 generator - expensive and solid looking device. To verify, if the input signal is free of the harmonics, we've connected another generator from the Agilent network analyser. Unfortunately, spectrum was looking the same. We did some additional measurements, for different frequencies, input voltage ranges, and signal amplitude, to find something that could help us in the investigation. The thing that made us suspicious was the harmonics level that did not differ due to the all shown above changes.
Next step was to build filter that would suppress the harmonics, to make sure that input signal if truly clean.