Distorion level in the FFT of the sine wave was to high!
Doing the measurement of the board we noticed, that harmonics level in the sine wave spectrum was to high.
As you can see, the second harmonics is only 55 dB below the first harmonic level. The sine wave source was the Rhode & Schwartz SML01 generator - expensive and solid looking device. To verify, if the input signal is free of the harmonics, we've connected another generator from the Agilent network analyser. Unfortunately, spectrum was looking the same, so the conclusion was to focus on the FmcAdc card. We did some additional measurements, for different frequencies, input voltage ranges, and signal amplitude, to find something that could help us in the investigation. The observation that made us suspicious was the harmonics level that did not differ due to the all changes shown above.
Next step was to build filter that would suppress the harmonics, to make sure that input signal is truly clean.
Frequency response - marker at the frequency that is subharmonic of the
sampling frequency (no "signal leakage" in FFT).
This is the frequency of our desired input
signal.
Marker at the second harmonics. Filter's attenuation at this frequency
is 46 dB (-3 dB for the base signal
frequency).
SUCCESS!!! FFT of the filtred signal indicates the second harmonics
level of -90 dB and third harmonics of -80 dB. I don't have to add, that
these values are much closer to our
estimations.