PCIe doesn't work with Reference Design on TE0803 StarterKit

[denial]

Hi,
I'm using a TE0803-03-4AE11-A on a TEBF0808-04A and try to run the StarterKit Reference Design available here. I use the boot.bin from the prebuilt/boot_images/4cg_2gb/u-boot folder and the image.ub from the prebuilt/os/petalinux/2GB folder. I tried three different PCIe cards (i210, ASM3242, RM500Q-GL), but in all cases the kernel reported "nwl-pcie fd0e0000.pcie: Link is DOWN" because the register at 0xFD480238 reads 0x2.

What am I doing wrong? Are there any dip switches which influence PCIe? Do I need a specific CPLD firmware version for this to work?
The SERDES Lane 0 PLL claims to have locked to the refclk (0xFD4023E4) and the refclk on the slot looks ok as well. Are there any other things I can check to narrow down the problem?

I noticed the TEBF0808 uses caps C93 and C94 to decouple the refclk, but the PCI Express Card Electromechanical Specification wants the absolute voltage and crossover point of the clock to be in a specific region. Could that cause any problems? The ZCU102 doesn't do it like that (see figure 3-39 in UG1182).

[JH]

Hi,

can you add this patch from Xilinx:
https://support.xilinx.com/s/article/72992?language=en_US
it's not included in our reference design (XAPP72992 was original needed only up  to 19.x, but xilinx extend to 20.x some weeks ago). The next update of our reference design will be come in Q1 22 with Vivado 2021.2

Check also that your TE0803 is proper connected on the carrier and if all pins are OK.

br
John

[denial]

Thanks, but the problem persists.

There appears to be a little bit of oil inside the PCIe connector of the TEBF0808. Is this normal?

[JH]

Oil? No that's not normal. Can you try to remove it?
br
John

[denial]

Removing more of the oil didn't help. But:

Looping the TX pins to the RX pins at the PCIe connector made the LTSSM (register documented in Xilinx Answer 71210) reach the Configuration state. It was previously stuck in the Polling state. So I figured it must be one of the other pins. Supply voltages looked fine. PERST# is within spec if you ignore that it is briefly high before the FSBL configures the Si5338. That left only the REFCLK.

As I wrote initially the REFCLK is not supposed to be capacitively coupled. I bridged C93 & C94, soldered 47 ohm resistors between the REFCLK pins on the PCIe connector and their adjacent ground pins and configured the first Si5338 output to HCSL instead of LVDS. Et voilà, the link is up and the cards are enumerated. Of course the termination is not ideal, but it works.

Why did you use capacitively coupled LVDS when the Si5338 can do HCSL? The Si5345 providing that clock on the TE0807 and TE0808 also supports HCSL.

[JH]

Hi,
I'll discuss this with our HW developers and get back to you.
br
John

[Vitali]

Hi, please replace C93, C94 to 0 Ohm Resistor and configure output PLL as HCSL. The SI5338 has internal HCSL termination.

[denial]

Thanks, removing the resistors improved the waveform.