PS->PL EMIO 1 GbE to SFP for TE0715-05 / TEBA0841 carrier issue

[paul]

The design uses Vivado 2023.2 and instantiates the PCS/PMA IP Core in 1000BASE-X mode which is connects to an SFP/SFP+ module installed on the carrier. The GEM GMII and MDIO is connected to the PCS/PMA core viea EMIO.

With an SFP+ 0 dB loopback module installed: https://www.10gtek.com/sfploopback
the Linux kernel is happy and tcpdump shows the transmitted packets successfully being received.

Code Select Expand

Configuring network interfaces... macb e000b000.ethernet eth0: PHY [e000b000.ethernet-ffffffff:01] driver [Xilinx PCS/PMA PHY] (irq=POLL)
macb e000b000.ethernet eth0: configuring for phy/gmii link mode
done.
macb e000b000.ethernet eth0: unable to generate target frequency: 125000000 Hz
macb e000b000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
And looped-back packets:

Code Select Expand

# tcpdump -r pkt_loopback_sfp.pcap
reading from file pkt_loopback_sfp.pcap, link-type EN10MB (Ethernet), snapshot length 262144
19:18:52.589688 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:52.589762 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46
19:18:53.593173 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:53.593226 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46
19:18:54.633168 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:54.633222 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46
19:18:56.590178 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:56.590235 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46
19:18:57.593172 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:57.593227 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46
19:18:58.153179 IP6 fe80::fe0f:e7ff:fe21:3703 > ip6-allrouters: ICMP6, router solicitation, length 16
19:18:58.153237 IP6 fe80::fe0f:e7ff:fe21:3703 > ip6-allrouters: ICMP6, router solicitation, length 16
19:18:58.633170 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 28
19:18:58.633219 ARP, Request who-has 192.168.1.221 tell 192.168.1.220, length 46

The problem occurs when I attempt to communicate with another board. I have tried a 1000BASE-T SFP module, multimode fiber SFP+ module, and a passive DAC twinax cable all with the same failed result:

• System only detects a link if autonegotiation is disabled with ethtool
• Regardless of the autonegotiation setting, no packets are successfully transmitted or received on either end with the TE0715 in the loop.
• I put an ILA on the transceiver status signals from the PCS/PMA core, and see RXNOTINTABLE errors occurring, followed by the link being reset periodically

I instantiated Xilinx's example 1G PS EMIO 1000BASE-X design on a ZCU-102 to communicate with on the other end. These same SFP/SFP+ modules in the ZCU-102 link up fine with a Mikrotik switch, but no luck with the TE0715 at the other end.

I used counters to measure the ratios of the clock signals from the synthesizer on the TE0715 and they look right. Do I need to program the clock synthesizer explicitly?
Any ideas to look at?

Thanks!
Paul

Devicetree:

Code Select Expand

&gem0 {
/delete-property/ local-mac-address;

nvmem-cells = <&eth_addr>;
nvmem-cell-names = "mac-address";

status = "okay";
compatible = "cdns,zynq-gem", "cdns,gem";

phy-mode = "gmii";
phy-handle = <&ethernet_phy1>;

mdio {
#address-cells = <1>;
#size-cells = <0>;

ethernet_phy1: ethernet-phy@1 {
reg = <1>;
xlnx,phy-type = <0x5>;
/* sfp = <&sfp0>; */
};
};
};


[mch]

Hi Paul,

    Are you using our reference design?
        https://wiki.trenz-electronic.de/display/PD/TE0715+Test+Board
        https://www.trenz-electronic.de/Downloads/?path=Trenz_Electronic/Modules_and_Module_Carriers/4x5/TE0715/Reference_Design/2023.2/test_board
    Which Vivado revision are you using?
    We program the PLL chip (U10 SI5338) in our reference design using FSBL during booting via I2C interface. You can find the configuration core of the  PLL in the FSBL source code in the "sw_lib" folder in our reference design folder.
    The problem that the ethernet phy chip in your design cannot be found may be due to resetting. You should reset the ethernet phy chip in FSBL. The reset pin of the ethernet phy chip is connected to MIO50 in the hardware. You can use the AMD API commands in the FSBL code to reset the ethernet phy chip. If you are using our reference design, we did not reset the ethernet phy chip in FSBL code.
    If you look in the "misc/si5338" folder, you will find the ClockBuilder project file for the PLL chip. We have selected IN3 with a frequency of 25MHz as the default input clock to generate a clock with a frequency of 125MHz.

Best regards,

Mohsen Chamanbaz

[paul]

For those who might come across this thread later: the issue was the polarity of the MGT diffpair used for 1000BASE-X (MGT_TX3_P/N and MGT_RX3_P/N) is swapped between the TE0715-05 SOM and the TEBA0841 carrier card.

Enabling the transceiver control and status ports (see attached) and driving both gt0_txpolarity_in and gt0_rxpolarity_in high solved the problem for me.

Clock configuration was fine and handled properly by Trenz's code in FSBL.

Thank you John and Mohsen for your help!

Paul