TEBF0808 / TE0807 SFP+ Ethernet reference design.

[tom_kean]

Hi,

I'm trying to make the TE0807/TEBF0808 combination work with Xilinx AXI Ethernet subsystem cores driving the two SFP+ ports with 1000Base-X modules.  Is there an 'official' reference design or device tree for Petalinux which configures all the necessary motherboard resources to use the SFP+ ports (i.e. SI5345 clock chip, I2C expander which drives the SFP+ control/status signals and SFP+ sockets I2C)?

Thanks in advance.

[JH]

Hi,
we haven't a example for this. I would recommend to start with short copper cable instead of optical. And maybe simple check one time trace with ADM ibert IP. --> the opposite must support PRBS generation for this or loop back to the second SFP port to see if you MGTs lock with your CLK setup.
br
John

[tom_kean]

I spent a couple of weeks working on this and I think I have the device tree for the PHY, SI5345 and the I2Cs which control the SFP socket pretty much worked out but I never got it to work. The extra eth connections come up in Petalinux with ip -s addr but there are no packets received and when I look at the reference clock in the Zynq it has no waveform i.e. it is stuck at 1 or 0.

In case somebody else wants to try and make the SFPs work I can share what I came up with so far. This is trying to use the Xilinx 1G/2.5G ethernet subsystem with the SFP+ connectors under Vivado 2023.1.  I need recent updates to Linux which are included in 2023.1 for other parts of my design so I don't have the option to use older versions of Petalinux.

1. In Vivado use 1000BaseX as the Physical Interface Selection and the LOC constraints for the axi_ethernet subsystems which connect to the SFP should be X0Y14 (sfp_(rx/ tx)(p/n) _1 and X0Y15 (sfp_rx/ tx)(p/n)_0

2. If you need both SFP+ one of the AXI Ethernet IPs should use shared logic so they can both run off the same transceiver clock.

3. The clock to use is output 7 from the SI5345 PLL mgt_clk_n_0 / mgt_clk_p_0 in Vivado.  Pins F9 and F10.

4. Two regulators and a clock source need to be added in the system-user.dtsi file for the use of the si5345 chip.

/* 50MHz crystal connected to XA,XB inputs of Si5345 chip in page 24 of the TE0807-7DE21-A schematic */
   ref50: ref50M{
         compatible = "fixed-clock";
         #clock-cells = <0>;
         clock-frequency = <50000000>;
   };
   
   // SI5345 driver needs a regulator connection to specify a 1.8V supply or it will default to 2.5V.
   clk_vdd: fixedregulator {
         compatible = "regulator-fixed";
         regulator-name = "1v8-supply";
         regulator-min-microvolt = <1800000>;
         regulator-max-microvolt = <1800000>;
   };
   // See TE0807 schematic page 24,
   clk_vdda: fixedregulator {
         compatible = "regulator-fixed";
         regulator-name = "3v3-supply";
         regulator-min-microvolt = <3300000>;
         regulator-max-microvolt = <3300000>;
   };
   

5. The si5345 goes in system-user.dtsi at i2cswitch@77 i2c@4. I'm only concerned about the SFP clock and I haven't worked out the settings for any other clocks generated by the SI5345. I'm not using the Trenz custom FSBL to set up the clock chip because it
   didn't work out of the box with Vivado 2023.1.

i2c@4 { //Module PLL Si5338 or SI5345
            reg = <4>;
            si5345: clock_generator@69 {
                reg = <0x69>;   // Table 14, TE807 TRM says "I2C address in current configuration 1101001b"
                compatible = "silabs,algsi5345";
                #clock-cells = <2>;
                #address-cells = <1>;
                #size-cells = <0>;
                clocks = <&ref50>;
                clock-names = "xtal";
                vdd-supply = <&clk_vdd>;
                vdda-supply = <&clk_vdda>;
                // No pin called VDDS on schemation so not using the vdds-supply keyword
               
                silabs,pll-m-num = <14000>; // PLL at 14.0GHz
                silabs,pll-m-den = <50>;
   
                // Copied from silabs,si5341.txt example device tree out@6 in their example but out@7 here.
                // Clock 7 in Trenz board goes to Zynq Ultrascale pins F10,F9 which are mgt_clk_0 in Vivado project.
                out@7 {
                     reg = <7>;
                     vddo7-supply = <&clk_vdd>;
                     silabs,format = <1>; // LVDS 1.8V
                     silabs,common-mode = <13>;
                     silabs,amplitude = <3>;
                     always-on;
                     // This code is based on the example for some-video-node in linux/Documentation/devicetree/bindings/clock/silabs,si5341.txt
                // Use synthesiser 3 as the parent for output 7 (0 => output, 1=> synthesiser)
                assigned-clocks = <&si5345 0 7>, <&si5345 1 3>;
                assigned-clock-parents = <&si5345 1 3>;
                // Set output 7 to 125MHz using a synth frequency of 500MHz
                assigned-clock-rates = <125000000>, <500000000>;
                };
           };
        };

6. As well as the clock chip the two sfp sockets get an entry in the device tree.


// See sheet 9 of TEBF0808 schematic
/ {  sfp1: sfp_1 {
   compatible = "sff,sfp";
   i2c-bus = <&i2c_sfp1>;
   /* Module prescence input - not used Xilinx AXI ethernet device driver won't look at it
   # mod-def0-gpios =
   # Loss of signal input GPIO - not used Xilinx device driver won't look at it
   # los-gpios =
   # TX fault - not used Xilinx device driver won't look at it
   # tx-fault-gpios =
   # Disable transmit output GPIO. This is connected to the CPLD on the TEBF0808 motherboard which is programmed to connect it to Constant 0
   # no device-tree code needed. wiki.trenz-electronic.de/display/PD/TEBF0808+CPLD  Signal SFP1_TX_DIS
   # tx-disable-gpios =
   */
        // The rate select signals are now handled by hog connections in the i2c chip entry.
   // Rate select for receive. 1= high speed 0 = low speed. Possibly ignored by 1000Base-X module. Signal SFP1_RS0. P5 on PCF8574
   //rate-select0-gpios = <&pcf8574 5 GPIO_ACTIVE_HIGH>;
   // Rate select for transmit 1= high speed 0 = low speed. Possibly ignored by 1000Base-X module. Signal SFP1_RS1. P4 on PCF8574
   //rate-select1-gpios = <&pcf8574 4 GPIO_ACTIVE_HIGH>;   
   };

   sfp2: sfp_2 {
   compatible = "sff,sfp";
   i2c-bus = <&i2c_sfp2>;
   // Rate select for receive. 1= high speed 0 = low speed. Possibly ignored by 1000Base-X module. Signal SFP2_RS0. P2 on PCF8574
   //rate-select0-gpios = <&pcf8574 2 GPIO_ACTIVE_HIGH>;
   // Rate select for transmit 1= high speed 0 = low speed. Possibly ignored by 1000Base-X module. Signal SFP2_RS1. P3 on PCF8574
   //rate-select1-gpios = <&pcf8574 3 GPIO_ACTIVE_HIGH>;   
   };
   

7. The I2C i2cswitch@73 / i2c@1 is used to put constant values on the rate select inputs to the SFP to ensure it selects the high rate. I am not sure this is necessary but I was feeling paranoid.

i2c@1 { // SFP TEBF0808 PCF8574DWR
            reg = <1>;
            pcf8574: gpio@26 {
               reg = <0x26>;
               // Address = 0x26 so we want pcf8574 (possible addresses 0x20-0x27)not pcf8574a (possible addresses 0x30..3f)
               compatible = "nxp,pcf8574";
               gpio-controller;
               #gpio-cells = <2>;
               sfp1_rs0-hog {
                  gpio-hog;
                  gpios = <5 0>;
                  output-high;
                  };
               sfp1_rs1-hog {
                  gpio-hog;
                  gpios = <4 0>;
                  output-high;
                  };
               sfp2_rs0-hog {
                  gpio-hog;
                  gpios = <2 0>;
                  output-high;
                  };
               sfp2_rs1-hog {
                  gpio-hog;
                  gpios = <3 0>;
                  output-high;
                  };
                 
            };
               
        };

8. Two other connections on the i2cswitch@73 switch are used with the SFP ports and need labelled so they can be referenced in the sfp device entries.  This lets the SFP driver read back small ROMs in the transceivers within the SFP sockets to get ID information. When you enable them you will see the transceiver ID in the boot log.

       i2c_sfp1: i2c@3{ // SFP1 TEBF0808
            reg = <3>;
        };
        i2c_sfp2: i2c@4 {// SFP2 TEBF0808
            reg = <4>;
        };

9. The transmit enable for the SFPs is coming from the CPLD on the TEBF0808 baseboard and will always be 1. Nothing is needed in the device tree.

10. The AXI ethernet should probably have the names of its clock changed to match the clock chip output. This is cosmetic.

      &axi_ethernet_1 {
        // Replace master_clk dummy clocks in Xilinx generated pl.dtsi with references to actual clock from the PLL.         
       // **** need to change the first few signal names according to the clock names in your design *****
   clocks = <&zynqmp_clk 73>, <&zynqmp_clk 73>, <&zynqmp_clk 74>, <&si5345 0 7>, <&si5345 0 7>;
       };

11. The PHY device for the axi ethernet should be tweaked from the version output by Vivado so it references the SFP that it uses.
  &axi_ethernet_1phy1 {
        compatible = "Xilinx PCS/PMA PHY";
        phy-mode = "1000base-x";
        sfp = <&sfp1>;
};

[JH]

Hi,
Hi, sorry that you have invested so much time so far and it still didn't work out.

The question is, is maybe SI5345 programming over Linux to late. This depends mostly on the protocols and AMD use sometime preliminary configuration inside the FSBL or maybe in your case PL IP reset is maybe released to early?
We add Si5345 runtime programming into our FSBL(see reference design download) before GTR are initialised.  Maybe you can try this also? Alternativle, SI5345 NVM can be programmed via user over I2C, see:
https://wiki.trenz-electronic.de/display/PD/Si5345

If you are lucky this helps, I don't know if it does, but it is at least a possible source of error.

br
John

[tom_kean]

Hi John,

I found this post on the forum which describes a similar problem to what I am seeing.

https://forum.trenz-electronic.de/index.php/topic,1775.0.html

I took the circuit from this post to get the transceiver clock onto normal interconnect and had a look at it with the ILA core and it isn't moving. I don't think a reset issue would do that because the complex ethernet core isn't involved, it's just a couple of buffers.  My guess at this point is that either the device tree entry for the clock chip is wrong in some way I don't understand or there is s similar hardware issue to what the person who wrote this post found. The obvious next step would be to probe output 7 from the clock chip with a scope but I am not set up to probe nets on these kind of find pitched surface mount devices.  In the end I decided to switch to another board which has multiple RJ45 1G ethernets connected to the PL using RGMII rather than transceivers so I could make progress on the main part of my project.

[JH]

I took the circuit from this post to get the transceiver clock onto normal interconnect and had a look at it with the ILA core and it isn't moving. I don't think a reset issue would do that because the complex ethernet core isn't involved, it's just a couple of buffers

You mean from the other post? Yes, there were only buffers in there. But if you don't connect them properly, you won't see anything.  As far as I can see, he had set the buffers correctly on the other post and then divided CLK down and simply checked whether the counter was working.
The other customer did not comment further on the lack of resistors. We do voltage tests here before delivery, so that should actually be covered.

In the end I decided to switch to another board which has multiple RJ45 1G ethernets connected to the PL using RGMII rather than transceivers so I could make progress on the main part of my project.

Ok, I'm sorry I couldn't be of more help. Good luck with your project.

br
John