TEBF0808 + TE0803-01 + PCIe ssd

[fpgaguy99]

So I have a setup where I'm using the TEBF0808 starter kit + a TE0803-01 4CG FPGA Module, 2017.4 Vivado/SDK/Petalinux.
I'm getting about an 90% failure rate on detecting my PCIe device on power up.
Every now and then it does work completely to where I can use it for hours until I finally reboot or turn off the system. I am not sure why it is working when it does actually work.
I have the modified FSBL with the PCIe reset requirements in place, and I have the Si5338 configuration in with a custom 5338 project that outputs 100 MHz on all the outputs. I am only using PCIe and don't need USB/Display Port/Firefly etc, so in order to make sure something was being output on the 5338, I set them all the 100 MHz and am looking at the 100 MHz output from the PCIe connector.

I am running out of ideas for why the PCIe card would only work sometimes in linux. Is there something I'm not doing correctly in the FSBL initialization?

I'm getting a proper FSBL boot log with no errors for PCIe or Si5338.

Xilinx Zynq MP First Stage Boot Loader Release 2017.4   Sep  7 2018  -  11:25:02 -------------------------------------------------------------------------------- TE0803 Board Initialisation SI5338 Init Function Si5338 Rev 1 Initialization          Done. PCIe Reset Complete -------------------------------------------------------------------------------- NOTICE:  ATF running on XCZU4CG/silicon v4/RTL5.1 at 0xfffea000, with PMU firmware WARNING: BL31: invalid exception level (3) NOTICE:  BL31: Secure code at 0x0 NOTICE:  BL31: Non secure code at 0x8000000 NOTICE:  BL31: v1.3(debug):4999a52 NOTICE:  BL31: Built : 11:11:52, Jun 27 2018 INFO:    ARM GICv2 driver initialized INFO:    BL31: Initializing runtime services INFO:    BL31: PM Service Init Complete: API v0.3 INFO:    BL31: Preparing for EL3 exit to normal world INFO:    Entry point address = 0x8000000 INFO:    SPSR = 0x3c9 Xilinx Zynq MP First Stage Boot Loader Release 2017.4   Sep  7 2018  -  11:25:02 -------------------------------------------------------------------------------- TE0803 Board Initialisation SI5338 Init Function Si5338 Rev 1 Initialization          Done. PCIe Reset Complete

I'm just not sure what else could be the problem. Is it really supposed to reset PCIe twice?

[fpgaguy99]

Update:

Seems it works great as long as I totally power down the whole board in between tests. I'm getting a pretty much 80% success rate now. Every now and then it doesn't detect it.

[JH]

Hi,
second reset from the module on the first power up is more ore less a fast workaround at the moment for pcie.
Because SI5338 outputs seems to late available after I2C programming for PCIe.

So on second boot the SI CLK is still available and normally this works. CPLD will only one time reboot, if a internal flag is not set --> also only after power down and system starts with modified FSBL and PCIe is detected.
CPLD Source Code and description:

• https://wiki.trenz-electronic.de/display/PD/TEBF0808+CPLD+Firmware
• https://shop.trenz-electronic.de/en/Download/?path=Trenz_Electronic/Modules_and_Module_Carriers/5.2x7.6/5.2x7.6_Carriers/TEBF0808/REV04/Firmware

For TE0808 you can program NVM instead of FSBL reconfiguration of the SI. Unfortunately this is not possible for TE0803 with SI5338.

br
John

[fpgaguy99]

Hey John,

Thanks for the quick response!
Is there any way I can use the FSBL to detect when the Si5338 clocks are ready? I'm surprised the PLL Lock signal isn't enough status for it to be ready.
Could I hold the PCIe reset until that specific time?

Sorry, I'm guess I'm still a little confused about what the best solution for getting PCIe to work 100% is.

Regards
Matt

[JH]

Hi,

there is no quick 100% solution, when I know one, I will implement it.

One way where to use external CLK. 2 of the reference CLK inputs are sourced from carrier.
One is used with a soldered 150MHz OsC for Sata (U23 on TEBF0808) and one is not soldered(U6  on TEBF0808). In case you has the possibility to solder a correct 100MHz Osc there, than you can use this one. Change on PS settings to the other reference clk.

In case you try to find a solution for FSBL, than it will be not only a boot delay after SI5338 initialisation. If I remember correctly, I've tried out this before I've implemented this reboot solution.
PS initialisation of the periphery starts before the SI5338 will be initialized, maybe some of the pcie registers will be set wrong, if the CLK is not available in this step. And than it starts with wrong settings. But this is only a guess at the moment. So maybe reinitialize PCIe part after SI5338 init can be help.  But what exactly is initialized in the default FSBL for PCIe, one would have to look for first in the FSBL code.

br
John

[fpgaguy99]

I appreciate your insights, John. Thank you -- it at least gives some areas to look into. If I figure out a fix, I'll definitely be posting here (or any new questions I can think of).

Thank you!
Matt

[JH]

Hi,

thanks for your offer to share the information, if you find out something. Please let me know if you have some questions or if you find out something which we can discuss to find a better solution for pcie.

br
John

[fpgaguy99]

Any update on status from your end on this problem? I'm finally back to working on this issue and I am not sure how I'm going to get it up and running.
Previously you stated that perhaps the clocks haven't been initiated yet so when the registers are initiated the values may be incorrectly clocked. Are we talking about the 100 MHz clock that the Si5338 generates to the PCIe connector, or the PCIE_CLK_REF that goes to the PCIe controller in the FPGA's PS?

Thanks!

[JH]

Hi, sorry I haven't had time.

   
• PCIe used REFCLK2 with 100MHz --> B505_CLK2 --> SI5338 CLK3 -->100MHz
• SATA REFCLK0 with 150MHz --> B505_CLK0 --> TEBF0808 B505_CLK1 --> 150MHz  (fix)
• DP REFCLK3 with 27 MHz --> B505_CLK3 -> SI5338 CLK2 -->27MHz
• USB0 REFLCLK2 with 100MHz --> B505_CLK2

   
• CLK for PCIe Connector on TEBF0808 is SI5338 CLK0 from TE0803 --> 100MHz

   
• B505_CLK1 --> TEBF0808 B505_CLK0 --> this Osc is not soldered

--> see PS configuration and TE0803 schematics page 3, 13 and 22 and TEB0808 schematic page 4 and 23

On ZynqMP FSBL, SI5338 will be initialised on xfsbl_board.c but PS initialization will be done earlier, so I think at the moment some of the GTP configuration of PCIe will be done not correctly because CLK is not available on this time. After Reboot without power reset it will be available and than it works. 

So goal is to find out if SI5338 initialisation earlier on FSBL helps or if it will be possible to reinitialise  GTP configuration of PCIe again with FSBL after SI configuration is done.--> source code of the current modification is available as SDK template in the Starterkit reference design

br
John

[fpgaguy99]

Thank you John!
I haven't gotten it to work yet -- but as a sanity check, I'm going to replace the Si5338 with a pre-programmed Si5338 and comment out the restart code just to see if that will work alright. If that's the case then I may just go down that route.

[JH]

Hi,

I haven't gotten it to work yet --

so you has an solution for you now? Good.

ut as a sanity check, I'm going to replace the Si5338 w

this will be possible, but maybe you solder a 100MHz Osz on the TEBF0808 and change GTP CLK configuration only:
B505_CLK1 --> TEBF0808 B505_CLK0 --> this Osc is not soldered --> DSC1123 oscillator, U6

PS:Custom order with  your preprogrammed SI5338 will be also possible, but I did not know the prices and the min. quantity which is need for this kind of job. You can ask Trenz Sales support, if you are interested in the future.

br
John

[fpgaguy99]

Ah that's true, then it would just be 100 MHz for PCIe, thank you, that makes sense!
I also bought the Si5338 programmer with some spare Si5338 parts - https://www.digikey.com/product-detail/en/silicon-labs/SI5338-56-PROG-EVB/336-1747-ND/2136037

I'll try a bunch of different things this way.
Thank you for your support, John!