TE0808 PS Rootport hanging

[liamm1978]

Hi, I have an issue when I activate the PS rootport on the TE0808. Linux does not fully boot and gets stuck in a probe cycle:

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.4.0-xilinx-v2020.2 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 SMP Fri Dec 17 01:59:22 UTC 2021
[    0.000000] Machine model: xlnx,zynqmp
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '115200n8')
[    0.000000] printk: bootconsole [cdns0] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 256 MiB at 0x0000000068400000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 21 pages/cpu s48600 r8192 d29224 u86016
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] Speculative Store Bypass Disable mitigation not required
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1031940
[    0.000000] Kernel command line:  earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/ram0 rw
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x7bf00000-0x7ff00000] (64MB)
[    0.000000] Memory: 3759452K/4193280K available (11324K kernel code, 676K rwdata, 3460K rodata, 704K init, 556K bss, 171684K reserved, 262144K cma-reserved)
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f902f000
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] random: get_random_bytes called from start_kernel+0x2a8/0x42c with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 33.33MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x7b00c47c0, max_idle_ns: 440795202120 ns
[    0.000003] sched_clock: 56 bits at 33MHz, resolution 30ns, wraps every 2199023255541ns
[    0.008242] Console: colour dummy device 80x25
[    0.012389] Calibrating delay loop (skipped), value calculated using timer frequency.. 66.66 BogoMIPS (lpj=133333)
[    0.022668] pid_max: default: 32768 minimum: 301
[    0.027384] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.034614] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.043255] ASID allocator initialised with 32768 entries
[    0.047804] rcu: Hierarchical SRCU implementation.
[    0.052727] EFI services will not be available.
[    0.057128] smp: Bringing up secondary CPUs ...
[    0.202290] Detected VIPT I-cache on CPU1
[    0.202318] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.579257] Detected VIPT I-cache on CPU2
[    0.579274] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.955399] Detected VIPT I-cache on CPU3
[    0.955416] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.955464] smp: Brought up 1 node, 4 CPUs
[    0.985316] SMP: Total of 4 processors activated.
[    0.989988] CPU features: detected: 32-bit EL0 Support
[    0.995092] CPU features: detected: CRC32 instructions
[    1.000231] CPU: All CPU(s) started at EL2
[    1.004270] alternatives: patching kernel code
[    1.009645] devtmpfs: initialized
[    1.015496] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    1.021659] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    1.034026] xor: measuring software checksum speed
[    1.072554]    8regs     :  2606.000 MB/sec
[    1.112578]    32regs    :  2989.000 MB/sec
[    1.152607]    arm64_neon:  2595.000 MB/sec
[    1.152642] xor: using function: 32regs (2989.000 MB/sec)
[    1.156540] pinctrl core: initialized pinctrl subsystem
[    1.162017] DMI not present or invalid.
[    1.165722] NET: Registered protocol family 16
[    1.170759] DMA: preallocated 256 KiB pool for atomic allocations
[    1.175988] audit: initializing netlink subsys (disabled)
[    1.181444] audit: type=2000 audit(1.124:1): state=initialized audit_enabled=0 res=1
[    1.181779] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    1.195878] Serial: AMBA PL011 UART driver
[    1.209199] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.210248] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    1.216918] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.223580] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    2.199200] cryptd: max_cpu_qlen set to 1000
[    2.219960] DRBG: Continuing without Jitter RNG
[    2.294421] raid6: neonx8   gen()  1695 MB/s
[    2.362463] raid6: neonx8   xor()  1603 MB/s
[    2.430501] raid6: neonx4   gen()  1620 MB/s
[    2.498549] raid6: neonx4   xor()  1557 MB/s
[    2.566612] raid6: neonx2   gen()  1232 MB/s
[    2.634632] raid6: neonx2   xor()  1287 MB/s
[    2.702730] raid6: neonx1   gen()   799 MB/s
[    2.770717] raid6: neonx1   xor()   966 MB/s
[    2.838783] raid6: int64x8  gen()  1275 MB/s
[    2.906792] raid6: int64x8  xor()   833 MB/s
[    2.974850] raid6: int64x4  gen()  1072 MB/s
[    3.042908] raid6: int64x4  xor()   805 MB/s
[    3.110989] raid6: int64x2  gen()   743 MB/s
[    3.178989] raid6: int64x2  xor()   649 MB/s
[    3.247095] raid6: int64x1  gen()   492 MB/s
[    3.315099] raid6: int64x1  xor()   494 MB/s
[    3.315134] raid6: using algorithm neonx8 gen() 1695 MB/s
[    3.319095] raid6: .... xor() 1603 MB/s, rmw enabled
[    3.324025] raid6: using neon recovery algorithm
[    3.328898] iommu: Default domain type: Translated
[    3.333648] SCSI subsystem initialized
[    3.337255] mc: Linux media interface: v0.10
[    3.341428] videodev: Linux video capture interface: v2.00
[    3.346877] pps_core: LinuxPPS API ver. 1 registered
[    3.351792] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    3.360881] PTP clock support registered
[    3.364775] EDAC MC: Ver: 3.0.0
[    3.368217] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI mbox with TX/RX channels.
[    3.376515] FPGA manager framework
[    3.379832] Advanced Linux Sound Architecture Driver Initialized.
[    3.386025] Bluetooth: Core ver 2.22
[    3.389342] NET: Registered protocol family 31
[    3.393746] Bluetooth: HCI device and connection manager initialized
[    3.400062] Bluetooth: HCI socket layer initialized
[    3.404905] Bluetooth: L2CAP socket layer initialized
[    3.409925] Bluetooth: SCO socket layer initialized
[    3.415023] clocksource: Switched to clocksource arch_sys_counter
[    3.420910] VFS: Disk quotas dquot_6.6.0
[    3.424754] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    3.435209] NET: Registered protocol family 2
[    3.436181] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    3.444389] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    3.452402] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    3.459835] TCP: Hash tables configured (established 32768 bind 32768)
[    3.466034] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    3.472698] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    3.479863] NET: Registered protocol family 1
[    3.484283] RPC: Registered named UNIX socket transport module.
[    3.489939] RPC: Registered udp transport module.
[    3.494607] RPC: Registered tcp transport module.
[    3.499277] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.505890] PCI: CLS 0 bytes, default 64
[    3.509667] Trying to unpack rootfs image as initramfs...
[    3.871342] Freeing initrd memory: 8528K
[    4.007695] Initialise system trusted keyrings
[    4.007821] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[    4.013638] NFS: Registering the id_resolver key type
[    4.017933] Key type id_resolver registered
[    4.022097] Key type id_legacy registered
[    4.026056] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    4.032723] jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
[    4.053410] NET: Registered protocol family 38
[    4.053450] Key type asymmetric registered
[    4.056288] Asymmetric key parser 'x509' registered
[    4.061150] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    4.068481] io scheduler mq-deadline registered
[    4.072978] io scheduler kyber registered
[    4.100105] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    4.101727] Serial: AMBA driver
[    4.105659] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    4.113851] brd: module loaded
[    4.117826] loop: module loaded
[    4.118488] mtdoops: mtd device (mtddev=name/number) must be supplied
[    4.123615] libphy: Fixed MDIO Bus: probed
[    4.127581] tun: Universal TUN/TAP device driver, 1.6
[    4.131684] CAN device driver interface
[    4.136584] rtc_zynqmp ffa60000.rtc: registered as rtc0
[    4.140647] i2c /dev entries driver
[    4.145628] Bluetooth: HCI UART driver ver 2.3
[    4.148481] Bluetooth: HCI UART protocol H4 registered
[    4.153581] Bluetooth: HCI UART protocol BCSP registered
[    4.158868] Bluetooth: HCI UART protocol LL registered
[    4.163959] Bluetooth: HCI UART protocol ATH3K registered
[    4.169333] Bluetooth: HCI UART protocol Three-wire (H5) registered
[    4.175578] Bluetooth: HCI UART protocol Intel registered
[    4.180924] Bluetooth: HCI UART protocol QCA registered
[    4.186222] EDAC MC: ECC not enabled
[    4.189748] EDAC DEVICE0: Giving out device to module edac controller cache_err: DEV edac (POLLED)
[    4.198696] EDAC DEVICE1: Giving out device to module zynqmp-ocm-edac controller zynqmp_ocm: DEV ff960000.memory-controller (INTERRUPT)
[    4.210891] sdhci: Secure Digital Host Controller Interface driver
[    4.216812] sdhci: Copyright(c) Pierre Ossman
[    4.221134] sdhci-pltfm: SDHCI platform and OF driver helper
[    4.226983] ledtrig-cpu: registered to indicate activity on CPUs
[    4.232770] zynqmp_firmware_probe Platform Management API v1.1
[    4.238527] zynqmp_firmware_probe Trustzone version v1.0
[    4.266124] alg: No test for xilinx-zynqmp-aes (zynqmp-aes)
[    4.267804] zynqmp_aes zynqmp_aes: AES Successfully Registered
[    4.267804]
[    4.273592] alg: No test for xilinx-keccak-384 (zynqmp-keccak-384)
[    4.281065] alg: No test for xilinx-zynqmp-rsa (zynqmp-rsa)
[    4.288689] ARM CCI_400_r1 PMU driver probed
[    4.289114] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager registered
[    4.296693] pktgen: Packet Generator for packet performance testing. Version: 2.75
[    4.303765] Initializing XFRM netlink socket
[    4.307563] NET: Registered protocol family 10
[    4.312266] Segment Routing with IPv6
[    4.315633] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    4.321711] NET: Registered protocol family 17
[    4.325836] NET: Registered protocol family 15
[    4.330252] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    4.343128] can: controller area network core (rev 20170425 abi 9)
[    4.349287] NET: Registered protocol family 29
[    4.353680] can: raw protocol (rev 20170425)
[    4.357918] can: broadcast manager protocol (rev 20170425 t)
[    4.363546] can: netlink gateway (rev 20190810) max_hops=1
[    4.369052] Bluetooth: RFCOMM TTY layer initialized
[    4.373839] Bluetooth: RFCOMM socket layer initialized
[    4.378949] Bluetooth: RFCOMM ver 1.11
[    4.382660] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    4.387932] Bluetooth: BNEP filters: protocol multicast
[    4.393126] Bluetooth: BNEP socket layer initialized
[    4.398053] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    4.403937] Bluetooth: HIDP socket layer initialized
[    4.408958] 9pnet: Installing 9P2000 support
[    4.413120] Key type dns_resolver registered
[    4.417547] registered taskstats version 1
[    4.421412] Loading compiled-in X.509 certificates
[    4.427028] Btrfs loaded, crc32c=crc32c-generic
[    4.438246] ff000000.serial: ttyPS0 at MMIO 0xff000000 (irq = 43, base_baud = 6249999) is a xuartps
[    4.447266] printk: console [ttyPS0] enabled
[    4.447266] printk: console [ttyPS0] enabled
[    4.451560] printk: bootconsole [cdns0] disabled
[    4.451560] printk: bootconsole [cdns0] disabled
[    4.460735] of-fpga-region fpga-full: FPGA Region probed
[   25.519013] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   25.524927] rcu:     1-...0: (0 ticks this GP) idle=a26/1/0x4000000000000000 softirq=348/348 fqs=1162
[   25.533960]  (detected by 0, t=5263 jiffies, g=-843, q=8)
[   25.539340] Task dump for CPU 1:
[   25.542552] kworker/1:1     R  running task        0    32      2 0x0000000a
[   25.549598] Workqueue: events deferred_probe_work_func
[   25.554721] Call trace:
[   25.557155]  __switch_to+0x1c4/0x288
[   25.560719]  0x3fae
[   88.543013] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   88.548924] rcu:     1-...0: (0 ticks this GP) idle=a26/1/0x4000000000000000 softirq=348/348 fqs=1769
[   88.557958]  (detected by 3, t=21019 jiffies, g=-843, q=8)
[   88.563424] Task dump for CPU 1:
[   88.566636] kworker/1:1     R  running task        0    32      2 0x0000000a
[   88.573670] Workqueue: events deferred_probe_work_func
[   88.578796] Call trace:
[   88.581230]  __switch_to+0x1c4/0x288
[   88.584786]  0x3fae

I am using Vivado, Vitis and PetaLinux 2020.2 on a custom carrier board. I have verified the hardware pins and clocks and I am using MIO31 for active low PCI reset in the TE adapted FSBL.

Even with the endpoint device(Samsung NVMe) removed, the linux boot has the same behaviour.

Any help appreciated!

[JH]

Hi,
I'm not sure if your boot problem belong to pcie at this point.
I would suggest you remove at first PCIe from your PS IP and regenerate linux again (or create new linux) with the new XSA to see if it boots than.

PS: for PCIe on GTR see also: https://www.xilinx.com/support/answers/72992.html

br
John

[liamm1978]

Hi John,

Thanks for your response.

So, my design is booting petalinux if I disable the PCI on the block design.

The next step I am taking is to uprev to the 2021.1 tools, as per the design advisory.

Thanks,

Liam.