PetaLinux Kernel Oops on memset reserved memory

[DanielH]

Subject: PetaLinux Kernel Oops on memset shared memory

Hi there,

(sorry if I posted in the wrong sub-forum)
I'm using a TE0807-02-07EV-1E with a ZynQMP on a TEBF0808. I'm building/using PetaLinux 2020.2.

For the project we are working on, I need to use Shared Memory between the APU and the RPU, but I cannot even get some memory which I allocate to a PetaLinux Kernel module to be set to 0 without receiving a Kernel Oops.
I posted the question to the Xilinx support forum already [1] and the Xilinx colleague couldn't reproduce my error, so I assume it might be related somehow to the Trenz devices.

Is there a possibility for someone to reproduce my setup and test it?

Basically, what I do is:

Code Select Expand

petalinux-create -n testproject -t project --template zynqMP
[Copy StarterKit_7ev_1e_4gb.xsa from [2] into project directory]
petalinux-config --get-hw-description=StarterKit_7ev_1e_4gb.xsa
[Don't change anything in the config menu]
petalinux-create -t modules --name dummy --enable
[Copy over my module source file and the corresponding system-user.dtsi]
petalinux-build
cd image/linux
petalinux-package --boot --fsbl ./zynqmp_fsbl.elf --fpga ./system.bit --u-boot --atf ./bl31.elf --kernel ./image.ub
[Copy to the two partitions of SD card]

The part from the boot output where the dummy module is being loaded:

Code Select Expand

[    6.188917] dummy: loading out-of-tree module taints kernel.
[    6.198746] <1>Hello module world.
[    6.202253] <1>Module parameters were (0xdeadbeef) and "default"
[    6.208544] dummy reserved-driver@0: Device Tree Probing
[    6.214125] dummy reserved-driver@0: Allocated reserved memory, vaddr: 0xFFFF800015000000, paddr: 0x3ED80000
[    6.223989] Unable to handle kernel paging request at virtual address ffff800015000000
[    6.231909] Mem abort info:
[    6.234689]   ESR = 0x96000061
[    6.237745]   EC = 0x25: DABT (current EL), IL = 32 bits
[    6.243056]   SET = 0, FnV = 0
[    6.246129]   EA = 0, S1PTW = 0
[    6.249270] Data abort info:
[    6.252161]   ISV = 0, ISS = 0x00000061
[    6.255990]   CM = 0, WnR = 1
[    6.258945] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000001004000
[    6.265671] [ffff800015000000] pgd=000000087ffff003, pud=000000087fffe003, pmd=00000008764e7003, pte=006800003ed80707
[    6.276294] Internal error: Oops: 96000061 [#1] SMP
[    6.281162] Modules linked in: dummy(O+)
[    6.285081] CPU: 2 PID: 383 Comm: udevd Tainted: G           O      5.4.0-xilinx-v2020.2 #1
[    6.293418] Hardware name: xlnx,zynqmp (DT)
[    6.297588] pstate: 40000005 (nZcv daif -PAN -UAO)
[    6.302378] pc : __memset+0x16c/0x188
[    6.306049] lr : dummy_probe+0xf0/0x130 [dummy]
[    6.310570] sp : ffff8000140238f0
[    6.313868] x29: ffff8000140238f0 x28: 0000000000000100
[    6.319171] x27: ffff80001013f510 x26: 000000000000002b
[    6.324466] x25: ffff000879ff4080 x24: 0000000000000029
[    6.329761] x23: 000000003ed80000 x22: ffff00087a2d2080
[    6.335056] x21: ffff800015000000 x20: 0000000000000000
[    6.340350] x19: ffff00087ab10c10 x18: 0000000000000010
[    6.345645] x17: ffff00087a804888 x16: ffff00087a8048a8
[    6.350940] x15: ffff00087a2d24a8 x14: 3030303030353130
[    6.356235] x13: 3030384646464678 x12: 30203a7264646176
[    6.361529] x11: 202c79726f6d656d x10: 2064657672657365
[    6.366824] x9 : 0000000000000000 x8 : ffff800015000000
[    6.372119] x7 : 0000000000000000 x6 : 000000000000003f
[    6.372795] random: fast init done
[    6.377414] x5 : 0000000000000040 x4 : 0000000000000000
[    6.386102] x3 : 0000000000000004 x2 : 0000000000ffffc0
[    6.391397] x1 : 0000000000000000 x0 : ffff800015000000
[    6.396692] Call trace:
[    6.399128]  __memset+0x16c/0x188
[    6.402435]  platform_drv_probe+0x50/0xa0
[    6.406435]  really_probe+0xd8/0x2f8
[    6.410001]  driver_probe_device+0x54/0xe8
[    6.414080]  device_driver_attach+0x6c/0x78
[    6.418246]  __driver_attach+0x54/0xd0
[    6.421980]  bus_for_each_dev+0x6c/0xc0
[    6.425808]  driver_attach+0x20/0x28
[    6.429367]  bus_add_driver+0x148/0x1e0
[    6.433194]  driver_register+0x60/0x110
[    6.437013]  __platform_driver_register+0x44/0x50
[    6.441704]  dummy_init+0x44/0x1000 [dummy]
[    6.445878]  do_one_initcall+0x50/0x190
[    6.449704]  do_init_module+0x50/0x1f0
[    6.453444]  load_module+0x1ca4/0x2218
[    6.457176]  __do_sys_finit_module+0xd0/0xe8
[    6.461429]  __arm64_sys_finit_module+0x1c/0x28
[    6.465945]  el0_svc_common.constprop.0+0x68/0x160
[    6.470728]  el0_svc_handler+0x6c/0x88
[    6.474467]  el0_svc+0x8/0xc
[    6.477335] Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428)
[    6.483417] ---[ end trace 3bace3fb81fd9481 ]---
[    6.488999] udevd[368]: worker [383] terminated by signal 11 (Segmentation fault)
[    6.496542] udevd[368]: worker [383] failed while handling '/devices/platform/reserved-driver@0'

My dummy.c Kernel module code:

Code Select Expand

/*  dummy.c - The simplest kernel module.

* Copyright (C) 2013 - 2016 Xilinx, Inc
*
*   This program is free software; you can redistribute it and/or modify
*   it under the terms of the GNU General Public License as published by
*   the Free Software Foundation; either version 2 of the License, or
*   (at your option) any later version.

*   This program is distributed in the hope that it will be useful,
*   but WITHOUT ANY WARRANTY; without even the implied warranty of
*   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
*   GNU General Public License for more details.
*
*   You should have received a copy of the GNU General Public License along
*   with this program. If not, see <http://www.gnu.org/licenses/>.

*/
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/io.h>
#include <linux/interrupt.h>

#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_platform.h>

/* Standard module information, edit as appropriate */
MODULE_LICENSE("GPL");
MODULE_AUTHOR
    ("Xilinx Inc.");
MODULE_DESCRIPTION
    ("dummy - loadable module template generated by petalinux-create -t modules");

#define DRIVER_NAME "dummy"

/* Simple example of how to receive command line parameters to your module.
   Delete if you don't need them */
unsigned myint = 0xdeadbeef;
char *mystr = "default";

module_param(myint, int, S_IRUGO);
module_param(mystr, charp, S_IRUGO);

struct dummy_local {
        int irq;
        unsigned long mem_start;
        unsigned long mem_end;
        void __iomem *base_addr;
};

static irqreturn_t dummy_irq(int irq, void *lp)
{
        printk("dummy interrupt\n");
        return IRQ_HANDLED;
}

static int dummy_probe(struct platform_device *pdev)
{
        struct resource *r_irq; /* Interrupt resources */
        struct resource *r_mem; /* IO mem resources */
        struct device *dev = &pdev->dev;
        struct dummy_local *lp = NULL;
        struct resource r;
        void *paddr;
        void *vaddr;
        struct device_node *np = NULL;

        int rc = 0;
        dev_info(dev, "Device Tree Probing\n");

        /* Get reserved memory region from Device-tree */
        np = of_parse_phandle(dev->of_node, "memory-region", 0);
        if (!np) {
          dev_err(dev, "No %s specified\n", "memory-region");
          return -ENODEV;
        }

        rc = of_address_to_resource(np, 0, &r);
        if (rc) {
          dev_err(dev, "No memory address assigned to the region\n");
          return -ENODEV;
        }

        paddr = (void *)r.start;
        vaddr = devm_memremap(&pdev->dev, r.start, resource_size(&r), MEMREMAP_WT);
        dev_info(dev, "Allocated reserved memory, vaddr: 0x%0llX, paddr: 0x%0llX\n", (u64)vaddr, paddr);

        memset(vaddr, 0, resource_size(&r));
        dev_info(dev, "Memset done\n");

        return 0;
}

static int dummy_remove(struct platform_device *pdev)
{
        struct device *dev = &pdev->dev;
        struct dummy_local *lp = dev_get_drvdata(dev);
        free_irq(lp->irq, lp);
        iounmap(lp->base_addr);
        release_mem_region(lp->mem_start, lp->mem_end - lp->mem_start + 1);
        kfree(lp);
        dev_set_drvdata(dev, NULL);
        return 0;
}

#ifdef CONFIG_OF
static struct of_device_id dummy_of_match[] = {
        { .compatible = "vendor,dummy", },
        { /* end of list */ },
};
MODULE_DEVICE_TABLE(of, dummy_of_match);
#else
# define dummy_of_match
#endif


static struct platform_driver dummy_driver = {
        .driver = {
                .name = DRIVER_NAME,
                .owner = THIS_MODULE,
                .of_match_table = dummy_of_match,
        },
        .probe          = dummy_probe,
        .remove         = dummy_remove,
};

static int __init dummy_init(void)
{
        printk("<1>Hello module world.\n");
        printk("<1>Module parameters were (0x%08x) and \"%s\"\n", myint,
               mystr);

        return platform_driver_register(&dummy_driver);
}


static void __exit dummy_exit(void)
{
        platform_driver_unregister(&dummy_driver);
        printk(KERN_ALERT "Goodbye module world.\n");
}

module_init(dummy_init);
module_exit(dummy_exit);

My corresponding system-user.dtsi file:

Code Select Expand

/include/ "system-conf.dtsi"
/{
        reserved-memory
        {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;

                reserved: buffer@0x3ed80000 {
                        no-map;
                        reg = <0x0 0x3ed80000 0x0 0x1000000>;
                };
        };

        reserved-driver@0 {
                compatible = "vendor,dummy";
                memory-region = <&reserved>;
    };
};



[1] https://support.xilinx.com/s/question/0D52E00006ihQaJSAU/how-to-disable-caching-in-petalinux-kernel-module-shm-written-on-apu-not-visible-on-rpu?language=en_US
[2]
https://shop.trenz-electronic.de/de/Download/?path=Trenz_Electronic/Modules_and_Module_Carriers/5.2x7.6/TE0807/Reference_Design/2020.2/StarterKit

Thanks and BR
Daniel

[JH]

Hi,

when you create a new petalinux project you didn't has all the changes we have there, maybe you problem is not related to your module, it's rather related to some other wrong configuration?
https://wiki.trenz-electronic.de/display/PD/TE0807+StarterKit#TE0807StarterKit-SoftwareDesign-Vitis
or the wrong fsbl, because our fsbl changes are not included on petalinux
https://wiki.trenz-electronic.de/display/PD/TE0807+StarterKit#TE0807StarterKit-Application

please start with our reference design:

--> use the  prebuilt xsa file(like you has done) and generate linux with our template (it's included in the download, otherwise you must setup everything manually, like on the documentation)
--> Note: we use Vitis to generate the FSBL and boot.bin (template is also included -->but you can also use our scripts to regenerate your boot.bin with your uboot, when you change later something).

I would suggest to generate the design one time without any changes and check if it works (maybe something is wrong with your petalinux built environment).
If this works, you the same project and add your kernel module. Clean up the project and generate files again.



br
John

[DanielH]

Thanks for the answer John,

I don't fully understand how you mean I should use additional templates besides the XSA file. What I now tried was:

• Unpacked TE0807-StarterKit-vivado_2020.2-build_1_20210208093620 zip file
• Copied folder StarterKit/os/petalinux onto my build environment
• Copied .xsa file from StarterKit/prebuilt/hardware/7ev_1e_4gb to build environment in subfolder petalinux/project-spec/hw-description
• Added dummy kernel module via petalinux-create -t modules --name dummy --enable
• Exchanged the dummy.c kernel module source file with my own
• Added my memory-block devicetree contents to the system-user.dtsi which came with your zip file (see below)
• petalinux-build
• (in images/linux subfolder) petalinux-package --boot --fsbl ./zynqmp_fsbl.elf --fpga ./system.bit --u-boot --atf ./bl31.elf --kernel ./image.ub
• Copied generated files onto SD card (image.ub, BOOT.BIN, boot.scr to boot partition, rootfs.cpio and contents of rootfs.tar.gz to sytem partition)
• Startup the Trenz system with this SD card

I still receive the same error.
If you meant that I need to use additional contents from the TE0807-StarterKit-vivado_2020.2-build_1_20210208093620 zip file, i would appreciate if you could point me to them more specifically.

My device tree contents I added:

Code Select Expand

reserved-memory
        {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;

                reserved: buffer@0x3ed80000 {
                        no-map;
                        reg = <0x0 0x3ed80000 0x0 0x1000000>;
                };
        };

        reserved-driver@0 {
                compatible = "vendor,dummy";
                memory-region = <&reserved>;
    };


Thanks & BR
Daniel

[JH]

Hi,

Copied .xsa file from StarterKit/prebuilt/hardware/7ev_1e_4gb to build environment in subfolder petalinux/project-spec/hw-description

that's wrong, correct is
Copied .xsa file from StarterKit/prebuilt/hardware/7ev_1e_4gb to build environment in subfolder petalinux/
run  "petalinux-config --get-hw-description"      --> this will setup xsa correctly
run "petalinux-build"
if this works (I expect that this works), than you can add you changes.
PS: You must use our fsbl from vitis with our changes or SI5345 initialisation.  Petalinux generated fsbl didn't included this changes(only if you generate a patch).
br
John

[DanielH]

Thanks for the swift answer and for poining out my error. I did it correctly in the past, but with that test I simply forgot this and did it wrong.

When you say I have to use your fsbl, do you mean that instead of building it in Vitis I can also simply use the prebuilt one from the StarterKit zip package:

Code Select Expand

TE0807-StarterKit-vivado_2020.2-build_1_20210208093620.zip\StarterKit\prebuilt\software\7ev_1e_4gb\fsbl.elf
and refer to that one when issuing the petalinux-package command? Like this:

Code Select Expand

petalinux-package --boot --fsbl ./<your fsbl.elf from the zip> --fpga ./system.bit --u-boot --atf ./bl31.elf --kernel ./image.ub
That's what I did now and still, I get my initial error on memset in the kernel module.

BTW: That's the beginning of my serial console output I receive when booting up:

Code Select Expand

--------------------------------------------------------------------------------
TE0807 TE_XFsbl_HookPsuInit_Custom
Configure Carrier I2C Switch 0x77
Configure PLL: SI5345-B
Si534x Init Registers Write.
Si534x Init Complete.
PLL Status Register   0xC:0x0, 0xE:0x0, 0xD:0x0, 0x11:0, 0xF:0.or calibration...256 us).
USB Reset Complete
PCIe Reset Complete

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Xilinx Zynq MP First Stage Boot Loader (TE modified)
Release 2020.2   Feb  8 2021  -  06:53:43
Device Name: XCZU7EV

--------------------------------------------------------------------------------
TE0807 TE_XFsbl_BoardInit_Custom
Configure Carrier I2C Switch 0x73 for EEPROM access

--------------------------------------------------------------------------------
NOTICE:  ATF running on XCZU7EV/silicon v4/RTL5.1 at 0xfffea000
NOTICE:  BL31: v2.2(release):xlnx_rebase_v2.2_2020.3
NOTICE:  BL31: Built : 05:15:13, Oct  8 2021


U-Boot 2020.01 (Oct 08 2021 - 05:19:06 +0000)

Board: Xilinx ZynqMP
DRAM:  4 GiB
PMUFW:  v1.1
EL Level:       EL2
Chip ID:        zu7ev
NAND:  0 MiB
MMC:   mmc@ff160000: 0, mmc@ff170000: 1
In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: SD_MODE1
Reset reason:   EXTERNAL
Net:
ZYNQ GEM: ff0e0000, mdio bus ff0e0000, phyaddr 1, interface rgmii-id

BR
Daniel

[JH]

Hi,
you can use our prebuilt fsbl as long as your PS IP setup, PS-PL CLKs/Interfaces and PL AXI Register mapping is the same. if you change something, than you must also regnerate the fsbl again (and depending on changes also different stuff on linux).

Regarding reserved memory. My colleagues have noticed that with 2020.2 there is a warning that memory cannot be reserved, but then it worked (with 19.2 it still worked):
https://wiki.trenz-electronic.de/display/PD/TE0727+Zynqberry+Demo1#TE0727ZynqberryDemo1-ReleaseNotesandKnowIssues
We didn't find a workaround until now, on our design it was still working.
Maybe you has a similar problem here?
That's our device tree entry on the 7 series zynq:
https://wiki.trenz-electronic.de/display/PD/TE0727+Zynqberry+Demo1#TE0727ZynqberryDemo1-DeviceTree

Maybe you should write one time on Xilinx forum, it's a zynq specific problem and the community is bigger there.

br
John