"Verbose" mode for memap.py

[javier-moreno-tridonic-com]

Note: This is just a template, so feel free to use/remove the unnecessary things

Description

Type: Feature request
Priority: Minor

Feature Request

To analyse memory usage and identify possible optimisation points, I've been modifying memap.py in the following manner:

original code

    @staticmethod
    def path_object_to_module_name(txt):
        """ Parse a path to object file to extract it's module and object data
        Positional arguments:
        txt - the path to parse the object and module name from
        """

        txt = txt.replace('\\', '/')
        rex_mbed_os_name = r'^.+mbed-os\/(.+)\/(.+\.o)$'
        test_rex_mbed_os_name = re.match(rex_mbed_os_name, txt)

        if test_rex_mbed_os_name:

            object_name = test_rex_mbed_os_name.group(2)
            data = test_rex_mbed_os_name.group(1).split('/')
            ndata = len(data)

            if ndata == 1:
                module_name = data[0]
            else:
                module_name = data[0] + '/' + data[1]

            return [module_name, object_name]
        else:
            return ['Misc', ""]

My code

    @staticmethod
    def path_object_to_module_name(txt):
        """ Parse a path to object file to extract it's module and object data

        Positional arguments:
        txt - the path to parse the object and module name from
        """

        txt = txt.replace('\\', '/')

        # print('txt: ' + txt)

        rex_mbed_os_name = r'^.+mbed-os\/(.+)\/(.+\.o)$'
        test_rex_mbed_os_name = re.match(rex_mbed_os_name, txt)

        if test_rex_mbed_os_name:
            object_name = test_rex_mbed_os_name.group(2)
            data = test_rex_mbed_os_name.group(1).split('/')
            ndata = len(data)

            if ndata == 1:
                module_name = data[0]
            else:
                module_name = data[0] + '/' + data[1]

            return [module_name, object_name]
        else:
            rex_obj_name = r'^.+\/(.+\.o\)*)$'
            test_rex_obj_name = re.match(rex_obj_name, txt)
            if test_rex_obj_name:
                object_name = test_rex_obj_name.group(1)
                return [object_name, ""]       

            return ['Misc', ""]

With this I get and output similar to this:

Module	.text	.data	.bss
BufferedSerial.o	12	0	0
Fill	713	34	67
at24mac.o	134	0	20
crt0.o	116	0	0
crtbegin.o	92	12	28
crtend.o	4	4	0
crti.o	8	0	0
crtn.o	16	0	0
driverAtmelRFInterface.o	2541	1	8
driverRFPhy.o	2334	15	196
features/FEATURE_CLIENT	39077	11	53
features/FEATURE_COMMON_PAL	19454	93	8412
features/frameworks	3827	92	848
features/mbedtls	75976	51	8755
features/net	140911	302	13841
hal/common	5689	20	421
hal/targets	10767	4	1472
libc.a(lib_a-__atexit.o)	164	0	0
libc.a(lib_a-__call_atexit.o)	232	4	0
libc.a(lib_a-abort.o)	16	0	0
libc.a(lib_a-abs.o)	8	0	0
libc.a(lib_a-assert.o)	123	0	0
libc.a(lib_a-atexit.o)	12	0	0
libc.a(lib_a-atof.o)	8	0	0
libc.a(lib_a-atoi.o)	8	0	0
libc.a(lib_a-atol.o)	8	0	0
libc.a(lib_a-calloc.o)	16	0	0
libc.a(lib_a-callocr.o)	96	0	0
libc.a(lib_a-closer.o)	36	0	0
libc.a(lib_a-ctype_.o)	257	4	0
libc.a(lib_a-cxa_atexit.o)	20	0	0
libc.a(lib_a-div.o)	64	0	0
libc.a(lib_a-dtoa.o)	4044	0	0
libc.a(lib_a-environ.o)	0	4	4
libc.a(lib_a-errno.o)	12	0	0
libc.a(lib_a-exit.o)	32	0	0
libc.a(lib_a-fclose.o)	148	0	0
libc.a(lib_a-fflush.o)	420	0	0
libc.a(lib_a-findfp.o)	456	0	0
libc.a(lib_a-fini.o)	44	0	0
libc.a(lib_a-fiprintf.o)	40	0	0
libc.a(lib_a-flags.o)	116	0	0
libc.a(lib_a-fopen.o)	152	0	0
libc.a(lib_a-fputwc.o)	212	0	0
libc.a(lib_a-freer.o)	588	0	0
libc.a(lib_a-fseek.o)	4	0	0
libc.a(lib_a-fseeko.o)	696	0	0
libc.a(lib_a-fstatr.o)	40	0	0
libc.a(lib_a-fvwrite.o)	792	0	0
libc.a(lib_a-fwalk.o)	160	0	0
libc.a(lib_a-gdtoa-gethex.o)	1716	0	0
libc.a(lib_a-gdtoa-hexnan.o)	388	0	0
libc.a(lib_a-getc.o)	52	0	0
libc.a(lib_a-getchar.o)	16	0	0
libc.a(lib_a-getenv_r.o)	152	0	0
libc.a(lib_a-gettzinfo.o)	8	64	0
libc.a(lib_a-gmtime_r.o)	364	0	0
libc.a(lib_a-impure.o)	6	1068	0
libc.a(lib_a-init.o)	80	0	0
libc.a(lib_a-isalpha.o)	20	0	0
libc.a(lib_a-isattyr.o)	36	0	0
libc.a(lib_a-isdigit.o)	20	0	0
libc.a(lib_a-isspace.o)	20	0	0
libc.a(lib_a-iswspace.o)	28	0	0
libc.a(lib_a-lcltime.o)	16	0	0
libc.a(lib_a-lcltime_r.o)	600	0	0
libc.a(lib_a-locale.o)	38	92	0
libc.a(lib_a-lseekr.o)	40	0	0
libc.a(lib_a-makebuf.o)	224	0	0
libc.a(lib_a-malloc.o)	32	0	0
libc.a(lib_a-mallocr.o)	1316	1040	52
libc.a(lib_a-mbrtowc.o)	93	0	0
libc.a(lib_a-mbtowc_r.o)	44	4	0
libc.a(lib_a-memchr.o)	148	0	0
libc.a(lib_a-memcmp.o)	100	0	0
libc.a(lib_a-memcpy.o)	308	0	0
libc.a(lib_a-memmove.o)	200	0	0
libc.a(lib_a-memset.o)	156	0	0
libc.a(lib_a-mktime.o)	1512	0	0
libc.a(lib_a-month_lengths.o)	96	0	0
libc.a(lib_a-mprec.o)	2432	0	0
libc.a(lib_a-openr.o)	40	0	0
libc.a(lib_a-printf.o)	40	0	0
libc.a(lib_a-rand.o)	84	0	0
libc.a(lib_a-readr.o)	40	0	0
libc.a(lib_a-realloc.o)	16	0	0
libc.a(lib_a-reallocr.o)	1004	0	0
libc.a(lib_a-reent.o)	0	0	4
libc.a(lib_a-refill.o)	292	0	0
libc.a(lib_a-rget.o)	52	0	0
libc.a(lib_a-sbrkr.o)	36	0	0
libc.a(lib_a-sccl.o)	120	0	0
libc.a(lib_a-setbuf.o)	16	0	0
libc.a(lib_a-setvbuf.o)	248	0	0
libc.a(lib_a-sf_nan.o)	8	0	0
libc.a(lib_a-signal.o)	104	0	0
libc.a(lib_a-signalr.o)	44	0	0
libc.a(lib_a-siscanf.o)	88	0	0
libc.a(lib_a-snprintf.o)	156	0	0
libc.a(lib_a-sprintf.o)	76	0	0
libc.a(lib_a-sscanf.o)	88	0	0
libc.a(lib_a-stdio.o)	136	0	0
libc.a(lib_a-strchr.o)	204	0	0
libc.a(lib_a-strcmp.o)	844	0	0
libc.a(lib_a-strcpy.o)	188	0	0
libc.a(lib_a-strerror.o)	2781	0	0
libc.a(lib_a-strlen.o)	92	0	0
libc.a(lib_a-strncmp.o)	152	0	0
libc.a(lib_a-strrchr.o)	36	0	0
libc.a(lib_a-strstr.o)	1232	0	0
libc.a(lib_a-strtod.o)	4095	0	0
libc.a(lib_a-strtol.o)	308	0	0
libc.a(lib_a-strtoll_r.o)	460	0	0
libc.a(lib_a-strtoul.o)	320	0	0
libc.a(lib_a-strtoull_r.o)	472	0	0
libc.a(lib_a-svfiprintf.o)	260	0	0
libc.a(lib_a-svfiscanf.o)	2846	0	0
libc.a(lib_a-svfprintf.o)	5406	0	0
libc.a(lib_a-svfscanf.o)	3699	0	0
libc.a(lib_a-tzcalc_limits.o)	396	0	0
libc.a(lib_a-tzlock.o)	8	0	0
libc.a(lib_a-tzset.o)	12	0	0
libc.a(lib_a-tzset_r.o)	887	0	26
libc.a(lib_a-tzvars.o)	4	8	8
libc.a(lib_a-u_strerr.o)	4	0	0
libc.a(lib_a-ungetc.o)	120	0	0
libc.a(lib_a-vfiprintf.o)	3431	0	0
libc.a(lib_a-vfprintf.o)	5402	0	0
libc.a(lib_a-vsnprintf.o)	148	0	0
libc.a(lib_a-vsprintf.o)	76	0	0
libc.a(lib_a-wbuf.o)	168	0	0
libc.a(lib_a-wcrtomb.o)	84	0	0
libc.a(lib_a-wctomb_r.o)	28	4	0
libc.a(lib_a-writer.o)	40	0	0
libc.a(lib_a-wsetup.o)	196	0	0
libgcc.a(_aeabi_ldivmod.o)	56	0	0
libgcc.a(_aeabi_uldivmod.o)	48	0	0
libgcc.a(_arm_addsubdf3.o)	880	0	0
libgcc.a(_arm_cmpdf2.o)	272	0	0
libgcc.a(_arm_fixdfsi.o)	80	0	0
libgcc.a(_arm_fixunsdfsi.o)	64	0	0
libgcc.a(_arm_muldivdf3.o)	1060	0	0
libgcc.a(_arm_truncdfsf2.o)	160	0	0
libgcc.a(_ashldi3.o)	24	0	0
libgcc.a(_ashrdi3.o)	28	0	0
libgcc.a(_divdi3.o)	668	0	0
libgcc.a(_dvmd_tls.o)	4	0	0
libgcc.a(_fixunssfdi.o)	76	0	0
libgcc.a(_lshrdi3.o)	24	0	0
libgcc.a(_udivdi3.o)	620	0	0
libgcc.a(bpabi.o)	96	0	0
libgcc.a(libunwind.o)	428	0	0
libgcc.a(pr-support.o)	892	0	0
libgcc.a(unwind-arm.o)	2640	0	0
libm.a(lib_a-s_fpclassify.o)	88	0	0
libnosys.a(getpid.o)	16	0	0
libnosys.a(kill.o)	16	0	0
libstdc++.a(atexit_arm.o)	12	0	0
libstdc++.a(bad_alloc.o)	116	0	0
libstdc++.a(class_type_info.o)	338	0	0
libstdc++.a(del_op.o)	4	0	0
libstdc++.a(del_opv.o)	4	0	0
libstdc++.a(eh_alloc.o)	136	0	2052
libstdc++.a(eh_arm.o)	400	0	0
libstdc++.a(eh_call.o)	304	0	0
libstdc++.a(eh_catch.o)	280	0	0
libstdc++.a(eh_exception.o)	226	0	0
libstdc++.a(eh_globals.o)	16	0	12
libstdc++.a(eh_personality.o)	1508	0	0
libstdc++.a(eh_term_handler.o)	0	4	0
libstdc++.a(eh_terminate.o)	120	0	0
libstdc++.a(eh_throw.o)	308	0	0
libstdc++.a(eh_unex_handler.o)	0	4	0
libstdc++.a(functexcept.o)	376	0	0
libstdc++.a(guard.o)	64	0	0
libstdc++.a(guard_error.o)	115	0	0
libstdc++.a(locale-inst.o)	356	4	48
libstdc++.a(locale_facets.o)	136	4	0
libstdc++.a(new_handler.o)	16	0	4
libstdc++.a(new_op.o)	68	0	0
libstdc++.a(new_opv.o)	20	0	0
libstdc++.a(si_class_type_info.o)	381	0	0
libstdc++.a(snprintf_lite.o)	396	0	0
libstdc++.a(stdexcept.o)	402	0	0
libstdc++.a(string-inst.o)	1337	0	16
libstdc++.a(system_error.o)	473	12	0
libstdc++.a(tinfo.o)	77	0	0
libstdc++.a(wlocale-inst.o)	196	4	48
main.o	4285	21	410
rtos/rtos	575	4	0
rtos/rtx	7037	20	2683
Subtotals	386365	3008	39488

Allocated Heap: unknown
Allocated Stack: unknown
Total Static RAM memory (data + bss): 42496 bytes
Total RAM memory (data + bss + heap + stack): 42496 bytes
Total Flash memory (text + data + misc): 389373 bytes

I think it would be nice to have such print as optional for the mbed-cli (maybe even for mbed-os features so we could help you identify and fix issues).

[MarceloSalazar]

Hi Javier,

This feature looks very useful :)
A few suggestions:

• Add a verbose option to generate more information when invoking memap from the command line (by default should be disabled):

  mbed-os/tools/memap.py

  Line 597 in 5c82db8

  def main():

• Display Misc to make the output more consistent with the default behavior:
  libc.a(lib_a-__atexit.o) --> Misc\libc.a(lib_a-__atexit.o)

It would be great if you could send us a PR and we'll be happy to review it.

Thanks,
Marcelo

[javier-moreno-tridonic-com]

Well I'd be happy to prepare it, just I would need some pointers for this:

• Should be this a memap only feature? I mean, should this feature be also accessible by using a "mbed compile" flag? If so, any flag name suggestion that can be shared across both scripts?

[MarceloSalazar]

I'd suggest this to be a memap only feature for now (e.g. invoke the tool using python memap ...).
This will let us prepare the interfaces and do further testing. We will consider calling it from the mbed compile script.

[nuket]

This is great, I'm also looking for a way to have a deeper view into flash / stack / heap memory allocations.

[sg-]

@meriac Anything you want to share here?

[meriac]

@nuket : I had good results parsing arm-non-eabi-nm output - feel free to have a look on an early prototype for d3.js based linker visualization.

You can find the command line settings I chose here. Please note that I had to enable debug information to get line number resolution in the mbed release build.
You can find a quick patch for GCC in my tool repo. The official pull request is #2139 for adding debug symbol support to all mbed supported release builds.

[javier-moreno-tridonic-com]

Since this functionality was merged already (see #2845) this can be closed