headers: Refactor kernel and arch headers.

[stephanosio]

This commit refactors kernel and arch headers to establish a boundary
between private and public interface headers.

The refactoring strategy used in this commit is detailed in the issue

This commit introduces the following major changes:

1. Establish a clear boundary between private and public headers by
  removing "kernel/include" and "arch/*/include" from the global
  include paths. Ideally, only kernel/ and arch/*/ source files should
  reference the headers in these directories. If these headers must be
  used by a component, these include paths shall be manually added to
  the CMakeLists.txt file of the component. This is intended to
  discourage applications from including private kernel and arch
  headers either knowingly and unknowingly.

  - kernel/include/ (PRIVATE)
    This directory contains the private headers that provide private
   kernel definitions which should not be visible outside the kernel
   and arch source code. All public kernel definitions must be added
   to an appropriate header located under include/.

  - arch/*/include/ (PRIVATE)
    This directory contains the private headers that provide private
   architecture-specific definitions which should not be visible
   outside the arch and kernel source code. All public architecture-
   specific definitions must be added to an appropriate header located
   under include/arch/*/.

  - include/ AND include/sys/ (PUBLIC)
    This directory contains the public headers that provide public
   kernel definitions which can be referenced by both kernel and
   application code.

  - include/arch/*/ (PUBLIC)
    This directory contains the public headers that provide public
   architecture-specific definitions which can be referenced by both
   kernel and application code.

2. Split arch_interface.h into "kernel-to-arch interface" and "public
  arch interface" divisions.

  - kernel/include/kernel_arch_interface.h
    * provides private "kernel-to-arch interface" definition.
    * includes arch/*/include/kernel_arch_func.h to ensure that the
     interface function implementations are always available.
    * includes sys/arch_interface.h so that public arch interface
     definitions are automatically included when including this file.

  - arch/*/include/kernel_arch_func.h
    * provides architecture-specific "kernel-to-arch interface"
     implementation.
    * only the functions that will be used in kernel and arch source
     files are defined here.

  - include/sys/arch_interface.h
    * provides "public arch interface" definition.
    * includes include/arch/arch_inlines.h to ensure that the
     architecture-specific public inline interface function
     implementations are always available.

  - include/arch/arch_inlines.h
    * includes architecture-specific arch_inlines.h in
     include/arch/*/arch_inline.h.

  - include/arch/*/arch_inline.h
    * provides architecture-specific "public arch interface" inline
     function implementation.
    * supersedes include/sys/arch_inline.h.

3. Refactor kernel and the existing architecture implementations.

  - Remove circular dependency of kernel and arch headers. The
   following general rules should be observed:

    * Never include any private headers from public headers
    * Never include kernel_internal.h in kernel_arch_data.h
    * Always include kernel_arch_data.h from kernel_arch_func.h
    * Never include kernel.h from kernel_struct.h either directly or
     indirectly. Only add the kernel structures that must be referenced
     from public arch headers in this file.

  - Relocate syscall_handler.h to include/ so it can be used in the
   public code. This is necessary because many user-mode public codes
   reference the functions defined in this header.

  - Relocate kernel_arch_thread.h to include/arch/*/thread.h. This is
   necessary to provide architecture-specific thread definition for
   'struct k_thread' in kernel.h.

  - Remove any private header dependencies from public headers using
   the following methods:

    * If dependency is not required, simply omit
    * If dependency is required,
      - Relocate a portion of the required dependencies from the
       private header to an appropriate public header OR
      - Relocate the required private header to make it public.

This commit supersedes #20047, addresses #19666, and fixes #3056.

Signed-off-by: Stephanos Ioannidis <[email protected]>

[zephyrbot]

All checks are passing now.

checkpatch (informational only, not a failure)

-:2355: WARNING:NEW_TYPEDEFS: do not add new typedefs
#2355: FILE: include/arch/x86/intel64/thread.h:79:
+typedef struct x86_tss64 x86_tss64_t;

-:2376: WARNING:NEW_TYPEDEFS: do not add new typedefs
#2376: FILE: include/arch/x86/intel64/thread.h:100:
+typedef struct _callee_saved _callee_saved_t;

-:2390: WARNING:LINE_SPACING: Missing a blank line after declarations
#2390: FILE: include/arch/x86/intel64/thread.h:114:
+	u64_t r11;
+	char __aligned(X86_FXSAVE_ALIGN) sse[X86_FXSAVE_SIZE];

-:2393: WARNING:NEW_TYPEDEFS: do not add new typedefs
#2393: FILE: include/arch/x86/intel64/thread.h:117:
+typedef struct _thread_arch _thread_arch_t;

-:3207: WARNING:NEW_TYPEDEFS: do not add new typedefs
#3207: FILE: include/sys/arch_interface.h:43:
+typedef struct _k_thread_stack_element k_thread_stack_t;

-:5385: WARNING:NEW_TYPEDEFS: do not add new typedefs
#5385: FILE: subsys/testsuite/ztest/include/arch/cpu.h:13:
+typedef struct _callee_saved _callee_saved_t;

-:5390: WARNING:NEW_TYPEDEFS: do not add new typedefs
#5390: FILE: subsys/testsuite/ztest/include/arch/cpu.h:18:
+typedef struct _thread_arch _thread_arch_t;

- total: 0 errors, 7 warnings, 3850 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

Your patch has style problems, please review.

NOTE: Ignored message types: AVOID_EXTERNS BRACES CONFIG_EXPERIMENTAL CONST_STRUCT DATE_TIME FILE_PATH_CHANGES MINMAX NETWORKING_BLOCK_COMMENT_STYLE PRINTK_WITHOUT_KERN_LEVEL SPLIT_STRING VOLATILE

NOTE: If any of the errors are false positives, please report
      them to the maintainers.

Tip: The bot edits this comment instead of posting a new one, so you can check the comment's history to see earlier messages.

[stephanosio]

This PR supersedes the previous PR #20047, which was an attempt at fixing #3056.

The changes in this PR fix #3056, while addressing all the essential changes detailed in the issue #19666 as well.

Once this gets merged, the following hacks in arch/arm and arch/x86 should be fixed:

zephyr/include/arch/x86/ia32/arch.h

Line 238 in cc1b94c

/* FIXME prefer these inline, but see GH-3056 */

zephyr/arch/x86/core/ia32/irq_manage.c

Lines 41 to 48 in 8ffff14

	/* FIXME: IRQ direct inline functions have to be placed here and not in
	* arch/cpu.h as inline functions due to nasty circular dependency between
	* arch/cpu.h and kernel_structs.h; the inline functions typically need to
	* perform operations on _kernel. For now, leave as regular functions, a
	* future iteration will resolve this.
	*
	* See https://github.com/zephyrproject-rtos/zephyr/issues/3056
	*/

zephyr/include/arch/arm/irq.h

Line 93 in 8ffff14

/* FIXME prefer these inline, but see GH-3056 */

zephyr/arch/arm/core/irq_manage.c

Lines 160 to 167 in 8ffff14

	/* FIXME: IRQ direct inline functions have to be placed here and not in
	* arch/cpu.h as inline functions due to nasty circular dependency between
	* arch/cpu.h and kernel_structs.h; the inline functions typically need to
	* perform operations on _kernel. For now, leave as regular functions, a
	* future iteration will resolve this.
	*
	* See https://github.com/zephyrproject-rtos/zephyr/issues/3056
	*/

[aescolar]

@aescolar What do you think about adding a dummy dts

@stephanosio I presume you are seeing something broken in master, and I want to look into it asap. If that is the case, let's discuss it outside of this PR to not create confusion. If something is broken in master it should be fixed on its own, not in an unrelated PR.

[stephanosio]

Force pushing to address the change requested by @SebastianBoe and update hal_nxp revision from PR ref to merged commit hash in west.yml.

[ioannisg]

Last merge conflicts are due to the 64-bit updates on the syscalls, am I correct?

[nashif]

Last merge conflicts are due to the 64-bit updates on the syscalls, am I correct?

grrrrrrr

[nashif]

@stephanosio last rebase maybe? :) thanks.

[andrewboie]

@stephanosio last rebase maybe? :) thanks.

@stephanosio sorry that was my fault

[stephanosio]

Rebased onto the latest master.

[wentongwu]

@stephanosio sorry, just see the PR, why we re-write subsys/debug/tracing/CMakeLists.txt?

[stephanosio]

@stephanosio sorry, just see the PR, why we re-write subsys/debug/tracing/CMakeLists.txt?

Somehow missed this comment.

Because zephyr_library_include_directories must be used to prevent global inclusion of the private header directories and this requires the use of zephyr_library_* functions.