Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort a few python dicts for deterministic builds, even with python 3.5 and earlier. #14119

Merged
merged 2 commits into from
Mar 7, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion scripts/dts/extract_dts_includes.py
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,9 @@ def yaml_inc_error(msg):
def generate_defines():
# Generates #defines (and .conf file values) from DTS

for node_path in reduced.keys():
# sorted() otherwise Python < 3.6 randomizes the order of the flash
# partition table
for node_path in sorted(reduced.keys()):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I missed this earlier, but could get rid of the .keys(). x in dict, for x in dict, sorted(dict), etc., always uses the keys.

Copy link
Collaborator Author

@marc-hb marc-hb Mar 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed but I'd like to keep this particular commit as minimal as possible :-) Not just for aesthetic reasons but also cherry-picking for testing / backporting etc.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, fine with me.

generate_node_defines(node_path)

if not defs:
Expand Down
11 changes: 10 additions & 1 deletion scripts/elf_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
import struct
from distutils.version import LooseVersion

from collections import OrderedDict

import elftools
from elftools.elf.elffile import ELFFile
from elftools.elf.sections import SymbolTableSection
Expand Down Expand Up @@ -518,7 +520,14 @@ def find_kobjects(self, syms):
ret[addr] = ko

self.debug("found %d kernel object instances total" % len(ret))
return ret

# 1. Before python 3.7 dict order is not guaranteed. With Python
# 3.5 it doesn't seem random with *integer* keys but can't
# rely on that.
# 2. OrderedDict means _insertion_ order, so not enough because
# built from other (random!) dicts: need to _sort_ first.
# 3. Sorting memory address looks good.
return OrderedDict(sorted(ret.items()))

def get_symbols(self):
for section in self.elf.iter_sections():
Expand Down
41 changes: 26 additions & 15 deletions scripts/gen_kobject_list.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,26 +11,33 @@
import struct
from elf_helper import ElfHelper, kobject_to_enum

from collections import OrderedDict

# Keys in this dictionary are structs which should be recognized as kernel
# objects. Values should either be None, or the name of a Kconfig that
# indicates the presence of this object's definition in case it is not
# available in all configurations.

kobjects = {
"k_mem_slab": None,
"k_msgq": None,
"k_mutex": None,
"k_pipe": None,
"k_queue": None,
"k_poll_signal": None,
"k_sem": None,
"k_stack": None,
"k_thread": None,
"k_timer": None,
"_k_thread_stack_element": None,
"net_context": "CONFIG_NETWORKING",
"device": None
}
# Regular dictionaries are ordered only with Python 3.6 and
# above. Good summary and pointers to official documents at:
# https://stackoverflow.com/questions/39980323/are-dictionaries-ordered-in-python-3-6
kobjects = OrderedDict ([
("k_mem_slab", None),
("k_msgq", None),
("k_mutex", None),
("k_pipe", None),
("k_queue", None),
("k_poll_signal", None),
("k_sem", None),
("k_stack", None),
("k_thread", None),
("k_timer", None),
("_k_thread_stack_element", None),
("net_context", "CONFIG_NETWORKING"),
("device", None),
])



subsystems = [
"adc_driver_api",
Expand Down Expand Up @@ -269,10 +276,14 @@ def main():
parse_args()

if args.gperf_output:
assert args.kernel, "--kernel ELF required for --gperf-output"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe

if not args.kernel:
    sys.exit("--kernel must be passed if --gperf-output is")

would be a bit more purist.

It's more of a usage error than a runtime error.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a usage error but... a pretty stupid, "trial-and-error" one so I didn't want to spend too many lines of code on it.

eh = ElfHelper(args.kernel, args.verbose, kobjects, subsystems)
syms = eh.get_symbols()
max_threads = syms["CONFIG_MAX_THREAD_BYTES"] * 8
objs = eh.find_kobjects(syms)
if not objs:
sys.stderr.write("WARNING: zero kobject found in %s\n"
% args.kernel)

thread_counter = eh.get_thread_counter()
if thread_counter > max_threads:
Expand Down
3 changes: 3 additions & 0 deletions scripts/gen_priv_stacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ def main():
syms = eh.get_symbols()
max_threads = syms["CONFIG_MAX_THREAD_BYTES"] * 8
objs = eh.find_kobjects(syms)
if not objs:
sys.stderr.write("WARNING: zero kobject found in %s\n"
% args.kernel)

thread_counter = eh.get_thread_counter()
if thread_counter > max_threads:
Expand Down