Bluetooth: Cannot establish security and discover GATT when using Split LL

[MarekPieta]

Describe the bug
When connecting nrf_desktop Bluetooth Peripheral and Central I noticed the following error log (on Central):
[07095681] <err> bt_keys: Failed to save keys (err -5)
After the log Central was unable to establish security and complete GATT discovery.

To Reproduce

1. Use nrf_desktop (application from NordicPlayground/fw-nrfconnect-nrf repository - the application uses nrf Zephyr fork).
2. Build and flash firmware for Central (pca10059) and Peripheral (pca10056) with Split LL selected.
3. Turn on the devices.
4. Central will automatically try to connect to Peripheral.
5. Observe the problems.

Expected behavior
Split LL should work without issues (like other LL).

Impact
Split LL cannot be used for Bluetooth Central (Peripherals using split LL work fine with other hosts - tested with Windows10)

Screenshots or console output
Console logs (Central):

[06327395] <inf> ble_scan: Scan started
[06327398] <inf> event_manager: e: led_event led_id:1 effect:0x3b5e4
[06327400] <inf> event_manager: e: ble_peer_search_event active
[06327404] <inf> event_manager: e: led_event led_id:1 effect:0x3b5f4
[06327407] <inf> event_manager: e: led_ready_event led_id:1 effect:0x3b5e4
[06346154] <inf> ble_scan: Filters matched. fd:cc:ef:f5:a3:68 (random) connectable
[06346163] <inf> ble_scan: Scan stopped
[06346173] <inf> event_manager: e: ble_peer_search_event inactive
[06346176] <inf> event_manager: e: led_event led_id:1 effect:0x3b5e4
[06346184] <inf> ble_scan: Connecting done
[06346189] <inf> event_manager: e: led_ready_event led_id:1 effect:0x3b5e4
[06348553] <inf> ble_state: Connected to fd:cc:ef:f5:a3:68 (random)
[06348565] <inf> event_manager: e: ble_peer_event id=0x200029bc CONNECTED
[06348569] <inf> event_manager: e: led_event led_id:1 effect:0x3b5ec
[06348574] <inf> event_manager: e: led_ready_event led_id:1 effect:0x3b5ec
[06351803] <wrn> bt_ctlr_hci: type: 0x14
[06360014] <inf> ble_state: Security with fd:cc:ef:f5:a3:68 (random) level 2
[06360021] <inf> event_manager: e: ble_peer_event id=0x200029bc SECURED
[06360026] <inf> event_manager: e: led_event led_id:1 effect:0x3b5ec
[06360030] <inf> event_manager: e: led_ready_event led_id:1 effect:0x3b5ec
[06360461] <inf> ble_state: MTU exchange done
[07095681] <err> bt_keys: Failed to save keys (err -5)
[07255841] <inf> ble_scan: Discovery in progress

Environment (please complete the following information):

• Zephyr commit hash: 40175fd

Additional context

• enabling ticker compatibility mode seems to solve the issue

[jhedberg]

Looks like a controller vs flash access coexistence issue? -5 is -EIO and it's being returned from the settings subsystem.

[jhedberg]

@carlescufi FYI