Skip to content

Commit

Permalink
Update Windows installer to create a user-defined network for modules (
Browse files Browse the repository at this point in the history
…#627)

Currently on Windows, we create all modules (edgeAgent, edgeHub, and any others) on the default 'nat' network, due to Docker for Windows constraints IIRC.

On Linux, we provide useful isolation by creating edgeAgent on the default 'bridge' network so it can talk to `iotedged`, but then all other modules are on their own, user-defined network, named 'azure-iot-edge'. They can freely communicate with one another without the need to expose ports to the host.

We want our network topology on Windows to align with what we do on Linux. These changes accomplish that.
  • Loading branch information
damonbarry authored Dec 10, 2018
1 parent 8763c8e commit 6d5b95a
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 39 deletions.
44 changes: 20 additions & 24 deletions doc/networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,53 +4,49 @@ IoT Edge uses the networking capabilities of the Moby runtime to connect to IoT
In a basic IoT Edge setup, the default configuration should be sufficient.
However, if there are additional requirements for firewalls or network topology, it is helpful to understand IoT Edge's network setup and dependencies.

On Windows, all containers are started on the Moby [nat network][3]. The only requirement for IoT Edge is that this nat network has outbound internet connectivity to IoT Hub, a container registry, and optionally the Device Provisioning Service.

The remainder of this document discusses the setup on Linux, which provides better isolation, but is more complex.

# Default Topology

![IoT Edge network][network]

# Bridge Networks
# Networks

By default, IoT Edge uses two bridge networks for connectivity between containers.
By default, IoT Edge uses two networks for connectivity between containers.
This allows an additional form of isolation between the host network, the Edge Agent, and application containers.

The Edge Agent attaches to the [default docker bridge network][1] (named `bridge`).
This bridge network device shows up as `docker0` when running `ifconfig`.
The Edge Agent attaches to the [default docker network][1] (name and type are `bridge` on Linux, `nat` on Windows).
Inside a Linux container, this default network device shows up as `docker0` when running `ifconfig`.

The remaining containers, including the Edge Hub, are placed on a [user-defined bridge network][2] (named `azure-iot-edge`).
The remaining containers, including the Edge Hub, are placed on a [user-defined network][2] named `azure-iot-edge`. Like the default network, this network's type is `bridge` on Linux, `nat` on Windows.

## Default Bridge Network (docker0)
## Default Network

The Edge Agent is started by `iotedged` and is started without any explicit networking configuration.
By default, the Moby runtime places all containers without a defined network configuration on the default bridge network (`bridge`).
By default, the Moby runtime places all containers without a defined network configuration on the default network (named `bridge` on Linux, `nat` on Windows).

The Edge Agent requires outbound internet connectivity to IoT Hub to function properly.
This means that a route from the default docker bridge subnet to the internet must exist, no firewall rules are setup to block traffic, and IP forwarding is enabled.
All three of these conditions are met in a standard Docker installation.
This means that a route from the default docker subnet to the internet must exist, no firewall rules are setup to block traffic, and IP forwarding is enabled.
All three of these conditions are met in the standard Moby installation.

### Configuring `bridge`
### Configuring the Default Network

The Docker Engine creates and configures the default bridge network.
Changing this configuration is done at the Docker Engine level, by modifying the `daemon.json` config file and restarting the Docker Engine.
The Moby Engine creates and configures the default network.
Changing this configuration is done at the Moby Engine level, by modifying the `daemon.json` config file and restarting the Moby Engine.
This file is normally located at `/etc/docker/daemon.json` on Linux.

Reasons for changing this configuration include:
* Modifying the IP block used for assigning IPs to containers
* Changing the MTU
* Changing the gateway address

The usual reason for modifying this configuration is that some other subnet on the network clashes with the bridge network subnet.
The usual reason for modifying this configuration is that some other subnet on the network clashes with the default docker subnet.

## `azure-iot-edge` Network
## User-defined Network

Modules (containers), including the Edge Hub, are started by the Edge Agent and placed on a user-defined bridge network named `azure-iot-edge`.
Modules (containers), including the Edge Hub, are started by the Edge Agent and placed on a user-defined network named `azure-iot-edge`.
This network is created when the `iotedged` boots for the first time.

The Edge Hub requires outbound internet connectivity to IoT Hub to function properly.
This means that a route from the bridge subnet to the internet must exist and no firewall rules are setup to block traffic.
This means that a route from the `azure-iot-edge` subnet to the internet must exist and no firewall rules are set up to block traffic.

### Configuring `azure-iot-edge`

Expand All @@ -62,17 +58,17 @@ moby_runtime:
network: "azure-iot-edge"
```
Any changes to the specific settings of this network must be made out of band, via the Docker Engine.
Any changes to the specific settings of this network must be made out of band, via the Moby Engine.
Read the Docker [networking guide][4] for more information.
This network can be configured before starting IoT Edge, as the `iotedged` does a "get or create" operation on the network when booting.
This network can be configured before starting IoT Edge, as the `iotedged` does a "get or create" operation on the network when starting.
Pre-configuring a network and updating the network in the `config.yaml` allows complete control over its settings.

# Ports

IoT Edge does not require any inbound ports to be open for proper operation.

IoT Edge does require several ports to be open for outbound connectivity.
Depending on the scenario, IoT Edge does require several ports to be open for outbound connectivity.
The following table describes these requirements:

|Protocol | Port | Inbound | Outbound |
Expand All @@ -89,7 +85,7 @@ If no downstream devices are to connect to the edge device as a gateway, then al

*The `Edge Agent` and `Edge Hub` require one of ports 5671, 8883, or 443 open for outbound connectivity to IoT Hub.
The `iotedged` requires outbound connectivity on port 443 to IoT Hub, and optionally to the Device Provisioning Service (DPS), if DPS is used for provisioning.
The Docker Engine requires outbound connectivity on port 443 to a container registry.
The Moby Engine requires outbound connectivity on port 443 to a container registry.

# Upstream Protocol

Expand Down
2 changes: 1 addition & 1 deletion edgelet/contrib/config/windows/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -164,4 +164,4 @@ homedir: "C:\\ProgramData\\iotedge"

moby_runtime:
uri: "npipe://./pipe/iotedge_moby_engine"
# network: "nat"
# network: "azure-iot-edge"
2 changes: 1 addition & 1 deletion edgelet/iotedged/src/config/windows/default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ homedir: "C:\\ProgramData\\iotedge"

moby_runtime:
uri: "npipe://./pipe/iotedge_moby_engine"
network: "nat"
network: "azure-iot-edge"
23 changes: 10 additions & 13 deletions scripts/windows/setup/IotEdgeSecurityDaemon.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Set-Variable EdgeServiceName -Value 'iotedge' -Option Constant

Set-Variable MobyDataRootDirectory -Value 'C:\ProgramData\iotedge-moby-data' -Option Constant
Set-Variable MobyInstallDirectory -Value 'C:\ProgramData\iotedge-moby' -Option Constant
Set-Variable MobyLinuxNamedPipeUrl -Value 'npipe://./pipe/docker_engine' -Option Constant
Set-Variable MobyNamedPipeUrl -Value 'npipe://./pipe/iotedge_moby_engine' -Option Constant
Set-Variable MobyServiceName -Value 'iotedge-moby' -Option Constant

Expand Down Expand Up @@ -961,25 +962,21 @@ function Set-GatewayAddress {
function Set-MobyEngineParameters {
$configurationYaml = Get-Content "$EdgeInstallDirectory\config.yaml" -Raw
$selectionRegex = 'moby_runtime:\s*uri:\s*".*"\s*#?\s*network:\s*".*"'
$replacementContentWindows = @(
'moby_runtime:',
" uri: '$MobyNamedPipeUrl'",
' network: ''nat''')
$replacementContentLinux = @(
'moby_runtime:',
' uri: ''npipe://./pipe/docker_engine''',
' network: ''azure-iot-edge''')
switch ($ContainerOs) {
$mobyUrl = switch ($ContainerOs) {
'Linux' {
($configurationYaml -replace $selectionRegex, ($replacementContentLinux -join "`n")) | Set-Content "$EdgeInstallDirectory\config.yaml" -Force
Write-HostGreen 'Set the Moby runtime network to ''azure-iot-edge''.'
$MobyLinuxNamedPipeUrl
}

'Windows' {
($configurationYaml -replace $selectionRegex, ($replacementContentWindows -join "`n")) | Set-Content "$EdgeInstallDirectory\config.yaml" -Force
Write-HostGreen 'Set the Moby runtime network to ''nat''.'
$MobyNamedPipeUrl
}
}
$replacementContent = @(
'moby_runtime:',
" uri: '$mobyUrl'",
' network: ''azure-iot-edge''')
($configurationYaml -replace $selectionRegex, ($replacementContent -join "`n")) | Set-Content "$EdgeInstallDirectory\config.yaml" -Force
Write-HostGreen "Configured device with Moby Engine URL '$mobyUrl'."
}

function Get-AgentRegistry {
Expand Down

0 comments on commit 6d5b95a

Please sign in to comment.