Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AUTO] Optimize the device selection logic for auto-plugin #5549

Closed
wants to merge 1 commit into from
Closed

[AUTO] Optimize the device selection logic for auto-plugin #5549

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 74 additions & 5 deletions inference-engine/src/auto_plugin/auto_plugin.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,65 @@ namespace {
return config;
}

DeviceInformation SelectDevice(const std::vector<DeviceInformation>& metaDevices) {
std::string GetNetworkPrecision(const InferenceEngine::CNNNetwork &network) {
for (auto&& layer : network.getInputsInfo()) {
auto precision = layer.second->getPrecision();
auto name = std::string(precision.name());
if (name == "I8") {
name = "INT8";
}
return name;
}
return {};
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not network precision. It's precision of input layer..

Copy link
Contributor

@mashoujiang mashoujiang May 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, I remembered Maxim provided some code like:
auto nGraphFunc = network.getFunction();
for (auto &node : nGraphFunc->get_ordered_ops()) {
// auto type0 = node->input_value(0).get_element_type(); //input
auto type1 = node->input_value(1).get_element_type(); //weights
}
so is that expected? should we use typ1 or type0 as the network precision? there are lots of ops, should we use only the first one? thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

int8 network - if we have FakeQuantize operations in the graph
fp16 network - if we have convolution weights stored in fp16
fp32 network - if we have convolution weights stored in fp32

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused, as this PR overlaps heavily with (much larger) #5545, let's focus on 5545, OR change the 5545 to be merged into the branch from THIS requets.
When both requests are trying to merege to the master, we are reviewing the overlapping delta twice

ok, I will change the 5545 to be merged to this branch. the reason I create 5545 and cherry-pick this commit is because I want to test jenkins CI, it seems 5545 CI is OK, I will merge it to this. thanks!

Let's focus on #5545 , which already contains this PR's commit, I will close this PR later.


DeviceInformation SelectDevice(const InferenceEngine::CNNNetwork &network,
const std::vector<DeviceInformation>& metaDevices,
const std::vector<std::string>& optCap) {
if (metaDevices.empty()) {
IE_THROW(NotFound) << "No available device to select in AUTO plugin";
}
if (metaDevices.size() == 1) {
return metaDevices.at(0);
}

std::vector<DeviceInformation> CPU;
std::vector<DeviceInformation> GPU;

for (auto& item : metaDevices) {
if (item.deviceName.find("CPU") == 0) {
return item;
CPU.push_back(item);
continue;
}
if (item.deviceName.find("GPU") == 0) {
GPU.push_back(item);
continue;
}
}

auto networkPrecision = GetNetworkPrecision(network);
auto getCap = [&](std::string&& substr){
auto capability = std::find_if(optCap.begin(), optCap.end(),
[&](const std::string& c)->bool{ return (c.find(substr) != std::string::npos);});
return capability;
};

if (CPU.empty() && GPU.empty()) {
IE_THROW(NotFound) << "No available device found";
}

std::sort(GPU.begin(), GPU.end(), [](const DeviceInformation& a, const DeviceInformation& b)->bool{return b.deviceName < a.deviceName;});

if (!GPU.empty()) {
auto capability = getCap("GPU");
if (capability != optCap.end() && capability->find(networkPrecision) != std::string::npos) {
return GPU[0];
}
}
coneypo marked this conversation as resolved.
Show resolved Hide resolved
IE_THROW(NotFound) << "No available device could be used";
if (CPU.empty()) {
IE_THROW() << "Cannot select any device";
}
return CPU[0];
}
} // namespace

Expand All @@ -50,9 +102,9 @@ IE::ExecutableNetworkInternal::Ptr AutoInferencePlugin::LoadExeNetworkImpl(const

auto fullConfig = mergeConfigs(_config, config);
auto metaDevices = GetDeviceChoice(fullConfig);
auto optCap = GetOptimizationCapabilities();

// FIXME: always select CPU device now
DeviceInformation selectedDevice = SelectDevice(metaDevices);
DeviceInformation selectedDevice = SelectDevice(network, metaDevices, optCap);
IE::ExecutableNetwork executableNetwork;
try {
executableNetwork = GetCore()->LoadNetwork(network, selectedDevice.deviceName, selectedDevice.config);
Expand Down Expand Up @@ -179,6 +231,23 @@ std::vector<AutoPlugin::DeviceInformation> AutoInferencePlugin::GetDeviceChoice(
return metaDevices;
}

std::vector<std::string> AutoInferencePlugin::GetOptimizationCapabilities() const {
// FIXME: workaround to get devicelist.
std::unordered_set<std::string> capabilities;
std::vector<std::string> queryDeviceLists{"CPU", "GPU"};
for (auto &item : queryDeviceLists) {
try {
std::vector<std::string> device_cap =
GetCore()->GetMetric(item, METRIC_KEY(OPTIMIZATION_CAPABILITIES));
for (auto &dc : device_cap) {
capabilities.insert(dc);
}
} catch (...) {
}
}
return {capabilities.begin(), capabilities.end()};
}

//////////////////////////////////// private & protected functions ///////////////////
ConfigType AutoInferencePlugin::GetSupportedConfig(const ConfigType& config,
const std::string& deviceName) const {
Expand Down
1 change: 1 addition & 0 deletions inference-engine/src/auto_plugin/auto_plugin.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ class AutoInferencePlugin : public IE::InferencePluginInternal {

private:
std::vector<AutoPlugin::DeviceInformation> GetDeviceChoice(const ConfigType& config) const;
std::vector<std::string> GetOptimizationCapabilities() const;

protected:
ConfigType GetSupportedConfig(const ConfigType& config, const AutoPlugin::DeviceName & deviceName) const;
Expand Down