-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Iox #415 Make ServiceRegistry
aware of complete ServiceDescription
#860
Iox #415 Make ServiceRegistry
aware of complete ServiceDescription
#860
Conversation
Codecov Report
@@ Coverage Diff @@
## master #860 +/- ##
==========================================
+ Coverage 75.62% 75.81% +0.18%
==========================================
Files 335 335
Lines 12219 12269 +50
Branches 2037 2052 +15
==========================================
+ Hits 9241 9302 +61
+ Misses 2194 2176 -18
- Partials 784 791 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First part reviewed - except the tests
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
/// @brief Removes given service description from registry | ||
/// @param[in] serviceDescription, service to be removed | ||
/// @return true, if service description was removed, false otherwise | ||
bool remove(const capro::ServiceDescription& serviceDescription); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am confused here. If we remove a service the service is definitely removed after this call! And when the services was not added it is still removed.
Therefore, independent of the preconditions the user will always get the same result - a removed service. So why do we have to return a bool here?
This method will hardly be used to check if a service is added - for this we can use find.
I think we have to be cautious some time since such return values can increase the error handling overhead which brings in some cases no benefit and only drawbacks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is intended to indicate whether the service was there and removed or was not there and not removed. The logical state of the registry is the same in both cases, so we do not really need to communicate this. Personally I do those kind of things as well for more information if the user wants (the information was already computed and passing it is inexpensive). The user is technically free to ignore the return value (I know there are differing opinions on this one).
Regarding error handling overhead: the return value does not indicate a possible error, just further information on what actually happened.
This kind of style is definitely used in several APIs, but I am fine either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is intended to indicate whether the service was there and removed or was not there and not removed.
Exactly, that's the idea. There shall be the possibility to check whether remove was successful or remove was unsuccessful, when the element was not contained in the registry.
And when the services was not added it is still removed.
That's not entirely correct. It will not be removed if it wasn't added before. The bool
is used to signal this. I'll keep the additional info to be able to pass this to the user via the new ServiceDiscovery
API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mossmaurice I disagree. The reason is, what can a user do with this kind of information: One removes a service and after the call it is not there - this is the contract. If you return false I would assume that you tried to remove the service and failed therefore it is still present.
In my opinion it does not make sense to return here a bool. Do you have an argument for that? And if so, why not return an enum so that it is more readable and one really knows what is going on!
Please also consider that we should try to reduce unnecessary code bloat and error handling. If we do return here something we are forced by our standards to handle it and we turn this line
serviceRegistry.remove(myService);
into
if ( serviceRegistry.remove(myService) ) {
//...
} else {
//...
}
And as final argument: YAGNI, you aint gonna need it. I do not see the use case and if the use case arises we can still add it.
{ | ||
if (element == serviceDescription) | ||
{ | ||
return cxx::error<ServiceRegistryError>(ServiceRegistryError::SERVICE_DESCRIPTION_ALREADY_ADDED); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This error just increases the error handling overhead and has no value for the user. If a user calls add
, there are two possibilities:
- it is not added, but after the call -> it is added
- it is added, and after the call -> it is still added
Therefore, the user does not require any action here and if they would like to know if a certain service was registered they always can use find
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is also not an error if it already exists. Whether the user should be informed is debatable. I am fine with either way, just do not classify it as error.
{ | ||
cxx::set::remove(m_serviceMap[service].instanceSet, instance); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use the suggested data structure: std::map<std::map<CaproIdString_t, std::set<CaproIdString_t>>, CaproIdString_t> m_services
and refactor this method.
The implementation would be.
- Remove event entry from
std::set<CaproIdString_t>
- If the set from 1. is empty remove the instance entry from
std::map<CaproIdString_t, std::set<CaproIdString_t>>
- If the map from 2. is empty remove the service entry from `m_services.
You can safely remove m_serviceDescriptionVector
, m_instanceMap
and m_serviceMap
since the data structure from above is sufficient enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As described in another comment this structure is inefficient for storage (considering we would need static capacities for all of them) and also and does not lend itself for efficient operations, especially wildcard search. Furthermore the way it is proposed it it does not allow us to locate a full ServiceDescription
object which will contain more than 3 strings in the foreseeable future - we will need to embed custom information of DDS topics I think (like QoS), have not fully thought it through. But even if we do not need it for QoS, it is good to have to attach all information about a service here in one place, or at least point to it.
I am also not sure how a map as key would really behave, since it has no natural operator<
which is a requirement for key types (would maybe fall back to addresses of the keys here, but this could be problematic). Besides any map where keys or values are more than lightweight objects is doomed efficiency wise, since insertion and removal require copies of internal data (which would consist of maps of sets here). This is just not going to work efficiently
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my answer in the other thread
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To elaborate: Map keys need operator<
, but if it does not exist it will use std::less
in the map I think and use comparison on pointers to dynamically allocated data objects (allocated by the map). In other words: pointers to memory are the real key in this case, which is fine. Not fully sure though, our own map would have to behave in some reasonable way there as well and I do not know yet what this would be.
The storage size problem would be severe in any case for nested fixed-size containers.
This means we would need to implement a complex solution with a lot of drawbacks. We can have a simpler (still fairly complex) solution which at least avoids some of the drawbacks, is faster, generalizes in a better way to e.g. more keys, allows prefix based search and more.
The data structure itself is also somewhat easier to implement. But I fully intend to create the fixed-size map as well, just that we can design better algorithms use suitable abstractions instead for other problems.
const CaproIdString_t& service, | ||
const CaproIdString_t& instance) const | ||
{ | ||
if (instance == iox::cxx::string<100>(capro::AnyInstanceString)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please refactor find as well with the suggested data structure: std::map<std::map<CaproIdString_t, std::set<CaproIdString_t>>, CaproIdString_t> m_services
.
Then you can write the algorithm with much less code, somehow like this:
cxx::vector<uint64_t, MAX_SERVICE_DESCRIPTIONS> searchResult;
auto serviceStartIter = (service == capro::AnyServiceString) ? m_service.begin() : m_services.find(service);
auto serviceEndIter = (service == capro::AnyServiceString) ? m_service.end() : serviceStartIter + 1;
for(auto iter = serviceStartIter; iter != m_service.end() && iter != serviceEndIter; ++iter) {
auto instanceStartIter = (service == capro::AnyInstanceString) ? iter.begin() : iter.find(service);
auto instanceEndIter = (service == capro::AnyInstanceString) ? iter.end() : instanceStartIter + 1;
for(auto iter2 = instanceStartIter; iter2 != iter.end() && iter2 != instanceEndIter; ++iter2) {
for(auto event : iter2 )
searchResult.emplace_back(iter.first, iter2.first, event);
}
}
// maybe you have to rename iter to serviceIter, iter2 to instanceIter but this could would be much cleaner and faster
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to look a at the current implementation but I am sure that any solution with maps and sets is inferior to prefix tree implementations I propose (and the latter is easier to implement than a map or set, especially with our memory constraints).
The reason is simple: completing most operations in O(length of string to insert), as opposed to anything with maps which at least need O(log(#elements in container)*length of keys to compare) - and this is for find only, not considering for loops or anything else. These structures are much harder to implement with the logarithmic guarantees as well.
I have not checked the correctness of the above code, but if it is correct the estimate still holds true and gets worse with any nested loop iteration with find, as above.
Also we should not mix up short concise code with runtime-efficient code. For cases like this it is at least worth considering order of complexity of the algorithm (as above, I just gave lower bounds for comparison but the prefix tree order is tight). There are a lot of nested iterations going on, this is a bad sign. Of course we can argue that the results are small in most cases, but not in the worst and especially the wildcard case.
Nevertheless we can have concise and efficient code here at the same time with a combination of prefix trees for search for specific attributes (strings) and the right container abstraction for the elements themselves, which would be something like a minimal wrapper around a vector
(actually array
) or the ObjectPool
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elfenpiff That's a neat and small implementation of find()
! However, as stated in the other threads I'm not convinced that we will be able to build a fixed size cxx::map
, that can be place in shared memory. What is your take on that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be feasible. I see a map implementation as a challenge but not the static memory part and in some previous company I already have seen such a map. The tree can be implemented in the same fashion like the linked list which we already have (from the memory point of view).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is true, it is feasible with the same approach as the list, but it would still be the wrong structure for the job due to this nesting of maps and sets.
And it is a challenge I intend to tackle in the near future when we have a working approach. We would even be able to compare efficiency then, and my bet is against any map based approach (we can try with std::map
, which is likely better optimized then what we will be able to come up in cxx::map
without serious time investment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
service registry tests not yet reviewed
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
if (it->second == index) | ||
{ | ||
removedValue = it->second; | ||
it = m_serviceMap.erase(it); | ||
removedEntry = true; | ||
continue; | ||
} | ||
else if (removedEntry && it->second > removedValue) | ||
{ | ||
// update index due to removed element | ||
it->second--; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is broken. What happens when there is an entry with it->second > removedIndex
before the key was removed from the map?
removedValue
and removedEntry
are also not necessary. Just use index
and maybe rename it to removedIndex
, then the previous mentioned defect will also be fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The latter branch is only taken if a matching index was found aka removedEntry == true
. However, you're right that things are a bit entangled here. Refactored to make it more understandable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In your new fix PR I see that the iterator is always incremented but not part of the for loop header, could you please change this: #917
auto range = m_serviceMap.equal_range(service); | ||
for (auto entry = range.first; entry != range.second; ++entry) | ||
{ | ||
searchResult.push_back(m_serviceDescriptionVector[entry->second]); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is used quite often, can we have a lambda which could be used like
storeIndices(containerToStore, range);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With only two occurrences, I prefer to keep things how they are. ServiceRegistry::find
is very readable and fits well on one monitor currently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well, I count 4 ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any input on this? Resolving would be fine since the duplicated code is minimal, but you might revise your decision with four occurrences ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any input on this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still only count two occurrences 🤔 Let's leave it as it is 😇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elBoberido can you list your for occurrences here please so that @mossmaurice can see them as well.
… database efficently (WIP) Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
… and write down new test cases Signed-off-by: Simon Hoinkis <[email protected]>
…dServiceSingleInstance Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
…rviceRegistry::Error Signed-off-by: Simon Hoinkis <[email protected]>
…olete forward declaration Signed-off-by: Simon Hoinkis <[email protected]>
…Registry fails Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
94ef01d
to
cef63b5
Compare
…stry Signed-off-by: Simon Hoinkis <[email protected]>
…emove} Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
cd37e47
to
27271df
Compare
Signed-off-by: Simon Hoinkis <[email protected]>
Please have a look at the test case |
…removing a ServiceDescription from the ServiceRegistry and add related test cases Signed-off-by: Simon Hoinkis <[email protected]>
…#415-make-service-registry-aware-of-complete-service-description
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
cxx::vector<uint64_t, MAX_SERVICE_DESCRIPTIONS> intersection; | ||
|
||
// Find (K1, K2) | ||
// O(log n + log n + max(#PossibleServices + #possiblesInstances) + #intersection) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can write it much simpler, it is just: O(log(n))
Here is why:
log n + log n = log ( n * n ) = log ( n ^ 2 )
log ( n ^ 2 ) = 2 * log n
See: https://en.wikipedia.org/wiki/List_of_logarithmic_identities
And the big-O-notation: https://en.wikipedia.org/wiki/Big_O_notation states
that O(log(n))
that the Find(K1, K2)
function has an upper limit which is log(n)
with some constant.
Find(K1, K2) = O(log(n)) => Find(K1, K2) <= M * log(n)
with M > 0
. So the M
constant contains the factor 2 and all the constants with max(#PossibleServices + #possibleInstances) + #intersection)
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elfenpiff The first part is true, but we do not need the logarithm to argue this since it holds for any function f and constant C that O(C*f(n)) = O(f(n))
simply by definition.
The second part is wrong, the result set still may dominate this so it is O(log n + #resultset)
if we want to be exact. #resultset is not bounded by a constant and may depend on n. In the worst case it is n itself and naively we would then have O(n)
(no algorithm can be better in this case). To get a better estimate we introduce #resultset
as a second variable and essentially use multivariate O-notation.
In general #resultset will be small so we can think of this as O(log n)
for practical purposes but strictly speaking it is not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We hence can simplify this to O(log n + #resultset)
, but not O(log n)
. It maybe helps to see #resultset as some function of n which depends on the input query as well but is at most linear. We do not know more for general input and cannot improve the estimate for this reason.
Note that #resultset is the size of the intersection above. Computing the intersection takes linear time in the size of the operand sets if they are sorted. Note that the #resultset estimate I gave is also slightly wrong for this reason since computing the resultset can be more costly (say the result contains only 1 element but one of the operands has say 0.7*n candidates, computation will be linear in n in this case). If we keep the max as before this is accounted for, just remove one of the log n
terms.
TLDR: the estimate above with one log n term removed is as correct and cannot really be simplified without cheating.
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
iceoryx_posh/include/iceoryx_posh/internal/roudi/service_registry.hpp
Outdated
Show resolved
Hide resolved
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
Signed-off-by: Simon Hoinkis <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, tests still not reviewed.
There are also quite a few issues which are deferred to a follow up PR. Please document these like the the other todo #issueNumber
else it is quite hard to keep track of this
@@ -43,23 +43,175 @@ class ServiceRegistry_test : public Test | |||
} | |||
} | |||
iox::roudi::ServiceRegistry registry; | |||
iox::roudi::ServiceRegistry::InstanceSet_t searchResults; | |||
|
|||
iox::roudi::ServiceRegistry::ServiceDescriptionVector_t searchResults; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be named with sut
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, will do this with the next PR ;)
|
||
registry.find(searchResults, "a", "c"); | ||
EXPECT_THAT(searchResults.size(), Eq(0)); | ||
} | ||
|
||
TEST_F(ServiceRegistry_test, RemoveSingleFromMultipleServices) | ||
TEST_F(ServiceRegistry_test, | ||
AddingMultipleServiceDescriptionWithDifferentServicesAndRemovingSpecificDoesNotFindSpecific) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can surely be reworded 😄
AddingMultipleServiceDescriptionWithDifferentServicesAndRemovingSpecificDoesNotFindSpecific) | |
ServiceNotFoundAfterAddingAndRemovingToServiceRegistry) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will reword it in the next PR.
@elBoberido I've reviewed the tests. |
// after | ||
discoveryInfo.offerService(myServiceDescription); | ||
discoveryInfo.stopOfferService(myServiceDescription); | ||
discoveryInfo.findService("ServiceA", Wildcard); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that we now call findService
, provide a service ID and what will be returned is a container with full CaProServiceDescription? The Service IDs in this Service Description would be some kind of redundant information as hopefully you only get CaPro IDs that match the requested service ID
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's correct. Service IDs in this case would all be "ServiceA". IMHO this is beneficial to the old design as the user will now be able to read all the other members of a ServiceDescription
. This could in future be also e.g. QoS settings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be precise: The return value is a vector wrapped in an expected cxx::expected<ServiceContainer, FindServiceError>
and using ServiceContainer = iox::cxx::vector<capro::ServiceDescription, MAX_NUMBER_OF_SERVICES>;
renaming test cases and member variable Signed-off-by: Simon Hoinkis <[email protected]>
… fixing ServiceDescription deserialisation Signed-off-by: Simon Hoinkis <[email protected]>
renaming test cases and member variable Signed-off-by: Simon Hoinkis <[email protected]>
… fixing ServiceDescription deserialisation Signed-off-by: Simon Hoinkis <[email protected]>
Pre-Review Checklist for the PR Author
iox-#123-this-is-a-branch
)iox-#123 commit text
)git commit -s
)task-list-completed
)Notes for Reviewer
This pull request implements a new
ServiceRegistry
, which stores the completeServiceDescription
and takes n:m communication into account.Will be done in follow-up PR:
std::multimap
withcxx::trie
(will be done in Implement a prefix tree for fast string search #859)Checklist for the PR Reviewer
Post-review Checklist for the PR Author
References