Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flow Classification Rule outcome calculator needs to be direction aware #7036

Closed
Tracked by #7031
davidwatkins73 opened this issue Mar 20, 2024 · 1 comment
Closed
Tracked by #7031
Assignees
Labels
fixed (test & close) An issue has been fixed, merged into master and is ready for further testing
Milestone

Comments

@davidwatkins73
Copy link
Member

davidwatkins73 commented Mar 20, 2024

image

Changes needed (at first glance)

Step Changes Needed
1 Needs to discriminate between direction (possibly just a param)
2 This should probably be all under a tx (but may be slow)
3 Would need the direction from 1 propagating to it
4 Needs to know which field to target on the decorator table (outbound or inbound rule ref)
6 As 4
@davidwatkins73 davidwatkins73 changed the title Flow outcome calculator Flow outcome calculator needs to be direction aware Mar 20, 2024
@davidwatkins73 davidwatkins73 changed the title Flow outcome calculator needs to be direction aware Flow Classification Rule outcome calculator needs to be direction aware Mar 20, 2024
@davidwatkins73
Copy link
Member Author

Rewrtiting the evaluator to prevent hundreds of database interactions (see step 5).
Initial outlook is good - now only takes 10 seconds (as opposed to several minutes).

I made a further attempt to speed it up by removing flows which could not be covered by any statement because there are not statements for the flows datatypes. This only reduced the flow population by a small amount, therefore the increased complexity is not worth the minor performance lift. The code I tried (and am now removing is repeated below (just in case!)


        HashSet<Long> allInscopeDTIds = ruleVantagePoints
                .stream()
                .map(FlowClassificationRuleVantagePoint::dataType)
                .distinct()
                .reduce(
                        new HashSet<Long>(),
                        (acc, dt) -> {
                            acc.addAll(findChildren(dtHierarchy, dt.id()));
                            return acc;
                        },
                        (a, b) -> {
                            a.addAll(b);
                            return a;
                        });

        Set<FlowDataType> inScopePopulation = SetUtilities.filter(population, p -> allInscopeDTIds.contains(p.dtId()));
        LOG.debug("Initial Population: {}, Trimmed Population: {}", population.size(), inScopePopulation.size());

deutschebank-sync pushed a commit to deutschebank/waltz that referenced this issue Mar 27, 2024
- re-writing to an in-memory algo rather than lot's of database hits
- not ready for prime time - still in a harness

#CTCTOWALTZ-3091
finos#7036
deutschebank-sync pushed a commit to deutschebank/waltz that referenced this issue Mar 27, 2024
- re-writing to an in-memory algo rather than lot's of database hits
- not ready for prime time - still in a harness

#CTCTOWALTZ-3091
finos#7036
@davidwatkins73 davidwatkins73 added the fixed (test & close) An issue has been fixed, merged into master and is ready for further testing label Apr 9, 2024
@davidwatkins73 davidwatkins73 added this to the done : 1.60 milestone Apr 9, 2024
@davidwatkins73 davidwatkins73 modified the milestones: done : 1.60, 1.60.1 May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixed (test & close) An issue has been fixed, merged into master and is ready for further testing
Projects
None yet
Development

No branches or pull requests

2 participants