WIP: FastADC implementation #470
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ol-imorozko
force-pushed
the
FastADC
branch
2 times, most recently
from
6395e4a
to
394229a
Compare
This class represents an operator used in predicates for Denial Constrains (DC) representation. Predicates used there are less, greater, eq, neq, geq and leq. This C++ implementation is 100% bad, there is whole bunch of objects being created, but conceptually all of them are the same. Object "Operator '+'" and another object "Operator '+'" represents the same thing. ---------------------------------------------------------------------------- This is just a copy of a java code from https://github.com/RangerShaw/FastADC. I will refactor and think about better implementation later. I'll be just copying java code to get working algorithm ASAP, and after that I'll start thinking about good implementation.
This commit adds test_dc_structures.cpp file, which will be used to test different data structures which are required for DC representation (there are a lot).
This class represents a column operand within a predicate for FastADC.
FastADC processes Denial Constraints (DCs) that involve comparisons between
pairs of rows within a dataset. A typical DC example, derived from a Functional
Dependency (FD) such as A -> B, is expressed as: ∀𝑡, 𝑠 ∈ 𝑟, ¬(𝑡.𝐴 = 𝑠.𝐴 ∧ 𝑡.𝐵 ≠ 𝑠.𝐵).
This denotes that for any pair of rows in the relation, it should not be the case
that while the values in column "A" are equal, the values in column "B" are unequal.
A predicate in this context (e.g., 𝑡.𝐴 = 𝑠.𝐴) comprises three elements to be fully
represented: the column operand from the first tuple ("t.A"), the comparison operator
("="), and the column operand from the second tuple ("s.A"). The `ColumnOperand` class
encapsulates the column operand part of a predicate, such as "t.A" or "s.A".
First step in FastADC algorithm is to build so-called "Predicate Space". This is a long process during which many places in the code wants to get a Predicate. But each predicate is stored in a global storage -- map. In Java code this class (and other similar "provider" classes) are singletons. BaseProvider class is the class, from which a *Provider class should be derived. It ensures that only a PredicateBuilder class can initialize and free these singletons. I'm sure there exists a better approach, where we will store Provider classes in some fields to bind their lifetime more explicitly, but this is how it's done in Java, and I don't have much time to devise perfect architecture.
This class acts as a centralized storage to manage and provide access to Predicate objects. A Predicate is defined as "t1.A_i op t2.A_j", where t1 and t2 represent different rows, and A_i and A_j are columns (which may be the same or different) The FastADC algorithm first will build a so-called "Predicate Space", which is a set of all predicates that are allowed on R (set of rows, basically a table). In order to create and store predicates, this commit implements a singleton class with a hashmap storage.
FastADC processes Denial Constraints (DCs) that involve comparisons between
pairs of rows within a dataset. A typical DC example, derived from a Functional
Dependency such as A -> B, is expressed as:
`forall t, s in r, not (t.A = s.A and t.B != s.B)`
This denotes that for any pair of rows in the relation, it should not be the case
that while the values in column "A" are equal, the values in column "B" are unequal.
A predicate in this context (e.g., t.A == s.A) comprises three elements to be fully
represented: the column operand from the first tuple ("t.A"), the comparison operator
("="), and the column operand from the second tuple ("s.A").
This simple test creates two predicates on a 2x2 table and evaluates them. We're checking for mo::GetPredicate function ability to correctly create a predicate
In the original FastADC pull request this class manages creation of predicates, so it initializes PredicateProvider. But in this pr this class is not required for DC verification. Hence adding a temorary class just to make the tests work
TypedColumData kInt type is int64_t, and FastADC algorithm uses 64-bit long types
FastADC algorithm for mining approximate Denial Constraints will be implemented here.
IndexProvider assigns unique indices to each distinct object of type T added to it. It will be used later for two main operations: 1. Map out all predicates to numbers to use dynamic bitsets for quick intersection/etc. 2. Hash all values in the table keeping their relative order (ignoring columns of DC-unsupported types. Only ints, doubles and strings are allowed). That is, the same values are substituited by the same integers, and higher values are replaced by larger integers
FastADC algotithm decides which column pairs to use to create predicate with by checking whether they are comparable with `==, !=, <, >, >=, <=` or with `==, !=`. In both cases when the columns are of expected type (string, int or double) but different, we need to assert some kind of similarity between them. Otherwise the predicate space will be too big and not really interesting from the DC finding stadpoint, since there will be predicates like `!=` in between two completely different data attributes. These metrics are: - "shared percentage" Measures the overlap between two columns by considering the frequency of each unique element. It calculates the frequency of each unique value in both columns and determines the ratio of the shared values to the total values. - "average ratio" Computes the average value of each column and then returns the ratio of the smaller average to the larger average.
Generates and categorizes predicates for the future evidence set construction
This test builds predicate space from the provided data (CSV file) and compares the list of predicates that will be used later for DC discovery with the expected one. The expected list of predicates was built manually from running the FastADC Java implementation. The next test check that inverse and mutex maps are being built correctly.
This commit introduces the Position List Indexes (Pli) building. It's working with hashed column data, such that that equal values are represented by identical keys, and values are sorted by their natural order. We also build a so-called PliShards, which are just Pli's for a specific segment of the dataset, splitting whole dataset into a bunch of shards. This will allow us to be more efficient later.
This class organizes predicates into packs and creates a correction map, which will be used for optimizing predicate comparisons in derived clasees, that will actually build clues from PLIs
Inherits from CommonClueSetBuilder and builds clues based from one PLI shard
Inherits from CommonClueSetBuilder and builds clues based from two PLI shards
Validates the number of bits in the clue, the structure of the predicate packs. And the correction map which stores predicate-to-bitset mappings
This is a class for constructing clues from PliShards.
The expected values are, once again, are taken from Java implementation
For now this class builds necessary structures to build Evidences later. The structures are clue set, correction map and cardinality mask.
This is class that maps 1to1 with Clue. The ApproximateEvidenceInversion algorithm (AEI) that will build approximate denial constraints is using Evidences as it's input
EvidenceSet is basically just a vector of evidences. The only thing that's adding is a method to get total count (I probably can publically inherit from std::vector<Evidence>...?)
Add the building of evidences
Java code sometimes uses LongBitSet to store predicates, which is like boost::dynamic_bitset, but Java's implementation restructs number of bits in the clue to 64. We need to investigate further whether the Java's algorithm could work with predicate space more than 64. But for now we use 64 as maxumum amount of predicates
This class reorders predicates by evidence coverage to accelerate trie later
This class allows to efficently store bitsets and find whether one bitset is a subset of the stored bitsets
I've used `namespace model` as a placeholder before, we should use proper one
- "util" for structures with some complex logic - "model" for representation of concepts needed for denial constraint - "misc" for miscellaneous functions
Previously providers were defined as sigleton classes with static duration. That leaded to persisting state in between a googletest runs, plus this will make running two instances of FastADC in parallel impossible. Made them a normal class that should be created on FastADC algo creation and cleaned after, and we're passing pointers to related structures
Now there are a lot of common initialization code, so moved that to a googletest fixture
This way we can get rid of strange deriving in clue builder that was in Java. Now the separate class manages creation of auxiliary strucutres that we can just pass to clue/evidence set builders
This structure is renamed PackAndCorrectionMapBuilder, as it now builds not only packs and correction map, but also a correction map. So renamed that to EvidenceAuxStructuresBuilder
The zero clue presented a lot in the clues vector
ol-imorozko
force-pushed
the
FastADC
branch
2 times, most recently
from
f7a5ca2
to
3cc3060
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| return bset; | ||
| } | ||
|
|
||
| std::unordered_map<uint64_t, size_t> expected_clue_set = { |
There was a problem hiding this comment.
warning: use of undeclared identifier 'uint64_t' [clang-diagnostic-error]
std::unordered_map<uint64_t, size_t> expected_clue_set = {
^
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.