Migrate containers from Vec<(Tuple)> to columnar containers - #54
Migrate containers from Vec<(Tuple)> to columnar containers#54frankmcsherry wants to merge 1 commit into
Vec<(Tuple)> to columnar containers#54Conversation
|
The plan is to start by migrating out the current structural requirements that e.g. we refer to data with The first PR removes the requirement of |
|
☔ The latest upstream changes (possibly #57) made this pull request unmergeable. Please resolve the merge conflicts. |
The
Relation<Tuple>type wraps aVec<Tuple>that is sorted and deduplicated. This is great, except for the opinion that it must be aVec<Tuple>. This is a stronger opinion about the layout of the data than we require. By constrast, thecolumnarcrate is able to lay out sequences of structured types using large allocations of simple types. There are a few advantages here:Copytypes become less painful. Strings, JSON, other non-trivial types avoid allocating all over the place and inducing lots of random accesses.(Key, Val)segregate the key information, which makes searching for keys that much easier. Only key data are brought in, and various forms of compression (e.g. run-length, for sorted data) can make the data that much smaller. Merging and matching keys is often a large source of work.(A, B, C, .. K)is a k-tuple of containers, and one can reshape it into a container for((A, B), (C, .. K))at essentially no cost. This allows a sorted list of tuples to be treated as a trie, and serve as indexes for any prefix of the sorted list. This avoids needing e.g. separate indexes for a(A, B)collection on each of(),(A), and(A, B). You'd need them onBand(B, A)additionally, but that would only be one additional relation/variable as well.NB: Draft for the time being; no action required. To be honest, I thought I was pointing at my fork and definitely don't mean to cause any work to happen looking at this yet. Oops. :D