Skip to content

Zachmu/indexed joins #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 72 commits into from
Dec 6, 2020
Merged

Zachmu/indexed joins #236

merged 72 commits into from
Dec 6, 2020

Conversation

zachmu
Copy link
Member

@zachmu zachmu commented Dec 1, 2020

This needs more unit tests for the new join planner and more extensive engine tests for various kinds of N-table joins, but otherwise this is complete and I'm confident it works (for some values of works)

zachmu added 30 commits November 6, 2020 17:29
@zachmu
Removed unused params from indexed_join.go (made unnecessary by recen…
2cb379e 
…t refactoring to how indexed table access lookups work)

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Clarified comment
51ad88c 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Sketching outline of getting indexes for all join conditions in a tre…
75c4e6e 
…e at once

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Merge branch 'master' into zachmu/indexed-joins
f1cdbba
@zachmu
Introduced joinIndexes type
20045a3 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
More renaming
1d69d6a 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Rewrote index analysis functions to combine joinColExpr with index in…
2fd4c38 
…fo, so that we have all the information we need in a single data structure.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Small refactoring
53172eb 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Renames and comments
cfb4c3f 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Bunch of test scripts
af8a1ab 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
More refactoring of join optimizations
6b3e636 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Better support for native indexes in the memory harness
89eb92d 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
New getTableNames method and small refactoring of existin table analy…
ae5a1b6 
…sis code

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Ordering all tables in a join to get an access plan
77215d5 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Removed unnecessary fields from IndexedJoin. Took some notes about a …
a329ed6 
…couple different strategies to use for getting table order correct in the join tree.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Large amount of sketching toward getting search algorithm working
661108d 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Mostly complete sketch of search algorithm
63b7a93 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Changing search params to track used tables and join conditions, rath…
4e1f729 
…er than removing them from the list of available ones.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Simplified the search params
e9e0c7f 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Filled in the base case, ready for testing
59d9907 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Tests for building a join tree, fixed tests for indexed join construc…
e8490d6 
…tors

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
First working join plan test
c5d0e28 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added debug printing, fixed bug in right-branch recurion
3ca4c60 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Tests for all 6 table order combinations for linear 3-table joins
bf9e66d 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Four table test case, and cleaned up tests a little
f271c04 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added tests for joins to the same table
6d3eba1 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
More tests for 4-table star joins
7f79377 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Tests for 5-table joins
f24d52c 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added some tests of compound join conditions
ce6ec91 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Moved join search code into its own file
230fbdd 
Signed-off-by: Zach Musgrave <[email protected]>
zachmu added 10 commits November 28, 2020 20:19
@zachmu
Fixed bug in join condition assembly, got all 3, 4, and 5 table index…
0187a3d 
… joins working. Now only left and right join logic remains.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added join position (left or right) to join index struct
48e41ff 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Fixed a bug in field indexes for plans including certain kinds of nod…
a463d4d 
…es that wrap the top-level join

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
First pass at ruling out illegal table access orders for left and rig…
6ad7410 
…ht joins

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Fully working left and right joins (almost)
c9cf5a0 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Changed move_join_conditions_to_filter rule to apply apply filter nod…
26fbe5e 
…es above join nodes, rather than above individual tables (pushdown will get to those). The new join optimization logic needs only tables and joins in the join part of the tree, and these intermediate filter nodes were interfering.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Generalized move_join_conditions_to_filter so that it works with arbi…
94c85d2 
…trarily deeply nested join nodes.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Fixed and added tests for move_join_conditions_to_filter
84d93fb 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Killed off the old optimize_joins file, and moved its remnants into i…
7526951 
…ndexes.go. Removed a bunch of unused functionality from joinColExpr

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Moved N-table indexed join tests into main scripts file
59c3b01 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu zachmu requested review from reltuk and Hydrocharged December 1, 2020 05:16
Hydrocharged
Hydrocharged approved these changes Dec 1, 2020
View reviewed changes
Copy link
Contributor

@Hydrocharged Hydrocharged left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small stuff! I didn't really see anything else in the meat and potatoes.

reltuk
reltuk approved these changes Dec 1, 2020
View reviewed changes
Copy link
Contributor

@reltuk reltuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! All questions and nits I think...

zachmu added 14 commits December 3, 2020 14:12
@zachmu
Changed BinaryNode's fields to be private and replaced with Right() a…
2185c0d 
…nd Left() accessors

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Changed join plan cost estimate to int64
98678d8 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
PR feedback, extracted a method and added some comments
3a12665 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Removed debug logging and rewrote permutation code (h/t aaron)
9085a5d 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Cleanup and comments
8a5cb32 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Replaced tree inspection to determine whether we replaced a table wit…
cc18743 
…h indexed access with direct return from the function that does the replacing.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Rewrote n table join queries to cover more cases, made script tests u…
9b94ffa 
…se indexes

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Simplified and made more foolproof the join index analysis
893acfc 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added a bunch more tests and fixed a determinism bug. LEFT / RIGHT jo…
35a9c48 
…ins of 3+ tables have some issues that we will need to address.

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Merge branch 'master' into zachmu/indexed-joins
b6df177
@zachmu
Formatting.
1d9ba66 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
One more query
f750662 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Fixed broken tests that were masked by the ORDER BY regex being broken
a688dd2 
Signed-off-by: Zach Musgrave <[email protected]>
@zachmu
Added DescribeQuery to indexed_joins.go logic, without which we can't…
f486d0d 
… see the actual query plan using EXPLAIN

Signed-off-by: Zach Musgrave <[email protected]>
@zachmu zachmu merged commit 59fe20b into master Dec 6, 2020
@Hydrocharged Hydrocharged deleted the zachmu/indexed-joins branch December 8, 2020 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reltuk reltuk reltuk approved these changes

@Hydrocharged Hydrocharged Hydrocharged approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zachmu @reltuk @Hydrocharged