-
Notifications
You must be signed in to change notification settings - Fork 1.9k
[Build][Spark-Dsv2]Create module SparkDsv2 for codes for Kernel based dsv2 connector #4885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
huan233usc
wants to merge
48
commits into
delta-io:master
Choose a base branch
from
huan233usc:create-module
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
cefc2de
add test catalog
huan233usc 2e30924
add basic test
huan233usc 668d91a
fix build
huan233usc 7e36962
fix attempt2
huan233usc 3400758
fix attempt3
huan233usc 111e86d
fix attempt4
huan233usc 947c606
fix attempt5
huan233usc 926b37c
fix 6
huan233usc 8e1e632
fix 7
huan233usc ecbdc09
fix 8
huan233usc 895af13
fix 9
huan233usc e8bc4c8
fix 19
huan233usc ff5453b
fix 11
huan233usc 79e840d
fix 12
huan233usc ef7f427
fix python test
huan233usc 44d2b63
fix build
huan233usc 658aa6e
debug log
huan233usc 1c5eaec
debug log
huan233usc ae7229d
fix
huan233usc 37fbcf3
fix
huan233usc 34e5905
revert and see what happens
huan233usc c28511a
revert and see what happens
huan233usc b9dc13f
fix attempt 2
huan233usc 47659b7
fix attempt 3
huan233usc fcf0d75
fix attempt 4
huan233usc e7baf3a
fix attempt 5
huan233usc e23cc4d
fix attempt 6
huan233usc c27743a
fix attempt 7
huan233usc 7384217
fix attempt 8
huan233usc 6bb9912
fix attempt 10
huan233usc 0852e20
fix attempt 11
huan233usc 3715bf0
fix attempt 12
huan233usc 778eec7
fix attempt 1234
huan233usc 31243a9
fix attempt 1235
huan233usc f28ddcd
fix attempt 1236
huan233usc 4070b45
fix
huan233usc d5df56b
revert
huan233usc 35f3c09
revert
huan233usc cbf839f
skip kernel
huan233usc c1cb547
skip kernel
huan233usc 6aec27e
skip kernel
huan233usc feef5aa
update workflow
huan233usc 465470a
update workflow
huan233usc 090a45c
update workflow
huan233usc 1fb90fb
update workflow
huan233usc f170ee2
update workflow
huan233usc 255e47c
update workflow
huan233usc d0341cc
update workflow
huan233usc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
44 changes: 44 additions & 0 deletions
44
spark-dsv2/src/test/java/io/delta/spark/dsv2/Dsv2BasicTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| /* | ||
| * Copyright (2025) The Delta Lake Project Authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package io.delta.spark.dsv2; | ||
|
|
||
| import static org.junit.Assert.assertThrows; | ||
| import static org.junit.Assert.assertTrue; | ||
|
|
||
| import org.apache.spark.SparkConf; | ||
| import org.apache.spark.sql.SparkSession; | ||
| import org.junit.Test; | ||
|
|
||
| public class Dsv2BasicTest { | ||
|
|
||
| private final SparkSession spark = | ||
| SparkSession.builder() | ||
| .master("local[*]") | ||
| .config( | ||
| new SparkConf() | ||
| .set("spark.sql.catalog.dsv2", "io.delta.spark.dsv2.catalog.TestCatalog")) | ||
| .getOrCreate(); | ||
|
|
||
| @Test | ||
| public void loadTableTest() { | ||
| Exception exception = | ||
| assertThrows( | ||
| Exception.class, () -> spark.sql("select * from dsv2.test_namespace.test_table")); | ||
|
|
||
| assertTrue(exception instanceof UnsupportedOperationException); | ||
| assertTrue(exception.getMessage().contains("loadTable method is not implemented")); | ||
| } | ||
| } |
68 changes: 68 additions & 0 deletions
68
spark-dsv2/src/test/java/io/delta/spark/dsv2/catalog/TestCatalog.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| /* | ||
| * Copyright (2025) The Delta Lake Project Authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package io.delta.spark.dsv2.catalog; | ||
|
|
||
| import java.util.*; | ||
| import org.apache.spark.sql.connector.catalog.*; | ||
| import org.apache.spark.sql.connector.expressions.Transform; | ||
| import org.apache.spark.sql.types.StructType; | ||
| import org.apache.spark.sql.util.CaseInsensitiveStringMap; | ||
|
|
||
| public class TestCatalog implements TableCatalog { | ||
|
|
||
| private String catalogName; | ||
|
|
||
| @Override | ||
| public Identifier[] listTables(String[] namespace) { | ||
| throw new UnsupportedOperationException("listTables method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public Table loadTable(Identifier ident) { | ||
| throw new UnsupportedOperationException("loadTable method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public Table createTable( | ||
| Identifier ident, StructType schema, Transform[] partitions, Map<String, String> properties) { | ||
| throw new UnsupportedOperationException("createTable method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public Table alterTable(Identifier ident, TableChange... changes) { | ||
| throw new UnsupportedOperationException("alterTable method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public boolean dropTable(Identifier ident) { | ||
| throw new UnsupportedOperationException("dropTable method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public void renameTable(Identifier oldIdent, Identifier newIdent) { | ||
| throw new UnsupportedOperationException("renameTable method is not implemented"); | ||
| } | ||
|
|
||
| @Override | ||
| public void initialize(String name, CaseInsensitiveStringMap options) { | ||
| this.catalogName = name; | ||
| } | ||
|
|
||
| @Override | ||
| public String name() { | ||
| return catalogName; | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed for DSP workflow