add aws hadoop to DeltaIO - #39617
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
| "fs.s3a.endpoint", | ||
| localstack.getEndpointOverride(LocalStackContainer.Service.S3).toString()); | ||
| configuration.set("fs.s3a.access.key", localstack.getAccessKey()); | ||
| configuration.set("fs.s3a.secret.key", localstack.getSecretKey()); |
There was a problem hiding this comment.
Are all these properties required ? If not we should fine the minimum set that is required to use S3.
There was a problem hiding this comment.
Not all required. Some are for the test, but in real situations we can get down to a minimum set.
| permitUnusedDeclared "com.google.cloud.bigdataoss:gcsio:$bigdataoss_gcs_connector_version" | ||
| permitUnusedDeclared "com.google.cloud.bigdataoss:util:$bigdataoss_gcs_connector_version" | ||
|
|
||
| runtimeOnly library.java.hadoop_aws |
There was a problem hiding this comment.
Do we not need "aws-java-sdk-bundle" ?
There was a problem hiding this comment.
I think its pulled in as a transitive dependency from hadoop_aws
| @@ -0,0 +1,286 @@ | |||
| /* | |||
There was a problem hiding this comment.
Let's confirm that this passes for "beam_PostCommit_Java_Delta_IO_Dataflow" by modifying the trigger file.
|
Probably good to get this into 2.76.0 since this adds S3 support for a key feature (Delta Lake I/O). |
|
Assigning reviewers: R: @ahmedabu98 for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| outputs.upToDateWhen { false } | ||
|
|
||
| include '**/*IT.class' | ||
| exclude '**/DeltaIOS3IT.class' |
There was a problem hiding this comment.
Ah, didn't notice that we had to exclude here. Were you able to verify this end-to-end with real S3 data ?
Also, which Github action will pick up this test ?
There was a problem hiding this comment.
I was trying to in JB but couldn't due to the hadoop-aws dependency missing. I haven't tried again. Was hoping to get this in and try again.
I have an extra step in the post commit test to run the DeltaIOS3IT as a local simulated aws run via LocalStackContainer.
| resolutionStrategy.force 'org.apache.hadoop:hadoop-client:3.3.6' | ||
| resolutionStrategy.force 'org.apache.hadoop:hadoop-hdfs:3.3.6' | ||
| resolutionStrategy.force 'org.apache.hadoop:hadoop-hdfs-client:3.3.6' | ||
| resolutionStrategy.force 'org.apache.hadoop:hadoop-aws:3.3.6' |
There was a problem hiding this comment.
How much does this increase the io-expansion-service jar size by ?
There was a problem hiding this comment.
It actually decreases the jar size from 1.1GB to 0.8GB due to forcing it from 3.4.2 to 3.3.6.
2 participants
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.