Fix Spark 3.5.6 compatibility: Add truncate with overwrite support to StagedDeltaTableV2

[gkpanda4]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Summary

Added support for TRUNCATE operations in Delta tables by:

• Implementing the SupportsTruncate interface in DeltaV1WriteBuilder
• Adding TRUNCATE and OVERWRITE_BY_FILTER capabilities to the table's supported capabilities
• Importing required SupportsTruncate class from Spark SQL connector API

This change enables proper handling of TRUNCATE TABLE operations on Delta tables.

Root Cause

Spark’s change apache/spark#50739 in 3.5.6 version

Spark 3.5.6 introduced stricter validation in the V2Writes rule (PR #50739) that requires tables to properly implement overwrite interfaces, not just declare capabilities.

1. Added stricter capability checks for TRUNCATE TABLE operations

The issue came from Delta tables not declaring TRUNCATE capability, leading to compatibility problems with Spark's enhanced validation system which became stricter with 3.5.6.

Fix

Updated StagedDeltaTableV2 class in DeltaCatalog.scala to properly support overwrite operations

Resolves #4671

How was this patch tested?

Tested locally by running scripts to Create and Replace Table

Does this PR introduce any user-facing changes?

No

[hvanhovell]

@GeetKrishna how is this tested? Manual tests tend to break.

[vkorukanti]

@GeetKrishna how is this tested? Manual tests tend to break.

Upgraded the Spark version used in CI builds to 3.5.6 which breaks the existing tests without this fix. So we got the coverage now.

[gkpanda4]

Thanks @vkorukanti