databricks serverless syncer DELTA_FAILED_TO_MERGE_FIELDS

[perlego-maares]

First Stop

• I have read and agreed to the ThoughtSpot LICENSE.
• I understand that if I have a critical issue with ThoughtSpot, I should reach out to my designated Account Executive and Customer Success Manager instead of cs_tools.

Platform Configuration

                              │                                                                                                                      │                               
                              │           Info snapshot taken on 2025-11-04                                                                          │                               
                              │                                                                                                                      │                               
                              │               CS Tools: 1.6.5                                                                                        │                               
                              │         Python Version: Python 3.9.6 (default, Apr 30 2025, 02:07:17)                                                │                               
                              │    [Clang 17.0.0 (clang-1700.0.13.5)]                                                                                │                               
                              │            System Info: Darwin (detail: macOS-15.4.1-arm64-arm-64bit)                                                │                               
                              │      Configs Directory: /Users/ {anonymous} /Library/Application Support/cs_tools                                    │                               
                              │    Activate VirtualEnv: source "/Users/ {anonymous} /Library/Application Support/cs_tools/.cs_tools/bin/activate"    │                               
                              │          Platform Tags: macosx-10.9-universal2                                                                       │                               
                              │                                                                                                                      │                               
                              ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯     

Description

Initial setup shows MERGE issues, catalog, schema are empty, step command bellow

SYNCER='databricks://server_hostname=xxx.cloud.databricks.com'\
'&http_path=/sql/1.0/warehouses/xxx'\
'&access_token=dapixxx'\
'&catalog=thoughtspot'\
'&schema=cs_tools'\
'&load_strategy=TRUNCATE'\

cs_tools tools searchable metadata \
  --syncer "$SYNCER" \
  --config xxx > log.log

Additional info

Serverless compute connection, ensured have proper library

cs_tools                   1.6.5
databricks-sql-connector   4.1.4
databricks-sqlalchemy      2.0.8

Possibly related to #212, close if duplicate.

Considering adding a log file.

Truncated logs

cs_tools/.cs_tools/lib/python3.9/site-packages/databricks/sql/backend/thrift_backend.py:635 in            │                
               │ _check_command_not_in_error_or_closed_state                                                                                                        │                
               │                                                                                                                                                    │                
               │    632 │   ):                                                                                                                                      │                
               │    633 │   │   if get_operations_resp.operationState == ttypes.TOperationState.ERROR_STATE:                                                        │                
               │    634 │   │   │   if get_operations_resp.displayMessage:                                                                                          │                
               │ ❱  635 │   │   │   │   raise ServerOperationError(                                                                                                 │                
               │    636 │   │   │   │   │   get_operations_resp.displayMessage,                                                                                     │                
               │    637 │   │   │   │   │   {                                                                                                                       │                
               │    638 │   │   │   │   │   │   "operation-id": op_handle                                                                                           │                
               │                                                                                                                                                    │                
               │ ╭──────────────────────────────────────────────────────────────────── locals ────────────────────────────────────────────────────────────────────╮ │                
               │ │ get_operations_resp = TGetOperationStatusResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None,                   │ │                
               │ │                       errorMessage=None, displayMessage=None, errorDetailsJson=None), operationState=5, sqlState='22005', errorCode=0,         │ │                
               │ │                       errorMessage="org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_FAILED_TO_MERGE_FIELDS]          │ │                
               │ │                       com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields       │ │                
               │ │                       'org_id' and 'org_id'.\n\tat                                                                                             │ │