Snowflake / Databricks SQL support via Ibis #100

hershelm · 2025-03-19T01:59:03Z

hershelm
Mar 19, 2025

Hey @rich-iannone, I am a long-time user of your package from R, and love to see it in Python, thank you!

I am having a similar issue to the user here:

#57

except mine is for Snowflake / Databricks. ibis supports both, but I am getting:

ValueError: The provided table object cannot be converted to a Narwhals DataFrame.

am totally down to help debug / support both of these backends further. thanks

Answered by rich-iannone

Mar 24, 2025

I think the PySpark error with preview() is fixed now in #112. I'll release pretty soon, so it's somewhat easier to test.

View full answer

rich-iannone · 2025-03-19T04:28:23Z

rich-iannone
Mar 19, 2025
Maintainer

Hey, thanks for using the package! What I’ve been doing is slowly enabling the use of different Ibis backends to whoever asks for it. The reason the support isn’t there in the first place is because I haven’t yet tested it.

So I’ll make a change (tomorrow) to enable the Databricks and Snowflake backends so you test it out from main. Thanks for volunteering with the testing/troubleshooting! Hoping these backends just work out of the box though.

4 replies

rich-iannone Mar 19, 2025
Maintainer

@hershelm from the just-merged PRs #101 and #102 , you can now get past the blocking ValueError and see how well the Ibis backends (snowflake, databricks, and pyspark) work with Pointblank!

hershelm Mar 20, 2025
Author

thanks @rich-iannone!

snowflake works, when testing pb.preview(TABLE_NAME)

both databricks and pyspark connections fail when testing pb.preview(TABLE_NAME), with the same error

---------------------------------------------------------------------------
RelationError                             Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) pb.preview(sales)

File /helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:450, in preview(data, columns_subset, n_head, n_tail, limit, show_row_numbers, max_col_width, incl_header)
    [447](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:447) if incl_header is None:
    [448](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:448)     incl_header = global_config.preview_incl_header
--> [450](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:450) return _generate_display_table(
    [451](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:451)     data=data,
    [452](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:452)     columns_subset=columns_subset,
    [453](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:453)     n_head=n_head,
    [454](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:454)     n_tail=n_tail,
    [455](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:455)     limit=limit,
    [456](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:456)     show_row_numbers=show_row_numbers,
    [457](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:457)     max_col_width=max_col_width,
    [458](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:458)     incl_header=incl_header,
    [459](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:459)     mark_missing_values=True,
    [460](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:460) )

File /helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:573, in _generate_display_table(data, columns_subset, n_head, n_tail, limit, show_row_numbers, max_col_width, incl_header, mark_missing_values, row_number_list)
    [571](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:571) # Convert either to Polars or Pandas depending on the available library
    [572](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:572) if df_lib_name_gt == "polars":
--> [573](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:573)     data = data_subset.to_polars()
    [574](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:574) else:
    [575](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/pointblank/validate.py:575)     data = data_subset.to_pandas()

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:611, in Expr.to_polars(self, params, limit, **kwargs)
    [583](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:583) @experimental
    [584](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:584) def to_polars(
    [585](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:585)     self,
   (...)
    [589](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:589)     **kwargs: Any,
    [590](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:590) ) -> pl.DataFrame:
    [591](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:591)     """Execute expression and return results as a polars dataframe.
    [592](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:592) 
    [593](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:593)     This method is eager and will execute the associated expression
   (...)
    [609](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:609)         A polars dataframe holding the results of the executed expression.
    [610](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:610)     """
--> [611](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:611)     return self._find_backend(use_default=True).to_polars(
    [612](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:612)         self, params=params, limit=limit, **kwargs
    [613](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/types/core.py:613)     )

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:265, in _FileIOHandler.to_polars(self, expr, params, limit, **kwargs)
    [240](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:240) """Execute expression and return results in as a polars DataFrame.
    [241](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:241) 
    [242](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:242) This method is eager and will execute the associated expression
   (...)
    [261](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:261) 
    [262](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:262) """
    [263](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:263) import polars as pl
--> [265](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:265) table = self.to_pyarrow(expr.as_table(), params=params, limit=limit, **kwargs)
    [266](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/__init__.py:266) return expr.__polars_result__(pl.from_arrow(table))

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:561, in Backend.to_pyarrow(self, expr, params, limit, **kwargs)
    [550](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:550) def to_pyarrow(
    [551](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:551)     self,
    [552](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:552)     expr: ir.Expr,
   (...)
    [557](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:557)     **kwargs: Any,
    [558](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:558) ) -> pa.Table:
    [559](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:559)     self._run_pre_execute_hooks(expr)
--> [561](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:561)     sql = self.compile(expr, limit=limit, params=params, **kwargs)
    [562](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:562)     with self._safe_raw_sql(sql) as cur:
    [563](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/databricks/__init__.py:563)         res = cur.fetchall_arrow()

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:135, in SQLBackend.compile(self, expr, limit, params, pretty)
    [108](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:108) def compile(
    [109](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:109)     self,
    [110](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:110)     expr: ir.Expr,
   (...)
    [115](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:115)     pretty: bool = False,
    [116](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:116) ) -> str:
    [117](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:117)     """Compile an expression to a SQL string.
    [118](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:118) 
    [119](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:119)     Parameters
   (...)
    [133](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:133)         Compiled expression
    [134](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:134)     """
--> [135](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:135)     query = self.compiler.to_sqlglot(expr, limit=limit, params=params)
    [136](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:136)     sql = query.sql(dialect=self.dialect, pretty=pretty, copy=False)
    [137](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/__init__.py:137)     self._log(sql)

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:597, in SQLGlotCompiler.to_sqlglot(self, expr, limit, params)
    [594](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:594) if params is None:
    [595](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:595)     params = {}
--> [597](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:597) sql = self.translate(table_expr.op(), params=params)
    [598](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:598) assert not isinstance(sql, sge.Subquery)
    [600](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:600) if isinstance(sql, sge.Table):

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:633, in SQLGlotCompiler.translate(self, op, params)
    [631](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:631) if self.lowered_ops:
    [632](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:632)     op = op.replace(reduce(operator.or_, self.lowered_ops.values()))
--> [633](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:633) op, ctes = sqlize(
    [634](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:634)     op,
    [635](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:635)     params=params,
    [636](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:636)     rewrites=self.rewrites,
    [637](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:637)     post_rewrites=self.post_rewrites,
    [638](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:638)     fuse_selects=options.sql.fuse_selects,
    [639](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:639) )
    [641](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:641) aliases = {}
    [642](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/compilers/base.py:642) counter = itertools.count()

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:381, in sqlize(node, params, rewrites, post_rewrites, fuse_selects)
    [379](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:379) # apply the backend specific rewrites
    [380](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:380) if rewrites:
--> [381](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:381)     node = node.replace(reduce(operator.or_, rewrites))
    [383](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:383) # lower the expression graph to a SQL-like relational algebra
    [384](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/backends/sql/rewrites.py:384) context = {"params": params}

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:525, in Node.replace(self, replacer, filter, context)
    [521](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:521)     kwargs[k] = v
    [523](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:523) # Call the replacer on the node with any rewritten nodes (or None
    [524](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:524) # if unchanged).
--> [525](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:525) result = fn(node, kwargs if changed else None)
    [526](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:526) if result is not node:
    [527](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:527)     # The node is changed, store it in the mapping of replacements
    [528](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:528)     replacements[node] = result

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:226, in _coerce_replacer.<locals>.fn(node, kwargs)
    [221](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:221) ctx = context or {}
    [222](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:222) # need to first reconstruct the node from the possible rewritten
    [223](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:223) # children, so we can match on the new node containing the rewritten
    [224](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:224) # child arguments, this way we can propagate the rewritten nodes
    [225](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:225) # upward in the hierarchy
--> [226](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:226) recreated = node.__recreate__(kwargs) if kwargs else node
    [227](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:227) if (result := obj.match(recreated, ctx)) is NoMatch:
    [228](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/graph.py:228)     return recreated

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:126, in Annotable.__recreate__(cls, kwargs)
    [122](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:122) @classmethod
    [123](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:123) def __recreate__(cls, kwargs: Any) -> Self:
    [124](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:124)     # bypass signature binding by requiring keyword arguments only
    [125](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:125)     kwargs = cls.__signature__.validate_nobind(cls, kwargs)
--> [126](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/common/grounds.py:126)     return super().__create__(**kwargs)

File /helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:353, in Set.__init__(self, left, right, **kwargs)
    [351](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:351)     if missing_from_right:
    [352](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:352)         msgs.append(f"Columns missing from the right:\n{missing_from_right}.")
--> [353](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:353)     raise RelationError("\n".join(msgs))
    [355](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:355) if left.schema.names != right.schema.names:
    [356](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:356)     # rewrite so that both sides have the columns in the same order making it
    [357](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:357)     # easier for the backends to implement set operations
    [358](https://vscode-remote+attached-002dcontainer-002b7b22636f6e7461696e65724e616d65223a222f616666656374696f6e6174655f70617265227d.vscode-resource.vscode-cdn.net/helix-llama/.venv/lib/python3.10/site-packages/ibis/expr/operations/relations.py:358)     cols = {name: Field(right, name) for name in left.schema.names}

RelationError: Table schemas must be equal for set operations.
Columns missing from the left:
ibis.Schema {
  ibis_ibis_row_number_5txakd6oxzdabeat2l3b3s5yre  int64
}.

rich-iannone Mar 24, 2025
Maintainer

I think the PySpark error with preview() is fixed now in #112. I'll release pretty soon, so it's somewhat easier to test.

Answer selected by hershelm

hershelm Mar 25, 2025
Author

works now, thanks!

MarcoGorelli · 2025-03-19T10:11:52Z

MarcoGorelli
Mar 19, 2025

Hey

ValueError: The provided table object cannot be converted to a Narwhals DataFrame.

In case it helps, I just wanted to point out that we have a PR in progress to support Ibis in Narwhals narwhals-dev/narwhals#2000
I'm hoping that by mid-April we can ship it and support Ibis across the (lazy-only) Narwhals API

1 reply

rich-iannone Mar 19, 2025
Maintainer

This is amazing and we will definitely migrate over to that once it ships!

Navigation Menu

Repository navigation

Snowflake / Databricks SQL support via Ibis #100

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Snowflake / Databricks SQL support via Ibis #100

Uh oh!

hershelm Mar 19, 2025

Replies: 2 comments · 5 replies

Uh oh!

rich-iannone Mar 19, 2025 Maintainer

Uh oh!

Uh oh!

rich-iannone Mar 19, 2025 Maintainer

Uh oh!

Uh oh!

hershelm Mar 20, 2025 Author

Uh oh!

rich-iannone Mar 24, 2025 Maintainer

Uh oh!

hershelm Mar 25, 2025 Author

Uh oh!

MarcoGorelli Mar 19, 2025

Uh oh!

rich-iannone Mar 19, 2025 Maintainer

hershelm
Mar 19, 2025

Replies: 2 comments 5 replies

rich-iannone
Mar 19, 2025
Maintainer

rich-iannone Mar 19, 2025
Maintainer

hershelm Mar 20, 2025
Author

rich-iannone Mar 24, 2025
Maintainer

hershelm Mar 25, 2025
Author

MarcoGorelli
Mar 19, 2025

rich-iannone Mar 19, 2025
Maintainer