I noticed that in your CREATE TABLE DDL statements, the LOCATION fields point to individual files instead of S3 folders. This will lead to the "zero records returned" issue you encountered.
This is because for distributed systems that handles large amount of data, they operate on folders of files instead of files themselves. For example, in Presto (the underlying engine of Athena), a table maps to a folder of files with the same schema. Same in Spark (the underlying engine of Glue ETL jobs), a DataFrame represents a folder of files with the same schema. Based on the way these engines operate, the underlying SerDe used to read/write the data expect the specified LOCATION to be a folder.
When you create the table in Glue Catalog, the issue won't emerge yet, because at this point, you've only created metadata in the Catalog. However, when the engine (Athena) tries to use the metadata (including the SerDe) to read the files, it'd fail to recognize them.
SnowSQL config password needs to be enclosed in double quotes
-
Today I tested how to use snowsql to connect to SnowFlake. I put the
connection credential in the config file. But I encountered the following
error mess...
Acum o zi
Niciun comentariu:
Trimiteți un comentariu