space ocuppied in amazon s3 is more than expected

there is an issue with failed multipart upload.

check the link below

It was aborted multipart uploads. S3 keeps every uploaded part of every failed multipart upload indefinitely by default! A process had been failing and retrying multipart uploads without explicitly cleaning up the failed transfers.

We remedied this by temporarily enabling versioning, setting a lifecycle rule to remove aborted multipart upload chunks after 1 day, then waited a day, disabling versioning again once the chunklets were cleared.

https://serverfault.com/questions/965716/aws-s3-bucket-size-has-exploded-but-i-cant-figure-out-how

amazon athena = presto

Athena uses Presto and ANSI SQL to query on the data sets. It also uses HiveQL for DDL statements.

I noticed that in your CREATE TABLE DDL statements, the LOCATION fields point to individual files instead of S3 folders. This will lead to the "zero records returned" issue you encountered. This is because for distributed systems that handles large amount of data, they operate on folders of files instead of files themselves. For example, in Presto (the underlying engine of Athena), a table maps to a folder of files with the same schema. Same in Spark (the underlying engine of Glue ETL jobs), a DataFrame represents a folder of files with the same schema. Based on the way these engines operate, the underlying SerDe used to read/write the data expect the specified LOCATION to be a folder. When you create the table in Glue Catalog, the issue won't emerge yet, because at this point, you've only created metadata in the Catalog. However, when the engine (Athena) tries to use the metadata (including the SerDe) to read the files, it'd fail to recognize them.

Execute failed: Illegal character in opaque part at index 2

this error can appear in knime when you work with parquet files

the problem is that you try to store using the windows convention names (C:\..\..\)

Try to use the linux file path instead:   /../../

aws glue for beginners

https://data.solita.fi/aws-glue-tutorial-with-spark-and-python-for-data-developers/

OCD

https://www.psychiatry.org/patients-families/ocd/what-is-obsessive-compulsive-disorder

Obsessive-compulsive disorder (OCD) is an anxiety disorder in which time people have recurring, unwanted thoughts, ideas or sensations (obsessions) that make them feel driven to do something repetitively (compulsions). The repetitive behaviors, such as hand washing, checking on things or cleaning, can significantly interfere with a person’s daily activities and social interactions.

Many people have focused thoughts or repeated behaviors. But these do not disrupt daily life and may add structure or make tasks easier. For people with OCD, thoughts are persistent and unwanted routines and behaviors are rigid and not doing them causes great distress. Many people with OCD know or suspect their obsessions are not true; others may think they could be true (known as poor insight). Even if they know their obsessions are not true, people with OCD have a hard time keeping their focus off the obsessions or stopping the compulsive actions.

A diagnosis of OCD requires the presence of obsession and/or compulsions that are time-consuming (more than one hour a day), cause major distress, and impair work, social or other important function. About 1.2 percent of Americans have OCD and among adults slightly more women than man are affected. OCD often begins in childhood

best explanation for setting with copy warning in pandas

https://www.dataquest.io/blog/settingwithcopywarning/