![]() #SQL Timestamp String (default format) to DateType Spark.sql("select to_date(current_timestamp) as date_type") If you are from an SQL background these come in handy. ![]() # Using Cast to convert TimestampType to DateTypeĭf.withColumn('date_type', to_timestamp('input_timestamp').cast('date')) \įollowing are similar examples using with PySpark SQL. # Using Cast to convert Timestamp String to DateTypeĭf.withColumn('date_type', col('input_timestamp').cast('date')) \ Here is another way to convert TimestampType (timestamp string) to DateType using cast function. withColumn("datetype",to_date(col("ts"))) \ This example converts the PySpark TimestampType column to DateType.ĭf.withColumn("ts",to_timestamp(col("input_timestamp"))) \ The input to this function should be timestamp column or string in TimestampType format and it returns just date in DateType column.ĭf.withColumn("date_type",to_date("input_timestamp")) \ĭf.withColumn("date_type",to_date(current_timestamp())) \ĭf.select(to_date(lit('06-24-2019 12:01:19.000'),'MM-dd-yyyy HH:mm:ss.SSSS')) \ In this example, we will use to_date() function to convert TimestampType (or string) column to DateType column. Using to_date() – Convert Timestamp String to Date ![]() |- input_timestamp: string (nullable = true) Use to_date() function to truncate time from Timestamp or to convert the timestamp to date on DataFrame column. PySpark timestamp ( TimestampType) consists of value in the format yyyy-MM-dd HH:mm:ss.SSSS and Date ( DateType) format would be yyyy-MM-dd. To_date() – function formats Timestamp to Date. In this tutorial, I will show you a PySpark example of how to convert timestamp to date on DataFrame & SQL. PySpark functions provide to_date() function to convert timestamp to date (DateType), this ideally achieved by just truncating the time part from the Timestamp column.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |