When Not to Use Apache Spark for Data Processing: A Guide for Data EngineersRecently, I got asked by a professional on LinkedIn that is building his startup, whether or not they should use Apache Spark to scrape and…Jan 29, 2023Jan 29, 2023
How to convert integer/seconds to HH:MM:SS format in a Spark Dataframe using spark SQLAt the time when this solution was developed as part of an ETL pipeline on Spark in Databricks, Spark did not provide a time data type…Mar 22, 2021Mar 22, 2021
