When Not to Use Apache Spark for Data Processing: A Guide for Data Engineers

Ioudom Foubi Jephte
4 min readJan 29, 2023

Recently, I got asked by a professional on LinkedIn that is building his startup, whether or not they should use Apache Spark to scrape and process data for job search purposes. Instead of focusing on telling them when or how to use Apache spark, I felt it will more beneficial for them to know when not to use Apache spark, that way they don’t fall into a trap.

--

--