spark_jobs +---------+---------------------+-------+-----------+---------+ | job_id | submit_time | user | memory_gb | status | +---------+---------------------+-------+-----------+---------+ | 101 | 2023-10-20 10:35:00 | alice | 8 | success | | 102 | 2023-10-20 11:10:00 | bob | 16 | running | | 103 | 2023-10-20 12:40:00 | carol | 32 | failed | +---------+---------------------+-------+-----------+---------+ ##### Scenario Big-data discussion on Spark and cloud pipelines ##### Question Compare Spark RDDs, DataFrames and Spark SQL and state their advantages. What benefits does Spark offer versus classic MapReduce? Explain Spark’s lazy evaluation and how it optimizes execution. Describe how you would submit and monitor jobs on AWS EMR or a similar managed cluster. ##### Hints Mention in-memory processing, Catalyst and Tungsten optimizers, DAG scheduling, cost/scalability on the cloud.