Compare Spark RDDs, DataFrames, and SQL Performance Gains
Company: Experian
Role: Data Scientist
Category: Data Manipulation (SQL/Python)
Difficulty: medium
Interview Round: Technical Screen
SparkJobs
+---------+---------+---------------------+----------+
| job_id | user_id | submit_time | status |
+---------+---------+---------------------+----------+
| 1001 | 17 | 2023-10-01 10:15:00 | running |
| 1002 | 21 | 2023-10-01 10:20:00 | failed |
| 1003 | 17 | 2023-10-01 10:25:00 | success |
| 1004 | 42 | 2023-10-01 10:30:00 | running |
+---------+---------+---------------------+----------+
##### Scenario
Discussion of distributed data processing tools used on big-data projects.
##### Question
Compare Spark RDDs, DataFrames, and Spark SQL. What performance gains come from Spark’s lazy evaluation model? When would you choose each abstraction?
##### Hints
Highlight memory management, query optimization, and developer ergonomics.
Overview: This question evaluates understanding of Spark abstractions (RDDs, DataFrames, Spark SQL), lazy evaluation, memory management, query optimization, and developer ergonomics within the Data Manipulation (SQL/Python) domain of distributed data processing.
You are given `SparkJobs`, a table of submitted Spark jobs.
Return one row per job `status` with:
- `status`
- `job_count`: number of jobs in that status
- `first_submit_time`: earliest `submit_time`, formatted as `YYYY-MM-DD HH24:MI:SS`
- `last_submit_time`: latest `submit_time`, formatted as `YYYY-MM-DD HH24:MI:SS`
Order the result alphabetically by `status`.
Tables
SparkJobs(job_id INTEGER, user_id INTEGER, submit_time TIMESTAMP, status VARCHAR(10))
Hints
- Group by status.
- Use COUNT(*), MIN(submit_time), and MAX(submit_time).