Compare Spark RDDs, DataFrames, and SQL Performance Gains

Q: How do I practice SQL interview questions?

PracHub provides an interactive SQL console where you can write and test queries against real database schemas. Get instant feedback and compare your solution with the expected output.

Q: What difficulty level is this coding question?

This is a Medium difficulty Data Manipulation (SQL/Python) question, commonly asked during Technical Screen rounds at Experian.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Experian during technical interviews.

Question

SparkJobs

+---------+---------+---------------------+----------+
| job_id  | user_id | submit_time         | status   |
+---------+---------+---------------------+----------+
| 1001    | 17      | 2023-10-01 10:15:00 | running  |
| 1002    | 21      | 2023-10-01 10:20:00 | failed   |
| 1003    | 17      | 2023-10-01 10:25:00 | success  |
| 1004    | 42      | 2023-10-01 10:30:00 | running  |
+---------+---------+---------------------+----------+

##### Scenario

Discussion of distributed data processing tools used on big-data projects.

##### Question

Compare Spark RDDs, DataFrames, and Spark SQL. What performance gains come from Spark’s lazy evaluation model? When would you choose each abstraction?

##### Hints

Highlight memory management, query optimization, and developer ergonomics.

PracHub · Accepted Answer

This question evaluates understanding of Spark abstractions (RDDs, DataFrames, Spark SQL), lazy evaluation, memory management, query optimization, and developer ergonomics within the Data Manipulation (SQL/Python) domain of distributed data processing.

Quick Overview

Quick Overview