Compare Data Infrastructure Storage Choices
Company: XPeng
Role: Data Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Technical Screen
You are designing data infrastructure for an autonomous-driving company. Discuss the following storage and processing concepts and trade-offs:
1. In PySpark, what is the difference between row-wise and column-wise operations, and when is each appropriate?
2. What is Parquet, and why is it commonly used in analytical data pipelines?
3. When would you choose MongoDB or another NoSQL database instead of a lakehouse table format such as Iceberg?
4. When would you choose Iceberg instead of MongoDB or a traditional Hive table?
5. Why might a company migrate from Hive tables to Iceberg tables? What concrete improvements does Iceberg provide over Hive-style tables?
Quick Answer: This question evaluates a candidate's competency in data storage and processing choices, covering PySpark row-wise versus column-wise operations, columnar file formats like Parquet, the trade-offs between NoSQL databases and lakehouse table formats such as Iceberg, and motivations for migrating from Hive-style tables.