Design Data Quality and Observability Pipeline

Design a production data orchestration, data quality, and observability framework for batch and streaming data pipelines. Your design should address: 1. How would you orchestrate data pipelines with upstream and downstream dependencies? 2. What data quality checks would you implement, such as threshold checks, upstream dependency checks, schema checks, null checks, and primary-key checks? 3. What would a data observability framework include? 4. If a small percentage of streaming records, for example 2%, fails validation, would you stop the entire pipeline? Explain your decision process and handling strategy.