Discuss conflicts, deadlines, and persuasion

How to answer effectively - Structure: Use STAR (Situation, Task, Action, Result) + Learnings. For conflict: add 5 Whys to diagnose root cause; for deadlines: add MoSCoW (Must/Should/Could/Won't) and Critical Path. - Be specific: Quantify scope, quality, speed, and cost (e.g., freshness SLA, p95 latency, failure rate, $/TB, incident rate, build time). - Anticipate follow-ups: Know exact definitions, baselines, trade-offs, risks, and how you validated outcomes. (a) Conflict with a colleague/stakeholder Framework - Diagnose: 5 Whys, shared goals, data/definitions review, assumptions audit. - Resolve: Clarify decision owner, propose options with trade-offs, agree on success metrics and a follow-up check. Sample answer (Data Engineer) - Situation: Data Science escalated that our "Daily Active User" dashboard was inconsistent with their model input by 10–15% across markets. - Task: Resolve the discrepancy and restore trust without blocking model retraining. - Action: - Reproduced the issue by running both pipelines against the same 7-day sample; built an incident timeline. - Root cause via 5 Whys: (1) mismatched DAU numbers → (2) different sessionization windows → (3) conflicting SQL in ETL vs feature store → (4) undocumented metric contract → (5) no ownership for DAU definition. - Led a 45-minute working session: aligned on a single DAU definition and wrote a one-page data contract (owner, logic, SLOs, test cases, and deprecation policy). - Implemented: standardized SQL in a shared library, added unit tests and Great Expectations checks, and set a pre-merge validation job. Published an RFC; scheduled a 30-day deprecation for the old definition with versioned tables. - Result: Reduced metric mismatch from 12% to <0.5% within 2 weeks; restored on-time model retraining; closed 3 related incidents; freshness SLO improved from 95% to 99.5%. Stakeholder CSAT (postmortem survey) improved from 3.2 to 4.6/5. - Learning: Conflicts often stem from unclear contracts and ownership; document, test, and version metrics. Follow-ups to prepare - Exact DAU SQL before/after. Test cases and thresholds (e.g., alert if drift >1%). Who owned the metric afterward and where it’s documented. - What you would do if disagreement persisted (tie-breaker via DRI, decision log). (b) Project with an immovable deadline Framework - Plan: Backward plan from the deadline; identify critical path; estimate and buffer; choose MoSCoW scope. - Communicate: Weekly status, risk register with owners, explicit trade-offs and rollback plan. Sample answer (Data Engineer) - Situation: A feature launch required a new streaming events pipeline live by quarter-end (hard date tied to marketing). Missing the date would block launch. - Task: Deliver real-time ingest (p95 < 5 min), two gold tables, and a dashboard; maintain 99.9% freshness SLO. - Action: - Decomposed DAG: ingest → schema registry → bronze → dedup/enrich (silver) → gold → BI. Identified critical path: schema + streaming job + gold tables. - Scope: Must (ingest, 2 gold tables, freshness alerts), Should (exactly-once semantics), Could (cost optimizations, 5 more dimensions). Set freeze: T-10 days. - Resourcing: Paired an SRE for IaC; staged environments; added 20% buffer on critical tasks. - Risks and mitigations: Schema churn → data contracts and a breaking-change window; throughput spikes → autoscaling and backpressure tests; dashboard performance → pre-aggregations. - Communication: Twice-weekly updates with a red/yellow/green status and a live trade-off log; pre-approved rollback (fail to batch if streaming unstable >30 min). - Outcome: Shipped Must scope on time; p95 latency 3.8 minutes (SLO 5), 0 launch-day incidents, 99.93% freshness in week 1. We deferred 5 non-critical dimensions and cost optimizations (delivered in the next sprint). Business KPIs: +6% CTR on the feature within 2 weeks. - Learning: Ruthless prioritization and pre-agreed trade-offs protect immovable dates without silent scope creep. Follow-ups to prepare - Your estimates vs actuals; critical path tasks and buffers used. The exact trade-offs you documented and who approved them. Rollback criteria and how you monitored them. (c) Most important end-to-end project Framework - Cover: problem and scale, stakeholders, architecture choices, your unique contributions, measurable impact, and operational excellence (monitoring/on-call). Sample answer (Data Engineer) - Situation: Batch analytics with 6–8 hour delays hurt product decisions; incidents averaged 3 per month. - Task: Design and lead a new unified events platform (ingest → storage → transformation → serving) with <10-minute latency, 99.9% freshness SLO, and lower costs. - Action (your contributions): - Authored the design doc and RFC; selected streaming over hourly micro-batch after replay tests showed 45% lower staleness for the same cost at p95. - Implemented Kafka ingestion with idempotent producers; wrote schema registry validators and a contract CI check to block breaking changes. - Built transformations with exactly-once via watermarking and dedup keys; created standardized gold table patterns with partitioning and Z-ordering. - Added data quality: 24 assertions (row counts, null %, distribution drift); lineage and dashboarding; on-call runbook and alerts. - Migration: dual-write and shadow-reads for 3 weeks; cutover only after p95 parity within 1%. - Impact: Reduced data delay from 6h to 6 min (p95), cut incident rate from 3/month to 0.4/month, improved query cost by 32% via pruning and pre-aggregations, and saved ~$24k/month by storage class tuning and spot autoscaling. - Learning: Contracts + observability are the backbone of reliable platforms; shadow traffic de-risks cutovers. Follow-ups to prepare - Exact dedup key, watermark policy, and late-data handling. Cost breakdown: ingest, storage, transform. What failed in the migration and how you mitigated it. (d) Persuading others using data Framework - Define the decision and alternatives; build a small but correct analysis; if possible, run a safe experiment (shadow, A/B, replay). Present trade-offs with quantified impact and a mitigation plan. Sample answer (Data Engineer) - Situation: BI team wanted to add 12 high-cardinality dimensions to a core fact table, increasing joins and storage. I believed it would degrade performance and raise costs for little benefit. - Task: Influence the design toward a star schema with 3 dimensions now and a lookup pattern for the rest. - Action: - Analyzed 90 days of query logs: only 18% of queries filtered on the proposed dimensions; high-cardinality would increase table size ~2.4×. - Modeled cost: queries/day × average scanned GB × $/TB. Example: 2,000 queries × 15 GB × $5/TB ≈ $150/day; with 2.4× size → ~$360/day (+$6.3k/month). - Prototype: built both schemas on a 5% sample and ran a replay. Results: star schema reduced p95 query latency by 29% and scan by 41% with identical answers on validation tests. - Addressed pushback: kept a compatibility view for 60 days; versioned metrics; provided a migration guide and identified dashboards affected; committed to monitoring and rollback. - Result: We adopted the star schema. Realized 38% lower compute cost and 26% faster p95 query latency over 60 days with zero correctness regressions; BI satisfaction improved in the post-change survey. - Learning: Decision logs + reproducible experiments build trust and speed up consensus. Follow-ups to prepare - Exact validation queries and thresholds; sample size and replay method; define p95 latency; how you ensured no correctness regressions (e.g., checksums, row counts, stratified sampling). Pitfalls to avoid - Vague impact ("helped a lot"); blaming individuals; skipping trade-offs; not defining metrics (e.g., freshness SLO vs SLA); ignoring rollback/mitigation plans. Guardrails and definitions - SLO vs SLA: SLO is your internal target; SLA is an external commitment with consequences. - Freshness: now − last_success_timestamp; latency: ingestion-to-availability time (report p95/p99). - Capacity quick math: storage/day ≈ events/sec × avg bytes × 86,400; ensure headroom for p95 spikes. Practice checklist - Write one STAR per prompt with concrete numbers, diagrams-in-words of your DAG, and who decided what. Time yourself to 2 minutes per story, with 2–3 layers of ready follow-ups.