Design an ML inference orchestration platform

Read the full interview experience this question came from →

Quick Overview

This question evaluates system-design and ML infrastructure competencies, focusing on orchestration of multi-model inference workflows, distributed data flow and storage, versioning and model management, scaling and deployment, failure handling, observability, and multi-tenant security and quota management.

Design an ML inference orchestration platform

Company: Palo Alto Networks

Role: Software Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Onsite

Your company exposes multiple ML models as independent services (e.g., classification, embeddings, re-ranking). Design a platform that orchestrates these models into end-to-end workflows for external users. Describe request routing/orchestration, data flow between models, schemas and validation, intermediate storage/caching, final results storage, model/version management, failure handling and retries, monitoring/traceability, scaling and deployment, and multi-tenant auth/rate limiting. Justify when to use RPC versus REST between services and the trade-offs in latency, schema evolution, observability, and backward compatibility.

Overview: This question evaluates system-design and ML infrastructure competencies, focusing on orchestration of multi-model inference workflows, distributed data flow and storage, versioning and model management, scaling and deployment, failure handling, observability, and multi-tenant security and quota management.

Read the full Palo Alto Networks Software Engineer interview experience this question came from

|Home/ML System Design/Palo Alto Networks
Palo Alto Networks logo
Palo Alto Networks
Sep 6, 2025
hardSoftware EngineerOnsiteML System Design
12
0

System Design: ML Inference Orchestration Platform

Context

You are designing a multi-tenant platform that exposes several ML models as independent services (for example: text classification, embeddings generation, and re-ranking). External clients should be able to invoke end-to-end workflows that chain these models. The platform must support both low-latency synchronous requests and higher-latency asynchronous jobs.

Assume:

  • External clients send requests with an input document (text or metadata), a chosen workflow, and optional parameters (model versions, thresholds, etc.).
  • Individual model services are independently deployed and versioned.
  • Workflows may include branching and parallel steps (e.g., compute embeddings while running classification, then re-rank results).

Task

Design the orchestration platform and address the following:

  1. Request Routing and Orchestration
    • How do external requests arrive and get routed to workflow execution?
    • How are workflows represented (DAG/state machine) and executed?
    • Synchronous vs. asynchronous execution.
  2. Data Flow Between Models
    • How are inputs/outputs passed between steps (in-memory vs. references)?
    • Handling large payloads and parallel branches.
  3. Schemas and Validation
    • Define request/response schemas and validation strategy across services.
    • Versioning and backward compatibility.
  4. Intermediate Storage and Caching
    • Where to store intermediate artifacts and how to cache reusable results (e.g., embeddings by content hash)?
    • TTLs and invalidation.
  5. Final Results Storage
    • How to persist final outputs for retrieval, analytics, and audit.
    • Retention and multi-tenant partitioning.
  6. Model and Version Management
    • Model registry, version pinning, canary/AB testing, and rollback.
  7. Failure Handling and Retries
    • Timeouts, retries with backoff/jitter, idempotency, partial failures, and fallbacks.
  8. Monitoring and Traceability
    • Metrics, logs, and distributed tracing across the workflow.
    • Per-tenant visibility and cost/QPS attribution.
  9. Scaling and Deployment
    • Autoscaling model services (CPU/GPU), concurrency controls, warm pools.
    • Multi-region, high availability, and deployment strategies.
  10. Multi-tenant Auth and Rate Limiting
  • Authentication/authorization, quotas, rate limiting, and isolation.
  1. RPC vs. REST Between Services
  • Justify when to use RPC versus REST for internal calls.
  • Discuss trade-offs in latency, schema evolution, observability, and backward compatibility.

Be explicit about assumptions and provide rationales for key choices.

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More ML System Design•More Palo Alto Networks•More Software Engineer•Palo Alto Networks Software Engineer•Palo Alto Networks ML System Design•Software Engineer ML System Design