Design an ML inference orchestration platform

Read the full interview experience this question came from →

Quick Overview

This question evaluates system-design and ML infrastructure competencies, focusing on orchestration of multi-model inference workflows, distributed data flow and storage, versioning and model management, scaling and deployment, failure handling, observability, and multi-tenant security and quota management.

Design an ML inference orchestration platform

Company: Palo Alto Networks

Role: Software Engineer

Category: ML System Design

Difficulty: hard

Interview Round: Onsite

Your company exposes multiple ML models as independent services (e.g., classification, embeddings, re-ranking). Design a platform that orchestrates these models into end-to-end workflows for external users. Describe request routing/orchestration, data flow between models, schemas and validation, intermediate storage/caching, final results storage, model/version management, failure handling and retries, monitoring/traceability, scaling and deployment, and multi-tenant auth/rate limiting. Justify when to use RPC versus REST between services and the trade-offs in latency, schema evolution, observability, and backward compatibility.

Overview: This question evaluates system-design and ML infrastructure competencies, focusing on orchestration of multi-model inference workflows, distributed data flow and storage, versioning and model management, scaling and deployment, failure handling, observability, and multi-tenant security and quota management.

Read the full Palo Alto Networks Software Engineer interview experience this question came from

|Home/Palo Alto Networks/ML System Design
Palo Alto Networks logo
Palo Alto Networks
Sep 6, 2025
hardSoftware EngineerOnsiteML System Design
15
0

System Design: ML Inference Orchestration Platform

Context

You are designing a multi-tenant platform that exposes several ML models as independent services (for example: text classification, embeddings generation, and re-ranking). External clients should be able to invoke end-to-end workflows that chain these models. The platform must support both low-latency synchronous requests and higher-latency asynchronous jobs.

Assume:

  • External clients send requests with an input document (text or metadata), a chosen workflow, and optional parameters (model versions, thresholds, etc.).
  • Individual model services are independently deployed and versioned.
  • Workflows may include branching and parallel steps (e.g., compute embeddings while running classification, then re-rank results).

Task

Design the orchestration platform and address the following:

  1. Request Routing and Orchestration
    • How do external requests arrive and get routed to workflow execution?
    • How are workflows represented (DAG/state machine) and executed?
    • Synchronous vs. asynchronous execution.
  2. Data Flow Between Models
    • How are inputs/outputs passed between steps (in-memory vs. references)?
    • Handling large payloads and parallel branches.
  3. Schemas and Validation
    • Define request/response schemas and validation strategy across services.
    • Versioning and backward compatibility.
  4. Intermediate Storage and Caching
    • Where to store intermediate artifacts and how to cache reusable results (e.g., embeddings by content hash)?
    • TTLs and invalidation.
  5. Final Results Storage
    • How to persist final outputs for retrieval, analytics, and audit.
    • Retention and multi-tenant partitioning.
  6. Model and Version Management
    • Model registry, version pinning, canary/AB testing, and rollback.
  7. Failure Handling and Retries
    • Timeouts, retries with backoff/jitter, idempotency, partial failures, and fallbacks.
  8. Monitoring and Traceability
    • Metrics, logs, and distributed tracing across the workflow.
    • Per-tenant visibility and cost/QPS attribution.
  9. Scaling and Deployment
    • Autoscaling model services (CPU/GPU), concurrency controls, warm pools.
    • Multi-region, high availability, and deployment strategies.
  10. Multi-tenant Auth and Rate Limiting
  • Authentication/authorization, quotas, rate limiting, and isolation.
  1. RPC vs. REST Between Services
  • Justify when to use RPC versus REST for internal calls.
  • Discuss trade-offs in latency, schema evolution, observability, and backward compatibility.

Be explicit about assumptions and provide rationales for key choices.

Browse More Questions

More ML System Design•More Palo Alto Networks•More Software Engineer•Palo Alto Networks Software Engineer•Palo Alto Networks ML System Design•Software Engineer ML System Design