Design backend to score and classify tweets

You are asked to build a small backend application that processes Twitter data and uses an external LLM-style API to score and classify each tweet.

Requirements

1. Data Source
   • Use a public Twitter dataset from Kaggle (assume it is provided as CSV/JSON files with fields like tweet id, user id, timestamp, text, etc.).
2. Data Modeling and Ingestion
   • Define a strict data model for a tweet (e.g., using a schema-validation library like Pydantic in Python or an equivalent in your language of choice).
   • Validate each record from the Kaggle dataset against this model.
   • Ingest the validated tweets into a local database (you may choose a concrete technology, e.g., PostgreSQL, SQLite, or another relational/NoSQL store, but be prepared to justify your choice).
3. Inference Worker
   • Implement an inference worker service that:
     • Fetches tweets from the database that have not yet been processed.
     • Calls a third-party LLM-like API (referred to as the Grok API ) to:
       • Assign a numeric score to each tweet (e.g., sentiment score, relevance score, or toxicity score).
       • Classify each tweet into one or more categories (e.g., sentiment classes like positive/neutral/negative, or topical categories such as sports/politics/entertainment). You may define a reasonable category scheme.
     • Stores the score and classification results back into the database, associated with the corresponding tweet.
   • Your worker should be robust:
     • Handle API failures, timeouts, rate limits, and retries.
     • Avoid double-processing the same tweet.
4. Validation / Quality Check
   • Propose and implement at least one method to validate or sanity-check the quality of the scores and classifications. Examples:
     • Simple heuristic checks (e.g., tweets containing obvious positive/negative words vs. model sentiment).
     • Manual spot-checking tools (e.g., an endpoint or script that samples random tweets and displays their text and labels for review).
     • Aggregate statistics (e.g., distribution of sentiment scores; flag anomalies).
5. Interface / Demo (Conceptual)
   • Assume you will record a short demo of your system. You do not need to design a full UI here, but:
     • Describe what API endpoints, CLI commands, or simple views you would expose to demonstrate the system working end-to-end (from raw Kaggle data to classified tweets stored in the DB).
6. Non-Functional Requirements
   • Discuss how your design would handle:
     • Scalability if the Kaggle dataset is large (millions of tweets) or if you later switch from a one-time batch to continuous ingestion.
     • Reliability (e.g., preventing data loss, reprocessing after failures).
     • Observability (logging, metrics, and monitoring of ingestion and inference pipelines).

What to Deliver (in the interview)

In the interview, you should:

• Describe the overall architecture of your system: components, their responsibilities, and data flows.
• Justify your choices of technologies (language, database, libraries, queueing mechanisms, etc.).
• Detail the data model for tweets and for storing inference results.
• Explain the design of the inference worker , including error handling and concurrency.
• Explain your validation approach and how you would use it in practice.
• Optionally, outline how you would extend this system if you needed to support:
  • Different scoring/classification tasks.
  • Multiple LLM/ML providers instead of just the Grok API.