Problem
You are asked to implement a small program that retrieves data from a remote web service protected by token-based authentication, parses the response, and stores the parsed data into a database.
Requirements
-
Remote request with token-based authentication
-
The program should send an HTTP(S) request to a given URL (e.g., provided via a config file or command-line argument).
-
The remote service uses
token-based authentication
(e.g., a bearer token).
-
The program should attach the given token to the request in the appropriate HTTP header (for example,
Authorization: Bearer <token>
).
-
Parse response content
-
Assume the remote service returns a structured response (e.g., JSON) containing a list of items.
-
Each item has a few fields such as an
id
, a
timestamp
, and a
message
(you can assume reasonable names and types if not specified exactly).
-
The program should parse the response body and extract these fields into in-memory objects/records.
-
Store into a database
-
The program should connect to a relational database (e.g., PostgreSQL or MySQL) using a connection string or config.
-
Create (or assume the existence of) a table with appropriate columns to store the parsed fields (e.g.,
id
,
timestamp
,
message
).
-
Insert the parsed records into the table.
-
Handle duplicates in a sensible way (for example, avoid inserting the same
id
more than once, or perform an upsert).
-
Error handling and robustness
-
Handle common errors such as:
-
Network failures or timeouts when calling the remote service.
-
Non-2xx HTTP status codes.
-
Invalid or unexpected response formats.
-
Database connection or insertion errors.
-
Log errors or print meaningful messages so that a user or operator can understand what went wrong.
-
Execution interface
-
The program can be a command-line tool.
-
It should accept at least:
-
The remote service URL.
-
The authentication token.
-
Database connection information.
Describe how you would design and implement this program, including:
-
How you would structure the code (e.g., separation between HTTP client, parser, and database layer).
-
How you would manage configuration (URL, token, DB credentials).
-
How you would test it (unit tests, integration tests, mocking the remote service and DB).