Prepare for realtime delivery first: state the stream boundary, track the last applied position, and explain what happens after a gap or lost continuity. The new original reconnect drill includes actual local checks and explicit limitations.
The existing usage rollup, SQL migration, quota and webhook exercises remain useful transferable practice. They teach concurrency, isolation and operational judgment; they do not establish Ably-specific hiring questions.
Recruiter Conversation
reportedHalf of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.
What to demonstrate
- Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
- Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
- Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not
How to prepare
- Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
- Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
- If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window
Technical Assessments
reportedMost of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.
What to demonstrate
- Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
- Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
- Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
- Whether a failing case is isolated and explained before any edit is made to the code
How to prepare
- From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
- Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
- Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.
Take-Home Assignment
reportedThe README is read before the code, and a follow-up conversation is usually built from it, so treat every sentence you put there as a question you have agreed to answer. It needs the command that runs the thing, the assumptions you made where the prompt was ambiguous, and the limits of what you built stated with the preconditions that make them true. Overclaiming is the expensive mistake here. Writing that something is thread-safe, or constant-time, or handles files larger than memory invites a reader to check that exact line, and a claim the code cannot support costs more than silence would have.
What to demonstrate
- Whether the run instructions work from a clean clone, naming the exact commands, the language version you tested on, and any environment variable the program expects
- Whether ambiguities in the prompt are resolved in writing, with the interpretation you picked and the reason, rather than settled silently in the code
- Whether documented limits match the implementation, so a stated input bound is one the code enforces or at least does not contradict
- Whether the trade-offs you list come with the condition that would make you choose the other way, instead of reading as a list of alternatives you happened to consider
How to prepare
- Write the README before the final hour, then read the code against it claim by claim and correct or delete every statement the implementation does not back
- For each ambiguity in the prompt, write one sentence fixing your interpretation and keep it; those sentences become the assumptions section and your answer when someone asks why you did it that way
- Give the repository to someone who has not seen the prompt and ask them to run it using only what is written down, treating every question they have to ask you as a gap in the document
Final Leadership Interviews
reportedNobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.
What to demonstrate
- Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
- Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
- Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
- Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience
How to prepare
- Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
- For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
- Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed
PracHub editorial advice for the preparation topics above.
One shared connection pool for every tenant and every query class
A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.
Checking a quota with a select and then writing
Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.
Assuming the input fits in memory
Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.
Assuming fixed-width integer arithmetic cannot overflow
In languages with fixed-width integers, including C, C++, Java, Go and Rust, computing a midpoint as (lo + hi) / 2 overflows once the sum passes the type's maximum, so write lo + (hi - lo) / 2 instead. Say which language you are in: arbitrary-precision integers, as in Python or Ruby, remove this specific hazard and none of the others.
Choose a category, try a prompt, then open its approach, worked solution or follow-up when you need it.
Apply an ordered stream through gaps, replay and reconnect
Original fictional acme/orders stream: start at cursor100,total10. Receive103(+7),101(+2), replay101, then102(-1). Maintain a contiguous applied cursor and a two-record pending cap. On continuity loss, invalidate old callbacks and install an authoritative snapshot before resuming. These integers are application-owned sequences, not Ably timeserials.
Teaching visuals: delivery trace and recovery decisions.
Approach
- Separate the applied cursor from the largest received sequence.
- Buffer a bounded gap; an in-order missing record can drain a full queue.
- On continuity loss or overflow, enter RESYNCING and invalidate prior callback generations.
- Install consistent snapshot state and its cursor through an explicit snapshot/live handoff.
Worked solution 35 min
Original fictional acme/orders stream: start at cursor100,total10. Receive103(+7),101(+2), replay101, then102(-1). Maintain a contiguous applied cursor and a two-record pending cap. On continuity loss, invalidate old callbacks and install an authoritative snapshot before resuming. These integers are application-owned sequences, not Ably timeserials.
- Contract: one fictional application stream per (tenant,channel); immutable integer sequences and integer deltas. Initial cursor 100,total 10. These are not Ably timeserials. The cursor tracks applied progress, not the largest received event.
- Trace:103(+7) buffers with cursor 100,total 10. 101(+2) gives cursor 101,total 12. Replay101 is ignored. 102(-1) applies102 then buffered103, producing cursor 103,total 18. Replay103 is ignored.
- Continuity loss: enter RESYNCING, clear pending state, increment callback generation1→2. Old-generation callbacks cannot mutate the new replica. Install an authoritative snapshot108,total27, then109(-3) gives cursor 109,total 24.
- Overflow:with cursor 100 and capacity2, pending103/104 fill the cap. Arrival105 triggers RESYNCING and a new generation. Snapshot105,total 30, then106(+3), gives cursor 106,total 33. In-order missing events can drain a full buffer; duplicates consume no new slot.
- Snapshot/live handoff is outside the code: obtain an authoritative snapshot with a high-water mark, retain the live tail from that position, then install and drain it. Bound the handoff buffer and restart recovery if it overflows. A local generation is not authorization.
- 5,942 local checks passed, including 873 permutations of one to six events with immediate replay. Concrete traces, overflow, stale handlers, conflicting pending data and wrong-stream rejection were tested. No Ably SDK/API/network, durability or throughput test is claimed.
- The pending cap counts records, not bytes. Add payload-byte limits and a gap deadline: a missing predecessor can stall forever without filling the buffer. Monitor applied lag, oldest gap age, pending bytes and repeated resyncs.
- Single-threaded, in-memory scope:the constructor expects a validated snapshot and stream identity. Already-applied duplicate content is trusted from the immutable source. Durable state and cursor must commit atomically or use idempotent effects; the model does not guarantee exactly-once external delivery.
- Teaching diagrams: received versus applied progress and reconnect recovery decisions. These visuals are original fictional practice material.
class Replica:
def __init__(self, stream, cursor=100, total=10, capacity=2):
if type(capacity) is not int or capacity <= 0:
raise ValueError("positive capacity required")
self.stream = stream
self.cursor, self.total = cursor, total
self.capacity = capacity
self.epoch = 1
self.state = "LIVE"
self.pending = {}
def discontinuity(self):
self.epoch += 1
self.pending.clear()
self.state = "RESYNCING"
return self.epoch
def snapshot(self, epoch, cursor, total):
if epoch != self.epoch:
return "stale"
if self.state != "RESYNCING":
raise ValueError("snapshot requires resync")
if type(cursor) is not int or type(total) is not int or cursor < self.cursor:
raise ValueError("invalid or backward snapshot")
self.cursor, self.total = cursor, total
self.state = "LIVE"
return "installed"
def receive(self, epoch, stream, sequence, delta):
if stream != self.stream:
raise ValueError("wrong tenant/channel")
if type(sequence) is not int or type(delta) is not int or sequence <= 0:
raise ValueError("invalid message")
if epoch != self.epoch:
return "stale", []
if self.state != "LIVE":
return "snapshot-needed", []
if sequence <= self.cursor:
return "duplicate", []
if sequence in self.pending:
if self.pending[sequence] != delta:
raise ValueError("conflicting immutable sequence")
return "duplicate", []
if sequence != self.cursor + 1:
if len(self.pending) >= self.capacity:
self.discontinuity()
return "snapshot-needed", []
self.pending[sequence] = delta
return "buffered", []
applied = []
while True:
self.total += delta
self.cursor = sequence
applied.append(sequence)
sequence += 1
if sequence not in self.pending:
break
delta = self.pending.pop(sequence)
return "applied", applied
Scroll sideways to view long lines.
Follow-up
- How do you bound pending bytes and gap age?
- What transaction boundary makes a persisted application cursor reliable?
- Which authorization checks are required before accepting stream data?
Schedule ordered webhook retries with a heap of subscription queues
Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.
Approach
- Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's
next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, incrementattempt_countand setnext_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.- Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
- Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
- All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on
(subscription_id, next_attempt_at) where status in ('pending','failed_retryable')claimed withFOR UPDATE SKIP LOCKED, and the write-back must be fenced onlease_tokenso a worker that stalled and resumed cannot overwrite a newer attempt.
Follow-up
- One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
- A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
- The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?
Fold a deduplicated usage stream into hourly rollups
You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.
Approach
- Bucket on
occurred_at, neveringested_at:hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions.occurred_atsays which hour the customer is billed for;ingested_atsays how current the fold is. Using the second for the first makes late data invisible instead of correctable. - The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over
(tenant_id, idempotency_key)at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning byhash(tenant_id) % Pso each shard holds 1/P of the set and no tenant's keys straddle shards. - Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
- Accumulate in scaled integers, not binary floating point.
numeric(20,6)admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree. - Carry
source_max_ingested_at = max(ingested_at)over the events folded into each cell, and countevent_countover accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks. - State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes
stagingbills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.
Worked solution 25 min
- Write both key tuples down before any code: dedup key
(tenant_id, idempotency_key), cell key(tenant_id, workspace_id, sku, hour_start), withhour_startderived fromoccurred_atin UTC. - Build a 10,000-row fixture containing one event duplicated three times under the same
idempotency_key, two events sharing anidempotency_keyacross differenttenant_idvalues, one event whoseoccurred_atis two hours before itsingested_at, and onestagingevent inside an otherwise production cell. - Fold it and assert each of those four expectations separately rather than eyeballing a grand total.
- Re-run with the input shuffled and diff the output files.
- Size the dedup set for 250M keys using the load-factor arithmetic and write the number down next to the fixture.
Follow-up
- A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
- The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
- What makes a re-run over the same day produce byte-identical rollups?
Find peak concurrent sandbox usage from run intervals
Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.
Approach
- Turn each run into two sweep events,
(started_at, +1)and(end, -1), then sort the 2n events by timestamp with-1ordered before+1at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap. - Decide each null out loud before sweeping, because each choice moves the answer. A null
started_atmeans queued and contributes nothing. A nullfinished_atwith statusrunningorleasedis clipped to the window end. Statuslosthas no observed end at all, so clip it atstarted_at + wall_clock_limit_secondson the grounds that the supervisor owns the timeout, and record that you did. The table'scheck (finished_at is null or started_at is not null)guarantees you never see an end without a start. - Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update
peak_atonly on a strict increase, or you will report the last such instant instead of the earliest). Capture the firstrun_idwhose+1takes the counter to C+1 during the same sweep rather than in a second pass. - Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by
started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on(tenant_id, started_at). - If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.
Follow-up
- Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
- The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
- How would you answer 'peak concurrency within any 5-minute window' without re-sorting?
Enforce a concurrent-run quota that survives simultaneous requests
A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.
Approach
- Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
- Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as
insert ... select ... where (select count(*) ...) < 20narrows the race to the statement's snapshot but does not close it under read committed. - Give the version that holds at read committed: serialise on a row both transactions must touch.
update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning runningupdates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract. - State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
- Close the leak the status enum implies: a run can end as
lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule againststatus = 'running' and leased_until < now(), and treat the counter as a fast path overjob_run, which stays the system of record.
Follow-up
- Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
- Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
- The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?
Migrate a live partitioned event table without blocking ingest
usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.
Approach
- Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so
alter column ... typerewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement. - Expand: add
quantity_numeric numeric(20,6)andenvironmentwith its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each withlock_timeoutset to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage. - Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
- Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
- Make NOT NULL cheap with the three-step form:
add constraint ... check (environment is not null) not valid(brief ACCESS EXCLUSIVE, no scan), thenvalidate constraint(SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), thenset not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent. - Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.
Worked solution 45 min
- On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
- Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
- Run the expand step with
lock_timeout = '2s'while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers. - Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
- Run the not-valid, validate, set-not-null sequence and confirm from
pg_stat_activityand timings that nothing held an exclusive lock through a full scan. - Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.
Follow-up
- A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
- The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
- Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.
What are the differences between strongly and weakly typed languages?
What are the differences between strongly and weakly typed languages?
Approach
- “Strong” and “weak” typing are informal labels. Make the question concrete: which conversions occur implicitly, which operations are rejected, and when are checks performed? Static versus dynamic checking is a separate dimension. Python is dynamically typed, but adding a string to an integer fails; JavaScript permits some coercions that can surprise an application.
- For a realtime payload, demonstrate schema validation at the boundary: sequence numbers must be integers, IDs must have defined scope, and incompatible message versions need a documented response. A language label does not replace validation of external data.
Follow-up
- Which implicit conversion would break your payload contract?
- Why is static versus dynamic typing a separate distinction?
What was your relationship with Product like at your last role?
What was your relationship with Product like at your last role?
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
How do random number generators work?
How do random number generators work?
Approach
- A pseudorandom generator updates internal state using a deterministic algorithm. Its seed fixes a reproducible sequence, which is useful for simulation and tests. It does not become cryptographically secure because the output looks irregular. Security-sensitive tokens require an appropriate cryptographic generator backed by trusted entropy.
- Python's
randommodule uses Mersenne Twister for its general-purpose generator and explicitly warns against security use. A bounded integer also needs careful mapping: applying% nto a uniform finite integer range can bias outcomes unless the range divides evenly. Rejection sampling discards the extra tail. For a uniform byte andn=10, accept values 0–249, reject 250–255, then reduce modulo 10; each result has 25 accepted representatives. - For an interview answer, distinguish statistical quality, reproducibility, period, unpredictability, and sampling bias. If the task is reconnect jitter, discuss avoiding synchronized retries. If it is credential generation, choose the security API instead of inventing a generator. A deterministic seed is a test tool, not a secrecy guarantee.
Follow-up
- Explain why byte modulo10 is biased without rejecting250–255.
- When would you require a cryptographic generator rather than reproducible PRNG output?
Recover a realtime subscriber after a gap or continuity loss
Define ordering, replay and backpressure behavior for a realtime subscriber that misses an update and reconnects. Which evidence tells you the application is current?
Approach
- Define the unit of ordering, durability boundary, permitted duplicates, maximum interruption, and consumer capacity. Draw one failure trace before choosing a database or queue. Then name the evidence that would disprove your design: gap age growing, repeated snapshots, cross-tenant delivery, or an acknowledgement that advances without application.
Follow-up
- What if a gap never fills but the pending queue stays small?
- Can a local callback generation replace authorization?
Webhook fan-out with per-endpoint isolation and backoff
One domain event fans out to every matching subscription, producing a webhook_delivery row per (subscription_id, event_id, redelivery_seq). Peak unique event rate is 20k/second; attempts run five to ten times that once fan-out and retries are counted. One customer endpoint has returned 503 for six hours and its backlog holds days of events; every other customer must be unaffected. Design the delivery system: how a worker claims work, the backoff schedule, the per-endpoint circuit breaker, the queue partitioning, and whether you offer ordering per subscription. State the delivery guarantee in one sentence.
Approach
- State the guarantee first, because it determines the rest: at-least-once with a stable event_id, and the consumer documented as responsible for idempotency. Exactly-once over HTTP is not deliverable - the 200 can be lost after the customer has already committed - so any design that promises it is either lying or is really offering at-most-once.
- Partition work per subscription rather than into one global pool, with a concurrency cap per subscription. With a shared pool, the endpoint that has been dead for six hours consumes workers on retries that will fail, and every other customer's delivery latency rises: head-of-line blocking across tenants is the exact failure being designed against here.
- Claim by compare-and-set with a fencing token: UPDATE webhook_delivery SET status = 'in_flight', lease_token = $new, leased_until = now() + interval '60 seconds' WHERE delivery_id = $1 AND status IN ('pending','failed_retryable') AND (leased_until IS NULL OR leased_until < now()), and make the terminal write carry AND lease_token = $new so a paused worker's late write is rejected rather than overwriting a newer attempt. Find due work through the partial index on next_attempt_at WHERE status IN ('pending','failed_retryable'), so the scan is proportional to live rows rather than to the terminal rows that outnumber them by orders of magnitude.
- Use full jitter: sleep uniformly in [0, min(cap, base x 2^(attempt-1))]. Plain exponential backoff hands a recovering endpoint its entire backlog as one synchronised herd and knocks it over again; full jitter de-correlates it. Then check the schedule actually spans the retention you promise - with base 1 s and a 3,600 s cap, twenty attempts have an expected total elapsed time of only about 4.6 hours, so a twenty-four-hour promise needs roughly fifty-nine attempts or a larger cap.
- Trip a circuit per endpoint on consecutive failures or a failure ratio over a rolling window: stop dispatching, push next_attempt_at out or mark new deliveries dropped_circuit_open, and half-open with exactly one probe rather than a batch. Bound the backlog explicitly with a per-subscription cap or retention, and decide in advance whether a recovered endpoint receives six hours of events at full rate or a pointer telling it to fetch what it missed.
- Offer ordering only as an opt-in mode of one in-flight attempt per subscription, and price it honestly: with parallel attempts a retried event overtakes a newer one, so ordering requires serialisation, and serialisation means one slow endpoint blocks its own queue entirely. That converts a shared problem into that customer's own problem, which is the right place for it, but it is still a real cost.
Worked solution 35 min
- Compute the attempt rate: 20k unique events/second x mean fan-out x retry multiplier, and size worker pools and the per-subscription concurrency cap from it.
- Write the claim statement and the terminal write, and point at the clause that rejects a resumed worker's stale write.
- Tabulate the backoff for attempts 1 to 20 with base 1 s and cap 3,600 s, take the expected value of each full-jitter sleep as half its ceiling, and sum to get total expected elapsed coverage.
- Decide the policy for a subscription down six hours: events buffered, bytes held, and what the customer actually receives when it returns.
Follow-up
- The endpoint recovers. Does it receive six hours of events at full rate, and what does that do to it?
- Trace the exact code path by which an event belonging to one tenant could be signed and sent to another tenant's endpoint.
- A customer insists they never received an event your row marks delivered. What evidence do you have, and what does payload_digest let you prove?
Regional error rate explodes after a dependency merely slows
A control-plane read replica in one region degrades from 4 ms to 120 ms. Within ninety seconds that region's gateway error rate rises from 0.01% to 40% and its p99 becomes bimodal, one mode near the old p99 and one at the client timeout. The other two regions are unaffected. The gateway retries control-plane reads three times with exponential backoff and no jitter. Give an ordered checklist that separates trigger from amplifier, the offered-load arithmetic, and the controls that break the loop.
Approach
- Split the incident into three questions before touching a control: what started it, what amplified it, and what would make recovery slow. Here they are the replica slowdown, the retry policy interacting with queueing, and a synchronised unjittered herd at recovery. They are different mechanisms and each needs its own control.
- Read the distribution rather than the mean. A bimodal p99 with one mode pinned at the client timeout is two populations, not one degraded path; split latency by cache hit and miss and confirm the fast mode is hits and the timeout mode is misses that reached the replica.
- Do the load arithmetic. Three retries turn one client request into up to four upstream requests, so offered load reaches roughly 4x on a dependency that is already slower, and it arrives at the worst moment. With utilisation approaching one, queueing delay grows superlinearly, which is why a 30x latency increase upstream does not produce a 30x increase downstream, it produces timeouts.
- Break the loop with controls that bound offered load rather than with more attempts: a concurrency limit on the control-plane client so at most N calls are in flight and the remainder fail fast, a circuit breaker scoped per dependency and region, and a retry budget capping retries at a small fraction of base traffic so amplification has a ceiling that does not depend on how many clients are retrying.
- Add full jitter to whatever retries survive, sleeping uniformly in [0, min(cap, base x 2^attempt)], so attempts de-correlate instead of arriving in waves aligned to the moment of failure.
- Decide the unreachable-dependency behaviour in advance, because it is the actual product decision underneath: serving from an expired credential cache keeps the product available while extending a revoked key's life past the stated bound, and failing closed converts a dependency degradation into a total outage. State the mode and the staleness number rather than letting the timeout choose.
Follow-up
- The replica recovers. Describe what happens in the first ten seconds with your controls in place versus without them.
- Which single metric would have paged before the error rate moved, and why is upstream latency by itself not it?
- Requests that fail fast under the concurrency limit still need an answer. What does the gateway return, and what does it do to the usage event it would otherwise have emitted?
Choose coding or senior design depth from your confirmed loop. Start with the delivery trace and tested model; retain database and webhook exercises for isolation and operational practice.
Prepare, practise & reflect
One practical outcome each day. Spend longer where you need it.
0 / 7 done01Define delivery scope
- Name tenant/channel boundary, immutable event identity, ordering scope and acknowledgement meaning.
- Read current official connection/channel/history docs; distinguish SDK recovery from application persistence.
Deliverable: A stream contract and source notes.
Practice prompt ↗Worked solution ↗02Work the gap and replay trace
- Manually derive cursor100→101→103,total10→12→18.
- Implement the model and add duplicate/conflicting-payload checks.
Deliverable: Correct trace, code and expected-result tests.
Practice prompt ↗Worked solution ↗03Recover after continuity loss
- Invalidate stale callback generations.
- Draw the snapshot/high-water/live-tail handoff and name the overflow restart boundary.
Deliverable: A recovery trace ending at109,total24.
Practice prompt ↗Worked solution ↗04Bound slow consumers
- Test two pending records followed by overflow.
- Add a design for gap deadlines, byte limits and fairness; state what is modeled versus deferred.
Deliverable: Backpressure policy and explicit limitations.
Practice prompt ↗Practice prompt ↗Worked solution ↗05Revisit SQL and usage drills
- Retain the existing quota, usage and migration practice.
- Explain tenant isolation, exact arithmetic and application/cursor transaction boundaries.
Deliverable: One transferable database exercise reviewed.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Worked solution ↗Worked solution ↗06Explain conceptual and project choices
- Answer RNG mechanics with seed/state, security and rejection sampling.
- Rehearse a real project story and one tradeoff with evidence.
Deliverable: Specific conceptual answers and honest project notes.
Practice prompt ↗Practice prompt ↗Practice prompt ↗07Defend a changed requirement
- Run a mock that changes retention or consumer speed.
- Review where the guarantees changed and rerun the weakest case.
Deliverable: A changed design and precise follow-up tasks.
Practice prompt ↗Practice prompt ↗Worked solution ↗Worked solution ↗Expand any day for tasks and deliverables. Your progress is saved on this device.
Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.
Estimate a tenant-leading index migration you have never run
Someone needs a date. usage_event carries an index on (occurred_at) and needs (tenant_id, occurred_at); the largest tenant holds roughly a hundred times the median tenant's rows, the table is partitioned daily with years of retention, and you have never run a migration on a table this large. Give an estimate you would defend: how you decompose the work, the two or three numbers you would go and measure first, the range and confidence you state, and what you commit to when the person asking needs a single date today.
Approach
- Refuse the bare number and then give one anyway, in the form that is actually useful: a range plus the measurement that collapses it. 'Four to eleven days; one afternoon building this index on a restored copy of the largest partition takes that to within a day' is an answer, while 'it depends' is not.
- Decompose by failure mode rather than into equal chunks, because that is where estimates go wrong. On a partitioned parent you create the index ON ONLY the parent, build each partition's index with CREATE INDEX CONCURRENTLY, then ALTER INDEX ... ATTACH PARTITION, at which point the parent index becomes valid. CONCURRENTLY does not block writes but scans each partition twice, waits out older transactions, cannot run inside a transaction block, and on failure leaves an invalid index you must drop concurrently and retry.
- Name the two unknowns that dominate and price them: build time on one restored partition of realistic size, and whether the planner actually chooses the new index for the skewed tenant, since selectivity for a tenant holding most of the rows is a different question from selectivity for the median tenant. Both are half-day measurements against a replica, and both are cheaper than being wrong by a week.
- State the assumptions the range is conditional on, because that is what makes a slip a re-estimate instead of a credibility event: no partition above a stated row count, one concurrent build at a time so it does not compete with ingest for I/O, and an ingest backlog that can absorb the added write amplification while both indexes exist.
- Budget the step nobody budgets: verification and the old index's removal. Dropping the old index is fast, but deciding it is safe to drop means confirming no plan still uses it, and that confirmation waits on real traffic across a full weekly cycle rather than on your patience.
- Answer the single-date request honestly. Commit to a date for the first checkpoint — the measured build number from the replica — and to re-estimating on that date, and say plainly what you are not committing to yet. A date with a scheduled re-estimate is worth more to the asker than a confident wrong one, and you should say why in those words.
Follow-up
- The concurrent build fails half way through the largest partition. What is the state of the database and what do you do next?
- Your estimate slips by sixty percent. Which assumption broke, and at what point would you have known?
- The person asking needs the date for a customer commitment. Does your answer change?
Argue against failing open when the control plane is unreachable
The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.
Approach
- Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
- Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
- Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
- Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
- State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
- Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.
Follow-up
- Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
- The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
- You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?
Disclose a cross-tenant webhook delivery to affected customers
An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.
Approach
- Bound the population before saying anything externally. The affected set is deliveries in the window where the event's tenant and the subscription's tenant differ; the ones that actually left are those with delivered_at set and a 2xx in last_response_code. Attempted and delivered are two different counts and a disclosure has to use the right one in the right sentence.
- Separate what the records prove from what they do not, and say both halves rather than the flattering one. They prove which payloads were signed, where they went, and — through payload_digest — exactly which bytes. They do not prove what the receiving system did with them, and they do not bound the window more precisely than your deploy timestamps do.
- Communicate on the facts you hold, with the scope stated as an upper bound: 'at most eleven payloads, four recipient endpoints, these fields, this window' is more useful and more honest than waiting a day for certainty. The field list matters more than the event count, because a customer cannot assess exposure from 'an event'.
- Name the code change precisely, because this class never originates in the delivery worker. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before the payload is signed, and make the second comparison drop the delivery rather than log a warning. Say why one check is insufficient: the enqueue check protects against the bug you know about, the pre-signing check protects the boundary itself.
- Run the history question in parallel and say so: a query over historical deliveries for the same mismatch tells you whether this was nineteen minutes or a year, and you would rather find the second case yourself than have a customer find it after your disclosure.
- Split the response into workstreams with owners — recipients asked to delete, affected customers notified, the check landed with a test, history swept — and say which you personally drove and which you handed off. Claiming all four is not credible and claiming none is not ownership.
Follow-up
- The historical sweep finds two more instances from last year. What changes in what you have already told people?
- Who approves the wording, and what do you do when you are asked to soften the scope?
- A customer asks you to prove a redelivery contained the same bytes as the original. What do you show them?
- 01
Someone needs a date. usage_event carries an index on (occurred_at) and needs (tenant_id, occurred_at); the largest tenant holds roughly a hundred times the median tenant's rows, the table is partitioned daily with years of retention, and you have never run a migration on a table this large. Give an estimate you would defend: how you decompose the work, the two or three numbers you would go and measure first, the range and confidence you state, and what you commit to when the person asking needs a single date today.
- 02
The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.
- 03
An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.
Is this an official Ably interview guide?
No. It is PracHub's own research and practice material for the Software Engineer role at Ably. Rounds and questions reflect what candidates have reported, not a process Ably has published, and they change over time. Confirm the current format and scope with your recruiter.
PracHub interview research ↗What should I practice for Ably software engineer interviews?
Start with realtime stream scope, reconnect continuity, gap and duplicate handling, backpressure and tenant isolation. The new worked model includes actual local tests. The existing SQL, usage and webhook exercises remain transferable practice. Match your depth to the stages recruiting confirms.
PracHub interview research ↗What is the best way to prepare for the take-home assignment?
Focus on clean code, thorough documentation, and clear communication of your design decisions. Candidates report that interviewers want to see how you think, not just the final result.
PracHub interview research ↗Does this guide establish a fixed Ably interview timeline?
No. Its stage descriptions are candidate-reported editorial material, not an employer-published schedule. Ask recruiting for your actual sequence, timing, assessment scope and allowed tools.
PracHub interview research ↗Does the worked model guarantee exactly-once delivery?
It applies a contiguous prefix once in a single-threaded in-memory model. It does not guarantee durable or external effects. A production design needs application state and cursor atomicity or idempotent effects, authorization, replay retention, and an explicit snapshot/live handoff.
PracHub interview research ↗Sources & methodology 7 sources ↗
No official company page is cited. Rounds and questions come from candidate reports and PracHub editorial material; each source shows the date it was read.
- 01PracHub interview research ↗
PracHub editorial research into this company and role, maintained with this guide. Candidate-reported, not an employer publication.
PracHub page · Accessed 2026-09-22 - 02PracHub Software Engineer practice ↗
Cross-company practice questions for this role.
PracHub page · Accessed 2026-09-22 - 03PracHub interview preparation framework ↗
The framework the preparation plan follows.
PracHub page · Accessed 2026-09-22 - 04Ably connection state and recovery ↗
Official technical documentation for practice context, not hiring-process evidence.
PracHub page · Accessed 2026-10-08 - 05Ably channel states ↗
Official technical documentation for practice context, not hiring-process evidence.
PracHub page · Accessed 2026-10-08 - 06Ably message history ↗
Official technical documentation for practice context, not hiring-process evidence.
PracHub page · Accessed 2026-10-08 - 07Python random documentation ↗
Official technical documentation for practice context, not hiring-process evidence.
PracHub page · Accessed 2026-10-08
