GitHub API Integration: Authentication, Pagination, and Webhook Recovery

A GitHub API integration needs a clear identity, a complete data retrieval path, and a recovery process for missed events. A successful request to one public repository proves very little about access to a customer's private repositories.

This guide develops a read-only repository issue mirror. The local example checks webhook signatures without contacting GitHub or changing any repository. Authentication setup, pagination, durable processing, and reconciliation remain separate responsibilities.

For an interview exercise, use the public paginated API ingestion prompt. Its upstream API differs from GitHub's, so explain which pagination and retry rules belong to the particular service.

1. Choose the principal before choosing the token

Start by writing down whose authority each operation needs. A service that maintains an organization's repository mirror usually needs installation-scoped access. A feature that performs an action for a signed-in person needs that user's authorization.

GitHub's app selection guide generally favors GitHub Apps. They support both installation and user authorization. Acting for a user therefore does not automatically require an OAuth app.

For the issue mirror, request only the permissions its endpoints need and install the app on selected test repositories. Consult the endpoint reference rather than guessing from a token's name. With installation authentication, an app JWT obtains a token for a particular installation; that installation token authenticates subsequent repository requests and expires after one hour.

Store private keys and tokens securely, refresh tokens before expiry, and never log them. Keep installation credentials separated by tenant. A repository name supplied by a caller is not proof that the caller owns that installation.

If an OAuth app is required, follow its authorization flow. Bind the callback to the initiating session with validated state, use PKCE (an S256 code challenge and its verifier), exchange the code on the server, and handle the token's actual expiration behavior.

2. Define the dataset and follow every page

For a small mirror, REST makes individual resource operations easy to inspect. GraphQL can select nested relationships in one query, but each connection still needs pagination and the response can contain errors alongside partial data. Its point and query limits are separate from REST's primary request budget. One HTTP response does not imply one cheap or complete query.

A first REST request for the mirror has this shape. OWNER, REPO, and the token are placeholders; do not paste a real credential into a shared shell transcript.

GET /repos/OWNER/REPO/issues?state=all&per_page=100 HTTP/1.1
Host: api.github.com
Accept: application/vnd.github+json
Authorization: Bearer INSTALLATION_ACCESS_TOKEN
X-GitHub-Api-Version: 2026-03-10
User-Agent: issue-mirror-example

The pinned version shown here appears in GitHub's current versioning documentation. Omitting the header currently selects the documented default, 2022-11-28; send the header explicitly so a later change of default cannot silently change your client's behavior. Pin and test the version your integration supports, then review changes before upgrading.

Follow the rel="next" URL in the response's Link header until it is absent, as the pagination guide explains. Use a maintained parser or SDK paginator. Do not infer completion from an arbitrary item count. Restrict authenticated requests to trusted API destinations, including redirect handling, so credentials cannot follow an untrusted URL.

The repository issues endpoint can return pull requests as well as issues. For an issue-only mirror, exclude entries containing pull_request. Store stable API identifiers and repository identity instead of parsing identifiers out of display URLs.

Consider a synthetic two-page response. Page one contains issue IDs 101 and 102; page two contains 102 and 103 because the underlying collection changed. Upserting by stable ID produces three records. Appending each response blindly produces four rows. Conversely, deduplication cannot repair an item skipped while a mutable collection shifted between pages.

Commit each processed batch before its checkpoint. If the process fails after data commit but before checkpoint commit, replaying the batch must be safe. A saved page URL is a restart aid, not a permanent snapshot. Use overlapping incremental windows where supported and periodic reconciliation; never infer deletion merely because an incremental batch omitted an item.

3. Budget requests and classify failures

Use response headers and the current REST rate-limit reference, rather than treating every token as a fresh quota. Unauthenticated public reads normally allow 60 requests per hour per IP. User authentication normally shares a 5,000-request hourly budget across requests made for that user. Installation limits can scale, and Enterprise Cloud exceptions differ. Search and other resource buckets can impose additional constraints.

Primary capacity does not remove secondary limits. A conservative queue, bounded work, and serial requests are a sensible starting point. Increasing concurrency until an advertised ceiling is reached is not a capacity plan.

For a confirmed rate-limit response, honor retry-after; when remaining quota is zero, wait until the reset time as well. Otherwise use at least a minute before retrying a secondary-limit failure, then bounded exponential backoff. GitHub's REST best practices also describes authenticated conditional GETs: a qualifying 304 avoids primary quota usage. Cache the representation together with its validator; do not parse a 304 as a new JSON document.

Not every 403 is throttling. Inspect the error and relevant headers. GitHub can also return 404 for a private resource when authentication or authorization is insufficient. Its troubleshooting reference separates permission, installation, and resource-access failures. Repeatedly retrying with the same insufficient permission will not fix them.

For reads, apply timeouts and bounded recovery for transient failures. For a write with an uncertain outcome, establish whether it happened before blindly repeating it; an issue creation or comment is a side effect. Log sanitized request IDs, status, installation identity, latency, and quota state without private payloads or credentials.

Practice scheduling work under an external budget with the public third-party API data-puller question.

4. Verify webhook bytes before accepting work

Webhooks provide timely changes while reconciliation repairs gaps. Keep the HTTP receiver small: authenticate the original bytes, validate the expected installation and supported payload shape, persist recoverable work, and acknowledge successful acceptance.

The following complete Python example checks the SHA-256 HMAC format used by GitHub. Save it as github_signature_demo.py and run it with Python 3. Its main block uses the public test vector from GitHub's signature validation reference; that sample secret is for testing only.

import hashlib
import hmac
import re

def valid_signature(raw_body, signature, secret):
    if not isinstance(raw_body, bytes):
        raise TypeError("preserve the original request bytes")
    if not isinstance(secret, bytes) or not secret:
        raise ValueError("a configured byte secret is required")
    if not isinstance(signature, str):
        return False
    if re.fullmatch(r"sha256=[0-9a-f]{64}", signature) is None:
        return False
    expected = hmac.new(secret, raw_body, hashlib.sha256).digest()
    supplied = bytes.fromhex(signature[7:])
    return hmac.compare_digest(expected, supplied)

if __name__ == "__main__":
    body = b"Hello, World!"
    signature = (
        "sha256="
        "757107ea0eb2509fc211221cce984b8a3"
        "7570b6d7586c22c46f4379c8b043e17"
    )
    print(valid_signature(body, signature, b"It's a Secret to Everybody"))
    print(valid_signature(body + b"!", signature, b"It's a Secret to Everybody"))

The output is True, then False. Signature verification does not establish that the body is valid JSON or that your application supports its event schema. Apply an appropriate request-size limit in the HTTP layer, preserve the raw bytes, then parse and validate only after authentication. This helper is not an HTTP server, queue, or complete replay defense.

GitHub's HMAC authenticates the body, not arbitrary HTTP routing headers. Do not authorize a tenant solely from an unchecked header. Match authenticated payload context against the receiver's expected installation and permissions. A valid signature also does not establish freshness: a captured valid body and signature can be replayed. Header delivery-ID deduplication alone cannot prevent a malicious replay that changes that unsigned header. Business operations must remain idempotent and constrained by their authorized resource and current state.

5. Make acknowledgement and recovery honest

GitHub's webhook best practices calls for a successful response within ten seconds and recommends asynchronous processing. Acknowledge after durable acceptance. Starting a background thread after returning success leaves a crash window in which accepted work disappears.

Use a durable inbox with a uniqueness constraint scoped to the receiver and delivery ID. Store the authenticated payload and processing status together. The worker must retry pending or failed processing rather than treating an existing receipt as completed business work. Retention and access controls should reflect the private repository data that an inbox may contain.

Here is a synthetic failure sequence for the issue mirror:

1. Delivery D1 changes issue 101. The receiver commits its inbox row, then responds successfully.
2. The worker updates the mirror but crashes before marking D1 complete. A retry safely upserts issue 101 again.
3. A requested redelivery of D1 reuses its delivery ID. The inbox recognizes it without creating another work item.
4. An older update arrives later. Serialize work per resource or use a guarded reconciliation protocol so a stale worker cannot overwrite newer state. Fetching current state helps, but concurrent fetches can still finish in the wrong order.

An inbox and a database upsert do not make external emails or comments happen exactly once. Give any downstream side effect its own recoverable business identity and outcome tracking.

Crucially, GitHub does not automatically redeliver failed webhooks. Build an explicit recovery process using authorized redelivery mechanisms and scheduled canonical reads. Redelivery covers only deliveries from the past 3 days, so an outage longer than that must be repaired by canonical reads rather than redelivery. Removed repository access also needs an explicit policy, rather than interpreting an authorization failure as an instruction to delete all local data.

The public webhook and metric aggregation prompt gives a broader delivery-system exercise. Use the system architecture article to discuss the receiver, inbox, workers, and reconciliation boundaries.

6. Test the boundaries before widening access

Use recorded synthetic fixtures for deterministic local tests. Keep a separate sandbox installation exercise for real authentication, endpoint permissions, pagination, and webhook delivery. Local signature tests do not verify those external systems.

Monitor inbox age, failed processing, reconciliation lag, authentication failures, and request budgets. A low HTTP error rate can coexist with a stalled worker. Define what complete synchronization means for your chosen dataset, record a checkpoint only after durable progress, and alert when that progress stops.

For more preparation, the REST API interview article covers HTTP semantics and safe retries. The inspection-in-testing article offers a structured way to review these acceptance criteria before implementation.

Keep practicing

• Implement paginated API ingestion
• Design a rate-limited third-party API data puller
• Design webhooks and metric aggregation
• REST API questions with HTTP examples