> ## Documentation Index
> Fetch the complete documentation index at: https://control.datarelay.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect from an HTTP API

> Configure API polling, sample extraction, pagination and checkpoints before routing to a Destination.

# Collect from an HTTP API

**Goal:** collect a real vendor API's events through an existing HTTP API Connector and deliver them through a normal Stream/Route.

> [!NOTE]
>
> **Screen version:** The numbered images are historical Control UI test captures. Follow the current five-step flow (Connect → Sample & Record Selection → Destinations → Route Processing → Deploy), not older step ordering. See the [numbered screenshot guide](https://control.datarelay.run/use-cases/screen-walkthrough.md).

## End-to-end workflow and acceptance

1. **Prepare requirements:** consult the vendor's official API/integration guide for endpoint, authentication, least-privilege scope, pagination/checkpoint and sample response. Obtain sanitized test events and actual Destination connectivity information.
2. **Use ChatGPT for unknowns:** supply public documentation and redacted samples; request a checklist of Connector inputs, request parameters, event root, timestamp/ID, cursor and unknowns. **Never share secrets.** Validate the response with the vendor guide and Run Test.
3. **Create Connector:** Connectors → Create Connector, choose this page's source type, enter real connection/auth details, save and test connectivity.
4. **Create Destination:** Destinations → Create, choose SYSLOG\_UDP/TCP/TLS or WEBHOOK\_POST, and configure the actual receiver's host/port/URL, TLS and authentication.
5. **Create Stream:** Streams → Create → Connect, choose the saved Connector and configure source collection/request settings.
6. **Confirm samples and schema:** Sample & Record Selection → Run Test; verify extracted records, Record Root, Union Schema, pagination and checkpoint.
7. **Process events:** select Destination/Route; configure Mapping, timestamp, enrichment, Masking/Drop/Quarantine in Route Processing. Use Tokenization **only if the deployed product supports it**. Verify effective Route state and actual output.
8. **Deploy and prove delivery:** resolve readiness findings, start the Stream, and correlate Run ID, Route Logs and the **independent Destination receipt**. All three are required for PASS.
9. **Diagnose:** for connection/auth check API scope, TLS, network and 429; for zero records check record root/pagination; for failed delivery follow [Delivery Incident](https://control.datarelay.run/use-cases/delivery-incident.md); for changed fields follow [Schema Drift](https://control.datarelay.run/use-cases/schema-change.md).

### Suggested ChatGPT prompt

> I want to collect this vendor's logs using DataRelay Control. Based only on the public API guide and redacted JSON sample below, list required Connector connection/auth fields, API request/pagination, event root, timestamp/ID/checkpoint, mapping and sensitive-field protection, and failure diagnostics. Mark undocumented information 'needs confirmation' rather than inventing values.

## Required input from your API owner

Have an **actual reachable** base URL, API path and method, auth scheme, a sanitized representative response, and documentation describing pagination and the event cursor. The example values below are placeholders, not a working public API.

| Input | Example | Where to use |
| - | - | - |
| Base URL | `https://vendor.example.invalid` | Connector **Host / Base URL** |
| Endpoint | `/v1/audit/events` | request path in Connect/Run Test |
| Auth | API key/Bearer as documented | Credential/auth configuration |
| Event array | `$.events` | Record Selection, only if response truly uses it |
| Cursor field | `event_time` | Checkpoint, only if stable and monotonic for this API |

## 1. Register and test the Connector

1. Go to **Connectors → Create Connector** and choose **HTTP API Polling**.
2. Enter a meaningful **Connector Name** and the real **Host / Base URL**.
3. Select the actual supported authentication mode; store secrets using the product's credential configuration, not in exported samples or public YAML.
4. Save; run the Connector connectivity test when supported. Confirm the vendor API, TLS, network, and authorization are correct.

## 2. Choose the collection contract

1. Open `/streams/new`, choose the saved Connector in **Connect**, and enter the actual request path/method/query parameters.
2. Configure pagination according to vendor documentation (for example page, offset, or cursor); do not assume every endpoint supports the same pagination.
3. In **Sample & Record Selection → Run Test**, retrieve response data. Inspect HTTP status and actual JSON, not just a green connection indicator.
4. Switch to **Record Selection**. Select the array/record root that contains the events. If the response is `{"events":[...]}`, `$.events` is a candidate **only if the UI preview confirms the extracted records**.
5. Review the Union Schema: required fields, rarely appearing fields, and sensitive-field suggestions. The UI may suggest a record or checkpoint path; **you must confirm it**.
6. Configure checkpoint mode and source field using the vendor's ordering semantics. If no reliable cursor exists, use the supported collection method rather than inventing an incremental checkpoint.

![select-connector — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/select-connector.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=595bbed2cfdba81a0a246ed84116cc33)

*Figure 1. **Selecting an existing Connector.** ① Select your saved Connector (Generic REST is a test example). ② Continue to Sample. The Connector tab is already active.*

![run-test — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/run-test.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=b608f03405b24c6b130a215599571177)

*Figure 2. **Testing collection and record count.** ① Run Test → ② verify HTTP status and extracted record count → ③ open Record Selection. An HTTP success is not proof of Destination delivery.*

![record-selection — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/record-selection.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=196de1ecaf541296095434008afc05ad)

*Figure 3. **Choosing record path and checkpoint.** ① Review the suggested Record Path, ② inspect actual fields in JSON Tree, ③ select the Checkpoint Field. This capture shows the not-yet-confirmed state.*

## 3. Route and deploy

1. Select an existing Destination in **Destinations** (create a test sink first using [Lab setup](https://control.datarelay.run/use-cases/lab-setup.md) if necessary).
2. In **Route Processing**, review Transform mapping, optional Protection, and the output preview.
3. In **Deploy**, resolve blocking readiness findings. Deploy, start the Stream when required, and record its initial checkpoint.
4. Allow a real collection run and verify delivery logs **and** actual Destination receipts.

![field-mapping — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/field-mapping.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=15ae79d8a5249262f236f997f253d286)

*Figure 4. **Mapping an event to output fields.** ① Inspect the source sample, ② configure Field Mapping, ③ compare Final Event Preview. A preview does not prove real delivery.*

![stream-actions — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/stream-actions.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=f5462ebde85e05d0c95ab32096a1a24e)

*Figure 5. **Start and verify a newly created Stream.** ① Review Stream creation, ② use Start Stream only when runtime is available, ③ open View logs to verify execution. This historical capture says Runtime API unavailable.*

## 4. Verify safe incremental collection

* Record the run ID, extracted event IDs/time range, and checkpoint before the run.
* Run another collection cycle with no new vendor events. Check for unexpected duplicates; do not rely on a visual preview as a cursor correctness test.
* Add a new *synthetic* vendor event if the testing API permits it. Confirm extraction, delivery, and checkpoint advance **after** the required successful delivery.
* If the API uses unstable ordering or non-unique timestamps, resolve its pagination/cursor strategy before production rollout.

## If it fails

| Symptom | Inspect |
| - | - |
| 401/403 | Credential, auth mode, expired token, vendor permissions |
| 429 / timeout | Vendor rate limits, backoff, Connector/API Health |
| Test returns JSON but 0 records | Event array/record root, response envelope, request filters |
| Records repeat | Cursor/checkpoint and vendor ordering semantics |
| Records collected but not delivered | Route, Destination, protection/policy outcome, Logs |

**PASS =** a newly collected event is found in the independent Destination and its corresponding Route delivery record; no unexpected checkpoint advance on failed delivery.

**Related:** [Schema change](https://control.datarelay.run/use-cases/schema-change.md), [Delivery incident](https://control.datarelay.run/use-cases/delivery-incident.md).

![source-api-error — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/source-api-error.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=c66a05ef28da079c34e48a36ed217113)

*Figure 6. **HTTP 503 during Source API Test.** ① Read the HTTP 503 error, ② repair the source and Retry, ③ do not proceed while blocked. This is a Source API Test failure, not a Destination failure.*


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.