> ## Documentation Index
> Fetch the complete documentation index at: https://control.datarelay.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect Files over SFTP

> Configure secure remote-file polling and verify events extracted from a remote JSONL file.

# Collect Files over SFTP

**Goal:** collect events from a test file on an SSH/SFTP server. SFTP-compatible SCP mode is a specific supported option; it is **not** arbitrary legacy SCP directory polling.

> [!NOTE]
>
> **Screen version:** The numbered images are historical Control UI test captures. Follow the current five-step flow (Connect → Sample & Record Selection → Destinations → Route Processing → Deploy), not older step ordering. See the [numbered screenshot guide](https://control.datarelay.run/use-cases/screen-walkthrough.md).

## End-to-end workflow and acceptance

1. **Prepare requirements:** consult the vendor's official API/integration guide for endpoint, authentication, least-privilege scope, pagination/checkpoint and sample response. Obtain sanitized test events and actual Destination connectivity information.
2. **Use ChatGPT for unknowns:** supply public documentation and redacted samples; request a checklist of Connector inputs, request parameters, event root, timestamp/ID, cursor and unknowns. **Never share secrets.** Validate the response with the vendor guide and Run Test.
3. **Create Connector:** Connectors → Create Connector, choose this page's source type, enter real connection/auth details, save and test connectivity.
4. **Create Destination:** Destinations → Create, choose SYSLOG\_UDP/TCP/TLS or WEBHOOK\_POST, and configure the actual receiver's host/port/URL, TLS and authentication.
5. **Create Stream:** Streams → Create → Connect, choose the saved Connector and configure source collection/request settings.
6. **Confirm samples and schema:** Sample & Record Selection → Run Test; verify extracted records, Record Root, Union Schema, pagination and checkpoint.
7. **Process events:** select Destination/Route; configure Mapping, timestamp, enrichment, Masking/Drop/Quarantine in Route Processing. Use Tokenization **only if the deployed product supports it**. Verify effective Route state and actual output.
8. **Deploy and prove delivery:** resolve readiness findings, start the Stream, and correlate Run ID, Route Logs and the **independent Destination receipt**. All three are required for PASS.
9. **Diagnose:** for connection/auth check API scope, TLS, network and 429; for zero records check record root/pagination; for failed delivery follow [Delivery Incident](https://control.datarelay.run/use-cases/delivery-incident.md); for changed fields follow [Schema Drift](https://control.datarelay.run/use-cases/schema-change.md).

### Suggested ChatGPT prompt

> I want to collect this vendor's logs using DataRelay Control. Based only on the public API guide and redacted JSON sample below, list required Connector connection/auth fields, API request/pagination, event root, timestamp/ID/checkpoint, mapping and sensitive-field protection, and failure diagnostics. Mark undocumented information 'needs confirmation' rather than inventing values.

## 1. Prepare the remote host

1. On a dedicated test SSH server, create an allowed directory such as `/srv/control-demo` and place `events.jsonl` containing:

```json theme={null}
{"id":"sftp-001","message":"remote file record","severity":"low"}
```

2. Create a least-privilege read-only SSH user. Confirm the account can list and read the file using your normal `sftp` client.
3. Record the trusted server SSH host key fingerprint. Do **not** disable host verification to make a test pass.

## 2. Configure Remote File Connector

1. Choose **Connectors → Create Connector → Remote file polling (SFTP / SFTP-compatible SCP mode)**.
2. Set **Connector Name** = `Lab SFTP`, **Protocol** = `sftp`.
3. Fill in the remote **Host**, **Port** (usually 22), and **Username**.
4. Provide a permitted **Password or Private key** as supported by the form. Configure the server identity through **known-hosts** using the trusted fingerprint; choose the **strict** host-key verification policy.
5. Configure remote directory/file selection (and file pattern where offered) for `/srv/control-demo/events.jsonl`. Save, then run the supported remote connectivity/list test.

## 3. Test records and deliver

1. Create a Stream in `/streams/new`, choose `Lab SFTP`, and **Run Test**.
2. Verify a parsed record with `id=sftp-001`; use **Record Selection** to confirm the event structure.
3. Choose the [test Destination](https://control.datarelay.run/use-cases/lab-setup.md), review Route Processing and Deploy settings, and start the Stream.
4. In **Logs**, correlate the run with the independent Destination receipt.

**PASS:** SFTP listing/read succeeds, a sample record is extracted, and an independent receiver contains the corresponding event.

![record-selection — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/record-selection.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=196de1ecaf541296095434008afc05ad)

*Figure 1. **Choosing record path and checkpoint.** ① Review the suggested Record Path, ② inspect actual fields in JSON Tree, ③ select the Checkpoint Field. This capture shows the not-yet-confirmed state.*

![stream-actions — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/stream-actions.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=f5462ebde85e05d0c95ab32096a1a24e)

*Figure 2. **Start and verify a newly created Stream.** ① Review Stream creation, ② use Start Stream only when runtime is available, ③ open View logs to verify execution. This historical capture says Runtime API unavailable.*

## If it fails

* Host verification error: fix the recorded host key and trust chain. Never switch to insecure verification to make production pass.
* Permission denied: verify the SSH account's file/directory permissions.
* No extracted records: inspect remote file encoding/format and Record Selection.
* Duplicate records: inspect source checkpoint/position and file rotation strategy.

**Related:** [S3 collection](https://control.datarelay.run/use-cases/s3-object-collection.md), [Delivery incident](https://control.datarelay.run/use-cases/delivery-incident.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.