Skip to main content

​
Collect Files over SFTP

Goal: collect events from a test file on an SSH/SFTP server. SFTP-compatible SCP mode is a specific supported option; it is not arbitrary legacy SCP directory polling.
Screen version: The numbered images are historical Control UI test captures. Follow the current five-step flow (Connect → Sample & Record Selection → Destinations → Route Processing → Deploy), not older step ordering. See the numbered screenshot guide.

​
End-to-end workflow and acceptance

  1. Prepare requirements: consult the vendor’s official API/integration guide for endpoint, authentication, least-privilege scope, pagination/checkpoint and sample response. Obtain sanitized test events and actual Destination connectivity information.
  2. Use ChatGPT for unknowns: supply public documentation and redacted samples; request a checklist of Connector inputs, request parameters, event root, timestamp/ID, cursor and unknowns. Never share secrets. Validate the response with the vendor guide and Run Test.
  3. Create Connector: Connectors → Create Connector, choose this page’s source type, enter real connection/auth details, save and test connectivity.
  4. Create Destination: Destinations → Create, choose SYSLOG_UDP/TCP/TLS or WEBHOOK_POST, and configure the actual receiver’s host/port/URL, TLS and authentication.
  5. Create Stream: Streams → Create → Connect, choose the saved Connector and configure source collection/request settings.
  6. Confirm samples and schema: Sample & Record Selection → Run Test; verify extracted records, Record Root, Union Schema, pagination and checkpoint.
  7. Process events: select Destination/Route; configure Mapping, timestamp, enrichment, Masking/Drop/Quarantine in Route Processing. Use Tokenization only if the deployed product supports it. Verify effective Route state and actual output.
  8. Deploy and prove delivery: resolve readiness findings, start the Stream, and correlate Run ID, Route Logs and the independent Destination receipt. All three are required for PASS.
  9. Diagnose: for connection/auth check API scope, TLS, network and 429; for zero records check record root/pagination; for failed delivery follow Delivery Incident; for changed fields follow Schema Drift.

​
Suggested ChatGPT prompt

I want to collect this vendor’s logs using DataRelay Control. Based only on the public API guide and redacted JSON sample below, list required Connector connection/auth fields, API request/pagination, event root, timestamp/ID/checkpoint, mapping and sensitive-field protection, and failure diagnostics. Mark undocumented information ‘needs confirmation’ rather than inventing values.

​
1. Prepare the remote host

  1. On a dedicated test SSH server, create an allowed directory such as /srv/control-demo and place events.jsonl containing:
  1. Create a least-privilege read-only SSH user. Confirm the account can list and read the file using your normal sftp client.
  2. Record the trusted server SSH host key fingerprint. Do not disable host verification to make a test pass.

​
2. Configure Remote File Connector

  1. Choose Connectors → Create Connector → Remote file polling (SFTP / SFTP-compatible SCP mode).
  2. Set Connector Name = Lab SFTP, Protocol = sftp.
  3. Fill in the remote Host, Port (usually 22), and Username.
  4. Provide a permitted Password or Private key as supported by the form. Configure the server identity through known-hosts using the trusted fingerprint; choose the strict host-key verification policy.
  5. Configure remote directory/file selection (and file pattern where offered) for /srv/control-demo/events.jsonl. Save, then run the supported remote connectivity/list test.

​
3. Test records and deliver

  1. Create a Stream in /streams/new, choose Lab SFTP, and Run Test.
  2. Verify a parsed record with id=sftp-001; use Record Selection to confirm the event structure.
  3. Choose the test Destination, review Route Processing and Deploy settings, and start the Stream.
  4. In Logs, correlate the run with the independent Destination receipt.
PASS: SFTP listing/read succeeds, a sample record is extracted, and an independent receiver contains the corresponding event. record-selection — numbered Control historical UI capture Figure 1. Choosing record path and checkpoint. ① Review the suggested Record Path, ② inspect actual fields in JSON Tree, ③ select the Checkpoint Field. This capture shows the not-yet-confirmed state. stream-actions — numbered Control historical UI capture Figure 2. Start and verify a newly created Stream. ① Review Stream creation, ② use Start Stream only when runtime is available, ③ open View logs to verify execution. This historical capture says Runtime API unavailable.

​
If it fails

  • Host verification error: fix the recorded host key and trust chain. Never switch to insecure verification to make production pass.
  • Permission denied: verify the SSH account’s file/directory permissions.
  • No extracted records: inspect remote file encoding/format and Record Selection.
  • Duplicate records: inspect source checkpoint/position and file rotation strategy.
Related: S3 collection, Delivery incident.
Last modified on October 18, 2026