Skip to main content

​
Collect Logs from S3 or MinIO

Goal: read a synthetic event file from an S3-compatible bucket and deliver its records through a Stream.
Screen version: The numbered images are historical Control UI test captures. Follow the current five-step flow (Connect → Sample & Record Selection → Destinations → Route Processing → Deploy), not older step ordering. See the numbered screenshot guide.

​
End-to-end workflow and acceptance

  1. Prepare requirements: consult the vendor’s official API/integration guide for endpoint, authentication, least-privilege scope, pagination/checkpoint and sample response. Obtain sanitized test events and actual Destination connectivity information.
  2. Use ChatGPT for unknowns: supply public documentation and redacted samples; request a checklist of Connector inputs, request parameters, event root, timestamp/ID, cursor and unknowns. Never share secrets. Validate the response with the vendor guide and Run Test.
  3. Create Connector: Connectors → Create Connector, choose this page’s source type, enter real connection/auth details, save and test connectivity.
  4. Create Destination: Destinations → Create, choose SYSLOG_UDP/TCP/TLS or WEBHOOK_POST, and configure the actual receiver’s host/port/URL, TLS and authentication.
  5. Create Stream: Streams → Create → Connect, choose the saved Connector and configure source collection/request settings.
  6. Confirm samples and schema: Sample & Record Selection → Run Test; verify extracted records, Record Root, Union Schema, pagination and checkpoint.
  7. Process events: select Destination/Route; configure Mapping, timestamp, enrichment, Masking/Drop/Quarantine in Route Processing. Use Tokenization only if the deployed product supports it. Verify effective Route state and actual output.
  8. Deploy and prove delivery: resolve readiness findings, start the Stream, and correlate Run ID, Route Logs and the independent Destination receipt. All three are required for PASS.
  9. Diagnose: for connection/auth check API scope, TLS, network and 429; for zero records check record root/pagination; for failed delivery follow Delivery Incident; for changed fields follow Schema Drift.

​
Suggested ChatGPT prompt

I want to collect this vendor’s logs using DataRelay Control. Based only on the public API guide and redacted JSON sample below, list required Connector connection/auth fields, API request/pagination, event root, timestamp/ID/checkpoint, mapping and sensitive-field protection, and failure diagnostics. Mark undocumented information ‘needs confirmation’ rather than inventing values.

​
1. Prepare a controlled object

Create a test bucket in MinIO or AWS S3 and upload a file called security/tutorial.jsonl containing:
Your Control runtime must be able to reach the endpoint. Use non-production access keys with read/list permissions only; avoid embedding them in package files.

​
2. Create the Connector

  1. Go to Connectors → Create Connector → S3 Object Polling (MinIO / AWS-compatible).
  2. Enter Connector Name = Lab S3 Events.
  3. Under S3 endpoint & bucket, enter the actual Endpoint URL, Bucket, and Region.
  4. Set Prefix = security/ and Object key pattern = *.jsonl as a candidate filter. Adjust to your actual object paths.
  5. Enter Access key and Secret key in protected Connector inputs. For MinIO use path-style addressing where appropriate. Enable HTTPS/SSL with the correct endpoint/certificates in secured environments.
  6. Save and run the supported S3 connectivity/list test. Confirm the object is visible; do not assume a successful credential test implies it matches Prefix/Pattern.

​
3. Create the Stream

  1. In /streams/new, choose Lab S3 Events under Connect.
  2. In Sample & Record Selection, run a sample test and confirm the event is parsed as an individual JSON record.
  3. Verify Event Root and schema; choose a supported checkpoint/object progression strategy appropriate to the source.
  4. Select the test HTTP Destination, review Route Processing and Deploy readiness, then start the Stream.
  5. Check Logs plus the independent HTTP receiver file for s3-001.
PASS: the expected object is discovered, a record is extracted, and the test Destination receives the corresponding event. record-selection — numbered Control historical UI capture Figure 1. Choosing record path and checkpoint. ① Review the suggested Record Path, ② inspect actual fields in JSON Tree, ③ select the Checkpoint Field. This capture shows the not-yet-confirmed state. stream-actions — numbered Control historical UI capture Figure 2. Start and verify a newly created Stream. ① Review Stream creation, ② use Start Stream only when runtime is available, ③ open View logs to verify execution. This historical capture says Runtime API unavailable.

​
If it fails

  • Empty listing: verify bucket/Prefix/Object key pattern and read/list authorization.
  • TLS/endpoint failure: inspect actual MinIO/AWS regional URL and runtime network.
  • Object fetched but zero events: check file structure (JSONL vs array/object) and Record Selection.
  • Unexpected repeated files: inspect checkpoint/object identity and retry behavior.
Related: First flow, Schema change.
Last modified on October 18, 2026