> ## Documentation Index
> Fetch the complete documentation index at: https://control.datarelay.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Collect Logs from S3 or MinIO

> Configure S3 object polling, filter event objects, and verify delivery into a test Destination.

# Collect Logs from S3 or MinIO

**Goal:** read a synthetic event file from an S3-compatible bucket and deliver its records through a Stream.

> [!NOTE]
>
> **Screen version:** The numbered images are historical Control UI test captures. Follow the current five-step flow (Connect → Sample & Record Selection → Destinations → Route Processing → Deploy), not older step ordering. See the [numbered screenshot guide](https://control.datarelay.run/use-cases/screen-walkthrough.md).

## End-to-end workflow and acceptance

1. **Prepare requirements:** consult the vendor's official API/integration guide for endpoint, authentication, least-privilege scope, pagination/checkpoint and sample response. Obtain sanitized test events and actual Destination connectivity information.
2. **Use ChatGPT for unknowns:** supply public documentation and redacted samples; request a checklist of Connector inputs, request parameters, event root, timestamp/ID, cursor and unknowns. **Never share secrets.** Validate the response with the vendor guide and Run Test.
3. **Create Connector:** Connectors → Create Connector, choose this page's source type, enter real connection/auth details, save and test connectivity.
4. **Create Destination:** Destinations → Create, choose SYSLOG\_UDP/TCP/TLS or WEBHOOK\_POST, and configure the actual receiver's host/port/URL, TLS and authentication.
5. **Create Stream:** Streams → Create → Connect, choose the saved Connector and configure source collection/request settings.
6. **Confirm samples and schema:** Sample & Record Selection → Run Test; verify extracted records, Record Root, Union Schema, pagination and checkpoint.
7. **Process events:** select Destination/Route; configure Mapping, timestamp, enrichment, Masking/Drop/Quarantine in Route Processing. Use Tokenization **only if the deployed product supports it**. Verify effective Route state and actual output.
8. **Deploy and prove delivery:** resolve readiness findings, start the Stream, and correlate Run ID, Route Logs and the **independent Destination receipt**. All three are required for PASS.
9. **Diagnose:** for connection/auth check API scope, TLS, network and 429; for zero records check record root/pagination; for failed delivery follow [Delivery Incident](https://control.datarelay.run/use-cases/delivery-incident.md); for changed fields follow [Schema Drift](https://control.datarelay.run/use-cases/schema-change.md).

### Suggested ChatGPT prompt

> I want to collect this vendor's logs using DataRelay Control. Based only on the public API guide and redacted JSON sample below, list required Connector connection/auth fields, API request/pagination, event root, timestamp/ID/checkpoint, mapping and sensitive-field protection, and failure diagnostics. Mark undocumented information 'needs confirmation' rather than inventing values.

## 1. Prepare a controlled object

Create a test bucket in MinIO or AWS S3 and upload a file called `security/tutorial.jsonl` containing:

```json theme={null}
{"id":"s3-001","message":"object event one","severity":"low"}
```

Your Control runtime must be able to reach the endpoint. Use non-production access keys with read/list permissions only; avoid embedding them in package files.

## 2. Create the Connector

1. Go to **Connectors → Create Connector → S3 Object Polling (MinIO / AWS-compatible)**.
2. Enter **Connector Name** = `Lab S3 Events`.
3. Under **S3 endpoint & bucket**, enter the actual **Endpoint URL**, **Bucket**, and **Region**.
4. Set **Prefix** = `security/` and **Object key pattern** = `*.jsonl` as a candidate filter. Adjust to your actual object paths.
5. Enter **Access key** and **Secret key** in protected Connector inputs. For MinIO use path-style addressing where appropriate. Enable HTTPS/SSL with the correct endpoint/certificates in secured environments.
6. Save and run the supported S3 connectivity/list test. Confirm the object is visible; do not assume a successful credential test implies it matches Prefix/Pattern.

## 3. Create the Stream

1. In `/streams/new`, choose `Lab S3 Events` under **Connect**.
2. In **Sample & Record Selection**, run a sample test and confirm the event is parsed as an individual JSON record.
3. Verify Event Root and schema; choose a supported checkpoint/object progression strategy appropriate to the source.
4. Select the [test HTTP Destination](https://control.datarelay.run/use-cases/lab-setup.md), review Route Processing and Deploy readiness, then start the Stream.
5. Check **Logs** plus the independent HTTP receiver file for `s3-001`.

**PASS:** the expected object is discovered, a record is extracted, and the test Destination receives the corresponding event.

![record-selection — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/record-selection.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=196de1ecaf541296095434008afc05ad)

*Figure 1. **Choosing record path and checkpoint.** ① Review the suggested Record Path, ② inspect actual fields in JSON Tree, ③ select the Checkpoint Field. This capture shows the not-yet-confirmed state.*

![stream-actions — numbered Control historical UI capture](https://mintcdn.com/datarelay-control/Tl9_UibGAprPlCr8/images/screenshots/annotated/stream-actions.webp?fit=max\&auto=format\&n=Tl9_UibGAprPlCr8\&q=85\&s=f5462ebde85e05d0c95ab32096a1a24e)

*Figure 2. **Start and verify a newly created Stream.** ① Review Stream creation, ② use Start Stream only when runtime is available, ③ open View logs to verify execution. This historical capture says Runtime API unavailable.*

## If it fails

* Empty listing: verify bucket/Prefix/Object key pattern and read/list authorization.
* TLS/endpoint failure: inspect actual MinIO/AWS regional URL and runtime network.
* Object fetched but zero events: check file structure (JSONL vs array/object) and Record Selection.
* Unexpected repeated files: inspect checkpoint/object identity and retry behavior.

**Related:** [First flow](https://control.datarelay.run/use-cases/webhook-first-flow.md), [Schema change](https://control.datarelay.run/use-cases/schema-change.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.