LM-Kit OneDocs2026.11.1lm-kit.comEULA
Search & RAG/The Search Engine

Syncing From S3 Buckets

Documents often already live in object storage: contracts in an AWS S3 bucket, scans dropped into MinIO, exports landing in Cloudflare R2. A source connects such a bucket to a Search collection. Each sync lists the bucket, indexes the objects that are new or changed, leaves the others alone and removes from the collection the documents whose object was deleted. Nothing is copied by hand and nothing is indexed twice.

Any store that speaks the S3 API works: AWS S3, MinIO, Cloudflare R2, Wasabi, Backblaze B2, Ceph, Scaleway and OVHcloud Object Storage. Requests are signed with AWS Signature Version 4.

Add a source#

In the admin panel, open Search services, pick the tenant, and click Sources on the collection the documents belong in. Add source asks for:

Field What to enter
Id A name for the source: lower-case letters, digits and dashes. It cannot change later.
Endpoint https://s3.<region>.amazonaws.com for AWS, http://minio:9000 for MinIO, the account endpoint for R2 or Wasabi.
Bucket The bucket's name.
Prefix The keys the source covers, such as contracts/2026/. Empty covers the whole bucket.
Region The region requests are signed for. us-east-1 suits most S3-compatible stores.
Sync every Minutes between two syncs. 0 syncs only when you click Sync now.
Access key id and secret A key allowed to list the bucket and read its objects.
Path-style addressing On for MinIO and most self-hosted stores (endpoint/bucket/key); off for the virtual-hosted style (bucket.endpoint/key).

Test connection lists the first objects under the prefix with these credentials and says how many are in a format the collection indexes, before anything is saved. The first sync starts as soon as the source is added.

The key needs read access only. On AWS, a policy granting s3:ListBucket on the bucket and s3:GetObject on its objects is enough:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::contracts-bucket" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::contracts-bucket/*" }
  ]
}

The secret is sealed at rest like every other secret of the server, never returned by the API and never shown again in the panel. To rotate it, edit the source and enter the new one; leaving the field empty keeps the stored secret.

What a sync does#

A sync compares the listing with what it indexed last time, object by object, using the version (the ETag) the store reports:

The object What happens
Appeared since the last sync Read and indexed.
Changed (new ETag) Read and indexed again, replacing the previous version.
Unchanged Left alone: not even downloaded.
Deleted from the bucket Its document is removed from the collection.
In a format the collection cannot index, or larger than the upload limit Skipped and counted.
Failed to read or index Counted with its reason; the next sync tries it again.

Every object is indexed through the same path as an upload: the tenant's OCR setting, parsing effort and embedding model apply, and office formats are converted first when LibreOffice is installed. Each document keeps where it came from: its source URI (s3://bucket/key), and the source_id and source_key metadata fields, returned with every hit.

A listing the store refuses (wrong key, missing bucket, unreachable endpoint) fails the run and changes nothing: the collection keeps what it has, and the run's error names the store's reason.

Watch the syncs#

The Sources list shows, for each source, how many objects it has indexed, the last run (when it ended, how many objects were added, updated, removed, skipped or failed) and the first error when there was one. Sync now starts a sync within half a minute; Remove stops syncing and leaves the indexed documents in the collection, where you delete them like any others.

The same operations are on the admin API, under /lmkit/v1/admin/clusters/search/{cluster}/tenants/{tenant}/collections/{collection}/sources: GET lists the sources and their last runs, POST adds one, PUT /{id} changes one, DELETE /{id} removes one, POST /{id}/sync asks for a sync, and POST /test checks credentials.

On a farm#

Sources are shared objects of the domain: added on one node, they are known to every node. One node runs the syncs (the search-sources job holds a lease, listed on Jobs). A long sync renews that lease and records its progress as it goes, so when its node stops, another node takes the job over and resumes where the bucket and the recorded versions say it should, without indexing again what was already done.