> ## Documentation Index
> Fetch the complete documentation index at: https://docs.autosage.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Document ingestion

> Upload documents, understand the processing pipeline, and track status through to indexing.

Ingestion turns a raw file into searchable, grounded knowledge. This page covers the developer flow — requesting an upload URL, uploading bytes, and polling status — plus what happens in the background pipeline and how quotas apply.

## Upload flow

Uploads use a two-step presigned pattern so file bytes go straight to object storage, and processing begins automatically once the file lands.

<Steps>
  <Step title="Request an upload URL">
    `POST /documents/presign` returns a `document_id`, an `upload_url`, and an expiry.
  </Step>

  <Step title="Upload the file">
    `PUT` the raw file bytes to `upload_url` before it expires.
  </Step>

  <Step title="Processing starts automatically">
    Once the bytes land, the background pipeline begins on its own — no extra call needed.
  </Step>

  <Step title="Poll status">
    `GET /documents/status?document_id=...` until the status is `processed`.
  </Step>
</Steps>

```bash theme={null}
# 1. Request an upload URL
curl -X POST "$AUTOSAGE_URL/documents/presign" \
  -H "Authorization: Bearer $AUTOSAGE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "kb_id": "KB_ID",
    "tenant_id": "TENANT_ID",
    "filename": "handbook.pdf",
    "mime_type": "application/pdf",
    "size_bytes": 482913
  }'

# 2. Upload the bytes to the returned upload_url
curl -X PUT "UPLOAD_URL" \
  -H "Content-Type: application/pdf" \
  --data-binary @handbook.pdf

# 3. Poll until processed
curl "$AUTOSAGE_URL/documents/status?document_id=DOCUMENT_ID" \
  -H "Authorization: Bearer $AUTOSAGE_KEY"
```

<Info>
  The `upload_url` is time-limited. Upload the file promptly after presigning; if the URL expires before you upload, request a new one.
</Info>

## The processing pipeline

Once a file lands, it moves through a background pipeline that prepares it for grounded retrieval. Each stage runs on background workers, so large files process without blocking your integration.

<Steps>
  <Step title="Parse">
    Text is extracted from the source. PDFs — including **scanned documents via OCR** — audio files (**transcribed to text**), and Office and structured documents are all supported.
  </Step>

  <Step title="Summarize">
    A concise summary is generated so agents can see at a glance what the document covers.
  </Step>

  <Step title="Chunk">
    Content is split into **content-aware** chunks that respect the document's structure, which improves retrieval quality.
  </Step>

  <Step title="Embed">
    Each chunk is embedded into a vector representation for semantic search.
  </Step>

  <Step title="Index">
    Chunks are indexed into the knowledge base's **private vector partition**, isolated from every other knowledge base.
  </Step>
</Steps>

## Status values

The `status` field tracks a document through ingestion:

| Status           | Meaning                                                   |
| ---------------- | --------------------------------------------------------- |
| `pending_upload` | Presigned; waiting for the file bytes to be uploaded.     |
| `queued`         | Bytes received; queued for processing.                    |
| `processing`     | Moving through parse → summarize → chunk → embed → index. |
| `processed`      | Fully indexed and ready to query.                         |
| `failed`         | Processing did not complete.                              |

<Tip>
  Wait until a document is `processed` before querying against it — chunks are searchable only after indexing completes. If a document ends up `failed`, or you abandon an upload at `pending_upload`, remove it: incomplete documents still count toward your quota.
</Tip>

## Other ingestion methods

Beyond file uploads, you can bring in web content and manage existing documents:

<AccordionGroup>
  <Accordion title="Scrape a single URL" icon="link">
    `POST /documents/upload-url` fetches a URL and ingests its readable text as a document.
  </Accordion>

  <Accordion title="Discover site URLs" icon="sitemap">
    `POST /documents/upload-url-site` discovers URLs from a site. It does not ingest those pages; submit each selected URL separately to `POST /documents/upload-url`.
  </Accordion>

  <Accordion title="Reprocess" icon="rotate">
    `POST /documents/:id/reprocess` runs a document through the pipeline again — useful after a `failed` status.
  </Accordion>

  <Accordion title="Download" icon="download">
    `GET /documents/:id/download` returns a presigned URL to retrieve the original file.
  </Accordion>

  <Accordion title="Delete" icon="trash">
    `DELETE /documents/:id` soft-removes the document from search and cleans up its stored bytes and index entries.
  </Accordion>
</AccordionGroup>

## Quotas

Uploads are checked against your tenant's limits before they are accepted:

* **Document count** — the number of documents the tenant may hold.
* **Storage** — the total bytes the tenant may store.

When a limit is exceeded, the API returns a clear `403` so you can surface the reason and prompt the user to free up space.

<Warning>
  `failed` and abandoned `pending_upload` documents count toward your document and storage quotas until removed. Clean them up as part of your ingestion flow to avoid hitting limits unexpectedly.
</Warning>

## Endpoint summary

| Action                  | Method & path                           |
| ----------------------- | --------------------------------------- |
| Request an upload URL   | `POST /documents/presign`               |
| Upload file bytes       | `PUT <upload_url>`                      |
| Check processing status | `GET /documents/status?document_id=...` |
| Scrape a single URL     | `POST /documents/upload-url`            |
| Discover site URLs      | `POST /documents/upload-url-site`       |
| Reprocess a document    | `POST /documents/:id/reprocess`         |
| Download original file  | `GET /documents/:id/download`           |
| Delete a document       | `DELETE /documents/:id`                 |

## Next steps

<CardGroup cols={2}>
  <Card title="Documents in a knowledge base" icon="book" href="/knowledge-bases/documents">
    How documents power the RAG corpus.
  </Card>

  <Card title="Documents API" icon="code" href="/developers/api/documents">
    Full request and response schemas.
  </Card>

  <Card title="Retrieval & grounding" icon="magnifying-glass" href="/developers/lifecycles/rag">
    How indexed chunks become cited answers.
  </Card>

  <Card title="Quickstart" icon="rocket" href="/developers/quickstart">
    Upload a document and run a chat end to end.
  </Card>
</CardGroup>