> ## Documentation Index
> Fetch the complete documentation index at: https://docs.soarlabs.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Add a URL

> Register web sources that should be crawled, cleaned, and indexed for retrieval.

## Overview

URLs let you ingest external web content without uploading files. Each URL is crawled, normalized, and processed through the ingestion pipeline to extract text, generate chunks, and create vector embeddings. Submit multiple URLs in a single request for efficient batch ingestion.

<Info>
  **Perfect for**: Documentation sites, blog posts, knowledge bases, status pages, and any web content you want to make queryable.
</Info>

<Warning>
  URL content is fetched at ingestion time. Changes to the source page won't automatically update - you'll need to re-add the URL to refresh content.
</Warning>

## Authentication

Requires valid JWT token or session authentication. You must be the owner of the target corpus.

## Request Body

<ParamField body="corpora" type="UUID" required>
  ID of the corpus that will own these URLs. Must be a corpus you created and have access to.
</ParamField>

<ParamField body="urls" type="array<object>" required>
  Array of URL objects to crawl and ingest. Each object represents one web resource.

  **Batch size recommendations:**

  * Optimal: 5-20 URLs per request
  * Maximum: Check your instance configuration (typically 50)

  <Expandable title="URL object structure">
    <ParamField body="urls[].url" type="string (URL)" required>
      Absolute URL to crawl (must include protocol: `https://` or `http://`).

      **Supported URL types:**

      * Public web pages (HTML)
      * Documentation sites
      * Blog posts and articles
      * Public APIs returning HTML/text
      * Sitemap URLs (when `scrape_sitemap` is enabled)

      **Unsupported:**

      * Authentication-required pages
      * JavaScript-heavy SPAs (limited support)
      * PDF files served directly (use file upload instead)
    </ParamField>

    <ParamField body="urls[].scrape_sitemap" type="boolean" default="false">
      Set to `true` to automatically discover and enqueue additional URLs from the page's sitemap.

      **How it works:**

      1. Soar Labs fetches the provided URL
      2. Searches for `sitemap.xml` or sitemap links
      3. Automatically enqueues all discovered URLs
      4. Processes each URL as a separate resource

      **Use cases:**

      * Documentation sites with full sitemaps
      * Blog archives
      * Product catalogs
      * Knowledge base articles

      **Caution**: Large sitemaps (>100 URLs) may take significant time to process. Consider adding specific sections instead of entire sites.
    </ParamField>
  </Expandable>
</ParamField>

## Example request

```bash theme={null}
curl -X POST https://{your-host}/api/data/urls/ \
  -H "Authorization: Bearer $SOAR_LABS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "corpora": "8d0f0a5d-4b5e-4c09-9db6-0e9d2aa8a9fd",
    "urls": [
      {"url": "https://docs.example.com/support/escalations"},
      {"url": "https://status.example.com/incidents", "scrape_sitemap": true}
    ]
  }'
```

## Response

Returns an array of URL objects (one for each submitted URL):

<ResponseField name="id" type="UUID">
  Unique identifier for the URL resource. Use for tracking, retrieval, or deletion.
</ResponseField>

<ResponseField name="created_at" type="timestamp">
  ISO 8601 timestamp when the URL was added.
</ResponseField>

<ResponseField name="updated_at" type="timestamp">
  Last update timestamp. Changes when crawling or indexing completes.
</ResponseField>

<ResponseField name="indexed_on" type="timestamp | null">
  Timestamp when indexing completed successfully. `null` while processing.
</ResponseField>

<ResponseField name="indexing_status" type="string">
  Current processing status:

  * `PRS` - Processing (crawling and indexing in progress)
  * `IND` - Indexed (content ready for queries)
  * `ERR` - Error (crawling or processing failed)
  * `PND` - Pending (queued for crawling)
</ResponseField>

<ResponseField name="url" type="string (URL)">
  The original URL as submitted.
</ResponseField>

<ResponseField name="clean_url" type="string (URL)">
  Normalized/canonical version of the URL (removes tracking parameters, standardizes format).
</ResponseField>

<ResponseField name="scrape_sitemap" type="boolean">
  Whether sitemap scraping was enabled for this URL.
</ResponseField>

<ResponseField name="metadata" type="object">
  Extracted metadata from the crawled page.

  <Expandable title="metadata fields">
    <ResponseField name="content_type" type="string">
      MIME type of the fetched content (e.g., `text/html`, `application/xhtml+xml`).
    </ResponseField>

    <ResponseField name="title" type="string">
      Page title extracted from `<title>` tag or Open Graph metadata.
    </ResponseField>

    Additional fields may include:

    * `description` - Meta description or OG description
    * `author` - Content author if available
    * `published_date` - Publication date if found
  </Expandable>
</ResponseField>

<ResponseField name="corpora" type="UUID">
  ID of the parent corpus containing this URL.
</ResponseField>

## Example Response

```json theme={null}
[
  {
    "id": "f0d6fe08-87c8-4eb0-80d8-7a2de638514b",
    "created_at": "2024-09-01T12:05:11.337Z",
    "updated_at": "2024-09-01T12:05:11.337Z",
    "indexed_on": null,
    "indexing_status": "PRS",
    "url": "https://docs.example.com/support/escalations",
    "clean_url": "https://docs.example.com/support/escalations",
    "scrape_sitemap": false,
    "metadata": {
      "content_type": "text/html",
      "title": "Escalation Playbook"
    },
    "corpora": "8d0f0a5d-4b5e-4c09-9db6-0e9d2aa8a9fd"
  }
]
```

## Important Notes

<Warning>
  **URL Validation**: Malformed URLs are rejected with `400 Bad Request` and field-level errors. Ensure all URLs are absolute and properly formatted.
</Warning>

<Tip>
  Track crawling progress via `GET /api/data/urls/?corpora={id}`. Monitor `indexing_status` transitions from `PRS` → `IND`.
</Tip>

<Note>
  **Deletion Impact**: Deleting a URL via `DELETE /api/data/urls/{id}/` immediately removes all derived chunks from the vector store. This action cannot be undone.
</Note>

## Client examples

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    import os
    import requests

    BASE_URL = "https://your-soar-instance.com"
    TOKEN = os.environ["SOAR_LABS_TOKEN"]
    CORPUS_ID = "8d0f0a5d-4b5e-4c09-9db6-0e9d2aa8a9fd"

    payload = {
        "corpora": CORPUS_ID,
        "urls": [
            {"url": "https://docs.example.com/support/escalations"},
            {"url": "https://status.example.com/incidents", "scrape_sitemap": True},
        ],
    }

    response = requests.post(
        f"{BASE_URL}/api/data/urls/",
        headers={
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=30,
    )
    response.raise_for_status()
    urls = response.json()
    ```
  </Tab>

  <Tab title="TypeScript / JavaScript">
    ```ts theme={null}
    const BASE_URL = "https://your-soar-instance.com";
    const token = process.env.SOAR_LABS_TOKEN!;

    async function addUrls(corpusId: string) {
      const response = await fetch(`${BASE_URL}/api/data/urls/`, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${token}`,
        },
        body: JSON.stringify({
          corpora: corpusId,
          urls: [
            { url: "https://docs.example.com/support/escalations" },
            { url: "https://status.example.com/incidents", scrape_sitemap: true },
          ],
        }),
      });

      if (!response.ok) {
        throw new Error(`Add URLs failed: ${response.status}`);
      }

      return response.json();
    }
    ```
  </Tab>

  <Tab title="Java">
    ```java theme={null}
    import java.net.URI;
    import java.net.http.HttpClient;
    import java.net.http.HttpRequest;
    import java.net.http.HttpResponse;

    var BASE_URL = "https://your-soar-instance.com";
    var token = System.getenv("SOAR_LABS_TOKEN");
    var corpusId = "8d0f0a5d-4b5e-4c09-9db6-0e9d2aa8a9fd";

    var json = "{" +
        "\"corpora\":\"" + corpusId + "\"," +
        "\"urls\":[{" +
            "\"url\":\"https://docs.example.com/support/escalations\"" +
        "},{" +
            "\"url\":\"https://status.example.com/incidents\",\"scrape_sitemap\":true" +
        "}]" +
    "}";

    var request = HttpRequest.newBuilder(URI.create(BASE_URL + "/api/data/urls/"))
        .header("Authorization", "Bearer " + token)
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(json))
        .build();

    var response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());

    if (response.statusCode() >= 400) {
        throw new RuntimeException("Add URLs failed: " + response.statusCode());
    }
    ```
  </Tab>
</Tabs>

## Best Practices

<AccordionGroup>
  <Accordion title="Choosing URLs Wisely" icon="bullseye">
    Select URLs that provide maximum value:

    **Good candidates:**

    * Documentation pages with stable content
    * Knowledge base articles
    * Blog posts and tutorials
    * Product specifications
    * API reference pages

    **Avoid:**

    * Dynamic pages with frequently changing content
    * Pages behind authentication/paywalls
    * JavaScript-heavy SPAs (content may not extract properly)
    * Pages with primarily images/videos (limited text content)
    * Auto-generated index pages with little content
  </Accordion>

  <Accordion title="Sitemap Scraping Strategy" icon="sitemap">
    Use `scrape_sitemap: true` strategically:

    **When to enable:**

    * Well-structured documentation sites
    * Blog archives (when you want all posts)
    * Product catalogs with many pages
    * Knowledge bases with comprehensive sitemaps

    **When to avoid:**

    * Sites with 100+ pages (submit sections instead)
    * Frequently updated news sites (content becomes stale)
    * Sites with dynamic pagination (submit specific URLs)

    **Best practice**: Start with a single page, verify content quality, then enable sitemap scraping if needed.
  </Accordion>

  <Accordion title="Handling Crawl Failures" icon="triangle-exclamation">
    Common crawl failures and solutions:

    **404 Not Found:**

    * Verify URL is correct and accessible
    * Check if page was moved or deleted
    * Try accessing in a browser first

    **403 Forbidden / 401 Unauthorized:**

    * Page requires authentication
    * Site blocks bots/crawlers
    * Consider uploading content as file instead

    **Timeout Errors:**

    * Page is too slow to load
    * Server is experiencing issues
    * Try again later or contact site admin

    **Content Extraction Failures:**

    * JavaScript-heavy page (content not in initial HTML)
    * Page has anti-scraping measures
    * Consider using an alternative format (PDF, markdown)
  </Accordion>

  <Accordion title="Content Freshness" icon="clock-rotate-left">
    Managing URL content updates:

    **Strategy 1: Manual Refresh**

    * Delete the old URL resource
    * Re-add the URL to fetch latest content
    * Best for infrequently changing content

    **Strategy 2: Scheduled Updates**

    * Implement periodic refresh via API calls
    * Delete and re-add URLs on a schedule
    * Good for documentation that updates regularly

    **Strategy 3: Webhooks (Advanced)**

    * Set up webhooks to trigger updates on content changes
    * Requires integration with content management system
    * Best for real-time accuracy requirements
  </Accordion>

  <Accordion title="Robots.txt Compliance" icon="robot">
    Soar Labs respects `robots.txt` directives:

    * Crawlers follow site-specific rate limits
    * Disallowed paths are not crawled
    * User-agent identification: `SoarLabsBot`

    **If crawling fails:**

    1. Check the site's `robots.txt`
    2. Verify your URLs aren't disallowed
    3. Consider reaching out to site owner for permission
    4. Alternative: Upload content as files if you have access
  </Accordion>
</AccordionGroup>

## Management Operations

<AccordionGroup>
  <Accordion title="List All URLs" icon="list">
    Retrieve all URLs in a corpus:

    ```bash theme={null}
    curl -X GET "https://{your-host}/api/data/urls/?corpora={corpus-id}" \
      -H "Authorization: Bearer $SOAR_LABS_TOKEN"
    ```

    Supports pagination with `page` and `page_size` parameters.
  </Accordion>

  <Accordion title="Update a URL" icon="pen-to-square">
    Modify URL settings (e.g., enable sitemap scraping):

    ```bash theme={null}
    curl -X PATCH "https://{your-host}/api/data/urls/{url-id}/" \
      -H "Authorization: Bearer $SOAR_LABS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"scrape_sitemap": true}'
    ```

    **Note**: Changing settings triggers re-crawling and re-indexing.
  </Accordion>

  <Accordion title="Refresh URL Content" icon="arrows-rotate">
    To refresh stale content, delete and re-add the URL:

    ```bash theme={null}
    # Step 1: Delete old version
    curl -X DELETE "https://{your-host}/api/data/urls/{url-id}/" \
      -H "Authorization: Bearer $SOAR_LABS_TOKEN"

    # Step 2: Re-add with fresh content
    curl -X POST "https://{your-host}/api/data/urls/" \
      -H "Authorization: Bearer $SOAR_LABS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "corpora": "{corpus-id}",
        "urls": [{"url": "https://example.com/updated-page"}]
      }'
    ```
  </Accordion>

  <Accordion title="Delete URLs" icon="trash">
    Remove URLs from the corpus:

    ```bash theme={null}
    curl -X DELETE "https://{your-host}/api/data/urls/{url-id}/" \
      -H "Authorization: Bearer $SOAR_LABS_TOKEN"
    ```

    **Warning**: Deletion immediately removes all crawled content and vector embeddings. Cannot be undone.
  </Accordion>
</AccordionGroup>

## Crawling Performance

<Tip>
  **Typical Crawl Times:**

  * Simple pages (\< 100KB HTML): 2-5 seconds
  * Documentation pages (100-500KB): 10-30 seconds
  * Complex pages with many images: 30-60 seconds
  * Sitemap crawls (10-50 pages): 1-5 minutes
</Tip>

<Warning>
  Rate limiting may apply to prevent overwhelming target sites. Large sitemap crawls are processed in batches with delays between requests.
</Warning>


## OpenAPI

````yaml POST /api/data/urls/
openapi: 3.0.3
info:
  title: Soar Labs API - Insider Dev preview
  version: 0.1.0
  description: <b>Soar Labs Advanced RAG platform</b>
servers: []
security: []
paths:
  /api/data/urls/:
    post:
      tags:
        - Resources
      description: >-
        URL creation API using Django Rest Framework.


        This viewset provides CRUD operations for URL objects associated with a
        Corpora.
      operationId: api_data_urls_create
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/URL'
          application/x-www-form-urlencoded:
            schema:
              $ref: '#/components/schemas/URL'
          multipart/form-data:
            schema:
              $ref: '#/components/schemas/URL'
        required: true
      responses:
        '201':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/URL'
          description: ''
      security:
        - jwtHeaderAuth: []
        - jwtCookieAuth: []
        - cookieAuth: []
        - basicAuth: []
components:
  schemas:
    URL:
      type: object
      properties:
        id:
          type: string
          format: uuid
          readOnly: true
        created_at:
          type: string
          format: date-time
          readOnly: true
          description: The date and time the organization was created
        updated_at:
          type: string
          format: date-time
          readOnly: true
          description: Last updated time
        indexed_on:
          type: string
          format: date-time
          readOnly: true
          nullable: true
        indexing_status:
          allOf:
            - $ref: '#/components/schemas/IndexingStatusEnum'
          readOnly: true
        url:
          type: string
          format: uri
          description: URL of the resource
          maxLength: 200
        clean_url:
          type: string
          format: uri
          nullable: true
          description: Canonical form of the URL - free of tracking parameters
          maxLength: 200
        scrape_sitemap:
          type: boolean
          description: Whether to scrape the sitemap for additional URLs
        metadata:
          nullable: true
          description: Additional metadata for the URL
        corpora:
          type: string
          format: uuid
          description: Corpora to which the Maps to
      required:
        - corpora
        - created_at
        - id
        - indexed_on
        - indexing_status
        - updated_at
        - url
    IndexingStatusEnum:
      enum:
        - PND
        - IQE
        - PRS
        - DEX
        - DER
        - IND
        - CMP
        - ERR
      type: string
      description: |-
        * `PND` - Pending
        * `IQE` - In Queue
        * `PRS` - Processing
        * `DEX` - Data Extracted Successfully
        * `DER` - Data Extraction Error
        * `IND` - Indexed
        * `CMP` - Completed
        * `ERR` - Error
  securitySchemes:
    jwtHeaderAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
    jwtCookieAuth:
      type: apiKey
      in: cookie
      name: soar-app-auth
    cookieAuth:
      type: apiKey
      in: cookie
      name: sessionid
    basicAuth:
      type: http
      scheme: basic

````