Tools for agents: UTF-8 byte-budget text chunker

Split Unicode text into lossless byte-bounded chunks with exact code-point and UTF-8 offsets.

Connect with MCP

{
  "mcpServers": {
    "super-agent-tools": {
      "url": "https://app.getsupers.com/sites/agent-utf8-chunks/mcp"
    }
  }
}
Streamable HTTP · No API key · Stateless

Use this URL in any client supporting remote Streamable HTTP MCP. Calls return structured JSON; inputs are processed without persistence.

Call chunk_text_by_utf8_bytes

max_bytes is a caller-supplied positive integer budget for each chunk encoded as raw UTF-8 without a BOM. Greedy chunks contain as many consecutive complete Unicode scalar values as fit. If any single scalar needs more bytes than the budget, the whole call fails; no oversized chunk is emitted. Joining returned text fields in order reconstructs the exact original text. Offsets are zero-based half-open positions in the original text. Empty text returns zero chunks. Newlines, whitespace and normalization are preserved. Scalar boundaries do not guarantee grapheme-cluster, word, sentence, Markdown, JSON or tokenizer boundaries: a combining mark or joined emoji sequence may be split between chunks. This is not an LLM token counter. JSON escaping and transport framing may require more bytes than the raw UTF-8 text budget. Unpaired surrogates and boolean budgets are rejected. No text is persisted or sent to another service.

Input and output schemas
{
  "name": "chunk_text_by_utf8_bytes",
  "title": "UTF-8 byte-budget text chunker",
  "description": "Greedily split text into nonempty Unicode-scalar-aligned chunks within max_bytes UTF-8 bytes each, with half-open code-point and byte offsets.",
  "inputSchema": {
    "properties": {
      "text": {
        "title": "Text",
        "type": "string"
      },
      "max_bytes": {
        "title": "Max Bytes",
        "type": "integer"
      }
    },
    "required": [
      "text",
      "max_bytes"
    ],
    "title": "chunk_text_by_utf8_bytesArguments",
    "type": "object"
  },
  "outputSchema": {
    "additionalProperties": true,
    "title": "chunk_text_by_utf8_bytesDictOutput",
    "type": "object"
  },
  "icons": null,
  "annotations": {
    "title": null,
    "readOnlyHint": true,
    "destructiveHint": false,
    "idempotentHint": true,
    "openWorldHint": false
  },
  "meta": null,
  "execution": null
}

Run the example to see the actual result.

HTTP alternative

POST https://app.getsupers.com/sites/agent-utf8-chunks/call
Content-Type: application/json

{
  "text": "A\u00e9\u6f22B\nC",
  "max_bytes": 5
}

The HTTP and MCP interfaces execute the same implementation. Validation errors are returned explicitly. See the tool notes above for its supported inputs and behavior.

UTF-8 Byte-Budget Text Chunker

Read the explanation

The tool greedily partitions Unicode text into chunks bounded strictly by a maximum raw UTF-8 byte budget. Different Unicode scalars consume different byte lengths. ASCII 'A' takes one byte, while e acute requires two bytes, filling three of the five byte capacity. The three-byte han character would push the total to six bytes, breaching the limit. The chunker immediately seals chunk zero at three bytes, placing han into chunk one. Submitting the form outputs structured JSON with zero-based, half-open byte and scalar ranges, guaranteeing exact lossless reconstruction of the source string.

UTF-8 Byte-Budget Text Chunker Mechanics

How does the chunker partition Unicode text into byte-bounded chunks without breaking multi-byte code points?

The chunk_text_by_utf8_bytes tool iterates through raw Unicode scalar values and computes each character's exact UTF-8 byte length (1 to 4 bytes). It greedily packs complete scalar values into the active chunk until adding the next character would exceed max_bytes. Whenever the current budget would be exceeded, it emits that chunk with exact half-open code-point and byte offsets, then opens a fresh chunk. Concatenating all chunk texts in order reconstructs the source string losslessly.

The partitioning is strictly scalar-aligned, not grapheme-cluster-aligned; multi-scalar graphemes such as combining accents or emoji ZWJ sequences can be split across chunk boundaries. If any individual scalar requires more bytes than max_bytes, the whole invocation raises a ValueError instead of creating an oversized chunk. The budget also measures raw UTF-8 octets, not LLM token counts or JSON transport overhead.

Try a worked example

Submitting the default JSON arguments with text 'A\u00e9\u6f22B\nC' and max_bytes 5 evaluates each character's byte footprint: 'A' (1 byte), '\u00e9' (2 bytes), '\u6f22' (3 bytes), 'B' (1 byte), '\n' (1 byte), and 'C' (1 byte). Chunk 0 takes 'A\u00e9' (3 bytes, code points [0, 2), bytes [0, 3)); adding '\u6f22' would require 6 bytes, exceeding 5. Chunk 1 receives '\u6f22B\n' (5 bytes, code points [2, 5), bytes [3, 8)). Chunk 2 receives 'C' (1 byte, code points [5, 6), bytes [8, 9)), producing 3 total chunks across 6 code points and 9 UTF-8 bytes.

Unicode Scalar Encoding and Byte Footprints

Under RFC 3629, standard UTF-8 represents valid Unicode scalar values (U+0000 to U+10FFFF, excluding the surrogate code points U+D800 to U+DFFF) in sequences of 1 to 4 octets. Standard ASCII characters consume 1 byte, extended Latin and accented characters consume 2 bytes, common CJK ideographs take 3 bytes, and supplementary plane symbols like emoji require 4 bytes. RFC 3629 - UTF-8, a transformation format of ISO 10646

The tool validates that the input text contains only Unicode scalar values and rejects surrogate pairs or isolated surrogates that violate the UTF-8 specification. RFC 3629 - UTF-8, a transformation format of ISO 10646

Deterministic Greedy Partitioning

The implementation processes characters sequentially, tracking the running byte total. A chunk is finalized as soon as the addition of the next scalar value would cause the chunk's cumulative byte count to exceed max_bytes. Zero-based, half-open intervals [start, end) are recorded for both Python code points and UTF-8 octet offsets.

Sources and further reading