> For the complete documentation index, see [llms.txt](https://methara.gitbook.io/methara-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://methara.gitbook.io/methara-docs/methara-developer-documentation/processor.md).

# Processor

The Processor module compresses and cleans text before it is sent to an LLM.

Removing stopwords from casual messages typically reduces token count by 20 to 40 percent.

### Processor.shrink(text, options?)

This is the full compression pipeline.

It removes email signatures, normalizes whitespace, and strips stopwords.

```ts
Processor.shrink('Hey, I was just wondering if you could maybe help me out with my tasks?');
// 'help tasks'

// Disable stopword removal
Processor.shrink(text, { removeStopwords: false });
```

### Processor.truncate(text, maxTokens)

Trims text to a maximum token count while preserving the most recent content.

Use it before sending long conversation histories to an LLM.

```ts
const trimmedHistory = Processor.truncate(conversation.history, 400);

const before = Processor.countTokens('This is a fairly long message with many words in it.');
// 13
```

### All methods

| Method                                | Does                                                              | Returns |
| ------------------------------------- | ----------------------------------------------------------------- | ------- |
| `Processor.shrink(text, options?)`    | Signature removal, whitespace normalization, and stopword removal | string  |
| `Processor.removeStopwords(text)`     | Strips common English stopwords                                   | string  |
| `Processor.normalizeWhitespace(text)` | Collapses repeated spaces, tabs, and newlines                     | string  |
| `Processor.removeSignature(text)`     | Cuts text at the first email signature marker                     | string  |
| `Processor.countTokens(text)`         | Estimates token count with a `~4 chars/token` heuristic           | number  |
| `Processor.truncate(text, maxTokens)` | Trims to a token limit while keeping recent content               | string  |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://methara.gitbook.io/methara-docs/methara-developer-documentation/processor.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.