Legacy library
This page describes the old TypeScript @sylphx/coderag library. It is not the Locus MCP server. The live server is local Rust BM25 in @sylphx/locus. Start at the quickstart.
AST Chunking
Locus uses AST-based chunking to split code at semantic boundaries (functions, classes, etc.) using the Synth parser library.
chunkCodeByAST()
Split code into semantic chunks using AST analysis.
async function chunkCodeByAST(
code: string,
filePath: string,
options?: ASTChunkOptions
): Promise<readonly ChunkResult[]>Parameters
code string - Source code to chunk
filePath string - File path (used for language detection)
options ASTChunkOptions (optional)
interface ASTChunkOptions {
maxChunkSize?: number // Max chunk chars (default: 1000)
minChunkSize?: number // Min chunk chars (default: 100)
preserveContext?: boolean // Include imports/types (default: true)
nodeTypes?: string[] // Custom AST node types to chunk
parseEmbedded?: boolean // Parse code in markdown (default: true)
}Returns
Promise<readonly ChunkResult[]> - Array of semantic chunks
interface ChunkResult {
content: string // Chunk content
type: string // AST node type
startLine: number // Start line (1-indexed)
endLine: number // End line (inclusive)
metadata: Record<string, unknown> // Additional metadata
}Example
import { chunkCodeByAST } from '@sylphx/coderag'
const code = `
import { z } from 'zod'
export function validateUser(user: unknown) {
const schema = z.object({
name: z.string(),
email: z.string().email()
})
return schema.parse(user)
}
export class UserService {
async createUser(data: unknown) {
const validated = validateUser(data)
return this.db.users.create(validated)
}
}
`
const chunks = await chunkCodeByAST(code, 'user-service.ts')
for (const chunk of chunks) {
console.log(`Type: ${chunk.type}`)
console.log(`Lines: ${chunk.startLine}-${chunk.endLine}`)
console.log(chunk.content)
console.log('---')
}
// Output:
// Type: FunctionDeclaration
// Lines: 3-10
// import { z } from 'zod'
//
// export function validateUser(user: unknown) { ... }
// ---
// Type: ClassDeclaration
// Lines: 12-17
// import { z } from 'zod'
//
// export class UserService { ... }chunkCodeByASTSimple()
Simplified wrapper that returns chunk content only.
async function chunkCodeByASTSimple(
code: string,
filePath: string,
options?: ASTChunkOptions
): Promise<readonly string[]>Returns
Promise<readonly string[]> - Array of chunk content strings
Example
const chunks = await chunkCodeByASTSimple(code, 'example.ts')
// Returns: ['function a() { }', 'class B { }']getSupportedLanguages()
Get list of supported languages for AST chunking.
function getSupportedLanguages(): string[]Returns
string[] - Array of supported language names
Example
import { getSupportedLanguages } from '@sylphx/coderag'
const languages = getSupportedLanguages()
console.log(languages)
// ['javascript', 'typescript', 'python', 'rust', 'go', ...]Supported Languages
Locus supports AST chunking for the following languages:
JavaScript/TypeScript
- Extensions:
.js,.mjs,.cjs,.jsx,.ts,.tsx,.mts,.cts - Boundaries: Functions, classes, methods, arrow functions
- Context: Imports, type definitions
Python
- Extensions:
.py - Boundaries: Functions, classes, methods
- Context: Imports, class definitions
Rust
- Extensions:
.rs - Boundaries: Functions, structs, impls, traits
- Context: Use statements, type definitions
Go
- Extensions:
.go - Boundaries: Functions, methods, structs
- Context: Imports, type declarations
Java
- Extensions:
.java - Boundaries: Classes, methods, constructors
- Context: Imports, class declarations
C/C++
- Extensions:
.c,.h,.cpp,.hpp,.cc,.hh - Boundaries: Functions, structs, classes
- Context: Includes, type definitions
Ruby
- Extensions:
.rb - Boundaries: Modules, classes, methods
- Context: Requires, module definitions
PHP
- Extensions:
.php - Boundaries: Classes, functions, methods
- Context: Namespace, use statements
Markdown
- Extensions:
.md,.mdx - Boundaries: Headings, code blocks
- Embedded: Recursively parses code blocks
Markup/Data
- HTML:
.html,.htm - XML:
.xml - JSON:
.json - YAML:
.yaml,.yml - TOML:
.toml
Chunking Behavior
Semantic Boundaries
Code is split at meaningful boundaries based on AST structure:
const code = `
function greet() {
console.log('hello')
}
function farewell() {
console.log('goodbye')
}
`
const chunks = await chunkCodeByAST(code, 'example.js')
// Returns 2 chunks (one per function)Context Preservation
By default, context (imports, types) is prepended to each chunk:
const code = `
import { User } from './types'
export function createUser(data: User) {
return { ...data, id: generateId() }
}
export function deleteUser(id: string) {
return db.users.delete(id)
}
`
const chunks = await chunkCodeByAST(code, 'user.ts', {
preserveContext: true // Default
})
// Each chunk includes: import { User } from './types'Disable context preservation:
const chunks = await chunkCodeByAST(code, 'user.ts', {
preserveContext: false
})
// Chunks contain only the function codeLarge Chunk Handling
Chunks exceeding maxChunkSize are recursively split:
const code = `
export function processLargeData(items: Item[]) {
// 2000 lines of code...
}
`
const chunks = await chunkCodeByAST(code, 'process.ts', {
maxChunkSize: 1000
})
// Returns multiple sub-chunks from the function bodySmall Chunk Merging
Small non-semantic chunks are merged to reach minChunkSize:
const code = `
const a = 1
const b = 2
const c = 3
function main() { }
`
const chunks = await chunkCodeByAST(code, 'example.js', {
minChunkSize: 50
})
// Constants merged into single chunk, function as separate chunkFallback Behavior
If AST parsing fails, falls back to character chunking:
const invalidCode = `
function incomplete(
`
const chunks = await chunkCodeByAST(invalidCode, 'broken.js')
// Returns character-based chunks with metadata.fallback = trueEmbedded Code Parsing
Markdown code blocks are recursively parsed:
const markdown = `
# Example
\`\`\`typescript
function hello() {
console.log('hi')
}
\`\`\`
`
const chunks = await chunkCodeByAST(markdown, 'example.md', {
parseEmbedded: true // Default
})
// Returns chunk with:
// - type: 'FunctionDeclaration'
// - metadata.embeddedIn: 'CodeBlock'
// - metadata.embeddedLanguage: 'typescript'Disable embedded parsing:
const chunks = await chunkCodeByAST(markdown, 'example.md', {
parseEmbedded: false
})
// Returns code block as-is without parsingCustom Node Types
Override default semantic boundaries:
const code = `
const config = {
host: 'localhost',
port: 3000
}
function start() { }
`
const chunks = await chunkCodeByAST(code, 'config.js', {
nodeTypes: ['VariableDeclaration', 'FunctionDeclaration']
})
// Chunks both config and function separatelyChunk Metadata
Each chunk includes metadata about its origin:
const chunks = await chunkCodeByAST(code, 'example.ts')
const chunk = chunks[0]
console.log(chunk.metadata)
// Common metadata:
// - name: Function/class name (if available)
// - async: true/false for async functions
// - export: true/false for exported symbols
// - fallback: true if character chunking was used
// - split: true if chunk was split from larger node
// - merged: true if merged from small chunks
// - embeddedIn: Parent node type for embedded code
// - embeddedLanguage: Language of embedded codePerformance
Parsing Speed
Approximate parsing speeds:
- JavaScript/TypeScript: ~5000 lines/sec
- Python: ~3000 lines/sec
- Rust/Go/Java/C++: ~2000 lines/sec (WASM-based)
- Markdown: ~10000 lines/sec
Memory Usage
- In-memory AST: ~1KB per 100 lines of code
- Chunk overhead: ~100 bytes per chunk
- Total: ~50-100MB for large codebases
Error Handling
Invalid Syntax
try {
const chunks = await chunkCodeByAST(invalidCode, 'broken.js')
// Falls back to character chunking
if (chunks[0].metadata.fallback) {
console.log('AST parsing failed, using fallback')
}
} catch (error) {
console.error('Chunking failed:', error)
}Unknown Language
const chunks = await chunkCodeByAST(code, 'unknown.xyz')
// Logs: [WARN] Unknown language, falling back to character chunking
// Returns character-based chunksBest Practices
Choose appropriate chunk sizes:
// For embeddings (typical token limit: 8192)
const chunks = await chunkCodeByAST(code, file, {
maxChunkSize: 1000, // ~500 tokens
minChunkSize: 100
})
// For LLM context (larger is better)
const chunks = await chunkCodeByAST(code, file, {
maxChunkSize: 2000,
minChunkSize: 200
})Preserve context for better search:
const chunks = await chunkCodeByAST(code, file, {
preserveContext: true // Include imports/types
})Enable embedded parsing for documentation:
const chunks = await chunkCodeByAST(markdown, 'README.md', {
parseEmbedded: true // Parse code examples
})Handle fallback gracefully:
const chunks = await chunkCodeByAST(code, file)
const validChunks = chunks.filter(c => !c.metadata.fallback)Integration with Indexer
CodebaseIndexer automatically uses AST chunking:
import { CodebaseIndexer } from '@sylphx/coderag'
const indexer = new CodebaseIndexer({
codebaseRoot: './src'
})
await indexer.index()
// Automatically chunks all files using chunkCodeByAST()Search results include chunk metadata:
const results = await indexer.search('authentication')
for (const result of results) {
console.log(`${result.chunkType} at ${result.path}:${result.startLine}`)
// FunctionDeclaration at src/auth.ts:42
}Language Configuration
View language-specific settings:
import { getLanguageConfig } from '@sylphx/coderag'
const config = getLanguageConfig('typescript')
console.log(config)
// {
// parser: '@sylphx/synth-js',
// boundaries: ['FunctionDeclaration', 'ClassDeclaration', ...],
// contextTypes: ['ImportDeclaration', 'TypeAlias', ...],
// embedded: [{ nodeType: 'CodeBlock', ... }]
// }