Skip to content

Legacy library

This page describes the old TypeScript @sylphx/coderag library. It is not the Locus MCP server. The live server is local Rust BM25 in @sylphx/locus. Start at the quickstart.

AST Chunking ​

Locus uses AST-based chunking to split code at semantic boundaries (functions, classes, etc.) using the Synth parser library.

chunkCodeByAST() ​

Split code into semantic chunks using AST analysis.

typescript
async function chunkCodeByAST(
  code: string,
  filePath: string,
  options?: ASTChunkOptions
): Promise<readonly ChunkResult[]>

Parameters ​

code string - Source code to chunk

filePath string - File path (used for language detection)

options ASTChunkOptions (optional)

typescript
interface ASTChunkOptions {
  maxChunkSize?: number       // Max chunk chars (default: 1000)
  minChunkSize?: number       // Min chunk chars (default: 100)
  preserveContext?: boolean   // Include imports/types (default: true)
  nodeTypes?: string[]        // Custom AST node types to chunk
  parseEmbedded?: boolean     // Parse code in markdown (default: true)
}

Returns ​

Promise<readonly ChunkResult[]> - Array of semantic chunks

typescript
interface ChunkResult {
  content: string              // Chunk content
  type: string                 // AST node type
  startLine: number           // Start line (1-indexed)
  endLine: number             // End line (inclusive)
  metadata: Record<string, unknown>  // Additional metadata
}

Example ​

typescript
import { chunkCodeByAST } from '@sylphx/coderag'

const code = `
import { z } from 'zod'

export function validateUser(user: unknown) {
  const schema = z.object({
    name: z.string(),
    email: z.string().email()
  })
  return schema.parse(user)
}

export class UserService {
  async createUser(data: unknown) {
    const validated = validateUser(data)
    return this.db.users.create(validated)
  }
}
`

const chunks = await chunkCodeByAST(code, 'user-service.ts')

for (const chunk of chunks) {
  console.log(`Type: ${chunk.type}`)
  console.log(`Lines: ${chunk.startLine}-${chunk.endLine}`)
  console.log(chunk.content)
  console.log('---')
}

// Output:
// Type: FunctionDeclaration
// Lines: 3-10
// import { z } from 'zod'
//
// export function validateUser(user: unknown) { ... }
// ---
// Type: ClassDeclaration
// Lines: 12-17
// import { z } from 'zod'
//
// export class UserService { ... }

chunkCodeByASTSimple() ​

Simplified wrapper that returns chunk content only.

typescript
async function chunkCodeByASTSimple(
  code: string,
  filePath: string,
  options?: ASTChunkOptions
): Promise<readonly string[]>

Returns ​

Promise<readonly string[]> - Array of chunk content strings

Example ​

typescript
const chunks = await chunkCodeByASTSimple(code, 'example.ts')
// Returns: ['function a() { }', 'class B { }']

getSupportedLanguages() ​

Get list of supported languages for AST chunking.

typescript
function getSupportedLanguages(): string[]

Returns ​

string[] - Array of supported language names

Example ​

typescript
import { getSupportedLanguages } from '@sylphx/coderag'

const languages = getSupportedLanguages()
console.log(languages)
// ['javascript', 'typescript', 'python', 'rust', 'go', ...]

Supported Languages ​

Locus supports AST chunking for the following languages:

JavaScript/TypeScript

  • Extensions: .js, .mjs, .cjs, .jsx, .ts, .tsx, .mts, .cts
  • Boundaries: Functions, classes, methods, arrow functions
  • Context: Imports, type definitions

Python

  • Extensions: .py
  • Boundaries: Functions, classes, methods
  • Context: Imports, class definitions

Rust

  • Extensions: .rs
  • Boundaries: Functions, structs, impls, traits
  • Context: Use statements, type definitions

Go

  • Extensions: .go
  • Boundaries: Functions, methods, structs
  • Context: Imports, type declarations

Java

  • Extensions: .java
  • Boundaries: Classes, methods, constructors
  • Context: Imports, class declarations

C/C++

  • Extensions: .c, .h, .cpp, .hpp, .cc, .hh
  • Boundaries: Functions, structs, classes
  • Context: Includes, type definitions

Ruby

  • Extensions: .rb
  • Boundaries: Modules, classes, methods
  • Context: Requires, module definitions

PHP

  • Extensions: .php
  • Boundaries: Classes, functions, methods
  • Context: Namespace, use statements

Markdown

  • Extensions: .md, .mdx
  • Boundaries: Headings, code blocks
  • Embedded: Recursively parses code blocks

Markup/Data

  • HTML: .html, .htm
  • XML: .xml
  • JSON: .json
  • YAML: .yaml, .yml
  • TOML: .toml

Chunking Behavior ​

Semantic Boundaries ​

Code is split at meaningful boundaries based on AST structure:

typescript
const code = `
function greet() {
  console.log('hello')
}

function farewell() {
  console.log('goodbye')
}
`

const chunks = await chunkCodeByAST(code, 'example.js')
// Returns 2 chunks (one per function)

Context Preservation ​

By default, context (imports, types) is prepended to each chunk:

typescript
const code = `
import { User } from './types'

export function createUser(data: User) {
  return { ...data, id: generateId() }
}

export function deleteUser(id: string) {
  return db.users.delete(id)
}
`

const chunks = await chunkCodeByAST(code, 'user.ts', {
  preserveContext: true  // Default
})

// Each chunk includes: import { User } from './types'

Disable context preservation:

typescript
const chunks = await chunkCodeByAST(code, 'user.ts', {
  preserveContext: false
})
// Chunks contain only the function code

Large Chunk Handling ​

Chunks exceeding maxChunkSize are recursively split:

typescript
const code = `
export function processLargeData(items: Item[]) {
  // 2000 lines of code...
}
`

const chunks = await chunkCodeByAST(code, 'process.ts', {
  maxChunkSize: 1000
})
// Returns multiple sub-chunks from the function body

Small Chunk Merging ​

Small non-semantic chunks are merged to reach minChunkSize:

typescript
const code = `
const a = 1
const b = 2
const c = 3

function main() { }
`

const chunks = await chunkCodeByAST(code, 'example.js', {
  minChunkSize: 50
})
// Constants merged into single chunk, function as separate chunk

Fallback Behavior ​

If AST parsing fails, falls back to character chunking:

typescript
const invalidCode = `
function incomplete(
`

const chunks = await chunkCodeByAST(invalidCode, 'broken.js')
// Returns character-based chunks with metadata.fallback = true

Embedded Code Parsing ​

Markdown code blocks are recursively parsed:

typescript
const markdown = `
# Example

\`\`\`typescript
function hello() {
  console.log('hi')
}
\`\`\`
`

const chunks = await chunkCodeByAST(markdown, 'example.md', {
  parseEmbedded: true  // Default
})

// Returns chunk with:
// - type: 'FunctionDeclaration'
// - metadata.embeddedIn: 'CodeBlock'
// - metadata.embeddedLanguage: 'typescript'

Disable embedded parsing:

typescript
const chunks = await chunkCodeByAST(markdown, 'example.md', {
  parseEmbedded: false
})
// Returns code block as-is without parsing

Custom Node Types ​

Override default semantic boundaries:

typescript
const code = `
const config = {
  host: 'localhost',
  port: 3000
}

function start() { }
`

const chunks = await chunkCodeByAST(code, 'config.js', {
  nodeTypes: ['VariableDeclaration', 'FunctionDeclaration']
})
// Chunks both config and function separately

Chunk Metadata ​

Each chunk includes metadata about its origin:

typescript
const chunks = await chunkCodeByAST(code, 'example.ts')

const chunk = chunks[0]
console.log(chunk.metadata)

// Common metadata:
// - name: Function/class name (if available)
// - async: true/false for async functions
// - export: true/false for exported symbols
// - fallback: true if character chunking was used
// - split: true if chunk was split from larger node
// - merged: true if merged from small chunks
// - embeddedIn: Parent node type for embedded code
// - embeddedLanguage: Language of embedded code

Performance ​

Parsing Speed ​

Approximate parsing speeds:

  • JavaScript/TypeScript: ~5000 lines/sec
  • Python: ~3000 lines/sec
  • Rust/Go/Java/C++: ~2000 lines/sec (WASM-based)
  • Markdown: ~10000 lines/sec

Memory Usage ​

  • In-memory AST: ~1KB per 100 lines of code
  • Chunk overhead: ~100 bytes per chunk
  • Total: ~50-100MB for large codebases

Error Handling ​

Invalid Syntax ​

typescript
try {
  const chunks = await chunkCodeByAST(invalidCode, 'broken.js')
  // Falls back to character chunking
  if (chunks[0].metadata.fallback) {
    console.log('AST parsing failed, using fallback')
  }
} catch (error) {
  console.error('Chunking failed:', error)
}

Unknown Language ​

typescript
const chunks = await chunkCodeByAST(code, 'unknown.xyz')
// Logs: [WARN] Unknown language, falling back to character chunking
// Returns character-based chunks

Best Practices ​

Choose appropriate chunk sizes:

typescript
// For embeddings (typical token limit: 8192)
const chunks = await chunkCodeByAST(code, file, {
  maxChunkSize: 1000,  // ~500 tokens
  minChunkSize: 100
})

// For LLM context (larger is better)
const chunks = await chunkCodeByAST(code, file, {
  maxChunkSize: 2000,
  minChunkSize: 200
})

Preserve context for better search:

typescript
const chunks = await chunkCodeByAST(code, file, {
  preserveContext: true  // Include imports/types
})

Enable embedded parsing for documentation:

typescript
const chunks = await chunkCodeByAST(markdown, 'README.md', {
  parseEmbedded: true  // Parse code examples
})

Handle fallback gracefully:

typescript
const chunks = await chunkCodeByAST(code, file)
const validChunks = chunks.filter(c => !c.metadata.fallback)

Integration with Indexer ​

CodebaseIndexer automatically uses AST chunking:

typescript
import { CodebaseIndexer } from '@sylphx/coderag'

const indexer = new CodebaseIndexer({
  codebaseRoot: './src'
})

await indexer.index()
// Automatically chunks all files using chunkCodeByAST()

Search results include chunk metadata:

typescript
const results = await indexer.search('authentication')

for (const result of results) {
  console.log(`${result.chunkType} at ${result.path}:${result.startLine}`)
  // FunctionDeclaration at src/auth.ts:42
}

Language Configuration ​

View language-specific settings:

typescript
import { getLanguageConfig } from '@sylphx/coderag'

const config = getLanguageConfig('typescript')
console.log(config)

// {
//   parser: '@sylphx/synth-js',
//   boundaries: ['FunctionDeclaration', 'ClassDeclaration', ...],
//   contextTypes: ['ImportDeclaration', 'TypeAlias', ...],
//   embedded: [{ nodeType: 'CodeBlock', ... }]
// }

Released under the MIT License.