Skip to content

Legacy library

This page describes the old TypeScript @sylphx/coderag library. It is not the Locus MCP server. The live server is local Rust BM25 in @sylphx/locus. Start at the quickstart.

Storage ​

Locus provides two storage implementations: PersistentStorage for SQLite-backed persistence and MemoryStorage for in-memory usage.

PersistentStorage ​

SQLite-based storage using LibSQL and Drizzle ORM. Supports chunk-level indexing with BM25 scoring.

Constructor ​

typescript
new PersistentStorage(config?: DbConfig)

Parameters ​

config DbConfig (optional)

typescript
interface DbConfig {
  codebaseRoot?: string  // Project root (default: process.cwd())
  dbPath?: string        // Custom database path (optional)
}

If dbPath is not provided, uses ~/.coderag/projects/<hash>/ based on the codebase root path.

Example ​

typescript
import { PersistentStorage } from '@sylphx/coderag'

// Automatic path (recommended)
const storage = new PersistentStorage({
  codebaseRoot: '/path/to/project'
})
// Creates: ~/.coderag/projects/a3d5f8b2/index.db

// Custom path
const storage = new PersistentStorage({
  dbPath: '/custom/path/index.db'
})

Storage Interface ​

Both PersistentStorage and MemoryStorage implement the Storage interface:

typescript
interface Storage {
  storeFile(file: CodebaseFile): Promise<void>
  storeFiles?(files: CodebaseFile[]): Promise<void>
  getFile(path: string): Promise<CodebaseFile | null>
  getAllFiles(): Promise<CodebaseFile[]>
  deleteFile(path: string): Promise<void>
  clear(): Promise<void>
  count(): Promise<number>
  getChunkCount?(): Promise<number>
  exists(path: string): Promise<boolean>
}

File Operations ​

storeFile() ​

Store a single file in the index.

typescript
async storeFile(file: CodebaseFile): Promise<void>

Parameters ​

typescript
interface CodebaseFile {
  path: string
  content: string
  size: number
  mtime: number | Date
  language?: string
  hash: string
}

Example ​

typescript
await storage.storeFile({
  path: 'src/index.ts',
  content: 'export function hello() { }',
  size: 30,
  mtime: Date.now(),
  language: 'typescript',
  hash: 'abc123'
})

storeFiles() ​

Store multiple files in a batch (more efficient than individual stores).

typescript
async storeFiles(files: CodebaseFile[]): Promise<void>

Example ​

typescript
const files: CodebaseFile[] = [
  { path: 'src/a.ts', content: '...', /* ... */ },
  { path: 'src/b.ts', content: '...', /* ... */ }
]

await storage.storeFiles(files)

getFile() ​

Retrieve a file by path.

typescript
async getFile(path: string): Promise<CodebaseFile | null>

Example ​

typescript
const file = await storage.getFile('src/index.ts')
if (file) {
  console.log(file.content)
}

getAllFiles() ​

Get all indexed files.

typescript
async getAllFiles(): Promise<CodebaseFile[]>

Example ​

typescript
const files = await storage.getAllFiles()
console.log(`Total files: ${files.length}`)

deleteFile() ​

Delete a file from the index.

typescript
async deleteFile(path: string): Promise<void>

Example ​

typescript
await storage.deleteFile('src/removed.ts')

deleteFiles() ​

Delete multiple files in a batch (PersistentStorage only).

typescript
async deleteFiles(paths: string[]): Promise<void>

Example ​

typescript
await storage.deleteFiles(['src/a.ts', 'src/b.ts'])

clear() ​

Clear all files from the index.

typescript
async clear(): Promise<void>

Example ​

typescript
await storage.clear()

count() ​

Get total number of indexed files.

typescript
async count(): Promise<number>

Example ​

typescript
const fileCount = await storage.count()
console.log(`Indexed ${fileCount} files`)

exists() ​

Check if a file exists in the index.

typescript
async exists(path: string): Promise<boolean>

Example ​

typescript
if (await storage.exists('src/index.ts')) {
  console.log('File is indexed')
}

Chunk Operations ​

These methods are specific to PersistentStorage and support chunk-level indexing.

storeChunks() ​

Store chunks for a file (replaces existing chunks).

typescript
async storeChunks(filePath: string, chunks: ChunkData[]): Promise<number[]>

Parameters ​

typescript
interface ChunkData {
  content: string
  type: string            // AST node type (e.g., 'FunctionDeclaration')
  startLine: number
  endLine: number
  metadata?: Record<string, unknown>
}

Returns ​

Promise<number[]> - Array of chunk IDs.

Example ​

typescript
const chunkIds = await storage.storeChunks('src/index.ts', [
  {
    content: 'export function hello() { }',
    type: 'FunctionDeclaration',
    startLine: 1,
    endLine: 3
  }
])

storeManyChunks() ​

Store chunks for multiple files in a batch.

typescript
async storeManyChunks(
  fileChunks: Array<{ filePath: string; chunks: ChunkData[] }>
): Promise<Map<string, number[]>>

Returns ​

Promise<Map<string, number[]>> - Map of file paths to chunk ID arrays.

Example ​

typescript
const chunkIdMap = await storage.storeManyChunks([
  {
    filePath: 'src/a.ts',
    chunks: [{ content: '...', type: 'FunctionDeclaration', startLine: 1, endLine: 5 }]
  },
  {
    filePath: 'src/b.ts',
    chunks: [{ content: '...', type: 'ClassDeclaration', startLine: 1, endLine: 10 }]
  }
])

const aChunkIds = chunkIdMap.get('src/a.ts')

getChunksForFile() ​

Get all chunks for a file.

typescript
async getChunksForFile(filePath: string): Promise<StoredChunk[]>

Returns ​

typescript
interface StoredChunk extends ChunkData {
  id: number
  fileId: number
  filePath: string
}

Example ​

typescript
const chunks = await storage.getChunksForFile('src/index.ts')
for (const chunk of chunks) {
  console.log(`${chunk.type} (${chunk.startLine}-${chunk.endLine}): ${chunk.content}`)
}

getChunkCount() ​

Get total number of indexed chunks.

typescript
async getChunkCount(): Promise<number>

Example ​

typescript
const chunkCount = await storage.getChunkCount()
console.log(`Indexed ${chunkCount} chunks`)

Vector Operations ​

These methods manage TF-IDF vectors for chunk-level search.

storeChunkVectors() ​

Store TF-IDF vectors for a chunk.

typescript
async storeChunkVectors(
  chunkId: number,
  terms: Map<string, { tf: number; tfidf: number; rawFreq: number }>,
  tokenCount?: number
): Promise<void>

Example ​

typescript
const terms = new Map([
  ['function', { tf: 0.5, tfidf: 1.2, rawFreq: 2 }],
  ['export', { tf: 0.25, tfidf: 0.8, rawFreq: 1 }]
])

await storage.storeChunkVectors(123, terms, 4)

storeManyChunkVectors() ​

Store vectors for multiple chunks in a batch.

typescript
async storeManyChunkVectors(
  chunkVectors: Array<{
    chunkId: number
    terms: Map<string, { tf: number; tfidf: number; rawFreq: number }>
    tokenCount?: number
  }>
): Promise<void>

getChunkVectors() ​

Get TF-IDF vectors for a chunk.

typescript
async getChunkVectors(
  chunkId: number
): Promise<Map<string, { tf: number; tfidf: number; rawFreq: number }> | null>

getAllChunkVectors() ​

Get all chunk vectors in a single query.

typescript
async getAllChunkVectors(): Promise<
  Map<number, Map<string, { tf: number; tfidf: number; rawFreq: number }>>
>

Returns a map of chunk IDs to their term vectors.

IDF Operations ​

Manage inverse document frequency scores.

storeIdfScores() ​

Store IDF scores for all terms.

typescript
async storeIdfScores(
  idf: Map<string, number>,
  docFreq: Map<string, number>
): Promise<void>

getIdfScores() ​

Get all IDF scores.

typescript
async getIdfScores(): Promise<Map<string, number>>

getIdfScoresForTerms() ​

Get IDF scores for specific terms only.

typescript
async getIdfScoresForTerms(terms: string[]): Promise<Map<string, number>>

rebuildIdfScoresFromVectors() ​

Recalculate IDF scores from stored vectors.

typescript
async rebuildIdfScoresFromVectors(): Promise<void>

Uses smoothed IDF formula: log((N+1)/(df+1)) + 1

Search Operations ​

searchByTerms() ​

Search chunks by query terms using SQL.

typescript
async searchByTerms(
  queryTerms: string[],
  options?: { limit?: number }
): Promise<SearchCandidate[]>

Returns ​

typescript
interface SearchCandidate {
  chunkId: number
  filePath: string
  content: string
  type: string
  startLine: number
  endLine: number
  matchedTerms: Map<string, { tfidf: number; rawFreq: number }>
  magnitude: number
  tokenCount: number
}

Example ​

typescript
const candidates = await storage.searchByTerms(
  ['function', 'export'],
  { limit: 20 }
)

for (const candidate of candidates) {
  console.log(`${candidate.filePath}:${candidate.startLine}`)
  console.log(`Matched: ${Array.from(candidate.matchedTerms.keys()).join(', ')}`)
}

Metadata Operations ​

setMetadata() ​

Store metadata key-value pair.

typescript
async setMetadata(key: string, value: string): Promise<void>

getMetadata() ​

Retrieve metadata value.

typescript
async getMetadata(key: string): Promise<string | null>

getAverageDocLength() ​

Get average chunk token count (for BM25).

typescript
async getAverageDocLength(): Promise<number>

updateAverageDocLength() ​

Recalculate and store average chunk length.

typescript
async updateAverageDocLength(): Promise<number>

Maintenance Operations ​

recalculateTfidfScores() ​

Update all TF-IDF scores using current IDF values.

typescript
async recalculateTfidfScores(): Promise<void>

updateChunkMagnitudes() ​

Recalculate pre-computed magnitudes for cosine similarity.

typescript
async updateChunkMagnitudes(): Promise<void>

Magnitude formula: sqrt(sum(tfidf^2)) for each chunk.

getAllFileMetadata() ​

Get file metadata without content (for incremental updates).

typescript
async getAllFileMetadata(): Promise<Map<string, { mtime: number; hash: string }>>

getTermsForFiles() ​

Get all terms used in chunks of specified files.

typescript
async getTermsForFiles(paths: string[]): Promise<Set<string>>

Useful for tracking affected terms during incremental updates.

MemoryStorage ​

In-memory storage implementation. Useful for testing or temporary indexes.

Constructor ​

typescript
new MemoryStorage()

Example ​

typescript
import { MemoryStorage } from '@sylphx/coderag'

const storage = new MemoryStorage()

await storage.storeFile({
  path: 'test.ts',
  content: 'console.log("test")',
  size: 20,
  mtime: Date.now(),
  hash: 'abc'
})

const file = await storage.getFile('test.ts')

Database Schema ​

PersistentStorage uses the following tables:

files

  • id, path (unique), content, hash, size, mtime, language, indexed_at

chunks

  • id, file_id, content, type, start_line, end_line, metadata, magnitude, token_count

document_vectors

  • chunk_id, term, tf, tfidf, raw_freq

idf_scores

  • term (unique), idf, document_frequency

index_metadata

  • key (unique), value, updated_at

Performance Tips ​

Batch Operations ​

Use batch methods for better performance:

typescript
// Good: Single transaction
await storage.storeFiles(files)
await storage.storeManyChunks(fileChunks)

// Bad: Multiple transactions
for (const file of files) {
  await storage.storeFile(file)
}

Incremental Updates ​

Use metadata comparison to avoid unnecessary work:

typescript
const metadata = await storage.getAllFileMetadata()
const existing = metadata.get(filePath)

if (existing?.hash === newHash) {
  // Skip, content unchanged
}

Low Memory Mode ​

For large codebases, use SQL-based search:

typescript
const indexer = new CodebaseIndexer({
  storage: new PersistentStorage(),
  lowMemoryMode: true  // Uses searchByTerms() instead of in-memory index
})

Released under the MIT License.