Skip to content

Legacy library

This page describes the old TypeScript @sylphx/coderag library. It is not the Locus MCP server. The live server is local Rust BM25 in @sylphx/locus. Start at the quickstart.

File Watching ​

Locus provides real-time file watching using @parcel/watcher for automatic index updates when files change.

@parcel/watcher Usage ​

@parcel/watcher is a native file watcher that uses platform-specific APIs for efficient change detection.

Why @parcel/watcher?

  • Native performance (FSEvents on macOS, inotify on Linux, ReadDirectoryChangesW on Windows)
  • Low CPU overhead (kernel-level notifications, not polling)
  • Recursive directory watching
  • Built-in ignore patterns
  • Reliable event delivery

Alternatives comparison:

LibraryPerformanceReliabilityPlatform Support
@parcel/watcherExcellent (native)HighmacOS, Linux, Windows
chokidarGood (fallback to polling)MediumAll platforms
fs.watchPoor (OS-dependent)LowNode.js built-in

Installation:

bash
npm install @parcel/watcher

@parcel/watcher is a native module with prebuilt binaries for major platforms.

Debouncing (500ms) ​

File watchers emit events immediately, but multiple events often occur for a single logical change (e.g., save triggers multiple write events).

Debouncing behavior:

Locus waits 500ms after the last event before processing changes. This consolidates rapid-fire events into a single update.

Example without debouncing:

User saves file
  ├─ 0ms: change event
  ├─ 5ms: change event
  ├─ 10ms: change event
  └─ 15ms: change event
Result: 4 index updates (wasteful)

Example with debouncing (500ms):

User saves file
  ├─ 0ms: change event → start timer
  ├─ 5ms: change event → reset timer
  ├─ 10ms: change event → reset timer
  └─ 15ms: change event → reset timer
  ... 500ms pass ...
  └─ 515ms: process update once
Result: 1 index update (efficient)

Implementation:

typescript
private pendingUpdates = new Map<string, NodeJS.Timeout>()

private handleFileChange(type: 'add' | 'change' | 'unlink', absolutePath: string): void {
  const relativePath = path.relative(this.codebaseRoot, absolutePath)

  // Clear existing timeout
  const existing = this.pendingUpdates.get(relativePath)
  if (existing) {
    clearTimeout(existing)
  }

  // Set new timeout (500ms)
  const timeout = setTimeout(async () => {
    this.pendingUpdates.delete(relativePath)
    await this.processFileChange(type, relativePath, absolutePath)
  }, 500)

  this.pendingUpdates.set(relativePath, timeout)
}

Debounce duration:

500ms is chosen to balance responsiveness and efficiency:

  • Too short (100ms): Multiple updates for single edit
  • Too long (2000ms): Feels unresponsive
  • 500ms: Good balance for typical edit workflows

Incremental Updates ​

When files change, Locus updates only affected chunks rather than rebuilding the entire index.

Update algorithm:

  1. Detect change: File watcher emits event
  2. Debounce: Wait 500ms for additional events
  3. Hash comparison: Check if content actually changed
  4. Delete old chunks: Remove chunks for changed file
  5. Re-chunk: Parse file into new chunks
  6. Update vectors: Compute TF-IDF vectors for new chunks
  7. Rebuild IDF: Recalculate global IDF scores (affected by all changes)
  8. Recalculate TF-IDF: Update TF-IDF scores using new IDF
  9. Update metadata: Recalculate chunk magnitudes and average doc length

Change detection flow:

typescript
async processFileChange(type: 'add' | 'change' | 'unlink', path: string) {
  if (type === 'unlink') {
    // File deleted
    await storage.deleteFile(path)
    await rebuildIndex()
    return
  }

  // File added or changed
  const content = await fs.readFile(absolutePath, 'utf-8')
  const newHash = simpleHash(content)

  // OPTIMIZATION: Compare hash to detect real changes
  const existingFile = await storage.getFile(path)
  if (existingFile && existingFile.hash === newHash) {
    console.log('File unchanged (same hash), skipping')
    return  // No actual change
  }

  // Content changed, update index
  const file: CodebaseFile = {
    path,
    content,
    hash: newHash,
    size: stats.size,
    mtime: stats.mtime,
    language: detectLanguage(path)
  }

  await storage.storeFile(file)
  await rebuildIndex()
}

Hash-based optimization:

File watcher events trigger on mtime changes, but content may be unchanged (e.g., file touched without edits). Hash comparison prevents unnecessary reindexing.

typescript
// Simple hash function (fast, not cryptographic)
function simpleHash(content: string): string {
  let hash = 0
  for (let i = 0; i < content.length; i++) {
    hash = ((hash << 5) - hash + content.charCodeAt(i)) | 0
  }
  return hash.toString(36)
}

Event Handling ​

@parcel/watcher emits events for file system changes. Locus maps these to index operations.

Event types:

Watcher EventLocus TypeAction
createaddIndex new file
updatechangeRe-index changed file
deleteunlinkRemove file from index

Subscription:

typescript
import * as watcher from '@parcel/watcher'

async startWatch(): Promise<void> {
  this.watcher = await watcher.subscribe(
    this.codebaseRoot,
    (err, events) => {
      if (err) {
        console.error('[WARN] File watcher error:', err.message)
        return
      }

      for (const event of events) {
        const absolutePath = event.path
        const relativePath = path.relative(this.codebaseRoot, absolutePath)

        // Skip ignored files
        if (this.shouldIgnore(relativePath)) {
          continue
        }

        // Map event type
        const eventType =
          event.type === 'create' ? 'add' :
          event.type === 'delete' ? 'unlink' :
          'change'

        this.handleFileChange(eventType, absolutePath)
      }
    },
    {
      // Auto-detect best backend (FSEvents, inotify, etc.)
      backend: undefined,

      // Ignore common directories
      ignore: [
        '**/node_modules/**',
        '**/.git/**',
        '**/dist/**',
        '**/build/**',
        '**/.next/**',
        '**/.turbo/**',
        '**/.cache/**',
        '**/coverage/**',
        '**/*.log',
      ]
    }
  )

  this.isWatching = true
  console.error('[SUCCESS] File watcher started (native FSEvents)')
}

Event batching:

@parcel/watcher batches events internally and delivers them in groups, reducing overhead.

typescript
// Single callback receives all events in a batch
(err, events) => {
  // events = [
  //   { type: 'create', path: '/path/to/file1.ts' },
  //   { type: 'update', path: '/path/to/file2.ts' },
  //   { type: 'delete', path: '/path/to/file3.ts' }
  // ]
}

Watch Lifecycle ​

Starting watch mode:

typescript
import { CodebaseIndexer } from '@sylphx/coderag'

const indexer = new CodebaseIndexer({
  codebaseRoot: '/path/to/project',
  watch: true,  // Auto-start watching after indexing
  onFileChange: (event) => {
    console.log(`File ${event.type}: ${event.path}`)
  }
})

await indexer.index()
// Watcher automatically started after indexing completes

// Or start manually:
await indexer.startWatch()

Stopping watch mode:

typescript
await indexer.stopWatch()
// Cleans up:
// - Unsubscribes from watcher
// - Clears pending update timers
// - Releases file handles

Watch state:

typescript
const isWatching = indexer.isWatchEnabled()
// Returns: true if watching, false otherwise

Ignore Patterns ​

Locus respects .gitignore and custom ignore patterns.

Gitignore support:

typescript
import { loadGitignore } from '@sylphx/coderag/utils'

const ignoreFilter = loadGitignore('/path/to/project')

// Check if file should be ignored
if (ignoreFilter.ignores('node_modules/package.json')) {
  console.log('File ignored')
}

Uses ignore library (same as git).

Built-in ignore patterns:

@parcel/watcher automatically ignores common directories:

typescript
ignore: [
  '**/node_modules/**',  // Dependencies
  '**/.git/**',          // Git metadata
  '**/dist/**',          // Build output
  '**/build/**',         // Build output
  '**/.next/**',         // Next.js cache
  '**/.turbo/**',        // Turborepo cache
  '**/.cache/**',        // General cache
  '**/coverage/**',      // Test coverage
  '**/*.log',            // Log files
]

These are in addition to .gitignore patterns.

Custom ignore filter:

typescript
private shouldIgnore(relativePath: string): boolean {
  // Skip empty paths
  if (!relativePath) return true

  // Check gitignore
  if (this.ignoreFilter?.ignores(relativePath)) {
    return true
  }

  // Custom rules
  if (relativePath.startsWith('.')) {
    return true  // Ignore hidden files
  }

  return false
}

Performance Characteristics ​

Event latency:

PlatformNative APILatencyCPU Usage
macOSFSEvents50-100ms<1%
Linuxinotify10-50ms<1%
WindowsReadDirectoryChanges100-200ms<2%

Overhead:

  • No polling (0% CPU when idle)
  • Kernel-level notifications (efficient)
  • Minimal memory footprint (~1-2MB)

Scalability:

@parcel/watcher handles large directory trees efficiently:

FilesMemoryEvent Rate
1k files1MB1000+ events/sec
10k files2MB1000+ events/sec
100k files5MB500+ events/sec

Example Usage ​

Basic watching:

typescript
import { CodebaseIndexer } from '@sylphx/coderag'

const indexer = new CodebaseIndexer({
  codebaseRoot: './src',
  watch: true,
  onFileChange: (event) => {
    console.log(`[${event.type.toUpperCase()}] ${event.path} at ${new Date(event.timestamp).toISOString()}`)
  }
})

// Index and start watching
await indexer.index()

// Keep process alive
process.on('SIGINT', async () => {
  await indexer.stopWatch()
  process.exit(0)
})

Manual watch control:

typescript
const indexer = new CodebaseIndexer({
  codebaseRoot: './src',
  watch: false  // Don't auto-start
})

await indexer.index()

// Start watching later
console.log('Starting file watcher...')
await indexer.startWatch()

// Perform searches...
const results = await indexer.search('async function')

// Stop when done
await indexer.stopWatch()

File change callbacks:

typescript
const indexer = new CodebaseIndexer({
  watch: true,
  onFileChange: async (event) => {
    if (event.type === 'add') {
      console.log(`New file: ${event.path}`)
    } else if (event.type === 'change') {
      console.log(`Modified: ${event.path}`)
      // Notify external service
      await notifyWebhook({ type: 'file_changed', path: event.path })
    } else if (event.type === 'unlink') {
      console.log(`Deleted: ${event.path}`)
    }
  }
})

Conditional watching:

typescript
// Only watch in development
const isDevelopment = process.env.NODE_ENV === 'development'

const indexer = new CodebaseIndexer({
  watch: isDevelopment,
  onFileChange: isDevelopment ? (event) => {
    console.log(`[DEV] File changed: ${event.path}`)
  } : undefined
})

Released under the MIT License.