Cursor indexing that works on large codebases

A concrete indexing and context blueprint for making Cursor reliable on large monorepos: what to index, what to ignore, and how to repair stale context.

Cursor indexing on big repos: the short version If your repository is more than a small app, turning on Cursor’s codebase indexing and spamming @codebase is the wrong move. Cursor builds an embedding index of every indexed file and then pays to send chunks of it into a ~200k token context window for each request, according to its token‑based pricing, where you are billed on total tokens processed per request rather than a flat request count. On a large existing repo, that only stays reliable and affordable if you: Index only behavioural source-of-truth (core src/ , services, tests, key docs). Aggressively ignore build artefacts, dependencies, generated code, logs, data and secrets. Default to surgical context ( @file , @folder , Inline Edit) and reserve @codebase for diagnostics. Treat stale index repair (reindex, deletes, Rules updates) as a normal ops task after big changes. This guide walks through that configuration step by step, using a realistic SaaS monorepo as the running example. Why Cursor’s codebase indexing matters for large existing repositories Cursor’s own description of securely indexing large codebases is clear: when codebase indexing is enabled, it uploads contents-derived data from every file in your project to build a semantic index. That index powers fast repository-level understanding and search, instead of the model re-reading the entire repo each time. Internally, Cursor computes embeddings per file as part of this index, as described in the official Cursor Materials . Those embeddings are then used whenever you: call @codebase in chat, ask questions like “where is X implemented?”, run multi-file edits or refactors guided by the agent, use semantic search. Cursor’s docs on working with context describe ‘context’ as combining your current instructions with the state of your codebase, and they recommend using targeted @code , @file and @folder handles instead of relying solely on automatic retrieval. That recommendation gets more important as your repo size grows. Cursor and third‑party case studies describe it being used on very large repositories in practice, but specific file counts like “550,000+ files” currently come from secondary reports rather than Cursor’s own documentation and should be treated as illustrative, not official capacity guarantees. With token-metered pricing , every extra chunk the model reads adds cost. Cursor’s pricing policy explains that Model API fees and any Cursor Token Rate surcharges are calculated on the basis of total tokens (prompt plus completion) multiplied by the applicable per‑token prices. Larger and noisier context → more tokens → higher bill, even if the answers are worse due to irrelevant retrieval. If you are still deciding whether to adopt Cursor at all, especially for large repositories, the broader trade-offs against Copilot are covered in the GitHub Copilot vs Cursor comparison , the general AI coding stack guide , and the focused Cursor review . This article assumes Cursor is chosen and focuses purely on how to configure it on a real production repo. Mental model: how Cursor builds and uses a repo index What happens when you enable codebase indexing When you open a project in Cursor and enable codebase indexing, Cursor scans the workspace folders (VS Code concept), discovers files, and uploads contents-derived data to its backend endpoint at api2.cursor.sh for embedding and search, as documented by the secure indexing blog and a public capture of indexing traffic on GitHub. The Working with Context guide visually reinforces how @file and @folder provide targeted, surgical context instead of relying solely on broad @codebase retrieval in large repositories. According to Cursor’s materials, it creates embeddings for each file once. That gives you a semantic “map” of the repo that can be searched and retrieved much faster than raw file reads. Incremental updates: saves, deletes, renames After the initial pass, Cursor performs incremental updates. The secure indexing blog notes that it avoids full rebuilds for each user/machine by updating only changed files: File save : new contents-derived data is uploaded, embeddings refreshed. File delete : the indexed representation is removed. File rename / move : the index is updated to point to the new path. Branch switches and large refactors are just combinations of these events. In practice, directory-level moves and massive deletions can leave the index in a temporarily inconsistent state until Cursor has processed all changes. Where to control indexing and ignores Cursor’s current documentation shows indexing controls under Cursor Settings, in the Indexing & Docs section, where you can enable or disable indexing, view sync state, configure ignored files/patterns, and delete a project’s index. This is where you treat indexing as a curated retrieval layer, not a mirror of git ls-files . Think of it as a more security- and cost-sensitive twin of .gitignore , and as one piece of a wider AI repo setup alongside guardrails like those in Codex repository setup for existing prod repos. How the index actually feeds Cursor’s AI Once the embeddings exist, the index is used in several ways: @codebase : tells Cursor to pull semantically relevant chunks from the whole indexed repo. Search / "find where X happens" : uses the embeddings to locate usages or definitions. Repository-level chat : questions like “How does billing work?” are answered by retrieving relevant files from the index. Multi-file edits : when planning a change, the agent can look up related files via the index. Cursor’s models page indicates that Cursor normally uses a context window of around 200k tokens for its default agents and chat models. @‑context that the agent deems relevant is drawn from the index and included in that budget, subject to Cursor’s own trimming and summarization heuristics. If your index is full of noisy, huge, or misleading files, those are more likely to crowd out the useful ones. Inline Edit, documented in the Inline Edit overview , adds another nuance: when you use Cmd/Ctrl+K on a selection, Cursor automatically includes the selected code and surrounding context. That is already a form of surgical context, independent of @codebase . In production workflows this pairs well with a safe PR-driven process like the one in the AI coding agent PR workflow . A concrete blueprint: what to index in a mature SaaS monorepo Consider a realistic monorepo: The Rules documentation illustrates how project rules in .cursor/rules act as long-lived constraints and guidance for the AI, aligning with the article’s recommendation to encode policies like preferring auth_v2 over legacy auth_v1. apps/web – Next.js frontend (TypeScript) services/api – Node or Go backend services/worker – Python job workers packages/shared – shared TypeScript utilities infra – Terraform / Pulumi / CloudFormation docs – ADRs and domain design docs In practice, the most useful configuration is for Cursor’s index to primarily capture the behavioural source‑of‑truth (the code that actually runs in production) and the closest explanatory docs, rather than everything in the repo. A practical default: Always index : apps/web/src , apps/web/app services/**/src or cmd / internal (for Go) packages/shared/src tests , __tests__ , e2e where they directly describe behaviour docs/adr , docs/domain , migration docs tightly coupled to current code infra/app , infra/services modules that define current deployed topology Maybe index (case by case): Legacy modules that are still read but rarely changed One canonical source directory for generated types if the generation pipeline is not easily discoverable Language- and stack-specific guidance Normalised across common stacks, a conservative “index these by default” list for a production monorepo is: Stack Index by default Notes TypeScript / JS (Next.js, Node) src , app , pages , components , lib , tests, domain docs Include next.config.* , tsconfig.* , eslint / jest configs in root as they affect behaviour. Python Package directories ( pkg_name/ ) excluding __pycache__ , src , tests Index pyproject.toml , setup.cfg , linters. Go cmd , internal , pkg , config , tests Infra modules that wire services should be indexed. Java / Kotlin src/main , src/test , main config files Exclude target , build , generated sources unless essential. Ruby / Rails app , lib , config , db/migrate , tests Index Gemfile , routes.rb for routing context. PHP / Laravel app , routes , config , tests Index main .env.example (without secrets) if it encodes config patterns. Monorepos and workspaces For Nx/Turborepo/Lerna or similar: Treat each apps/* and packages/* subtree as a separate mental “project”, but still within one Cursor workspace. Index only the apps and packages that are actively developed. Old experiments can be excluded. Keep shared packages indexed if they are commonly edited; otherwise, index only their public API and key implementations. When deciding whether to apply heavy curation per workspace or a simple global pattern, follow the “what changes the decision” rule from the brief: if the repo has clearly separated workspaces with distinct teams, per-workspace indexing and ignore strategies are usually superior. What to exclude aggressively (and why) Cursor’s indexing controls support ignored files and patterns, configured via a Cursor‑specific ignore mechanism (now documented under Cursor Settings → Indexing → Ignore Files ), which plays a similar role to .gitignore but with its own semantics for how patterns apply to indexing. For a large existing codebase, this is where most of the economic and quality gains sit. Cursor’s official ignore-file documentation shows how to curate what the index can see, mirroring the article’s advice to aggressively exclude high-churn, low-signal paths from codebase indexing. High-churn, low-signal directories These almost always belong in the ignore list: Directory / pattern Why ignore node_modules , .pnpm , .yarn/cache Third-party code you rarely want AI to edit; huge token sink; noisy retrieval. dist , build , .next , .nuxt Build artefacts; regenerate instead of editing; high churn and duplication. .venv , .tox , .pytest_cache Python environments and caches, no value for semantic understanding. coverage , target , bin / obj Compiler or test artefacts; noisy and redundant. vendor (where it holds vendored third-party code) Same rationale as dependencies; keep out unless it’s truly first-party. logs , tmp , cache Runtime logs and temporary files; risk of leaking data; irrelevant for reasoning. Security- and privacy-sensitive areas Cursor’s secure indexing article explains that code is processed into contents-derived data on Cursor’s infrastructure. Even when designed securely, organisations with strict policies will not want obvious secrets and data dumps embedded. As a default, always ignore : .env , .env.* , any plaintext secrets files secrets , keys , or credential directories customer data exports (CSV, Parquet, JSON dumps) analytics or event logs that contain PII The decision flips further towards maximal exclusion when compliance prohibits this data leaving your network at all. In that case, even anonymised dumps and some vendor trees should be excluded, as described in the brief’s “security-first” scenario and the more general security guidance in Codex production repository guardrails . Generated code and SDKs Generated protobufs, OpenAPI clients, ORM models, and large language bindings can overwhelm retrieval. Over-indexing them has two consequences: Semantic similarity often pulls in generated files instead of their human-authored source, diluting answers. When @codebase is used, large generated trees can dominate the token budget. The practical approach: Ignore entire generated directories by default. If AI must edit some generated code directly (for pragmatic reasons): include only that subtree, or index a single canonical generated output that accurately reflects the interface, and document in Rules where the true source of generation lives. Binary and large data files Binary assets and large data files (images, models, large CSV/Parquet) are poor candidates for semantic embedding. They slow initial indexing and do not meaningfully contribute to code understanding. Default to ignoring: *.png , *.jpg , *.gif *.pdf (unless they are key design docs and you explicitly want them embedded) *.csv , *.parquet , *.db , *.sqlite ML models ( *.pt , *.onnx , etc.) Pinning and surgical context: using @file, @folder, @codebase and Rules Cursor’s “working with context” guide says to prefer targeted @file , @folder and @code to relying on automatic context gathering. On a large existing repo, that becomes a discipline. How the @-context handles differ @file : include a specific file in the AI request. @folder : include files from a specific folder (usually a small subtree). @codebase : ask Cursor to retrieve from anywhere in the indexed repo. @workspace (when available): focus on the current workspace root rather than the full repo. On top of that, Inline Edit automatically includes selected code plus surrounding context, and is ideal for localised changes. Rules as persistent “guardrails” Cursor provides project-level Rules which act as persistent prompt context attached to the workspace. They can encode durable constraints such as: auth_v1 is legacy; prefer auth_v2 for new work. Do not modify files under legacy/ unless explicitly asked. When updating API handlers, also update their corresponding tests in tests/api . Rules are particularly effective during long-running refactors or migrations, where stale code remains indexed but should be deprioritised. For a deeper pattern library of production-safe rules, see the dedicated Cursor Rules playbook. Screenshot guidance: using surgical context Suggested screenshot: a Cursor chat window where the user types something like “Refactor this controller to use the new service” and adds @file apps/web/src/controllers/user.ts and @folder packages/shared/src/services/user . The screenshot should highlight the @file and @folder annotations as examples of surgical context, rather than @codebase . Operational blueprint for a large SaaS monorepo This section applies the rules above as a concrete configuration for the example monorepo. Step 1 – Configure indexing and ignores Open the monorepo in Cursor. Go to Settings → Features → Codebase Indexing . Enable indexing for the workspace. In “Ignored files/patterns”, add at least: **/node_modules/** **/dist/** , **/build/** , **/.next/** **/.venv/** , **/.tox/** , **/__pycache__/** **/coverage/** , **/target/** , **/logs/** , **/tmp/** **/.env* , **/secrets/** , **/data_dumps/** **/generated/** , unless there is a clear reason to include a subset Confirm that core directories ( apps/web/src , services/**/src , packages/shared/src , docs/adr ) are not in the ignore list. Suggested screenshot: Cursor’s Codebase Indexing settings panel showing: Indexing enabled and sync state healthy. Ignored patterns list populated as above. Step 2 – Encode long-lived Rules Next, open Settings → Rules for the project and define a small set of durable rules such as: “ apps/web is a Next.js app using the app router; follow existing patterns in apps/web/app .” “ services/api is the main public API; services/worker handles async jobs; don’t create new job queues from the API layer.” “ auth_v1 is legacy; prefer auth_v2 . Don’t create new calls to auth_v1 .” “When you change behaviour in a handler, also update its tests under tests/ in the same package.” Suggested screenshot: the Rules configuration panel showing these examples, with one rule specifically encoding “ auth_v1 is legacy, prefer auth_v2 ”. Step 3 – Context discipline by workflow Use different context strategies for common workflows: Feature work in a known area (e.g. expanding a dashboard): Open the main component and related service files. Use Inline Edit ( Cmd/Ctrl+K ) plus @file for any adjacent utilities. Avoid @codebase ; lean on Rules for broader guidance. Debugging a specific bug : Start from the failing file/test; add @file for the handler, service and repository layer. Add @folder for a narrow subtree ( services/api/src/auth ), not the entire repo. Only use @codebase when you truly don’t know where a behaviour lives. Cross-cutting refactor (e.g. renaming a core type): Use search to locate the core definition and index it via @file . Use @folder on key packages that depend on it. For initial scoping, one @codebase query is acceptable; then work locally. This approach aligns with Cursor’s guidance to combine intent and state deliberately rather than letting automatic retrieval run the show. It also matches the PR-first safety patterns in the AI coding agent PR workflow and the broader AI development workflow . Token cost implications: why curation matters Cursor’s pricing policy explains that model API fees are based on total tokens (prompt + completion) multiplied by the model’s pricing rate, and that on some paid plans Cursor also applies its own per‑million‑token ‘Cursor Token Rate’ surcharge on top of the underlying model cost. The exact token counts per request are not published, so the analysis here is directional. Scenario 1 – Naive @codebase on full monorepo Assume a large monorepo where indexing includes everything: source, dependencies, build artefacts, generated code. When you ask for “Refactor billing to support coupons” with @codebase : Cursor queries the embedding index across hundreds of thousands of files. It selects many semi-relevant chunks (old generated SDKs, legacy billing paths). Those chunks occupy a large portion of the ~200k token window. The underlying model reads and processes substantially more tokens than necessary, and you pay per token. Responses may be slower and less accurate, because the model’s attention is diluted by noise. Scenario 2 – Curated index with occasional @codebase Same question, but the repo is indexed per the blueprint above (only core behavioural code and key docs): The embedding search runs over an order-of-magnitude smaller set of files. Retrieved chunks are more likely to be human-authored, up-to-date code. The overall token count per request drops because fewer, more relevant chunks are pulled. You still have the convenience of @codebase , but retrieval is better and cheaper. Scenario 3 – Curated index and surgical context only If you avoid @codebase almost entirely and rely on Inline Edit plus @file / @folder , the model context typically only includes: the files you’re actively editing, a small surrounding neighbourhood, any pinned Rules. That constrains tokens per request close to the minimum required for accurate edits. Under token-based billing, this directly reduces monthly spend for the same or better quality edits. Cursor’s pricing deep dive covers the token mechanics and Cursor Token Rate in more detail. The important point for indexing strategy is that retrieval noise converts directly into token waste. Detecting and repairing stale or misleading context Academic work on repository-level editing has shown that retrieving stale snippets from older repo states can materially hurt completion quality. One paper on “When Retrieval Hurts Code Completion” reports that retrieval of outdated code leads to worse suggestions compared with no retrieval at all. Another, on the importance of reasoning for retrieval, shows that intelligent context selection is critical on large repos. These findings align with what many teams observe when Cursor keeps pulling in deleted or deprecated code after a big change. How stale index issues show up Common symptoms that your Cursor index is out of date: Cursor suggests imports from files that were deleted or renamed. It references functions or types that no longer exist on your branch. @codebase answers mention v1 modules after a migration to v2 that has been merged. Semantic search returns results in directories you know are dead. Triage: is it index state or current branch? Before forcing a full reindex: Check that your local working copy is clean and on the expected branch. Confirm that the misleading files are genuinely gone or marked as legacy. Check Codebase Indexing settings to ensure those directories are either included or ignored as intended; misconfigured ignore patterns can cause partial visibility. Repair playbook Once you are confident the repo is correct, but the index is stale: Go to Settings → Features → Codebase Indexing for the project. If only a subset of paths are problematic (e.g. services/api/auth_v1 ): Temporarily add them to the ignore list. Trigger a reindex or wait for incremental sync. Optionally remove them from the ignore list if they must remain visible as legacy but should be deprioritised via Rules. If the whole repo has undergone a major reorganisation (mass renames, folder moves): Use the per-project delete option described in the mirrored docs to delete the existing index. Re-enable indexing to force a fresh full pass based on the new structure. Update Rules to encode any migrations (e.g. “Prefer auth_v2 ; treat auth_v1 as legacy.”). This actively repairs misleading embeddings and aligns AI behaviour with the new repository reality, rather than waiting for slow, partial incremental updates to converge. When to build index health checks into operations It is practical to treat index maintenance like any other repo-level operational task. A simple decision tree: Minor changes (feature branches, small refactors): rely on incremental indexing; no action. Large refactor within a directory (e.g. services/api/auth ): after merge, trigger a per-project reindex and verify that semantic search aligns with the new structure. Mass renames / moves (monorepo rearrange, workspace split): delete and recreate the index. Branch divergence for long-lived feature branches : consider disabling indexing on experimental branches if they drift far from trunk and AI confusion becomes a problem. For enterprises, this cadence pairs well with the idea in the brief: slightly more operational overhead, but lower risk of AI-generated regressions based on stale code. Mapping repo size and shape to indexing complexity Repo type Typical size Recommended indexing strategy Small app Tens of thousands of lines, single app Index almost everything except obvious artefacts/deps; minimal ignore list; light use of @codebase . Medium monolith Hundreds of thousands of lines, one main service Curated index of core source + tests + docs; aggressive ignores; routine use of @file / @folder ; limited @codebase . Large monorepo Millions of lines, multiple apps/services Heavy curation; per-workspace strategy; strong Rules; @codebase mainly for discovery; regular reindex after big merges. The brief’s “what changes the decision” conditions apply directly: smaller repos justify simpler indexing, while large monorepos repay the complexity of a curated retrieval layer. When this strategy might not be worth it There are cases where full-blown curation is overkill: If you are working on a small or medium-sized app with a relatively clean structure, the risk of noisy retrieval is lower. In that context, indexing almost everything (minus obvious artefacts and secrets) and using @codebase freely is acceptable. If the team rarely uses repository-wide features and mostly relies on Inline Edit and local @file , index curation delivers diminishing returns. Focus on rules and habits instead. If security policy forbids any external indexing, your problem is deployment architecture, not ignore patterns. In that case, consider local-only tools or terminal-first agents like Claude Code, as discussed in Cursor vs Claude Code vs terminal agents and the related comparison in Claude Code vs Codex . Treat Cursor indexing as production infrastructure Cursor’s codebase index is not a cosmetic feature; it is a persistent semantic layer that governs what the AI can