Skip to main content

Pre-processing & Chunking Stage

For larger repositories, this platform runs an additional pre-processing & chunking stage before files are submitted to the AI for analysis. This page explains what happens during that stage, how it shows up in the UI, and what to do if it stalls.

Why pre-processing exists​

Submitting every code file in a large repository straight to the security-analysis AI would:

  • Exceed the per-request token limit, causing the AI to truncate files mid-way
  • Trigger LLM Guard scanner limits on oversized inputs
  • Result in unpredictable findings for files that are only partially analyzed

To avoid all of those problems, on the first analysis of each repository, every file passes through a pre-processing step that:

  1. Runs LLM Guard scanning to detect and reject clearly-malicious content (for example: prompt-injection patterns, secrets, known malware signatures)
  2. Counts tokens using Anthropic's per-request token accounting
  3. Skip-classifies files that are too small, too large, binary, or otherwise out of scope (the existing skip_reason values continue to apply)
  4. Chunks files larger than the per-request token limit into multiple analysis pieces so the AI can score them in full

This work runs on the worker between "file accepted" and "file submitted to Anthropic". It can take a few seconds per file, which adds up on a multi-thousand-file repo.

What you'll see in the UI​

A new Pre-processing status has been added between "Queued" and "Analyzing":

StatusWhat it means
QueuedFile accepted and waiting for the worker to pick it up
Pre-processingWorker is mid-flight running guardrails, token-limit checks, and chunking for this file
AnalyzingFile has been submitted to Claude; waiting on AI results
Complete / Failed / SkippedTerminal states — see View Security Reports

Where the Pre-processing count is shown​

  • Admin dashboard — a dedicated "Pre-processing" tile sits between the "Queued" and "Completed" tiles so you can see at a glance how many files are mid-flight.
  • Repository status page — each file row in the file list shows a PRE-PROCESSING badge (blue) while it is in this state.
  • Admin files view (admin_files.php?status=preprocessing) — a filterable view of every file currently in this state, useful for diagnosing stalls.

How long does it normally take?​

The Pre-processing stage is fast per file (typically under a few seconds), but on a large repository the total time can be substantial. The dashboard counts down as files complete pre-processing; if the count is stuck on a non-zero value for more than ~30 minutes, that's a strong signal something has gone wrong (see Troubleshooting below).

What happens after pre-processing​

Once pre-processing finishes for a file:

  • If the file is clean and within the token limit → moves to Analyzing (the existing queued batch flow)
  • If the file is too large → chunked into multiple pieces, each analyzed separately
  • If the file is rejected by LLM Guard → marked as a skip with a guard-specific reason (llm_guard_blocked)
  • If the file is out of scope by size, type, or detection rules → marked as a skip with the existing reason

For a chunked file, RepoRisk combines the findings from every piece. It also selects the file summary deterministically from the chunk descriptions that best match the file's name and purpose, so an internal library section does not appear as the summary of the whole bundled file merely because that chunk finished last. Existing reports keep their saved summaries until the file is analyzed again.

The transition is automatic; no user action is required.

While these requests are being built, the platform keeps the entire batch in a protected preparation state. Scheduled submission recovery cannot send that batch early. Only after every request is stable does the creating worker reserve the batch and revalidate that every queued file is still assigned to it.

If a worker stops during preparation, automatic recovery claims the whole abandoned batch before returning all of its unfinished files to Queued. The replacement batch remains tied to the same scan, so it cannot select files from another scan or attribute results to the wrong report. Preparation and recovery cannot each take part of the same batch, so a completed AI batch cannot silently leave a file behind.

Troubleshooting​

The Pre-processing count won't move down​

If the Pre-processing tile has been non-zero for more than 30 minutes:

  1. Check the worker is alive — go to the Admin dashboard and confirm recent batch submissions have been logged
  2. In rare cases a worker mid-flight will be killed — automatic recovery atomically releases the entire abandoned preparation and returns its unfinished files to Queued. The next worker tick retries them as a new batch.
  3. If recovery isn't helping — contact support with the affected Repo ID

A file is permanently in Pre-processing​

If a single file has been in Pre-processing for much longer than expected:

  • The auto-recovery path (PREPROCESSING_STUCK_THRESHOLD_MINUTES = 30) will move it back to Queued and a later worker tick will retry it
  • If the same file keeps stalling, the file contents may be unusually large or contain sequences that cause timeout in token-counting — please report the file path to support

How do I re-queue Pre-processing files manually?​

Run the standard re-queue path on the Admin dashboard ("Re-queue All Stuck Files") — it covers Pre-processing rows in addition to Queued rows and clears stale completion markers before retrying. You can also target Pre-processing rows directly with admin_files.php?status=preprocessing followed by the file-level requeue button.