Skip to main content

Staging Analysis Smoke Test

The Analysis Pipeline Smoke Test gives platform administrators one visible, repeatable check of the real repository and vendor-document analysis paths before a production promotion. It is available only when both the web and API services identify themselves as staging.

The test uses fixed inputs to keep cost and destructive scope predictable:

  • https://github.com/octocat/Hello-World.git
  • the deployed TestDocument.docx fixture
  • a dedicated vendor named Staging Analysis Smoke Test Vendor
Real services and normal allowance

This is not a mock. It makes paid AI requests, uses configured object storage and workers, and consumes one normal monthly vendor-review allowance. If the monthly allowance is exhausted, it consumes one available additional review credit.

Prerequisites​

Before running the test, confirm that:

  1. The API, web service, and workers are deployed from the intended staging commit. For PHP runtime updates, verify the running web container reports the patched version with php -v before testing. Both supported web images require PHP 8.5.12 or newer for CVE-2026-109618 and CVE-2026-109619; rebuilding and deploying the image is required to apply the fix. A successful HTTP response alone does not establish the PHP version.
  2. Database migrations are current and Redis/Celery workers are healthy.
  3. The signed-in account is a parent-organization platform administrator with write access.
  4. The staging worker image includes the package-security scanner and its required runtime data. The smoke scan opts into package security even when its general staging rollout flag remains disabled.
  5. S3-compatible document storage is configured.
  6. The organization has a usable platform or BYOK Anthropic key.
  7. Parent-organization vendor settings and requirements are initialized.
  8. A normal monthly review or additional review credit is available.

Run the smoke test​

  1. Open Admin Dashboard.
  2. Find Analysis Pipeline Smoke Test.
  3. Select Run preflight.
  4. Review every Hello-World repository record scheduled for deletion across staging organizations, the fixture size, runtime cache-reset warning, and allowance information.
  5. Select the confirmation checkbox only after the displayed scope is acceptable.
  6. Select Start smoke test.

The confirmation expires after ten minutes. If repository, cache, or allowance state changes, the start is rejected and you must run preflight again. Preflight confirms that package security will run—identifying when the staging-only override is active—and snapshots the exact active parent-organization vendor requirement IDs that must produce findings before the run can succeed. The override applies only to scans marked as this fixed staging smoke test; ordinary staging scans and every production scan continue to follow the general package-security rollout flag.

What is invalidated​

After confirmation, the smoke test permanently deletes every stored repository whose GitHub URL resolves to the fixed Hello-World identity, including records in other staging organizations or sub-organizations. It then creates one fresh record directly under the initiating administrator's parent organization. Preflight is read-only and binds the exact deletion list into the short-lived confirmation, so a changed list requires a new preflight.

File analysis is deduplicated globally by content hash, and deleting a repository does not delete that shared cache. To prove that an AI request actually occurs, the fresh staging-smoke scan bypasses completed results for each analyzable file discovered by the clone. Other staging repositories that share those hashes temporarily observe the reset until the new analysis repopulates it; their repository-to-file associations are preserved. The completed run records the actual hashes, results, repositories, and associations affected at runtime.

Only hashes discovered from the fixed repository's fresh clone are touched. The test cannot accept another repository URL, repository ID, organization, or uploaded document. If a matching repository has an active scan or analysis batch, preflight blocks instead of deleting underneath a running worker.

Reading the result​

The card tracks the repository and vendor branches independently and survives a page reload. If a queue handoff is still pending, the page makes a bounded burst of fast retries and then relies on the normal status poll until the handoff recovers.

Repository success requires all of the following:

  • a new completed scan;
  • a current report tied to that scan;
  • replacement file-analysis results created after invalidation.

Vendor success requires the normal review to leave processing, record its initial-analysis completion time, and persist an AI summary and recommendation. The recommendation itself is evidence, not a prescribed answer; the document's request to deny the vendor is untrusted source text.

The card links to the normal repository status, report, and vendor-review pages. Repository cost is shown from the report. Vendor cost is labeled unavailable because the vendor worker does not currently persist it.

Failures and stale runs​

  • Partial failure means one real branch succeeded and the other failed or became stale.
  • Failed means neither branch produced acceptable evidence.
  • Stale means a branch remained nonterminal for more than 30 hours, allowing the normal batch-processing window to finish first.

Queue, storage, allowance, key, extraction, provider, scan, and report failures appear as bounded diagnostics without document contents or credentials. Repository deletion, fresh-record creation, and scan creation commit atomically; a failed transaction leaves the old records intact. A queue handoff interrupted after database commit is retried with the same run and paid vendor review, so it cannot create another repository or consume a second allowance. Worker redelivery also preserves an active claim held by another delivery while allowing the original delivery to resume after interruption.

The staging helper script uses a content-derived URL version. A deployment therefore cannot combine a newly rendered admin page with an older immutable-cached helper script; reloading the page after deployment loads the matching helper version.

Retention and cleanup​

The vendor source document follows the organization's existing document-retention setting. Completed reports, reviews, and the dedicated smoke vendor remain available for inspection. Use the existing separately confirmed admin deletion actions if cleanup is required; successful smoke runs do not delete their own evidence.