Full-stack observability
for production
language models.
AI Signals™ gives engineering and compliance teams end-to-end visibility into LLM calls, prompt versions, and evaluation outcomes — so you can ship AI confidently and govern it continuously.
Beta access open. AI Signals™ is in active Beta. Core tracing, prompt management, and evaluation pipelines are production-ready. Join the Beta to get early access and influence the roadmap.
Beta includes
- End-to-end LLM call tracing (OTLP)
- Prompt versioning with approvals
- LLM-as-judge evaluation pipelines
- Human feedback & labeling queues
- Dataset regression runs
- AI Security Core — REDTEAM, SAST, MODEL_SCAN, MCP TEST
- MCP Governance — server inventory & policy enforcement
- Reports — OWASP LLM TOP 10, NIST AI RMF, EU AI ACT
- OpenAI, Anthropic, Bedrock support
Six modules. Full AI governance lifecycle.
AI Signals™ covers observability, prompt governance, evaluation, AI security testing, MCP governance, and compliance reporting — in one platform.
Observability
Capture every LLM call, retrieval step, tool invocation, and agent action with structured spans, session replay, and model cost tracking.
- OTLP-compatible trace ingestion
- Token usage & latency percentiles
- Multi-step agent trace stitching
- Session-level replay viewer
Prompt Management
Version-control prompts with approval workflows, rollback, and a live playground. Change audit trails satisfy compliance evidence requirements.
- Git-style prompt versioning
- Approval gates before production
- One-click rollback
- Playground with trace-linked runs
Evaluation
Score outputs with LLM-as-judge and custom metrics. Collect human feedback, build golden datasets, and run regression tests on every release.
- LLM-as-judge scoring
- Human annotation queue
- Dataset regression runs
- Model-to-model compare views
Security Core
Queue additive AI security runs without touching your Observe data flows. Target Agents, RAG pipelines, Prompts, Models, Repositories, or MCP Servers.
- REDTEAM, SAST, MODEL_SCAN runs
- MCP TEST & COMPARE run types
- Customer-managed runner provisioning
- Findings feed into reports
MCP Governance
Inventory MCP servers, enforce tool-access policy modes, and track operational health. Policy rule sets are JSON-defined and scoped per asset or project.
- Server registry (HTTP, SSE transports)
- Monitor, Audit, Block policy modes
- JSON allowTools / denyTools rule sets
- Operational health dashboard
Compliance Reports
Generate framework-mapped evidence reports from AI Signals findings and security run artifacts. Machine-collected evidence ensures reproducibility.
- OWASP LLM TOP 10 coverage
- NIST AI RMF 1.0 mapping
- EU AI ACT evidence bundles
- Exportable PDF & JSON reports
Built for AI teams that ship to production.
From developer-level tracing to CISO-level audit evidence — one platform for the full AI governance lifecycle.
LLM Call Tracing
Instrument any model provider — Anthropic, OpenAI, AWS Bedrock, Azure OpenAI, or open-source — and capture structured traces with zero code change via SDK auto-instrumentation.
Prompt Version Control
Store, diff, and deploy prompt templates with semantic versioning. Gate production deploys behind approval workflows. Roll back in one click when output quality regresses.
LLM-as-Judge Scoring
Configure a judge model to score outputs on correctness, tone, groundedness, and custom rubrics. Run automatically on every trace or on-demand against evaluation datasets.
Human Feedback Loop
Route flagged outputs to labeling queues for SME review. Feed structured feedback back into eval pipelines for continuous improvement and compliance evidence.
Interactive Playground
Iterate on prompts and model parameters in a side-by-side playground that integrates directly with your prompt library and logs every experiment as a traceable run.
Python & JS/TS SDKs
Drop-in SDKs for Python and TypeScript with OpenTelemetry-compatible tracing. LangChain, LlamaIndex, and Vercel AI SDK callbacks included.
AI Security Runs
Queue REDTEAM, SAST, MODEL_SCAN, MCP TEST, and COMPARE runs against Agent, RAG, Prompt, Model, Repository, or MCP Server targets — without touching your live Observe streams.
MCP Policy Enforcement
Register MCP servers by endpoint, set per-asset policy modes (Monitor, Audit, Block), and define JSON rule sets controlling which tools agents may call at runtime.
Framework Evidence Reports
Generate reproducible compliance evidence reports mapped to OWASP LLM TOP 10, NIST AI RMF 1.0, and EU AI ACT — built entirely from machine-collected findings, not manual notes.
Evidence-ready AI governance for auditors.
AI Signals™ generates machine-collected compliance reports and verifiable audit trails for every prompt change, security run, and model deployment — aligned to the frameworks regulators are asking about now.
OWASP LLM TOP 10
Security risk framework for LLM-powered applications
- Security Core runs (REDTEAM, SAST, MODEL_SCAN) generate findings mapped to OWASP LLM TOP 10 categories
- One-click evidence report generation from accumulated security run results
- Framework coverage score tracks which TOP 10 risks have tested controls
NIST AI RMF 1.0
AI Risk Management Framework — Govern, Map, Measure, Manage
- Evaluation metrics mapped to AI RMF Measure function outcomes
- AI system documentation for the Govern and Map functions via trace metadata
- Human feedback and labeling align to the Manage function's oversight requirements
EU AI ACT
Regulation (EU) 2024/1689 — Risk classification and compliance obligations
- AI Signals reports generate EU AI ACT evidence bundles from machine-collected security and evaluation runs
- Prompt governance change logs provide technical documentation for high-risk AI system dossiers
- Human annotation and oversight workflows satisfy Articles 14/16 human oversight obligations
SOC 2 — AI Service Controls
Trust Services Criteria for AI-assisted systems
- Centralizes LLM traces, prompt change logs, and eval outcomes as evidence artifacts
- Tracks approval chains for prompt promotions (CC6 change management)
- Provides export-ready audit bundles for SOC 2 Type II periods
Answers for AI, security, and procurement teams.
What model providers are supported?
AI Signals™ works with any OTLP-compatible provider: Anthropic Claude, OpenAI GPT-4+, AWS Bedrock (Claude, Titan, Llama), Azure OpenAI, Google Vertex AI, and open-source models via LiteLLM. Framework callbacks for LangChain, LlamaIndex, and Vercel AI SDK ship out of the box.
How is trace data stored and what are the privacy controls?
Traces are stored in AiVRIC's ISO 27001-aligned infrastructure with AES-256 at rest and TLS in transit. Configurable PII redaction masks sensitive fields before ingestion. Self-hosted deployment keeps all trace data within your network. Review Trust Center.
What deployment options are available?
AiVRIC SaaS (multi-tenant, managed), single-tenant SaaS (your cloud account), self-hosted Docker/Kubernetes (air-gapped supported). The Beta program runs on managed SaaS; self-hosted tiers are planned for GA.
How does SDK instrumentation work?
The Python and JS/TS SDKs use OpenTelemetry-compatible spans. One import and one init call instruments all supported model provider clients automatically — no manual span creation needed for standard LLM calls.
Can evaluation results be used as SOC 2 evidence?
Yes. AI Signals™ exports evaluation run results, prompt change logs, and human review records as timestamped PDF and JSON bundles. These are accepted as SOC 2 Type II operational evidence for AI-assisted system controls.
Is there SSO and RBAC support?
SAML 2.0 and OIDC SSO are supported (Okta, Azure AD, Google Workspace). Role-based access controls with custom roles for viewers, evaluators, prompt owners, and admins. SCIM provisioning is on the GA roadmap.
What Security Core run types and targets are available?
Security Core supports five run types: REDTEAM (adversarial prompt attack simulation), SAST (static analysis of prompts and pipeline code), MODEL_SCAN (model safety and alignment checks), MCP TEST (policy gate verification for MCP servers), and COMPARE (side-by-side safety comparison across model versions). Targets include Agent, RAG, Prompt, Model, Repository, and MCP Server. Customer-managed runners can be provisioned for private environments.
How does MCP Governance work?
Register any MCP server by name, endpoint URL, and transport (HTTP or SSE). Set a policy mode per server or per project scope — Monitor (observe only), Audit (log all tool calls), or Block (enforce allow/deny rules). Policy rule sets are defined in JSON with allowTools and denyTools arrays. v1 focuses on observe and policy gate flows; full server lifecycle management is on the roadmap.
What compliance report formats does AI Signals™ generate?
Reports are generated from machine-collected evidence only — security run findings, evaluation outcomes, and prompt change artifacts — so they remain reproducible and auditor-verifiable. Reports are downloadable as PDF and JSON. Current framework templates: OWASP LLM TOP 10, NIST AI RMF 1.0, and EU AI ACT. Framework Coverage scores per framework are shown on the Reports dashboard.
Ship AI you can govern and audit.
AI Signals™ Beta is open. Join now to get early access, shape the roadmap, and start building AI observability into your development workflow.