arya.ai

Synthesis
This feature automates model deployment spec generation for BFSI workflows using structured interviews, embedding regulatory guardrails by default. It is NOT a model validator, runtime monitor, or replacement for legal review. Our downside case: $568K/year at 40% adoption still justifies build costs (est. $310K).

Primary Metrics

Guardrail Metrics

What We Are NOT Measuring

1. Number of specs generated: Measures vanity output, not quality
2. Tool adoption rate: Confounds with org-specific rollout policies
3. Raw generation speed: Could incentivize skipping compliance checks

Strategic Decisions Log
Decision: Support non-BFSI use cases?
Choice: Phase 1: BFSI-only (credit, fraud, KYC)
Rationale: 78% of revenue from BFSI; generic solution increases compliance risk

Decision: Real-time schema validation?
Choice: Require sample data upload
Rationale: Prevents hypothetical schemas; rejected "manual entry only" as error-prone

Premortem
It is 6 months post-launch and this feature failed. Top 3 reasons:

1. Compliance teams rejected generated clauses due to missing RBI sub-section 5.8 updates
2. Users spent more time correcting schemas than writing specs manually
3. Competitor embedded equivalent feature in MLOps suite for free

What success looks like:
Product teams deploy models in 48 hours with zero compliance tickets. Engineering VP says: "We redirected 9 FTEs to high-impact work." Auditors cite arya.ai specs as compliance benchmarks.

Assumptions vs Validated

Core Objective
Generate production-ready deployment specs in under 1 hour that:

1. Reduce manual effort by 85% vs. current baseline (8.2 hours → 45 min)
2. Achieve ≥98% coverage of RBI/FATF compliance controls for target use cases
3. Embed P0 monitoring thresholds (e.g., "alert if PSI > 0.25")

Competitive Landscape

Our wedge is compliance-by-design because competitors treat regulation as afterthoughts.

Quantified Baseline

Before/After Narrative
Before: Priya (Lead Data Eng, NeoBank) spends 3 days manually translating a fraud model’s Python notebook into a 40-page deployment spec. She misses RBI’s new requirement to monitor gender bias in false negatives, causing a 2am incident when biased rejections spike.
After: Priya answers 6 questions about the fraud model’s inputs, outputs, and risk class. The generator produces a compliant spec with pre-configured bias monitors. She deploys in 4 hours with explicit signoff from Legal.

Adversarial Validation

1. Attack: User selects "fraud detection" but inputs mismatched data types
   Defense: Type coercion checks with user confirmations
   Limitation: Cannot resolve semantic mismatches (e.g., "income" vs. "revenue")

2. Attack: Malicious actor injects script tags in field descriptions
   Defense:* Sanitize outputs to plaintext
   Limitation:* Loses rich formatting

3. Attack: User omits high-risk dependencies (e.g., "no fallback needed")
   Defense:* Require justification for P1/P2 model exemptions
   Limitation:* Cannot force fallbacks for non-critical models

Phased Acceptance Criteria
Phase 1 — MVP (6 weeks)
US#1 — Generate input/output schemas

• Given user completes "Data Sources" section
• When system ingests sample payload
• Then output spec defines schema with 100% field coverage and data types
• Failure Consequence: Schema mismatches cause production data breaks
• Validated by QA against 12 real BFSI models

US#2 — Embed RBI bias checks

• Given user selects "credit scoring" use case
• When generator runs
• Then spec includes mandated SC/ST/OBC disparity monitors
• Failure Consequence: Regulatory action for bias violations
• Validated by Compliance against RBI Circular DBOD.No.FSD.BC.7/24.01.050/2022-23

Out of Scope (Phase 1)

Mandatory Oversight Points

1. Pre-deployment:

   • Legal must sign off on generated regulatory clauses
   • Senior ML engineer must certify fallback logic

2. Runtime:

   • Threshold breaches require human confirmation before model rollback
   • Compliance drift detected in >2% of inferences triggers spec review

Override Mechanics

• Users can edit ANY generated content (audit trail enabled)
• Edits to P0 clauses (bias monitors, fallbacks) require justification memo

Risk Register

1. Risk: Generator omits RBI Master Direction 5.2.3(c) for rural credit models
   Probability: Medium | Impact: High
   Mitigation: Preload jurisdiction-specific clauses (Owner: Compliance Lead by 9/30)
   Fallback: If unvalidated by deadline, restrict rural model deployment

2. Risk: Generated thresholds cause false alerts (e.g., PSI < 0.25 too sensitive)
   Probability: High | Impact: Medium
   Mitigation:* Embed threshold calculators for common metrics (Owner: ML Eng by 10/15)

3. Risk: Adversarial inputs exploit schema generator (e.g., fake field names)
   Probability: Low | Impact: Critical
   Mitigation:* Input sanitization with allowlists (Owner: Security by 9/25)

Kill Criteria

1. 15% of D90 specs require manual remediation for compliance gaps

2. Generated latency thresholds cause ≥5% false-positive alerts in first 200 deployments
3. RBI Digital Lending Guidelines update not incorporated within 14 days of publication

Embedded Controls

1. Credit Models:

   • Auto-add "rejection rate disparity by gender/region" monitors
   • Require explicit fairness thresholds (e.g., "≤1.5x disparity ratio")

2. KYC Models:

   • Flag document verification bias against non-English names
   • Enforce RBI circular 12/2023: "Alternative verification for low-income applicants"

Validation Protocol

• Generated specs tested against 15 historical bias incidents (e.g., rural loan rejections)
• 100% coverage of RBI’s "High-Risk AI Checklist" items 3.1-3.7