Live in productionv1.0.0Open source · Apache 2.0

Brok's Forge

The Engineering Platform for AI Agents.

Frameworks help you build an agent. Brok’s Forge is the platform that helps you engineer one — register it, version it, point real datasets and prompts at it, evaluate it against objective metrics, benchmark variants, catch regressions, and report on cost, latency and quality over time.

Open the live platform
Source & architecture docs
  • 22feature & infra modules
  • 27REST controllers
  • 29append-only migrations
  • ~335backend Java files
  • 13LLM providers modeled
  • 17architecture decision records
built by one engineer — documented like a team did it
Fig. 01 — watch a run

This is what the platform does.

One evaluation job, replayed exactly as it runs in production: items fan out, every output is scored, and downstream readers get one summary row.

evaluation-job 7f3a92

○ pending

  1. Job queued

    agent + dataset + prompt + profile, pinned by version id

  2. Invoke

    provider-agnostic SPI calls the agent endpoint, per item

  3. Score

    9 metric strategies judge every output

  4. Summarize

    one precomputed row — benchmarks read this, not 4,500 results

0

runs / 500

0

results scored

—

pass rate

$0.00

cost · p95 —

1 cell = 10 dataset items■ passing · ■ failures to diagnose
drawingevaluation job — fan-out
items500
metrics9 per run
sourcejob → run → result

fig. 01 — deterministic replay of one job. The 29 failures are the interesting part: the root-cause engine turns them into diagnoses.

The problem

Building an agent is easy. Engineering one is not.

Every framework — LangGraph, CrewAI, AutoGen, Spring AI — helps you build an agent. Almost nothing helps you after that: which prompt version is live, whether the new model is actually better, what a deploy did to cost and latency, why run #482 failed.

Teams answer these questions with spreadsheets, ad-hoc scripts and gut feel. That works for a demo. It collapses the moment agents carry production traffic.

  • No stable registry of agents, versions and credentials
  • Evaluations that are unreproducible because inputs drift
  • No objective comparison between prompts, models or versions
  • Regressions discovered by users instead of by pipelines
  • Cost and latency invisible until the invoice arrives
The platform

One system for the whole engineering loop.

Brok's Forge treats the agent as the central aggregate root — a stable, framework-agnostic identity every other capability attaches to. Datasets and prompts are immutable, versioned artifacts; evaluation jobs pin exact versions, so every result is reproducible forever.

It is provider-agnostic and framework-agnostic by construction: an agent is described by metadata, not by any framework’s types, and every LLM provider is reached through one SPI. Adding a provider is a code-only change — no schema migration.

  • Reproducible by construction — evaluations pin immutable versions
  • Objective comparison — leaderboards over precomputed summaries
  • Regressions caught by the pipeline, before users see them
Core modules

Everything after the agent is built.

Nine product capabilities over 22 backend modules — the full engineering loop from registration to advice.

01

Agent registry

Framework-agnostic agents with versioning, encrypted credentials and provider-aware health checks. The single source of truth for "the thing under test."

02

Versioned datasets & prompts

Immutable, append-only versions with {{variable}} templating, activate/rollback and version comparison. Evaluations pin versions, so results never shift under you.

03

Evaluation engine

Jobs fan out into runs and per-metric results — deterministic metrics plus judge-family metrics (LLM judge, semantic similarity, hallucination and citation checks).

04

Benchmarking & leaderboards

Six comparison axes — agent, version, prompt, model, dataset, profile — ranked from precomputed job summaries. Nothing is re-run to build a leaderboard.

05

Regression detection

Baseline-vs-candidate checks against thresholds, so a worse deploy is caught by the pipeline — not by your users.

06

Cost, latency & token analytics

Historical trends across every run: spend concentration, latency spikes, token bloat — per agent, per model, per project.

07

AI Engineering Advisor

Five pure sub-advisors (prompt, model, cost, agent, RAG) produce recommendations with why, what changed, how to fix and expected improvement — computed on read, never stale.

08

Root cause & AI debugger

Failed runs become diagnoses (timeout, HTTP error, empty output, JSON-invalid…) and a 7-stage execution timeline that honestly marks unobserved stages NOT_INSTRUMENTED.

09

Engineering knowledge graph

A queryable graph of failure modes, regressions and remediations (20 seeded nodes, 20 typed edges) that learns — every surfaced pattern increments its occurrence count.

Fig. 02 — evaluation architecture

Job → runs → results → summary.

The pipeline is a fan-out tree with an insert-only hot path: one job, one run per dataset item, one result per metric per run. Summaries are precomputed, so benchmarks and regression checks read one row — not millions.

  1. 01

    EvaluationJob

    agent + dataset + prompt + profile, all pinned by version id

  2. 02

    JobExecutor

    the queue-ready seam — fans the job out per dataset item

  3. 03

    ModelInvocation

    provider-agnostic SPI calls the agent endpoint

  4. 04

    EvaluationRuns

    × dataset items

    output, latency, cost, tokens — one per item

  5. 05

    Results × metrics

    × metrics per run

    one atomic score per metric, per run

  6. 06

    Job summary

    one precomputed row feeds benchmarks, regressions, analytics

Insert-only hot path

A running job appends rows; nothing is updated in place except progress counters. High-volume inserts are the workload the schema is tuned for.

Designed for millions of results

Runs and results partition naturally by job id. "Millions" is the explicit design target the hierarchy is sized for — a target, not a measured benchmark.

Queue-ready by seam, not rewrite

The executor is the only component that fans jobs out. Moving it behind a queue and a worker fleet is a contained change — schema and API stay put.

Metric strategies

  • EXACT_MATCH
  • CONTAINS
  • REGEX_MATCH
  • JSON_VALID
  • LENGTH
  • LATENCY
  • COST
  • TOKEN_COUNT
  • NON_EMPTY
  • LLM_JUDGE
  • SEMANTIC_SIMILARITY
  • HALLUCINATION_DETECTION
  • CITATION_VERIFICATION

One strategy bean per type, resolved from a registry — adding a metric is an enum constant plus one class, never a migration. Judge-family metrics (highlighted) use a configurable LLM or embedding provider.

Benchmark comparisons

AGENT_VS_AGENT

different agents, same dataset / prompt / profile

VERSION_VS_VERSION

did the new deploy actually improve things?

PROMPT_VS_PROMPT

prompt versions against the same agent

MODEL_VS_MODEL

different providers or models behind the agents

DATASET_VS_DATASET

the same agent on different datasets

PROFILE_VS_PROFILE

the same runs under different thresholds

Fig. 03 — architecture

A modular monolith with microservice fault lines.

One deployable today; strict module boundaries so extraction later is mechanical, not archaeological. No cross-module JPA associations, no shared repositories — modules reference each other by UUID and published services only.

Every module, layered the same way

web/

thin controllers · record DTOs · MapStruct mappers · OpenAPI

service/

use cases · transactions · access guards · invariants

domain/

JPA entities · enums · value objects — matches the schema exactly

repository/

Spring Data JPA · private to its module, never shared

  • No cross-module JPA associations — evaluation stores a plain UUID agentId, never a @ManyToOne Agent
  • No shared repositories — reads go through the owning module’s published service
  • The database is the source of truth — Flyway owns the schema, Hibernate only validates it

The modules that turn a registry into an engineering platform.

com.broksforge.modules.evaluation

EvaluationJob → EvaluationRun → EvaluationResult, plus reusable metric/threshold profiles.

domain/ · repository/ · service/ · web/dto/ — dependencies point downward only

Fig. 04 — provider & framework agnostic

One SPI. Any provider. Any framework.

Business logic never touches a provider SDK. Callers invoke through ModelInvocationService; concrete invokers are an extension point — adding one is code-only, because providers and frameworks are text-backed enums with no schema coupling.

EvaluationJobExecutor · health checks · judges

callers — never reference a concrete invoker

ModelInvocationService

registry + dispatcher — the single entry point

interface ModelInvoker

prompt + credentials in → output, latency, cost, tokens out

AgentEndpointInvoker

shipped — any agent that speaks HTTP

OpenAiInvoker · AnthropicInvoker · …

extension point — code-only, no migration

Providers modeled

  • OpenAI
  • Anthropic
  • Google Gemini
  • Groq
  • Ollama
  • OpenRouter
  • DeepSeek
  • Azure OpenAI
  • AWS Bedrock
  • Google Vertex
  • Mistral
  • Cohere
  • Hugging Face

Providers are text-backed enums — adding one touches code, never the schema.

Agent frameworks described

  • Spring AI
  • LangGraph
  • CrewAI
  • AutoGen
  • PydanticAI
  • Semantic Kernel
  • Custom REST / HTTP

An agent is metadata plus an HTTP endpoint — the platform never imports a framework's types.

Security

Multi-tenant isolation, enforced — not assumed.

A platform that stores other teams' credentials and calls their endpoints outbound has to treat security as architecture, not review feedback.

IDOR as 404

Every aggregate resolves by its full (id, projectId, organizationId) tuple. A foreign id is indistinguishable from a missing one — the API never confirms a resource the caller cannot see.

Encryption doctrine

Verification secrets are hashed (BCrypt, SHA-256); usage secrets the platform must present upstream are encrypted (AES-256-GCM, versioned ciphertext for key rotation). Never logged, never returned.

SSRF defence in depth

Agent endpoints are user-supplied URLs the platform calls outbound. Syntactic validation on write; a runtime OutboundUrlGuard re-resolves every call and blocks private, loopback and metadata targets.

Mass assignment: impossible

Request DTOs are records that omit every server-controlled field — a client physically cannot set ids, tenancy keys, status or audit columns.

RBAC + stateless auth

OWNER > ADMIN > MEMBER enforced centrally in the service layer; short-lived JWTs with rotating refresh tokens; password change revokes every session; API keys hashed and shown once.

Leak-free error contract

One GlobalExceptionHandler renders a stable ApiError shape. No stack trace ever leaves the process; correlation IDs make incidents traceable without exposing secrets.

Technology

The stack, as it is layered in production.

Backend

  • Java 21 · Spring Boot 3.4records, sealed types, virtual-thread-ready
  • PostgreSQL + Flywayschema as source of truth, ddl-auto=validate
  • Redisrate limiting, caching, token revocation
  • Spring Security + JWTrotating refresh tokens, API keys
  • MapStruct · springdoc-openapigenerated mappers, typed API surface

Frontend

  • Next.js 15 · React 19 · TypeScriptApp Router
  • Tailwind + shadcn/RadixCSS-variable design system
  • TanStack Query · Zustandserver state with transparent token refresh
  • React Hook Form + Zodschemas shared between form and API

AI layer

  • ModelInvoker SPI13 providers modeled — OpenAI, Anthropic, Gemini, Groq, Ollama…
  • Evaluation enginedeterministic + LLM-judge metric strategies
  • Benchmark engineleaderboards from precomputed summaries

Infrastructure

  • Docker Composeapi · postgres · redis · web
  • AWS EC2 + Nginx + Let’s Encryptself-hosted API, TLS, reverse proxy
  • Vercelfrontend at broksforge.gokul.quest
  • Prometheus + structured logsMicrometer metrics, ECS-JSON logging, correlation IDs
Fig. 05 — production

Deployed and operated, end to end.

Not a localhost project. The API runs on AWS EC2 behind Nginx with Let’s Encrypt TLS; the frontend ships from Vercel. Postgres and Redis are never exposed publicly — only Nginx binds host ports.

  1. Browser

    broksforge.gokul.quest

  2. Vercel

    Next.js 15 frontend

  3. Nginx + TLS

    api.broksforge.gokul.quest

  4. Spring Boot API

    Docker · AWS EC2

  5. Postgres · Redis

    internal network only

  • 12-factor config — the app fails fast if a required secret is missing
  • Flyway migrates on boot; an entity/schema mismatch aborts startup
  • Stateless app tier — scales horizontally without sticky sessions
  • Kubernetes-grade liveness/readiness probes; readiness reflects DB reachability
there is no staging on friday
Engineering philosophy

Non-negotiables, reflected in code and schema.

Eight rules from the Master Architecture document — each one is enforced in review, not aspirational.

  1. 01

    The agent is the centre of gravity

    One stable aggregate root every module attaches to — nothing invents its own notion of "the thing under test."

  2. 02

    The database is the source of truth

    Flyway owns the schema; entities conform to it; migrations are append-only and never edited.

  3. 03

    Provider neutrality is a hard rule

    Anything provider-specific lives behind the SPI. Text-backed enums keep new providers migration-free.

  4. 04

    Secrets are sacred

    Hash what you verify, encrypt what you must present upstream, log neither.

  5. 05

    Immutability where correctness depends on it

    Dataset, prompt and agent versions never change — results stay reproducible and attributable.

  6. 06

    Fail safe, fail loud, leak nothing

    Optimistic locking, fail-fast startup on missing secrets, sanitized error contract.

  7. 07

    Computed on read, never stale

    Recommendations, diagnoses and leaderboards are derived from current data each request — they can never drift.

  8. 08

    Honest observability

    Stages the platform cannot see yet are NOT_INSTRUMENTED — reported, never faked.

Delivery

Four phases to v1.0.

  1. Phase 1 — Foundation

    Auth, users, organizations, projects, API keys. Multi-tenancy and RBAC from day one.

  2. Phase 2 — Agent Registry

    Agent as the central aggregate: versioning, encrypted credentials, health checks.

  3. Phase 3 — Intelligence Layer

    Datasets, prompts, the invocation SPI, evaluation, benchmarking, regression, analytics, reports, search, dashboard.

  4. Phase 4 — AI Engineering Advisor

    Advisor, root-cause engine, AI debugger, knowledge graph — measurement becomes advice.

  5. v1.0 — Production hardening

    Prometheus metrics, structured logging, health probes, OTP flows, provider-aware health checks, EC2 deployment.

Roadmap

What v1.x is for.

  • Async evaluation workers

    Move the executor seam behind a queue and a horizontally-scaled worker fleet.

  • Live tracing

    Drive the TraceRecorder seam with real per-stage spans — lighting up the AI Debugger’s NOT_INSTRUMENTED stages.

  • OpenTelemetry export

    IDs and structure are already in place; wiring exporters is config, not domain change.

  • Provider-direct invokers

    OpenAI/Anthropic/Gemini clients as additional ModelInvoker implementations behind the same SPI.

  • Backend test suite

    JUnit + Testcontainers coverage for RBAC, IDOR-as-404, SSRF and encryption paths. Documented strategy; honest gap today.

  • SDK & CLI

    Programmatic access for CI pipelines — evaluate on every merge.

Calibrated claims, on purpose: counts on this page are from the repo; "millions of evaluations" is the stated design target of the schema, not a measured benchmark; the backend test suite is documented but not yet implemented.

See it running

The platform is live. The architecture is public.

Register an agent, run an evaluation, read the Master Architecture document and the 17 ADRs behind it — everything on this page is verifiable.

Open Brok's ForgeRead the source
Back to portfolio