Invocation Log & Analytics
PromptShuttle provides detailed logging of every request and three levels of analytics aggregation for understanding cost, performance, and usage patterns.
Invocation log
Every request is logged with timing, cost, and metadata. Full prompt and answer text, original input sequences, and template parameter values, are kept by default. A tenant can reduce that period or keep text for only a sample of runs. Set this on Tenant Profile under Request / response text. Cost, model and status stay in all cases. Debug runs always keep full text. Already-stored runs are stripped on the next cleanup pass. The run row itself stays for the platform request-retention window.
Browse the log
GET /api/v1/llm-logsFilters
cursor
string
Pagination cursor (from previous response)
pageSize
integer
Results per page (default 50)
from
datetime
Start time filter
to
datetime
End time filter
environments
string array
Filter by environment name
flows
string array
Filter by flow name
statuses
string array
Filter by status (Pending, Succeeded, Failed)
tags
string array
Filter by tags (any match)
customerId
string
Filter by end-customer external ID
rootOnly
boolean
Show only root-level requests (exclude sub-agents)
parentRequestId
string
Show children of a specific parent request
Log entry fields
Each log entry includes:
Request details — Messages, model, parameters, environment, tags
Response details — Text output, tool calls, citations, finish reason
Tool calls — For each call, the endpoint it was actually placed against and where that origin came from: the tool definition, an environment binding, or the caller's
X-Shuttle-Callback-UrlUsage — Tokens in/out, reasoning tokens, cost in credits and USD
Timing — Duration in milliseconds
Hierarchy — Parent/root request IDs, agent depth, agent role
Customer — End-customer ID and external ID
Feedback — Feedback score if submitted
Detail dropped —
detailDroppedis true when the tenant's sample rate skipped persisting this run's prompt/response textCancellation —
cancelRequestedAt/cancelledAt/cancelRequestedBy, if the run was stoppedProgress —
progressScore/progressCertainty/minProgressScore, if the run opted into verification. Absent means unscored, which is not the same as a score of zero
Stopping a run
Runs still executing show as running in the log, and the list refreshes on its own while any of them are on screen. Each has a stop control, as does the run detail — stopping a run stops every sub-agent beneath it. When anything is running, a Stop everything running button appears above the list; it stops every run in the workspace, including ones started by other people and by your applications.
Stopping is not instant: a run notices at its next step, so a model call already in flight finishes first. What it had produced by then is kept and still billed — stopping saves the work that had not happened yet. A stopped run shows a grey cancelled badge and carries a Cancelled warning explaining how far it got.
To find stopped runs, filter on the Cancelled flag; the status filter cannot express it, because a run that was stopped after producing something still records Succeeded. The dashboard's Running now counter links straight to the runs it counts.
The same controls are available over the API (POST /api/v1/runs/{runId}/cancel) and over MCP (cancel_run, cancel_all_runs).
Request hierarchy
For multi-agent requests, view the full hierarchy:
Returns:
Breadcrumb — Ancestor path from root to current request
Parent — Direct parent summary
Children — Direct child agent summaries
Streaming events
View the SSE events emitted during a request:
Filter by event type: ?types=agentStarted,agentCompleted
Get event type counts: GET /api/v1/llm-logs/{id}/events/stats
Log entries (debug)
For requests with elevated log levels, view detailed internal log entries:
Analytics
PromptShuttle provides analytics at three levels of granularity, each answering different questions.
Level 1: Per-inference stats
What it answers: How are individual LLM calls performing?
Metrics per model:
Average tokens in/out
Average cost and latency
Percentiles (P50, P90, P95, P99) for tokens, latency, and cost
Token and latency histograms
Level 2: Per-request stats
What it answers: How are complete requests performing (including tool-calling loops)?
Metrics:
Total cost per request (across all LLM calls and tool invocations)
Total function calls per request
Request latency distribution
Broken down by primary model
Level 3: Per-tree stats (multi-agent)
What it answers: How are entire agent trees performing?
Metrics:
Total tree cost (root + all sub-agents)
Agent count per tree
Maximum depth reached
Depth distribution
Agent count distribution
All three levels at once
Returns all three levels plus time series data for invocations and cost.
Tenant-wide statistics
Query parameters:
period
TimeSpan
Time window (e.g. 7.00:00:00 for 7 days)
resolution
string
Time bucket size: Minutes5, Minutes15, Hour, Day, Week
Returns:
Invocation count (current vs. previous period)
Per-flow invocation time series
Cost time series
Token usage time series
Top flows by invocation count
Credit usage comparison
Account overview
Returns cost and request counts broken down by model and by flow, with percentage distribution.
Cost by model over time
Answers "did switching models actually cut unit cost". One cell per time bucket × served model: call count, billed cost, raw provider cost, and average billed cost per call. Buckets are hour / day / week from the period length (the same rule as the dashboard). Reads the hourly usage rollup, so a 180-day window stays cheap.
The Usage page and each flow's Analysis tab render this as a stacked bar plus a heatmap table (total cost, avg $/call, or calls). Long Analysis windows (90d / 180d) show only this map — the percentile histograms still scan every request and do not survive that range.
Hours from before the live rollup writer started are filled in once from stored inference rows. Embedding traffic has no per-request history and is not reconstructed.
Cost per tag
Answers "what did this unit of work cost". If your application stamps its own correlation tag on every request — an order id, a document id, a tender id — this returns that tag's total spend as a flow × model matrix: how many LLM calls each flow made against each model, and what they cost.
Because a tagged job can run for weeks and the model behind a flow may change during it, the breakdown keys on the model that actually served each call. A model swapped mid-job simply shows up as a second column rather than silently replacing the first.
Accepts API keys as well as session tokens, so your own application can display the figure.
tag
string
Required. The exact tag, e.g. TenderId:12313. Not a prefix match.
grouping
string
split (default) or rollup — see below
from
DateTime
Optional start. Unbounded by default
to
DateTime
Optional end
Sub-agent requests inherit their parent's tags, so one tag covers an entire agent tree. grouping decides how that tree is presented — the totals are identical either way:
split— one row per flow that actually ran, so a sub-agent's own flow is visible with its own model and cost. Use this to find which agent is expensive.rollup— sub-agent spend folds into the entry-point flow you invoked. Use this to answer "what did each feature cost me".
Each cell reports costUsd (billed), rawCostUsd (provider cost before markup), calls, requests, token counts, and pricingUncertain — the last flags a cell whose served model is not in the model catalogue, so its price is an estimate.
One row per distinct tag under a key prefix, ranked by spend — "which tender cost the most". Only tags starting with the prefix are reported, which also keeps PromptShuttle's own internal marker tags (Agent:, Panel, DebugUrl:) out of the result.
prefix
string
Required. Anchored prefix, e.g. TenderId:
from
DateTime
Optional start. Defaults to 90 days ago; the window is capped at 365
to
DateTime
Optional end
limit
int
Rows to return, 1–200 (default 50). truncated says if more exist
Both are also available in the UI under Tag Costs.
Caveats. Panel deliberations run as their own top-level requests with fixed tags, so they do not inherit a caller tag and will not appear in a tag report.
Data export
Export your flows and tools for backup or migration:
Returns all flows (with all versions and templates) and tools in a single JSON document.
Import into another tenant:
Duplicate flows and tools are skipped (matched by name).
Last updated