Simulated Bedrock

Simulated Bedrock answers model invocations from responses declared against a prompt or a model. A test says what the model says, and no model runs.

Bedrock-specific types are imported from the @kensio/yulin/bedrock subpath.

Answering a conversation

Converse answers with the response declared for the prompt it carries. The prompt is the text of the last user message.

/**
 * Declaring what a model answers one prompt with, and conversing with it.
 */

import { ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws.bedrock().responses().onPrompt("Summarise entry 1042", {
  text: "Entry 1042 covers the tone sandhi rules.",
});

const answered = await simAws.bedrock().converse(
  new ConverseCommand({
    modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages: [{ role: "user", content: [{ text: "Summarise entry 1042" }] }],
  }),
);

console.log(answered.output.message.content.at(0)?.text);
// "Entry 1042 covers the tone sandhi rules."
console.log(answered.stopReason); // "end_turn"

A conversation with several turns matches on its last user message, so a rule keeps matching as the conversation grows.

Answering every call to one model

onModel covers every invocation of a model that no prompt rule matched first. This is the rule for a test that cares about the code around the call rather than about one exchange.

/**
 * Declaring one answer for every call to a model.
 */

import { ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws
  .bedrock()
  .responses()
  .onModel("amazon.nova-pro-v1:0", { text: "A short summary." });

const answered = await simAws.bedrock().converse(
  new ConverseCommand({
    modelId: "amazon.nova-pro-v1:0",
    messages: [{ role: "user", content: [{ text: "Anything at all" }] }],
  }),
);

console.log(answered.output.message.content.at(0)?.text); // "A short summary."

byDefault covers everything else again. An invocation matching no rule at all answers with a built-in line of text that says it is simulated.

Answering with a tool call

A declared response carries content blocks as they were written, so a tool call reaches the code that handles one. The response stops for tool_use unless the declaration names another reason.

/**
 * Declaring a tool call for the code under test to handle.
 */

import { ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws
  .bedrock()
  .responses()
  .onPrompt("What is in entry 1042?", {
    content: [
      {
        toolUse: {
          toolUseId: "tooluse-1",
          name: "lookUpEntry",
          input: { entryId: "1042" },
        },
      },
    ],
  });

const answered = await simAws.bedrock().converse(
  new ConverseCommand({
    modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages: [{ role: "user", content: [{ text: "What is in entry 1042?" }] }],
  }),
);

console.log(answered.stopReason); // "tool_use"
console.log(answered.output.message.content.at(0)?.toolUse?.name);
// "lookUpEntry"

Streaming a conversation

ConverseStream answers from the same rules, and sends the response as the events real Bedrock sends. Declaring chunks says where the deltas fall.

/**
 * Declaring a response in chunks and accumulating the stream.
 */

import { ConverseStreamCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws
  .bedrock()
  .responses()
  .onPrompt("Summarise entry 1042", {
    chunks: ["Entry 1042 covers ", "the tone sandhi rules."],
  });

const answered = await simAws.bedrock().converseStream(
  new ConverseStreamCommand({
    modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages: [{ role: "user", content: [{ text: "Summarise entry 1042" }] }],
  }),
);

let accumulated = "";

for await (const event of answered.stream) {
  accumulated += event.contentBlockDelta?.delta.text ?? "";
}

console.log(accumulated); // "Entry 1042 covers the tone sandhi rules."

The events arrive in the order real Bedrock sends them. messageStart, then one contentBlockDelta per chunk, then contentBlockStop, messageStop carrying the stop reason, and metadata carrying the token counts. A tool call adds a contentBlockStart before its delta, carrying the tool use id and name.

The same declaration serves both APIs. Converse answers with the chunks joined, and a response declared as text streams as a single delta. A stream is readable once, and reading it again raises.

InvokeModelWithResponseStream streams the declared body as a single chunk:

for await (const event of answered.body) {
  accumulated += new TextDecoder().decode(event.chunk?.bytes);
}

Invoking a model directly

InvokeModel answers with the body declared for the request, serialized as JSON. The body is the shape the model behind the id uses, which is why there is no built-in default for it.

/**
 * Declaring a model-specific response body and invoking the model.
 */

import { InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws
  .bedrock()
  .responses()
  .onModel("amazon.titan-text-express-v1", {
    body: { results: [{ outputText: "Entry 1042 covers tone sandhi." }] },
  });

const answered = await simAws.bedrock().invokeModel(
  new InvokeModelCommand({
    modelId: "amazon.titan-text-express-v1",
    body: JSON.stringify({ inputText: "Summarise entry 1042" }),
  }),
);

console.log(JSON.parse(new TextDecoder().decode(answered.body)));
// { results: [ { outputText: "Entry 1042 covers tone sandhi." } ] }

An InvokeModel request matches a prompt rule on its request body decoded as UTF-8. A model rule is usually the one to reach for.

Reporting token counts

usage comes from the declaration, and a response that declares none reports fixed counts.

/**
 * Declaring what a response cost, for code that meters token spend.
 */

import { ConverseCommand } from "@aws-sdk/client-bedrock-runtime";

import { SimAws } from "@kensio/yulin";

const simAws = new SimAws();

simAws
  .bedrock()
  .responses()
  .byDefault({
    text: "A short summary.",
    usage: { inputTokens: 1400, outputTokens: 220 },
  });

const answered = await simAws.bedrock().converse(
  new ConverseCommand({
    modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages: [{ role: "user", content: [{ text: "Summarise entry 1042" }] }],
  }),
);

console.log(answered.usage.totalTokens); // 1620

Authorizing an invocation

An invocation authorizes bedrock:InvokeModel against the model it names. A base model id becomes a foundation model ARN for the Region the call was made in, and an inference profile ARN or a provisioned model ARN is authorized against as it was written.

SDK interception

An intercepted BedrockRuntimeClient reaches the simulated Bedrock of the Account and Region the client is configured for.

/**
 * Answering production code that holds its own Bedrock Runtime client.
 */

import {
  BedrockRuntimeClient,
  ConverseCommand,
} from "@aws-sdk/client-bedrock-runtime";

import { SimSdk } from "@kensio/yulin/sdk";

const simSdk = new SimSdk();
simSdk.intercept(BedrockRuntimeClient);

simSdk.simAws
  .account()
  .region("eu-west-2")
  .bedrock()
  .responses()
  .byDefault({ text: "Entry 1042 covers the tone sandhi rules." });

const client = new BedrockRuntimeClient({ region: "eu-west-2" });

const answered = await client.send(
  new ConverseCommand({
    modelId: "anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages: [{ role: "user", content: [{ text: "Summarise entry 1042" }] }],
  }),
);

console.log(answered.output?.message?.content?.at(0)?.text);
// "Entry 1042 covers the tone sandhi rules."

client.destroy();
simSdk.restoreAll();

Available functionality

• Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream, through simAws.bedrock() and through an intercepted BedrockRuntimeClient.
• Responses declared with onPrompt, onModel and byDefault. A prompt rule wins, then a model rule, then the default.
• A declared response carries text, chunks or content blocks for Converse, a body for InvokeModel, and optionally a stopReason and a usage.
• Streamed responses, with chunks deciding where the deltas fall and one declaration serving the streaming and non-streaming APIs alike.
• Tool calls, as a declared toolUse content block.
• IAM authorization of bedrock:InvokeModel against the foundation model, inference profile or provisioned model ARN the request names.
• Rules held per Account and Region, so two Regions answer the same prompt differently.

Limitations

• No model runs. A response is whatever the matching rule declared, and the prompt is read only as a key to match on.
• Every event of a stream is ready as soon as the call returns. Real Bedrock sends them as the model generates them. No simulated clock advance separates them here.
• A response is split into deltas only where the declaration says so. Text declared any other way arrives in one delta. The split a real model streams comes from its own tokenizer.
• A streamed tool call sends its arguments in one delta. Real ConverseStream sends them as fragments of JSON to be concatenated. A tool call with no declared arguments sends {}.
• InvokeModelWithResponseStream sends the declared body as a single chunk. Real Bedrock sends a chunk per generated fragment, in the shape the model behind the id uses.
• A serialized Bedrock request is refused. Bedrock speaks REST-JSON, and the wire path answers only the AWS JSON protocol services. A function bundling its own SDK reaches Bedrock through module interception. S3 and every other non-JSON-protocol service are reached the same way.
• The Bedrock control plane is unsimulated. ListFoundationModels, guardrail management, inference profile management and provisioned throughput all belong to BedrockClient.
• Bedrock Agents and knowledge bases are unsimulated. InvokeAgent, Retrieve and RetrieveAndGenerate arrive on a client of their own.
• ApplyGuardrail is unsimulated, and a guardrailConfig or a guardrailIdentifier on an invocation is refused outright. Answering without the guardrail would make one look applied here and be applied in production.
• system, inferenceConfig, toolConfig and additionalModelRequestFields are accepted and decide nothing. maxTokens truncates no declared response, and a toolConfig naming no tools still gets a declared tool call.
• Token counts are fixed unless the declaration carries them. Counting them needs the tokenizer of the model the request names. Declare a usage where the code under test meters spend.
• metrics.latencyMs is always zero. No time passes during an invocation.
• A Converse with no messages is refused. Real Bedrock accepts one where the modelId names a prompt version from Prompt management, which is unsimulated.
• The model id goes unchecked. AWS publishes no enumerable table of model ids, so refusing one would be failing closed against Yulin’s own gaps.
• InvokeModel has no built-in default response body, and an invocation matching a rule that declares none is refused. A response body is model-specific, and one family’s shape served for every other one would parse into something the caller cannot read.
• An InvokeModel body that arrives as a stream or a Blob matches no prompt rule. Reading it would consume the caller’s own request body, so such a request falls through to a model rule or to the default.
• A Converse call matching a response declared only as a body is refused for the same reason. It does not fall back to the default.
• There are no CloudFormation resource types for Bedrock, and Bedrock is absent from serveSimAws.