Collect Cloud Sessions, Evaluate Locally: Analyzing AI Agents on Amazon Bedrock AgentCore with Apache SkyWalking AI Sessionizer

This post is also published on AWS Builder Center. 中文版 / Chinese version

This walkthrough uses a release assistant to demonstrate the full workflow: ask an agent to change configuration, deploy an application, and query the result, then check whether the work met the requirements. The agent runs on Amazon Bedrock AgentCore. Apache SkyWalking AI Sessionizer (asz) records its execution in the cloud. We pull those records onto a development machine, inspect them, and run a judge through a local CLI to check whether the agent met the requirements.

We will follow one example through five steps: design the agent → deploy and collect → pull and inspect → run the judge → read results against the goals.

1. What the agent should do

The agent’s job is to release demo-v1. We give it tools for file access, configuration validation, building, deployment, and status queries. The model decides how to use them.

The agent uses LangGraph, runs in AgentCore Runtime, and makes real calls to the Nova Pro model on Amazon Bedrock. The business system is simulated: configuration, bundles, deployment state, and acceptance requests are stored as files. It does not operate a real production application.

We designed seven conversation turns covering normal work, revised requirements, denied permissions, and cancellation:

These requests are stored in scenarios/release.json and sent as real user messages. The right-hand column is our checklist for reading the judge report later.

The simulator deliberately denies production deployment, and acceptance submission returns no approval conclusion. Correct behavior therefore includes reporting limits accurately when work cannot be completed. Whether deployment succeeded and whether the agent reported failure honestly are separate questions.

2. Deploy and collect

Deploy the example

Prepare Git, Go 1.27+, uv, OpenSSL, AWS CLI, OpenTofu, and a running Docker engine. Your AWS account needs infrastructure deployment permissions and access to us.amazon.nova-pro-v1:0 in us-east-1.

git clone --branch main https://github.com/wu-sheng/aws-agentcore-asz-demo.git
cd aws-agentcore-asz-demo/examples/pull-judge
bash scripts/bootstrap.sh
export PATH="$PWD/.work/bin:$PATH"

bootstrap.sh installs the example dependencies and builds asz. This example has its own directory, preserving the code and deployment process from the previous article.

If you are not signed in to AWS, run aws login first. Run the deployment commands below from examples/pull-judge/, replacing YOUR_PUBLIC_IPV4 with your public egress IPv4 address. If you use a proxy, supply its egress address.

aws sts get-caller-identity
.venv/bin/python scripts/prepare-cloud.py --client-ip YOUR_PUBLIC_IPV4

AGENT_IMAGE=ghcr.io/wu-sheng/aws-agentcore-asz-pull-judge-agent:b3e872bfb9a26fbd8bfac6eab6073d241b79c904 \
  bash scripts/up.sh

This uses the image tested in this walkthrough. The script checks model access, copies the image into ECR, and creates the AgentCore Runtime, asz on Fargate, EFS, and networking. Infrastructure and model calls incur AWS charges. The deployment writes its resource manifest to .work/cloud/resources.txt.

Collect execution records and file changes

Agent in AgentCore
  ├─ Conversation, model calls, tool calls ──→ cloud asz
  └─ File changes ──→ shared EFS ──→ cloud asz
                                       │
                                  pull on demand
                                       ↓
                              Local viewing and judging

The example configures two collection paths:

• Execution records: LangGraph tracing sends user input, model responses, and tool calls to the asz receiver through LANGSMITH_ENDPOINT.
• File changes: The asz recording component watches write_file, build_release, deploy, and submit_acceptance, writing changes to shared EFS for asz to read.

These paths preserve model responses, tool results, and file changes in the same conversation. The agent calls Bedrock using its Runtime IAM role. The cloud runs the agent and asz collection service; Claude Code or Codex runs on your local machine for evaluation.

Now send the seven requests from section 1:

.venv/bin/python scripts/cloud.py

The script invokes the agent turn by turn in one AgentCore session, waits for collection, and automatically pulls snapshots A, B, and C after turns 1, 5, and 7. It prints a .work/cloud-TIMESTAMP directory containing:

• turn-*.json: Each request and response.
• checkpoints.json: Conversation IDs and tags for the three snapshots.
• client/asz.yaml: The local configuration for the next step.

3. Pull the records and inspect them locally

Remote pull copies a cloud conversation into a local snapshot. Cloud collection continues, while local viewing and evaluation use the fixed version you pulled.

Connect to cloud asz

Enter the printed run directory, substituting your actual timestamp:

cd .work/cloud-TIMESTAMP/client

The script has already generated asz.yaml. Its remote connection section looks like this; the actual file has the address and paths filled in, so you do not need to copy this snippet:

remotes:
  - name: demo
    url: https://YOUR_NLB_HOST:8788
    token_file: /path/to/demo.token
    ca_file: /path/to/demo-ca.pem
    server_name: asz-pull-judge.internal

demo names the remote service, and token_file supplies authentication. The last two settings verify the example’s self-signed certificate. The deployment script also configures HTTPS and restricts the endpoint to your client IP. See the remote pull configuration guide.

List cloud conversations, then pull one:

asz pull -list -remote demo
asz pull demo/CONVERSATION_ID

Replace CONVERSATION_ID with an ID from the list. The pull output provides SNAPSHOT_TAG; checkpoints.json in the parent directory contains the A, B, and C tags already pulled by the script. Later, use demo/CONVERSATION_ID:SNAPSHOT_TAG to select that exact data. Pulling again with no new records returns unchanged.

See what the agent actually did

asz view

Open the local address printed by the command and select the pulled conversation. Start with turn 2, where the user asked to change the timeout from 300 to 600. Find three things:

1. The user’s change request.
2. The write_file change to configs/staging.json.
3. The configuration value returned by validate_config.

This is the file change pulled from our cloud run:

The final validation read 600, confirming the update. However, the record also shows an earlier validation running alongside the write, reading the old value, and requiring a retry. The final result can be correct while the execution still contains avoidable rework. The judge helps organize these findings.

Viewing an existing snapshot does not require cloud asz to be online. Pull again when you need newer data. When finished viewing, press Ctrl+C to stop asz view and continue with the evaluation commands in the same directory.

4. Evaluate the agent against its goals and generate an HTML report

The judge identifies tasks from the user’s requests, then checks whether tool results, file changes, and replies support completion. We can also provide the design goals from section 1, so the report follows our own checklist.

Prepare the local CLI and goal list

Install and sign in to Claude Code, and make sure your terminal can run claude. asz starts it for evaluation; conversation content it reads goes to its model service. For Codex, replace -evaluator claude with -evaluator codex in the commands below.

The bootstrap.sh command in section 2 installs an asz CLI with HTML reporting support (commit 07a2d95); no additional build is needed. The cloud experiment and recording components used 341f8c0; existing snapshots do not need to be collected again.

Continue in .work/cloud-TIMESTAMP/client/. The example’s scenarios/judge-goals.yaml organizes the seven requests into six goals: configuration, staging deployment and verification, production deployment, acceptance, autoscaling and cancellation, and the final summary. Copy it into the current directory:

cp ../../../scenarios/judge-goals.yaml goals.yaml

The file is a simple list. For example, one entry reads:

goals:
  - Build and deploy staging, then query it to verify the release, replica count and timeout.

Use the complete file copied above, which contains six goals. For your own scenario, specify the expected result, such as querying after deployment to confirm the release, replica count, and timeout. This list organizes report tasks for comparison with the design in section 1.

Run the evaluation

After all requests in section 1 have finished, including the final request to cancel scaling and summarize the results, evaluate the conversation snapshot pulled in section 3. This gives the judge the full history of requirement changes and cancellation. From client/, run:

asz judge -evaluator claude -goals goals.yaml \
  -remote-conversation demo/CONVERSATION_ID:SNAPSHOT_TAG

Replace CONVERSATION_ID and SNAPSHOT_TAG with the conversation ID and snapshot tag from the pull output in section 3. If you use the snapshots pulled automatically by the script, choose entry C in checkpoints.json; it contains the conversation after all requests have finished. The command evaluates the local records and groups the identified tasks under our goals. Each evaluation is saved as a numbered record called a run. It does not execute the agent’s release tools again or require cloud asz to be online.

If a judge run already exists and you only want to organize it by this goal list, use that saved run:

asz judge -regroup 1 -goals goals.yaml -evaluator claude \
  -remote-conversation demo/CONVERSATION_ID:SNAPSHOT_TAG

-regroup calls the evaluator to regroup existing tasks while preserving their criteria, evidence, and scores. -goals provides a goal list, not scoring rules. When reading the report, check whether the completion criteria identified by the judge match the original requests. The report shown here uses existing run 1 from the complete conversation snapshot, regrouped under six goals and exported.

Export the HTML report and open it in a browser

List the saved evaluations, then export the run you want to inspect. This example selects run 1; replace the number in -run 1 to select another record:

asz judge -list \
  -remote-conversation demo/CONVERSATION_ID:SNAPSHOT_TAG

asz judge -show -run 1 -format html \
  -remote-conversation demo/CONVERSATION_ID:SNAPSHOT_TAG > judge-report.html

Open judge-report.html in your browser. It needs no web server and includes its own styles, making it convenient to retain or share the detailed report. Export reads saved results without calling the model again.

-show defaults to the latest closed run; -run 1 selects run 1 explicitly. Use -format markdown instead of HTML for a concise summary, or -show -run 1 -json for the raw stage data. See the judge command documentation.

5. Review evaluation results in the browser

Start by selecting a task in the Goals table, expand its completion criteria and evidence, then read the overall scores and summary. We will use this order to check the six goals from section 1.

The report below evaluates all requests in this AgentCore conversation and organizes the results under the six goals from section 1. The HTML report is included in the repository and can be saved and opened locally.

First: Find the work in the Goals table

The table has Goal, Target, Status, and Score columns. A Goal is the result we wanted. A Target is a specific task identified by the evaluator. Status describes its outcome, and Score gives its score. Select Check details to inspect the task.

Staging configuration is marked achieved with 100; staging deployment and verification is marked contradicted with zero. The first means the judge found the task completed. The second means the agent’s completion claim conflicts with the recorded evidence. Expand both tasks to see why.

Second: Expand Criteria, Done, and Missing

This is the expanded staging deployment task. Criteria, completed work, and missing work appear side by side for direct comparison.

Configuration: completed, with rework. Check the criteria for two replicas, 600 seconds, and validation after the update. File changes and the final validation support them. Done describes the configuration work. Missing lists two rounds of rework caused by parallel writes and validation, along with avoidable usage. A score of 100 means the result met the requirements, not that execution was error-free.

Staging: deployment returned success, but verification was incomplete. Done shows the built bundle and a successful deployment tool result with saved simulated state. Missing shows that the query ran alongside deployment, returned not_deployed, and was never repeated. Compare that with section 1’s requirement to verify after deployment: the gap is clear.

The task shows two scores: 40 by the achievement record, and a final score of zero after accounting for the conflicting completion claim. Read both the build work in Done and the missing post-deployment verification in Missing.

This is a remote pull snapshot. The HTML includes saved criteria and evidence excerpts but does not embed the original conversation steps. Use asz view from section 3 to inspect full tool requests, responses, and file changes.

Third: Compare criteria and evidence with the original requirements

The production task illustrates a different issue: the judge’s completion criteria are weaker than the user’s request. Section 1 asks for deployment to production, but the evaluation defined one condition as attempting deployment. The tool returned denied and deployed: false, yet the report marked the task achieved because an attempt satisfied that condition.

Against the original request, the conclusions are: production deployment was not completed; the agent correctly reported the denial. Grouping the task under the production goal does not correct that loose criterion. Read the evaluation alongside the original request, Criteria, and tool evidence.

Use the same approach for the remaining goals:

Fourth: Review overall scores, evaluation completeness, and the summary

Scroll to the bottom of the report for Overall assessment and Overall result. These sections summarize the whole conversation; the Goals table above shows individual task results.

Read the four values in Overall assessment above as follows:

These numbers describe completion and execution efficiency in this run, not model accuracy on other tasks.

This run also reports grouping: the judge agreed with 0 of your 6 goals; 6 merged. The original evaluation grouped all six tasks under one release goal, while our checklist separates them. This is a grouping difference, not six failed goals.

Also check whether the evaluation itself is complete. Two completion judgments relied only on the agent’s own statements, so the report reduced the credit given to them. Efficiency analysis also left work unclassified, and the report is partial. This describes an incomplete evaluation; keep these limits in mind when reading the scores. See the scoring and evidence rules.

Overall result summarizes achieved, contradicted, and cancelled tasks in words, and distinguishes agent usage from judging usage. The original conversation used about 48.5 thousand tokens; the original evaluation used about 263.8 thousand. Regrouping is an additional evaluator call, recorded separately from the original run. The evaluation usage above excludes that additional regrouping call.

Decide what to change

The clearest improvement is to sequence dependent tools: write configuration → validate → build → deploy → query and confirm. Also require the final summary to distinguish completed, denied, and still-unconfirmed work.

After making changes, run the same seven requests, pull a new snapshot, and judge again. Keep using the checklist from section 1: was verification completed, did rework decrease, and is the summary accurate?

Clean up the experiment

When finished, preserve the snapshots and reports you need, then return to examples/pull-judge/ to clean up:

bash scripts/down.sh --yes
bash scripts/down.sh check

Snapshots have retention limits. Exclude tokens and TLS private keys from archives. If network cleanup is waiting for AgentCore to release interfaces, follow the check output. The example README has detailed cleanup instructions.

Find the code in the example repository and additional results in the experiment record. Full usage instructions are in the Apache SkyWalking AI Sessionizer documentation.

• ← Previous
Next →