AWS & NVIDIA Hackathon 2025: Research-as-a-Code, UDR, and the $1,800 Cloud Bill

I have always had a special appreciation for long-running hackathons. While 24-hour weekend sprints (like LlamaCon or WeaveHacks 2) are exhilarating adrenaline rushes, their compressed timelines force teams to rely on quick shortcuts and pre-packaged templates.

A multi-week competition offers something fundamentally different: the luxury of time to dive deep into unexplored technical territory, experiment with competing state-of-the-art architectures, push cloud infrastructure to its limits, and understand the subtle engineering trade-offs of cutting-edge research.

That was the journey behind the AWS & NVIDIA Agentic AI Unleashed Hackathon 2025, which ran from October 13 through November 4, 2025.

My project for the competition was Research-as-a-Code (RaaC) (Devpost submission): a two-level hierarchical agentic platform deployed on AWS EKS that explored and contrasted two fundamentally distinct deep research paradigms: NVIDIA Universal Deep Research (UDR) and Google Test-Time Diffusion Deep Researcher (TTD-DR).

Along the way, I wrestled with GPU node warmup latency, prompt-engineered small 8B parameter models into reliable code compilers, debugged diffusion convergence loops, and learned a painful lesson about enterprise cloud costs: running a multi-node GPU Kubernetes cluster on AWS for a month ended up costing over $1,800 USD in cloud bills!

The Vision: Pitting Two Deep Research Paradigms Against Each Other

Traditional Retrieval-Augmented Generation (RAG) pipelines follow a rigid, linear formula: embed user query, fetch top-$k$ document chunks from a vector database, stuff them into an LLM prompt, and generate an answer.

For complex, ambiguous, or multi-domain inquiries (such as “Perform a comparative cost-benefit analysis of deploying NVIDIA NIMs on AWS EKS versus Serverless container endpoints”), naive RAG collapses completely. The system needs to formulate a research plan, conduct exploratory search, identify information gaps, synthesize partial findings, and iteratively dive deeper.

I wanted to investigate how modern LLMs best perform this multi-step deep research. Specifically, I wanted to benchmark two cutting-edge paradigms:

• 1. NVIDIA Universal Deep Research (UDR) — “Strategy-as-Code”:
  • Core Pipeline: Natural Language Plan → Compile to Python Code → Sandboxed Execution Once
  • Key Properties: Deterministic, transparent, highly efficient execution, but rigid branching and syntax-sensitive
• 2. Google Test-Time Diffusion Deep Researcher (TTD-DR) — “Iterative Denoising”:
  • Core Pipeline: Noisy/Coarse Outline → Targeted Gap Queries → Multi-Turn Denoise & Refine
  • Key Properties: Adaptive, self-evolving, exploratory discovery, but higher token usage and slower convergence

1. NVIDIA Universal Deep Research (UDR): “Strategy-as-Code”

The core thesis behind NVIDIA’s UDR prototype is that LLMs are exceptionally proficient at writing source code. Instead of forcing an LLM to navigate complex, branching research workflows through chained English prompts, UDR frames the entire research strategy as an executable Python script.

The LLM acts as a compiler: it translates a high-level research intent into clean Python code that programmatically invokes vector search tools, executes web queries, filters tabular data, and calls summarization endpoints. That script is then executed in a single, high-speed, deterministic pass inside an execution sandbox.

2. Google Test-Time Diffusion Deep Researcher (TTD-DR): “Iterative Denoising”

Borrowing inspiration from generative diffusion models, Google’s Test-Time Diffusion approach treats research report generation as an iterative denoising process.

Rather than compiling code, the system begins with an intentionally coarse, “noisy” draft or structural outline. At each diffusion step, an evaluation engine analyzes gaps in the draft, formulates targeted questions, executes fresh searches, and merges the new evidence to “denoise” the document. This loop continues iteratively until the draft reaches a quantitative convergence score.

System Architecture: Research-as-a-Code (RaaC)

To bring these two paradigms to life, I designed a two-level hierarchical architecture:

(Architectural overview of Research-as-a-Code: Level 1 LangGraph planner directing user requests to either NVIDIA UDR Strategy-as-Code or Google Test-Time Diffusion, backed by AWS EKS and NVIDIA NIM microservices.)

• Level 1 Orchestrator: Built with LangGraph and served via FastAPI, this top-level planner analyzes the incoming research query. For straightforward factual questions, it routes directly to a standard enterprise RAG pipeline. For complex, multi-faceted inquiries, it engages Level 2.
• Level 2 Strategy Engine: Allows the user or orchestrator to choose between NVIDIA UDR or Google TTD-DR.
• Cloud Infrastructure: Hosted on AWS EKS with containerized NVIDIA NIM (Inference Microservices) for LLM reasoning and NeMo Retriever vector search, backed by Milvus vector database and object storage.
• Agentic User Interface: Built with Next.js and CopilotKit AG-UI, streaming agent thoughts, execution logs, and intermediate research steps to the browser in real time via Server-Sent Events (SSE).

(The CopilotKit AG-UI interface showing strategy selection between NVIDIA UDR and Google Diffusion, with real-time progressive streaming of agent execution state.)

The AWS Infrastructure & The $1,800+ Reality Check

Because the hackathon mandated deploying NVIDIA NIM microservices in an AWS environment, I provisioned an enterprise AWS EKS (Elastic Kubernetes Service) cluster using Terraform.

Accelerating Cluster Warmup with Karpenter

Anyone who has worked with GPU containers in Kubernetes knows the cold-start pain: a single NVIDIA NIM container image can be 15 GB to 30 GB in size. Pulling layers, allocating GPU drivers via the NVIDIA GPU Operator, and warming up PyTorch runtimes on standard AWS Auto Scaling Groups (ASGs) can easily take 15 to 20 minutes per node!

To solve this, I deployed Karpenter for just-in-time GPU node provisioning:

• Karpenter directly provisions raw EC2 instances without the overhead of ASG abstraction.
• It dynamically scheduled AWS g5.2xlarge instances (featuring NVIDIA A10G Tensor Core GPUs with 24 GB of VRAM) the moment a NIM pod requested GPU capacity.
• Node warmup and pod readiness dropped from ~18 minutes down to under 4 minutes.

The Cost Battle: Spot Instances and Sleep Automation

Because this was a multi-week hackathon and I was balancing development with my daily work commitments, I couldn’t afford to leave high-end GPU nodes sitting idle all day.

I initiated a series of aggressive cost-reduction engineering sprints:

1. Automated Sleep/Wake Scripts: Built orchestration scripts to scale down stateless deployments and idle GPU workers to zero when I wasn’t actively testing.
2. Karpenter Spot Instance Migration: Migrated the GPU worker pools from On-Demand instances ($4.02/hr baseline) to Spot instances ($1.40/hr), achieving a ~65% to 70% cost reduction on GPU compute.
3. Eviction Protection for State: While NIMs are stateless and can be safely terminated with Spot warnings, stateful databases like Milvus (holding embeddings for 97+ curated research PDFs), etcd, and MinIO were pinned to protected On-Demand nodes with Pod Disruption Budgets (PDBs) to prevent data corruption.

# Monitoring Karpenter Spot capacity and cost metrics
kubectl get nodes -L karpenter.sh/capacity-type
aws ce get-cost-and-usage --time-period Start=$(date -d '7 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY --metrics BlendedCost

Yet despite these aggressive cost-cutting measures, running a multi-GPU Kubernetes cluster continuously over several weeks—accounting for storage volumes, NAT gateways, data transfer, and test runs—drove my AWS bill past $1,800 USD by the time the competition concluded.

It was an unforgettable reminder of the hidden infrastructure overhead required to run true enterprise-grade GenAI backends.

Engineering War Stories: Model Pragmatism & The UDR Code Compiler

Building Research-as-a-Code surfaced deep engineering challenges that never appear in standard tutorials.

1. The NIM Reality Check: Downsizing to Nemotron Nano 8B

The original architectural plan called for deploying massive 70B parameter models (Llama-3.3-70B-Instruct) for the planning and code compilation steps.

However, reality intervened quickly:

• Hosting a 70B parameter model in FP16 or AWQ requires multiple high-end GPUs (e.g. g5.12xlarge with 4x A10G GPUs).
• AWS GPU quotas on standard accounts are heavily restricted, and running multi-GPU instances would have caused the cloud bill to exceed $5,000.
• Furthermore, initial NIM stability for large models on single-node setups was unpredictable during early hackathon preview releases.

I made the pragmatic engineering pivot to llama-3.1-nemotron-nano-8b-v1.

Operating with a lightweight 8B model kept the cluster lean and within single-GPU g5.2xlarge bounds. But it introduced an even steeper obstacle: getting an 8B model to generate zero-defect executable Python code.

2. The UDR Code Generation Minefield

When an LLM writes code for a human programmer, minor errors (like an unused import, a slight parameter name mismatch, or a missing dictionary key) are easily corrected.

In NVIDIA UDR, the code is executed autonomously by a machine sandbox. If the 8B model hallucinated even a single invalid function parameter or tried to import an uninstalled package, the entire research workflow crashed instantly!

Early test logs were littered with compilation and runtime failures:

• The model would invent fantasy methods like retriever.search_by_text(query, top_k=5) instead of retriever.search(query=query, limit=5).
• It would fail to handle asynchronous coroutines, causing unawaited Future objects to crash downstream synthesizers.
• It suffered from UnboundLocalError when variable scopes in complex generated conditionals were not properly initialized.

3. Building the AST Validator and Self-Healing Loop

To make UDR robust on smaller models, I implemented a two-tier verification layer:

1. Static AST Analysis: Before executing any generated code, an Abstract Syntax Tree (AST) validator scanned the Python code. It verified that all imports were restricted to safe built-ins and pre-approved tools (tavily, milvus_client, requests), checked variable assignments, and ensured function signatures matched our tool API contracts.
2. Runtime Self-Healing Loop: If the execution sandbox encountered a runtime exception, the error traceback, the offending code snippet, and the original research goal were bundled into a diagnostic prompt and fed back to the Nemotron NIM:

“The research code generated for query ‘X’ failed with TypeError: unexpected keyword argument 'top_k'. Here is the execution traceback. Rewrite only the affected code block to resolve the parameter error while preserving the remaining pipeline logic.”

With this self-healing correction loop in place, the compilation success rate leaped from ~45% to over 96%, enabling even the compact 8B Nemotron model to reliably compile and execute complex research strategies!

(Detailed execution logs showing the UDR engine compiling natural language strategies into Python code, invoking retrieval tools, and synthesizing comprehensive cited reports.)

4. The Diffusion Research Quirk: Managing Denoising Loops

Google’s Test-Time Diffusion approach brought its own unique set of surprises.

During early testing, the diffusion loop would occasionally terminate prematurely after just one iteration. Because the convergence detector was misinterpreting empty gap lists as “complete understanding,” the system would exit early and fall back to generic placeholder summaries.

By systematically instrumenting variable tracking across diffusion stages (as detailed in my TTD_DR_DEBUG_FINDINGS.md), I discovered that the issue wasn’t prompt wording—it was variable flow through the LangGraph state. Once the state flow was hardened and convergence thresholds were tuned between 0.85 and 0.95, TTD-DR produced remarkably rich, highly nuanced research drafts that expanded organically across iterations.

Video Demonstration

I recorded a full end-to-end video walkthrough demonstrating the architecture, EKS infrastructure, Karpenter autoscaling, and live strategy comparisons:

(Note: There was some background audio noise in the microphone during the recording, but the technical walkthrough covers the complete backend execution and UI streaming flow!)

Key Takeaways & Open Source Resources

Participating in the AWS & NVIDIA Agentic AI Unleashed Hackathon was an extraordinary learning experience that reinforced several timeless engineering principles:

1. Strategy-as-Code is Incredibly Powerful: NVIDIA’s UDR paradigm demonstrated that compiling research plans into executable Python code dramatically cuts down token consumption and latency compared to open-ended LLM multi-agent conversations.
2. Smaller Models Need Self-Healing Guardrails: You don’t always need a 70B or 400B model if you provide structured output constraints, AST static analysis, and automated runtime error-correction loops.
3. Respect Cloud Infrastructure Costs: Running real enterprise GPU workloads on AWS is expensive. When designing AI architectures, cost-awareness (Spot instances, sleep automation, lightweight model selection) must be a first-class architectural concern from day one.

The complete codebase, Terraform scripts, Helm charts, and comprehensive engineering documentation are fully open-source:

• Source Code Repository: github.com/Research-as-a-Code/Research_as_a_Code — Contains the LangGraph backend, UDR compiler, TTD-DR diffusion loop, and CopilotKit Next.js frontend.
• Infrastructure Blueprint: Check the /infrastructure and /memories folders in the repo for all Terraform modules, Karpenter provisioners, and spot migration guides.
• Devpost Submission: Research as a Code on Devpost
• Official Competition: AWS & NVIDIA Agentic AI Unleashed Hackathon 2025

A heartfelt thank you to AWS and NVIDIA for hosting such an intellectually challenging competition and giving developers the playground to build the future of agentic intelligence!