Learn Weave with Serverless Inference - Weights & Biases Documentation

This guide shows you how to use W&B Weave with Serverless Inference to learn Weave fundamentals. Use Serverless Inference to build and trace LLM applications using live open source models without setting up your own infrastructure or managing API keys from multiple providers. With your W&B API key, you can interact with all models hosted by Serverless Inference. By the end of this guide, you’ll have traced LLM calls, compared models, and run an evaluation that you can review in the Weave UI.

What you’ll learn

This guide shows you how to:

Set up Weave and Serverless Inference.
Build a basic LLM application with automatic tracing.
Compare multiple models.
Evaluate model performance on a dataset.
View your results in the Weave UI.

Prerequisites

A W&B account
Python 3.10+ or Node.js 18+
Required packages installed:
- Python: pip install weave openai
- TypeScript: npm install weave openai
An OpenAI API key set as an environment variable.

Trace your first LLM call

This section shows you how to make a single LLM call and have Weave trace it automatically, so you can confirm your setup is working before moving on to more complex examples. To begin, copy and paste the following code example. The code example uses Llama 3.1-8B from Serverless Inference. When you run this code, Weave:

Traces your LLM call automatically.
Logs inputs, outputs, latency, and token usage.
Provides a link to view your trace in the Weave UI.

Python
TypeScript

import weave
import openai

# Initialize Weave. Replace [YOUR-TEAM] with your team name.
weave.init("[YOUR-TEAM]/inference-quickstart")

# Create an OpenAI-compatible client pointing to Serverless Inference
client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',
    api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
    project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
)

# Decorate your function to enable tracing. Use the standard OpenAI client
@weave.op()
def ask_llama(question: str) -> str:
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ],
    )
    return response.choices[0].message.content

# Call your function - Weave automatically traces everything
result = ask_llama("What are the benefits of using W&B Weave for LLM development?")
print(result)

import * as weave from 'weave';
import OpenAI from 'openai';

// Initialize Weave. Replace values enclosed in "[]" with your own.
await weave.init("[YOUR-TEAM]/inference-quickstart")

// Create an OpenAI-compatible client pointing to Serverless Inference
const client = new OpenAI({
    baseURL: 'https://api.inference.wandb.ai/v1',  // Serverless Inference endpoint
    apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY', // Replace with your API key or set the WANDB_API_KEY environment variable
});

// Wrap your function with weave.op to enable tracing
const askLlama = weave.op(async function askLlama(question: string): Promise<string> {
const response = await client.chat.completions.create({
    model: 'meta-llama/Llama-3.1-70B-Instruct',
    messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: question }
    ],
});
return response.choices[0].message.content || '';
});

// Call your function - Weave automatically traces everything
const result = await askLlama('What are the benefits of using W&B Weave for LLM development?');
console.log(result);

Build a text summarization application

Now that you’ve traced a single LLM call, this section shows how Weave traces nested operations across multiple functions, so you can see how a real multi-step LLM application is captured in the UI. Next, run this code, which is a basic summarization app that shows how Weave traces nested operations:

Python
TypeScript

import weave
import openai

# Initialize Weave. Replace values enclosed in "[]" with your own.
weave.init("[YOUR-TEAM]/inference-quickstart")

client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',
    api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
    project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
)

@weave.op()
def extract_key_points(text: str) -> list[str]:
    """Extract key points from a text."""
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "Extract 3-5 key points from the text. Return each point on a new line."},
            {"role": "user", "content": text}
        ],
    )
    # Returns response without blank lines
    return [line for line in response.choices[0].message.content.strip().splitlines() if line.strip()]

@weave.op()
def create_summary(key_points: list[str]) -> str:
    """Create a concise summary based on key points."""
    points_text = "\n".join(f"- {point}" for point in key_points)
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "Create a one-sentence summary based on these key points."},
            {"role": "user", "content": f"Key points:\n{points_text}"}
        ],
    )
    return response.choices[0].message.content

@weave.op()
def summarize_text(text: str) -> dict:
    """Main summarization pipeline."""
    key_points = extract_key_points(text)
    summary = create_summary(key_points)
    return {
        "key_points": key_points,
        "summary": summary
    }

# Try it with sample text
sample_text = """
The Apollo 11 mission was a historic spaceflight that landed the first humans on the Moon 
on July 20, 1969. Commander Neil Armstrong and lunar module pilot Buzz Aldrin descended 
to the lunar surface while Michael Collins remained in orbit. Armstrong became the first 
person to step onto the Moon, followed by Aldrin 19 minutes later. They spent about 
two and a quarter hours together outside the spacecraft, collecting samples and taking photographs.
"""

result = summarize_text(sample_text)
print("Key Points:", result["key_points"])
print("\nSummary:", result["summary"])

import * as weave from 'weave';
import OpenAI from 'openai';

// Initialize Weave. Replace with your-team/your-project
await weave.init('[YOUR-TEAM]/inference-quickstart');

const client = new OpenAI({
baseURL: 'https://api.inference.wandb.ai/v1',
apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY',  // Replace with your API key or set the WANDB_API_KEY environment variable
});

const extractKeyPoints = weave.op(async function extractKeyPoints(text: string): Promise<string[]> {
const response = await client.chat.completions.create({
    model: 'meta-llama/Llama-3.1-8B-Instruct',
    messages: [
    { role: 'system', content: 'Extract 3-5 key points from the text. Return each point on a new line.' },
    { role: 'user', content: text }
    ],
});
// Returns response without blank lines
const content = response.choices[0].message.content || '';
return content.split('\n').map(line => line.trim()).filter(line => line.length > 0);
});

const createSummary = weave.op(async function createSummary(keyPoints: string[]): Promise<string> {
const pointsText = keyPoints.map(point => `- ${point}`).join('\n');
const response = await client.chat.completions.create({
    model: 'meta-llama/Llama-3.1-8B-Instruct',
    messages: [
    { role: 'system', content: 'Create a one-sentence summary based on these key points.' },
    { role: 'user', content: `Key points:\n${pointsText}` }
    ],
});
return response.choices[0].message.content || '';
});

const summarizeText = weave.op(async function summarizeText(text: string): Promise<{key_points: string[], summary: string}> {
const keyPoints = await extractKeyPoints(text);
const summary = await createSummary(keyPoints);
return {
    key_points: keyPoints,
    summary: summary
};
});

// Try it with sample text
const sampleText = `
The Apollo 11 mission was a historic spaceflight that landed the first humans on the Moon 
on July 20, 1969. Commander Neil Armstrong and lunar module pilot Buzz Aldrin descended 
to the lunar surface while Michael Collins remained in orbit. Armstrong became the first 
person to step onto the Moon, followed by Aldrin 19 minutes later. They spent about 
two and a quarter hours together outside the spacecraft, collecting samples and taking photographs.
`;

const result = await summarizeText(sampleText);
console.log('Key Points:', result.key_points);
console.log('\nSummary:', result.summary);

Compare multiple models

A common use case for Weave is comparing how different models respond to the same prompt. Serverless Inference provides access to multiple models. Use the following code to compare the performance of Llama and DeepSeek’s respective responses:

Python
TypeScript

import weave
import openai

# Initialize Weave. Replace with your-team/your-project
weave.init("[YOUR-TEAM]/inference-quickstart")

client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',
    api_key="YOUR_WANDB_API_KEY",  # Replace with your actual API key
    project="[YOUR-TEAM]/my-first-weave-project",  # Required for usage tracking
)

# Define a Model class to compare different LLMs
class InferenceModel(weave.Model):
    model_name: str
    
    @weave.op()
    def predict(self, question: str) -> str:
        response = client.chat.completions.create(
            model=self.model_name,
            messages=[
                {"role": "user", "content": question}
            ],
        )
        return response.choices[0].message.content

# Create instances for different models
llama_model = InferenceModel(model_name="meta-llama/Llama-3.1-8B-Instruct")
deepseek_model = InferenceModel(model_name="deepseek-ai/DeepSeek-V3.1")

# Compare their responses
test_question = "Explain quantum computing in one paragraph for a high school student."

print("Llama 3.1 8B response:")
print(llama_model.predict(test_question))
print("\n" + "="*50 + "\n")
print("DeepSeek V3 response:")
print(deepseek_model.predict(test_question))

import * as weave from 'weave';
import OpenAI from 'openai';

// Initialize Weave. Replace with your-team/your-project
await weave.init("[YOUR-TEAM]/inference-quickstart")

const client = new OpenAI({
  baseURL: 'https://api.inference.wandb.ai/v1',
  apiKey: process.env.WANDB_API_KEY || 'YOUR_WANDB_API_KEY', // Replace with your API key or set the WANDB_API_KEY environment variable
});

// Create model functions using weave.op (weave.Model is not supported in TypeScript)
function createModel(modelName: string) {
  return weave.op(async function predict(question: string): Promise<string> {
    const response = await client.chat.completions.create({
      model: modelName,
      messages: [
        { role: 'user', content: question }
      ],
    });
    return response.choices[0].message.content || '';
  });
}

// Create instances for different models
const llamaModel = createModel('meta-llama/Llama-3.1-8B-Instruct');
const deepseekModel = createModel('deepseek-ai/DeepSeek-V3.1');

// Compare their responses
const testQuestion = 'Explain quantum computing in one paragraph for a high school student.';

console.log('Llama 3.1 8B response:');
console.log(await llamaModel(testQuestion));
console.log('\n' + '='.repeat(50) + '\n');
console.log('DeepSeek V3 response:');
console.log(await deepseekModel(testQuestion));

Evaluate model performance

Going beyond ad-hoc comparisons, this section shows you how to run a structured evaluation across a dataset so you can systematically measure and compare model quality. Evaluate how well a model performs on a Q&A task using Weave’s built-in EvaluationLogger. This provides structured evaluation tracking with automatic aggregation, token usage capture, and rich comparison features in the UI. Append the following code to the script you used in the prior section:

Python
TypeScript

from typing import Optional
from weave import EvaluationLogger

# Create a simple dataset
dataset = [
    {"question": "What is 2 + 2?", "expected": "4"},
    {"question": "What is the capital of France?", "expected": "Paris"},
    {"question": "Name a primary color", "expected_one_of": ["red", "blue", "yellow"]},
]

# Define a scorer
@weave.op()
def accuracy_scorer(expected: str, output: str, expected_one_of: Optional[list[str]] = None) -> dict:
    """Score the accuracy of the model output."""
    output_clean = output.strip().lower()
    
    if expected_one_of:
        is_correct = any(option.lower() in output_clean for option in expected_one_of)
    else:
        is_correct = expected.lower() in output_clean
    
    return {"correct": is_correct, "score": 1.0 if is_correct else 0.0}

# Evaluate a model using Weave's EvaluationLogger
def evaluate_model(model: InferenceModel, dataset: list[dict]):
    """Run evaluation on a dataset using Weave's built-in evaluation framework."""
    # Initialize EvaluationLogger BEFORE calling the model to capture token usage
    # This is especially important for Serverless Inference to track costs
    # Convert model name to a valid format (replace non-alphanumeric chars with underscores)
    safe_model_name = model.model_name.replace("/", "_").replace("-", "_").replace(".", "_")
    eval_logger = EvaluationLogger(
        model=safe_model_name,
        dataset="qa_dataset"
    )
    
    for example in dataset:
        # Get model prediction
        output = model.predict(example["question"])
        
        # Log the prediction
        pred_logger = eval_logger.log_prediction(
            inputs={"question": example["question"]},
            output=output
        )
        
        # Score the output
        score = accuracy_scorer(
            expected=example.get("expected", ""),
            output=output,
            expected_one_of=example.get("expected_one_of")
        )
        
        # Log the score
        pred_logger.log_score(
            scorer="accuracy",
            score=score["score"]
        )
        
        # Finish logging for this prediction
        pred_logger.finish()
    
    # Log summary - Weave automatically aggregates the accuracy scores
    eval_logger.log_summary()
    print(f"Evaluation complete for {model.model_name} (logged as: {safe_model_name}). View results in the Weave UI.")

# Compare multiple models - a key feature of Weave's evaluation framework
models_to_compare = [
    llama_model,
    deepseek_model,
]

for model in models_to_compare:
    evaluate_model(model, dataset)

# In the Weave UI, navigate to the Evals tab to compare results across models

import { EvaluationLogger } from 'weave';

// Create a simple dataset
interface DatasetExample {
  question: string;
  expected?: string;
  expected_one_of?: string[];
}

const dataset: DatasetExample[] = [
  { question: 'What is 2 + 2?', expected: '4' },
  { question: 'What is the capital of France?', expected: 'Paris' },
  { question: 'Name a primary color', expected_one_of: ['red', 'blue', 'yellow'] },
];

// Define a scorer
const accuracyScorer = weave.op(function accuracyScorer(args: {
  expected: string;
  output: string;
  expected_one_of?: string[];
}): { correct: boolean; score: number } {
  const outputClean = args.output.trim().toLowerCase();
  
  let isCorrect: boolean;
  if (args.expected_one_of) {
    isCorrect = args.expected_one_of.some(option => 
      outputClean.includes(option.toLowerCase())
    );
  } else {
    isCorrect = outputClean.includes(args.expected.toLowerCase());
  }
  
  return { correct: isCorrect, score: isCorrect ? 1.0 : 0.0 };
});

// Evaluate a model using Weave's EvaluationLogger
async function evaluateModel(
  model: (question: string) => Promise<string>,
  modelName: string,
  dataset: DatasetExample[]
): Promise<void> {
  // Initialize EvaluationLogger BEFORE calling the model to capture token usage
  // This is especially important for Serverless Inference to track costs
  // Convert model name to a valid format (replace non-alphanumeric chars with underscores)
  const safeModelName = modelName.replace(/\//g, '_').replace(/-/g, '_').replace(/\./g, '_');
  const evalLogger = new EvaluationLogger({
    name: 'inference_evaluation',
    model: { name: safeModelName },
    dataset: 'qa_dataset'
  });
  
  for (const example of dataset) {
    // Get model prediction
    const output = await model(example.question);
    
    // Log the prediction
    const predLogger = evalLogger.logPrediction(
      { question: example.question },
      output
    );
    
    // Score the output
    const score = await accuracyScorer({
      expected: example.expected || '',
      output: output,
      expected_one_of: example.expected_one_of
    });
    
    // Log the score
    predLogger.logScore('accuracy', score.score);
    
    // Finish logging for this prediction
    predLogger.finish();
  }
  
  // Log summary - Weave automatically aggregates the accuracy scores
  await evalLogger.logSummary();
  console.log(`Evaluation complete for ${modelName} (logged as: ${safeModelName}). View results in the Weave UI.`);
}

// Compare multiple models - a key feature of Weave's evaluation framework
const modelsToCompare = [
  { model: llamaModel, name: 'meta-llama/Llama-3.1-8B-Instruct' },
  { model: deepseekModel, name: 'deepseek-ai/DeepSeek-V3.1' },
];

for (const { model, name } of modelsToCompare) {
  await evaluateModel(model, name, dataset);
}

// In the Weave UI, navigate to the Evals tab to compare results across models

After you run these examples, you have traced LLM calls, a nested summarization pipeline, a model comparison, and a full evaluation logged to Weave. Running these examples returns links to the traces in the terminal. Click any link to view traces in the Weave UI. In the Weave UI, you can:

Review a timeline of all your LLM calls.
Inspect inputs and outputs for each operation.
View token usage and estimated costs (automatically captured by EvaluationLogger).
Analyze latency and performance metrics.
Navigate to the Evals tab to see aggregated evaluation results.
Use the Compare feature to analyze performance across different models.
Page through specific examples to see how different models performed on the same inputs.

Available models

For a complete list of available models, see the Available Models section in the Serverless Inference documentation.

Next steps

With the basics in place, the following resources help you go deeper with Weave and Serverless Inference:

Use the Playground: Try models interactively in the Weave Playground
Build evaluations: Learn about systematic evaluation of your LLM applications
Try other integrations: Weave works with OpenAI, Anthropic, and many more

​What you’ll learn

​Prerequisites

​Trace your first LLM call

​Build a text summarization application

​Compare multiple models

​Evaluate model performance

​Available models

​Next steps

​Troubleshooting