Sherlock Tool: Monitor LLM Requests & Tokens Live

Discover Sherlock, the powerful debugging tool that monitors LLM API requests in real-time, tracks token usage, and saves everything as markdown files.

What Is Sherlock and Why You Need It

Sherlock is a revolutionary debugging tool designed specifically for developers working with Large Language Models (LLMs). As AI applications become increasingly complex, understanding what your LLM is actually sending and receiving becomes crucial for optimization and troubleshooting. Sherlock acts as a transparent middleware layer that intercepts every API request between your application and the LLM service. This visibility is essential for developers who want to understand token consumption patterns, debug unexpected responses, or simply gain insights into their AI application's communication flow. The tool provides real-time monitoring capabilities that were previously difficult to achieve without custom logging solutions.

Real-Time Request Monitoring Capabilities

One of Sherlock's standout features is its ability to display every LLM request as it happens in real-time. This live monitoring functionality allows developers to see exactly what prompts are being sent, what responses are received, and how the conversation flows between their application and the AI model. The tool captures all request metadata, including headers, timestamps, and response codes, providing a comprehensive view of the communication process. This real-time visibility is particularly valuable during development and testing phases, where understanding the exact nature of API interactions can help identify issues, optimize prompts, and ensure that your application is behaving as expected. The live feed creates an intuitive debugging experience that eliminates guesswork.

Token Usage Tracking and Visualization

Token consumption is a critical factor in LLM applications, directly impacting both performance and costs. Sherlock provides an engaging real-time token counter that updates as requests are processed, giving developers immediate feedback on their token usage patterns. This feature helps identify expensive operations, optimize prompt engineering strategies, and manage API costs more effectively. The visual representation of token consumption makes it easy to spot trends and anomalies in usage patterns. Developers can quickly identify which parts of their application consume the most tokens and make informed decisions about optimization. This real-time tracking eliminates the need to wait for billing statements or manually calculate token usage, providing instant visibility into resource consumption.

Markdown Export and Data Persistence

Sherlock automatically saves all captured request and response data in markdown format, creating a permanent record of your LLM interactions. This documentation feature is invaluable for maintaining audit trails, sharing debugging information with team members, and analyzing historical patterns in AI behavior. The markdown format ensures that the exported data is human-readable and can be easily integrated into documentation systems, version control, or shared via collaboration platforms. This persistent storage capability transforms ephemeral API interactions into tangible documentation that can be referenced later for troubleshooting, compliance purposes, or performance analysis. The structured markdown output makes it simple to search through historical data and identify patterns or recurring issues.

Integration and Setup Process

Implementing Sherlock into your existing LLM workflow is designed to be straightforward and minimally invasive. The tool acts as a proxy layer, requiring minimal configuration changes to your existing codebase. Most implementations involve simply routing your LLM API calls through Sherlock's monitoring interface rather than directly to the AI service provider. This architecture means you can add comprehensive monitoring capabilities without significantly refactoring your application. The setup process typically involves configuring endpoints, authentication credentials, and output preferences. Once configured, Sherlock operates transparently, ensuring that your application's functionality remains unchanged while gaining powerful debugging and monitoring capabilities. The tool supports various LLM providers and can be customized to match different workflow requirements.