Milwaukee SPIN

The Software Process Improvement Network is comprised of individuals dedicated to improving the software engineering practice. Regional groups called "SPINs" meet and share their experiences - initiating and sustaining software process improvement programs.

Mastering LLM Instrumentation and Test Automation

Building and scaling LLM-powered applications is no walk in the park! Unlike traditional software, LLMs can unpredictably generate different outputs from the same input, making it incredibly tricky to optimize, troubleshoot, and maintain these systems. Even the smallest tweak could cause the entire chain to break. So, how can we turn this wild unpredictability into a reliable system at scale? Enter LangSmith, a tool designed to tame the complexities of LLM instrumentation and test automation.
 
In this session, we’ll dive into how LangSmith offers deep insights into the behavior of large language models, revealing performance bottlenecks, bias patterns, and other anomalies—all in real-time! We’ll also roll up our sleeves and get hands-on with live demos, showcasing how to implement automated testing pipelines that validate LLMs across various tasks and domains.
 
This isn't just theory—this is battle-tested automation designed to make life easier for AI engineers and DevOps pros working in the fast-paced world of LLMs. Whether you're optimizing for performance, debugging unpredictable behavior, or ensuring ethical compliance, LangSmith (or tools like it) will be your secret weapon to streamline development cycles and raise the bar on reliability.
 
By the end of this talk, you’ll walk away with actionable, cutting-edge strategies to productionalize LLM applications with confidence. Expect a fast-paced, demo-rich session full of insights that will change how you manage and optimize your AI-driven projects!
Date and Time
Tuesday, 11/12/2024
5:30 PM - 6:00 PM Free Food and Networking 6:00 PM - 6:15 PM Announcements 6:15 PM - 7:30 PM Presentation and Q&A
 
Location
Milwaukee Tool, 13135 W Lisbon Rd, Brookfield, WI 53005 Click here for map.
 
Cost
FREE - Thanks to our sponsors!
Speaker Bio:
Cameron Vetter is a technologist with 25 years of experience using Microsoft tools and technologies to create products. He has experience in Technical, Leadership, and Managerial roles in many different sized organizations.
 
Cameron is the AI Practice Director at New Resource Consulting, a technological innovation leader focused on delivering transformative business impact to clients. He helps clients build technology roadmaps and Architect, Design, and Develop software focusing on Deep Learning, Artificial Intelligence, Cloud Architecture, and Mixed Reality.
 
Cameron enjoys sharing what he has learned by speaking at national, regional, and local conferences, including THAT Conference, Microsoft Ignite, The Midwest Architect Community Conference, AI DevWorld, Codemash, and many others.
 
In 2019, he was named a Microsoft MVP for Artificial Intelligence– one of the first honorees in the U.S. and he is now a 6 time MVP award winner for evangelism in the Deep Learning and Generative AI space.