Fine Tuning LLMs for Function Calling w/Pawel Garbacki

Fine-tuning large language models for function calling requires selecting an objective—ranging from simple forced calls to complex multi-turn chats—that aligns with specific use cases. Preserving existing instruction-following capabilities is best achieved by tuning on top of instruct-optimized models rather than base models. LoRa tuning offers a resource-efficient path for these iterations, especially when working with limited, high-quality datasets. Implementing a special function-call token improves parsing reliability, while constrained generation using context-free grammars effectively eliminates hallucinations and accelerates inference by short-circuiting token generation. When scaling to hundreds of functions or complex agentic systems, developers should prioritize high-quality synthetic data, leverage KV cache pre-population for efficiency, and consider semantic retrieval to manage context window limitations. Pawel Garbacki from Fireworks AI emphasizes that while fine-tuning is rewarding, it remains an iterative, labor-intensive process that demands rigorous hyperparameter tuning and evaluation.