This is a nice analysis, as expected, from Hamel and Antaripa. Good illustration of more traditional analysis in the newer domains of LLM evals and decision models.
For critique I would say it’s pushing the boundaries of decision models by using a very broad prompt and single decision vs breaking the trace eval into several smaller decisions.
Really appreciate the ongoing work from @HamelHusain and @doesdatmaksense. They are so helpful for advancing this eval field.