A lot of SaaS and API platforms now publish agent skills. We did too, but I cannot tell how useful they are for real users.
We have evals for testing that the skills work under ideal conditions, but I'm interested in runtime behaviour in the wild. For instance:
How often did an agent fall back to web search instead of following what the skill said?
How often does an installed skill get triggered for an average user?
Is anyone tracking real use, beyond the install counts that platforms like skills.sh report? Does it even make sense to try and get these insights?
We have evals for testing that the skills work under ideal conditions, but I'm interested in runtime behaviour in the wild. For instance:
How often did an agent fall back to web search instead of following what the skill said? How often does an installed skill get triggered for an average user?
Is anyone tracking real use, beyond the install counts that platforms like skills.sh report? Does it even make sense to try and get these insights?