Improve workshop instructions and instrumentation examples - #1
DaRubberDuckieee wants to merge 7 commits into
Conversation
Yash Vedanaparti (yashv19)
left a comment
Yash Vedanaparti (yashv19)
left a comment
There was a problem hiding this comment.
I like the restructuring of these exercises so that they walk you through step by step. Left some comments on wordsmithing.
Open question: give that these exercises walk you through the solution step by step, do we need solution.md files anymore?
| and seed traces after each one. This makes it clear what each change adds. | ||
|
|
||
| Seed traces with: | ||
| `scripts.seed` is a traffic generator, not tracing code. For each run, it |
There was a problem hiding this comment.
Minor tweak, but I'd remove the "not tracing code" bit.
| `llm` span, with metrics automatically parsed. | ||
| `@traced` captures function arguments and return values automatically. Use | ||
| `current_span().log()` when you need an extra field for filtering or a smaller, | ||
| safer representation of large data. |
There was a problem hiding this comment.
i think we should also call out here that you can use span.log() for full control over what gets logged
| handles a local path and an attachment hydrated from a dataset row. | ||
|
|
||
| Keep `has_attachments`. The boolean gives you a fast filter. The attachment in | ||
| the root input gives a reviewer the actual file to open. |
There was a problem hiding this comment.
I think we should remove lines 36-40, they're a bit redundant.
| ## Step 4: Export one row per run with SQL | ||
|
|
||
| ## Task | ||
| Open the **SQL sandbox**. Start with this query: |
There was a problem hiding this comment.
we should call out why we would do this and the use cases, along w what the actual query is fetching
|
|
||
| Before you can evaluate the agent, you need a set of test cases to run it | ||
| against. In practice the best test cases are the requests real users already sent. So you curate them from production logs. | ||
| Turn real agent traces into a focused dataset for one behavior: drafting customer emails. Create a dataset named **email-drafting**. You will use it in the next two exercises. |
There was a problem hiding this comment.
here also, rather than just saying "do this", we should call out why we would do something like this - e.g. production carries the most signal, curate datasets for specific scenarios, etc.
| Register both on the `learn-bt` project with `project.scorers.create(...)`, then | ||
| push: | ||
| ~~~python | ||
| async def valid_email(trace=None): |
There was a problem hiding this comment.
Maybe we do this in a different PR, but one thing we can do here is make the valid_email scorer a span level scorer and the resolution scorer a trace level. That way, the exercise also shows the different contexts in which scorers can run
| Replace the commented-out **Eval** template with: | ||
|
|
||
| ~~~python | ||
| Eval( |
There was a problem hiding this comment.
might be worth calling out here what our Eval code does
| metadata.feedback | ||
| ~~~ | ||
|
|
||
| If the write did not land on the root span, revise the view in Loop and test again. A useful custom view improves the path to a decision and reliably stores review evidence. |
There was a problem hiding this comment.
A summary sentence like this at the end of each exercise would be awesome - it clearly calls out what we did here and what value it serves
| # 4.2 Create a human review scorer [UI] | ||
|
|
||
| ## Task | ||
| Automated scorers give you repeatable checks. A human-review scorer captures the judgment a domain expert makes when the rubric is too nuanced for code or an LLM judge. |
There was a problem hiding this comment.
this is slightly inaccurate. The use case for human review scorers isn't only if "the rubric is too nuanced for an llm judge". Its very common for teams to use both human and llm judge reviews
| ## Step 1: Create the saved parameters | ||
|
|
||
| ### 1. Create and push a parameters object | ||
| Open **evals/parameters.py**. Remove the unused **create_model** import. Replace the commented-out template with: |
There was a problem hiding this comment.
Should add a sentence or two on what parameters are
2 participants
Summary
Validation
git diff --check