Before it ships
Four checks. Match the method they use now, test it on later years, read the bill, and have a person sign it.
First
Match the spreadsheet they already trust
Before I claim a model is better, I reproduce the plant’s current method to within half a percent.
Second
Test it on years it has never seen
The model is scored on later years, not the years it trained on, and ranked by what a failure would cost.
Third
Read the bill before you scale it
On one real bill of 1,781 charges, 94% of the money went to reading long documents. Chat was 6%.
Fourth
A person signs the result
The write-up can only cite numbers the model actually produced. Someone accountable approves it before it changes a decision.
One real bill, 1,781 charges
Almost all of the money was spent reading long documents, not answering questions. The fix is a page limit that stops the job before it runs.
94% of the bill
Reading documents, at $0.028 a page
6% of the bill
Chat, at a fraction of a cent