Artifact 01

Enterprise policy questions in five file formats

Ask a question about a company's policies and get an answer that points to the pages it came from. The same policies exist as PDF, Word, PowerPoint, HTML and Markdown files, so the artifact also tests whether the answer changes when only the file format does.

Status: built, not certified. It can be tried after sign-in. Sign-in is required because each question uses paid GPU time.

Sign in to try itDevelopment measurements
Known limits

No sealed acceptance test has been run, so there is no certificate.

No laptop, workstation or other customer hardware profile has been tested.

Development scores come from an automated judge that has not been validated.

Some answers on the development set stated facts the documents do not support. The rate is on the performance page.

The system does not yet reliably decline questions the documents cannot answer.

What it is built from

documents

236 policy documents, each in 5 file formats

text pieces searched

1,105

companies

5, all fictional

licence

MIT, fully synthetic

search

meaning-based and keyword search, merged, then re-ranked

search model

ibm-granite/granite-embedding-english-r2

re-ranking model

Alibaba-NLP/gte-reranker-modernbert-base

answer model

qwen3-4b-tuned (qwen3-8b-tuned also available)

where it runs

hosted GPU on Modal, not on a laptop