AI Benchmark Engineer, Native Language Specialist (Indonesian, freelance)
LILT AI (United States, remote)
- Oct 2026 to Present
- L2:lilt-ai-2026
I design and build Indonesian-language tasks for Terminal-Bench, an agentic coding benchmark used to evaluate frontier AI models (OpenAI GPT, Google Gemini, Anthropic Claude) on real-world terminal and programming work. The tasks test whether AI agents handle Indonesian language and cultural conventions in realistic workplace data: government records, family registration, finance and procurement.
Highlights
- Built each task end to end: native-language datasets, Dockerized environments, Python reference and control solutions, and pytest verifiers with held-out data, sandboxed reruns and tamper checks
- Ran agentic probe experiments across multiple LLMs to measure difficulty before submission, and used failure analysis to separate language-understanding gaps from generic coding errors
- Worked through rigorous peer review: reproduced reviewer findings, closed specification gaps without weakening difficulty and kept documentation and evaluation evidence consistent
- Set up a multi-agent R&D workflow (research, red-teaming and verification agents) to discover and validate new evaluation ideas
Tech stack
Python
Docker
pytest- Harbor
- Terminal-Bench 2.0
- LLM-as-judge
- LLM evaluation