Home › Jobs › Falls Technology
Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI
Falls Technology · McLean, Virginia · Posted October 28, 2026 (22 days ago)
- McLean, Virginia
Job description
Active TS/SCI required. U.S. citizenship required.
Location: On-site in McLean, VA (Full-Time).
Prove which LLMs actually work for the mission. You'll design RAG and LLM evaluation experiments, tune models on custom data, and run GPU-accelerated benchmarks so the team ships AI systems they can trust.
Responsibilities
Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)
Design and run experiments that benchmark LLMs against defined measures
Tune neural networks / LLMs on custom datasets and apply results to concrete use cases
Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)
Use deep learning frameworks such as PyTorch, TensorFlow or Keras
Leverage GPUs for accelerated training and evaluation
Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets
Work in GitHub and Jenkins-style CI for reproducible experiments
Communicate methodological choices and model results to technical and non-technical stakeholders
What you bring (required)
Active TS/SCI
Hands-on RAG testing experience (must-have)
Tuning neural nets / LLMs on custom datasets for real use cases
Building AI/ML models and ensembles in Python or R
PyTorch, TensorFlow or Keras (key)
GPU-accelerated computing (key)
Testing, evaluating or validating LLM-based AI systems (key)
Experiment design to benchmark LLMs (key)
SQL including CTEs, set ops, aggregates and nested subqueries
GitHub and Jenkins (or equivalent) version control / CI
Preferred qualifications
Hugging Face Transformers library and hub
Building benchmarks that measure LLM performance
Cloud computing development and architecture
Flask or similar front-end/web frameworks
Text classification and topic modeling (scikit-learn or deep learning)
Semantic search applications
Tableau dashboards
Verification/validation test benches, Explainable AI (XAI), ONNX
Why Falls
A 9% retirement contribution, 100% of family medical premiums paid, and 4 weeks of PTO. We're a small business that's been delivering since 2019, so your work is visible and your voice counts.
Falls Technology is an Equal Opportunity Employer. All qualified applicants will receive consideration without regard to race, color, religion, sex, national origin, disability, or protected veteran status.
Posting 133 on bamboohr. Source: the employer’s listing. Valid through December 12, 2026 unless closed earlier.