FAR.AI · Berkeley Office · Remote · Posted 13d ago
$290k - $450k
Frontier alignment research to ensure the safe development and deployment of advanced AI systems.
ABOUT US
FAR.AI http://FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response.
We’re structured to support that work from early research through real-world adoption:
Independent by design. We can pursue what's most impactful based on our theory of change and share what we find publicly.
A portfolio approach. Rather than focus on one single direction, we run diverse bets across the safety stack. We take promising ideas from initial experiments to deployment, informed by red-team partnerships with frontier labs and governments.
Serious infrastructure for ambitious research. A dedicated engineering team runs our compute cluster and experiment-scaling stack, so researchers spend their time on research instead of on infra.
Setting the standard. Our events convene key decision makers; our red-team works with frontier developers and governments; and our communications inform the public. Together, this drives adoption and sets the new standard in safety.
Since our founding in July 2022, we've grown to 50+ staff https://www.far.ai/about/team, published 40+ academic papers https://scholar.google.com/citations?user=FVJ24k8AAAAJ, and convened leading AI safety events https://far.ai/events/. Our work is recognized globally, with publications at premier venues such as NeurIPS, ICML including a Best Paper Honorable Mention in 2026 https://icml.cc/virtual/2026/oral/71065, and ICLR, and features in the Financial Times https://www.ft.com/content/175e5314-a7f7-4741-a786-273219f433a1, Nature News https://www.nature.com/articles/d41586-024-02218-7, Wired Magazine https://www.wired.com/story/jailbreaking-ai-models-google-anthropic-openai-spacexai/ and MIT Technology Review https://www.technologyreview.com/2020/02/28/905615/reinforcement-learning-adversarial-attack-gaming-ai-deepmind-alphazero-selfdriving-cars/. We conduct pre-deployment testing on behalf of frontier developers such as OpenAI and independent evaluations for governments including the EU AI Office https://www.far.ai/news/far-ai-selected-to-lead-eu-ai-act-cbrn-risk-consortium and publish the AI Security Leaderboard https://leaderboard.far.ai/ based on our red-teaming expertise. We help steer and grow the AI safety field through developing https://arxiv.org/abs/2405.06624 research https://arxiv.org/abs/2506.20702 roadmaps https://www.researchgate.net/publication/396910034_Open_Technical_Problems_in_Open-Weight_AI_Model_Risk_Management with renowned researchers such as Yoshua Bengio; running FAR.Labs https://www.far.ai/programs/far-labs, an AI safety-focused co-working space in Berkeley housing 40+ members; and supporting the community through targeted grants https://www.far.ai/programs/grantmaking to technical researchers.
FAR.AI http://FAR.AI is hiring a Research Lead to develop and lead our work on pre-training safety, shaping models’ capabilities and internal representations at their source, rather than trying to fix them after the fact.
Our initial focus is capability control: removing harmful capabilities while preserving benign ones. We see this as a promising way to prevent misuse of open-weight models in areas such as CBRN and cyber by removing offensive capabilities, and reducing loss-of-control risks by removing knowledge of oversight mechanisms. We will validate approaches like pre-training data filtering at scale, drive adoption of successful methods, and explore techniques such as gradient routing and unlearning..
We are scaling methods like Deep Ignorance https://deepignorance.ai/ by over an order of magnitude (>100B parameter models with >1T tokens). You will direct this work, partner with our red team to stress-test the resulting models, and analyze how well the methods scale to frontier systems.
Our research directions include:
- Improved data filtering methods, such as using data attribution (e.g. influence-based selection) or more sophisticated classifiers
- Using methods like gradient routing to isolate dual-use capabilities in components of the model (e.g. specific MoE experts)
- Training to actively remove harmful capabilities, such as interleaving next-token prediction with unlearning, as opposed to simply filtering data
- Adding synthetic data to pre-training or mid-training to shape the representations and behavior of the model
You'll build and lead the team, set its research direction, mentor Members of Technical Staff to scale your vision, and remain hands-on enough to write code and run experiments yourself. This role offers high autonomy in an impact-driven environment, pursuing empirically grounded, scalable ML safety research.
ABOUT US
FAR.AI http://FAR.AI is a non-profit AI research institute working to ensure advanced AI is safe and beneficial for everyone. Our mission is to facilitate breakthrough AI safety research, advance global understanding of AI risks and solutions, and foster a coordinated global response.
We’re structured to support that work from early research through real-world adoption:
Independent by design. We can pursue what's most impactful based on our theory of change and share what we find publicly.
A portfolio approach. Rather than focus on one single direction, we run diverse bets across the safety stack. We take promising ideas from initial experiments to deployment, informed by red-team partnerships with frontier labs and governments.
Serious infrastructure for ambitious research. A dedicated engineering team runs our compute cluster and experiment-scaling stack, so researchers spend their time on research instead of on infra.
Setting the standard. Our events convene key decision makers; our red-team works with frontier developers and governments; and our communications inform the public. Together, this drives adoption and sets the new standard in safety.
Since our founding in July 2022, we've grown to 50+ staff https://www.far.ai/about/team, published 40+ academic papers https://scholar.google.com/citations?user=FVJ24k8AAAAJ, and convened leading AI safety events https://far.ai/events/. Our work is recognized globally, with publications at premier venues such as NeurIPS, ICML including a Best Paper Honorable Mention in 2026 https://icml.cc/virtual/2026/oral/71065, and ICLR, and features in the Financial Times https://www.ft.com/content/175e5314-a7f7-4741-a786-273219f433a1, Nature News https://www.nature.com/articles/d41586-024-02218-7, Wired Magazine https://www.wired.com/story/jailbreaking-ai-models-google-anthropic-openai-spacexai/ and MIT Technology Review https://www.technologyreview.com/2020/02/28/905615/reinforcement-learning-adversarial-attack-gaming-ai-deepmind-alphazero-selfdriving-cars/. We conduct pre-deployment testing on behalf of frontier developers such as OpenAI and independent evaluations for governments including the EU AI Office https://www.far.ai/news/far-ai-selected-to-lead-eu-ai-act-cbrn-risk-consortium and publish the AI Security Leaderboard https://leaderboard.far.ai/ based on our red-teaming expertise. We help steer and grow the AI safety field through developing https://arxiv.org/abs/2405.06624 research https://arxiv.org/abs/2506.20702 roadmaps https://www.researchgate.net/publication/396910034_Open_Technical_Problems_in_Open-Weight_AI_Model_Risk_Management with renowned researchers such as Yoshua Bengio; running FAR.Labs https://www.far.ai/programs/far-labs, an AI safety-focused co-working space in Berkeley housing 40+ members; and supporting the community through targeted grants https://www.far.ai/programs/grantmaking to technical researchers.
ABOUT FAR.RESEARCH
We explore promising research directions in AI safety and scale up only those showing a high potential for impact. When an approach proves effective, we develop it into a minimum viable demonstration and work with AI developers and governments to support real-world adoption.
Our recent and ongoing research includes:
Adversarial Robustness: working to rigorously solve security problems through building a science of security and robustness for AI, from demonstrating superhuman systems can be vulnerable https://far.ai/post/2023-07-superhuman-go-ais/, to scaling laws for robustness https://www.far.ai/news/does-robustness-improve-with-scale and jailbreaking constitutional classifiers https://arxiv.org/abs/2506.24068.
Mechanistic Interpretability: finding https://arxiv.org/abs/2502.12892 issues https://arxiv.org/abs/2508.16560 with https://arxiv.org/abs/2505.11756 Sparse Autoencoders, probing deception using AmongUs https://arxiv.org/abs/2504.04072, understanding learned planning https://far.ai/post/2024-07-learned-planners/ in SokoBan, and interpretable data attribution.
Red-teaming: conducting pre- and post-release adversarial evaluations of frontier models (e.g. Claude 4 Opus https://x.com/ARGleave/status/1926138376509440433, ChatGPT Agent https://cdn.openai.com/pdf/839e66fc-602c-48bf-81d3-b21eacc3459d/chatgpt_agent_system_card.pdf, GPT-5 https://cdn.openai.com/gpt-5-system-card.pdf); developing novel attacks https://www.far.ai/news/defense-in-depth to support this work.
Evals: developing evaluations for new threat models, e.g. persuasion https://arxiv.org/abs/2506.02873 and tampering risks https://arxiv.org/abs/2507.11630, and launching a new research agenda on eval awareness
Mitigating AI deception: studying when lie detectors induce honesty or evasion https://www.far.ai/news/avoiding-ai-deception, and developing approaches https://www.far.ai/research/the-obfuscation-atlas-mapping-where-honesty-emerges-in-rlvr-with-deception-probes to deception and sandbagging.
Applied Interpretability: using interpretability to tackle concrete safety problems (better probes, backdoor detection, deception monitoring), aiming for fast feedback loops, often in collaboration with our other pods.
ABOUT THE ROLE
Research Leads define and own a research workstream end-to-end. Day-to-day, that means:
- Articulate a research agenda with a clear theory of change for mitigating catastrophic risks from human-level or superhuman AI systems, and/or vastly increasing the upside of such systems.
- Grow and lead a team of technical staff in pursuit of this agenda, either directly or in partnership with an engineering co-lead.
- Lead novel research projects where there may be unclear markers of progress or success.
- Share your research findings through written content (e.g. academic publications, blog posts) and presentations (e.g. ML conferences, policymaker briefings) to drive adoption and change.
- Mentor and coach junior team members in research skills and ML engineering.
- Contribute to the FAR.AI http://FAR.AI intellectual environment and research culture, for example by giving feedback on early-stage proposals.
- Build a research field around your agenda through FAR.AI http://FAR.AI's grantmaking and events, and connect it to real-world deployments through our independent testing and government advising.
This role would be a great fit if you:
- Want to work on the most impactful research directions, alongside mission-driven colleagues who'll push them forward with you.
- Wish to pursue empirically grounded, scalable research directions that lean, technically strong teams can drive forward.
- Value the ability to speak freely. We don't censor our researchers. We just ask that you protect confidential information and make clear when you're speaking personally or on behalf of the organization.
- Want to advise and collaborate with governments, leading AI companies, and academics. We're a small organization that punches above its weight by working closely with these partners: through red-teaming, technical standards work, and research collaborations.
This role would be a poor fit if you: