Why you should write your own LLM benchmarks — with Nicholas Carlini, Google DeepMind

Linking pages

Emulating Humans with NSFW Chatbots - with Jesse Silver https://www.latent.space/p/nsfw-chatbots 1 comment
Efficiency is Coming: 3000x Faster, Cheaper, Better AI Inference from Hardware Improvements, Quantization, and Synthetic Data Distillation https://www.latent.space/p/nyla 0 comments
From API to AGI: Structured Outputs, OpenAI API platform and O1 Q&A — with Michelle Pokrass & OpenAI Devrel + Strawberry team https://www.latent.space/p/openai-api-and-o1 0 comments

Linked pages

GitHub - carlini/printf-tac-toe: tic-tac-toe in a single call to printf https://github.com/carlini/printf-tac-toe 259 comments
How I Use "AI" https://nicholas.carlini.com/writing/2024/how-i-use-ai.html 181 comments
[2311.17035] Scalable Extraction of Training Data from (Production) Language Models https://arxiv.org/abs/2311.17035 120 comments
The International Obfuscated C Code Contest http://ioccc.org/ 119 comments
[2302.10149] Poisoning Web-Scale Training Datasets is Practical https://arxiv.org/abs/2302.10149 95 comments
[2403.06634] Stealing Part of a Production Language Model https://arxiv.org/abs/2403.06634 51 comments
Yet Another Doom Clone (In 13kb of JavaScript) https://nicholas.carlini.com/writing/2019/javascript-doom-clone-13k.html 40 comments
The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka https://www.latent.space/p/yitay 4 comments
My benchmark for large language models https://nicholas.carlini.com/writing/2024/my-benchmark-for-large-language-models.html 2 comments
Segment Anything 2: Demo-first Model Development https://www.latent.space/p/sam2 2 comments
AI Fundamentals: Benchmarks 101 https://www.latent.space/p/benchmarks-101 1 comment
Cursor.so: The AI-first Code Editor — with Aman Sanger of Anysphere https://www.latent.space/p/cursor 1 comment
Llama 2, 3 & 4: Synthetic Data, RLHF, Agents on the path to Open Source AGI https://www.latent.space/p/llama-3 1 comment
Digital Logic Gates on Conway's Game of Life - Part 1 https://nicholas.carlini.com/writing/2020/digital-logic-game-of-life.html 0 comments
[2012.07805] Extracting Training Data from Large Language Models https://arxiv.org/abs/2012.07805 0 comments
Latent Space | swyx | Substack https://www.latent.space/ 0 comments
GitHub - carlini/yet-another-applied-llm-benchmark: A benchmark to evaluate language models on questions I've previously asked them to solve. https://github.com/carlini/yet-another-applied-llm-benchmark 0 comments
AI Magic: Shipping 1000s of successful products with no managers and a team of 12 — Jeremy Howard of Answer.ai https://www.latent.space/p/answerai 0 comments
Is finetuning GPT4o worth it? - Latent Space https://www.latent.space/p/cosine 0 comments