Bringing Whisper and LLaMA to the masses

This week we’re talking with Georgi Gerganov about his work on Whisper.cpp and llama.cpp. Georgi first crossed our radar with whisper.cpp, his port of OpenAI’s Whisper model in C and C++. Whisper is a speech recognition model enabling audio transcription and translation. Something we’re paying close attention to here at Changelog, for obvious reasons. Between the invite and the show’s recording, he had a new hit project on his hands: llama.cpp. This is a port of Facebook’s LLaMA model in C and C++. Whisper.cpp made a splash, but llama.cpp is growing in GitHub stars faster than Stable Diffusion did, which was a rocket ship itself.

Discuss on Changelog News

Changelog++ members get a bonus 12 minutes at the end of this episode and zero ads. Join today!

Sponsors:

• Postman – Build APIs together — More than 20 million developers use Postman for building and using APIs. Postman simplifies each step of the API lifecycle and streamlines collaboration so you can create better APIs—faster.
• Sentry – Session Replay! Rewind and replay every step of the user’s journey before and after they encountered an issue. Eliminate the guesswork and get to the root cause of an issue, faster. Use the code CHANGELOG and get the team plan free for three months.
• Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com
• Typesense – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You iterlly can’t get any faster!

Featuring:

• Georgi Gerganov – Mastodon, Twitter, GitHub, Website
• Adam Stacoviak – Mastodon, Twitter, GitHub, LinkedIn, Website
• Jerod Santo – Mastodon, Twitter, GitHub, LinkedIn

Show Notes:

• ggerganov/whisper.cpp
• examples/main
• Arm Neon technology
• Apple’s secret M1 coprocessor
• ggerganov/llama.cpp
• Introducing LLaMA: A foundational, 65-billion-parameter large language model
• facebookresearch/llama
• Ludacris Llama Llama Red Pajama Freestyle
• The Changelog #506: Stable Diffusion breaks the internet with Simon Willison
• Large language models are having their Stable Diffusion moment

Something missing or broken? PRs welcome!

Timestamps:

(00:00) - This week on The Changelog
(01:20) - Sponsor: Postman
(04:09) - Start the show!
(12:03) - Why is Whisper interesting to us?
(17:04) - What's involved in making a port?
(22:55) - Sponsor: Sentry
(24:51) - One layer deeper
(27:57) - Examples of Whisper.cpp
(31:49) - Whisper.cpp and speaker detection
(39:25) - What did you learn about Apple Silicon?
(42:26) - Apple's secret M1 coprocessor
(44:56) - GPU support on the roadmap
(47:06) - Cultivating contributions
(48:49) - Ludacris Llama Llama Red Pajama
(52:57) - What is Llama.cpp so interesting?
(57:01) - What are you going from here?
(58:22) - How can this be extended?
(1:01:22) - How did you learn this stuff?
(1:08:48) - Wrapping up
(1:10:09) - Outro

Episode source