Loading Events

« All Events

  • This event has passed.

Mixture of Experts: Harnessing the Hidden Architecture of GPT4

October 25 @ 6:00 pm - 7:00 pm

Mixture of Experts with GPT4

UPDATE – this meetup was recorded and can be watched here. There is also a transcript of the meetup provided by Huntsville AI Transcribe.

We’re back at HudsonAlpha this week to talk about Mixture of Experts. Josh Phillips has been working with MoE across several LLM based projects over the last few months. He will give us an overview of MoE and the current progress on this pattern among the open source community.

In today’s AI landscape, we often face a trade-off between the capabilities of dense language models and the computational cost associated with running them. The Mixture of Experts (MoE) architecture offers an intriguing solution to this dilemma. By replacing standard linear layers with a Gating Mechanism that routes tasks to specialized ‘experts,’ MoE models manage to keep computational costs stable while expanding the model’s memory footprint. This approach allows for high performance without the usual spike in inference time. This however comes at the cost of increased complexity during training and inference.

The Mixture of Experts architecture is widely known to be the “secret sauce” behind the GPT4 model currently available from OpenAI, although concrete details of their implementation of the architecture are not currently known.

In this talk we will explore the building blocks of the mixture of experts architecture and how it compares to traditional transformers. We will also explore the current paradigms for implementing mixture of experts, example projects, and the challenges being faced by each.



Repos / Code

Misc :



October 25
6:00 pm - 7:00 pm


601 Genome Way Northwest
Huntsville, AL 35806
+ Google Map