What is Minecraft Companion?

Minecraft Companion (or MC Companion) is an interactive AI-powered companion for Minecraft. Controlled through a textual user interface (TUI), you can customize MC Companion's name, backstory, appearance, and voice. You can communicate with your companion through voice or in-game chat. Furthermore, MC Companion can respond to you through voice and in-game chat. Additionally, your companion is fully aware of its surroundings and can interact with blocks and entities in the game.

How does it work?

MC Companion is built using several connected systems. These systems meet at the Node.js application, which acts as the bridge between external systems and the Minecraft client. To interface with Minecraft, we use the Mineflayer library on NPM. It is an incredibly powerful library that allows you to fully control a Minecraft client using code.

For our AI layer, we primarily use ChatGPT for player interactions and controlling the Minecraft bot. To allow an LLM more structured control of the Minecraft client, we developed a skill layer that enables the model to call functions in chains to complete complex tasks. For example, the model could receive a relatively simple-sounding command such as "collect some wood" and break the request down into multiple actions, executing them in succession. For example, the previous task could be broken down into:

1. Identify nearby wood blocks
2. Move to the closest wood block
3. Mine the block

For our voice layer, we used the ElevenLabs API to convert ChatGPT's output into audio, and we used on-device language processing to achieve seamless speech-to-text functionality, creating a smooth voice interaction experience with your companion. Additionally, we added an optional ESP32 extension to our project that uses LEDs to indicate your companion's current mood.

The Building System

Our building system is one of the most complex components of MC Companion. When the model is prompted to build something, it enters building mode. In this mode, the companion uses a multi-step approach to iteratively refine the current build. It starts with a rough outline of what you asked it to build and then iterates through multiple steps to add detail and refine the structure. The initial sketch is generated using a faster, less performant model such as Gemini Flash, while later additions are handled using more advanced models such as Gemini Pro.

Features

• Skill-powered interactions with the Minecraft world
• Advanced building generation algorithm
• Text and voice interaction
• Step-based goal decomposition
• Pathfinding and terrain navigation
• Block collection and inventory awareness
• Hardware integration through ESP32 to indicate mood (optional)

My Contribution

My main responsibility was implementing the skill-based framework that allowed AI agents to utilize Mineflayer's capabilities. Furthermore, I implemented the advanced pathfinding and terrain navigation system that enabled the agents to effectively navigate the Minecraft world. Lastly, I was responsible for refining the step-based goal decomposition and planning algorithms, ensuring that the agent could break down complex tasks into smaller, manageable steps and execute them effectively.

Inspiration

The main inspiration for Minecraft Companion was the recently published AI Minecraft build generator. We wanted to implement something more advanced that could create buildings from scratch without extensive prior training. Additionally, we wanted to create a more interactive and engaging experience than simply providing a tool that builds structures for you. As a result, we decided to create an AI companion that could play Minecraft with you and assist you in your adventures.

My team after we won 3rd place at EurekaHacks 2026

Extremely proud to have won 3rd place at EurekaHacks 2026 out of 42 projects!