ElevenLabs 纽约峰会发布多项更新：Interaction Engine、ElevenAgents 升级与 ElevenAI

AI is more capable than ever, but to make that intelligence widely accessible, systems need to be easy for everyone to understand and use. That depends on those interactions feeling natural and intuitive. ElevenLabs is building that interaction layer for AI. 

At its New York Summit today, ElevenLabs introduced new models, agents, and deployment options to help businesses interact with people through AI and find new ways to generate revenue.

“Within a few years, conversational agents will become critical infrastructure for every enterprise, as essential as cloud or payments,” said Mati Staniszewski, CEO and co-founder of ElevenLabs. “Every business will need to reinvent how it interacts with its customers, employees, and the wider world.”

Here’s what ElevenLabs announced at the Summit:

Advances in the models powering every interaction

We started with voice and built the industry’s leading models for natural conversation, and our work is now expanding to the models and tools powering all the ways people interact with technology.

Onstage, Staniszewski previewed several updates that will make interactions between AI systems, customers, and employees feel more natural and better suited to the situation. “Our mission at ElevenLabs is to solve human-technology interaction, and for the first time, it’s solvable,” he said. 

Fine-tune open-source models and ElevenLabs voice models for your domain

Every industry has its own language and norms. How you serve customers, and the best channel for that conversation, differs from one industry to the next. That’s why ElevenLabs has worked to build dedicated models for major industries. Domain-specific Speech to Text models learn the terms that industries a business actually uses, like EBITDA in finance, or HIPAA in government. ElevenLabs used this approach to build Scribe v3 Medical, a transcription model for healthcare. It can recognize thousands of complex medical terms with much greater accuracy, and captures the difference between drug names like cycloserine and cyclosporine, so a patient ends up with a tuberculosis treatment, not an immunosuppressant.  

This work won’t be slowing down, and Staniszewski said that a domain-optimized model for the financial services industry is in the works. But we also believe businesses should be able to refine the models they use not just for their industry, but for their specific context and customers.   

In the keynote, Staniszewski announced that in the coming months, we will help businesses fine-tune certain open-source language models, as well as ElevenLabs voice models, for their own use. “We will not solve human-AI interaction without reflecting what makes people and businesses distinctive,” he said.

Teams will be able to fine-tune models to the specific idiosyncrasies that makes their company unique. Businesses can bring their own data from their live deployments and work with ElevenLabs researchers to refine the models powering them. Each business can pick the metrics that matter to them, such as a customer satisfaction score (CSAT) or reliability, and ElevenLabs researchers will help you take data from your live deployments to fine-tune the models that power them. You can continuously optimize based on each new interaction; Staniszewski compared it to each company essentially having their own reinforcement-learning stack.  

“We will continue shipping more domain-optimized models to ensure your AI is more accurate and more useful, but we also think businesses should be able to go further, with models refined not just for their industry, but for their company and customers.,” Staniszewski said. “It’s one of the best ways to make your data work for you, and extend your own unique advantage.”

Interaction Engine orchestrates the whole conversation 

Not every agentic task requires a frontier model at every step to succeed. Determining which models to use, how, and when to drive a specific outcome is an art in itself, one that ElevenLabs researchers have been working to solve. 

ElevenLabs previewed a new system, called Interaction Engine, which starts with the rhythm of the conversation. It handles turn-taking and interruptions, so the agent knows when to jump in and when to stop talking. It can also pick  the right LLM for each part of an exchange. A cheaper model can handle triage, keeping token use in check, and a frontier model steps in to handle more complex tasks as needed.

Major upgrades to ElevenAgents and more

ElevenAgents is the platform for configuring, deploying, and improving voice and chat agents across sales, support, recruitment, and operations. It understands who it’s talking to and adapts to what they need. 

Customers and partners like Yahoo, Westgate, and KPN are already putting ElevenAgents to work and seeing that agents who better understand context and culture deliver better outcomes for everyone.

Staniszewski shared several updates coming to ElevenAgents that will make interactions between agents and customers more seamless and natural. 

Memory helps ElevenAgents remember returning customers

In customer support conversations, people often share valuable information that isn’t captured or acted on.  “One of the hallmarks of a great conversation with a person is that they remember what you told them and act on it. Agents should do the same,” Staniszewski said.

Memory will let an ElevenAgents agent develop a record of a customer over time, recognizing a returning authenticated customer and recalling what they said before. While the CRM remains the primary customer record, Memory captures the details shared in conversation that never had a field, things like preferences, constraints, upcoming occasions. If a customer prefers Spanish and can only talk after 5pm, the agent remembers and acts on it.

After each interaction, the agent distills the conversation into a structured record and updates it when the customer returns. It also holds on to details that matter later. A phone subscriber who mentioned he was moving in March doesn’t need to bring it up again when he calls in April: the agent automatically knows to ask whether he’d like to transfer service to the new address, and retires the old one. The result is an understanding of each customer that compounds with every conversation, so every interaction feels more personal than the last.

Routines handle the work around conversations

Most of the follow-up work for a customer support call happens after the call ends. It’s often done by people and tools that weren’t part of the original conversation. Routines, the new automation layer in ElevenAgents, will allow users to run multi-step work before, during, and after every conversation. It connects to tools and systems your team already works in. 

A Routine can pre-screen a mortgage applicant and call back within minutes, or remind patients ahead of a visit, and offer a canceled slot from the waitlist.They can also be built to email a report, write to a CRM, or start a call, all natively in ElevenAgents.

With Routines, ElevenAgents takes on more of the work around every conversation, so your team spends less time on prep and follow-up and more on customers.

ElevenAI helps people stay present in the conversations that matter

In a sneak preview during his keynote, Staniszewski introduced ElevenAI, a new app from ElevenLabs. It functions as an AI teammate for enterprise teams; it listens to spoken conversations, including in-person meetings, and turns those into notes and next steps. If ElevenAgents is the interaction layer between businesses and customers, ElevenAI will be that layer for employees getting work done. 

ElevenAI offers dictation and meeting notes powered by Scribe v3, ElevenLabs’ newest and most accurate Speech to Text model. In meetings, it takes notes with everyone’s consent, then delivers a summary and action items for each attendee when the call ends. It can also give live assistance during a call, sharing context and enhancing collaboration between people and agents. Notice and consent mechanisms are built in, so the people on the other end feel comfortable.

As people rely on personal agents more in their daily lives, they’ll need support at work that’s built for them as employees. ElevenAI adapts to a company’s tools and processes and stays with people throughout their workday. It also passes the right context between agents and people, so the two can work together as one team. Free early access to ElevenAI is opening today. 

Expanding the enterprise ecosystem with the ElevenLabs Partner Program and ElevenLabs Marketplace

ElevenLabs’ partner network has grown to more than 550 partners in 44 countries, including Google Cloud, IBM, Deloitte, DXC, and Genesys. Today, we’re launching the ElevenLabs Partner Program to make it easier for enterprises to deploy ElevenAgents with the consultancies, systems integrators, software vendors, and agencies that already know their business. Partners handle the integrations into your existing tech stack, or work alongside ElevenLabs FDEs on complex deployments, whatever helps you realize value fastest.

As part of the program, ElevenLabs is also announcing an alliance with McKinsey, pairing ElevenAgents with the firm’s enterprise transformation expertise, and QuantumBlack, AI by McKinsey. Businesses can now build on ElevenLabs, build with a partner, or buy from one to scale agents across complex systems.

Additionally, ElevenLabs and NVIDIA announced they are expanding their partnership. Nemotron 3.5 Super VL is now available in ElevenAgents, and ElevenLabs is also adding 15 MW of new Vera Rubin compute capacity. The two companies will also work on an on-premises voice agent for Poland’s national e-Health platform. The agent will work on appointment scheduling for patients who only have landlines. 

Today, we’re introducing the ElevenLabs Marketplace, a curated group of industry-specific AI and voice applications built on ElevenLabs by our technology partners, for industries including healthcare, financial services, insurance, education, and retail. Enterprises can browse agents and applications and connect directly with the partners behind them. Starting next year, businesses will be able to buy these agents through the Marketplace using their existing ElevenLabs commitment. 

What happens when interaction is solved

Staniszewski closed the keynote by asking what happens when interaction between people and AI is solved. His answer was a world where technology adapts to people, the limits of language, geography, and scale fall away, and local nuance and culture are preserved.

“People are understandably concerned that AI could centralize and homogenize everything. That businesses won’t be able to protect their moats. That each new iteration of AI regresses to the mean,” Staniszewski said. “We’re building the opposite. Models, products, and a community to amplify what makes your business unique, and help you extend your advantage.” 

Today’s announcements are steps toward that future, and ElevenLabs is building it with the customers, partners, and developers who joined in New York. To get started, begin with ElevenAgents.