Voice Prompts are provided to enable speaker control.
TTS-Llama is a TTS-only system. MoLE-Llama is a multimodal LLM capable of both TTS and text QA tasks.
Large language models (LLMs) have revolutionized natural language processing (NLP) with impressive performance across various text-based tasks. However, the extension of text-dominant LLMs to with speech generation tasks remains under-explored. In this work, we introduce a text-to-speech (TTS) system powered by a fine-tuned Llama model, named TTS-Llama, that achieves state-of-the-art speech synthesis performance. Building on TTS-Llama, we further propose MoLE-Llama, a text-and-speech multimodal LLM developed through purely late-fusion parameter-efficient fine tuning (PEFT). Extensive empirical results demonstrate MoLE-Llama’s competitive performance on both text-only question-answering (QA) and TTS tasks, without catastrophic forgetting in either modality. Finally, we further explore MoLE-Llama in text-in-speech-out QA tasks, demonstrating its potential as a multimodal dialog system capable of speech generation.
Voice Prompts are provided to enable speaker control.
TTS-Llama is a TTS-only system. MoLE-Llama is a multimodal LLM capable of both TTS and text QA tasks.
| Traget Text Transcript | Voice Prompt | YourTTS | VoiceCraft | XTTS v2 | TTS-Llama (Ours) | MoLE-Llama (Ours) |
|---|---|---|---|---|---|---|
| well, his want of valour was but a deeper appeal to her tenderness. to thrill with response to it she had only to remember all the ladies she herself had, as they called it, funked. | ||||||
| wow, that's really cool. you never know what kind of experiences you'll have when you travel. | ||||||
| what a symphony of sound fills the air as birds sing their melodious tunes at the break of dawn! | ||||||
| Comprehending the influence of emotions and feelings is vital to mastering the art of magic. | ||||||
| i'm in disbelief that we achieve that! it's a significant accomplishment. how did our team unite to make it possible? | ||||||
| tickets to moma, The Museum of Modern Art, are eighteen dollars. | ||||||
| please note that all comments including hyperlinks automatically get dumped into the moderation or spam folders. | ||||||
| the American Civil War was fought in the United States from eighteen sixty one, to eighteen sixty five. the result of a long standing controversy over slavery, war broke out in April eighteen sixty one, when Confederates attacked Fort Sumter in South Carolina, shortly after Abraham Lincoln was inaugurated. | ||||||
| Wikipedia is a free online encyclopedia, with the aim to allow anyone to edit articles. Wikipedia is the largest and most popular general reference work on the Internet, and is ranked the fifth most popular website. | ||||||
| deep water *is* deceptively chilly, despite the big lake's sun exposure, so it's better for photographing than swimming. | ||||||
| the tested criteria are among other organizational objectives, curricula, personnel, quality management and facilities. | ||||||
| Instinctively, we marched Sammy up to the tar barrel, and I liberally daubed his hateful and, wonderful to relate, still clean collar with its contents, taking more pains to get it on his collar, than to keep it off his clothes. | ||||||
| i shall go away for a few weeks, for no doubt a journey will set me up again. | ||||||
| Fatal Dose. in an adult two grains, same as arsenic. Fatal Period. death follows in eight to twelve hours, from exhaustion. | ||||||
| yes, Gramps, Bud said meekly. they broke out of the other side of the thicket and came upon the place where Old Yellowfoot had left the hemlocks to slink into a stand of yellow birch. | ||||||
| Oscar tightened his belt and waited for orders, while Armitage and Claiborne conferred in quick pointed sentences. | ||||||
| this all comes from his love of thrill and novelty and is innocent enough. but it leads to misunderstandings and broken homes unless the Thoracic marries the right type of person. | ||||||
| they all stepped to one side at the sound of the Ah yo of our leader, except a band of coolies carrying the monstrous trunk of a pine tree, chanting as they swung the mast between them, and keeping step with the chant. | ||||||
| here there was another grand reception. all the nobility and gentry of the surrounding country came in to honor the king's arrival, and the ceremonies attending the entrance of the royal cottage were extremely magnificent. | ||||||
| every drop of blood had fallen from Jane's face and the rough hands on her knee were locked together. |
Samples in this task contain at least two special characters, such as digits, dates, ZIP code and currency.
TTS-Llama is a TTS-only system. MoLE-Llama is a multimodal LLM capable of both TTS and text QA tasks.
Both TTS-Llama and MoLE-Llama take raw text transcripts as inputs, without relying on any frontend system.
| Traget Text Transcript | YourTTS | VoiceCraft | XTTS v2 | TTS-Llama (Ours) | MoLE-Llama (Ours) |
|---|---|---|---|---|---|
| The New Year's Eve party starts at 8 PM on December 31, 2022, and goes until 1 AM on January 1, 2023. About 100,000 people plan to join in this annual celebration! | |||||
| The Paris 2024 Summer Olympics, officially the Games of the XXXIII Olympiad and branded as Paris 2024, were an international multi-sport event that occurred between July 16 - August 11 2024 in France, with the opening ceremony having taken place on July 26th. | |||||
| The letter was addressed to Mr. John Smith, 123 Main Street, Palo Alto, CA 12345. | |||||
| ABC Corporation is headquartered in 901 S Broadway St., Suite 200, Seattle, WA 98104. | |||||
| Please visit our campus at 456 Memorial Dr., Apt 101, New York, NY 10001. | |||||
| Dr. Maria Rodriguez will host a science fair for K-12 students interested in STEM studies at the Museum of Science, located at 789 W Oak Ave., San Francisco, CA 94102. | |||||
| My driver's license number is DL123456, and my license plate number is CAL1234, which I need to renew next month. | |||||
| The company's profit margin is expected to be around 15%-20% for the quarter, with net income estimated to top $200,000. | |||||
| The company has 1,234 employees, and it was founded in 1999, which was the 25th anniversary of the industry reborn after World War II. | |||||
| Mary finished 1st place in the 5th annual city marathon, with a time of 2 hrs and 45 mins. |
Extend MoLE-Llama beyond TTS for handling both speech QA and text QA tasks.
| Text Question | MoLE-Llama's Speech Response |
|---|---|
| As an AI system, introduce what you can do. | |
| Tell me a joke then. | |
| I am doing my mathematics homework, I need your help to solve some problems. | |
| If a bag contains 5 red balls, 3 blue balls and 2 green balls, what is the probability of drawing a red ball? | |
| I need help with this problem: Trevor buys several bouquets of carnations. The first included 9 carnations; the second included 14 carnations; the third included 13 carnations. What is the average number of carnations in the bouquets? |
| Text Question | MoLE-Llama's Speech Response |
|---|---|
| What part of some furniture could you put some stamps in? | |
| Where do you buy food shaped like Mickey mouse? | |
| Sarah noticed that she was out of cookies. She was going to serve some with dinner, so she needed more. Where might she go? | |
| Where must a television be kept quiet? | |
| What do children love to do with scissors and paper? |
| Text Question | MoLE-Llama's Speech Response |
|---|---|
| What year was George W. Bush elected? | |
| Where English is spoken? | |
| What music period did Beethoven live in? | |
| Where is the best place to vacation in the Dominican republic? | |
| Where to exchange euros in New York city? |