Matt Cook joins Kenny Jahng on the Church Tech Today Podcast

We’re excited to share that our President, Matt Cook, was recently featured on the Church Tech Today Podcast hosted by Kenny Jahng — one of the leading voices in digital ministry and AI innovation for churches. The conversation dove into how Aberdeen’s real-time AI captioning and translation tools are making worship services more accessible, inclusive, and effective for everyone in the pews.

Watch the full episode

Read the article on ChurchTechToday.com: Church Tech Today Podcast Feature with Matt Cook

Why This Conversation Matters Now

The conversation wasn’t just about cool tech — it was about mission. AI is opening the door for churches to reach people who’ve been sitting quietly in their pews, unable to fully engage — whether due to hearing loss, language barriers, or generational differences in media consumption.

As Kenny noted, "Most churches are barely scratching the surface of what's possible." But with the right tools, accessibility can be seamless, affordable, and incredibly impactful.

Key Takeaways from the Interview

1. AI Makes Accessibility Affordable for Every Church

Matt shared how Aberdeen has taken technology once reserved for large, media-savvy churches — and made it accessible for congregations of all sizes. Even 100-member churches are now using AI captioning and translation to serve ESL communities and individuals with hearing loss.

“We’ve had churches start this just to help one deaf member, and suddenly realize there are more people quietly struggling to engage — and they begin reaching new audiences they never expected.”
— Matt Cook, Aberdeen

2. Captions Aren’t Just for the Hearing Impaired

More than 70% of Gen Z and over 50% of Millennials now watch content with captions on. It’s not an accessibility issue — it’s an engagement issue.

Captions help with focus, retention, and accessibility for everyone — not just those with hearing impairments.

“Captions help us absorb more. They’ve become part of how the next generation engages with content, including sermons.”
— Kenny Jahng

3. Live, In-Person Captions Are Easier Than Ever

Churches don’t need expensive gear or complex workflows. Aberdeen’s AI captioning and translation system works right from the audio feed churches already have. Viewers can scan a QR code and get real-time captions or listen to a voice dub in their preferred language — right on their phone.

“Setup takes just minutes. Most churches already have everything they need.”
— Matt Cook

4. Voice Dubbing in 60+ Languages

Beyond just captions, Aberdeen offers AI voice dubbing — allowing attendees to hear the sermon in real time in their own language, with natural inflection and clarity. This opens the door to multi-generational families, immigrant communities, and global reach without needing separate services.

A Tool for Growth, Not Just Accommodation

Kenny brought up an important shift in thinking: Captioning and translation aren't just for “accommodation” — they’re tools for growth and connection.

Whether it’s helping Gen Z absorb more of the message, allowing a Spanish-speaking grandmother to understand her grandson’s baptism, or making someone with hearing loss feel seen and included — accessibility is kingdom work.

Try It with Your Own Content

As Matt shared on the podcast, Aberdeen offers churches a free trial month and will even demonstrate the technology using the church’s own sermon video.

• Real-time captions
• Mobile access via QR code
• Translation and voice dubs in 60+ languages
• Seamless setup using your existing equipment

Ready to See It in Action?

Curious how it would work for your services? We’re now offering personalized live demos using your actual content so you can see how real-time captions and multilingual translations would look during your own worship services.

• Get guidance tailored to your tech setup
• Experience the interface with familiar voices and formats
• Evaluate translation quality in real-time

Schedule your one-on-one demo now using our calendar link below.

In a recent StreamingChurch.tv webinar, Aberdeen Broadcast Services President Matt Cook joined Phil Thompson to discuss the powerful role artificial intelligence is playing in making church services more accessible—both in-person and online.

As churches strive to reach broader and more diverse communities, the need for multilingual captioning and translation has never been greater. During the conversation, Matt shared how Aberdeen has spent years refining AI tools behind the scenes before confidently bringing them forward for real-time use.

“We’ve been using AI in post-production captioning for years—combining it with human editors to ensure top-tier quality,” Matt explained. “But we didn’t want to offer it live until we were confident it would meet the high standards our clients expect.”

A New Era of Real-Time Accessibility

What once required dedicated human captioners or translators can now be achieved instantly through AI—with support for over 60 languages in both subtitles and audio dubbing.

Whether you’re broadcasting to viewers across the globe or welcoming a multilingual congregation in your sanctuary, Aberdeen’s real-time solution is built to scale with you. During the live demo, attendees saw firsthand how captions and translations could be instantly toggled across multiple devices—no app, log-in, or special equipment required.

Designed for Churches—Not Just Tech Experts

One of the biggest takeaways from the webinar was that you don’t have to be a tech expert to bring this level of accessibility to your church. Aberdeen’s cloud-based tools work with any encoder and can be used in both live streams and in-person worship services.

“It’s as simple as scanning a QR code or clicking a link. Your viewers can then choose their preferred language and follow along in real-time,” said Matt.

Watch the Full Webinar Replay

If you missed the live event or want to revisit what was shared, you can now watch the full replay on demand here:

You’ll see:

• A live demo of the tech in action
• How churches are using it for both local and international outreach
• A walkthrough of Aberdeen’s 60+ language support
• The inspiration behind Aberdeen’s long-term commitment to accessibility

See It in Action—With Your Own Content

Curious how it would work for your services? We’re now offering personalized live demos using your actual content so you can see how real-time captions and multilingual translations would look during your own worship services.

• Experience the interface with familiar voices and formats
• Evaluate translation quality in real-time
• Get guidance tailored to your tech setup

Spots are limited and filling up quickly. Schedule your one-on-one demo now using our calendar link below.

The Bottom Line

Churches today have an unprecedented opportunity to reach global audiences, welcome newcomers, and serve members of all abilities and backgrounds. With trusted AI-powered tools from Aberdeen, you can break language barriers and make sure no one misses the message.

Learn more about ASR-powered accessibility tools for churches here.

Gather25, hosted by IF:Gathering, was a groundbreaking 25-hour global broadcast connecting audiences across every continent. With over 1.25 million online viewers and participation from more than 21,000 Gather Groups worldwide, the event aimed to inspire and unite the global church community.

Aberdeen Broadcast Services has partnered with IF:Gathering since 2020, providing both live captioning for streaming events and post-produced captioning for archived content. Gather25 marked an ambitious new chapter in that partnership—the first time Aberdeen was brought on to help scale the event’s global reach through real-time translation.

As far as we know, this was a first-of-its-kind undertaking: a multilingual livestream of this magnitude, requiring impeccable coordination across dozens of languages and platforms. Aberdeen was one of several trusted vendors, working alongside the technology and broadcast partners hired for the event to make it all possible.

The Challenge: Delivering 25 Hours of Multilingual, Real-Time Access

Supporting a continuous, multilingual broadcast of this scale introduced several technical challenges:

• Maintain uninterrupted captioning across a 25-hour live stream spanning 7 global broadcast hubs.
• Generate real-time captions and translations for 84 language streams with high accuracy and minimal latency.
• Implement a dynamic captioning system that allows viewers to pause, rewind, and review up to four hours of content during the live event ("rolling DVR" functionality).
• Collaborate across multiple teams — Element Media Group (master control), Sardius (video player delivery), SyncWords (captioning/subtitle platform), and Interprefy (live interpreters) — to ensure flawless live execution.
• Deliver caption files in a continuous VTT format, dynamically updating during the broadcast to support both live viewing and post-event on-demand access.

The Solution: Continuous Live Captioning and Translation Across 84 Languages

Before the event went live, a workflow that could support uninterrupted, real-time captioning and translation across dozens of simultaneous streams needed to be built. This required deep coordination with multiple teams to align audio sources, language feeds, and delivery endpoints. Our goal was to ensure every segment of the broadcast could be accurately captioned and translated with minimal manual intervention once the event began.

A key technology partner behind this workflow was SyncWords, whose platform Aberdeen leverages to manage real-time captioning and translation delivery. SyncWords played a vital behind-the-scenes role in not only powering the infrastructure we used to deploy captions across dozens of streams but also collaborating directly with engineers at Sardius and Elemental Media to implement specialized audio-isolation coding for the event. This coordination ensured that every audio feed we received was optimized for clean, accurate transcription and translation at scale.

Pre-Event Testing

To guarantee performance, Aberdeen conducted over 50 hours of pre-event testing, stress-testing multi-language streams, and simulating 25-hour sessions to ensure system endurance. A key priority was ensuring that VTT caption files would function as continuously updated feeds, rather than static uploads. This real-time updating was essential for Sardius’ platform to support both live captions and rolling DVR features, while also allowing seamless access to captions during on-demand playback after the event.

Streaming Caption Workflow

In a typical video workflow, captions are created after recording, uploaded separately, and synced to on-demand content. For Gather25’s livestream, captions had to be generated and delivered in real time, alongside the video stream.

The system worked like this:

• Live video was broken into segments (HLS .ts files) and stored in an S3 bucket, alongside a .m3u8 manifest file managing playback.
• Simultaneously, a VTT subtitle file was created and appended in real time—each new caption line was added to the file without restarting it.
• This continuous file structure allowed Sardius’ player to provide viewers with immediate access to live captions while also enabling DVR controls—viewers could rewind up to four hours and then return to the live stream without losing their place.
• Once the event ended, these same VTT files remained attached to the recorded content for full accessibility during on-demand viewing.

This architecture required precise timing, structured file delivery, and full alignment with Sardius’ streaming infrastructure.

Workflow Optimization

Aberdeen’s team engineered a sophisticated workflow to process 20 incoming HLS feeds. Working closely with Element Media Group, which managed master control and delivered stripped audio to Sardius, Aberdeen received feeds prepped for accessibility:

• Speech-Only Audio Extraction: Each of the 20 HLS streams included up to 16 audio tracks; Aberdeen pulled content from Track 16, where music had been removed to isolate spoken word, ensuring clean inputs for accurate captioning and translation.
• Language Processing:
  • Live English captioning was performed by Aberdeen’s professional human captioners via a broadcast encoder at Master Control, ensuring exceptional accuracy of the English captions for the television broadcast.
  • Translated English to Kinyarwanda, a language not supported by any of the major ASR engines, demonstrating a custom solution beyond standard automation.
  • Created native-language captions for 14 non-English spoken feeds, including Arabic, Mandarin, French, German, Portuguese, and more.
  • Translated stripped English audio into 68 other languages, selecting the best available machine translation engine per language (prioritizing: DeepL > OpenAI > Amazon > Google > Azure).
• File Management: All caption files were delivered to Sardius using clearly labeled S3 bucket destinations (e.g., EN for English, ES for Spanish), with structured naming conventions to ensure accurate mapping during the live event and in the post-event archive.

Impact: Real-Time Engagement and Global Reach Through Accessibility

Gather25’s accessibility efforts produced measurable results:

• Over 1 million minutes of English captions and translations were viewed during the live stream, equal to nearly 17,000 hours of real-time engagement.
• Continuous, real-time VTT streaming ensured seamless live accessibility, enabled four-hour rolling DVR, and extended caption access post-event.
• Aberdeen's workflow operated without any critical interruptions across the entire 25-hour window, validating the strength of its testing protocols and system reliability.

By combining AI-driven automation, human captioning expertise, and a deep integration with broadcast systems, Aberdeen Broadcast Services delivered scalable, high-quality accessibility at a truly global level.

Conclusion: Advancing Live Event Accessibility on a Global Scale

Gather25’s mission to unite believers around the world was made stronger through its commitment to accessibility. With real-time captioning and translation across 84 language streams, Aberdeen Broadcast Services helped make this global event inclusive, impactful, and available to all.

For organizations planning large-scale, multilingual broadcasts, Aberdeen’s tested and proven solutions offer the reliability and scalability needed to reach a worldwide audience.

Let’s talk about how we can support your next event. Contact us to learn more.

The recent pivot to embrace the perks of AI for captioning allowed us at Aberdeen to set our sights on integrating it into parts of our Spanish workflow via our Aberscript transcript editor application, something which we began to use for English programs over 5 years ago and have now fully implemented for Spanish content. This was developed, in part, as an effort to keep transcription in-house and reduce the typical time it would take to transcribe a program from scratch before preparing it for the finishing touches in MacCaption. 

Our Spanish captioning clients offer content that covers a variety of topics such as news, sports, entertainment, medicine, automotive sales, and Christian programming. The initial challenge was choosing the right field to begin testing our AI model. Initial test programs often included multiple speakers throughout the program who frequently code-switch with English or Spanglish phrases, adding complexity to the transcription process. Due to frequent language shifts, testing AI was not optimal for this variety. 

Our best bet was looking at the content that makes up the majority of our day-to-day work: Christian content, where it’s much easier to find a single speaker addressing his or her audience and where the formality of language is likely to remain consistent and free of informal slang, contractions, and colloquialisms. Given our success with English captioning, we decided to leverage AI’s many advantages—automating repetitive functions, enabling real-time generation and quicker turnarounds, maximizing speed and efficiency, and offering vastly improved and adaptable machine learning and speech recognition—perks that tend to outperform traditional methods.

By far, one of the biggest perks has been AI's ability to generate words instantly and accurately, helping to reduce some of the tediousness involved in typing out words, phrases, sentences, and so on. AI has now achieved optimal accuracy with the correct transcription of accent marks in Spanish vowels (a, e, i, o, u) as well as the ñ. This results in a more free-flowing and undisturbed editing process. 

For AI to be successful in this regard, it needs to know when to pick up the natural stress in syllables so that it generates the accent mark on top of the correct letter. Here are some of the common Christian terms and names that appear frequently in Spanish and were properly transcribed with the correct accent marks: amén, Jehová, Adán, oración, redención, apóstol, ángel, Espíritu Santo, Jesús, Lázaro, Noé, María, Salomón, Salvación. Knowing this, the next focus was implementing AI for a client whose content aligned with our goal of maximizing efficiency and achieving a quicker turnaround.

The first stage of incorporating AI into our Spanish workflow began in mid-2024, as we introduced the Spanish-dubbed version of one of our biggest clients. And given that we were now making this transition, with recurring episodes airing on many different networks, we were looking for ways to make progress without outsourcing. The workflow associated with this program was once destined for out-of-house transcription and captioning--with only a brief QC and final exports created in-house. In the past, a few business days for two different programs were generally required, given that the vendor needed ample time to transcribe from scratch and to caption before sending us the completed files for a final QC.

At the beginning of our AI pivot for this particular client, every episode required about 90-120 minutes to transcribe from scratch, and another 45 minutes to complete the post-transcription QC and captioning in MacCaption. However, after months of fine-tuning and consistent use of Aberscript, now it only takes us around 90 minutes in total to complete each — 60 minutes for transcription and 30 for post-transcription QC and captioning — while still maintaining the same level of quality expected for this product level.

For our clients, these efficiencies allow us to respond quickly and accommodate shifting deadlines without compromising quality. AI helps us ensure your programs are captioned with accuracy and delivered on schedule, all without sacrificing the attention to detail your content deserves. It enables quicker turnarounds and supports urgent delivery timelines, something that’s become increasingly important in fast-paced production environments.

We are currently working on running other Spanish programs through our AI workflow to boost efficiency, accuracy, and work time goals. Without a doubt, we will continue to maximize its potential in 2025.

We’ve streamlined our process. Let’s streamline yours.

With AI-enhanced tools and a refined in-house workflow, we’re helping clients get accurate, broadcast-ready Spanish captions faster than ever, without sacrificing quality. Whether you’re managing weekly programs or shifting timelines, we’re here to make your process faster and more efficient.

Contact us to see how we can help.

In today's world defined by constant digital engagement, younger generations increasingly rely on captions and subtitles to enhance their viewing experience. This trend, largely popularized by Gen Z and Millennials, isn’t just limited to streaming shows or watching social media content; it’s spilling over into live events, with a strong case for captions as a way to boost engagement and attendance in venues that historically may have overlooked them, such as churches.

Here’s a look at the data supporting this movement and how churches can use captioning to foster a more engaging environment.

Younger Generations and the Subtitle Revolution

Preply, a language learning platform, conducted a survey titled, “Why America is Obsessed with Subtitles,” to explore the growing trend of subtitle usage among Americans. The study involved over 1,200 participants, aiming to understand how and why individuals use subtitles in their media consumption. The findings revealed that 50% of Americans watch content with subtitles most of the time, with younger generations, particularly Gen Z, showing a greater preference for subtitle use.

• According to Preply's survey, 70% of Gen Z and 53% of Millennial viewers turn on subtitles most of the time when they watch content. This statistic highlights a major shift in media consumption patterns: it’s not just about watching but understanding and engaging with content that drives this trend.
• 74% of Gen Z also prefer subtitles when watching content in foreign languages. With the global reach of platforms like Netflix, young people are more accustomed than ever to encountering diverse media, where subtitles bridge language and comprehension gaps.
• Another 57% of younger viewers report watching content in public places, using captions to stay engaged while maintaining privacy or avoiding sound disruptions. This trend underscores their reliance on captions to stay connected to content in any setting.

This data reveals a generation that sees captions not as an add-on but as an essential part of the viewing experience. For churches, this could signal an opportunity: integrating captions into services may not only help with accessibility but also align with the viewing habits of younger generations.

Captioning Live Events: A Path to Higher Engagement

This enthusiasm for live captions reflects a shift in how younger generations want to consume live content. Captions at events enhance accessibility for everyone, regardless of hearing ability, and address concerns with acoustics or unclear speech, which often deter audiences. In the church context, offering captions during sermons, worship songs, or events could break down barriers that may otherwise prevent younger individuals from fully engaging.

Engaging a New Generation: How Captioning Can Help Churches Reconnect with Young Adults

Christian churches across the United States are increasingly challenged to capture the interest and attendance of younger generations, who are showing declining levels of religious affiliation and engagement." The Pew Research Center's 2019 article, "In U.S., Decline of Christianity Continues at Rapid Pace," highlights a significant decline in Christian affiliation among younger Americans. The data indicates that only 49% of Millennials (born between 1981 and 1996) identify as Christians, compared to 84% of the Silent Generation (born between 1928 and 1945).

With reports indicating a decline in church attendance in the U.S., many churches are seeking strategies to re-engage their congregations, especially young adults. Captions could be a powerful, practical solution.

Offering live captions during services could address several issues:

• Accessibility and Inclusivity: Young people who are accustomed to captions often view them as an essential tool for inclusion. By offering captions in church, congregations can signal a commitment to inclusivity, welcoming not only those with hearing impairments but also those who may benefit from captioning for other reasons, like understanding speakers with different accents or enhancing focus.
• Boosting Engagement: With 55% of Americans saying it’s harder to hear dialogue in shows than it used to be, similar issues with acoustics or clarity in large church spaces might deter engagement. Captions can solve this problem, making services clearer and more engaging for all attendees.
• Attracting a Tech-Savvy Generation: For a generation that turns to their phones for entertainment, information, and engagement, adding captions is a way of bridging familiar digital practices into the church environment. Just as captions boost engagement on platforms like Instagram and TikTok, they could play a similar role in church services.
• Providing a Multilingual Option: In a diverse country, language can be a barrier for many congregants. With nearly 1 in 5 Preply survey respondents using subtitles to learn a new language, churches could consider offering caption translations in key languages relevant to their community, creating a multilingual service environment.

Moving Forward: A Call for Churches to Embrace Captioning

By understanding the viewing habits of younger generations, churches have the opportunity to create an environment that aligns with their engagement preferences. Embracing captioning technology for in-person services and online sermon streams not only makes services more accessible but can also foster deeper engagement, particularly among younger congregants who see captions as an essential part of their everyday experience.

As churches consider how best to adapt to changing times, incorporating live captions could be a powerful step toward renewing attendance and helping younger generations feel seen, heard, and included in the community. It’s a practical, meaningful solution that could not only enhance accessibility but help bridge generational gaps, allowing churches to resonate with the next generation and grow their community in an inclusive and modern way.

Automatic Speech Recognition (ASR) is now making a substantial impact on how local churches connect with their communities, breaking down barriers once caused by financial limitations. With ASR, churches can now offer inclusive services to the Deaf and Hard of Hearing (D/HH) community. But it doesn’t stop there—when combined with Automatic Machine Translation (AMT), this powerful duo overcomes language hurdles, translating sermons into dozens of languages. Even the smallest congregations can now reach a global audience, sharing their message far and wide.

We previously explored the ethical and theological concerns with AI in the Church in our last blog post: The Role of AI in Enhancing Church Accessibility and Inclusion.

While human-generated captions and translations always offer the highest quality, ASR and AMT provide a cost-effective solution that can be utilized by churches and ministries of any size or budget. Imagine your sermon reaching the Deaf and Hard of Hearing (D/HH) community, allowing for full participation, or sharing your message in various languages to a worldwide audience that might otherwise have been unreachable. AI-powered closed captioning and translations help make this a reality. ASR captions and translations are not only a technological advancement; they are tools for inclusivity and global outreach.

Churches aiming to make a significant impact can turn to AI-powered accessibility tools, once considered out of reach, for preaching and teaching. Practical uses of ASR include:

• Live Streaming: AI-powered ASR can integrate with live streaming services to provide real-time captions, subtitles, or even real-time voice dubbing, making worship services accessible to a global audience. This allows viewers from different parts of the world to follow along with the sermon, regardless of their language or hearing ability, thereby expanding the church’s reach and inclusivity.
• In-Person Services: ASR technology can make live sermons more accessible by providing real-time text or audio dubs on congregants’ mobile devices. By making the sermon text available on personal devices, churches can enhance comprehension and engagement for in-person attendees.
• Notes/Summaries: After a sermon is transcribed, extend the message's reach with our AI-enhanced sermon extras. Whether you want to generate Bible studies or daily devotionals, curate key terms and highlight Bible verses, or create summaries and discussion questions, our enhanced transcript materials are designed to add value and extend the impact of your content. Create endless content from a single sermon.

Aberdeen’s new ASR solution, developed with ministry in mind, employs robust AI engines and large language models to provide a powerful advantage in delivering Christian content. Each solution is carefully crafted to fit your specific ministry needs, providing high-quality captions at a fraction of the cost.

Discover how Aberdeen’s ASR solution offers a cost-effective approach to closed captioning & translation. Learn more here: Aberdeen ASR Closed Captioning.

Closed captioning serves as a powerful tool that extends its impact far beyond aiding the deaf and hard-of-hearing community. Its significance transcends age, abilities, and background, making it an invaluable resource for both educators and learners. In the digital age, closed captioning has emerged as a transformative resource, with research revealing that students, English language learners, and children with learning disabilities who watch programs with closed captioning turned on improve their reading skills, increase their vocabulary, and enhance their focus and attention.

The scholarly article, Closed Captioning Matters: Examining the Value of Closed Captions for All Students (Smith 231) states that “Previous research shows that closed captioning can benefit many kinds of learners. In addition to students with hearing impairments, captions stand to benefit visual learners, non-native English learners, and students who happen to be in loud or otherwise distracting environments. In remedial reading classes, closed captioning improved students’ vocabulary, reading comprehension, word analysis skills, and motivation to learn (Goldman & Goldman, 1988). The performance of foreign language learners increased when captioning was provided (Winke, Gass, & Sydorenko, 2010). Following exams, these learners indicated that captions lead to increased attention, improved language processing, the reinforcement of previous knowledge, and deeper understanding of the language. For low-performing students in science classrooms, technology-enhanced videos with closed captioning contributed to post-treatment scores that were similar to higher-performing students (Marino, Coyne, & Dunn, 2010). The current findings support previous research and highlight the suitability of closed-captioned content for students with and without disabilities.”

Reading Rockets, a national public media literacy initiative provides resources and information on how young children learn and how educators can improve their students’ reading abilities. In the article, Captioning to Support Literacy, Alise Brann confirms that “Captions can provide struggling readers with additional print exposure, improving foundational reading skills.”

She states, “In a typical classroom, a teacher may find many students who are struggling readers, whether they are beginning readers, students with language-based learning disabilities, or English Language Learners (ELLs). One motivating, engaging, and inexpensive way to help build the foundational reading skills of students is through the use of closed-captioned and subtitled television shows and movies. These can help boost foundational reading skills, such as phonics, word recognition, and fluency, for a number of students.”

Research clearly demonstrates that “people learn better and comprehend more when words and pictures are presented together. The combination of aural and visual input gives viewers the opportunity to comprehend information through different channels and make connections between them” (The Effects of Captions on EFL Learners’ Comprehension of English-Language Television Programs).

From bolstering reading skills, to enhancing focus and language comprehension, the benefits of closed captioning are numerous. We at Aberdeen Broadcast Services are committed to providing quality closed captions for television (TV) and educational programming.

Here is the public service announcement (PSA) we released in 2016 on local broadcast stations, emphasizing how closed captioning can enhance children's literacy skills.

This article was co-written with the help of both ChatGPT and Google Bard as a demonstration of the technology discussed in this article. You can also read along with Aberdeen's President, Matt Cook in the recording below - but not really, this is Matt's voice cloned using a short clip of Matt's voice given to AI.

Artificial Intelligence (AI) has revolutionized numerous industries, and its influence on language-related technologies is particularly remarkable. In this blog post, we will explore how AI is transforming closed captioning, language translation, and even the creation of cloned voices. These advancements not only enhance accessibility and inclusion but also have far-reaching implications for communication in an increasingly globalized world.

AI in Closed Captioning

Closed captioning is an essential feature for individuals who are deaf or hard of hearing, enabling them to access audiovisual content. Traditional closed captioning methods rely on human transcriptionists, however, AI-powered speech recognition algorithms have made significant strides in this field.

Using deep learning techniques, AI models can more accurately transcribe spoken words into text, providing real-time closed captioning. This is not up to the FCC guidelines for broadcast but is oftentimes good enough for other situations where the alternative is to have no closed captions at all. These models continuously improve their accuracy by analyzing large amounts of data and learning from diverse sources. As a result, AI has made closed captioning more accessible, enabling individuals to enjoy online videos with greater ease.

Our team is working hard to develop and launch AberScribe, our new AI transcript application powered by OpenAI, sometime in mid-2024. From any audio/video source file, the AberScribe app will create an AI-generated transcript that can be edited in our online transcript editor and exported into various caption formats. AberScribe will also have added features for creating other AI-generated resources from that final transcript. Resources like summaries, glossaries of terms, discussion questions, interactive worksheets, and many more - the possibilities are endless.

Sign up to join the waitlist and be one of our first users: https://aberdeen.io/aberscribe-wait-list/

AI-Driven Language Translation

Language barriers have long hindered effective communication between people from different linguistic backgrounds. However, AI-powered language translation has emerged as a game-changer, enabling real-time multilingual conversations and seamless understanding across different languages.

Machine Translation (MT) models, powered by AI, have made significant strides in accurately translating text from one language to another. By training on vast amounts of multilingual data, these models can understand and generate human-like translations, accounting for context and idiomatic expressions. This has empowered businesses, travelers, and individuals to engage in cross-cultural communication effortlessly.

In addition to written translation, AI is making headway in spoken language translation as well. With technologies like neural machine translation (NMT), AI systems can listen to spoken language, translate it in real-time, and produce synthesized speech in the desired language. This breakthrough holds immense potential for international conferences, tourism, and fostering cultural exchange.

Cloned Voices and AI

The advent of AI has brought about significant advancements in speech synthesis, allowing for the creation of cloned voices that mimic the speech patterns and vocal identity of individuals. While cloned voices have sparked debates regarding ethical use, they also present exciting possibilities for personalization and accessibility.

AI-powered text-to-speech (TTS) models can analyze recorded speech data from an individual, capturing their vocal characteristics, intonations, and nuances. This data is then used to generate synthetic speech that sounds remarkably like the original speaker. This technology can be immensely beneficial for individuals with speech impairments, providing them with a voice that better aligns with their identity.

Moreover, cloned voices have applications in industries like entertainment and marketing, where celebrity voices can be replicated for endorsements or immersive experiences. However, it is crucial to navigate the ethical considerations surrounding consent and proper usage to ensure that this technology is used responsibly.

Conclusion

Artificial Intelligence continues to redefine the boundaries of accessibility, communication, and personalization in various domains. In the realms of closed captioning, language translation, and cloned voices, AI has made significant strides, bridging gaps, and enhancing user experiences. As these technologies continue to evolve, it is vital to strike a balance between innovation and ethical considerations, ensuring that AI is harnessed responsibly to benefit individuals and society as a whole.

Open captions and closed captions are both used to provide text-based representations of spoken dialogue or audio content in videos, but they differ in their visibility and accessibility options.

Here's the difference between closed and open captions:

Open Captions

• Open captions, also known as "burned-in" or "hardcoded" captions, are permanently embedded in the video itself.
• They are visible to all viewers and cannot be turned off or disabled.
• Open captions are typically added during the video production process and become a permanent part of the visual content.
• They are useful when the video is intended for a wide audience, including individuals who are deaf or hard of hearing, or when the video will be played in noisy environments where audio may not be easily heard (e.g. a museum).

Closed Captions

• Closed captions are separate text tracks that can be turned on or off by the viewer.
• They are not visible by default but can be enabled through the video player's settings or options.
• Closed captions provide flexibility for viewers who may not need or want the captions.
• They are commonly used in broadcast television, streaming platforms, and online videos to cater to diverse audiences and comply with accessibility regulations.
• Closed captions can be generated in real-time using a live writer or automatic speech recognition (ASR) technology or added manually during post-production.

Feature	Open Caption	Closed Captions
Visibility	Permanently embedded in the video	Separate text track that can be turned on or off
Accessibility	Cannot be turned off	Can be turned on or off by the viewer
Applications	Wide audiences, noisy environments	Diverse audiences, compliance with accessibility regulations
Creation	Added during video production	Generated in real-time or embedded manually during post-production or uploaded as a sidecar file

Both open and closed captions serve the purpose of making videos accessible to individuals who are deaf or hard of hearing, those who are learning a new language, or those who prefer to read the text alongside the audio.

The choice between open or closed captions depends on the specific requirements and preferences of the content creators and the target audience.

In the July ‘21 release of Premiere Pro, Adobe introduced its artificial intelligence (AI) powered speech-to-text engine to help creators make their content more accessible to their audiences. Their extensive toolset allows their users to edit, stylize, and export captions in all supported formats straight out of the sequence timeline of a Premiere Pro project. A 3-step process of auto-transcribing, generating, and stylizing captions all within the platform already familiar to its users delivers a seamless experience from beginning to end. But how accurate is the final product?

Today, at their best, AI captions have an error rate of 5-10% - much improved over the 80% accuracy we saw just a few years ago. High accuracy is crucial for the deaf and hard-of-hearing audience as each error adds to the possibility of confusing the message. To protect all audiences that rely on captioning to understand television programming, the Federal Communications Commission (FCC) set a detailed list of quality standards that all captions must meet to be acceptable for broadcast back in 2015. Preceding those standards, the Described and Captioned Media Program (DCMP) published its Captioning Key manual over 20 years ago and has since been a valuable reference for captioning of both entertainment and educational media targeted to audiences of all age groups. Simply having captions present on your content isn’t enough, it needs to be accurate and best replicate the experience for all audiences.

Adobe’s speech-to-text engine has been one of the most impressive that our team has seen to date, so we decided to take a deeper look at it and run some tests. We tasked our most experienced Caption Editor with using Adobe’s auto-generated transcript to create & edit the captions to meet the quality standards of the FCC and the deaf and hard of hearing community on two types of video clips: a single-speaker program and one with multiple speakers. Our editor used our Pop-on Plus+ caption product for these examples, which are our middle-tier quality captions that fulfill all quality standard requirements but are not always 100% free of errors.

Did using Adobe’s speech-to-text save time, or did it create more work in the editing process than needed? Here’s how it went…

Single Speaker Clip

In this example, we used the perfect scenario for AI: clear audio, a single speaker at an optimal words-per-minute (WPM) speaking rate, and no sound effects or music.

The captions contained the following issues that would need to be corrected by the Caption Editor:

• No speaker ID.
• Strange leading spaces before the second caption line - Caption Editor can delete easily in their software.
• A total of 10 cells out of 95 were formatted the way our Caption Editor did it and would have required no further attention at all.
• Several first words were not capitalized.
• Some sentences had end-sentence punctuation in the middle of a sentence.
• Many words were wrong.
• Most cells formatted poorly, resulting in cells with 2-3 words, or cells that were difficult to read.
  • Many of these cells can be optimized and lines combined for ease of reading.
• Several cells with a duration of less than 1 second.

Here’s the clip with Adobe’s speech-to-text captions overlayed on the top half of the video, and ours on the bottom half.

Multiple Speaker Clip

For the next clip, we went with a more realistic example of television programming where there are multiple speakers, an area where AI is known to struggle and has difficulties identifying the speakers. This clip also features someone with a pronounced accent, commentators speaking over one another, and proper names of athletes – all of which our editors take the time to research and understand.

The same errors detailed in the single-speaker example are present throughout, among the other difficulties we expected it to have. In fact, there were so many errors that our editor was unable to use the transcript from Adobe and started from the beginning using our own workflow.

Here’s a sample of the first 9 cells of captions with what Adobe transcribes in the first column, notes from our Caption Editor, and how it should look.

Adobe’s Automated SRT Caption File	Issue	Formatted by Aberdeen
something  you are never seen in your life, correct?	No speaker ID.	(Pedro Martinez) It's something you have never seen in your life,
	“Correct” is spoken by new speaker.	(Matt Vasgersian) Correct!
So it's.	Missing text.	So it's--so it's MVP of the year!
So we're all watching something  different. OK		(Pedro) We're all watching something different.
He gets the MVP.		Okay, he gets the MVP.
I'd be better off.	Completely misunderstood music lyrics.	♪ Happy birthday to you ♪
Oh, you, you guys.		(Matt) You guys.
Let me up here to dove into the opening night against the Hall of Fame.	Merged multiple sentences together.	Just left me up here to die.
		You left me up here to die against the hall of famer.

Take a look at the clip. Again, with Adobe's speech-to-text on the top and Aberdeen on the bottom.

The Verdict

Overall, the quality of the auto-generated captions exceeded expectations, and we found them to be in the top tier of speech-recognition engines available. The timing and punctuation were particularly impressive. However, when doing a true comparison to the captioning work that we would consider acceptable, AI does not meet Aberdeen’s broadcast quality standard.

Aberdeen's post-production Caption Editors are detail-oriented and grammar-savvy and always strive to portray every element of the program with 100% accuracy so that the viewer misses nothing. For our most experienced Caption Editor, it took a 5:1 ratio in time for them to edit and correct the single-speaker clip; meaning, for every minute of video, it took 5 minutes to clean up the transcript and captions. Assuming your team is educated in the proper timing of caption cells, line breaks, and grammar, a 30-minute program may take over 2.5 hours to bring up to standards with a usable transcript. In the second example, the transcript was unusable and would have taken more time to clean up than it did to transcribe from scratch. Double that timeline now.

Consider all of the above when using this service. Do you have the time and resources to train your staff to know how to edit auto-generated captions and get them up to the appropriate standards? How challenging may your content be for the AI? Whenever and however you make the choice, make sure you deliver the best possible experience to your entire audience.