YoutubeAudioLoader and updates to OpenAIWhisperParser #5772
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eyurtsev
reviewed
Jun 6, 2023
| # Split the audio into chunk_duration_ms chunks | ||
| for split_number,i in enumerate(range(0, len(audio), chunk_duration_ms)): | ||
|
|
||
| print(f"Transcribing part {split_number}!") |
Collaborator
There was a problem hiding this comment.
Suggested change
| print(f"Transcribing part {split_number}!") |
|
|
||
| with blob.as_bytes_io() as f: | ||
| transcript = openai.Audio.transcribe("whisper-1", f) | ||
| yield Document( |
Collaborator
There was a problem hiding this comment.
Should we yield a single document if the input is a single audio file and we're trying to hide the fact there's chunking under the hood? We can collect the transcripts and concatenate them. The only problem is that it's unclear on which delimiter to use to join on.
Contributor
Author
There was a problem hiding this comment.
It would be easy to do this. E.g., we can build a single blob from the combined docs:
combined_docs = [doc.page_content for doc in docs].join(strings)
But, as discussed, it's kind of nice to have the intermediate outputs.
(The latency is somewhat high - 15 min for 2 hr video.)
| with yt_dlp.YoutubeDL(ydl_opts) as ydl: | ||
| info = ydl.extract_info(url,download=False) | ||
| title = info.get('title', 'video') | ||
| print(f"Writing file: {title} to {self.save_dir}") |
Collaborator
There was a problem hiding this comment.
Suggested change
| print(f"Writing file: {title} to {self.save_dir}") |
rlancemartin
force-pushed
the
rlm/simple_audio_load_and_split
branch
7 times, most recently
from
01a5729 to
74326d6
Compare
eyurtsev
approved these changes
Jun 6, 2023
Collaborator
There was a problem hiding this comment.
replace with raise ValueError or ImportError
Collaborator
There was a problem hiding this comment.
Needs to be raised as well
rlancemartin
force-pushed
the
rlm/simple_audio_load_and_split
branch
from
74326d6 to
4f0e4ca
Compare
rlancemartin
force-pushed
the
rlm/simple_audio_load_and_split
branch
from
4f0e4ca to
e1fa1a4
Compare
Undertone0809
pushed a commit
to Undertone0809/langchain
that referenced
this pull request
Jun 19, 2023
) This introduces the `YoutubeAudioLoader`, which will load blobs from a YouTube url and write them. Blobs are then parsed by `OpenAIWhisperParser()`, as show in this [PR](langchain-ai#5580), but we extend the parser to split audio such that each chuck meets the 25MB OpenAI size limit. As shown in the notebook, this enables a very simple UX: ``` # Transcribe the video to text loader = GenericLoader(YoutubeAudioLoader([url],save_dir),OpenAIWhisperParser()) docs = loader.load() ``` Tested on full set of Karpathy lecture videos: ``` # Karpathy lecture videos urls = ["https://youtu.be/VMj-3S1tku0" "https://youtu.be/PaCmpygFfXo", "https://youtu.be/TCH_1BHY58I", "https://youtu.be/P6sfmUTpUmc", "https://youtu.be/q8SA3rM6ckI", "https://youtu.be/t3YJ5hKiMQ0", "https://youtu.be/kCc8FmEb1nY"] # Directory to save audio files save_dir = "~/Downloads/YouTube" # Transcribe the videos to text loader = GenericLoader(YoutubeAudioLoader(urls,save_dir),OpenAIWhisperParser()) docs = loader.load() ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces the
YoutubeAudioLoader, which will load blobs from a YouTube url and write them. Blobs are then parsed byOpenAIWhisperParser(), as show in this PR, but we extend the parser to split audio such that each chuck meets the 25MB OpenAI size limit. As shown in the notebook, this enables a very simple UX:Tested on full set of Karpathy lecture videos: