Mandatory Social Distancing is my third foray into the world of multi stream video virtual ensemble projects (besides a short stint of dabbling with that certain multi stream phone app). Before working on MSD, I did two projects for the Blue Stars Drum & Bugle Corps - one of which we shared widely (the corps song) and one of which we buried and will never see the light of day (my first go at these kinds of project, and quite a learning experience).
With the publication of the Blue Stars video and with my on-going social media commentary throughout the process of MSD, I've gotten a number of questions from folks who want to do something similar: "How did you do it?" "What program do you use?" - I decided to put together this blog post to answer those questions in one place and to offer a fair word of warning.
Probably in this space is where I should mention the rippling consequences of the COVID-19 situation, the various shut-downs around the country, and of course, the need for musicians to still find a way to perform in ensembles together. If you're here reading this, I don't need to say anything more about the situation: you already get it. You might be a band director with an administrator asking you to toss together a video of your kids - "How cool would that be??"
In the words of Samuel L. Jackson from Jurassic Park: Hold on to your butts.
In the words of Samuel L. Jackson from Jurassic Park: Hold on to your butts.
The things you'll need:
- A LOT of storage space. Like, a LOT.
- As much RAM as your computer can handle.
- Speaking of your computer, a consumer-grade computer is going to be hard to get to work - it will need to be fairly new and have pretty substantial specs. I'm using a mid-2017 iMac and it's quite a struggle (more later).
- Real, professional-grade video and audio software. Audacity won't work. Windows Movie Maker definitely will not work. iMovie and Garage Band will not work. You'll need Final Cut Pro (I'm also using Apple's Motion for MSD), Logic Pro, Adobe Premiere, Pro Tools, and the like. Yes, it costs real money.
- Also, an external video card (in the lingo, eGPU) is almost immensely helpful.
Step 1: Pick your piece of music
This step is pretty obvious. You might already have something in mind. The shorter, the better. If you're thinking of doing a multi-stream of a Beethoven Symphony, you need to... well, just don't do that.
Once you have your piece of music, you'll need to create a click track that includes the source music. MIDI is the easiest and best way. All those performers in your virtual ensemble need to perform along with something. They absolutely will not be in tune with one another in your final output without starting from a point of unifying your tuning tendencies (reference that aforementioned Blue Stars video that will not see the light of day). This might be the point where somebody would tout their ensemble's wonderful musicianship - however, musicians play in tune because they listen and adjust, not because they're great as an individual.
So, create a MIDI file that includes a very obvious metronome click.
This step is pretty obvious. You might already have something in mind. The shorter, the better. If you're thinking of doing a multi-stream of a Beethoven Symphony, you need to... well, just don't do that.
Once you have your piece of music, you'll need to create a click track that includes the source music. MIDI is the easiest and best way. All those performers in your virtual ensemble need to perform along with something. They absolutely will not be in tune with one another in your final output without starting from a point of unifying your tuning tendencies (reference that aforementioned Blue Stars video that will not see the light of day). This might be the point where somebody would tout their ensemble's wonderful musicianship - however, musicians play in tune because they listen and adjust, not because they're great as an individual.
So, create a MIDI file that includes a very obvious metronome click.
Step 2: Have your performers record
Also obvious.
The tips I've given to my performers for MSD: use earbuds, and only have one. Keep your other ear open to hear your instrument. Play in a very dead room: you don't want the beautiful acoustics of a large, open, reverb-heavy space for this. For bell-front brass: do not aim at the microphone, play about 45° to the side.
Also obvious.
The tips I've given to my performers for MSD: use earbuds, and only have one. Keep your other ear open to hear your instrument. Play in a very dead room: you don't want the beautiful acoustics of a large, open, reverb-heavy space for this. For bell-front brass: do not aim at the microphone, play about 45° to the side.
Step 3: Collect your videos
However you're doing this project (yourself or with your ensemble), you'll need a place to collect your videos.
Pro tip: You don't have enough storage space (no really, you need more).
I've found that my computer and my software all work better by storing the videos on an external, 2TB, USB3.0 solid state hard drive - rather than keeping the video local, on the same drive as the program that's running.
But seriously: go buy more storage.
However you're doing this project (yourself or with your ensemble), you'll need a place to collect your videos.
Pro tip: You don't have enough storage space (no really, you need more).
I've found that my computer and my software all work better by storing the videos on an external, 2TB, USB3.0 solid state hard drive - rather than keeping the video local, on the same drive as the program that's running.
But seriously: go buy more storage.
Step 4: ORGANIZE
I cannot emphasize this one enough. At the point I began collecting the videos for MSD, I spent 4 days simply just organizing files, data, and process (in fairness: my project is very large).
Create a filename structure that makes sense for you - be sure you clearly identify each part and each performer in a uniform manner so that you can easily keep track of what you've used and what you're using next.
Before you start working on the project, practice. For all the multi-stream projects I've done, I did two or three "trial runs" of various steps of the process to understand what I'll be doing once I started on the main thing.
I cannot emphasize this one enough. At the point I began collecting the videos for MSD, I spent 4 days simply just organizing files, data, and process (in fairness: my project is very large).
Create a filename structure that makes sense for you - be sure you clearly identify each part and each performer in a uniform manner so that you can easily keep track of what you've used and what you're using next.
Before you start working on the project, practice. For all the multi-stream projects I've done, I did two or three "trial runs" of various steps of the process to understand what I'll be doing once I started on the main thing.
Step 5: It's time to actually do the thing
So now you're ready to edit.
Somewhere along the way, folks got into the routine of doing something like an audible and visible "clap" on the video as a means of lining it up. I haven't done that.
If your performers are playing correctly in time with the click track, the timing will all line up.
Whichever movie software you're using, assuming it's pro-end, you'll be able to look at the audio waveform. Zoom all the way in and line up the audio waveforms of all your performers. If they're playing correctly, they'll have very similarly-shaped waveforms. Match the patterns and your videos will line up.
Depending on how big of a project you're doing, consider editing your audio separately. For MSD, I'm doing audio editing in Logic Pro X in order to get a great EQ and to edit the timing of individual performances. As musicians, an attack or release that's off by a few milliseconds will drive you nuts - and the performers aren't going to be perfect, so you're going to need to edit their timing.
Editing the timing isn't a function in your audio app: at least, not without doing a substantial amount of preparatory work. You'll need to put your parts all next to one another, aligned in time, and then zoom in so you can see the waveforms of each individual note and the transients (peaks) of the articulations/initiations. You'll need to split the audio tracks and nudge notes forward and backward in time to get them all to line up.
Video is easier, in a way, to edit. Decide on how you want the videos laid out, get them into their spots, and get the timing lined up. Boom.
If you want the videos to animate - you need more horsepower. Doing the Blue Stars video (the one that DID go out), I spent about 20 hours editing the video (and it was very simplistic). The majority of the video editing time was just letting the computer render the edits I made. It crashed several times.
After the Blue Stars video, I spent quite a bit of money to upgrade my computer in order to handle MSD, and it's still VERY inadequate. I'm getting through it, but it takes a lot of time. To make it even achievable, I'm grouping the performers' videos into sections to create "subvideos" that will get integrated into the final product. Each subvideo takes about an hour to export.
Also, in order to process this much video - like I said: a lot of computer upgrades. Also, I do a fresh restart of the computer, I close out any and all programs/utilities that are running, including all the background "helper" utilities, and I disconnect from the network to prevent any of the helpful cloud-based background tasks (during one export, my computer received a text message and crashed the whole process).
To save some of the rendering time, I switched from Final Cut Pro to Motion. In Motion, you can create simulated cameras (like on a movie set) that can pan, float, rotate, and so on. So rather than animating the videos themselves, I'm using the cameras to move across the performers. In this way, I can just create a "wall" of video and no worry about needing to do a lot of animation.
It takes a LOT of time and a LOT of horsepower.
So now you're ready to edit.
Somewhere along the way, folks got into the routine of doing something like an audible and visible "clap" on the video as a means of lining it up. I haven't done that.
If your performers are playing correctly in time with the click track, the timing will all line up.
Whichever movie software you're using, assuming it's pro-end, you'll be able to look at the audio waveform. Zoom all the way in and line up the audio waveforms of all your performers. If they're playing correctly, they'll have very similarly-shaped waveforms. Match the patterns and your videos will line up.
Depending on how big of a project you're doing, consider editing your audio separately. For MSD, I'm doing audio editing in Logic Pro X in order to get a great EQ and to edit the timing of individual performances. As musicians, an attack or release that's off by a few milliseconds will drive you nuts - and the performers aren't going to be perfect, so you're going to need to edit their timing.
Editing the timing isn't a function in your audio app: at least, not without doing a substantial amount of preparatory work. You'll need to put your parts all next to one another, aligned in time, and then zoom in so you can see the waveforms of each individual note and the transients (peaks) of the articulations/initiations. You'll need to split the audio tracks and nudge notes forward and backward in time to get them all to line up.
Video is easier, in a way, to edit. Decide on how you want the videos laid out, get them into their spots, and get the timing lined up. Boom.
If you want the videos to animate - you need more horsepower. Doing the Blue Stars video (the one that DID go out), I spent about 20 hours editing the video (and it was very simplistic). The majority of the video editing time was just letting the computer render the edits I made. It crashed several times.
After the Blue Stars video, I spent quite a bit of money to upgrade my computer in order to handle MSD, and it's still VERY inadequate. I'm getting through it, but it takes a lot of time. To make it even achievable, I'm grouping the performers' videos into sections to create "subvideos" that will get integrated into the final product. Each subvideo takes about an hour to export.
Also, in order to process this much video - like I said: a lot of computer upgrades. Also, I do a fresh restart of the computer, I close out any and all programs/utilities that are running, including all the background "helper" utilities, and I disconnect from the network to prevent any of the helpful cloud-based background tasks (during one export, my computer received a text message and crashed the whole process).
To save some of the rendering time, I switched from Final Cut Pro to Motion. In Motion, you can create simulated cameras (like on a movie set) that can pan, float, rotate, and so on. So rather than animating the videos themselves, I'm using the cameras to move across the performers. In this way, I can just create a "wall" of video and no worry about needing to do a lot of animation.
It takes a LOT of time and a LOT of horsepower.
Step 6: Enjoy!
So you're done. It's hard work and stressful work - but the final product is quite fulfilling.
With all the above - it seems that I'm saying "don't do it." I'm not saying to not do it, just setting some realistic expectations. It's not easy. It's not something for a novice to do. I don't have a LOT of video experience, but enough to crawl through the process. If you're running on the six-year-old laptop you got in your senior year of college, you're not going to handle this project. If you have an administrator asking you to do this, send them to Apple's Mac Pro website (you'll be in the $9000 range to have a computer ready for this project).
But seriously - if you do the project, the best of luck. Be patient. Take your time. Be prepared for stress. I'll be excited to see what you do.
So you're done. It's hard work and stressful work - but the final product is quite fulfilling.
With all the above - it seems that I'm saying "don't do it." I'm not saying to not do it, just setting some realistic expectations. It's not easy. It's not something for a novice to do. I don't have a LOT of video experience, but enough to crawl through the process. If you're running on the six-year-old laptop you got in your senior year of college, you're not going to handle this project. If you have an administrator asking you to do this, send them to Apple's Mac Pro website (you'll be in the $9000 range to have a computer ready for this project).
But seriously - if you do the project, the best of luck. Be patient. Take your time. Be prepared for stress. I'll be excited to see what you do.