Now at the end of the hackathon, we are very pleased to present… a Soundscape Generator!

The Idea

We were thinking of photos and what they’re lacking. It’s great to be able to look at pictures of moments and either imagine what it would be like, or remember what it was like to be there, but couldn’t they be a bit more immersive? Well of course! What if you could hear what it was like as well?

Users could simply choose or upload a photo and a list of objects in the image would be created. Then after the press of a button the app could find audio clips appropriate for each of the objects. Finally, playing all those clips together would create a realistic soundscape for the image!

The Tech

We decided to go about this by creating a Flask app with React on the frontend and use AWS Codestar to run it on an EC2 instance.

Image Scanning

We still needed a few more things though. First of all, how were we going to get the features from the images? Luckily AWS Rekognition was perfect for this. By making one simple call to it we were able to get the features no problem.

def detect_labels(self, key, bucket=BUCKET, max_labels=10, min_confidence=80,
                  region="us-east-1"):
    rekognition = boto3.client("rekognition", region)
    rek_results = rekognition.detect_labels(
        Image={
            "S3Object": {
                "Bucket": BUCKET,
                "Name": key,
            }
        },
        MaxLabels=max_labels,
        MinConfidence=min_confidence,
    )
    labels = []
    for label in rek_results['Labels']:
        labels.append(label['Name'])
    return labels

Sample Searching

Once we had the features the next step was to get the audio clips for them. Was there a sound library out there with an API? You bet there was. After a quick search we found Freesound and it was just what we needed.

With the features returned by Rekognition we could query the Freesound library for clips that had those terms in their names.

function getSoundSearchResults(keyword) {
    const url = `https://freesound.org/apiv2/search/text/?query=${keyword}`
    let headers = new Headers();
    headers.append("Authorization", "Token <token>");
    return fetch(url, {headers})
        .then((response) => response.json())
        .then(json => json.results)
        .catch((err) => {
            console.error("ERROR", err);
            throw new Error(err);
        });
}

After retrieving the audio clips they could then each be streamed from Freesound either one at a time or all together. Don’t like one of the clips? That’s okay just hit the shuffle button and another result will be fetched!

Room Settings

The last goal we had was to add an appropriate amount of effects to match the types of environments where pictures were taken. Google’s Resonance Audio just happened to fall into our laps thanks to a post on Hacker News and boy we’re glad it did! It provides the ability to add reverb, as well as gain control and panning for the audio clips. We do our best to guess the types of room materials in the pictures from the features returned by Rekognition but there is definitely room for improvement here.

What’s Next

Video

Just as we were wrapping this hackathon project up, AWS announced at their re:Invent conference that Rekognition will now be supporting video! This is really exciting for us because we can very soon add that support to our app as well. With this option we have considered that video producers could even use our app to quickly create a soundscape to drop into a scene they’re working on.

Sharing

At the moment, users only have the ability to hop on the app and listen back to the results while they’re there. We would love for people to export their creations and share them on social media as well.

Conclusion

So that’s our Soundscape Generator. It was a blast working on it and thanks to Product Hunt for the extra incentive as well as the helpful resources. You can check out the app here and our GitHub repo is over here. If you have any comments or questions about it please feel free to reach out :)

I would like to begin this new adventure with a post about a Javascript audio library and an open-source project I became involved with back in September. The project is called Waterbear and it is a visual programming language with a great mascot:

As one of the projects included in an interesting opportunity called UCOSP, as well as some others, many people have been involved with it throughout several years. It has already been through multiple iterations and continues to grow in all sorts of directions. The benefit of this being that each participant is welcome to contribute to it in any way he or she can imagine.

What to do?

For my contribution, it wasn’t long until I realized the opportunity I had to improve the sound blocks that were being offered to users. Originally, the audio library being used was Sound.js, a simple but limited option. After researching and considering other libraries I decided on Timbre.js, mainly because of the awesome examples in their documentation and the quality of their sounds.

We decided that out of all the functionality we could add to the sound portion of the Waterbear project, the top priorities were to be:

• Playback of files
• Generation of sound effects
• Building songs from notes

Fortunately, Timbre.js was capable of implementing all of these things so it was a perfect candidate for the job.

The Basics

Note: You can play with these code examples on JSFiddle after including this as an external resource or just modify the examples on the Timbre.js site.

I began with generating some waveforms. The most primitive example that provides a sine wave looks something like this:

T("sin", {freq:880}).play();

Here the Timbre Object (‘T’) is being created with parameters for the type of waveform and the frequency. The dictionary attribute containing the frequency can include a number of additional parameters such as volume and envelope to further customize the synth. The play method is then called on that object to generate the tone.

Timbre.js synthesis has the ability to play single notes as well as chords:

var sine1 = T("sin", {freq:440, mul:0.5});
var sine2 = T("sin", {freq:660, mul:0.5});
var sine3 = T("sin", {freq:880, mul:0.5});

T("perc", {r:500}, sine1, sine2, sine3).on("ended", function() {
  this.pause();
}).bang().play();

In this example, several sine waves are being combined inside of a "perc" Timbre object. on() is called on that object to add a listener to the end of the listeners array when the synth has "ended". When the Timbre object has ended, the listener will pause the synth so it can be started again later without reloading the script. Then bang() sets playback to be a short burst of sound rather than an ongoing signal and play() begins the processing. The release time is set to 500ms with "{r:500}".

The block that was created to define synths within Waterbear includes options for the waveform, attack time, and release time. This provides a reasonable amount of sound-shaping for people to get started with:

With this block a synth is being defined by the following:

var osc = T("sine");
var env = T("perc", {a:10, r:300});
var oscenv = T("OscGen", {osc:osc, env:env, mul:0.15}).play();

To see an advanced example of what Timbre.js can do with audio synthesis check out ChordWork as well as the other examples they provide.

Playback of files

Audio files can easily be played:

T("audio", {load:"/some/audio/file.wav"}).play();

But many effects such as chorus, delay, phaser, and reverb can be added to playback as well.

Generation of sound effects

For Waterbear there was a demand to have sound effects ready to play at certain points within a program. Things such as lasers and alarms can be used at key points to alert the user of an action. Thus, it was important for the chosen audio library to generate such sounds.

Here is how to create a laser sound effect with Timbre.js:

var table = [1760, [110, "200ms"]];

var freq = T("env", {table:table}).on("bang", function() {
    VCO.mul = 0.2;
}).on("ended", function() {
    VCO.mul = 0;
});
var VCO = T("saw", {freq:freq, mul:0}).play();
freq.bang();

Several sound effects were created in this way and a block was made with a drop-down menu to choose an effect from:

Building songs from notes

Another big feature request was to be able to define notes with specific frequencies and durations so that they could be used to drag-and-click out a melody. See the “Twinkling Song” example on Waterbear to get a feel for what it’s like to use a visual programming language to do so. For short parts it is quite convenient!

First, a synth must be defined, then optionally set the tempo (120bpm by default), and then notes can be added. When it is time to play the melody within the program, a play block must be dropped into Waterbear in the desired location:

In order to queue up notes for playback the information collected from the blocks has to first be converted into Music Markup Language:

global.runtime = {
	...
	addNote: function(note, octave, beats){
	    switch(note){
	        case "A":
	            note = "a";
	            break;
	        ...
	        case "Rest":
	            note = "r";
	            break;
	    }
	    if (octave > current_octave) {
	        var octave_diff = octave - current_octave;
	        for (var i = 0; i < octave_diff; i++) {
	            song += "<";
	        }
	    }
	    else if (octave < current_octave) {
	        var octave_diff = current_octave - octave;
	        for (var i = 0; i < octave_diff; i++) {
	            song += ">";
	        }
	    }
	    current_octave = octave;
	    var length;
	    switch(beats){
	        case "1/32":
	            length = "32";
	            break;
	        ...
	        case "1":
	            length = "1";
	            break;
	    }
	    var newNote = note + length;
	    song += newNote;
	},
	...
}

So that a string such as song="o4 l4 V12 cd" is created, then that string is passed to Timbre.js with the play block:

T("mml", {mml:song}, sound).on("ended", function() {
    sound.pause();
    this.stop();
}).start();

Alternatively, uses can enter in MML directly to a different block:

Conclusion

The code that used Timbre.js resides in runtime.js and the HTML for the sound blocks is in playground.html. There are even more sound blocks than discussed here so I urge you to go and explore the rest on your own.

I found this project to be a great experience with taking an audio library and implementing it in an interesting way. Visual programming languages are meant to teach people to think like coders so a lot of consideration was put into making all options visible and understandable in order for users to quickly grasp what’s available to them.

So thanks very much for reading and I hope you enjoyed my overerview of these musical technologies. If you have any thoughts or opinions then please feel free to share them in the comments below or send me a message!