Creating a browser-based recording tool with flask + recorder.js + p5.js on TypeScript

We will create a simple recorder using recorder.js, a wrapper for the Web Audio API. We will implement the UI by writing p5.js, the browser version of Processing, in TypeScript.

Field recording: Recording natural or environmental sounds outside a studio I made this to meet the niche demand of wanting to easily collect natural and environmental sounds, and send them to a PC for real-time processing.

Final

Image

It's just a screenshot, but it works like this.

You can (probably) quickly set up the environment and run it, so please try it if you are interested.

Recording

audio

on

the

front-end

with

recorderjs

GitHub: https://github.com/mattdiamond/Recorderjs

First, here is a demo: Simple Recorder.js demo

It's like a wrapper for the Web Audio API. It's an excellent tool that allows you to easily start, stop, and save recordings by passing an instance of AudioNode.

You can define a function to start recording like the following, so you just need to call it on any event.

let recorder: Recorder;

const startUserMedia = (stream: MediaStream) => {
  audio_context = new AudioContext();
  let input: AudioNode = audio_context.createMediaStreamSource(stream);
  recorder = new Recorder(input);
};

const startRecording = () => {
  recorder && recorder.record();
};

Then, you can get a wav Blob object with a single call to this Recorder.exportWAV() method, so you just need to POST it via ajax.

recorder &&
  recorder.exportWAV((blob: Blob) => {
    let url = URL.createObjectURL(blob);
    let fd = new FormData();
    fd.append("data", blob);
    $.ajax({
      type: "POST",
      url: "/",
      data: fd,
    }).done((data) => {
      recorder.clear();
    });
  });

Saving

the

POSTed

wav

file

with

flask

On the flask side, you can write it like this.

from flask import Flask, jsonify, request


@app.route('/', methods=['POST'])
def uploaded_wav():
    fname = "sounds/" + datetime.now().strftime('%m%d%H%M%S') + ".wav"
    with open(f"{fname}", "wb") as f:
        f.write(request.files['data'].read())
    print(f"posted sound file: {fname}")
    return jsonify({"data": fname})

With this, files like 1104235900.wav will accumulate directly under sounds/.

Sending

the

saved

file

path

via

osc

Personally, I want to use this application in a performance, so I'll try sending a message via osc when the sound file is saved. I believe this will make it easier to load sound files in real-time using Max/MSP or Max for Live on the local PC acting as a server.

We will use a package called pythonosc. (Install with pip install python-osc)

python-osc PyPI: https://pypi.org/project/python-osc/

from pythonosc import dispatcher, osc_message_builder, osc_server, udp_client


address = "127.0.0.1"
port = 5050
client = udp_client.UDPClient(address, port)


def send_osc(msg):
    msg_obj = osc_message_builder.OscMessageBuilder(address=address)
    msg_obj.add_arg(msg)
    client.send(msg_obj.build())

This is fine. Then, if you do send_osc(fname) inside the uploaded_wav() mentioned above, the file path will arrive as a message. In Max, if you do [udpreceive 5050], you can play it with open & sfplay~.

p5.js

p5js.org: https://p5js.org/

It's like Processing where you can manipulate the DOM, and since it draws on a Canvas element, you don't have to draw complex animations with CSS. It might be easier if the browser supports canvas. Also, there is a Web Editor (https://editor.p5js.org/), which is very easy to get started with because you can test the behavior without setting up an environment.

If you want to introduce TypeScript, you should first use the following repository (it was very easy).

And I referred to the following entries:

• Setting up a Processing (p5.js) environment with TypeScript + webpack - Qiita
• Created an environment where P5.js can be written in TypeScript for CreativeCoding. - Qiita

After that, just write it quickly. As an example, I'll leave the class for the recording button part of the UI...

class Button {
  private w: number;
  private h: number;
  private centerX: number;
  private centerY: number;
  private radius: number;
  private isRecording: boolean;
  private rectCircleRatio: number;
  private progress: number; // 0 ~ 300 value (about 5s)

  constructor(w: number, h: number, size: number) {
    this.w = w;
    this.h = h;
    this.centerX = w / 2;
    this.centerY = h / 2;
    this.radius = size;
    this.isRecording = false;
    this.rectCircleRatio = size / 2;
    this.progress = 0;
  }

  isTouched(x: number, y: number) {
    if ((x - this.centerX) ** 2 + (y - this.centerY) ** 2 < this.radius ** 2) {
      return true;
    }
    return false;
  }

  switchRecording() {
    this.isRecording = !this.isRecording;
    console.log(`switched to recording: ${this.isRecording}`);
    if (this.isRecording) {
      startRecording();
    } else {
      this.progress = 0;
      stopRecording();
    }
  }

  draw() {
    if (this.progress == 300) {
      this.progress = 0;
      this.switchRecording();
    }
    if (this.isRecording) {
      if (this.rectCircleRatio > 5) {
        clear();
        this.rectCircleRatio -= 5;
      }
      this.progress++;
    } else {
      if (this.rectCircleRatio <= this.radius / 2) {
        clear();
        this.rectCircleRatio += 5;
      }
    }
    drawCircleUI((this.progress * 2 * PI) / 300);
    noStroke();
    fill(mainColor);
    rect(
      this.centerX - this.radius / 2,
      this.centerY - this.radius / 2,
      this.radius,
      this.radius,
      this.rectCircleRatio,
    );
    // text
    fill(white);
    textAlign(CENTER, CENTER);
    textSize(16);
    if (this.isRecording) {
      text("STOP", this.centerX, this.centerY);
    } else {
      text("REC", this.centerX, this.centerY);
    }
  }
}

Repository

You can actually use it, so please do if you feel like it. Like osc-webapp, it is published by digging an https tunnel with ngrok. (You can't use the Web Audio API unless it's https)