Developers Club geek daily blog

3 years ago
As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1ГцIn this article I will tell about that as I had had idea to write the tuner and to what it has led. And also I will share the modest knowledge in the field of TsOS (digital signal processing) gained in HIGHER EDUCATION INSTITUTION and as they have helped me to solve some problems. And of course, I will share the source code and experience of programming on Swift which has received at implementation of this project.

Background


The idea to write the tuner for guitar at me has arisen for a long time, about 10 years ago when I studied at university. To that was available variety of reasons:

  • First, not so that is a lot of really qualitative programs tuners — some simply incorrectly determine frequency, or work extremely unstably at upper strings. Besides, all tuners which came across to me, à priori are configured on A440 (when la of the first octave is configured on 440 Hz). But there is still setup on C256 (to the first octave it is configured on 256 Hz). Many tuners do not allow to be configured on frets though if it is necessary to play on 5-7 harmony, it is better to configure the tool on the same frets.
  • Secondly, visualization of setup in the form of the toddler, range or the frequency oscillogram, in my opinion, is a little not fizichna. I wanted to show as the wave generated by the tool directly — as far as it pure, or on the contrary distorted looks. To make visualization visual and sympathetic.
  • Thirdly, there was a wish to apply on advantage of mankind the knowledge in area in TsOS gained in HIGHER EDUCATION INSTITUTION.

Generally, the idea long sat at me in the head, but to implement it the iOs-device and some experience of mobile development has turned out only this year as at last there was free time.

It is a little theory


Any person a little familiar with the European string instruments knows that each string is configured on some note, and to this note there corresponds some frequency. Difference between note and frequency that each note bears in itself some musical function. And this function is defined by the provision of note concerning other notes, so for example the note to - major plays role sequences tonics, i.e. the basic and the steadiest sound. If this note a little moves, it will lose the function. For this purpose what the elicited sounds corresponded to notes it is required to observe strict compliance between intervals of notes and ratio of frequencies. Compliance of the elicited sound of that function will depend on the accuracy and frequency stability the generated string that have been put in note.

What is the sound generated by the string? If to use ADSR model which was offered by the American researcher and the composer Vladimir Usachevsky for the first synthesizers, the sound of string represents modulated some bending around harmonic oscillation. This bending around has received the name ADSR since it has four characteristic moments: attack (English attack), recession (English decay), susteyn (English sustain) and attenuation (English release).

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


The interval of susteyn most accurately transfers the frequency since fluctuations happen practically without change of amplitude. If the guitar generated ideal monoharmonious sound, taking into account ADSR bending around range of such fluctuation would have appearance of narrow strip. The form of this strip would correspond to range of the bending around:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


But the real tool generates nonlinear oscillatory processes therefore there are additional harmonics which call overtones.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


These overtones are quite artful fellow travelers since they can prevail over the main tone and disturb determination of frequency. But nevertheless, usually main tone well is defined without additional manipulations.

So, on the basis of these representations it is possible to plan way on which the program has to work:

  1. To calculate Fourier transform of sound wave
  2. To find the main tone on range
  3. To calculate the frequency of the main tone

About standing waves


Usually the sound on the diagram is presented in the form by development on time. In point zero on abscissa axis is located value at the initial moment of supervision, further according to value which will be observed through 1 with, 2 with, etc. Measurement process in that case can be provided the imagined frame which evenly moves at the left on the right. This frame can be called on miscellaneous: watch window (English observation window), supervision interval (English observation interval), temporary window (English time window) — all these terms mean about one and too.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


So, the frame lets to us know as there is process of measurement and what moments appear at the beginning of frame, and what at the end. On the basis of it we can provide that will if compare coordinate system with frame — at us there will be feeling that the sound wave appears in the right part of the diagram and disappears in the left. Such wave is called running:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


But such representation of sound wave not informatively since the wave can move very much and very quickly. Our task somehow to stop wave. For this purpose that the wave would stop it is necessary that its speed would be equally 0. And since the wave has two speeds: phase and group, it is possible to receive two types of standing waves:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


The standing wave at which group velocity is equal to zero is characterized by that it bending around remains always on one place. But thus fluctuations do not stop — zero and gorbik continue to move along abscissa axis. It is obvious that such wave does not suit us since that is of interest that occurs in ADSR bending around, namely at the moment when we observe fluctuations in the mode of susteyn.

For this purpose there is other type of standing waves at which the phase velocity is equal to zero:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


The zero phase velocity guarantees that nodes and gorbik always remain on one place therefore we can easily make out form of harmonious fluctuation and evaluate as far as it is close to ideal sinusoidal form. The algorithm of receiving such wave is obvious:
  1. To find phase of the main tone
  2. To displace the displayed wave on phase value

Implementation


Record from microphone


Generally Apple gives many high-level opportunities for work with multimedia from Objective-C/Swift. But in fact now work with sound to turn Audio Queue Services around:

Audio Queue Services provides features similar to those previously offered by the Sound Manager and in OS X. It adds additional features such as synchronization. The Sound Manager is deprecated in OS X v10.5 and does not work with 64-bit applications. Audio Queue Services is recommended for all new development and as a replacement for the Sound Manager in existing Mac apps.
source

But unlike SoundManager'a which was pretty high-level solution, Audio Queue Services it is clumsy wrappers which simply repeat C-code on Swift:

    func AudioQueueNewInput(_ inFormat: UnsafePointer<AudioStreamBasicDescription>,
                      _ inCallbackProc: AudioQueueInputCallback,
                      _ inUserData: UnsafeMutablePointer<Void>,
                      _ inCallbackRunLoop: CFRunLoop!,
                      _ inCallbackRunLoopMode: CFString!,
                      _ inFlags: UInt32,
                      _ outAQ: UnsafeMutablePointer<AudioQueueRef>) -> OSStatus

The profit from low-level code in Swift is not present therefore I left capture of sound on payoff to auxiliary C-code. If to lower minor code for control of buffers, setup of record consists in initialization of structure of AQRecorderState by means of the AudioQueueNewInput function:

    void AQRecorderState_init(struct AQRecorderState* aq, double sampleRate, size_t count){
        aq->mDataFormat.mFormatID = kAudioFormatLinearPCM;
        aq->mDataFormat.mFormatFlags = kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsSignedInteger;
        aq->mDataFormat.mSampleRate = sampleRate;
        aq->mDataFormat.mChannelsPerFrame = 1;
        aq->mDataFormat.mBitsPerChannel = 16;
        aq->mDataFormat.mFramesPerPacket = 1;
        aq->mDataFormat.mBytesPerPacket = 2;// for linear pcm
        aq->mDataFormat.mBytesPerFrame = 2;

        AudioQueueNewInput(&aq-;>mDataFormat, HandleInputBuffer, aq, NULL, kCFRunLoopCommonModes, 0, &aq-;>mQueue);

        DeriveBufferSize(aq->mQueue,
                         &aq-;>mDataFormat,
                         (double)count / sampleRate,  // seconds
                         &aq-;>bufferByteSize);

        for (int i = 0; i < kNumberBuffers; ++i) {
            AudioQueueAllocateBuffer(aq->mQueue, aq->bufferByteSize, &aq-;>mBuffers[i]);
            AudioQueueEnqueueBuffer(aq->mQueue, aq->mBuffers[i], 0, NULL);
        }

        aq->mCurrentPacket = 0;
        aq->mIsRunning = true;

        aq->buffer = Buffer_new(32768);
        aq->preview_buffer = Buffer_new(5000);

        AudioQueueStart(aq->mQueue, NULL);
    }

Data record happens through the HandleInputBuffer function. Calls of Buffer_write_ints will transform data from int to float and saves in the buffer for later processing.

    static void HandleInputBuffer (
        void                                 *aqData,
        AudioQueueRef                        inAQ,
        AudioQueueBufferRef                  inBuffer,
        const AudioTimeStamp                 *inStartTime,
        UInt32                               inNumPackets,
        const AudioStreamPacketDescription   *inPacketDesc
    ) {
        struct AQRecorderState* pAqData = (struct AQRecorderState*)aqData;

        if(inNumPackets == 0 &&pAqData->mDataFormat.mBytesPerPacket != 0)
            inNumPackets = inBuffer->mAudioDataByteSize / pAqData->mDataFormat.mBytesPerPacket;

        const SInt16* data = inBuffer->mAudioData;
        Buffer_write_ints(pAqData->buffer, data, inNumPackets);
        Buffer_write_ints(pAqData->preview_buffer, data, inNumPackets);

        if (pAqData->mIsRunning == 0) return;

        AudioQueueEnqueueBuffer(pAqData->mQueue, inBuffer, 0, NULL);
    }

Performance concerns and Swift


Originally the idea consisted what for 100% to use the Swift language. In general so I have arrived, have copied all code on Swift except for FFT for which implementation from Accelerate library was used. But strangely enough, the version on Swift has given out the huge loading around 95% of processor time and delay in processing of signal which brought to awfully slow drawing.

In such look of course application was not suitable for use therefore it was necessary all processing of signal completely to transfer to Accelerate library. But even after that loading all the same remained high. It was necessary to transfer to C and those operations with arrays which demanded only one pass, i.e. linear time of execution. For illustration I will give identical code on Swift and C:

    class Processing{
        ...
        func getFrequency() -> Double {
            var peak: Double = 0
            var peakFrequency: Double = 0

            for i in 1..<spectrum.count/2 {
                var spectrumValue: Double = p->spectrum[i]
                var f: Double = fd * i / spectrum.count

                if (spectrumValue > peak) {
                    peak = spectrum[i]
                    peakFrequency = f
                }
            }
            return peakFrequency
        }
    }

    double get_frequency(p* processing){
        double peak = 0;
        double peakFrequency = 0;

        for(size_t i = 1; i < p.spectrumLength / 2; i ++){
            double spectrumValue = p->spectrum[i];
            double f = p->fd * i / p->spectrumLength;

            if (spectrumValue > peak) {
                peak = spectrum;
                peakFrequency = f;
            }
        }

        return peakFrequency;
    }

Generally, even trivial pass on array if it becomes in the scraper with frequency of 20 calls a seconds could load the device quite strongly.

Perhaps it was the problem of the first Swift version, but completely everything as a result was necessary to exclude from Swift that made step-by-step operations. So in Swift remained only responsible for creation of arrays, transfer to the support libraries written on C, and also drawing code on OpenGL ES 2.

Whether but there was advantage of Swift? Certainly, as for high-level tasks, Swift copes with it perfectly. It is much more pleasant to write the code, modern syntax which is not demanding constant points with commas it is a lot of intuitively clear and useful syntax sugar. So in spite of the fact that use of Swift has generated some problems, in general language has seemed to the quite pleasant.

Problem of sensitivity of microphone


So, having copied part of code on C, it seemed that the moment when I am able already to configure the guitar just about will come. But there was one more trouble of which I did not think at all. The microphone on iPhone awfully cut off low-frequency part of range. Of course, I assumed that microphones in smartphones are not ideal at all, but everything was much worse since the blockage began already on 200 Hz.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


As for setup of guitar, such blockage of AChH does impossible setup of the sixth string since it has frequency about 80 Hz. At such easing the main tone starts sinking in harmonics with frequency of 160 Hz, 240 Hz, etc.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


I have planned at once two possible solutions of this problem:
  • if basic frequency has twin with frequency of 0,5, it has to testify to that that the main tone is muffled and the resulting frequency has to be corrected and make 1/2f
  • to give the chance to the user to suppress overtones the low-frequency filter which will cut frequencies a little big than tone of string

The first option has seemed more interesting since did not demand from the user of additional efforts. Nevertheless, it was not absolutely well-founded since led to many bad situations. For example, basic frequency occasionally was cut by microphone so strongly that was at the level of 1-2% of the first harmonic. Besides, since the resonator of guitar is very non-linear circuit, quite often there was situation when the second and third, and even the fourth harmonic started competing among themselves on amplitude. It brought to that that tone four times higher than the main was taken.

In principle, it is problems could be solved in the program way. The main disappointment has been caused by that harmonics in guitar very strongly walk therefore software configuration will not provide them accuracy in ±0,1 Hz in any way. It is connected probably by that the main tone is set by string with quite stable frequency, on the contrary harmonics are supported generally due to fluctuations in the body of guitar and can deviate on some Hertz during sounding of string.

Therefore it was necessary to refuse the first solution to please more predictable and less convenient. So, the low-frequency filter has approximately such frequency response:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц

The blockage to the right of cut-off frequency provides suppression of overtones so the main tone again becomes prevailing. Payment for this general decrease in signal-to-noise ratio and as a result small decrease in accuracy, however within the admissible.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц

Accuracy vs high-speed performance


In digital signal processing the problem of choice of the size of watch window always emerges. The big watch window allows to collect more information, to do exact and stable assessment of parameters of signal. On the other hand, it creates number of problems because of that that is required to store and process bigger number of counting for time, plus leads to emergence to proportional delay in processing of signal.

In turn Accelerate allows to calculate range of sequences of not exceeding 32768 counting. But such number of counting means that the step of frequency grid is approximately equal in range 1,35 Hz. On the one hand it is admissible value, so far as concerns for example mi the first octave with frequency of 440 Hz, i.e. note which turns out on open first string (thinnest). But on the sixth string such error is fatal, since between mi big octave and, for example, re big octave only 3 Hz. I.e. the error in 1,35 Hz is error in half-tone.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц

Nevertheless, solution of this problem quite simple, but it also shows all force of time-and-frequency analysis. Since there is no opportunity to accumulate some seconds of signal, we can accumulate value of range at frequency of the main tone with repeated Fourier transform. Mathematically the result will be equivalent processings of 1,35 Hz the filter at frequency of the main tone. Having only 16 complex counting, we can increase result accuracy by 16 times, i.e. approximately to 0,08 Hz that is a little more exact than ±0,1 Hz.

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц

In other words, without having information on value of the main tone, it should us for obtaining accuracy in ±0,1 Hz increase temporary window to 5 with and process 163840 counting. But since at temporary window in 0.743 with we can already make frequency assessment about the accuracy of 1,35 Hz, for more exact assessment it is enough to accumulate counting from the extremely narrow band with sampling in 2,7 Hz. For this purpose 2,7 Hz * 5 with = there are enough 13,75 complex counting (or 16 if to round and take with stock).

Comparison of notes and frequencies


It is problem it is quite easily solved on Swift. I have created the special class Tuner in which has brought all information on the supported tools and rules of comparison. All these calculations are based on two formulas "baseFrequency * pow (2.0, (n — b)/12.0)" and "12.0 * log (f/baseFrequency)/log (2) + b",
where baseFrequency is the frequency of base of 440 Hz or 256 Hz, b — number of note in integral numbers, beginning from to subcounteroctave.

The code has turned out quite Chinese:

class Tuner {
    ...

    init(){
        addInstrument("guitar", [
            ("Standard", "e2 a2 d3 g3 b3 e4"),
            ("New Standard", "c2 g2 d3 a3 e4 g4"),
            ("Russian", "d2 g2 b2 d3 g3 b3 d4"),
            ("Drop D", "d2 a2 d3 g3 b3 e4"),
            ("Drop C", "c2 g2 c3 f3 a3 d4"),
            ("Drop G", "g2 d2 g3 c4 e4 a4"),
            ("Open D", "d2 a2 d3 f#3 a3 d4"),
            ("Open C", "c2 g2 c3 g3 c4 e4"),
            ("Open G", "g2 g3 d3 g3 b3 d4"),
            ("Lute", "e2 a2 d3 f#3 b3 e4"),
            ("Irish", "d2 a2 d3 g3 a3 d4")
        ])

        ...
    }

    ...

    func noteNumber(noteString: String) -> Int {
        var note = noteString.lowercaseString
        var number = 0
        var octave = 0

        if note.hasPrefix("c") { number = 0; }
        if note.hasPrefix("c#") { number = 1; }
        ...
        if note.hasPrefix("b") { number = 11; }

        if note.hasSuffix("0") { octave = 0; }
        if note.hasSuffix("1") { octave = 1; }
        ...
        if note.hasPrefix("8") { octave = 8; }

        return 12 * octave + number
    }

    func noteString(num: Double) -> String {
        var noteOctave: Int = Int(num / 12)
        var noteShift: Int = Int(num % 12)

        var result = ""
        switch noteShift {
            case 0: result += "c"
            case 1: result += "c#"
            ...
            default: result += ""
        }

        return result + String(noteOctave)
    }

    func noteFrequency(noteString: String) -> Double {
        var n = noteNumber(noteString)
        var b = noteNumber(baseNote)
        return baseFrequency * pow(2.0, Double(n - b) / 12.0);
    }

    func frequencyNumber(f: Double) -> Double {
        var b = noteNumber(baseNote);
        return 12.0 * log(f / baseFrequency) / log(2) + Double(b);
    }

    func frequencyDistanceNumber(f0: Double, _ f1: Double) -> Double {
        var n0 = frequencyNumber(f0)
        var n1 = frequencyNumber(f1)
        return n1 - n0;
    }

    func targetFrequency() -> Double {
        return noteFrequency(string) * fretScale()
    }

    func actualFrequency() -> Double {
        return frequency * fretScale()
    }

    func frequencyDeviation() -> Double {
        return 100.0 * frequencyDistanceNumber(noteFrequency(string), frequency)
    }
}

Visualization of standing wave


As for standing wave which allows to see form of sound wave of the tool, how I already wrote, the algorithm is absolutely trivial — on the found basic frequency wavelength is calculated and the phase assessment is made, and then on the found value shift is made. Data undertake from the auxiliary preview buffer which, unlike the basic, does not make accumulation. I.e. works on algorithm of "the jumping window" (English tumbling window):

    double waveLength = p->fd / f;
    size_t index = p->previewLength - waveLength * 2;
    double* src = &p-;>preview[index];

    // в цикле происходит разложение основного тона на реальную и мнимую составляющую
    double re = 0; double im = 0;
    for (size_t i = 0; i < waveLength*2; i++) {
        double t = (double)2.0 * M_PI * i / waveLength;
        re += src[i] * cos(t);
        im += src[i] * sin(t);
    }

    double phase = get_phase(re, im); // фаза волны относительно начала
    double shift = waveLength * phase / (2.0 * M_PI); // искомый сдвиг

    // положение начала стоячей волны
    double* shiftedSrc = &p-;>preview[index - (size_t)(waveLength - shift) - (size_t)waveLength];

Appearance


I have made appearance and navigation on the basis of built in player, only instead of switching on tracks there is switching on strings:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц


Conclusion


All way of application programming on Swift/C has taken about two months. Application was quite difficult in implementation. First, productivity of smartphones still leaves much to be desired and solutions "in forehead" on high-level language are absolutely unsuitable for household application by users. Secondly, sound processing subject awfully unpopular at developers under iOs therefore information should be collected on particles. Though it concerns, probably, any subject which is not connected with UI when developing for mobile applications. Thirdly, though Swift and not bad communicates with C-data, but all the same such way of development is awfully inconvenient and is awful trudozatraten.

In spite of the fact that article has turned out quite powerful, many subtleties and nuances remained dark. I hope the source code of application will help to clear the unclear moments:

As I wrote the guitar tuner under iOs on Swift. And it is also a little about TsOS, standing waves and how to achieve accuracy in ±0,1Гц github.com/kreshikhin/scituner

The source code is accompanied by the MIT license. Therefore can safely use the code locations interesting you or all code of the project in the purposes.

This article is a translation of the original post at habrahabr.ru/post/266471/
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here: sysmagazine.com@gmail.com.

We believe that the knowledge, which is available at the most popular Russian IT blog habrahabr.ru, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.
Best wishes.

comments powered by Disqus