VeeCad / Stripboard Revisited

I’m ready to make a permanent version of my DTMF detector. I’m only going to build 2, maybe 3, of these so it’s not worth it to me to design and manufacture a PCB for the project.

Instead, I decided to build the project on stripboard, like this, using VeeCad to layout the components.

I’ve used stripboard on a few small projects in the past and documented my process here, which includes instructions on making a reaming tool:

How to use VeeCAD Stripboard Editor with CadSoft Eagle

As I reviewed my old process, I recalled that it was quite tedious to correctly ream the stripboard and then get the components into the correct position.

After doing the design, I printed it out to make sure all of the components fit the footprints I had specified:

This is when I had my ‘EUREKA’ moment. Yesterday I was cleaning out all of my photo paper and noticed I had 8×11.5 vinyl stickers I had never used. I could print the design on the vinyl, stick it on the board and use it for component placement:

Further, I could print a template for the back of the board to use to ream out the copper strips. Here, I’ve attached the template to the back of the stripboard and started reaming out the X’s:

The completed board came up on the first try!

Here are some additional notes:

  • Place X marks at the corners of the design.
  • Print front of board to vinyl sticker. Cut sticker to just beyond the X marks.
  • Holding sticker to board, align X marks to board, then create a hinge using tape.
  • Flip sticker up on hinges, remove backing, then flip back down into position – I start applying pressure at center and work out.
  • Print the back of the board on normal copy paper, and tape it to the back. Punch a paper clip thru several holes to verify both templates are aligned.
  • Take reaming tool and begin reaming out X’s on the back of the board. Pull off mask, check reams visually. Then check with continuity tester to be sure they were all reamed properly.
  • Use a paper clip to punch out holes where all leads will be placed AND the X’s to make sure residue is removed from hole. I find it easier to go down columns rather than across rows.
  • Any drilling required should be done now before adding components.
  • Solder wires, resisters, diodes, caps, other stuff… essentially in order from shortest to tallest.

Now to get it all to fit into a little box…

Posted in c-electronics | Tagged , | 6 Comments

DTMF Detector / Decoder Circuit

Note1: The FCC frowns upon you connecting devices to the telephone system (the old copper line network a.k.a. POTS). In my testing, this device is connected to a small PBX. When I’m done it will be connected to a VoIP ATA, not a copper phone line.

Note2: If you make a circuit such as this to connect to a POTS line, make sure you install the 2 specified fuses to protect the device from voltage spikes / lightning strikes.

I am working on a project where I want to press keys on my telephone and send commands via WIFI to a computer. To do that, I must be able to detect and decode the DTMF tones generated when pressing a key on the telephone handset.

(What is DTMF?)

As I started researching decoding DTMF, I found this little board does just that which I could connect to an Microcontroller Unit (MCU) quite easily. I ordered one up. This ended up coming via slow boat from Thailand. Literally. It took a good month to show up and when it did the RJ connectors are the wrong size. I assume these are for the Australian market as that is where they are sold from.

The mini-board uses an MT-8870 chip to do the decoding, so I started looking at just sourcing the chip myself. There is an Arduino library already in place, but as I continued research I found the only MT-8870 chips available are surface mount. I have very little experience in SMT and at this point in the game am better off avoiding it. I could find some old through-hole DIP MT-8870 chips on eBay.

As I contemplated the problem, it occurred to me I had once written code to emulate a 1200 baud modem using a Teensy MCU, it should be possible to decode DTMF directly on the MCU if it is fast enough.

The normal way to decode DTMF would be to use Fast Fourier Transform (FFT). FFT requires a pretty fast CPU, though. Not sure a cheap MCU would be up to it. As I continued to research, I learned of the Goertzel Algorithm. This is a specialized case of FFT which can detect individual frequencies with much less processing.

As I learned about the Goertzel algorithm, I found this very clean explanation of how it works. Unfortunately, the link to the source code is dead. After several days I finally found it here. There is also an Arduino library module of the algorithm.  These two examples gave me a good starting point for the software side.

If I was going to use an MCU to decode DTMF, I was going to need to to processes the DTMF signal properly to present it as 0-3.2 volts to the MCU’s Analog to Digital converter (ADC). Anything beyond that voltage range would destroy the MCU.

Further, I was going to have to be able to greatly attenuate the ring signal. This can be as high as 120 Volts.

Looking at the Circuit

The basis for my circuit comes from here(Marantz PMD Recorder) and here.

The full schematic of my circuit can be found as a PDF here.

Note, there is a Eagle schematic below, in Nov 3 update.

A bill of materials (BOM) can be found here.

There are 3 main parts to the circuit: the DTMF processor, the Tone Injector, and a square to sine wave converter. The square to sine wave converter is not implemented and will not be discussed.

DTMF Processor Circuit

The DTMF (and RING) signal  comes from the telephone(tip and ring). 2 fuses, F1 and F2, protect the circuit from lightning. The capacitor C1 is non-polarized and high voltage to block DC voltage.

Transformer TR1 is a normal 600:600 transformer used in telephony to isolate our circuit from the telephone network. An example of this transformer can be found here.

The values for F1, F2, C1 and TR1 are critical especially for a POTS line.

There are two ways to look at this circuit – what happens when DTMF tone occurs, and what happens when a RING signal occurs. To design it, I started by  getting the circuit to work for the DTMF tone, and then added further protection from the RING signal.

The DTMF tone starts on the phone line as a signal oscillating from 6.6 to 9.8V. When it passes thru C1 and TR1, it becomes -0.9 to +0.9V. The signal needs to be positive, so we add 3.3V to it and get 2.5 to 4.1V on the input side of C2.

C2 will block the DC component of the signal (the 3.3V we added). R2 and R3 form a voltage divider. This will center the frequency. That is, the signal crosses the X axis at 3.3V/2 or 1.65 V. At this point in the circuit, the DTMF signal can be properly read by the MCU.

The RING signal is very high voltage which will destroy the MCU. My PBX generates a -76 to +74V RING signal. D1 and D2 bleed most of the voltage off so the RING signal is between -0.7 and 7.4V on the input side of C2.

The voltage divider attenuates the RING signal a bit, but it still peaks at 4.8V – way too high. The zener diodes D3 & D4 bleed off the rest of the excess voltage. The tested voltage of the RING signal at the MCU pin is 0.9 to 2.2V.

The DTMF output signal is now MCU safe and can be connected to an ADC port on the MCU.

Tone Injector

While this post is primarily concerned with DTMF detection, I also want to notify the user  their key sequences were accepted. This is done by generating a tone with the MCU and injecting it onto the phone line using the Tone Injector circuit.

As the tone comes in, R8 and R9 create a voltage divider to attenuate the level of the tone to an appropriate value. I determined these values by first using a potentiometer and and just listening to the tone until I got a pleasing volume.

C3 simply blocks the DC voltage from where I connect this sub-circuit to the DTMF circuit.

The RING signal can make its way back from the DTMF circuit, so D5 and D6 are used to limit the voltage of any signal coming back from the DTMF circuit.

The Software Environment

For this demo, I decided to use a Teensy 3.2 MCU. For the most part, code written for an Arduino will work on a Teensy and vice versa. I have done a lot of work with Teensy’s. They are much faster than Arduinos, and I have several just laying around so I started with them.

I expect to design the final version of the project on an Arduino 33 IoT. It too is much faster than an Atmel Arduino like the Nano or Uno and it includes an integrated WIFI module.

The speed of the CPU is a major concern. The ADC is sampled 8,000 times a second. This is easy for a 72MHz Teensy. An 8MHz Arduino simply cannot sample as fast.

I did not test my code on an Arduino, but I reduced the clock speed of the Teensy to see the effects. At 24MHz, the Teensy was still sampling OK. At 16MHz it started becoming flaky. At 8MHz it did not function.

It may be possible to use this software on an 8MHz Arduino, but the sampling rate would need to be reduced. I did not experiment to see if it would work.

Since this is a new project, I downloaded and installed the latest version of the Arduino IDE (1.8.16). I then installed the latest Teensy Teensyduino (1.55) over that.

I also found I needed a few libraries which were not installed in the IDE by default. I like to keep these libraries in the code directory rather than installed into the IDE.

In these libraries are a FIFO queue handler, cppQueue, a TM1637 LED display driver, and a file of the frequencies for various musical notes.

In reviewing the few C versions of code I could find which implement the Goertzel algorithm, none did what I wanted. Because other things could be happening on the MCU besides sampling the signal, I want the sampler to run as a timed interrupt.

The Code

The code for the project can be found here.

As I mentioned, this is the beginning of a bigger project, so there is some stub code in here already for that project. The main project, sjdtmf.ino, contains a state machine which gets far enough to allow testing of the dtmf.cpp module.

I use the code::blocks editor/IDE. sjdtmf.cbp is it’s project information file.

globals.h and globals.cpp contain constants and code that could be called from any module.

Most of the pertinent code, however, can be found in the dtmf.cpp module.

dtm_init initializes the module. You pass the sample rate and block size.  This module will calculate the Goertzel constants for each of the frequencies (stored in dtm_coeffs) and then start an interrupt timer with the correct interval to sample at 8,000 samples per second.

If you change the sample rate passed to dtm_init() you will also need to calculate the appropriate timer interval. This could be done automatically, I just didn’t do so myself.

dtm_tick is the heart of the algorithm – haha. It is called once every 125 microseconds which results in an 8,000 samples/second sample rate.

First, it reads the ADC to see what voltage is coming from the DTMF processor circuit:

sample = analogRead(pin_dtmfIn);

It then uses that value to recompute Q1 and Q2 values of the Goertzel algorithm for each DTMF frequency:

for(i = 0;i < dtm_toneLen; i = i + 1) {
    Q0 = dtm_coeffs[i] * dtm_Q1[i] - dtm_Q2[i] + (float) (sample - dtm_adcCenter);
    dtm_Q2[i] = dtm_Q1[i];
    dtm_Q1[i] = Q0;

Note the calculation (sample – dtm_adcCenter). This adjusts the ADC input to a proper value for the Goertzel algorithm.

If there is NO DTMF signal, the voltage to the ADC should be 1.65V (3.3V/2) and the ADC should read 512 (1023/2). Unless the resistors R2/R3 are perfectly matched, the ADC won’t quite read 512 when there is no signal. This is corrected with a proper value for dtm_adcCenter.

To determine the value for dtm_adcCenter, I monitored the ADC when there was no signal and got 505. If you cannot determine the corrected value, then using 512 is a good estimate.

Subtracting dtm_adcCenter (505) from the ADC gives values from -505 to 518 – the proper values for the algorithm.

The next chunk of code toggles a digital pin on the MCU:

if (flag)
    digitalWrite(pin_freqTest, HIGH);
    digitalWrite(pin_freqTest, LOW);
flag = !flag;

One of the implementations I reviewed did this and I thought it was quite clever. If I want to know exactly what the real sample rate is, I simply connect an oscilloscope or frequency counter to the MCU’s pin. Then multiply by 2 to get the sample rate. With my 72MHz Teensy I was getting exactly 8K samples / second.


dtm_sampleCount = dtm_sampleCount + 1;                      
if (dtm_sampleCount >= dtm_blockSize) {                     

This increments the sample count. If we’ve seen enough samples to fill a block, then we try to detect a key, and finally reset the algorithm to read another block of samples.

dtm_detect – If dtm_tick is the heart, then dtm_detect must be the brain of the  DTMF module. It determines which, if any, frequencies were seen, if they combine to create a key press, what the key pressed is, and then queue it for consumption by the user.

First, the magnitude for each DTMF frequency is calculated in the array magnitudes:

for(i = 0; i < dtm_toneLen; i = i + 1) {
     magnitudes[i] = sqrt(
        dtm_Q1[i] * dtm_Q1[i] +
        dtm_Q2[i] * dtm_Q2[i] -
        dtm_coeffs[i] * dtm_Q1[i] *dtm_Q2[i]
    } // for

This is taken straight from the algorithm.

Next, we determine if any of the thresholds exceed an arbitrary threshold. If so a bit is set in a bitmap of frequencies:

found = false;                                              
for (i = 0; i < dtm_toneLen; i = i + 1) { 
    if (magnitudes[i] >= dtm_threshold) {
        found = true;					                    
        bitmap = bitmap | bitIx;
    bitIx = bitIx << 1;
    } // for

The value for dtm_threshold was determined empirically using every handset I could find and I have a box full of old handsets! However, in my final code, this value will be stored in eeprom and be changeable by the end-user.

If any frequency signal exceeded the threshold above, we then compare the bitmap of the frequencies found to bitmaps of valid combinations of frequencies. If a match is found, we then locate the key would have been pressed to generate the frequencies seen and place it in the variable curKey.

if (found) {						                        
    for (i = 0; i < dtm_toneLen*2; i = i + 1)		        
        if (bitmap == dtm_bitmaps[i])                       
    if (i < dtm_toneLen*2)				                    
        curKey = dtm_keymap[i];				                
        curKey = '\0';					                    
    curKey = '\0';					                        

Note the usage of the boolean found here. This is a vestige of some old code and could be reduced to just checking if bitmap != 0.

The last chunk of code looks at the transition of the current and previous key pressed and determines what needs to be done. curKey/lastKey may contain the ASCII value of a key (0,1,2…) or it may contain ‘\0’ indicating no key pressed. There are 4 possibilities:

lastKey and curKey both EMPTY: There has been no change, so we do nothing.

lastKey EMPTY and curKey VALID: a new key was pressed. Save it in lastKey and start the timer.

lastkey VALID and curKey EMPTY: the key was released. Calculate duration of key press. If the key was pressed long enough (> dtm_durationTimeout), then consider this a valid keypress and enqueue it into the FIFO queue. Finally, set lastKey to EMPTY.

lastKey == curKey: Same key is still being pressed, do nothing.

If code makes it to this point, there was an unexpected condition. Reset everything.

dtm_read / dtm_available / dtm_peek – When keys are pressed they are detected and placed into the FIFO queue by a timer-based interrupt. This leaves the main program free to do other things. It can then look to see if there are any keys in the queue to process. The read/available/peek functions provided by the DTMF module function like the SERIAL library, allowing the main program to access the keys.

For example, in the module sjdtmf.ino and in the idle state (case st_idle:) , I look to see if a key has been pressed, and if so change the state to st_keyPressed so the key can be processed:

    case st_idle:

        if (swDebounce(pin_switch)) {                       // was reset switch pressed?
            state = st_swPressed;

        if (dtm_available() > 0) {                          // was a key pressed
            state = st_keyPressed;

        break; //st_idle

Demo Video

I put together a short video of my breadboarded version of this DTMF project. It can be seen here:

Nov 2, 2021 Update

Since posting blog I have created a proper prototype:

While doing this I found some problems:

The capacitor for C1 in the BOM is redonkulously large. The next BOM will have an alternate that is reasonably sized.

The cleaned up circuit was getting a “ringing” on the line every time I pressed a key. Finally tracked it down to the LED display I’m using. I put a 100uF capacitor between the Vcc going into the LED and ground. This resolved that problem.

Once I have the problems 100% resolved, I’ll update the BOM and schematic.

Nov 3 Update

Created Eagle CAD project for the schematic.

A PDF of Eagle CAD schematic for the project can be found by clicking on this picture:

The Eagle Project can be found here.

Note that the board has not been laid out since this version is just a breadboard prototype for me.

Change notes:

F3 and D7 have been added to protect the board from excessive current or a reversed power supply.

Added zero ohm resisters in parallel to all fuses. These can be used as jumpers if the fuses are not installed.

C4 was added which keeps the signal clean when the LED is being updated.

I attempted to compile my code on the Nano 33 IoT I expect to use and it immediately failed because intervalTimer.h is a teensy-only module. There is a special timer module for the Nano 33 which I will implement. It has been so long since I used a normal Arduino, I’m not sure what library one would use to replace intervalTimer, but I’m sure one must exist.

Nov 9 Update: Nano 33 IoT Notes

The obvious changes to replace the Teensy with a Nano 33 IoT were to implement a timer library (SAMDTimer), and a fast analogRead function, analogReadFast(). It would have been nice if the Nano would just start working after those changes, but that would be too simple! It would not properly sense the DTMF tones.

The main issue is the difference in clock speed. The Nano is 48MHz while the Teensy is 72MHz. Though the Teensy worked properly when I set it to use 48Mhz, I couldn’t get the Nano to work. The frequency on pin 4, indicating the actual sample rate, had a lot of jitter on it.

It finally occurred to me to time the dtm_tick() function. NOTE: because this is an interrupt called function, micros and millis do not work in it (took me a couple of trys to realized that). Instead, I had to manually call dtm_tick to test its performance.

I found dtm_tick() took 116 microsecs to complete when dtm_detect() was NOT called. The sample rate is once every 125 microseconds, so clearly the MCU was not up to the task. (Running the same test on the Teensy, it too only 24 microsecs to execute dtm_tick).

I then started reducing the sample rate to see how low I could get it. I got it down to 4K samples / sec (vs the original of 8K) with dtm_tick() now being called once every 250 microseconds.

Sensing the DTMF now worked fine, but the tone generator sounded pretty cruddy because even with dtm_tick() taking only half the cycles, it is still too much to get a clean tone. So I modified the program such than when I need to generate a tone, I turn off dtm_tick – the user is expecting a response at that momement anyway.

That worked great. The tone is very clean. Even on the teensy, it was noisy.

Now that I have DTMF working properly on the Nano, I can begin learning how to get it to communicate via WiFi to another system I wish to control via DTMF!

Posted in c-arduino, c-electronics, c-tinys | Tagged | 1 Comment

Review: “A Guide to COBOL Programming” (and others) by Daniel D. McCracken

I just wanted to stick this quick note in with my other COBOL posts.

Whilst reading the recently published book “The History of the FORTRAN Programming Language”,  mention was made of McCracken’s book, “Digital Computer Programming” written in 1957. This was the first book on the subject.

Intrigued, I went looking for “Digital Computer Programming” and found a copy on ebay for a reasonable price. It can be found online here. It was quite an interesting book and illustrated how bare to the bones programming was. Assemblers were still extremely primitive.

I went on to track down his book on FORTRAN, “A Guide to FORTRAN Programming”. Again available on ebay for a reasonable price. While I never wrote much FORTRAN, this version of FORTRAN was extremely close to what I remember the ‘old’ programmers using at my first job. VERY primitive compared to the FORTRAN ’66 I had learned.

His book, “A Guide to COBOL Programming” was more difficult to find, though still reasonably priced. It ended up coming from the UK from a company that was reselling old UK library books.

This book was written in 1963 and was based on the 1961 COBOL standard. I found it quite interesting and was surprised at how close it already was to the 1966/1974 versions I spent many years programming in. Not to dis FORTRAN, but COBOL’61 was so much easier to read/write than those early versions of FORTRAN.

If you are interested in Computer History, I can recommend all of the books mentioned. They give a great insight into what programming was like in the early days.

I’m still trying to get 2 other McCracken Books: “A Guide to IBM 1401 Programming” and (being a big Pascal fan) “A Guide to Algol Programming”. I’ve yet to see the 1401 book anywhere. The only time I found the Algol book, the owner clearly knew its rarity and it was priced accordingly 😦


Posted in c-gnuCOBOL | Tagged , , , | Leave a comment

Forcing WordPress Back to Allowing Classic Editor

Having had my last post trashed by WordPress’s block editor, I have been avoiding using WordPress at all; however, yesterday I needed to go have a look at a post while using my laptop (which I almost never use with WordPress).

When I got into the Posts view, the posts were being displayed in the old format. Lo and behold the Add | Classic Mode option was again available. Perplexing because I am sure there was no method to bring classic mode up in the past several months. I know I tried.

Then I noticed the ‘screen options’ button and found that would force WordPress back into the new way of displaying.

Went back to my desktop and now it too was magically back in Classic View mode.

I have to assume that has done something recent to make this option available again. I don’t know how long it will last as it has been my understanding they are getting rid of classic mode.

Nevertheless, here is how I switched from showing posts in the Default View (where Classic Mode is NOT available), to showing posts in the Classic View (where it is available).

Select ‘Posts’ from the left column and you will see posts listed the new way with a Screen Options drop down in the upper right corner. Click on this:

This will show the two display formats. Select classic view:

Now to add a new post using the classic editor, click on the drop down arrow and select Classic Editor.

As I said, I am nearly positive this was not available a few months ago. I’m still skeptical that the Classic Editor will be yanked away from me again so I’m not any too excited about posting anything else here.

Fool me once, shame on you. Fool me twice, shame on me.




Posted in c-Misc | Leave a comment

PMS5000 Air Quality Monitor Part 4 – Installing the Hardware (AKA WordPress BLOCK Mode F****** me)

I wrote a complete explanation of how I installed the PMS5000 and WordPress decided to discard it for me even though I was saving drafts every couple of paragraphs.

A few months ago WordPress decided to drop support for their ‘classic mode’ method for creating posts. Now you have to use their slick as chicken shit block mode editor or whatever the hell they call it. While parts of the new editor are easier to use, I find myself struggling often to do the simplest of things that I already knew how to do in classic mode.

As I have tried to explain to others many many times NEWER rarely means BETTER. It just means DIFFERENT and quite possibly WORSE.

WordPress, along with way too many other products continue to to spend a huge time working on form over function and the end result is a product that is far worse than what they started with.

I am so disappointed in loosing this post that it may very well by my last. I see no point in spending an hour or more writing something that has a small audience only to have it sent to the bit bucket. I keep copious notes in my own project file and I can refer back to those rather than attempt to share my projects here.


Posted in c-electronics | Tagged , , , | 2 Comments

PMS5003 Air Quality Monitor Part 3 – Data Presentation

After getting a week or so worth of data it was time to decide how I wanted to view it. I knew I wanted to see the output on a web page. The software for this project is going to be installed on the same Raspberry Pi I use to monitor my weather station. That already supports Apache so it will be easy enough to just create a new web page for Air Quality.

In the back of my head, I kind of knew how I wanted to see data – a series of graphs: one for today, one for the week, etc.

I started by extracting data from the SQLITE db in CSV and using LibreOffice’s Calc program to create Charts of what I thought I wanted to see. This gave me a good opportunity to see what queries I would need and how I would need to summarize the data.

For example, while I could produce a nice daily report using every data point in the database, that was too much data for the 30 day report. That data needed to be averaged. But when I averaged the data I lost the max values. After some experimenting I decided most of the reports needed the data points to be averaged, but then I would also graph the max for each each averaged sample.

Writing a script to extract data in CSV format using SQLITE3 was easy enough. But how to produce a graph for a web page?

Many years ago I had to do the same thing for a massive network monitoring project. Network performance data was summarized from a MySQL database and then gnuplot used to produce JPG files of graphs. I don’t know if that is still the best way to handle generating graphs of SQL data, but I know it works so that is how I proceeded.

Some Program Changes

While experimenting with the existing Pascal program, I found a couple of changes needed to be made from what I posted prior. The Pascal program needed to flush console output because now it’s output is to a log file that I monitor with the linux tail -f command.

The database was modified to correct a fieldname and an index was added to the data field since we will always be searching data based on date.

Further, a new script, was created to allow operation from cron. This script will verify the aqmonitor is not already running and will fixup the LCK..ttyUSB0 problem if it exists.

The new source code is here:

Building a Script to Graph 24 Hours of AQ Data

After messing with Calc, my goal was to have a graph roughly like this, except I would report only PM2.5:

I first created a CSV with pm2.5 and pm10 columns (though I will only chart the pm2.5):

# create and execute query that will output last 24 hours of data
# into aq24h.csv file

cat <<EOF | sqlite3 aqmonitor.db
.headers on
.mode csv
.output aq24h.csv
  strftime('%Y-%m-%d %H:%M', date) as date, pm25Env,
  (pm100Env-pm25Env) as pm100EOnly
from observations
where date > datetime('now','localtime','-24 hours')
order by date;

cat <<EOF | sqlite3 aqmonitor.db sends all of the commands until EOF is found to sqlite. Of these commands: .headers/.mode/.output create the proper CSV output file. Then the actual query occurs.

One thing about SQLITE that is a bit disconcerting. When you use the date ‘now’ it is always GMT. The data is the db is all store in local time. Therefore I must always use the ‘localtime’ modifier.

This creates the proper CSV file:

"2021-06-08 10:41",0,0
"2021-06-08 10:46",0,0
"2021-06-08 10:51",0,0
"2021-06-08 10:56",0,0
"2021-06-08 11:01",0,0
"2021-06-08 11:06",0,0

I created another query to will save the Maximum PM value in the bash variable ${maxPM} during this time period because I want to show that on the graph as well:

# This query determines the max Y value that will be plotted so we can 
# handle labels cleanly in the plot

maxPM=$(cat <<EOF | sqlite3 aqmonitor.db
  max(max(pm25Env), max(pm100Env-pm25Env)) as maxPM
from observations
where date > datetime('now','localtime','-24 hours')

I used gnuplot to generate the graph as a .PNG file. There is a lot going on in gnuplot so I will try to break it down. Overall, I feed commands to gnuplot the same way I fed them to sqlite3:

cat <<EOF | gnuplot 

Here is an explanation of the commands I sent to gnuplot. Note that I haven’t used gnuplot in over a decade. I found examples of what I needed online and changed them for my purposes. There may well be better ways and my understanding of what I’m doing here is limited.

I started by setting the gnuplot variable maxPM to bash’s ${maxPM} variable. I then created label 99 using that data. Next the CSV separator was specified and the graph’s Title defined:

    # display Max obs above and right of the graph
    set label 99 sprintf("Maximum Observation: %d", maxPM) at graph 1, graph 1 right offset 0, char 1
    set datafile separator ','							#CSV field separator
    set title "Particulate Matter for Last 24 Hours" font ",20"			#header

The X axis was formatted for using dates:

    set xdata time 								#tells gnuplot the x axis is time data
    set timefmt "%Y-%m-%d %H:%M" 						#specify our time string format
    set format x "%H:%M" 							#otherwise it will show only MM:SS

Here, the Y axis was labeled and the legend box defined:

    set ylabel "PM (µgrams / meter^{3})" 					#label for the Y axis    
    set ytics nomirror								#no Y ticks at top

    # setup legend box
    set style line 100 lt 1 lc rgb "dark-grey" lw 0.5                           # linestyle for the grid
    set grid ls 100 front                                                       # enable grid with specific linestyle

Here’s where it get’s a bit tricky. There are a series of ranges defined for PM2.5. For example, Good is 0 – 12 micro-grams/cubic meter:

In my graph, I want colored bars in the background for these ranges. Here, I defined each bar as a rectangle and gave it a color:

    # create colored background of bars 
    set object 50 rect from graph 0, graph 0  to graph 1, first 12        fc rgb "green"  lw 0 
    set object 51 rect from graph 0, first 12 to graph 1, first 35.4      fc rgb "yellow" lw 0
    set object 52 rect from graph 0, first 35.4  to graph 1, first 55.4   fc rgb "orange" lw 0
    set object 53 rect from graph 0, first 55.4  to graph 1, first 150.4  fc rgb "red"    lw 0
    set object 54 rect from graph 0, first 150.4 to graph 1, first 250.4  fc rgb "purple" lw 0
    set object 55 rect from graph 0, first 250.4 to graph 1, graph 1      fc rgb "brown"  lw 0

Further, I want to put text into each colored rectangle: Good, Moderate, etc. When I did this I had a lot of trouble because the text is a fixed font size and each colored bar would grow/shrink depending on the maximum values being graphed.

To deal with the problem, I decided to only display the colored bars that have data for them. To do that, I formatted the yrange and text based on the value of the maxPM variable:

    # Labels for the colored bars a defined based on which bars are actually displayed
    if (maxPM  12 && maxPM  35 && maxPM  55 && maxPM  150 && maxPM  250) {
      set yrange[0:500<*]
      set label 53 "Unhealthy"			    at graph 0.5, first 100  center tc rgb "black"
      set label 54 "Very Unhealthy"                 at graph 0.5, first 200  center tc rgb "black"
      if (maxPM <= 500) {
        set label 55 "Hazardous"                    at graph 0.5, first (500-250)/2+250    center tc rgb "black"
      else {
        set label 55 "Hazardous"	            at graph 0.5, first (maxPM-250)/2+250  center tc rgb "black" #Y position calculated to be centered

Finally, I setup the line styles for the plot, characteristics of the output graph (a PNG file), and do the actual plot:

    set style line 101               lw 1 lt rgb "black"                        #line color for PM2.5
    set style line 102 dashtype "-." lw 1 lt rgb "blue"				#line color for PM10
    # output graphic definitions
    set terminal pngcairo size 800,600 
    set output "aq24h.png"

    plot 'aq24h.csv' using 1:2 with lines ls 101 title 'PM2.5'#,\
         #''         using 1:3 with lines ls 102 title 'PM10'

When I run this for my current 24 hours, which has a maximum of 6 micro-grams/cubic meter I see:

Here, I tweak the data to create a single PM2.5 observation of 400. You can see how all of the appropriate color bars are added and the labels are removed from the skinny color bars:

Using this script as an example, I then proceeded to create scripts to produce plots for 7 days, 30 days, and 1 year.

Creating the Web Page

Now I have 4 graphs to display, plus a file that contains just text of the current time and the last observation recorded. I need an HTML file that ties this all together into a single web page.

Here is that page. The only thing even remotely tricky about this HTML file is using javascript to include the file aqCurrent.htm which contains the current observation.

Because the examples of the page I show are not yet on a web server, you won’t see this included file. But it will show up once I have the code running on a web server.

<!DOCTYPE html>
    BigDanz Air Quality Monitor
    BigDanz Air Quality Monitor
    <table border=0>
        <td align=center colspan="2">
          <div id="includedContent"></div>
          <img src="aq24h.png" style="max-width: 100%; height: auto">
          <img src="aq7d.png"  style="max-width: 100%; height: auto">
          <img src="aq30d.png" style="max-width: 100%; height: auto">
          <img src="aq1y.png"  style="max-width: 100%; height: auto">

Here is what the web page looks like. Note that the Last Year graph is a little funky because the average is for a full day, but there are only a few days of data.

The HTML file and the scripts to extract data and generate graphs are here:

In this project, I ended up graphing only PM2.5. I tried showing both on the graph, but the graph was just too sloppy. From what I’ve read, excessive PM2.5 is more critical than PM10, because it can embed itself further in the lungs. So I’ve focused only on PM2.5.

Posted in c-electronics, c-lazarus | Tagged , , , | Leave a comment

PMS5003 Air Quality Monitor Part 2 – Raspberry Pi and SQLite

After watching my little program monitor the PMS5003 air monitor properly for a few days, it was time to move on to the next steps: migrating the program to a Raspberry Pie and writing data to an SQLite database.

Moving to Raspberry PI (RPI)

The nice thing about Lazaras / Free Pascal is there is very little to do to move a program between architectures – just recompile the source on the ARM architecture.

Typically that works correctly, but the synapse library I’m using for serial I/O had a few issues recompiling on the ARM platform. Primarily it thinks there are some baud rates available that are not. Easy enough to just comment those out.

Connecting the FTDI breakout board was pretty easy as well. All drivers were already installed. After you plug the USB cable into the raspberry pi, use dmsg | grep FTDI to see the port assigned:

You can see that /dev/ttyUSB0 was assigned.

As in part 1, I used putty on the RPI to verify I was getting output from the PMS5003 sensor before going any further.

The only issue with the program written in part 1 was it was hardcoded for ports named COM. I changed the program to allow input of a string rather than a digit so I could use /dev/ttyUSB0 rather than just COM<n>.

With that change, I was receiving sensor data just as I had been with windows:

Allowing non-root Access to /dev/ttyUSB0

One difference with Linux is root access is required to use /dev/ttyUSB0. It is easy enough to allow your user access using usermod:

usermod -a -G dialout $USER

Fixing Control-C Issues

While playing around with the program, I quickly realized there was a very annoying problem. In Linux, if I use control-C to abort the program (which is the only way out), it will not properly close the serial port, making it impossible to restart the program.

When this happens you will see a message like this:

Communication error 9991: Port owned by other process
Unable to open communications port. Terminating.

At first, I was forced to reboot the RPI when I received this error, but I found that if you look at the /run/lock directory you will find a file called LCK..ttyUSB0. Delete this file and you will have access to the ttyUSB0 device again.

Although I used try/except to catch any program errors, control-C doesn’t get trapped. In Linux, the program just exits and leaves a mess unlike the Windows version.

To get around this problem, I trap for several Linux signals as the program starts:

// trap for signals that could cause abnormal termination so the com port
// can be properly closed
if FpSignal(SIGINT, @HandleSigInt) = signalhandler(SIG_ERR) then begin
    Writeln('Failed to install signal error: ', fpGetErrno);
if FpSignal(SIGHUP, @HandleSigInt) = signalhandler(SIG_ERR) then begin
    Writeln('Failed to install signal error: ', fpGetErrno);
if FpSignal(SIGTERM, @HandleSigInt) = signalhandler(SIG_ERR) then begin
    Writeln('Failed to install signal error: ', fpGetErrno);

These all install the same trap handler for all of the traps:

procedure handleSigInt(
    aSignal                             : LongInt


writeln('User requested program termination.');
abortRequested := true;

end; // handleSigInt

Then, the infinite loop that monitors the sensor output watches for abortedRequested to become true:

    while not abortRequested do begin

Installing SQLite3

I needed to install SQLite:

sudo apt install sqlite3 libsqlite3-dev

I also like to use the GUI SQL browser, so I installed that:

sudo apt install sqlitebrowser

Creating the Database

I created the database as follows:

>sqlite3 aqmonitor.db
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> CREATE TABLE observations (
   ...>                       NOT NULL
   ...>                       UNIQUE,
   ...>     date     DATETIME NOT NULL,
   ...>     duration INTEGER  NOT NULL,
   ...>     pm10Std  INTEGER  NOT NULL,
   ...>     pm25Std  INTEGER  NOT NULL,
   ...>     pm100Std INTEGER  NOT NULL,
   ...>     pm10Env  INTEGER  NOT NULL,
   ...>     pm25Env  INTEGER  NOT NULL,
   ...>     pm100E   INTEGER  NOT NULL,
   ...>     part03   INTEGER  NOT NULL,
   ...>     part05   INTEGER  NOT NULL,
   ...>     part10   INTEGER  NOT NULL,
   ...>     part25   INTEGER  NOT NULL,
   ...>     part50   INTEGER  NOT NULL,
   ...>     part100  INTEGER  NOT NULL
   ...> );
sqlite> .schema
CREATE TABLE observations (
                      NOT NULL
    date     DATETIME NOT NULL,
    duration INTEGER  NOT NULL,
    pm10Std  INTEGER  NOT NULL,
    pm25Std  INTEGER  NOT NULL,
    pm100Std INTEGER  NOT NULL,
    pm10Env  INTEGER  NOT NULL,
    pm25Env  INTEGER  NOT NULL,
    pm100E   INTEGER  NOT NULL,
    part03   INTEGER  NOT NULL,
    part05   INTEGER  NOT NULL,
    part10   INTEGER  NOT NULL,
    part25   INTEGER  NOT NULL,
    part50   INTEGER  NOT NULL,
    part100  INTEGER  NOT NULL
CREATE TABLE sqlite_sequence(name,seq);
sqlite> .quit

This creates fields for all of the data from the sensor. It also has a unique id which is the primary key. If you need to delete/modify a specific record, the Id uniquely identifies the record.

There are also fields for the date&time of the observation, and the length of time observations were averaged to get the record.

Modifying the Program to Create Database Records

SQLite access is easily integrated into Lazarus / Free Pascal using the db, sqldb, and sqlite3conn modules:

I added 3 functions to my original aqMonitor program to handle writing data to the database: dbOpen, dbClose, and dbAdd.

Opening and closing the db is very straightforward so I won’t cover that here. Adding is done each time we compute an average of the observations. When the timer pops, a new average is computed and we write it to the database using dbAdd:

if SecondsBetween(now, avgTimer) >= duration * 60 then begin        
    packet := historyAvg(duration * 60, history);                   
    dbAdd(duration, packet);                                    
    avgTimer := now;                                                

dbAdd is fairly straightforward as well. It creates an SQL insert statement, then executes it against the database.

The insert command is put into the variable s:

with packet do begin
    s := 'insert into observations (' +
            'date, duration, pm10Std, pm25Std, pm100Std, pm10Env, pm25Env, pm100E, ' +
            'part03, part05, part10, part25, part50, part100' +
            ')' +
        'values(' +
            '"' + FormatDateTime('yy-mm-dd hh:nn:ss', now) + '", ' +
            inttostr(duration) + ', ' +
            inttostr(pm10Std) + ', ' +
            inttostr(pm25Std) + ', ' +
            inttostr(pm100Std) + ', ' +
            inttostr(pm10Env) + ', ' +
            inttostr(pm25Env) + ', ' +
            inttostr(pm100Env) + ', ' +
            inttostr(particles03um) + ', ' +
            inttostr(particles05um) + ', ' +
            inttostr(particles10um) + ', ' +
            inttostr(particles25um) + ', ' +
            inttostr(particles50um) + ', ' +
            inttostr(particles100um) +
    end; // with

Once the string is created, a start transaction is executed (trans), the query is created (q), and finally executed (q.ExecSQL). Exceptions are handled, and if the Insert succeeds, everything is cleaned up in the finally block:

try try
    trans             := TSQLTransaction.Create(nil);
    trans.DataBase    := dbCB;

    q                 := TSQLQuery.Create(nil);
    q.DataBase        := dbCB;
    q.SQL.Text        := s;

    on e: EDatabaseError do begin
        writeln('Error: ' + e.Message);
        writeln('insert failed.');
    end; // try except;

    end; // try finally

Using sqlitebrowser to look at the table:

As I said, writing data to an SQL database is pretty easy.

Handling SQLite Locks

Well, there is one problem with SQLite – database locks. I’ve used SQLite pretty extensively and I like it a lot, but it is pretty stupid when it comes to locking, at least as far as I’m concerned. If it can’t obtain a lock, it will give up and give an error. There is no option to just wait until the other writer released the lock.

On a program like this, I do not want it aborting except for truly exceptional reasons. It is going to be running via cron in the background and I don’t intend to have to babysit it!

In other programs, I get around locking issues by locking my own semaphore before using SQLite. My semaphore lock would wait indefinitely rather than abort the program. That causes all database accessors to be singly threaded thru the semaphore before being allowed access to the database.

But that is too complicated to implement here, and requires all accessors use the semaphore which makes using tools like sqlitebrowser a bad idea.

Since the aqmonitor program has a single insert SQL command, it is fairly easy to simply do several retries myself. So rather than the simple q.ExecSQL I showed you above, what I actually do is this:

    // attempt to insert record. If dblocked occurs (error 5), then retry until
    // succeeds or # of retries are exceeded
    retries := 0;
    while true do begin
        try // insert
            on e: ESQLDatabaseError do begin                                
                if e.ErrorCode = 5 then begin                               
                    retries := retries + 1;
                    if retries > maxRetries then begin                      
                        writeln('Insert failed due to dblock after ', retries,
                            ' attempts.');
                        raise EInsertFailed.Create                          
                            ('Insert retries exceeded due to dblock');
                    sleep(retries * 1000);                                  
                end; // on
            end; //try insert
        end; // while

I try the insert (q.ExecSQL). If an exception occurs, I check to see if it is a lock error (errorCode = 5). If it is, I add 1 to the retries count, sleep for a while, then try again. If the number of retries is exceeded, then I raise EInsertFailed to let the caller know I gave up trying to add the record.

Further, in the while loop where dbAdd is called, I count the number of successive dbAdds that fail. If dbAdd fails 10 times in a row, then something is seriously wrong and I abort the program.

This concludes the data capture part of my Air Quality monitor.

The source the the program as it stands now, with the SQL code can be found at:

I’ll let this run for a few days to see if I can get dirtier air than I’ve had the past couple of days. Then I’ll start working on reading the database and generating some graphs to post on a website.

Posted in c-electronics, c-lazarus | Tagged , , , | 2 Comments

Reading PMS5003 Air Quality Sensors with Windows and Free Pascal

Some time in the past several months, the PMS5003 Air quality sensor came to my attention. This sensor allows you to monitor PM1.0, PM2.5, and PM10 particles in the air.

Some times the air quality around these parts gets quite bad due to dust storms or fires. Not only would I like to know just how bad the AQ (air quality) is at my house, I’d like to keep history. I’d also like to send myself a notification if it gets too bad so I know I need to shut up the house and turn on the HEPA filter.

I purchased this sensor at AdaFruit:

At $40, it isn’t cheap, but it is a pretty complex little device.

To start with, I wanted to play with this sensor using a windows PC. I needed a FTDI cable to do so. The FTDI cable converts a USB cable to the TTL serial protocol used by this board. Some quick research and I found this guy had connected the PMS5003 to his PC using an AdaFruit FT232H breakout board which can be found here:

I already cover how to connect this FTDI breakout board to your PC here:

Installing Adafruit FT232H Breakout of the FT232H USB to Serial Converter

Connecting AQ Sensor to FTDI Breakout Board

Since the sensor communicates via a normal serial interface, connection is very simple. You just need to connect power, ground, and TX & RX.

On the FTDI breakout, D0 is TX and D1 is RX. So connect D0 to RX and D1 to TX of the PMS board. For my project, I don’t intend to transmit to the sensor, but I’ll still connect the lead.

To see if you got the connections correct, just use putty to connect to the FTDI cable and you will see ‘garabage’ being transmitted. The data being transmitted is in binary so you won’t see anything useful at this point.

Reading and Formatting the Sensor Output

Once the sensor was functioning, I needed to read the binary values and  convert them into something useful.

Further, having the data spit out every second wasn’t real useful. I decided I needed to collect 5 minutes of data, average it, and output the average to get a better sense of what the air quality really was.

The program I wrote to do this is in Lazarus / Free Pascal. If you’ve seen my other posts, you know this is my language of choice. If you know C/C++ you won’t have much trouble reading the source code to understand how to implement the code in C.

The Pascal source and executable can be found at:

The Pascal source code for the program is in the file aqmonitor.lpr.

If you wish to compile this program, you will also need the synapse ‘synaser’ serial library which can be found at

Program Highlights

The sensor outputs a packet of data about once a second. I read that data and put it into this data structure:

aqPacketT                               =   record
    hdr1                                :   byte;
    hdr2                                :   byte;
    frameLen                            :   word;
    pm10Std                             :   word;
    pm25Std                             :   word;
    pm100Std                            :   word;
    pm10Env                             :   word;
    pm25Env                             :   word;
    pm100Env                            :   word;
    particles03um                       :   word;
    particles05um                       :   word;
    particles10um                       :   word;
    particles25um                       :   word;
    particles50um                       :   word;
    particles100um                      :   word;
    filler                              :   word;
    checksum                            :   word;
    end; // aqPacketT

In Pascal, ‘word’ is a 16bit unsigned integer.

There are 2 categories of ‘particulate matter’: Standard and environmental. So pm10Std is the PM1.0 standard data and pm10Env is the PM1.0 environmental data.

From others’ discussions, it appears that the standard values have something to do with calibration and it is the atmospheric environment values that should be used for actual measurements.

The variables for PM1.0, PM2.5, and PM10 will contain data for all particles <= that size. If it senses values for PM1.0, then those values will also show up in the PM2.5 and PM10 variables.

To create the data structure, I read the data stream until I see the ox42 and ox4D header bytes. Once I have those, I start reading data byte by byte and creating the data structure.

    pm10Std           := getword(comCB, calcChecksum);
    pm25Std           := getword(comCB, calcChecksum);
    pm100Std          := getword(comCB, calcChecksum);

As I’m populating the data structure, I’m also keeping track of the checksum so I can compare at the end of the packet. If they don’t match, I generate an exception. If the program gets 10 checksum errors in a row, it will assume a serious problem and abort.

    if finalChecksum <> checksum then begin
        // checksums don't match, throw exception
        raise EChecksum.Create(erMsgChecksum);

As each packet is read, it is added to a FIFO list which I have set to hold 360 samples (about 6 minutes) of data:

    historyAdd(history, packet);

Then every 5 minutes, I compute the average of all of the values for the past 5 minutes (300 seconds), and print those averages:

    if SecondsBetween(now, avgTimer) >= 300 then begin
        packet := historyAvg(300, history);
        avgTimer := now;

Here is an example of the program’s output. The air has been very clean here in my office since I started the monitor last night, so I lit a match at about 11:53 and let it smoke the room a bit:

Here is a much longer run from a few days ago when the wind was kicking up dust, then a rain storm cleaned the air. This graphs just the PM2.5 values:

Next Up

The plan is to migrate this project to a dedicated Raspberry Pi. The monitor program used here will be modified to write the data captured to a SQLite database. Once I have the data in a database, another program will periodically query the data and generate some graphics for a web page. Finally, I will need to fabricate some kind of enclosure.

Additional Resources

I am planning to use a USB cable and the documented FTDI breakout board for my final project because the Raspberry Pi will be in my garage and the sensor outside. It will be a long enough run of cable between the two, I don’t want to use straight serial TTL protocol. But if you want to fashion your own cable, this guy documents the correct connector you will need:

This guy connects the PMS5003 to an arduino and reports the current AQ on an LED display:

Pretty good article on the performance of the sensor:

Adafruit post regarding difference between Standard and Environmental values:

Posted in c-electronics, c-lazarus | Tagged , , , | Leave a comment

Installing Adafruit FT232H Breakout of the FT232H USB to Serial Converter

I recently purchased an air quality monitor I want to play with. It uses a TTL serial interface. While I could connect it directly to a Raspberry Pi (and eventually plan to do so), to start I want to have it connected to my PC to do some experimenting.

I found this guy had connected the AQ monitor using an Adafruit FT232H board rather than using the typical FTDI cable. The advantage to the FT232H board is it can provide the 5V the AQ monitor will need and yet it will handle the 3.3V data signals.

This post just covers getting this breakout board working on my PC. This should have been quick and easy but of course I had problems …

Once you have the breakout board and the pins soldered, you need to get the FTDI drivers installed. According to the manual, for Windows 10, when you connected the FT232H to the PC, the drivers should simply be downloaded from Microsoft.

This did not work automagically for either of my Win10 systems.

Configuring FT232H Drivers

When I inserted the FT232H, I received this in device manager:

To install the driver I right clicked on ‘Unknown Device’, selected Update Driver and then Search Automatically. The driver was located, but no new device showed up in the Ports section. Notice below, though, that USB serial converter does appear in the USB controllers section:

At this point, I wasted way too much time going down multiple rabbit holes trying to figure out why the COM port was not being assigned.

The trick was found here:

Click on right click the USB Serial Controller and select properties:

Select the advanced tab, then select Load VCP:

Finally, update the driver again and viola, the FT232H shows up as a COM port:

Testing the FT232H Breakout Board

To test, I put the breakout board on a small breadboard and tied the TX/RX lines together (D0/D1):

Next, start up Putty and configure it for serial operation:

Once connected, what you type will be echoed on the screen as the TX is looped back thru the RX line:

Posted in c-electronics | Tagged | 1 Comment

Accessing HP3000 with Telnet and puTTY

Telnet has never worked properly on my physical HP3000. I never needed it so I never fixed it. I decided to to track down the problem this evening.

Fixing the telnet server was fairly easy – in 2001 someone altered the Telnet Security settings to Deny *.*.*.* so no one was going to get in. Which made sense when it was in production. Now, this system runs maybe 4 hours a year and has nothing of value on it, so telnet is acceptable!

Once I had the telnet service running I found that accessing it from a Linux shell worked just fine but for some reason echo was failing when I accessed it with puTTY.

The logical answer was to change Local Echo from Auto to Forced On. Except when you do that, passwords are echoed.

The correct answer is to set Local line Editing to Forced Off:

Now, each character is echoed back from the host so passwords are not shown:

:hello manager.sys


HP3000  Release: C.65.00   User Version: C.65.00   FRI, DEC 18, 2020, 11:10 PM
MPE/iX  HP31900 C.25.06  Copyright Hewlett-Packard 1987.  All rights reserved.



Posted in c-retro | Tagged , | Leave a comment