N6OL

I expected that this morning, when the humidity dropped and the sun came out, the buzz would start up again, and it did.  This afforded me an opportunity to go out and try my direction-finding skills again with the Yagi and the HT.

This time, determining which pole it was with certainty was easy.

I set out to try both aiming the main lobe of the Yagi at a pole and then turning it 45 degrees to the null in the antenna pattern.  (I drew the 45 degree angles on the beam with a sharpie.)  The nulls in the pattern are about half as wide as the main lobe, meaning I could get better selectivity by looking for where the signal went away rather than where it was.

This turned out to be successful.  As it happened, there was a location on the sidewalk where turning 45 degrees to the left would put one pole in a null while pointing directly at the other pole.  And on my first try, I quickly determined which pole it was; it dropped into the null and that was it… nothing came even off the main lobe from the other pole.

From where I was standing, I had to point the antenna up some to pick up the buzz, so I was pretty sure it was coming from the pole in front of me and not one further away but in the same direction, but nonetheless, I walked around to the next block over just to make sure, and indeed, I had to turn 180 degrees to face where I had been standing to keep picking up the buzz.

I then called PG&E and had them add to the ticket that I’ve had open for the last 52 days the precise location of the pole from which the noise emanates.  I have now done more than half the work for them: I can tell them the conditions under which they’re likely to be causing interference, the time of day it’s most likely to be present (4 PM), and show them charts of when it has started and stopped and varied throughout the day.

The culprit pole is indeed along the powerline run that also runs behind our house which has already had multiple insulators replaced because of catastrophic failure.  I’m not letting PG&E off the hook until they fix this, because otherwise we’re almost certainly going to have wires down in someone’s back yard again, likely another pole fire, and someone could get seriously hurt or property could be damaged.  If that happens, all this data I’ve collected, all the information I’ve given PG&E, all the times I’ve warned them that this could be a serious problem, becomes somebody’s Plaintiff’s Exhibit A.

I AM IN RADIO CONTACT

W1EMI at the ARRL suggested that if I had a handheld Yagi and an HT that can demodulate AM on UHF, it could be very helpful to do some direction-finding and narrow it down to which pole is causing the trouble.  I didn’t have a Yagi before, but I have one now.

After assuring my neighbors that I’m not from space or anything, I walked around trying to pinpoint the noise tonight.  The noise itself was not very cooperative, being somewhat intermittent in the gusty winds, but I was able to narrow it down to two potential poles, and neither one was the one I originally thought it was.  I suspect this is why switching to UHF can really help.

Unfortunately both poles are in other people’s back yards (and just a few houses away from ours) and the main lobe of the antenna is just wide enough that with an intermittent sound, I can’t quite discern for certain which pole it is.  I suspect with a more constant sound I would be able to tell a little bit better, and a constant buzz might be enough to heat up an insulator so it shows up on a thermal image.

Noteworthy is the fact that both of the suspect poles are part of the same run of wire on which there have been two catastrophic insulator failures in the last three years, both of which resulted in powerlines down in people’s back yards and small pole fires.  Thus it would be irresponsible of me to let this go and not make PG&E fix it.

If you follow this blog, you are probably aware already of the difficulties I’m having operating on HF due to powerline noise apparently being generated by PG&E in my neighborhood, but I thought it would be a useful exercise to summarize the story so far in one place.  In this way, it is easy to see what has already been done, what’s been tried, and so-on, to hopefully save some time in chasing down and resolving the actual problem.

Here’s what is known about the offending signal so far:

• It’s precisely 120 pulses of noise per second
• These pulses line up naturally with the peaks of a 60 Hz sine wave, such as that employed by electrical utilities
• It’s a wideband mess, the kind you would expect from an accidental spark-gap transmitter.
• Quite prominent on the 80m band,  at peak intensity, it blankets everything from 1.5 to at least 54 MHz, which includes portions of the AM broadcast band, WWV on 2.5, 5, 10, 15, and 20 MHz, and even part of low-VHF OTA TV
• It varies significantly in intensity, though it seems generally to be worse in the late afternoon; however, it can start at any time of day
• It vaguely seems to be influenced by environmental conditions (like temperature, humidity, solar intensity, wind), but cannot (as yet) be entirely predicted
• Strong wind gusts seem to modulate the amplitude of the noise and sometimes the noise starts when the wind picks up in the afternoon; based on this, I feel fairly confident the noise source is exposed to the elements
• It is a local phenomenon whose intensity is strongest in one particular intersection
• It’s relatively weak and can only be received for about 180 meters in any direction
• It is not coming from my own house
• It does not appear to be coming from the PG&E substation about half a mile north of our house, nor the high tension powerlines about a quarter mile to the east

And here are things I’ve done while trying to chase it down:

• Turned off electrical power to my own house to ensure that’s not where the noise originates
• Put a portable radio and antenna in the car and driven away from the sound to verify whether it’s local or atmospheric (it’s local)
• Driven around the neighborhood with a portable radio, one block at a time, looking for the area where the noise is strongest
• In the general area indicated by the car trip, walked the area with the same portable radio and found the intersection with the strongest noise
• From this intersection, walked slowly away from the area of greatest intensity and observed a general reduction in noise with increasing distance
• Informed PG&E on March 12 that there appears to be a problem, and which pair of utility poles are present at the intersection where the noise originates
• Taken a thermal photo of one of the suspect utility poles and observed that possibly one insulator is warmer than the others at the top, but this is inconclusive
• Wrote software to sample audio from my radio tuned to 3.500 MHz LSB, analyzing the intensity of 120pps pulse noise relative to the noise floor, and automatically posting charts to this site
• Submitted informal complaints to the FCC and CPUC regarding PG&E’s persistent inaction and stonewalling
• Contacted the RFI email at ARRL

PG&E has done:

• After receiving my CPUC complaint about my ticket being ignored for a month, sent one general tech without any specialized equipment to investigate, who predictably was unable to do anything about the problem other than take some notes from me about the nature of the trouble
• Nothing else

I’ve attempted to follow up with PG&E on the following dates: March 20, March 28, April 5, April 11, April 25.  I’ve advised service folk to whom I’ve spoken that this kind of RFI could be a leading indicator that some piece of equipment is approaching a failure point, and this has fallen on deaf ears.  The culture at PG&E appears to be one of “wait and do nothing until it completely fails” which kind of explains how we ended up with so many wildfires caused by PG&E equipment in California.

At this point, I believe I’ve exhausted everything that I can do.  My data collection and charting will continue, looking for some kind of pattern or predictability that might aid PG&E technicians in tracking down the trouble, if only they’d make any serious attempt to do so at all.

As part of my effort to help to track down and, ultimately, have the powerline noise in my neighborhood eliminated, I felt it was necessary to collect a lot more data about the problem, especially to help predict when the noise would be most audible so PG&E technicians would stand a better chance of being able to find the source of the problem.  (To date, however, they’ve done exactly nothing.)

To that end, I did what I do best: I wrote some code. It turns out to be straightforward in Python to sample audio off a sound card, produce graphs, and upload them with sftp to a web server.  The problem of pulse detection was a fun one.  My initial approach was to use Fast Fourier Transforms (from the NumPy module) in an effort to look for a 120Hz signal and its harmonics.  This approach, however, was ill-suited to the problem. The noise PG&E is making comes in very brief pulses, lasting only 3-5ms, maybe a total of 24-48 samples from beginning to end at 16000 samples-per-second.  It turns out this isn’t great for analysis with FFT.

I decided to take a different approach and run my analysis in the time domain instead of the frequency domain.  Because what I am looking for is a very predictable and highly consistent exactly 120 pulses per second, it’s possible to create an array of coefficients with 1’s spaced exactly 1/120s apart and 0’s everywhere else.  Then this array can be “slid” along the sample array, multiplying the arrays together and summing them for each point.  Where the sum is the largest, the alignment is optimized for a repeating pulse at 120pps (because that’s where the most 1s lined up perfectly with the pulses).  I also found this method produced even better results with multiple 1’s in sequence; I therefore use a sequence of three 1’s.

Once the best alignment with 120pps can be found, it’s a simple matter to backtrack to the beginning of the sampled audio and run the same analysis again from that point to arrive at a raw sum of the sample amplitude.  Finding the noise floor is a similar operation.  I take a little bit of a shortcut here and analyze the audio exactly midway between the 120pps peaks.  This is an approximation, of course, and it has one significant limitation: the pulses have such a high relative amplitude that they de-sense the receiver to a small extent.  This has the effect of making a pulse with a high amplitude cause the noise floor to appear to be somewhat lower than it really is.  Another limitation is that a high noise floor can cause the amplitude of the 120pps signal to appear higher; I believe this is because it isn’t truly possible to completely discern a signal from noise; at every point along the time domain input, what we’re actually seeing is a sum of the signal and the noise.  To put it another way, the noise doesn’t stop being there just because there’s a signal present.  So the “signal” level is really a sum of the signal and the noise floor.

Luckily, for the most part, the noise floor is relatively steady and changes slowly over time.  That is, until there’s a solar flare:

Something notable happened to the noise floor just before 6 AM Pacific on April 22 and caused quite a ruckus for the remainder of the day. I happened to notice on Mastodon from the Solar Flare Alert bot reports of significant solar flares right around the time the noise floor went crazy.  It’s a fascinating result that I wasn’t expecting when I started on this project.

In this graph you can also see how the signal and noise levels affect each other to some extent, but the key important thing is noting when the signal (the 120pps noise) level exceeds the arbitrary threshold I chose, -48dB.  Just by observation, this seems to be about the level where the 120pps noise is significant enough to render the 80m band useless, and it also seems to be about the midpoint between “on” and “off”, which stands out a little bit better in this graph:

The symmetry in question stands out reasonably well in this chart, especially between about 5:30 AM and 8 AM.  One thing I’m hoping to find in the next several days or weeks of data is some pattern to when the noise is most likely to be present, in the hope that I can give PG&E’s technicians some idea of when they should come out looking for a problem.

The data is sampled once a minute, and the graph and the signal, noise, and SNR data along with some current weather conditions are recorded in a CSV file; the plot is regenerated, and both are immediately (re)uploaded to the n6ol.us server.  Then it’s a simple matter of updating the filename once a day to build up a complete set of charts and recorded data segmented by day.  You can view the complete set of files at https://n6ol.us/noise/

Remaining is the challenge of getting PG&E to actually do something about the problem.  So far they’ve been ignoring it as hard as they can or denying that it’s even possible for them to generate RFI that affects shortwave radio (which is, of course, nonsense).

I finally had a moment to pull up the waveform of The Buzz– PG&E’s faulty equipment producing radio frequency interference (RFI) in my neighborhood– and overlay a 60Hz sine wave on top of it:

As you can see, the bursts of noise align perfectly with a 60Hz sine wave.  It would be astonishing if this were not powerline noise.

I also found on the ARRL’s examples of utility noise page, under the heading “w4tdk-13690 Powerline Noise” is a noise that sounds very similar to mine.  From their description:

Torbjorn received excellent service from his local power utility. Power was turned off for several blocks around his QTH, and the noise was eventually isolated to a ground wire on pole-mounted lightning arrestors that had burned through in two places and were just barely making contact. The wire was repaired and the noise disappeared. The customer (W4TDK) is happy, and the power company fixes a potentially dangerous situation. Everyone wins!

Unfortunately, I have not received excellent service from my local power utility, PG&E.  They continue to ignore the problem, and just this morning they had a catastrophic failure of some piece of equipment (due to neglect, no doubt) on the other side of our neighborhood that cut power to about 900 customers.  One notable difference with W4TDK’s recording is that his noise comes in 60 pulses per second rather than 120; it’s only pulsing on one side of the sine wave; perhaps having a wire that’s just barely making contact was behaving a bit like a diode, sparking with current flowing in one direction, but not the other?

In my case, I suspect the problem is a faulty insulator, and I might have even located which one it is with my thermal camera.  I’m fairly sure my neighbors think I’m nuts, but I snapped this image last night while it was fairly cold outside, and I’d set the low end of the range of my FLIR camera high enough that most objects did not show up at all in infrared, except for a little heat given off by warm patches of road, houses, cars, trees, and this utility pole:

Note that this utility pole has a number of insulators on both the crossbar on the top and the middle.  You can’t see most of them because they’re just too cold.  So cold, in fact, that you can see a dark shadow cast by the two on the top right if you look carefully.  But on the top left, at the end of the green arrow, one insulator is just warm enough to be visible.

Now it’s certainly inconclusive, and in no way is this proof of a faulty insulator, let alone the one that seems to be warm.  There are many reasons this particular insulator might be showing up while the others do not; it could be something else that happens to be reflecting off its surface from the position where I happened to be standing, or it could be that had I stood in a different position, a different insulator would have been visible, just an accident of optics and positioning.  It’s not impossible, though, that this insulator is faulty, and that it’s warm because there’s a current arcing across it 120 times per second for 1.5ms – 3.75ms each time (I don’t know why sometimes there’s just one quick pop, and sometimes a double pop which you can see in the waveform, other than perhaps just random chance).

Another observation is that ambient temperature and humidity may be a factor, possibly temperature more than humidity.  Above about 60 degrees Fahrenheit, The Buzz is much less prominent than it is at 60 degrees and below, but I do not have enough data to say this is more than a coincidence; it could as easily be the case that it’s a function of how much electricity is being consumed in my neighborhood, and people consume more when it’s colder outside.  Or it might not be a real pattern at all.

At any rate, the saga continues, and in a few days I’ll make my weekly call to PG&E to check on the status of my open ticket.  As of last week, nobody had even looked at it, 24 days after it was opened…