A little over two weeks ago, I fractured my humerus in a cycling accident, and since that time I’ve been doing a lot of walking to try to minimize how much weight I put on while I’m stuck in a sling and unable to run, bike, or swim. So it happened on Tuesday after I got my sutures removed downtown that I decided to walk home (about 1.5 miles).
As I was nearing home, I heard a familiar sound: the sound of arcing 60Hz power. I looked around, and sure enough, I was standing underneath a PG&E power pole. It was arcing so loudly I could hear it over the traffic of the nearby 101 freeway.
Because arcing is a dangerous condition, I noted the address and pole number, and called it in to PG&E’s emergency line. They promised to send someone out to check on it. If indeed they did send anyone out, they did nothing. I went back over to the same pole yesterday, this time armed with my handheld Yagi and my HT, and confirmed both that the pole is still arcing, and that the arcing is definitely causing RFI.
I called it in to my PG&E “customer relations” specialist, who told me that the department responsible for investigating this kind of thing has been working on changing their processes so they don’t just ignore reports of arcing and RFI for months, but rather do something about them instead. We’ll see. I expect this pole to still be arcing two months from now, as is their usual custom.
It has now been 142 days since my initial reports of really bad RFI in my neighborhood to PG&E, and though they have fixed a few of the problematic poles, they still haven’t fixed them all. When you hear PG&E talking a big game about all the safety inspections they’re performing, know that it’s performative BS. If they really cared about powerline safety, they wouldn’t wait two months to investigate any report of RFI or power pole arcing.
Three days after they decided to send someone out again (finally), I got a call from PG&E’s RTVI hunter guy that he’d found a couple spots in the neighborhood, but unfortunately he called while I was in a meeting, so he got my voicemail. He was calling to check to see if the locations he found aligned with where I had picked up the noise myself, and strangely enough, no, they did not!
What this means is he found twomore locations that I didn’t even know about.
The following night the noise was unusually bad and choppy, so I decided to go out for a walk with my Yagi and my HT; this time I found yet another location, unfortunately behind someone’s house, but where the same type of sagging insulator was used that has been implicated in several other noisy poles.
And tonight, I decided to see if I could track down what was making the occasional snapping, crackling, or popping sound, and no it wasn’t a bowl of Rice Crispies. This turned out to be another location entirely, and what’s particularly disturbing about this location is that it’s off the levee and completely surrounded by dry vegetation and people’s homes. And once again, the same type of sagging insulators are implicated, this time on two poles which are connected.
Both of these two new locations only cause trouble when the wind is blowing; there seems to be some kind of fault that is triggered by the insulators moving slightly in the wind, which I could see from the ground. I have of course relayed all of this information to PG&E. Maybe some day they’ll act on it.
This brings the grand total of problematic utility poles in my neighborhood up to 6 now, including the original one that was close by, and that’s just the ones we know about.
It should come as no surprise to anyone that after insisting I open a new case for the secondary source of QRM from PG&E, nothing further has been done. I opened that case, as requested, back on May 16, and the QRM continues to follow a fairly predictable pattern, so much so that I can say with pretty high confidence that someone will hear it around 8 AM, 2 PM, and 8 PM most every day, more when it’s hot and dry.
A few things have happened though. Last week PG&E finally responded to the CPUC regarding my original complaint that they were dragging their heels and not doing their job to ensure their equipment was maintained properly and safely. I know it’ll come as a tremendous shock that PG&E misrepresented or omitted key facts in their response to the CPUC. Naturally I couldn’t allow a factual misrepresentation in their response stand, so I took advantage of the CPUC’s “one and only one opportunity” (how they put it) to set the record straight.
(You read that right: the CPUC will only allow you to respond once after the utility sends their response, and they warn you that unless you provide them with some new and compelling evidence, they’ll kick you to the curb. They’re not even vaguely nice about it.)
I corrected the record thusly:
My initial trouble report was made to PG&E on March12.
PG&E assigned someone to work on this case onlyAFTER my informal complaint to the CPUC was forwarded to them on April 12. Until that point, they had completely ignored my trouble report. My first contact from PG&E regarding my case was on April 16, and it still took them almost 30 days, until May 9, to assign someone to look into a safety issue that could have started a fire. “Complain to the CPUC just to get PG&E’s attention” is not a scaleable process.
Upon assigning telecom techs to the issue, PG&E found exactly what I said they would find, exactly where I said they would find it: high-voltage power was arcing across a failing insulator a short distance from our house.
The problem was severe enough that their techs told me they could actually hear the arcing when standing below the pole. This is rather different from PG&E’s initial characterization that there was nothing wrong.
On May 14, PG&E replaced failing insulators on the pole in question. There is no dispute that PG&E equipment WAS at fault. It took them more than 60 days from my initial problem report to get to this point.
However, a problem is still unresolved and threatens safety: there remains a secondary source of interference of exactly the same type, apparently originating within the Poplar substation (based on radio direction-finding I’ve undertaken on my own time and at my own expense).
PG&E insisted that I open a NEW case for the secondary source of interference, which persists in an unresolved state to this day. The new case number is [redacted], opened on May 16. It has not been addressed at all.
Given the interference is of exactly the same type, it is reasonable to suspect the cause is likely similar: high voltage electrical arcing, the type which can start fires, and which has started fires in the past.
There remains a threat to public safety any time PG&E’s infrastructure is experiencing electrical arcing. PG&E should not be ignoring these early warning signs.
The CPUC took about a week to respond, and when they finally did, they seemed to accept my “new evidence” and forwarded it on to PG&E.
The very next day someone from PG&E actually showed up at my door to ask about the issue; I explained that it’s been an ongoing problem for well over 100 days now (112 days when he turned up), that it’s a 120 pulse-per-second interference that appears to be originating at the nearby substation, and that it is possible to discern it now that the nearby pole has been fixed. He indicated that he’d arrange to have the telecom guys come out again and hunt for the source of interference.
We’re now up to 117 days since my original case was opened, and 52 days since my most recent case was opened regarding the substation QRM source.
To an outside observer, it appears that PG&E’s policy is to ignore these problems entirely for at least 2 months before taking any action, and I have no idea if they’d have ever taken action at all if I hadn’t opened a complaint with the CPUC. Even after a CPUC complaint, it can still take 3 weeks before someone is finally scheduled to come out and hunt for the arcing. These data fly in the face of PG&E’s ongoing PR effort in which they want to appear as though they place a high priority on safety.
If they really cared about safety, and someone was telling them they had equipment that was arcing near a field full of dry grass, you’d sure think they’d put a priority on finding the source and fixing it as quickly as possible.
Fixing problems costs money, though, and things that cost money cut into profits. For-profit corporations will never do the right thing voluntarily if the right thing costs them money. They can only be forced to do so by a regulatory body.
After getting connected with their comm techs a couple days ago, PG&E finally came out today (58 days after I opened my original ticket). And to give them credit, their crew was very knowledgeable and personable once I was finally able to get someone out here.
They managed to get access to the suspect pole, and just came back by after their noise hunting expedition to tell me they agree with me- it is that pole. One of the guys could even hear it arcing while he was standing underneath it.
They’re going to have a crew come out and “rebuild” what’s on that pole. No ETA, but at least we’re on a path to resolution.
This effort plausibly prevented a fire, property damage, or someone getting hurt. Also, I am not crazy! (Or at least not about RFI.)
I expected that this morning, when the humidity dropped and the sun came out, the buzz would start up again, and it did. This afforded me an opportunity to go out and try my direction-finding skills again with the Yagi and the HT.
This time, determining which pole it was with certainty was easy.
I set out to try both aiming the main lobe of the Yagi at a pole and then turning it 45 degrees to the null in the antenna pattern. (I drew the 45 degree angles on the beam with a sharpie.) The nulls in the pattern are about half as wide as the main lobe, meaning I could get better selectivity by looking for where the signal went away rather than where it was.
This turned out to be successful. As it happened, there was a location on the sidewalk where turning 45 degrees to the left would put one pole in a null while pointing directly at the other pole. And on my first try, I quickly determined which pole it was; it dropped into the null and that was it… nothing came even off the main lobe from the other pole.
From where I was standing, I had to point the antenna up some to pick up the buzz, so I was pretty sure it was coming from the pole in front of me and not one further away but in the same direction, but nonetheless, I walked around to the next block over just to make sure, and indeed, I had to turn 180 degrees to face where I had been standing to keep picking up the buzz.
I then called PG&E and had them add to the ticket that I’ve had open for the last 52 days the precise location of the pole from which the noise emanates. I have now done more than half the work for them: I can tell them the conditions under which they’re likely to be causing interference, the time of day it’s most likely to be present (4 PM), and show them charts of when it has started and stopped and varied throughout the day.
The culprit pole is indeed along the powerline run that also runs behind our house which has already had multiple insulators replaced because of catastrophic failure. I’m not letting PG&E off the hook until they fix this, because otherwise we’re almost certainly going to have wires down in someone’s back yard again, likely another pole fire, and someone could get seriously hurt or property could be damaged. If that happens, all this data I’ve collected, all the information I’ve given PG&E, all the times I’ve warned them that this could be a serious problem, becomes somebody’s Plaintiff’s Exhibit A.
W1EMI at the ARRL suggested that if I had a handheld Yagi and an HT that can demodulate AM on UHF, it could be very helpful to do some direction-finding and narrow it down to which pole is causing the trouble. I didn’t have a Yagi before, but I have one now.
After assuring my neighbors that I’m not from space or anything, I walked around trying to pinpoint the noise tonight. The noise itself was not very cooperative, being somewhat intermittent in the gusty winds, but I was able to narrow it down to two potential poles, and neither one was the one I originally thought it was. I suspect this is why switching to UHF can really help.
Unfortunately both poles are in other people’s back yards (and just a few houses away from ours) and the main lobe of the antenna is just wide enough that with an intermittent sound, I can’t quite discern for certain which pole it is. I suspect with a more constant sound I would be able to tell a little bit better, and a constant buzz might be enough to heat up an insulator so it shows up on a thermal image.
Noteworthy is the fact that both of the suspect poles are part of the same run of wire on which there have been two catastrophic insulator failures in the last three years, both of which resulted in powerlines down in people’s back yards and small pole fires. Thus it would be irresponsible of me to let this go and not make PG&E fix it.
If you follow this blog, you are probably aware already of the difficulties I’m having operating on HF due to powerline noise apparently being generated by PG&E in my neighborhood, but I thought it would be a useful exercise to summarize the story so far in one place. In this way, it is easy to see what has already been done, what’s been tried, and so-on, to hopefully save some time in chasing down and resolving the actual problem.
It’s a wideband mess, the kind you would expect from an accidental spark-gap transmitter.
Quite prominent on the 80m band, at peak intensity, it blankets everything from 1.5 to at least 54 MHz, which includes portions of the AM broadcast band, WWV on 2.5, 5, 10, 15, and 20 MHz, and even part of low-VHF OTA TV
It varies significantly in intensity, though it seems generally to be worse in the late afternoon; however, it can start at any time of day
It vaguely seems to be influenced by environmental conditions (like temperature, humidity, solar intensity, wind), but cannot (as yet) be entirely predicted
Strong wind gusts seem to modulate the amplitude of the noise and sometimes the noise starts when the wind picks up in the afternoon; based on this, I feel fairly confident the noise source is exposed to the elements
It’s relatively weak and can only be received for about 180 meters in any direction
It is not coming from my own house
It does not appear to be coming from the PG&E substation about half a mile north of our house, nor the high tension powerlines about a quarter mile to the east
And here are things I’ve done while trying to chase it down:
Turned off electrical power to my own house to ensure that’s not where the noise originates
Put a portable radio and antenna in the car and driven away from the sound to verify whether it’s local or atmospheric (it’s local)
Driven around the neighborhood with a portable radio, one block at a time, looking for the area where the noise is strongest
In the general area indicated by the car trip, walked the area with the same portable radio and found the intersection with the strongest noise
From this intersection, walked slowly away from the area of greatest intensity and observed a general reduction in noise with increasing distance
Informed PG&E on March 12 that there appears to be a problem, and which pair of utility poles are present at the intersection where the noise originates
Taken a thermal photo of one of the suspect utility poles and observed that possibly one insulator is warmer than the others at the top, but this is inconclusive
Wrote software to sample audio from my radio tuned to 3.500 MHz LSB, analyzing the intensity of 120pps pulse noise relative to the noise floor, and automatically posting charts to this site
Submitted informal complaints to the FCC and CPUC regarding PG&E’s persistent inaction and stonewalling
Contacted the RFI email at ARRL
PG&E has done:
After receiving my CPUC complaint about my ticket being ignored for a month, sent one general tech without any specialized equipment to investigate, who predictably was unable to do anything about the problem other than take some notes from me about the nature of the trouble
Nothing else
I’ve attempted to follow up with PG&E on the following dates: March 20, March 28, April 5, April 11, April 25. I’ve advised service folk to whom I’ve spoken that this kind of RFI could be a leading indicator that some piece of equipment is approaching a failure point, and this has fallen on deaf ears. The culture at PG&E appears to be one of “wait and do nothing until it completely fails” which kind of explains how we ended up with so many wildfires caused by PG&E equipment in California.
At this point, I believe I’ve exhausted everything that I can do. My data collection and charting will continue, looking for some kind of pattern or predictability that might aid PG&E technicians in tracking down the trouble, if only they’d make any serious attempt to do so at all.
As part of my effort to help to track down and, ultimately, have the powerline noise in my neighborhood eliminated, I felt it was necessary to collect a lot more data about the problem, especially to help predict when the noise would be most audible so PG&E technicians would stand a better chance of being able to find the source of the problem. (To date, however, they’ve done exactly nothing.)
To that end, I did what I do best: I wrote some code. It turns out to be straightforward in Python to sample audio off a sound card, produce graphs, and upload them with sftp to a web server. The problem of pulse detection was a fun one. My initial approach was to use Fast Fourier Transforms (from the NumPy module) in an effort to look for a 120Hz signal and its harmonics. This approach, however, was ill-suited to the problem. The noise PG&E is making comes in very brief pulses, lasting only 3-5ms, maybe a total of 24-48 samples from beginning to end at 16000 samples-per-second. It turns out this isn’t great for analysis with FFT.
I decided to take a different approach and run my analysis in the time domain instead of the frequency domain. Because what I am looking for is a very predictable and highly consistent exactly 120 pulses per second, it’s possible to create an array of coefficients with 1’s spaced exactly 1/120s apart and 0’s everywhere else. Then this array can be “slid” along the sample array, multiplying the arrays together and summing them for each point. Where the sum is the largest, the alignment is optimized for a repeating pulse at 120pps (because that’s where the most 1s lined up perfectly with the pulses). I also found this method produced even better results with multiple 1’s in sequence; I therefore use a sequence of three 1’s.
Once the best alignment with 120pps can be found, it’s a simple matter to backtrack to the beginning of the sampled audio and run the same analysis again from that point to arrive at a raw sum of the sample amplitude. Finding the noise floor is a similar operation. I take a little bit of a shortcut here and analyze the audio exactly midway between the 120pps peaks. This is an approximation, of course, and it has one significant limitation: the pulses have such a high relative amplitude that they de-sense the receiver to a small extent. This has the effect of making a pulse with a high amplitude cause the noise floor to appear to be somewhat lower than it really is. Another limitation is that a high noise floor can cause the amplitude of the 120pps signal to appear higher; I believe this is because it isn’t truly possible to completely discern a signal from noise; at every point along the time domain input, what we’re actually seeing is a sum of the signal and the noise. To put it another way, the noise doesn’t stop being there just because there’s a signal present. So the “signal” level is really a sum of the signal and the noise floor.
Luckily, for the most part, the noise floor is relatively steady and changes slowly over time. That is, until there’s a solar flare:
Something notable happened to the noise floor just before 6 AM Pacific on April 22 and caused quite a ruckus for the remainder of the day. I happened to notice on Mastodon from the Solar Flare Alert bot reports of significant solar flares right around the time the noise floor went crazy. It’s a fascinating result that I wasn’t expecting when I started on this project.
In this graph you can also see how the signal and noise levels affect each other to some extent, but the key important thing is noting when the signal (the 120pps noise) level exceeds the arbitrary threshold I chose, -48dB. Just by observation, this seems to be about the level where the 120pps noise is significant enough to render the 80m band useless, and it also seems to be about the midpoint between “on” and “off”, which stands out a little bit better in this graph:
The symmetry in question stands out reasonably well in this chart, especially between about 5:30 AM and 8 AM. One thing I’m hoping to find in the next several days or weeks of data is some pattern to when the noise is most likely to be present, in the hope that I can give PG&E’s technicians some idea of when they should come out looking for a problem.
The data is sampled once a minute, and the graph and the signal, noise, and SNR data along with some current weather conditions are recorded in a CSV file; the plot is regenerated, and both are immediately (re)uploaded to the n6ol.us server. Then it’s a simple matter of updating the filename once a day to build up a complete set of charts and recorded data segmented by day. You can view the complete set of files at https://n6ol.us/noise/
Remaining is the challenge of getting PG&E to actually do something about the problem. So far they’ve been ignoring it as hard as they can or denying that it’s even possible for them to generate RFI that affects shortwave radio (which is, of course, nonsense).
I finally had a moment to pull up the waveform of The Buzz– PG&E’s faulty equipment producing radio frequency interference (RFI) in my neighborhood– and overlay a 60Hz sine wave on top of it:
As you can see, the bursts of noise align perfectly with a 60Hz sine wave. It would be astonishing if this were not powerline noise.
I also found on the ARRL’s examples of utility noise page, under the heading “w4tdk-13690 Powerline Noise” is a noise that sounds very similar to mine. From their description:
Torbjorn received excellent service from his local power utility. Power was turned off for several blocks around his QTH, and the noise was eventually isolated to a ground wire on pole-mounted lightning arrestors that had burned through in two places and were just barely making contact. The wire was repaired and the noise disappeared. The customer (W4TDK) is happy, and the power company fixes a potentially dangerous situation. Everyone wins!
Unfortunately, I have not received excellent service from my local power utility, PG&E. They continue to ignore the problem, and just this morning they had a catastrophic failure of some piece of equipment (due to neglect, no doubt) on the other side of our neighborhood that cut power to about 900 customers. One notable difference with W4TDK’s recording is that his noise comes in 60 pulses per second rather than 120; it’s only pulsing on one side of the sine wave; perhaps having a wire that’s just barely making contact was behaving a bit like a diode, sparking with current flowing in one direction, but not the other?
In my case, I suspect the problem is a faulty insulator, and I might have even located which one it is with my thermal camera. I’m fairly sure my neighbors think I’m nuts, but I snapped this image last night while it was fairly cold outside, and I’d set the low end of the range of my FLIR camera high enough that most objects did not show up at all in infrared, except for a little heat given off by warm patches of road, houses, cars, trees, and this utility pole:
Note that this utility pole has a number of insulators on both the crossbar on the top and the middle. You can’t see most of them because they’re just too cold. So cold, in fact, that you can see a dark shadow cast by the two on the top right if you look carefully. But on the top left, at the end of the green arrow, one insulator is just warm enough to be visible.
Now it’s certainly inconclusive, and in no way is this proof of a faulty insulator, let alone the one that seems to be warm. There are many reasons this particular insulator might be showing up while the others do not; it could be something else that happens to be reflecting off its surface from the position where I happened to be standing, or it could be that had I stood in a different position, a different insulator would have been visible, just an accident of optics and positioning. It’s not impossible, though, that this insulator is faulty, and that it’s warm because there’s a current arcing across it 120 times per second for 1.5ms – 3.75ms each time (I don’t know why sometimes there’s just one quick pop, and sometimes a double pop which you can see in the waveform, other than perhaps just random chance).
Another observation is that ambient temperature and humidity may be a factor, possibly temperature more than humidity. Above about 60 degrees Fahrenheit, The Buzz is much less prominent than it is at 60 degrees and below, but I do not have enough data to say this is more than a coincidence; it could as easily be the case that it’s a function of how much electricity is being consumed in my neighborhood, and people consume more when it’s colder outside. Or it might not be a real pattern at all.
At any rate, the saga continues, and in a few days I’ll make my weekly call to PG&E to check on the status of my open ticket. As of last week, nobody had even looked at it, 24 days after it was opened…
I recently began dabbling in Ham Radio again after having my radios turned off for a few years; the endless prattle of right-wing nutjob conspiracy theorists moaning on every band had become tiresome, solar conditions weren’t great, and I just had a lot of other things going on. With the sun waking up again of late, and my husband insisting I spent some bonus money from work on something frivolous rather than something responsible, I got myself a shiny new Elecraft K4, a snazzy new antenna from HyPower Antennas, and got myself back on the air.
Things were going great for a few hours until The Buzz kicked in. The Buzz is, as best I can tell, some piece of failing PG&E equipment, probably an insulator or lightning arrestor, which has become an inadvertent spark-gap transmitter. It is somewhat intermittent, but almost always present in the late afternoon or early evening, and it generally persists well into the night. Once it starts, it nearly blanks out everything from 160m through 6m. It’s unclear what causes it to come and go, though there seems to be some correlation between moisture, temperature, and even wind.
Today after a few days of quite wet conditions followed by cold air at about 60% humidity, The Buzz was back with a vengeance. A new variant today is that we’re having strong, gusty winds, apparently adding some modulation to The Buzz. Here’s what it sounds like on 5MHz. I haven’t normalized the audio in this file; this is directly recorded off the K4. The signal strength was around -70dB:
You can hear some subtle interruptions in The Buzz. This is not coming from loose antenna connections. I suspect somewhere a powerline or bit of cabling is getting tossed around in the wind, and this in turn is introducing that modulation to the signal.
To prove it wasn’t just a bad connection on my antenna, and my antenna swinging around in the wind instead, I tuned to 15 MHz so I could try to listen to WWV along with The Buzz. In this recording, you can hear how the intermittent nature of The Buzz happens independently of the WWV signal; The Buzz comes and goes, but WWV stays more or less consistent throughout (though it can be a little hard to hear when The Buzz is in full force):
Here you can see exactly 12 pulses in 0.1 seconds, which equates to 120 pulses in 1 second.
Recall that the frequency of a sine wave- like that used in 60Hz utility power- refers to the time it takes to complete one full cycle, which includes both the positive and negative peaks. Thus, a 60 Hz wave while completing 60 cycles per second completes 120 peaks (and 120 zero crossings) per second. My suspicion is that PG&E has a faulty insulator somewhere and the arcing is happening across the insulator, between one phase and neutral or ground. Once the voltage reaches a high enough positive value, it begins to arc until the voltage comes down enough that the insulator is sufficient. Then the power crosses through 0 volts, and when it reaches a high enough (absolute) value, it again arcs until the (absolute) voltage comes back enough again. Then the process repeats. If we overlay a sine wave we can actually see what’s happening:
My alignment of my generated 60Hz sine wave to the recording isn’t perfect (Audacity on my ham radio PC is kind of a pain to work with), but you can see pretty effectively how each peak and trough of a 60Hz sine wave lines up with a burst of powerline noise. Given the highly predictable nature of the offending transmission, if we knew the RMS voltage of the source, we could probably even calculate the breakdown voltage at which the insulator is failing to prevent an arc.
Given the age of the infrastructure in my neighborhood — there are even still glass insulators on many of the utility poles here– it’s not exactly surprising that there would be some old equipment starting to break down. Indeed, in the two years prior to this year, we experienced lengthy power outages after insulators saturated with rain exploded, sending powerlines into trees and leaving crossbars smoldering.
With PG&E’s reputation of incinerating neighborhoods and even entire towns, you might think they’d be a little more interested than they are in taking preventative measures, especially when given the opportunity to fix a problem while it’s still just generating RFI but not yet fire. Sadly, this does not seem to be the case. I’ve contacted them (so far) on March 12, 20, 28, and April 5, for updates on my original case and to date, nobody has even looked at it, let alone gotten back to me. I am, of course, documenting everything, and after 60 days, I will make this an FCC RFI complaint, at which point I’m sure the FCC will remind PG&E once again that they are obligated to fix these problems, as they have had to do many times in the past.
