N6OL – November Six Oscar Lima

My Letter to PG&E’s CEO

January 17, 2025 / n6ol

It’s now been 10 months since I originally opened a ticket with PG&E regarding their RFI in my neighborhood, and though you’d never guess it by looking at Southern California right now, wildfire season is no longer a viable excuse for their inaction. Moreover, this winter storm season has been tragically very mild, with almost no rain to speak of in January and none on the horizon, so winter storms are not an excuse for inaction either.

My contact within PG&E is ghosting me. I haven’t heard back from the FCC or the ARRL in months either. So I decided to drop a letter in the postal mail to PG&E’s CEO outlining succinctly what the problem is, why it’s a problem, and what they really need to do about it. Here it is for your reading pleasure:

Ms Poppe:

Since March 12, 2024, I have been trying to get PG&E to resolve Radio Frequency Interference (RFI, also known as RTVI) that PG&E equipment is producing in the vicinity of my Amateur Radio station, beginning with the still-open case #5——–0.

PG&E does not appear to have a process for resolving this kind of problem. For example, it took until May 14, 2024 to resolve the closest source of interference to my station after the case was first ignored completely until May 9. Unfortunately, once that interference source was cleared, it came to light that there are several additional sources of interference nearby.

I have, on my own time and at my own expense, located these interference sources myself with my own equipment, and I’ve written monitoring software so your technicians can easily tell when the intermittent interference is active. Sadly, despite my efforts, nothing has been done.

PG&E is obligated to resolve RFI issues arising from its equipment under FCC rules under Title 47, CFR sections 15.5, 15.13, and 15.15.

Additionally, it is in PG&E’s best interest to locate and resolve these problems, as they can be an early indicator of an unsafe condition, such as current arcing across an insulator.

Current that is producing RFI is current that is not being delivered to customers and current that could be generating heat or sparks and thus may be a potential fire risk. Ignoring RFI is unwise.

I would like PG&E to meet its obligations both to safety and under federal law and fix these interference sources. Further, I strongly suggest that PG&E develop and implement consistent processes for this type of problem to ensure straightforward, timely resolution in the future.

I don’t think I’m being unreasonable, and I think 10 months is plenty of time to track down a handful of RFI sources and get them fixed, especially as I have already done all the legwork of figuring out (mostly) exactly where they are.

PG&E Serendipity

August 1, 2024 / n6ol

A little over two weeks ago, I fractured my humerus in a cycling accident, and since that time I’ve been doing a lot of walking to try to minimize how much weight I put on while I’m stuck in a sling and unable to run, bike, or swim. So it happened on Tuesday after I got my sutures removed downtown that I decided to walk home (about 1.5 miles).

As I was nearing home, I heard a familiar sound: the sound of arcing 60Hz power. I looked around, and sure enough, I was standing underneath a PG&E power pole. It was arcing so loudly I could hear it over the traffic of the nearby 101 freeway.

Because arcing is a dangerous condition, I noted the address and pole number, and called it in to PG&E’s emergency line. They promised to send someone out to check on it. If indeed they did send anyone out, they did nothing. I went back over to the same pole yesterday, this time armed with my handheld Yagi and my HT, and confirmed both that the pole is still arcing, and that the arcing is definitely causing RFI.

I called it in to my PG&E “customer relations” specialist, who told me that the department responsible for investigating this kind of thing has been working on changing their processes so they don’t just ignore reports of arcing and RFI for months, but rather do something about them instead. We’ll see. I expect this pole to still be arcing two months from now, as is their usual custom.

It has now been 142 days since my initial reports of really bad RFI in my neighborhood to PG&E, and though they have fixed a few of the problematic poles, they still haven’t fixed them all. When you hear PG&E talking a big game about all the safety inspections they’re performing, know that it’s performative BS. If they really cared about powerline safety, they wouldn’t wait two months to investigate any report of RFI or power pole arcing.

Some PG&E Progress, though not enough yet

July 16, 2024 / n6ol

Three days after they decided to send someone out again (finally), I got a call from PG&E’s RTVI hunter guy that he’d found a couple spots in the neighborhood, but unfortunately he called while I was in a meeting, so he got my voicemail. He was calling to check to see if the locations he found aligned with where I had picked up the noise myself, and strangely enough, no, they did not!

What this means is he found two more locations that I didn’t even know about.

The following night the noise was unusually bad and choppy, so I decided to go out for a walk with my Yagi and my HT; this time I found yet another location, unfortunately behind someone’s house, but where the same type of sagging insulator was used that has been implicated in several other noisy poles.

And tonight, I decided to see if I could track down what was making the occasional snapping, crackling, or popping sound, and no it wasn’t a bowl of Rice Crispies. This turned out to be another location entirely, and what’s particularly disturbing about this location is that it’s off the levee and completely surrounded by dry vegetation and people’s homes. And once again, the same type of sagging insulators are implicated, this time on two poles which are connected.

Both of these two new locations only cause trouble when the wind is blowing; there seems to be some kind of fault that is triggered by the insulators moving slightly in the wind, which I could see from the ground. I have of course relayed all of this information to PG&E. Maybe some day they’ll act on it.

This brings the grand total of problematic utility poles in my neighborhood up to 6 now, including the original one that was close by, and that’s just the ones we know about. Two utility poles situated over a great deal of dry vegetation

PG&E’s Resistance to Fixing QRM is Getting Old

July 7, 2024 / n6ol

It should come as no surprise to anyone that after insisting I open a new case for the secondary source of QRM from PG&E, nothing further has been done. I opened that case, as requested, back on May 16, and the QRM continues to follow a fairly predictable pattern, so much so that I can say with pretty high confidence that someone will hear it around 8 AM, 2 PM, and 8 PM most every day, more when it’s hot and dry.

A few things have happened though. Last week PG&E finally responded to the CPUC regarding my original complaint that they were dragging their heels and not doing their job to ensure their equipment was maintained properly and safely. I know it’ll come as a tremendous shock that PG&E misrepresented or omitted key facts in their response to the CPUC. Naturally I couldn’t allow a factual misrepresentation in their response stand, so I took advantage of the CPUC’s “one and only one opportunity” (how they put it) to set the record straight.

(You read that right: the CPUC will only allow you to respond once after the utility sends their response, and they warn you that unless you provide them with some new and compelling evidence, they’ll kick you to the curb. They’re not even vaguely nice about it.)

I corrected the record thusly:

My initial trouble report was made to PG&E on March 12.

PG&E assigned someone to work on this case only AFTER my informal complaint to the CPUC was forwarded to them on April 12. Until that point, they had completely ignored my trouble report. My first contact from PG&E regarding my case was on April 16, and it still took them almost 30 days, until May 9, to assign someone to look into a safety issue that could have started a fire. “Complain to the CPUC just to get PG&E’s attention” is not a scaleable process.

Upon assigning telecom techs to the issue, PG&E found exactly what I said they would find, exactly where I said they would find it: high-voltage power was arcing across a failing insulator a short distance from our house.

The problem was severe enough that their techs told me they could actually hear the arcing when standing below the pole. This is rather different from PG&E’s initial characterization that there was nothing wrong.

On May 14, PG&E replaced failing insulators on the pole in question. There is no dispute that PG&E equipment WAS at fault. It took them more than 60 days from my initial problem report to get to this point.

However, a problem is still unresolved and threatens safety: there remains a secondary source of interference of exactly the same type, apparently originating within the Poplar substation (based on radio direction-finding I’ve undertaken on my own time and at my own expense).

PG&E insisted that I open a NEW case for the secondary source of interference, which persists in an unresolved state to this day. The new case number is [redacted], opened on May 16. It has not been addressed at all.

Given the interference is of exactly the same type, it is reasonable to suspect the cause is likely similar: high voltage electrical arcing, the type which can start fires, and which has started fires in the past.

There remains a threat to public safety any time PG&E’s infrastructure is experiencing electrical arcing. PG&E should not be ignoring these early warning signs.

The CPUC took about a week to respond, and when they finally did, they seemed to accept my “new evidence” and forwarded it on to PG&E.

The very next day someone from PG&E actually showed up at my door to ask about the issue; I explained that it’s been an ongoing problem for well over 100 days now (112 days when he turned up), that it’s a 120 pulse-per-second interference that appears to be originating at the nearby substation, and that it is possible to discern it now that the nearby pole has been fixed. He indicated that he’d arrange to have the telecom guys come out again and hunt for the source of interference.

We’re now up to 117 days since my original case was opened, and 52 days since my most recent case was opened regarding the substation QRM source.

To an outside observer, it appears that PG&E’s policy is to ignore these problems entirely for at least 2 months before taking any action, and I have no idea if they’d have ever taken action at all if I hadn’t opened a complaint with the CPUC. Even after a CPUC complaint, it can still take 3 weeks before someone is finally scheduled to come out and hunt for the arcing. These data fly in the face of PG&E’s ongoing PR effort in which they want to appear as though they place a high priority on safety.

If they really cared about safety, and someone was telling them they had equipment that was arcing near a field full of dry grass, you’d sure think they’d put a priority on finding the source and fixing it as quickly as possible.

Fixing problems costs money, though, and things that cost money cut into profits. For-profit corporations will never do the right thing voluntarily if the right thing costs them money. They can only be forced to do so by a regulatory body.

Buzz-B-Gone!

May 16, 2024 / n6ol

Last week, PG&E finally sent a team of RFI hunters out, who agreed with my assessment regarding which pole seemed to be the trouble, and this week they had a crew out to replace the high voltage section of the pole:

The top of a utility pole featuring antiquated, sagging insulators.

The “before” picture- now it’s obvious how the insulators were sagging.

The offending crossbar and insulators removed by a crew.

Placing the new crossbar and insulators.

Job done!

The “after” picture; note how much further away from the pole these insulators keep the wiring

The RFI crew, once PG&E management finally relented in sending them, was very friendly, personable, and professional. They did a great job in confirming my suspicions and told me one of them was even able to hear arcing coming from that pole while standing underneath it. They reported back their findings, and a crew arrived this Tuesday to actually rebuild the top of the pole. This crew was fascinating to watch- so efficient, and they did such a nice, clean job of reworking that pole.

Here we can visually see the buzzing come to an end when power was shut down and the top of the pole was replaced:

Just after 9 AM, it’s gone, hopefully for good, and with also a reduced risk of fire, damage to property, or someone getting hurt. Unfortunately, this exercise also revealed a secondary source of interference, which is further away and picked up in the early afternoon. This one is proving more difficult to track down; from the bearings I’ve been able to get so far, it appears to originate inside the PG&E Poplar Creek substation. It’s also more intermittent, so I suspect it’s going to be more tricky to trace:

Data collected so far would seem to indicate that this is more of a mid-morning and early evening noise. Because it’s further away, it’s a little more difficult to get a fix on its location, but I was able to get a collection of points yesterday evening:

The area immediately around the Poplar substation is inaccessible to the public, so the best I could do was take a few readings from the closest I could get on public property. Note that there’s considerable uncertainty around the real origin of the noise; my antenna is directional, but the main lobe is still up to 30 degrees wide, and my headings are approximate, provided by a tool on my phone and an approximation of the direction my antenna was facing- they could all be off by a few degrees. Thus the real noise source could plausibly be anywhere in a 20 meter radius around these intersection points.

Unfortunately, PG&E wants me to start the entire process over again for the secondary source, so I’ll have to close out the original case, and open a new one now, which means probably another 60 days of dealing with this secondary noise before they send someone out again. At least being further away and less persistent, it’s somewhat less bothersome.

It’s kind of astounding how this utility doesn’t monitor these things on its own and drags its feet so aggressively in dealing with them when they’re reported. Once they actually relented in getting a crew out to investigate, the first problem was identified and resolved very quickly. It would stand to reason that they might want to proactively look for things arcing inside a substation…

Vindicated!

May 9, 2024 / n6ol

After getting connected with their comm techs a couple days ago, PG&E finally came out today (58 days after I opened my original ticket). And to give them credit, their crew was very knowledgeable and personable once I was finally able to get someone out here.

They managed to get access to the suspect pole, and just came back by after their noise hunting expedition to tell me they agree with me- it is that pole. One of the guys could even hear it arcing while he was standing underneath it.

They’re going to have a crew come out and “rebuild” what’s on that pole. No ETA, but at least we’re on a path to resolution.

This effort plausibly prevented a fire, property damage, or someone getting hurt. Also, I am not crazy! (Or at least not about RFI.)

I Have Located The Buzz

May 4, 2024 / n6ol

I expected that this morning, when the humidity dropped and the sun came out, the buzz would start up again, and it did. This afforded me an opportunity to go out and try my direction-finding skills again with the Yagi and the HT.

This time, determining which pole it was with certainty was easy.

I set out to try both aiming the main lobe of the Yagi at a pole and then turning it 45 degrees to the null in the antenna pattern. (I drew the 45 degree angles on the beam with a sharpie.) The nulls in the pattern are about half as wide as the main lobe, meaning I could get better selectivity by looking for where the signal went away rather than where it was.

This turned out to be successful. As it happened, there was a location on the sidewalk where turning 45 degrees to the left would put one pole in a null while pointing directly at the other pole. And on my first try, I quickly determined which pole it was; it dropped into the null and that was it… nothing came even off the main lobe from the other pole.

From where I was standing, I had to point the antenna up some to pick up the buzz, so I was pretty sure it was coming from the pole in front of me and not one further away but in the same direction, but nonetheless, I walked around to the next block over just to make sure, and indeed, I had to turn 180 degrees to face where I had been standing to keep picking up the buzz.

I then called PG&E and had them add to the ticket that I’ve had open for the last 52 days the precise location of the pole from which the noise emanates. I have now done more than half the work for them: I can tell them the conditions under which they’re likely to be causing interference, the time of day it’s most likely to be present (4 PM), and show them charts of when it has started and stopped and varied throughout the day.

The culprit pole is indeed along the powerline run that also runs behind our house which has already had multiple insulators replaced because of catastrophic failure. I’m not letting PG&E off the hook until they fix this, because otherwise we’re almost certainly going to have wires down in someone’s back yard again, likely another pole fire, and someone could get seriously hurt or property could be damaged. If that happens, all this data I’ve collected, all the information I’ve given PG&E, all the times I’ve warned them that this could be a serious problem, becomes somebody’s Plaintiff’s Exhibit A.

Direction Finding The Buzz

May 3, 2024 / n6ol

A tired looking guy holds up a Yagi antenna

I AM IN RADIO CONTACT

W1EMI at the ARRL suggested that if I had a handheld Yagi and an HT that can demodulate AM on UHF, it could be very helpful to do some direction-finding and narrow it down to which pole is causing the trouble. I didn’t have a Yagi before, but I have one now.

After assuring my neighbors that I’m not from space or anything, I walked around trying to pinpoint the noise tonight. The noise itself was not very cooperative, being somewhat intermittent in the gusty winds, but I was able to narrow it down to two potential poles, and neither one was the one I originally thought it was. I suspect this is why switching to UHF can really help.

Unfortunately both poles are in other people’s back yards (and just a few houses away from ours) and the main lobe of the antenna is just wide enough that with an intermittent sound, I can’t quite discern for certain which pole it is. I suspect with a more constant sound I would be able to tell a little bit better, and a constant buzz might be enough to heat up an insulator so it shows up on a thermal image.

Noteworthy is the fact that both of the suspect poles are part of the same run of wire on which there have been two catastrophic insulator failures in the last three years, both of which resulted in powerlines down in people’s back yards and small pole fires. Thus it would be irresponsible of me to let this go and not make PG&E fix it.

PG&E Powerline Noise: The Story So Far

April 28, 2024 / n6ol

If you follow this blog, you are probably aware already of the difficulties I’m having operating on HF due to powerline noise apparently being generated by PG&E in my neighborhood, but I thought it would be a useful exercise to summarize the story so far in one place. In this way, it is easy to see what has already been done, what’s been tried, and so-on, to hopefully save some time in chasing down and resolving the actual problem.

Here’s what is known about the offending signal so far:

It’s precisely 120 pulses of noise per second
These pulses line up naturally with the peaks of a 60 Hz sine wave, such as that employed by electrical utilities
It’s a wideband mess, the kind you would expect from an accidental spark-gap transmitter.
Quite prominent on the 80m band, at peak intensity, it blankets everything from 1.5 to at least 54 MHz, which includes portions of the AM broadcast band, WWV on 2.5, 5, 10, 15, and 20 MHz, and even part of low-VHF OTA TV
It varies significantly in intensity, though it seems generally to be worse in the late afternoon; however, it can start at any time of day
It vaguely seems to be influenced by environmental conditions (like temperature, humidity, solar intensity, wind), but cannot (as yet) be entirely predicted
Strong wind gusts seem to modulate the amplitude of the noise and sometimes the noise starts when the wind picks up in the afternoon; based on this, I feel fairly confident the noise source is exposed to the elements
It is a local phenomenon whose intensity is strongest in one particular intersection
It’s relatively weak and can only be received for about 180 meters in any direction
It is not coming from my own house
It does not appear to be coming from the PG&E substation about half a mile north of our house, nor the high tension powerlines about a quarter mile to the east

And here are things I’ve done while trying to chase it down:

Turned off electrical power to my own house to ensure that’s not where the noise originates
Put a portable radio and antenna in the car and driven away from the sound to verify whether it’s local or atmospheric (it’s local)
Driven around the neighborhood with a portable radio, one block at a time, looking for the area where the noise is strongest
In the general area indicated by the car trip, walked the area with the same portable radio and found the intersection with the strongest noise
From this intersection, walked slowly away from the area of greatest intensity and observed a general reduction in noise with increasing distance
Informed PG&E on March 12 that there appears to be a problem, and which pair of utility poles are present at the intersection where the noise originates
Taken a thermal photo of one of the suspect utility poles and observed that possibly one insulator is warmer than the others at the top, but this is inconclusive
Wrote software to sample audio from my radio tuned to 3.500 MHz LSB, analyzing the intensity of 120pps pulse noise relative to the noise floor, and automatically posting charts to this site
Submitted informal complaints to the FCC and CPUC regarding PG&E’s persistent inaction and stonewalling
Contacted the RFI email at ARRL

PG&E has done:

After receiving my CPUC complaint about my ticket being ignored for a month, sent one general tech without any specialized equipment to investigate, who predictably was unable to do anything about the problem other than take some notes from me about the nature of the trouble
Nothing else

I’ve attempted to follow up with PG&E on the following dates: March 20, March 28, April 5, April 11, April 25. I’ve advised service folk to whom I’ve spoken that this kind of RFI could be a leading indicator that some piece of equipment is approaching a failure point, and this has fallen on deaf ears. The culture at PG&E appears to be one of “wait and do nothing until it completely fails” which kind of explains how we ended up with so many wildfires caused by PG&E equipment in California.

At this point, I believe I’ve exhausted everything that I can do. My data collection and charting will continue, looking for some kind of pattern or predictability that might aid PG&E technicians in tracking down the trouble, if only they’d make any serious attempt to do so at all.

Graphing Powerline Noise

April 27, 2024 / n6ol

As part of my effort to help to track down and, ultimately, have the powerline noise in my neighborhood eliminated, I felt it was necessary to collect a lot more data about the problem, especially to help predict when the noise would be most audible so PG&E technicians would stand a better chance of being able to find the source of the problem. (To date, however, they’ve done exactly nothing.)

To that end, I did what I do best: I wrote some code. It turns out to be straightforward in Python to sample audio off a sound card, produce graphs, and upload them with sftp to a web server. The problem of pulse detection was a fun one. My initial approach was to use Fast Fourier Transforms (from the NumPy module) in an effort to look for a 120Hz signal and its harmonics. This approach, however, was ill-suited to the problem. The noise PG&E is making comes in very brief pulses, lasting only 3-5ms, maybe a total of 24-48 samples from beginning to end at 16000 samples-per-second. It turns out this isn’t great for analysis with FFT.

I decided to take a different approach and run my analysis in the time domain instead of the frequency domain. Because what I am looking for is a very predictable and highly consistent exactly 120 pulses per second, it’s possible to create an array of coefficients with 1’s spaced exactly 1/120s apart and 0’s everywhere else. Then this array can be “slid” along the sample array, multiplying the arrays together and summing them for each point. Where the sum is the largest, the alignment is optimized for a repeating pulse at 120pps (because that’s where the most 1s lined up perfectly with the pulses). I also found this method produced even better results with multiple 1’s in sequence; I therefore use a sequence of three 1’s.

Once the best alignment with 120pps can be found, it’s a simple matter to backtrack to the beginning of the sampled audio and run the same analysis again from that point to arrive at a raw sum of the sample amplitude. Finding the noise floor is a similar operation. I take a little bit of a shortcut here and analyze the audio exactly midway between the 120pps peaks. This is an approximation, of course, and it has one significant limitation: the pulses have such a high relative amplitude that they de-sense the receiver to a small extent. This has the effect of making a pulse with a high amplitude cause the noise floor to appear to be somewhat lower than it really is. Another limitation is that a high noise floor can cause the amplitude of the 120pps signal to appear higher; I believe this is because it isn’t truly possible to completely discern a signal from noise; at every point along the time domain input, what we’re actually seeing is a sum of the signal and the noise. To put it another way, the noise doesn’t stop being there just because there’s a signal present. So the “signal” level is really a sum of the signal and the noise floor.

Luckily, for the most part, the noise floor is relatively steady and changes slowly over time. That is, until there’s a solar flare:

Something notable happened to the noise floor just before 6 AM Pacific on April 22 and caused quite a ruckus for the remainder of the day. I happened to notice on Mastodon from the Solar Flare Alert bot reports of significant solar flares right around the time the noise floor went crazy. It’s a fascinating result that I wasn’t expecting when I started on this project.

In this graph you can also see how the signal and noise levels affect each other to some extent, but the key important thing is noting when the signal (the 120pps noise) level exceeds the arbitrary threshold I chose, -48dB. Just by observation, this seems to be about the level where the 120pps noise is significant enough to render the 80m band useless, and it also seems to be about the midpoint between “on” and “off”, which stands out a little bit better in this graph:

The symmetry in question stands out reasonably well in this chart, especially between about 5:30 AM and 8 AM. One thing I’m hoping to find in the next several days or weeks of data is some pattern to when the noise is most likely to be present, in the hope that I can give PG&E’s technicians some idea of when they should come out looking for a problem.

The data is sampled once a minute, and the graph and the signal, noise, and SNR data along with some current weather conditions are recorded in a CSV file; the plot is regenerated, and both are immediately (re)uploaded to the n6ol.us server. Then it’s a simple matter of updating the filename once a day to build up a complete set of charts and recorded data segmented by day. You can view the complete set of files at https://n6ol.us/noise/

Remaining is the challenge of getting PG&E to actually do something about the problem. So far they’ve been ignoring it as hard as they can or denying that it’s even possible for them to generate RFI that affects shortwave radio (which is, of course, nonsense).