Tuesday, August 11, 2020

Go: Convert errors.Wrap calls to fmt.Errorf

 I was a longtime fan of https://github.com/pkg/errors. It was a great way to add context to why an error was being returned which made tracing them easier. The need for pkg/errors has gone away with the new fmt.Errorf %w directive, errors.Is(), and errors.As().

I used errors.Wrap() a lot so naturally my code has lots of function calls I need to migrate. One repo had close to 300 calls to errors.Wrap() which is more than I'm willing to do by hand. I wrote a simple tool to take care of the most common case I have: errors.Wrap(err, "<message>").

  • Any line it doesn't know how to handle it leaves unchanged.
  • It looks specifically for errors.Wrap(err, "
  • On that same line it expects to find a double quote followed by a closing paren
  • The existing context string has : %w appended to it
  • It does not edit your imports; you should run goimports or a similar tool
  • By default the tool just outputs to stdout; use -o to overwrite the file in-place
  • Fix everything by doing for i in $(grep -R errors.Wrap `ls`); do errors_wrap_convert -in $i -o; end
  • Definitely make sure you have a snapshot of your code to revert back to in case this tool does bad things
  • This could have been done better using gofix but I was in too much of a hurry to learn how to extend gofix.
# errors_wrap_convert.go
package main

import (
        "bufio"
        "bytes"
        "flag"
        "fmt"
        "io"
        "io/ioutil"
        "log"
        "os"
        "strings"
)

var (
        fIn        = flag.String("in", "", "input file")
        fOverwrite = flag.Bool("o", false, "overwrite the existing file")
)

func fatalIfError(err error, msg string) {
        if err != nil {
                log.Fatal("error ", msg, ": ", err)
        }
}

func main() {
        flag.Parse()
        b, err := ioutil.ReadFile(*fIn)
        fatalIfError(err, "reading input file")

        var out io.WriteCloser = os.Stdout
        if *fOverwrite {
                out, err = os.Create(*fIn)
                fatalIfError(err, "opening output file")
        }
        defer out.Close()

        scanner := bufio.NewScanner(bytes.NewBuffer(b))
        for scanner.Scan() {
                fmt.Fprintln(out, Rewrite(scanner.Text()))
        }
        fatalIfError(scanner.Err(), "scanner error")
}


func Rewrite(in string) string {
        idx := strings.Index(in, `errors.Wrap(err, "`)
        if idx == -1 {
                return in
        }

        eIdx := strings.Index(in[idx:], ")")
        if eIdx == -1 {
                return in
        }
        eIdx += idx

        q1Idx := strings.Index(in[idx:], `"`)
        if q1Idx == -1 {
                return in
        }
        q1Idx += idx

        q2Idx := eIdx - 1
        if in[q2Idx] != '"' {
                return in
        }

        out := in[:idx] +
                `fmt.Errorf(` +
                in[q1Idx:q2Idx] +
                `: %w", err)` +
                in[eIdx+1:]
        return out
}

And a couple of basic tests:

# errors_convert_test.go
package main

import "testing"

func TestRewrite(t *testing.T) {
        t.Parallel()
        for in, want := range map[string]string{
                "": "",
                `               return nil, errors.Wrap(err, "bad thing") // foo bar`: `                return nil, fmt.Errorf("bad thing: %w", err) // foo bar`,
                `return nil, errors.Wrap(err, "foo " + blarg + " bar")`: `return nil, fmt.Errorf("foo " + blarg + " bar: %w", err)`,
        } {
                got := Rewrite(in)
                if got != want {
                        t.Fatalf("got %q, want %q, for %q", got, want, in)
                }
        }
}

I had searched for a tool to do this but it either doesn't exist or my searching ability failed me. If you would like to pick this up and generalize I'd happily refer to your version as canonical.

Sunday, August 2, 2020

The Autobucket Saga

The Leaking A/C and Early Failure

A few years ago our air conditioning started leaking. We discovered this when a stream of water started running from the corner of our kitchen's ballast lighting. Naturally we were alarmed. We found the location of the leak and got a plastic tub underneath it to catch the water. Until we could get a technician to the house we got to choose why we weren't sleeping well; either because it was too hot with the A/C off or every couple of hours one of us had to empty the tub with a wet vac.

This problem annoyed me. I'm a smart, technical guy, I should be able to solve this. I had taught myself some electronics and should be able to programmatically control a pump. I bought a little 5v USB pump and some float switches. I hooked it all up to a raspberry pi with the pump's power being controlled by the pi via an NPN transistor. Pump turns on when the high-water mark switch closes. Pump turns off when the low-water mark switch opens. Super simple, and for the life of me I couldn't get it to actually work.

The Student Elevates Himself

Since then I've learned a lot more about electronics, though I'm still a newbie. This year at BSides San Diego I bought an arduino-compatible microcontroller board and some other components as a way to help fund the event. Since I bought them I had to experiment with them!

The basic stuff is pretty easy! In the course of fiddling and experimenting I realized the problem with my original setup; with neither the pump control transistor nor the float switches was I tying them to a ground reference (via resistor, of course).

I spoke about my new understanding and new confidence to my loving partner. She noted how the condensation from the A/C just gets pumped out down the side of the house. We catch it with a bucket but rarely think to dump that water on our orange trees. It should would be nice to have the water moved over there automatically! I was inspired.

The Autobucket Is Born

This came together on a breadboard pretty quickly.

Random USB wires to your gaming laptop is fine, right?

Each float switch goes to a GPIO pin on a raspi zero w with the other side having a 10k resistor to ground and a 1k resistor to the pi's 3v3. I elected to have discrete pulldown resistors rather than integrated pulldown/pullup purely because I understand it better. The pump was directly connected to USB 5v with ground going to the NPN's collector. The base has 10k to ground and 1k to a GPIO pin. I wrote all the software in go with gobot including a feature to notify me via Telegram as the pump cycles on and off.

Amazingly, it just worked! On my bench I could move the float switches and see the program change state. Rather than powering on a dry pump I plugged in a device that charges via USB with a charging indicator LED. In the appropriate states it would turn on and off. Awesome!

Next up, the bucket. We have a bunch of orange buckets so hacking one up wasn't an issue:

Water, wires: besties!

I installed a couple of holes at different heights and installed the float switches. With their rubber gaskets it didn't even leak! I dropped the pump in there and for the time being just accepted that it didn't sit fully at the bottom but sufficiently below the low-water mark.

I didn't want the control system exposed to rain and sunlight and I needed it to be near power. I have a covered patio nearby with AC outlets, I just needed to establish connectivity between the two points. I needed 5 wires: USB+, USB ground, 3v3+, float switch 1, float switch 2. I have a supply of CAT-5 which is great since it even provides easy to differentiate individual wires. I wanted to be able to disconnect it so two wires went to a female USB connector and three into a molex hard drive power connector from my parts box. With this in place I could connect and disconnect as needed. Once things were settled I could shrink-wrap the connections for some weather protection.

I'm playing outside!

Testing again with the circuit on the board and again, success! You may note in this photo that power is supplied by exposed USB wires and gator clips. This is fine for a day of testing but not a workable long-term solution.

I'd like a neater board but don't keep perfboard handy. I do, however, have a 3D printer, a decent ability with CAD, and general lack of good judgement.


Sorry about your gag reflex.

Since I'm already using CAT-5 to connect to the bucket, why not RJ45? I had some RJ45 keystone jacks so I super glued a couple to my board. One was intended to connect to the bucket, the other to go to the raspi. Instead I ended up connecting to the raspi via a female header connector snipped in half then glued together so that I could easily connect and disconnect to GPIO. While I was at it, I hooked a BME280 sensor via I2C so I'd have an outdoor temperature/pressure/humidity sensor that I can expose as a webserver.

For power I grabbed a phone cable I had with only two wires. Part of the way through the conversion I had something that at least made me chuckle:

Windows is configuring your new magic smoke

This will need an enclosure but for the next step I started with a semi-disposable plastic storage container.

Pioneering Avant Garde project enclosures

And much to my surprise, it's still working at this stage. Before I go for a more permanent enclosure I want to let it run for a few days and make sure it doesn't need any changes.

Trouble In the Garden of E-Dumb

It does work, mostly. The pump is tiny and weak which is to be expected. I don't really care how fast the water makes its way to the orange trees, just that it gets there. Eventually though the pump is being started but not shut off. The pump is on, no water is flowing. It has to push the water through 1/4" inner diameter vinyl tubing up the height of the bucket, then a few yards over to the tree. I fluff the tubing and the water starts flowing. My hope at this point is that I'd only have to prime it after it's been idle for a while. The next step up in pump power is 12v and I'm reluctant to go there.

This problem persists. If I prime it, it gets going otherwise not much is happening. At first I thought maybe the primary purpose the pump is serving here is to get the initial water over the bucket height and then siphoning is taking care of the rest. It's not reliably doing even that though.

It eventually occurs to me that I can help the siphoning action by elevating the bucket. The path along the ground from the bucket to the tree is all flat. If the water source is elevated higher than the destination the siphoning should be more effective. I'm about to arrive...

The Autobucket: Passive Edition

I eventually realize I'm being pretty stupid. I've realized that gravity is doing most of the work here. I can ensure that the water reservoir is higher than the outlet.

I don't need the pump, the sensors, or the control at all. I need a hole near the bottom of the bucket and to seal the tube in that hole. Gravity will cause the water to drain through the tube. I had fixated on a solution to the complete neglect of the objective.

In summary:
  • What I built: A network-connected gray water reclamation and irrigation system
  • What I needed: A bucket with a hose glued into it

Sometimes the dumb solution is the right solution

Reflections

I basically took three lefts instead of a right but the journey wasn't all for naught. I got validation that I had overcome my gaps in electronics knowledge since the leaky air conditioner.

One thing that went very well was my process. In past projects I've had frustrating failures by pushing through to a complete solution. When something didn't work I had gone so far that troubleshooting meant tearing down and starting over. This time I worked much more incrementally, validating my progress at each stage and having the chance to make corrections. While I rarely needed corrections along the way, the anxiety that I might have screwed something up and wasted all my time was minimal.

I diagrammed lots of things. I put things together on the breadboard to make it work then translated that to a circuit diagram that I could follow more easily. When wiring up the CAT-5 I wrote down which colors would do what before I made any connections. From then on I could easily know which was the correct wire. I could then maintain that color scheme for portions downstream from the cable to keep it consistent and easier to wrap my head around. I haven't built up my electronics chops yet to keep what each wire does in my head.

Wire type really matters. I've often used internal CAT-5 strands as plentiful solid-core wire with color-coding. When I put that on the female pin header it didn't flex for anything and was very hard to work with. I had some stranded but only three colors, which complicated things. I ordered an assorted color stranded wire kit and for subsequent projects and it's been great.

The weather sensor was a great addition. I loved being able to check it from my phone. I ended up scrapping the raspi setup and setting up just weather sensors with the BME280, an Adafruit ESP8266 board, and some software that made the data available in Prometheus format.

Chibi Weather Station

Ultimately we don't learn a whole lot from our successes; we learn much more reflecting on failure. Maybe my missteps can be useful to you.

Thursday, February 27, 2020

Priority Channel in Go

I'm kind of impressed with this ugly monster:

package main

import (
 "context"
 "fmt"
 "sync"
 "time"
)

func main() {
 const levels = 3
 const sourceDepth = 5
 sources := make([]chan int, levels)
 for i := 0; i < levels; i++ {
  sources[i] = make(chan int, sourceDepth)
 }
 out := make(chan int)

 ctx, cancel := context.WithCancel(context.Background())

 wg := &sync.WaitGroup{}
 pc := New(ctx, sources, 10, out)
 wg.Add(1)
 go func() {
  defer wg.Done()
  defer close(out)
  pc.Prioritize()
 }()

 wg.Add(1)
 go func() {
  defer wg.Done()
  for i := range out {
   fmt.Println("i: ", i)
   time.Sleep(time.Second / 4)
  }
 }()

 for _, i := range []int{0, 2, 1, 0, 2, 1, 0, 2, 1} {
  fmt.Println("submitting ", i)
  pc.Submit(i, i)
 }
 time.Sleep(time.Second * 3)
 cancel()
 wg.Wait()
}

type PriorityChannel struct {
 notify  chan struct{}
 sources []chan int
 out     chan int
 ctx     context.Context
}

func New(ctx context.Context, sources []chan int, cap int, out chan int) PriorityChannel {
 pc := PriorityChannel{
  notify:  make(chan struct{}, cap),
  sources: sources,
  out:     out,
  ctx:     ctx,
 }
 for i := 0; i < cap; i++ {
  pc.notify <- struct{}{}
 }
 return pc
}

func (pc PriorityChannel) Prioritize() {
 for {
  // block until there's a value
  select {
  case pc.notify <- struct{}{}:
   // proceed
  case <-pc.ctx.Done():
   return
  }
 SOURCES:
  for _, rcv := range pc.sources {
   select {
   case i := <-rcv:
    pc.out <- i
    break SOURCES
   default:
    // keep looping
   }
  }
 }
}

func (pc PriorityChannel) Submit(i, priority int) {
 if priority < 0 || priority >= len(pc.sources) {
  panic("invalid priority")
 }
 pc.sources[priority] <- i
 <-pc.notify
}

Monday, January 13, 2020

The Tool Concert: A Synopsis

Drummer: <Bonk uh dunk
Bonk uh dunk
Bonk uh dunk tsh
Bonk uh dunk
Bonk uh dunk
Bonk uh dunk tsh>

Lead Guitar: <Grong gugga gug
Gug grong gugga gug
Gug grong gugga gug>

Bass Guitar: <Do doon doon do>
(no one knows what a bassist is doing)

Singer: I can't express the pain of being intellectually superior to everyone

Background Visuals: <Terence McKenna and David Cronenberg are fighting for control of the Winamp visualization plugins>

My conclusion: Tool is an alternate reality version of Phish where they dedicated themselves to rebelling against yuppies and neocons.


In all seriousness though, it was a really great show, and I'm not really a Tool fan. For the songs I was familiar with they were exactly as you hear on the radio; Rush level precision.

The opening act was awful and I won't dignify the name. Tool played a two and a half hour set which included a fifteen minute intermission. Also, I've never seen a crowd so engaged.

I was surprised by how much I liked the show. Not my favorite style of music but they really did earn my respect.