Unix Signals with Clojure (1.10-beta5), JDK11, and Kubernetes (1.12)

Kubernetes has a beta feature where you can share the process namespaces for a pod, allowing you to send, say, SIGHUP to another container in the pod, assuming you set the relevant permissions in the pod manifest.

Clojure and JDK11 allow you to catch signals, through sun.misc.Signal, (despite years of threatening to take it away).

So you can catch SIGHUP with the following:

(defn set-signal-handler!
  "Catch HUP, INT, or USR1, etc. and run a one argument handler that
  accepts a sun.misc.Signal"
  [sig f]
  (sun.misc.Signal/handle
   (sun.misc.Signal. sig)
   (proxy [sun.misc.SignalHandler] []
     (handle [received-signal]
       (f received-signal)))))

Just for reference, you can get your pid via

 (.pid (java.lang.ProcessHandle/current))

If you need to send a signal from Clojure, I'd suggest shelling out, as to the best of my knowledge, the JVM only lets you send SIGTERM and SIGKILL:

(clojure.java.shell/sh "/bin/kill" "-s" "SIGINT" "<some-pid>")

or

(clojure.java.shell/sh "/usr/bin/pkill" "-HUP" "some-process-name")

Here is the relevant Kubernetes information with the pod configuration, which basically amounts to setting shareProcessNamespace: true at the top of the pod spec, and adding the following to the container that will be sending the signal:

securityContext:
      capabilities:
        add:
        - SYS_PTRACE

Also, if you've come this far, you should take a look at https://github.com/pyr/signal

Note that java has a flag that seems to be related.  I found it after I wrote the original post.

-XX:+AllowUserSignalHandlers

Enables installation of signal handlers by the application. By default, this option is disabled and the application isn’t allowed to install signal handlers.

Referenced Javadocs:
java.lang.ProcessHandle

Friend, oauth2, JWT, and google.

A quick placeholder for showing how to put them together, without added extra dependencies.

View gist

I'll discuss it shortly.

Towards a better dir function in Clojure. (pt. 1)

Coming from Python, for the most part, I felt right at home in the Clojure REPL. However, one of my trustiest old tricks in the Python interpreter didn't work nearly as well in Clojure.

I'm referring, of course, to the "what the hell was that called again?" fixer: dir.

In Python, dir will get you the fns/classes in a namespace, but the first major obstacle in clojure is the lack of support for namespace aliases.

Let's fix that, shall we?

Looking at the source for the dir macro, we find:

user> (source dir)
(defmacro dir
  "Prints a sorted directory of public vars in a namespace"
  [nsname]
  `(doseq [v# (dir-fn '~nsname)]
     (println v#)))

Ok, this is really no more than a bit of sugar, to save us having to quote the namespace and to print the vars on their own line.  We'll have to dig deeper for some meat.

user> (source dir-fn)
(defn dir-fn
  "Returns a sorted seq of symbols naming public vars in
  a namespace"
  [ns]
  (sort (map first (ns-publics (the-ns ns)))))

One thing I really love about Lisps: many times when I look under the hood, I'm expecting a monstrous Rube Goldberg machine, but what I find is some beautifully simple thing, that will still be pretty much the same thing in 50 years.  (Go back and read some of the early algorithmic lisp code, it comes very easy.)

The first part of dir that bugs me is you must specify the full namespace, even if you've aliased it to something much simpler.

(ns yournamespace
  (:require [clojure.string as str]
            [clojure.repl :refer :all]))

(dir str)
=> Exception No namespace: str found  clojure.core/the-ns (core.clj:3830)

user> (dir clojure.string)
blank?
capitalize
escape
...

You can get aliases via ns-aliases, so...

(defn alias-dir-fn [nsname]
  (-> (ns-aliases *ns*)
      (get nsname) ; look up the alias or nil trying
      (or nsname)
      dir-fn))

(defmacro alias-dir
  "Prints a sorted directory of public vars in a namespace or an alias
  namespace."
  [nsname]
  `(doseq [v# (alias-dir-fn '~nsname)]
     (println v#)))

Now we can...

user> (alias-dir str)
blank?
capitalize
escape
...

But I'm not done with dir yet.  Stay tuned for part 2.

Edit: I've filed a ticket for this functionality to be added to Clojure

Using memoization to change a creation function into get-or-create.

N.B. This should work in any language with first class functions, memoization, and immutable values.

tl;dr:  In the past I'd always thought of memoization as a way to save the computer work. It hadn't occurred to me that it could also save me work.

I was re-reading "The Joy of Clojure" and came across a gem I'd missed the first time. Listing 14.12 is described as "A function to create or retrieve a unique Agent for a given player name".

(def agent-for-player
  (memoize
    (fn [player-name]
      (-> (agent [])
          (set-error-handler! #(println "ERROR: " %1 %2))
          (set-error-mode! :fail)))))

;; The above doesn't quite work for me, set-error-handler!
;; doesn't seem to return the agent. Doesn't make the pattern
;; less compelling, though.

The authors comment that this allows you to maintain player-name as an index into a table of agents without having to explicitly manager and lookup agents.

There are two caveats to this approach: 1) player-name must be immutable, and 2) You really need to understand the memoization mechanism.  clojure.core/memoize, for instance, will keep a internal map of args/response until the end of time. You could use http://clojure.github.io/core.memoize/ to modify the strategy if you so choose.

The place where I'd try this first is in what I call "micro-logs". Frequently as I'm working, I want to log some data to a side channel, and this pattern saves having to manage this manually and cluttering up my code.

(def get-or-create-micro-log
  (memoize
   (fn [file]
     (io/make-parents file)
     (.createNewFile file)
     (-> (io/writer file :append true)
         (agent
          :error-mode :fail
          :error-handler #(println "ERROR: " %1 %2))))))

(defn microlog
  "Useful micro-pattern to send off a write to various
  files via agents without having to maintain a lookup
  table. Symlinks can get you into trouble; at a 
  minimum they will duplicate the agent."
  [lg line]
  (let [a (-> (io/file lg)
              (.getAbsoluteFile)
              (get-or-create-micro-log))
        output (str (str/trim-newline line) "\n")]
    (send-off a
          (fn [writer]
            (doto writer
              (.write output)
              (.flush))))))

(defn microlog-all
  [lg all-data]
  (doseq [d all-data]
    (microlog lg d)))

Edits: Used io/make-parents instead of File calls. Changed send to send-off, since this is I/O. Cleaned up creation of agent.

Om examples: Scrubbing Calculator component

I'm starting to get the hang of Om.  While I've used cljs & jayq for a while, understanding the right way to use om + core.async has taken a bit of doing.  They're both great libraries, and have fairly small APIs, but I've been hungry for fairly simple examples.  In that vein, here are a couple, the second inspired by Bret Victor's Scrubbing Calculator.

Update: git repo with examples

https://github.com/jwhitlark/scrubber-example

Prelude: simple clock

Simple enough, derived from the animation example in om examples.

Scrubbing Int, Approach one: all local state

Two utility functions:

My first attempt entirely used internal state.  Not terribly useful, but it set the stage.

It works entirely in the span element.  Click and drag changes the value. If you drag out of the element before you release the mouse button, the state becomes inconsistent.  The snippet below will let you tune the sensitivity if you use it in the :onMouseMove handler.

Now

Approach two: Moving out the state

So here I've switched things to use an external atom for the value, and moved most of the logic into event listeners under IWillMount.  This works correctly even if you drag off the span element.  It's pretty much everything I set out to do with this.

The only drawback is the event listeners should be connected in the handler for onMouseDown, and removed when "mouseup" is received. I'll update it when I've figured out how to remove existing listeners.

Afterward: Stuff you should be looking at

• devcards are awesome, a great way to prototype this kind of stuff
• Prismatic/om-tools
• om-sweet-om-high-functional-frontend-engineering-with-clojurescript-and-react/
• Isomorphic Clojurescipt, and his example omelette

DataScript might be a great idea for Datomic

I was looking at DataScript, and the simple implementation gave me (I think) a lot of insight into Datomic.  If Tonsky's post is correct, and there is a core to Datomic that could be used as an in-memory Datomic-like thing, which could be used as easily as, say, core.logic, and that I could look under the hood to see how a simple case works, I would be in a much better place.  A big part of the problem is the catch-22 for learning datalog, easy to toy with, don't know where to go from there.

I'm not against spending money.  A few months back I went to an all-day Datomic class, and thought it well worth the $300 or so I spent on it.  I wouldn't be against talking people into paying for Datomic, but I'm not familiar with it to know when I should.  The main problem is that I haven't spent enough time playing with it to get a good feel for where it fits best.  Spinning up a separate server process just doesn't really work for that kind of playing.

Sort of like how I understood garbage collection, but once I implemented a toy lisp, I *got* it, and understood it's tradeoffs in a way I never had before.

I don't know what Datomic's sales numbers look like, but this seems like it would be a good strategy to increase interest/familiarity with the protocols and ideas involved, which would presumably increase the sales pipeline.

.emacs style Clojure rc files

The following will let you run clojure code in the current namespace, from a dotfile in your home directory.

(require '[clojure.java.io :as io])

(load-string (slurp (io/file (System/getenv "HOME") *rc-file*)))

The old, new job

Oh, yea.  FYI: I work for bittorrent these days.  I started around June.

Running Rake as a script

Rake seemed nice, but I needed some of it for a server side trick. Not difficult, but if you're new to it, it's tough to find just what you need.

https://gist.github.com/1228206/eca65818062f0a18a5c7923c94350f56153b32a6

Is all the boilerplate you need to run rake tasks as a script, from anywhere in the filesystem.

Controlling android via http(s) over usb

I've been playing with couchdb, and was playing with the paw web server for android (http://paw-android.fun2code.de/), and discovered something interesting.

* If you turn on usb teathering, then connect your phone to a linux box, it'll automatically add usb0 as a connection.
* If you grab the default route for usb0, you'll have the ip of your android phone.
* If you run paw, you'll be able to do things like send sms, control your phone, etc.
* If you run, (and tweak the config), you'll be able to connect to the couchdb on your phone
* You don't have to access these services via a web browser, (yay! SMS from emacs!)
* paw server also has stuff like "play music from phone", and "use phone as webcam", etc. etc. etc.

I suppose I knew all this was possible, but I didn't think it would be "done in 5 minutes" possible...

Python lockfiles

The python-daemon library has a nice implementation of pidfiles, much better than the standard lib lockfile. i.e.

import lockfile.pidlockfile as pidlock
lf = pidlock.PIDLockFile("/tmp/mqtask.pid")

# module methods
pidlock.read_pid_from_pidfile
pidlock.remove_existing_pidfile
pidlock.write_pid_to_pidfile

Airbnb sounds cool, but I really want Walk-in-closet-to-go

I don't like packing.

When I look around my house, I see the progress I've made towards living a less-stuff lifestyle, such as advocated by Paul Graham, among many others, (see Stuff.) Still, I don't think I'll reach the point where I can comfortably live out of a suitcase or a backpack without significantly reducing the quality of my life.

So what about a service more like AirBnB + PODS + A Closet organizer something like this + Robotic movers similar to these = Easy, comfortable, continuous travel.

Just playing with numbers on the back of an envelope, I think this would be both possible and cheap, in the near future.

Hmmmm....

Using Apache Camel from Clojure

I've been looking at Apache Camel for a project at work, and had a bit of trouble getting it working from Clojure, so I wrote this up in case anyone has the same issues.

The first example in Camel in Action watches a directory for new files, and copies them to an output directory. Here are the steps to get that running in Clojure, using leiningen.

• Create your input and output directories
• Create a new leiningen project
• Add the following to the project.clj

• run lein deps
  
• Add the following to your src/org/whitlark/fc.clj (or whatever your last name is ;-)

Part of what screwed me up was the fact that you need to include spring in the project in order to get camel to work properly.

You can also use a macro to make the code even cleaner, like this:



Which I think compares nicely to the original Java:



I'm sure there are more improvements possible, but this got me started.

Extending the range of a PS3 controller.

Just plug a short usb cable (I use 12"), and leave the other end unplugged. The cable seems to act like an antenna, giving you a better connection. I've not done any tests to see exactly how much you can get, but I get at least another 10'.

I'm thinking about opening up the controller and soldering an antenna to the usb enclosure. If I do that, I'll post instructions/video.

Quick post on working with large changes across your codebase

Stuff I wrote up for work. Ignore the p4 part, assuming you use a real version control system, *cough* git.

Strategies for making changes that affect a large number of files

emacs:

• recursive editing (C-r, C-M-c)
• find-grep-dired
• ibuffer (Used to save large number of buffers at once)
• dired-do-query-replace-regexp (find-replace across marked files in dired buffer)

shell:

• grep -rIl 'pattern' * > files_to_op_on (that's a capital i and a lowercase L)
  • grep option -r: recurse
  • grep option -I: (capital i) exclude binary files
  • grep option -l: (lowercase L) Show only filenames with matching content, no other output.
  • grep option -h: don't include filename in the match line.
• cat files_to_op_on | xargs sed -i backup_extension 's/old/new/g' #Note that bsd sed required a backup extension with the -i argument.
• cat files_to_op_on | xargs p4 edit

Watch out for symlinks! If you get them in your changelist, p4 won't accept submit, in which case you'll need to delete them via p4 change.

p4 submit > /dev/null

works well to just get the errors.

The general pattern of:

grep -rI 'something' * | sort -u

can show you the variations you'll need to fix. If you want to get rid of leading whitespace in your results, try piping through:

 sed -E -e 's/^[[:space:]]+//'

Clojure, Frozen Bubble, and how I learned to start worrying about my math education

So, I've been playing around with Clojure, reading the rough cut of Programming Clojure, (good book, so far), and was looking for a project to sink my teeth into in order to really get a feel for the language. Frozen Bubble, aka Bust A Move, has long been a favorite game of my wife, and while I tend to lose to my coworkers, should be interesting to implement.

Using a sample implementation of snake from the book as a template, I start hacking on the code, striping out all the snakey stuff, and just getting a minimal program that just displays a window. So, Frozen Bubble has only two real game objects: your target pointer, which ranges from -90 to +90 degrees, (a little less actually, no sense firing horizontally), and bubbles, which are in one of three states: not moving while in the initial position at the center of the bottom of the screen, stuck to the top of the game area (directly or via a chain of bubbles), and moving from the bottom of the screen to the top.

Well, the first state is easy, the third state is probably the same, but the second state has some unexpected complexity. First I think I'll give the bubble a location and direction, and handle the speed via the game tics. Seems reasonable, right? Then I run smack into something that makes me wish I had paid more attention in math class. The bubble has a direction, expressed in degrees, which I need to use to manipulate the location, which is expressed as [x,y].

Being a proper little reductionist, and stubborn enough to try to figure it out for myself instead of spending five minutes with google, I decide to think about the second simplest case, a 2x2 grid of pixels. (The simplest would be a single pixel, and I don't see how that would help at all. Perhaps I'm just not being clever enough, though.)

Anyway, this setup gives us the following:

Starting point: [0,0], 0 degrees (straight up or North) -> [0,1], 90 degrees (right or East) -> [1,0], and 45 degrees (NE) -> [1,1]

This says to me that if I want to work in single pixels, I have to do it in 45 degree increments. Not satisfactory. I can approximate closer and closer angles by using a larger grid, (why do I feel like this leads to calculus?) But that means that I'm not working with single pixels anymore.

I went and read some things at Better Explained, which while fascinating in their own right, (I love that site), didn't seem to help matters. I think the gap is that the actual position in real space would be floating point (well, kinda), but the game space in measured in integers (pixels).

Grabbing my trusty graph paper, I start diagramming. A 3x3 grid only buys me two new angles: 22.5 and 67.5. 4x4 gives me 0, 15, 30, 45, 60, 75, & 90.

I now have:
2x2 = 4 squares = 3 angles
3x3 = 9 squares = 5 angles
4x4 = 16 squares = 7 angles
5x5 = 25 squares = 9 angles, etc.
which generalizes to: AxA = A**2 squares, and 2A-1 angles
So to represent all 90 degrees, (which is actually kind of arbitrary anyway), I solve for 90 = 2A-1 = 45.5 squares to a side. My screen resolution on my laptop is 1024/768, so using the height as a guide, if I want to update to bubble position by individual degrees, I can only do it ~16 times from the bottom to the top of the screen. Clearly less than ideal, some sort of approximation is called for.

There are two ways that I can think of to do this off the top of my head for a 2x2 grid update: splitting the difference evenly, (i.e. a 20 degree angle will be bumped up to 45, or it can be rounded to the closer number, in this case 0. The rounding case seems more rational.) Obviously I want to stagger the updates to get the closes approximation to the actual angle I can, but a constraint I'm operating under is to have no other state saved between updates than location and direction. Ummmm.....

The only option I can see is to manipulate the direction each time so that after ~45 pixels, it arrives in the correct place. However, my laptop battery is about to die, and it's late, so I think I'll sleep on it and continue this later.

Nice trick with functools and os.path.join

Sometimes os.path.join('something', 'something_else', 'more' ... ) gets old. With functools.partial, you can preload os.path.join to clean up your code. For example:

import functools
my_path_plus = functools.partial( os.path.join, 'a', 'b', 'c')
m = my_path_plus('x', 'y', 'z')
# m is 'a/b/c/x/y/z'
n = my_path_plus()
# n is 'a/b/c', for when you don't want to add anything.

What cities whisper to us

Another great post by Paul Graham. I think I'll stop linking to his stuff unless I have something to add. Just go an read his essays.

I wonder what message Istanbul sends to residents. (edit) Having been to Istanbul again recently, I still don't know what it says to you, but can say it's a very sweet song, (if you've got money).

Distraction...

Paul Graham posted again in his usual good form. I canceled my cable years ago, and have never regretted it. Amusingly, I follow his formula for email at work, but only because Ironport uses exchange, and I haven't gotten around to setting up a windows vm on my Linux desktop. Hmm.. Thinking about it, I should try using Evolution instead. I'll give that a shot and report back...

Update: Evolution might get the job done, but I don't like it. Back to thunderbird, and ignoring meetings...

Unit tests force decoupling

I just read Vidar Hokstad's post on reducing coupling via unit tests, and I can't agree more. I'm currently trying to untangle some code that was written without unit tests, and is very tightly coupled. It's easier to fix in Python than it would be in, say, Java, but that still doesn't make it fun. A recent post on proggit makes the argument that unit testing is on the wane. I really hope he's wrong. I've been doing TDD for several years now, (since reading Kent Beck's book on the subject), and I find it much better than any other methodology.

I was also sad to hear that Agitar went under. I was never in their target market, but I always thought they had some cool tools.