j: deserialize from wordpress

In http://r-nd-m.blogspot.com/2018/02/jq-serialize-for-wordpress.html I was using jq to import json content into wordpress. There was significant follow-on work, as I was learning wordpress and mostly working at the database level, but the results were mostly satisfactory. (Though, in retrospect, maybe I should have been using javascript (node) rather than jq.)

Anyways, now I'm needing to do some more low-level work (need to migrate images to s3, because of wpengine storage constraints), but before I can adequately plan that out, I need to verify some assumptions.

And, for that, I want to build a list of all image urls for all crops (among other things, to see if any crops are shared across images, as a consequence of some earlier repair work).

So... time to build a wordpress deserializer in J.

Here's a draft:

fromwp=:3 :0
  'off val'=. 0 wp_unserialize y
  assert. off -: #y
  val
)

wp_unserialize=:4 :0
  NB. almost all numbers are character counts (numeric results are the exception)
  select. x { y
    case.'N' do.
      assert. 'N;' -: (x+0 1) { y
      (x+2);<a:
    case.'b' do.
      b=. (x+i.4) { y
      assert. (b -: 'b:0;') +. b-:'b:1;'
      (x+4);b-:'b:1;'
    case.'i' do.
      assert. 'i:' -: y{~x+0 1
      s=. (x+2) ';'findCh y
      (x+3+s);_ ". y{~x+2+i.s
    case.'d' do.
      assert. 'd:' -: y{~x+0 1
      s=. (x+2) ';'findCh y
      (x+3+s);_ ". y{~x+2+i.s
    case.'s' do.
      l=. _ ". y{~x+2+i. (x+2) ':'findCh y
      o=. #":l
      r=. y{~x+o+4+i.l
      assert. ('s:',(":l),':"',r,'";') -: y{~x+i.6+o+l
      (x+o+l+6);r
    case.'a' do.
      assert. 'a:' -: y{~x+0 1
      l=. _ ". y{~x+2+i. (x+2) ':'findCh y
      o=. #":l
      assert. ('a:',(":l),':{') -: y{~x+i.4+o
      'x r'=. (x+4+o) l wp_unserialize_array y
         NB. iterate through array - use a helper for this for clarity
      assert. '}'-:x{y
      (x+1);<r
    case. do.
      'unsupported serialization type' throw.
  end.
)

wp_unserialize_array=:1 :0
:
  kv=.2 0$''
  for.i.m do.
    'x k'=.x wp_unserialize y
    'x v'=.x wp_unserialize y
    kv=.kv,.k,&<v
  end.
  x;<kv
)
       
findCh=:1 :0
:
    peek=. y{~x+i.x-~(x+100)<.#y
    if. m e. peek do.
      peek i.m
    else.
      (x}.y)i.m
    end.
)

Some notes:

• J arrays are a bit different from php arrays. So I elected to represent php arrays as two row arrays in J (first row is keys, second is values).
• Technically wordpress arrays could include serialized objects. But those aren't relevant here, so I don't support them here.
• I had been working with a result exported from mysql (using the -s comman line option on mysqlclient) and read into J using readdsv. This mangled double quote characters. I now use (<;._2@,&TAB);._2 instead of readdsv.
• As an aside, mysql exports four characters with backslash escapes (backslash, newline, tab and ascii nul) - I handle this issue before running fromwp, using: 
• fromwp rplc&('\\';'\';'\0';({.a.);'\t';TAB;'\n';LF) exportedstring

Here's what I use to index from these two-row arrays which represent php arrays:

idx=:4 :0
  (a:,~{:y) {::~"1 0 ({. y) i. <^:(0=L.) x
)

Left arg is key, right arg is array, those arrays are always boxed arrays. This routine unboxes its result. Invalid keys give empty results.

Generally, when debugging this code, I enable suspensions (there's a 13!: foreign I could use, but there's also a cmd-K (or control-K) that turns on debugging under jqt). Usually just looking at the variables and character subsequences is enough to make problems obvious.

(I certainly don't memorize every little bit of this code. Yes, it's a bit ugly - parsers are like that. Either ugly, or so abstracted that you can't find the ugly stuff (and, thus, still not memorizable).)

Anyways, here's an example (or part of it):

   fromwp (<-1 1){::rawmeta
┌─────┬──────┬─────────────────────────────────────────────────┬────────...
│width│height│file                                             │sizes   ...
├─────┼──────┼─────────────────────────────────────────────────┼────────...
│4000 │2650  │outdoor-lighting-options-good-fences-sun-1016.jpg│┌───────...
│     │      │                                                 ││thumbna...
│     │      │                                                 │├───────...
│     │      │                                                 ││┌──────...
│     │      │                                                 │││file  ...
│     │      │                                                 ││├──────...
│     │      │                                                 │││outdoo...
│     │      │                                                 ││└──────...
│     │      │                                                 │└───────...
└─────┴──────┴─────────────────────────────────────────────────┴────────...

If it weren't clipped, that representation of the array would be 1241 characters wide.

jq: serialize for wordpress

This kind of just rolls off the tongue, right?

def towp: 
  if "null"==type then "N;"
  elif "boolean"==type then if . then "b:1;" else "b:0;" end
  elif "number"==type then if .==(.|floor) and .<9e15 then "i:"+(.|tostring)+";" else "d:"+(.|tostring)+";" end
  elif "string"==type then "s:"+(.|length|tostring)+":\""+.+"\";"
  else "a:"+(.|length|tostring)+":{"+(.|[to_entries|map(to_entries)[]|map(.value|towp)|add]|add)+"}"
  end;

For some reason, even though wordpress uses php, its serialized objects use a slightly different format from what's documented at http://php.net/manual/en/function.serialize.php -- see https://wpengine.com/support/wordpress-serialized-data/ for a decent description and https://codex.wordpress.org/Function_Reference/maybe_serialize for a more official description. (Edit: revisiting those pages shows that wordpress is now using php's serialize.)

Note, in particular, that in this wordpress format, strings get enclosing quotes but the reported length does not include those quotes. I do not yet know what this looks like for quotes and backslashes within a string - there are at least four possibilities (two hideously broken) for how that might be handled.

One issue, I imagine, is that when you delete an element from a php array, indices do not change for items after that point. The php serialize format does not preserve indices, but the wordpress serialize format does.

Anyways... it's a defining characteristic of software that "what works" generally takes precedence over "what's formally correct" (at least, if you want it to work).

Thus, although I keep harping on the distinction between "boolean" as used in software and its history:

• https://r-nd-m.blogspot.com/2014/04/the-boolean-disease.html
• https://r-nd-m.blogspot.com/2012/08/et-jaynes-and-boolean-algebra.html
• https://r-nd-m.blogspot.com/2012/02/truth-values.html

... I still have to use the word "boolean" in my code (and in my web searches when I want to be reminded of related syntactic issues or whatever else -- on the plus side, long words that are relatively meaningless do have some advantages when searching, as long as you know you need to search for tham).

But I guess that's related to one of the nice things about standards... there's so many to choose from.

And I guess that's also related to how you don't really want to use this on stuff that's already a string (unless sometimes it has to be something else, and you want to preserve that...).

And, also, related to how blogger's layout is so ornate that it pretty much has to be fixed width (which means it will be wrong for some desktops (they pretty much have to disable the theme support for phones - it's really that insane). [There is an option to revert the blog to "classic themes", but the preview on that is not really a preview at all, and I need to be doing other things now...]

And, I guess another loosely related issue is how incredibly difficult it can be to report bugs. (There's just so many people in the world - billions - and most of them do not have the frame of mind to understand what a meaningful bug report is. So as more and more people come online, things get more and more messed up and things that used to work start failing as a consequence. ... Eventually, the failures may become visible enough to get fixed anyways, but all too often huge issues can be neglected for decades, or longer. And, just figuring out where (or if) they should be fixed can be daunting.)

Anyways... here's the reverse transform:

def _fwp($P):
  $P.off as $j|
  .[$j:$j+1] as $typ|
  if "end"==$P.op then
    if ";"==$typ then
      $P|.off|=.+1
    else
      error("expected ';' at position "+($j|tostring)+" got '"+$typ+"' ")
    end
  elif "endarray"==$P.op then
    if "}"==$typ then
      $P|(.off|=.+1)|(.len=$P.stack[-1])|(.stack|=.[:-1])
    else
      error("expected '}' at position "+($j|tostring)+" got '"+$typ+"'")
    end
  elif "start"==$P.op then
    if "N"==$typ then
      _fwp($P|.op="end"|.r=null|.off|=.+1)
    elif ":"!=.[$j+1:$j+2] then
      error("expected : at position "+($j|tostring)+" got '"+.[$j+1:$j+2]+"'")
    elif "b" == $typ then
      .[$j+2:$j+3] as $t|
      if "0"==$t then
        _fwp($P|.op="end"|.r=false|.off|=.+3)
      elif "1"==$t then
        _fwp($P|.op="end"|.r=true|.off|=.+3)
      else
        error("expected 0 or 1 at position "+($j+2|tostring)+" got '"+$t+"'")
      end
    elif ("i"==$typ) or ("d"==$typ) then
      (.[$j+2:]|match("[^;]*")) as $match|
      _fwp($P|.op="end"|.r=($match.string|tonumber)|.off|=.+2+$match.length)
    elif "s"==$typ then
      (.[$j+2:]|match("[^:]*")) as $match|
      ($match.string|tonumber) as $strlen|
      ($j+4+$match.length) as $J|
      if ":\""==.[$J-2:$J] then
        .[$J:$J+$strlen] as $str|
        _fwp($P|.op="end"|.r=$str|.off=$J+$strlen+1)
      else
        error("expected ':' at "+($J|tostring)+" got '"+.[$J-1:$J]+"'")
      end
    elif "a"==$typ then
      (.[$j+2:]|match("[^:]*")) as $match|
      if 0==$match.length then
        error("invalid array length at position "+($j|tostring)+" got nothing")
      else
        ($match.string|tonumber) as $alen|
        ($j+3+$match.length) as $j|
        if ":"==.[$j-1:$j] then
          _fwp($P|.op="startarray"|(.stack|=(.+[$P.len]))|.len=$alen|.off=$j)
        else
          error("expected : at start of array at position "+($j|tostring)+" got '"+.[$j-1:$j]+"'")
        end
      end
    else
      error("unrecognized type "+$typ+" at position "+($j|tostring))
    end
  elif "startarray"==$P.op then
    if "{"==.[$j:$j+1] then
      if "i"==.[$j+1:$j+2] then
        _fwp($P|.op="array"|.r=[]|.off=$j+1)
      elif "s"==.[$j+1:$j+2] then
        _fwp($P|.op="object"|.r={}|.off=$j+1)
      elif "}"==.[$j+1:$j+2] then
        _fwp($P|.op="endarray"|.r={}|.off=$j+1)
      else
        error("invalid index type '"+.[$j:$j+1]+"' at position "+($j|tostring))
      end
    else
      error("expected { at start of array at position "+($j|tostring)+" got '"+.[$j:$j+1]+"'")
    end
  elif "array"==$P.op then
    if 0==$P.len then
      _fwp($P|.op="endarray")
    else
      $P.r as $r|
      _fwp($P|.op="start") as $P|
      $P.r as $key|
      if "number"==($key|type) then
        _fwp($P|.op="start") as $P|
        $P.r as $val|
        _fwp($P|.op="array"|.len|=(.-1)|.r|=$r+[$val])
      else
        error("invalid array index "+($key|tostring)+" at position "+($j|tostring))
      end
    end
  elif "object"==$P.op then
    if 0==$P.len then
      _fwp($P|.op="endarray")
    else
      $P.r as $r|
      _fwp($P|.op="start") as $P|
      $P.r as $key|
      if "string"==($key|type) then
        _fwp($P|.op="start") as $P|
        $P.r as $val|
        _fwp($P|.op="object"|.len|=(.-1)|.r=$r+{($key): $val})
      else
        error("invalid object index "+($key|tostring)+" at position "+($j|tostring))
      end
    end
  else
    error("program bug (this should never happen)")
  end;

def fromwp:
  _fwp({"op":"start","off":0,"len":"none","stack":[]}).r;

You wind up needing to a parser in jq for the reverse transform, and it mostly has to go into a single recursive function because functions can't refer to other functions which have yet to be defined, and I couldn't think of any way of breaking out significant chunks that made sense, with that limitation.

It doesn't help that (at least in version 1.5) jq's error handling is kind of useless (does not tell you where in the code the error occurred). So I threw in some forced errors to help track down problems (I could do better about reporting invalid where invalid numbers occurred, but try/catch in jq 1.5 mixed with error statements in code like this can lose track of context - in some cases which I found difficult to isolate I was seeing draft versions of this code trying to parse an error statement instead of the text it was supposed to be parsing.)

Another quirk here is that I threw in the support for double quotes around strings (why bother using a numeric string length that does not include those quotes? Scary design process there...) at the last minute, and I'm not bothering to check for a closing quote - I'm just skipping over that position without checking (which, ok, works just fine... but is sloppy and allows for future specification entropy in a bad way).

(Another issue with parsing nunbers is that I might be asking jq to inspect the entire rest of the unparsed string to see where the number ends - I tried limiting it, using .[$j+2:30] instead of .[$j:], but jq would decide in some cases that that was an error (near the end of the string). I decided I did not want the complexity if trying to micromanage that issue - things are messy enough already and I didn't want the hard-to-debug number parsing to be even more obscure, so just went with the simple .[$j:] approach.)

This is painfully slow on large (50k) objects, but it seems to work...

I should build a report_parse_error({offset, expected}) routine based on J's multi-line reporting style and use that here. That should make for easier to read error messages.

I also should build an extract_next_number($offset) filter and use it here - that would let me be smart about how big of a string I'm searching for the end of the number in (can do minimum of +30 or remainder of string), which should be a big performance win.

Vacuum barriers and wiring

Back when I was semi-seriously thinking about printing semiconductors, one of the issues which stalled me was figuring out how to put wires through the vacuum chamber without destroying its integrity.

I stumbled over one approach: http://www.douglaselectrical.com

No idea what pricing looks like at dev scales nor production scale.

I also have no idea about cutting holes (and cleanup) without wrecking everything. But I seem to remember some chambers come with holes already in them.

USA Voting

So...

Apparently, the Trump wants state voting rolls - details, names and voting history.

People are assuming that this is at the citizen level but given the level of laziness in most discussions I have no clue if that is correct.

But this got me to doing a little research on the history of voting and what the constitution says about it. Turns out that the constitution says nothing about how citizens vote. It's all about how the electors vote.  http://constitutionus.com/#a2s1c2

But, also, how the electors are supposed to vote has little to do with how citizens vote.

My reading of the constitutional voting process goes like this:

Given a list of candidates, the electors go through each pair of them and each elector chooses one over the other. Note that this becomes exponentially time consuming if the ballot is large.

Then, if there are ties, Congress chooses from the "top five". (Note the Senate, though - just the house.)

Do they do this? Probably. I'm not actually sure though...

You do not exactly want a relational database

SQL is touted as a relational database, with some historic infighting because of "relational" issues.

But mostly you do not want a relational database.

A "true" relational database would, in SQL, be like if all SELECT statements were really SELECT DISTINCT statements. (This is based on the mathematical definition of a "relation" which is where we get things like outer joins from. Of course, if we instead redefine "relational" to be whatever arbitrary collection of concepts got included in some particular SQL implementation or SQL standard there's no changes from that particular choice which remain.)

So, for example, if you were tracking money (or food, or really anything), and you happened to have multiple rows with the same value, some of that would vanish from your record keeping.

I think the fundamental structure of a database table is a named collection of columns (where each column is a sequence of the same length and each value in a column is the same type - stored the same way and interpreted using the same mechanics).

Though, for databases optimized for things like retail sales, there's a long history of pushing that structure down into the row and implementing the table as a collection of rows (with some waffling about the sequential aspect of these rows - they really are still a sequence, but the traditional database implementation reorders things rather often).

Business Case Issues for Semiconductor Printers

(This is not a formal business case, but preliminary thinking that might be useful for building a business case.)

Intuitively, Semiconductor Printers (probably mostly based on silicon wafers and molecular beam epitaxy, but other approaches can be viable for some purposes) seems like it should "fit" in today's economy. The underlying techniques are old, and proven (and a first generation of patents has expired). We've a variety of systemic issues where this approach seems like it could be fruitful. And, making this work is just hard enough that the prices on these systems is extremely high - ripe for disruptive harvesting.

But why?

#1) Entertainment value.

People like to build things. Construction games have historically been steady money makers, and there's a measurable fraction of our population (perhaps 16%?) who just enjoy making things.

#2) STEM Education

There's a lot of schlock information out there, and to distinguish between the meaningful and the trash you need to have some practical experiences. Done right, these printers can teach people some important things about physics and engineering which will later be useful in addressing real needs.

#3) Business Failures

Most businesses fail. People building businesses often need to cut corners for a variety of reasons. Entire lines of semiconductors can suddenly become not available or only available under adverse conditions. In many cases these issues can be solved through negotiation and/or finding other approaches. But having another fallback can help extend negotiation timetables and can sometimes be that "other approach".

#4) Tool building

Currently, there is a lot of red tape and cost involved in designing and building new semiconductor devices (unless you happen to be associated with the right people - who are often in some other country). We can change that.


Semiconductors (and related things: conductors and insulators) include things like sensors and things that make different kinds of light. The underlying fabrication process might also be used for making "nano-scale devices" - perhaps useful for fabrics or medical or biological work.

Finally, note that a major cost and design issue is the vacuum chamber. High vacuum chambers are readily available already, but getting power and materials into them can be a significant challenge. This all changes for space based systems, where vacuum is a "natural resource". This points at a variety of future possibilities and the people who understand how to make these things work well will have some advantages in getting space based computer manufacturing ready for market.

Printing Semiconductors

https://en.wikipedia.org/wiki/Molecular_beam_epitaxy

Our economy depends not only on mass scale popularity driven production, public education, solid work ethics and a healthy respect for experimentation but also on hobbiest activities where the future designers and decision makers learn from experience how things can actually be made to work.

So that, I think, means I should be leveraging the "3d printing" technology efforts and getting into semiconductor printing. Most of the relevant patents there have now expired, and there's quite a lot of solid precursor work dating from the early part of the 20th century.

Basically, it's taking the technologies behind the television tube (and other technologies - scanning electron microscopes, mass spectrometers) and using them to develop hobby scale circuitry. Or, that is my hope. (Now let's see if I can motivate myself enough to carry this through the obvious problems...)

Inspirations include:

Ben Krasnow's home built electron microscope

Charles Moore's OKAD vlsi design system

And hopefully I can get enough done to engage interested people at nova-labs

So, first off, that electron microscope is possibly simpler than what an initial build of the printer would be:

First, you need to have some way of isolating problems. For a reprap printer, the human eyeball is good enough, but for things like semiconductors you need something that can see on the microscopic scale, and the electron microscope is a plausible first approximation for what you need there.

Second, you need to be boiling off the materials that you are going to be depositing, and that means things that will need to be replaced over time.

And, of course, there's the need for the semiconductor blanks to be printed on. Almost certainly this will be silicon for any first efforts. So that's another cost item and something else where things can go wrong. (How do I check for problems on the surface of the blanks? If there are fingerprints, or other defects, how do I find a part of the surface which can still be used? Is there some way of cleaning or polishing away minor defects? What would that take?)

For dopants, I imagine initial efforts would focus on boron (p-type) and phosphorus and arsenic (n-type), and since I really don't know what I'm doing here, I'll guess that I'll need a vacuum of 10e-8 torr (though maybe that is too taxing for an initial effort) and dopant concentrations between 1% and 0.001%. (But what temperatures do I need, for these things? What voltages? How should focusing work?)

Also, reading various references, the temperatures involved for the wafer itself might range quite widely.  https://www.google.com/patents/US3615931 suggests 450..650 C for a variety of semiconductor substrate types, but another reference suggested temperatures as high as 850C..900C. And the melting point of silicon is slightly above 1400C.

Meanwhile, http://lase.ece.utexas.edu/mbe.php suggests that tightly controlled temperature ranges are important for repeatable results (which, in turn, are important for the results of destructive testing to be useful). [There's also a considerable body of knowledge to absorb, relating to various models of how semiconductors form and perform - but I think that having some practical examples to work with will help significantly in understanding that material. In fact, that's a significant part of the point of this exercise.]

That said: repeatability is *not* a priority for the initial implementation. For the initial implementation we need a relatively minimal goal. The point of building this thing in the first place is to get practical experience, turning it into something that can be useful of others is going to need that experience.

So, ok, honestly, the first thing to try to do is get a diode working. This is the absolute simplest thing that can demonstrate viability. (How do I test that I am putting it together right? How do I affix electrodes onto the result?)

After that would be a transistor, then perhaps a logic gate and then a binary counter. Another path gets into analog amplification. And if I can even get close to 1970's integrated circuit capabilities, I'll be happy that I understand those fundamental concepts.

For a source of interested people, I'm thinking makerspace community, educational community and maybe eventually business people (but ramping things up for popular use would require engagement with people who are very used to doing other things and with priorities which conflict with my own, so that's something for much later).

I spoke with one person at nova-labs, and he suggested I might need an x-ray license. So I spent some time trying to research virginia and federal (nuclear regulatory commission) regulations, and I currently think that I will not need a license. First off, though, I do not have enough specifics to even know what would be needed. But I am not planning on using this on people, and I'm not going to even approach 1e6 electron volts.

Though, also, there's a bit of an assumption there (I'm going to be working with volts, maybe 5 KV, and I think the implication with the electron volt system of measurement is that we are talking about the amount of energy imparted to one particle - so that's a mix of temperature and voltage and whatever else, and it all gets a bit fuzzy how much exactly is involved, especially when you start thinking about the energy which exists as mass - in principle though, the electron volt regulations are about difference from rest state, and are not about regulating mass itself).

Anyways... as long as the energies are low enough, no licensing should be needed. If licensing does wind up being needed, because of potential radiation hazards then I guess I would need to figure out which licenses would be needed from which agency. But, for now, this is looking more like a hand waving issue than a real issue.

Finally, there's the issue of how to install the connector pins that we traditionally use to hook up with integrated circuits. Conceptually, we might be able to just have some pads which we connect to, and that is something to play with. But "soldering" titanium wires using (for example) molybdenum deposition might be the place to start. This will apparently destroy the crystal structure underneath it (where it deposits), and it seems like perhaps something which has a cooler vapor phase than molybdenum might be the right place to start (lead? tin+lead? tin? ... probably tin, but apparently since lead atoms are so big they can be a carrier for a doping material and the lead will just sit on the surface and not become a part of the crystal...)

In fact, this is such a basic issue that probably the right place to start is first hooking two wires to a silicon wafer and verifying that they can conduct and that the thing is mechanically sound. Once that is doable, try making a semiconductor resistor. And only once that is possible does it make sense to try to make a diode. (And, then, after that: a transistor, some logic gates, a flip/flop, a counter, and be reviewing to make sure the techniques are documented, with some stream-of-thought blogging for things that might be useful memory jogs, later. Somewhere in this, repeatability and, thus, temperature control and high vacuum starts mattering. And, hypothetically, after this has been done it would make sense to bring in design tools, such as OKAD.)

(fifth draft of this page)

USB Mouse

For various reasons, I have been thinking about building a USB mouse.

This is tentative. I am not yet sure if I have the motivation to carry through on this, and I am uncertain about obtaining adequate switches and sensors.

Hypothetically, though, this is doable:

Mouse CPU would be a greenarrays F18 for cost and power reasons. $20-ish, last I looked (but prices change in either direction).

Case would be "3d printed" and then smoothed. If I can't do that on my own, I'd visit the local makerspace (NOVA Labs).

The initial idea would be to support standard mouse protocol and keyboard protocol. The primary use would be for gaming, and I'd want some useful fraction of a keyboard available on the mouse itself. Other mice do this already.

(I might have to get into non-standard protocols and specialized drivers, but not in the initial implementation.)

The other thing is that I would need to be able to isolate problems, so I would need some kind of display when keys are pressed/released so that I can distinguish between problems in the mouse itself (like, problems with the switches) and problems in the computer it is hooked up to.

Obviously, this would take quite some time to build. Time that I could instead spend playing games. Does that make sense economically? (Pro-tip: whenever someone uses an economic argument they probably do not know what they are talking about. Economics seems to be a mix of habits, peer pressure, religion, politics, laziness and pure old-fashioned bs.)

Anyways, this would take quite some time to build, so the real question is: am I motivated enough to expend that effort?

For starters: where can I find the sensors? For an optical mouse, you have a mini-camera (much less resolution, and cheaper, than a regular camera) taking pictures of the surface the mouse is on hundreds or thousands of times per second, and then some software to convert those images into information about position changes which then get packetized and sent to the computer. The trick is finding something that (a) I can buy (or make - but somehow semiconductor fabrication is not available at the hobbiest level anywhere I can find, which is silly given how important the related skills and physics are), and (b) that I can understand [find adequate documentation on] well enough to use.

Here are some sample spec sheets (and I hope that the link remain stable and active and do not fall prey to the type of manager who likes to destroy this kind of thing):

https://www.sparkfun.com/products/retired/12907
http://www.bidouille.org/files/hack/mousecam/ADNS2051.pdf

The biggest problem, right now, is that for example nothing is in stock at digikey.

"Standards"

"The nice thing about standards is that there are so many to choose from."

Standards can be a blessing, or a curse, or both. They are an outgrowth of a kind of problem that occurs when dealing with people.

Basically: when dealing with people you need some things which do not change so that you can deal with the "important stuff" (which itself varies depending on who and what you are dealing with). And  "standards" are the stuff that is held still.

Put differently: standards are useful at the interfaces between where one person is doing work and another person is doing work.

Also: standards that have been "designed" tend to be rather useless - what you want are standards which have evolved for dealing with problems similar to what you are dealing with. (Sometimes this includes "designed standards" but usually when that happens only a small part of the design is relevant. Or, at least, that has been my experience in the context of computing. This says something sad about the usefulness of a lot of people hours. But it also says something about how you should expect to be working if you want to get something useful done.)

Meanwhile: discussions about standards can become rather acrimonious. In my experience, this tends to be a mix of personality flaws in people holding the discussions and irrelevant wasted motion in the standards themselves.

Related is probably the cliche'd concept of "if you want something done right, you'll need to do it yourself" and its close [imperative] relative "take some responsibility". People tend to be frustrating to deal with, but that is often as much a fault of the person getting frustrated as it is a fault of the people they are getting frustrated with. Remedies which ignore either side of this kind of problem tend to fail.

It's often best to try to fail early so you can learn from your mistakes. But ...

War and China

The top 15 most populated countries on the planet are:

1) China
2) India
3) United States
4) Indonesia
5) Brazil
6) Pakistan
7) Nigeria
8) Bangladesh
9) Russia
10) Japan
11) Mexico
12) Philippines
13) Ethiopia
14) Vietnam
15) Egypt

Most computer hardware is made by China - the laws, regulations and traditions of the other countries are structured to favor this process.

Meanwhile, though, it seems to be extraordinarily difficult to get certain kinds of information out of China. We have a general idea of the size of its population (between 1.3 billion and 1.4 billion). We have a general idea of the life expectancy of this population (about 75 years), but it's difficult to get information about causes of death. Actually, it's difficult to get much of any good information out of China. I sometimes wonder if Chinese officials even know. But if we assume that these numbers are correct, we should expect that about 18 million Chinese die every year.

But what is killing them?

Historically speaking, we see a lot of disease coming out of China. I seem to recall reading reports that our annual flu epidemics originate in China (though I have also read about some coming from India). But, also, apparently the bubonic plague originated there. This kind of thinking leads me to speculate a lot about the nature of Chinese philosophy and morals.

But this also leads me to speculate a lot about my own background as a "computer professional". Seriously, what good have my efforts done for anyone?

Not much, I'm afraid...

But why did I write this?

Well... the USA loses tens of thousands of people each year to the flu. Japan (which has better overall health care than the USA - life expectancy is longer in Japan) loses hundreds of thousands of people each year to the flu. And, once upon a time, I remember hearing that flu typically comes from China (we get a new version each year). But I have a vague memory that maybe it sometimes comes from India. But when I try looking that up, on the internet? I can't find any discussions of the topic.

This suggests, if nothing else, that we do not have free speech on the internet. We have free bs, but that is not quite the same thing. For something this big to be hushed up - hundreds of thousands of deaths each year in one country - millions each year when you extrapolate that to other countries (and who knows how bad it is at the source) to be a topic which there is not available information on - there have to be active efforts going on to distract people from the topic.

Perhaps fear of war (where, also, most of the deaths historically have been through disease but I'm not sure that the deaths have ever been on this scale as a sustained thing  - something like WWII probably this flu death rate for the five years it ran, but without actual numbers for flu mortality rates even that is a guess which might not be true).

Anyways: if this kind of thing is "not war" that can only be because this kind of thing is "worse than war".

Meanwhile, most all fundamental computer fabrication has been moved to China. The reasons for this are involved, but it's not entirely unrelated.

CS for All...

I commented on https://computinged.wordpress.com/2016/04/01/we-need-to-better-justify-cs-for-all/ and on reflection, I am not all that happy with the response I posted there.

There's some significant work needed to flesh out what we mean by "Computer Science for All", and I feel I could have done much better.

One issue, of course, for the computing education blog is: what does the curriculum look like? And there's just so much to cover.

But let's consider English literacy, and how we teach that: we do not cover the "great works" when teaching the language. We do not even cover more than a tiny fraction of the dictionary. Instead, we try and cover the "basics", and we keep going over those, layer upon layer.

Meanwhile, some significant number of students drop out. Students in inner city schools tend to wind up with a significantly different understanding of the language than students in farm schools or students in prep schools. And a part of this has to do with their social life and their relations with the other students. And another aspect has to do with the student's interests and the problems they see needing to be addressed in their life outside the school. And these obviously vary considerably.

So... back to "CS for All".  What are the basics?

As the first few passes, I'd say:

• binary number system (perhaps also octal and hexadecimal)
• ASCII character system (perhaps also getting slightly into Unicode).
• Sequences (of numbers, of characters, of sequences).
• Searching (needs sequences)
• Sorting (needs sequences)
• Files and Directories
• Programs and Processes
• Touch Typing  (qwerty)
• The basics of the internet protocol - enough to understand machine addresses and network delays.
• pixels and images
• basic audio (d2a, a2d, wav file format, oscilloscopes)

This is obviously very rough, and leaves out the wondrous complexities of html, online gaming, email, office software, and so on. And at first blush you might even think that I'm leaving out programming or cramming that all into "Programs". But, I'm not. I think programming comes in to some degree as a part of the treatment of these topics.

Also, this would not be all taught in one class, but would be the desired end point after years of elementary school and high school instruction. And a part of the emphasis should be on "learning by doing" because trying to get students to absorb all of this on a purely theoretical basis is just asking for problems.

More specifically, when we place an emphasis on programming as an isolated concept we lose all the motivation for why someone would want to learn such a thing. But, also, I think that some amount of programming should be brought in when teaching the above. And, perhaps, some actual formal programming instruction (syntax, for example) should be included.

And obviously there's room for improvement here, as well as room for making this all boring and superficial. 

Still, when I think about computer literacy, the above topics are - roughly speaking - the topics I assume a literate person would understand.

Of course, that leaves us with a huge problem: most of our teachers do not have the background for this. So, how do we get from here to there?

It seems to me that this will require teachers adopt a different role. Instead of being subject matter experts, they should be instruction experts. It seems to me that the job of the teacher should become teaching study skills to the student, and that we need curriculum designed to walk the student through the specifics of what they will be learning. We should stop demanding that teachers must be masters of everything and let them instead focus on what they do best: education as a specialty. (It is quite true that some subject matter expertise can help in relating the subject to the student, but it's also quite true that the instructors will pick this up some of this as they go along. This will be especially true for pre-requisite subjects which are covered frequently.)

This does leave us still with some problems, one of which is designing curriculum that satisfies these needs. But we have plenty of subject matter experts - we just need to put some effort into closing the loop (and some care in dealing with the problems which of course always arise when dealing with large numbers of people).

An Implementation of J

The Iverson Software folks have put up a free copy of their original book documenting J source code:

http://www.jsoftware.com/books/pdf/aioj.pdf

It's all pictures, so not searchable as text without OCR (but text search traditionally has been horrible in its treatment of J so there is some poetic justice here...).

Related:

http://r-nd-m.blogspot.com/2012/03/2-3-nb-five.html

http://r-nd-m.blogspot.com/2012/03/1-2-3-nb-six.html

(I had a few people tell me they wanted to see more of that kind of thing. I have not mustered enough round tuits to go there, but this is a step in that direction, and this book has been an important part of my internal understanding of the language implementation.)

word searching

So... I was reading http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/WSDM09-keynote.pdf and I ran across this bit:

Ways of Index Partitioning

• By doc: each shard has index for subset of docs
• pro: each shard can process queries independently 
• pro: easy to keep additional per-doc information 
• pro: network traffic (requests/responses) small 
• con: query has to be processed by each shard 
• con: O(K*N) disk seeks for K word query on N shards  
• By word: shard has subset of words for all docs 
• pro: K word query => handled by at most K shards 
• pro: O(K) disk seeks for K word query 
• con: much higher network bandwidth needed 
• data about each word for each matching doc must be collected in one place 
• con: harder to have per-doc information

Hopefully they have moved beyond that. I mean, it's not like they don't have the time nor the talent. Also, some of the search algorithms I have heard described strongly suggest word-based indexing.

That said, here's some thoughts:

1. Hybrid approaches are valid. You can speed up doc searches if you only search docs which contain the word. This isn't all that useful when the word is "the" but can be quite useful when the word is "drachma".
2. You can have word indexes which select documents, and you can have word indexes which select elements within documents (paragraphs, sentences, table cells, titles, ...).
3. You can have word indices which also track adjacent words.
4. Words themselves are an exact list of characters but you can build out lists of "this word likely means this list of words given this context" for people where that makes sense. Identifying contexts takes work, of course - and feedback.

If you are looking for a single word, having a canned search result for that word means the search is trivial - precached. That said, you could also think of this from an oxford english dictionary approach - where words tend to have many definitions and examples, all of which are slight variations on each other. Here, you try to give the user a way of selecting the context they are currently interested in. (For example: are they researching spam, are they dealing with biochemistry, are they searching for cute cat pics, are they building a bridge, etc...)

But when you have multiple words, it gets more interesting. Here, you can start with the rarest word, and use that as the basis for your search result, constraining it by the more frequent words. If you are doing a phrase search (which should maybe be the default - falling back first to words in the same document element and then if that fails to words in the same document and possibly if that fails to words linking to the document... (but note that these need some description to the user of what they're getting and how to fall back to the slower approaches - and you can impose a small search delay on the fallbacks to hint at the underlying mechanical costs)) and if you have adjacent words stored, you might be able to conduct the search entirely without even visiting indices for the most common words in the phrase.

For example "office of the president" has a specific meaning which is different from other meanings involving the words "office" and "president", but "of" and "the" are typically "stop words", but if every entry in the "office" word index recorded its word position in the document and the words immediately preceding and following it, and if all word indices were structured the same way, you would just need to query the indices for "office" and "president" to complete this search.

Bandwidth? Seriously, how was that measured?

That said, there are and will be costs to this - having to do with normalization rules and so on. In addition to logical "word position" you also almost certainly need physical "byte offset" or maybe "character position" for some of the UI related tasks. And, these word indices never exclude the need for document indices. So you wind up having both and several versions of both), with a certain amount of aging...

So, yeah, hopefully Google is already long past the stage of doing things this way. 

If I were Google I'd be having several different competing ways of representing word searches and I'd be watching each of them for signs that they are returning substandard results. (And, yeah, that would have to be a mix of metrics (counting) and samples (random inspection), and I'd be wanting to contrast current presentation with those from years ago - both for signs of stability and signs of current areas of progress.)

[I'd also want to be hooking up with people whose areas of focus are outside the computer realm - farmers, plumbers, etc. People doing tangible work which directly benefits other people. Not for their impulse quips, but just to make sure I had my head on straight.]

Libertarianism

I've been thinking a lot about Libertarianism.

In one sense, it's advocating personal responsibility as a solution to society's needs. There's a lot of truth to that, but I'm not seeing it happen on the scale it needs to happen.

I have heard Libertarianism expressed as "Let the market solve it" - the idea being that markets are efficient at solving problems. The problem, though, is that - mathematically speaking - markets probably are not efficient: http://arxiv.org/pdf/1002.2284.pdf

I have heard Libertarianism expressed as deregulation. The problem with that, though, is that often the wrong things get deregulated.

Specifically: dollars are a construct of the government. Without the system of government regulations which demand their use for some things and place various restrictions on their use, dollars are just numbers. And if that is what you want, it's not clear why government should need to make any changes - we already have the ability to use numbers.

But we could also think of Libertarianism as a part of a reasonable system of checks and balances. When we think of questions like "who watches the watchers", Libertarianism apparently says: as few people as possible, let's just have police and military and get rid of courts, libraries, food quality regulations, nasa, college, school, etc. etc.

Um...

So, ok, there's some validity here. But what this demands is that (a) everyone work incredibly hard at making things right, and (b) that we put up with a lot of nastiness. Of course, to some degree we do that already.

I guess the problem is: taken to its logical extreme, libertarianism would be one of the most repressive forms of government you'd ever heard of.

We get to keep:

* Police (not much)

* Military (not much)

* Crime (um wait...)

So what's the distinction between that and what we've got now? The argument goes that our government is criminal. And, looking at incarceration rates, and various other problems such as civil forfeiture, we do indeed have big problems.

But those sorts of problems don't just go away. Those problems are people problems.

But if Libertarianism is the answer you don't get to the solution by opposing government action. Getting rid of libraries and regulations prohibiting the sale of rotten food is not going to make anything better. You get to the solution by helping people out as much as you can.

Put differently: "big government" is not just votes, but it's also dollars. If you get rid of government you get rid of dollars. But if that's going to work you have to be able to make things work - as much as possible - without dollars. And, to be frank, we need a lot of that kind of activity.

We have some sizable percent of our population apparently "idle" - not working, not employed. Or at least, not in the sense of exchanging their time for dollars. Libertarianism is valid to the degree that it can get work done without using dollars. And there's some significant examples of that, of course.

You could say that working for no pay as the ultimate expression of Capitalism - it's the limit condition of competitiveness. And setting up shelters for people, Feeding people, making sure they have productive uses for their time? That's all good.

But what this really means is that Libertarianism is advocating the idea that we ignore the economy - let the 1% have all the money, it's irrelevant. Riches are nothing to do with dollars, and people should have the right to solve problems without it.

Just don't expect it to be all pleasantness and beauty. It's going to be hard work.

Of course, this kind of thinking has relevance not only in the context of libertarianism but in the context of economics and of government and politics also. None of them are particularly valid or meaningful when taken in pure form. These lines of thought need context and connections before they can be thought of as valid.

It's also going to involve letting lots of people in the rest of the world kill each other. It's asking us to stand by and watch while people kill, maim and torture each other. It means not being afraid while nuclear weapons are developed and deployed. Or maybe it means the opposite. Maybe it means intervening using contractors and ... wait, isn't that what we're currently doing?

(And, I know - read this book... but what do I do about the mistakes in that book?)

Anyways, it's a head scratcher for me.

I agree with a lot of the principles advocated by Libertarianists. But at the same time, I'm not seeing the sorts of action taking place which those principles would suggest need to happen.

Yes, we need solutions to problems which government is not addressing, and which the dollar-based economy is not addressing. But you're not going to get those solutions using government activity nor using the economy, except in some sort of minimal sense. You're going to get those solutions by (a) getting people to cooperate and solve their problems, and (b) showing that you've done so and showing other people how to do it.

And a few people are doing that. But - personally - where I live I don't even know how to find people that need problems solved of the sort where I feel I can usefully contribute.

How about you?

The "boolean" disease

I was just reading the linux kernel coding style guide, and I am once again frustrated by how poorly people understand Boolean algebra.

Here's the offending quote:

Functions can return values of many different kinds, and one of the most common is a value indicating whether the function succeeded or failed.  Such a value can be represented as an error-code integer (-Exxx = failure, 0 = success) or a "succeeded" boolean (0 = failure, non-zero = success).

Mixing up these two sorts of representations is a fertile source of
difficult-to-find bugs.  If the C language included a strong distinction between integers and booleans then the compiler would find these mistakes for us... but it doesn't.

 So what's wrong with that?

It's essentially a failure of the entire educational system that a statement like that goes unchallenged.

Here's an example of Boolean multiplication on integers:
┌──┬─────────────────────────────┐
│*.│ _4  _3 _2 _1 0  1  2   3   4│
├──┼─────────────────────────────┤
│_4│  4  12  4  4 0 _4 _4 _12  _4│
│_3│ 12   3  6  3 0 _3 _6  _3 _12│
│_2│  4   6  2  2 0 _2 _2  _6  _4│
│_1│  4   3  2  1 0 _1 _2  _3  _4│
│ 0│  0   0  0  0 0  0  0   0   0│
│ 1│ _4  _3 _2 _1 0  1  2   3   4│
│ 2│ _4  _6 _2 _2 0  2  2   6   4│
│ 3│_12  _3 _6 _3 0  3  6   3  12│
│ 4│ _4 _12 _4 _4 0  4  4  12   4│
└──┴─────────────────────────────┘

And here's an example of Boolean addition on integers:
┌──┬─────────────────────┐
│+.│_4 _3 _2 _1 0 1 2 3 4│
├──┼─────────────────────┤
│_4│ 4  1  2  1 4 1 2 1 4│
│_3│ 1  3  1  1 3 1 1 3 1│
│_2│ 2  1  2  1 2 1 2 1 2│
│_1│ 1  1  1  1 1 1 1 1 1│
│ 0│ 4  3  2  1 0 1 2 3 4│
│ 1│ 1  1  1  1 1 1 1 1 1│
│ 2│ 2  1  2  1 2 1 2 1 2│
│ 3│ 1  3  1  1 3 1 1 3 1│
│ 4│ 4  1  2  1 4 1 2 1 4│
└──┴─────────────────────┘

Notice how, if we restrict ourselves to the 0 and 1 we get "binary and" and "binary or".

Anyways, somewhere along the line, someone (probably Henry Sheffer) decided that "logical not" should be a part of boolean algebra. Essentially, it's "proof by vigorous assertion". Algebras existed before Henry Sheffer, and Boole's work existed before Henry. Henry's work proved an equivalence between logic and Boole's system - for the case where we restrict ourselves to 0 and 1.

But what do we gain from this?

Answer: not much.

Meanwhile, there's another way of formulating logic using 0 and 1 which gracefully incorporates the idea of logical negation:

Instead of using Boole's multiplication (which is defined on integers) we can instead use regular multiplication. And, for logical not, we can use 1-x. And that's all we need, we can use Henry Sheffer's approach to define logical or: x OR y is 1-((1-x) * (1-y)).

And what do we gain from this?

Answer: probability and logic can be seen as two facets of the same system.

Something like this:

┌───┬──────────────────────────────────────────────────┐
│*  │0 0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1  │
├───┼──────────────────────────────────────────────────┤
│0  │0 0    0    0    0    0    0    0    0    0    0  │
│0.1│0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1│
│0.2│0 0.02 0.04 0.06 0.08 0.1  0.12 0.14 0.16 0.18 0.2│
│0.3│0 0.03 0.06 0.09 0.12 0.15 0.18 0.21 0.24 0.27 0.3│
│0.4│0 0.04 0.08 0.12 0.16 0.2  0.24 0.28 0.32 0.36 0.4│
│0.5│0 0.05 0.1  0.15 0.2  0.25 0.3  0.35 0.4  0.45 0.5│
│0.6│0 0.06 0.12 0.18 0.24 0.3  0.36 0.42 0.48 0.54 0.6│
│0.7│0 0.07 0.14 0.21 0.28 0.35 0.42 0.49 0.56 0.63 0.7│
│0.8│0 0.08 0.16 0.24 0.32 0.4  0.48 0.56 0.64 0.72 0.8│
│0.9│0 0.09 0.18 0.27 0.36 0.45 0.54 0.63 0.72 0.81 0.9│
│1  │0 0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1  │
└───┴──────────────────────────────────────────────────┘

┌───┬──────────────────────────────────────────────────┐
│or │0   0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1│
├───┼──────────────────────────────────────────────────┤
│0  │0   0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1│
│0.1│0.1 0.19 0.28 0.37 0.46 0.55 0.64 0.73 0.82 0.91 1│
│0.2│0.2 0.28 0.36 0.44 0.52 0.6  0.68 0.76 0.84 0.92 1│
│0.3│0.3 0.37 0.44 0.51 0.58 0.65 0.72 0.79 0.86 0.93 1│
│0.4│0.4 0.46 0.52 0.58 0.64 0.7  0.76 0.82 0.88 0.94 1│
│0.5│0.5 0.55 0.6  0.65 0.7  0.75 0.8  0.85 0.9  0.95 1│
│0.6│0.6 0.64 0.68 0.72 0.76 0.8  0.84 0.88 0.92 0.96 1│
│0.7│0.7 0.73 0.76 0.79 0.82 0.85 0.88 0.91 0.94 0.97 1│
│0.8│0.8 0.82 0.84 0.86 0.88 0.9  0.92 0.94 0.96 0.98 1│
│0.9│0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1│
│1  │1   1    1    1    1    1    1    1    1    1    1│
└───┴──────────────────────────────────────────────────┘

So why the mixup?

In part: people are lazy. I know I'm pretty lazy about this issue, and sometimes careless of how I research it. But I'm certainly not alone. People mostly just don't care. And *that* says a lot about the value of education.

Education has a lot to do with satisfying the teacher, and not so much about historical accuracy. People like to refer to names that most people don't bother looking up (who has the time?). And of course, this suggests that educational literature is rife with mistakes - most of it is crud (but not all).

Still, there's the issue of naming. If "boolean" is the wrong name for the concept people are getting at when they talk about distinguishing between "booleans and integers" what are some good names?

One obvious name might be "logical". That's closer to accurate than "boolean". And that leads to "true" and "false" being satisfying names for variables of this type. In the above kernel style guide excerpt, I think it would be good to replace "boolean" with "logical value".

Another obvious name might be "bit". And a variable of type "bit" really ought to be able to represent the values 0 and 1.

But that won't stop fans of "proof by vigorous assertion", will it? For them, historical accuracy and utility take back seat to vehemence. It's actually kind of amusing to watch.

Resume again

I don't really need more work. But it's addictive. So I've been thinking about that alot.

Here's the position writeup for Gannett - they're looking for someone to replace me. It's nice to feel missed, in a sort of sad way. (I miss my friends there, and more than occasionally regret deciding that it was time for me to move on.)

Anyways, here's their writeup for my old position:



Principal Developer, Development & Integrations

The principal developer oversees the development and implementation of code for highly complex software applications.  They have high-impact responsibilities within the team and are typically assigned to the most critical and strategic / complex / high-risk undertakings. A principal is responsible for architecting the systems that make up the framework the rest of the organization works with and builds upon.

A principal must ensure the integrity and durability of the code base by both protecting it from substandard changes and by designing defensively to mitigate user error and abuse. As a system designer, a principal must also educate and mentor the team that makes use of these systems to ensure adoption and also to identify gaps in the system. The principal is well versed on the latest advancements in his field and regularly evaluates new trends and technologies to improve the framework.

Responsibilities:
Define and enforce overarching architecture strategies
Identify high-level architectural issues and document/develop solutions
Review and merge high-risk changes
Develop solutions to mitigate user error (eg. automation, validation, etc.)
Mentor and develop best practices for developers within the team
Regularly ideate and develop new high-impact innovation projects
Provide team training on architecture, strategy, and technology
Define best practices and guidelines for lower level developers
Work with other disciplines to help shape new products from a technical perspective

 Required Technical Skills
Bachelor’s degree in Computer Science, Computer Engineering or related technical discipline. Masters degree a plus.
9+ total years of software development and architecture experience with at least 5+ years designing and architecting enterprise web applications.
Experience working directly with senior Development and IT groups in an advisory role.
Strong object-oriented design and development skills.
Expertise in multiple technologies (JavaScript, jQuery, Backbone.js, Ajax, HTML5, Django, Python, etc.) across software engineering, security, data management, etc.
Experience in integrating with 3rd party REST API’s.
Experience with Agile/Lean methodologies, continuous integration, GitHub, Git, Test Driven Development.

Upstart

I had an occasion to deal with "upstart" last week.

Upstart is a project which replaces "init". The design intent is pretty obvious: allow independent packages to have components which run at system boot. The expressed intent is somewhat different: add new "scanning" functionality to init (https://en.wikipedia.org/wiki/Upstart#Rationale).

Wait, what?

In other words, upstart has reduced functionality from sysv init. It no longer supports /etc/inittab and that's probably a good thing. Simplicity is a virtue. But documentation is hard to find and with all of its modules and added features (it also has added features) it can take some study before you can do anything useful with it.

A typical use for /etc/inittab was: you add a line specifying some program to run, and init runs it for you. If the program exits, init will restart it for you. Nice and simple.

For example you might add a line like this:

work:23:respawn:/bin/su username -c /home/username/bin/dowork

This would run /home/username/bin/dowork as the user 'username'. There's more details here, but that's the gist of it. And if the program exits, init would start it up again (if you read the man page for inittab you can see all sorts of details about this kind of thing).

Well, with upstart, there's more you have to do.

First, you have to create a configuration file such as /etc/init/work.conf

For this example that file might contain:
respawn
exec /bin/su username -c /home/username/bin/dowork

And then you also need to somehow get this started at machine boot. One possibility is to add a line to /etc/rc.local like:

service work start

Another possibility involves creating /etc/init.d/work (which might be the same) and then symbolically linking it from /etc/rc2.d/S99work and so on... You can get fancy here and create start/stop/restart commands. You probably do not want to get into this kind of complexity unless you are building packages for other people to use.

And, of course, there's lots more possibilities.

But upstart currently is not documented very well, in my opinion. Specifically, there is nothing in the manual page for upstart which even hints at the need to run the service program at boot time to achieve what init does automatically. Instead, the documentation reads like it would start the service automatically when a conf file was placed in /etc/init/. (And that might have been true for some versions of upstart - but it's not true for the version I'm using in ubuntu version 13.)

And this is something that both the free software communities and the professional software communities need to get better at: documenting things properly. And maybe this is something colleges need to do a better job of teaching.

It's really hard, as a developer, to think from the viewpoint of the user. And it's really easy for the user to not connect with the developer (note that I've not connected with any upstart developers about this issue - at least, not yet). So this is a difficult issue to think about, sometimes.

Google Search

I've been concerned, recently, about google search results. They've gotten... worse, much worse, than they used to be.

I'm also looking at google from a variety of other perspectives, as a user and as a potential employee. There's still a lot of good stuff there, but it looks like whoever is managing search either

(a) Has forgotten about long tail issues, or
(b) Is taking the concept of "fail as good business" too seriously, or
(c) Prefers other search companies to take the lead, or
(d) Is an idiot, or
(e) Is reacting to regulatory pressure, or
(f) Are overwhelmed by spam, or
(g) Something else...

Yeah, basically, I have no clue what is going on. But I am presuming some combination of some of the above.

But, it's pretty clear that if I want good search results I should be using other search engines, and falling back on google search only when my habits get the best of me.

And we should probably expect something similar from google's other services - and maybe that is the point. (Though, more likely, this is the result of "professional management" in the sense of "preventing work and taking money for the privilege of having presented the concept in an impressive fashion".)

On the other hand, philosophically speaking,  anyone working with computer systems needs to have contingency and backup plans.

Still, I am not sure how to view this - are they crying out for my help, or are they warning me to stay away (obviously neither of those, in a very real sense, but maybe you get the gist of how I need to think about this in terms of my own future plans).

Maybe it's time for me to give up on computers as a career path, and take up farming (in the sense of growing stuff out of the ground)?

Meanwhile, I'll point out that the failure of google search is something of a marketing and management failure as well as a technical failure.

As I understand it, one of the roots of the issue has been a focus on the first search result as a metric of success, and a focus on search speed as a metric of quality. And these are indeed important goals. But they cannot sustain the company by themselves, and while breadth of search is available in a variety of ways, those extra "useless" search hits provided a variety of useful information about potential related searches, and about the communities of information that one had access to.

But another issue is spam. To pursue its goals, Google needs to deal with money. And money - like any other human endeavor - leads to too much of some things and not enough of others. In other words, "spam". There's a huge element of people trying arbitrary things until they find something that works for them. And, another issue is that one person's good idea is another person's bad idea.

Personally, I'd take spam as early efforts at artificial intelligence - a subject worthy of study and research. And, of course, a good way of attracting attention to yourself if you are controlling or distributing it. (Pro tip: do not annoy too many people unless you actually want to attract the attention of the really violent types.)

Google has started to take a stab at this issue with google groups. Google groups makes a fair set of assumptions about how people organize their lives and their information. But somewhere in the process a lot of information that used to be available is no longer there.

Anyways, what Google currently conveys to me is that they do not have a clue and that companies that they service do not have a clue. And this is silly because the people within Google definitely have clues. So how is it that the corporation does not? This is something of a business opportunity - now is probably a good time to cash in on all the clueless businesses - but it's also something of a warning of things to come.

Good investment opportunities for the long term are probably not computer related at all, and probably not money related at all. Instead, make sure you have direct contracts with farmers and construction outfits and so on. If you want a safe future you probably need to make arrangements to bypass monetary transactions entirely and have direct access to the underlying services you need. This means food, water, something to keep you relatively dry, something to keep you warm, and so on.

Money, meanwhile, is looking more and more like it's mostly only going to be good for paying taxes.

I sure hope I am wrong about this, and that I am just being grumpy.