Anthony M Harrison

Last week marked the first major deadline that I’ve had here at NRL. The boss man was presenting our work at a social cognition workshop that was populated by cognitive and developmental psychologists plus some roboticists. From his report, it was an interesting interchange.

Our push leading up to it were some model fits of the monkey models plus demos of the robot running the models. The fits didn’t happen (bug in common reality that I’m working on currently), but the movies did (and should be posted soon). The final push towards the deadline has highlighted a few architectural divergences between jACT-R and ACT-R proper that I need to address. Those differences that are clearly mistakes will be patched, but those that are architectural decisions will likely be parameterized.

The post-workshop debriefing has produced some interesting discussions around here regarding the nature of theory of mind and meta-cognition more generally. Some of my early brain-storming regarding concurrent protocols seems like it really will set the stage for a general meta-cognitive capacity. Production compilation, threaded cognition, and a carefully designed declarative task description can get us really close to this goal. However, I suspect there are two pieces that need to be developed: variablized slot names and the ability to incrementally assemble a chunk specification (i.e. module request). I throw out this tantalizing bit with the promise that I will post a very lengthy consideration of this issue in the immediate future (I need to put together the pieces that currently exist and see how it plays out).

My insomnia development of the production static analyzer and visualizer is coming along nicely. Below are two screenshots from the analysis of the handedness model (which uses perspective taking to figure out which hand another person is holding up). The first shows all the positively related productions and their directions (i.e. which production can possibly follow which), with the selected production in yellow and its possible predecessors in blue and successors in green. The icons in the top-right corner show the filters for showing all, sequence, previous and next productions, plus positive, negative and ambiguous relationships. There are also configurations for layouts and zoom levels. Double clicking the production will switch the view to sequence which filters out all the other productions that are not directly related to the focus, across a configurable depth (1 currently).

I’m still not handling chunktype relationships, and I also need to provide buffer+chunktype transformations (i.e. +visual> isa move-attention doesn’t place a move-attention into the visual buffer, but a visual-object instead). Once those are in place, I’ll add a chunktype filter to the visualizer so that you can focus on all productions that depend upon a specific chunktype, which will help really complex models (the monkey model, with all of its contingency and error handling is around 90 productions, all nicely layered by hierarchical goals, but that’s still a boat load of productions to have to visualize at once)

I’m hoping to get a bunch of this menial development and fixes done on the flight to/fro Amsterdam for HRI. If all goes well, there should be a groovy new release in two weeks.

I’ve previously mentioned that I’m working on a static analysis tool for productions. Basically it gives you a score that reflects the match between two productions based on their conditions and actions. This allows you to visualize their relationships without ever having to run the model. A huge chunk of the basic logic is done and thanks to the joys of open source tools and my new favorite project, Zest, after a day of tinkering I’ve got a basic (but far from ideal) visualization tool:

Here we’ve got the unit one semantic model showing one of the three different view types: following. This shows which productions can follow the chain-category production. You can also view proceeding productions (those leading into chain-category, which only shows initial-retrieve in green), or All (showing all the productions and all their relationships).

It’s still not perfect. It’s not handling chunk type tests (simple addition), nor variables (bloody impossible without a hypothesis testing tool too), nor more comparison slot tests (less-than, etc). But all in all, pretty damned cool and useful.

In the weeks that have passed since I last posted on my work and ideas much has happened in the world of interfacing ACT-R with robots. As of Friday, jACT-R was piloting our in house robot, George. Yeah, there were a few glitches, but damn was it cool. The integration of jACT-R with actual robots and simulated has illustrated many interesting issues with respect to modeling in realtime domains (and I’ll use the softer definition of being able run within a 50ms cycle time on average, not deterministically). I’m thinking about putting together a talk for the ACT-R workshop this year on the topic. Could be interesting.

The test bed has been a social learning model (two actually, otherwise it wouldn’t be very social) of a chimp learning what food is safe to pursue in the presence of an alpha male. It’s quite entertaining. Two robo-chimps enter a space. The dominant will always pursue whatever food it can see and if a subordinate gets too close, the dominant will smack it. The subordinate needs to learn what food is safe to pursue based on distance and its visibility to the dominant. I’ve got the models up and running and interacting with each other in parallel (trivial for jACT-R, but not the Lisp) and the models are actually running faster than playerstage (i.e. they spend most of their time waiting for the simulation to advance the clock and send sensor data). The subordinate engages in perspective taking and can make the critical go/no-go decisions. The only problem is that the dominant has suddenly decided to pacifist on me. It refuses to smack the subordinate.

Much fun. The challenges that this work have presented have resulted in some serious improvements to jACT-R, to the point that I will state outright that I believe jACT-R is the better option if you need to work with real systems. Granted, it’s still not feature complete (compared to the Lisp) and there are still significant gaps in the theoretical validations - but all in all, she’s a fast and easily (and safely) extended system.

One element that I’ve been working on after hours has been a static production analysis tool. It hooks into the model compiler (unlike the Lisp which builds a live model, jACT-R models are entirely declarative and need to be parsed, compiled and built), and examines the possible end states of productions. From this it can generate probable execution paths. If you’ve ever had to read someone else’s model (hell, even your own after a month away from it) you’ll see the obvious appeal of such a tool. This would be particularly useful for inspecting a model after an extended run with production compilation enabled. Those productions can get down-right strange after enough time. I’m hoping to have a function GUI for this beast to show this summer. Hopefully I’ll be doing a tutorial at the ACT-R workshop to introduce folks to the whole system. A coming out party, if you will.

Anyway. Now back to the barrel and inject some aggression into those monkeys.

There’s been an idea percolating in the back of my mind for the past few weeks. It all started with the boss’s boss, Alan Schultz’s desire for a cognitively plausible mechanism for robots to explain what (and presumably why) they are doing what they are doing. While for Alan this was a practical desire, it struck me that from a cognitive psychology perspective what he was asking for was actually much deeper and more interesting.

What he really wanted was to have robots engage in concurrent verbal protocols (Ericsson & Simon, 1993). This is a general task that asks people to describe what they are doing while they are doing it with as little filtering or elaboration as possible. The idea is that these utterances represent the most basic description of the contents of working memory in service of the current goal. A great introspective tool (more so that retrospective protocols at least).

From a cognitive modeling perspective the challenge is to implement it in a general manner so that one can use the same productions across multiple models, regardless of goal structures. This means that any model of concurrent protocols should not be dependent upon interruptions of the current goal, but should operate in parallel (hence concurrent).

I do believe that I have a solution. Not surprisingly it relies upon one of my favorite contributed ACT-R modules, threaded cognitiion. Combined with the learning from instruction work that John and Niels have done, an interesting new system emerges.

Extending threaded cognition

Threaded cognition allows a model to engage in multiple goals at once, interleaving them at the production level. However, there is a level of isolation that prevents individual goals from being aware of each other (a good thing). For concurrent verbal protocols to work, there needs to be a mechanism to get at the other goal(s). A modest proposal is to add a query such as this:

(p what-else-am-i-doing
  =goal>
    isa verbal-protocol
    ....
  ?goal>
    other-goal =otherGoal
  ==>
   +retrieval>
    =otherGoal
   !output! (Im also doing =otherGoal))

The example is idiotic, but the idea is that when querying the goal buffer, you can request a reference to the other goal, which will bind to any goal in the buffer that is not the one that fired this production. With the other goal in hand, one can then begin to inspect it.

Learning from instruction

If the goal was learned via the learning from instruction work, you’d have a goal with a state slot. This state slot can then be used to retrieve from declarative memory the instructions that preceded and succeeds this one. In other words, you can launch a retrieval which will describe what it is doing. If one extended the learning from instruction work to include meta information for the justification for that instruction, you now have the ability to query the model for what and why it is doing what it is doing.

Levels of description

Of course, this will likely result in protocols that are very fine in their granularity. The states from learning from instruction are at the unit task level (i.e. reading a letter, pressing a button). This is not really that useful. If one extended the learning from instruction work to include progressively deepening goals (ala standard task analyses), the concurrent verbal protocols could then prefer to report at the level just above unit tasks:

(p avoid-unit-tasks
  =goal>
   isa verbal-protocol
   ...
  =retrieval>
   isa unit-task
   parent =parent
 ==>
  +retrieval>
   =parent)

One could then also query the model for more or less detail where it would then chain up or down the goal.

Beyond just a tool

In discussing this brainstorming, Greg Trafton realized that it may be more than just a damned useful tool. There may be genuine predictions and fits that can be evaluated with respect to the verbal protocol methodologies. Not only does this solution present a potential multi-tasking cost to performance of the primary goal, but the simple act of engaging in the protocols would change the state of the model.

With production compilation engaged, repeated protocols would become much faster as the instruction retrievals are compiled out (just as they were when learning how to do the core task). But more interestingly, the act of accessing the other goal via the retrieval buffer would lay down a memory trace for the partially completed goals. Without the concurrent protocol, the intermediate steps of the goal would be lost (or rather, never encoded). These partial goals may make further explanations easier, strengthen the instructions, provide further compilation opportunities, and (potentially) result in the improved learning that is seen when subjects are asked to explain their actions while learning a new task.

One could also look at the different predictions regarding concurrent and retrospective protocols.

What started as a potential solution to a practical problem, when situated within ACT-R, is now looking like a genuine theoretical construct that may get some serious mileage. Gotta love it when an idea germinates into something so much greater than expected.

Alrighty, the first functional commit of the new general motor module is in place. I have to say, I’m quite pleased with it. The implementation is general enough that it can be easily extended or adapted without having to fuss with threading or common reality interfacing. In fact I will be refactoring the vocal module to use it as well (although as a separate module).

The implementation includes a basic keyboard/mouse device on the common reality side (although the mouse elements are sketchy). While not all the motor commands are currently implemented (only punch, peck, peck-recoil), adding the remaining should be relatively straight forward. [code is in the repository, I have not uploaded to the update site yet]

In terms of major basic functionality gaps on the /PM side, there is just the missing SwingSensor that makes java guis visually available. I have no idea when I’ll get to that (any volunteers?) as my next focus is the embedded work I’m doing here at NRL.

The current goal is to get cracking on the compatibility tests over the winter holiday. Ideally, I’d like to have a fully verified compatible version for the next workshop this summer.

Last week the basic robotic visual system interface with jACT-R was finished and running fairly well (a tad inefficient, but.. whatever). Before launching into the final piece, motor control, I had to implement a long neglected issue: how to communicate efferent information from jACT-R to CommonReality.

Well, the extended weekend gave me some quality pondering time and yesterday I was able to whip up a viable solution. As a quick test-bed for it, I decided to implement a more general sensor/module pair (since motor control is necessarily yoked to a specific device) : the speech system.

There is now a general purpose speech system (org.commonreality.sensors.speech.DefaultSpeechSensor) and jACT-R’s vocal module (org.jactr.modules.pm.vocal.six.DefaultVocalModule6). Rock on.

Up next, some basic motor control and then I can get back to migrating the dissertation model to the robotics platform and then start carving up the dissertation for publication.

One thing I really like about working here at NRL is that they bring in researchers. It’s just like university research where you can actually keep abreast of current work. This morning we had David Kieras, father of EPIC, talk about some issues surrounding active visual systems.

One of the major differences between EPIC and ACT-R is that the former focuses much more heavily on embodied cognition and the interactions between perception and action (ACT-R’s embodiment system was largely modeled on EPIC’s).

Not surprisingly, one of the big issues with cognitive robotics is the perception/action link and how far up/down do we model it. Taking driving for instance, or rather, the simpler task of staying on the road. One could postulate a series of complex cognitive operations taking speed and position into account in order to adjust the steering wheel appropriately. Early drivers seem to do something like this, but experienced drivers rely on a much simpler solution. By merely tracking to visual points in space, one can just use an active monitoring process to make continual, minor adjustments.

In ACT-R this is modeled with rather simplistic goal-neutral productions (there are goals, but they aren’t really doing much cognitive processing) - it’s just a perception/action cycle. EPIC would accomplish it in a similar manner, but since productions fire in parallel, EPIC would be better able to interleave multiple tasks while driving (but the new threaded extension to ACT-R would accomplish something similar).

If we take embodiment seriously, then we have to ask how much of each task can (at an expert level of performance) be decomposed into these perceptual/action loops? and do these loops function as self monitoring systems without any need for conscious control (as ACT-R currently models it).

Let’s look at a visual navigation task - although navigation is probably a poor term. This is moving from one point to another where the destination is clearly visible. There isn’t much navigation, perhaps some obstacle avoidance, but certainly not “navigation” in the cognitive-mapping sense. Anyway..

A cognitive solution would be to extract the bearing and distance to the target, prepare a motor movement, and execute it. If something goes wrong, handle it. A perceptual/action loop solution would be much simpler, move towards the target, adjusting your bearing to maintain the object in the center of the FOV and stop once close enough.

The robot utilizes the first solution: it takes the bearing and distance to the target and off-loads the actual work to a SLAM-like processing module on the robot that does the navigation. Once it arrives, it notifies the model and all is good. This lets the model perform other work with no problems.. but the perceptual/action loop seems a more appropriate route. The challenge is in how much processing the model is performing. It’s work should be relatively limited so that it can perform other tasks..

Hmmm.. more thought is necessary.

The player/stage — CommonReality bridge is coming along nicely. I was able to start on it yesterday (finally my machine is on the network), and quickly got CR controlling very basic things. Today I got the execution and threading done so that sensors can be processed in parallel. Hell, I’ve even got a test model that is attending to the objects that its seeing in the environment. Very cool.

There do appear to be a handful of issues with respect to the generation of unique simulation objects, but it shouldn’t be too challenging to resolve. Tomorrow I should be able to start considering how to make it all work with the dissertation model.

I have to say this has gone much easier than expected. I do, however, have to figure out a more robust way to handle token/type/value relationships. While I could use the blobfinder and just assign each element to a unique channel (ugh), I think a better route would be to use the fiducial simulator which can give me an identifier and an orientation. This should just be as easy as building another ISensorProcessor (much like the one for blobs) - but I’m afraid I may have to use some ghettotastic sensor fusion to get this to work.

I’m thinking I’ll turn this process into a multi-part article for jactr. Speaking of which, I really ought to get the install instructions uploaded..

So, I couldn’t help it. I upgraded to Leopard. Overall, I have to agree with the major consensus: very solid, not a huge leap, but definitely worth the upgrade. Time Machine more than pays for the upgrade.

One problem though. I’d gotten Player/Stage set up for the modeling but that was on Tiger, so it’s now broken. Even worse, the install of the developer tools got screwed up somehow, so now I can’t even reinstall the MacPorts without first reinstalling the developer tools. Hopefully it will be resolved this evening.

knock on wood.

Things have been quiet as I have been getting acclimated to the new job at NRL. Due to government efficiencies, after two weeks I still don’t have a computer. While it had been ordered by my boss a month before I arrived, procurement didn’t order it until a week after I’d arrived. Supposedly it has been delivered, and so now it is “days, not weeks” until I will have it.

Anyway, a plan of attack has been assembled for the next few months. The first order of business is to get the long awaited Player/Stage simulator up and running so that ACT-R/S can actually operate within a dynamic spatial environment.

Up next will be building a model of someone else’s data and then migrating some preexisting robotic models to use ACT-R/S. Basically these first few months are slated for engineering problems as well as getting my feet wet with the current work of the others in the lab. This is fine by me, as the subsequent research path still needs to be considered in greater depth.

The spare time will also be filled with getting the jACT-R site back up and running with documentation and examples. The code base also needs some updating to handle the latest changes to Eclipse and its new execution environment (for macs).

Now that I’m back in the flow, look for weekly updates both here and at jACT-R.