Alt-BEAM Archive
Message #00990
To: beam@corp.sgi.com
From: Terry Newton wtnewton@nc5.infi.net
Date: Thu, 25 Feb 1999 12:07:33
Subject: [alt-beam] Another idea for Associative Memory
An idea for a hard-wired RL "brain" - use a static ram (2114 etc)
and a counter fed by an oscillator (to serve as a random move maker).
Reenforcement Learning (RL) works by keeping the moves that work and
randomising memories that don't. The read/write line of the ram and
the random move enable (4066 whatever) become the learn input, pulse
and the memory changes. When to pulse determines the behavior, for
example wire up diodes from the "bad" inputs, use similar diode logic
for output conditions that are not desirable. A slow oscillator can
provide repeated learning pulses to simplify the behavior logic.
Something like...
________ ______
Left feeler ------|a d|------------*------| |--M--. for L
Left light ------|a 2114 d|----------*-|------| |-----' rev L
Center light ------|a 6116 d|--------*-|-|------| 245 |-----. for R
Right light ------|a etc d|------*-|-|-|------|______|--M--' rev R
Right feeler ------|a______w|o-. _|_|_|_|_
| | 4066 | (or resistors ?)
.-*--------. __ + -100K--* |_________|----------*----------.
| | __ `-| \ .1u| _|_|_|_|_ __ | |
| `-| \ | )o-*-||--* | counter | / |-' __ |
| | )o-*-|__/ _|_ | |_________|-*-o( | / |---*
| .-|__/ | -.-2.2u| _|_ \__|-*-o( | |
| |slow osc`--220K--* ? | 100p -.- | \__|-. |
| `-----------------' | ? *---100K-'fast osc| |
`-------------------. | `-----------------' |
| `--|>o----------------------------'
|
learn signal
low=good move, high=bad move
Bigtime guessing on the parts values. Not tested, just an idea.
To get it to do what you want you really tell it what you don't want,
my base rules are try not to touch the feelers for too long and try
not to go in reverse or stand still when the feelers are not touching,
then do something about spinning in circles (an efficient way to
satisfy the first two conditions), phototropic or some other way.
Assuming positive logic on all inputs (use Nu to condition etc)
learn if either feeler goes high for too long, and when feelers
are both low learn if both motor forwards are low or if either motor
reverse is high. For phototropism, learn if left light is high too
long while motor forward left is high, and same for right.
A version of this was adopted for my speculative walker brains page.
In general directly driving motors as shown produces more reliable
motion and should allow for more reliable learning than controlling
a microcore walker, not necessarily better (can we tell?). More than
4 outputs are really needed for walker use, at least 5. A 6116 provides
8 outs, 4 bit counters are duals (string 'em together) and 8 bit buss
chips are common. Buss switching might not be not needed, resistors
can do the job of only feeding data when the memory is accepting.
With a 6116 cmos low power memory, cmos switching is more efficient
unless the resistors made high enough not to draw too much current
from every opposing output. Resistors limit the maximum clock speed,
which has to be high enough to ensure thermal drift etc in the caps
will produce moves that are random enough (or run the counter all
the time instead of gating). With a 2114 the extra drain wouldn't
matter much, the buss switch (4066 as I have it) could probably be
replaced with say 10K-47K resistors. An 8 bit counter chip with an
output enable pin would be an ideal solution if such a thing exists.
More circuitry is needed to pre-process the senses and trigger when
to learn, even with the extra stuff it should fit in 4-6 chips.
A thought... delay and feed extra bits to unused address lines to
encourage a notion of sequence to the memories instead of just pure
reaction. Just a few or learning time will increase too much.
Terry Newton
------------------------------------------------------------------------
eGroup home: http://www.eGroups.com/list/alt-beam
Free Web-based e-mail groups by eGroups.com
Home