Microgenics

ASSIGNMENT

Microgenics was a successful developer of immuno-assays. They had decided to expand into instrument development by buying a startup company that was developing a desktop general chemistry analyzer. The instrument under development did not have its own sample loader but was connected to a commercial general-purpose sample handler. Microgenics decided to create an integrated analyzer with sample handler by combining the analyzer under development with a copy of the sample handler.

The project had been going for two years when I was hired as the lead firmware engineer. Before taking the job, I examined the design and interviewed the engineers responsible for it. The architecture was unusual and the engineers had no explanations. I agreed to take the job and to do whatever I could without changing the hardware.

DESIGN PROCESS

The design that I inherited merged the analyzer and loader controllers into a single system. Each original unit was a typical single-board Z80-based controller. The merged design used multiple boards and a card cage. The two Z80s were moved to one board. Each CPU had its own independent system bus that went to the card-edge connector. Every other board in the system continued this same architecture. Essentially, the two systems were simply glued together, with no attempt made to share or normalize anything. This much I had seen in my initial inspection. I didn’t like it but I thought that we could make it work. However, with time to do a more thorough analysis, I discovered that the entire system interface logic was wrong. To avoid doing a timing analysis of backplane transactions, the engineers implemented a design in which each data transfer consisted of a series of transactions. First a CPU would write the address to a register. Then the CPU would write whether it wanted to read or write data. Finally, the data would be transferred. Essentially, all of the boards in the system were simulating what they should have been doing instead of simply doing it. I was unable to convince management that redoing the design properly would save time in the long run and perhaps save the entire project. I accepted their decision but I knew it was a mistake.

The inherited design not only used four times as much hardware as should have been used and ran ten times slower but still had severe timing violations, such as 50 nsec data hold violations that were masked by the capacitance of the bus holding the data. I hired an electrical engineer to assist me and together we spent months drawing timing diagrams by hand and later with Timing Designer. Eventually, our corrections fixed the implementation problems without changing the architecture.

As we corrected control problems, other problems were exposed. For example, we were able to prove that the data collected to show feasibility was, in fact, nothing but noise. When it became obvious that my firmware team was doing all of the electrical engineering, the project leader made me the manager of the entire electrical engineering group. Most of the engineers were good but inexperienced and, under my direction, began to perform adequately.

Microgenics’ marketing manager had been trying to show the instrument long before it was ready and had spent much of the design budget on a beautiful “designer” box. When we finally were able to determine how much rinsing would be required to avoid carry-over (Microgenics’ own rather viscous Cedia chemistry was notorious for demanding rinsing requirements) we realized that the designer waste containers would have to be emptied every 15 minutes. This sort of outside-in thinking continually plagued the project. After buying Microgenics, Boehringer Mannheim carefully examined this project and tried to determine some way to save it. They correctly concluded that it was hopeless.

PID SERVO CONTROLLER

The inherited design used only stepper motors. Loading was sufficiently consistent for steppers to be reliable in all but one case. One vertical axis actuator had significant load variations and a tendency to resonate at various load-dependent positions in its trajectory. Further, the only feasible place for a home flag was in the mechanism itself. Installing the flag required dismantling and reassembling an OEM mechanism at a manufacturing cost of 200 dollars. For this cost alone, I reasoned that I could replace the stepper motor, its controller, and the flag with a servo motor with a sensitive current rise monitor to detect when the end of travel had been reached. I implemented a PID (Proportional-Integral-Derivative) servo controller and the current rise detector in one six dollar Analog Devices ADSP2105 DSP. I would have used the chip’s serial port to communicate with the Z80 process controller but, to live within the bounds of the existing architecture, I implemented a parallel interface through the system bus. I implemented a command language for the master that included the ability to adjust the PID coefficients dynamically to tune the electromechanical system. At one end of travel, the mechanism was supposed to hold briefly, which was easily done with the stepper. I simulated this by significantly increasing the I and D coefficients at that moment so that they fought each other, locking the motor position harder than any stepper. This was practical only because the hold time was brief. The current flows were small but the microscopic vibrations of the motor caused the bearings to quickly overheat.

SMALLTALK DRIVERS

The control computer software developers had elected to use SmallTalk, thinking that object orientation was a magic bullet that would make up for the fact that none of them had much experience designing real systems. They made the typical neophyte mistake of making everything, including each byte of a message, an object. Their program ran so slowly that it was unusable. The SmallTalk/V they were using adequately provided for making primitives in a variety of languages, including SmallTalk, but the programmers didn’t understand this capability. I built them a small library of primitives for the most common objects. I decided to use (Z80) ASM because it was the easiest language to match up to SmallTalk/V’s stack frame as well as being considerably faster than the alternatives.