Build and Test Cycle
From CajunBot Wiki
The process of building and testing is very important in any system. Even for a static system, there must be a process to ensure the quality of the final product. Developing the CajunBot platform is no different. When coordinating a large group of people working on the same project, a set of practices and procedures that everyone must follow help to ensure both safety and success. Below is a description of some of those processes followed for software and hardware.
Contents |
[edit] Software Changes
[edit] Design
One of the first steps in being flexible to software changes is to start of with a good software design. Anticipating what needs there will be of the system, as well as reducing the coupling between different modules helps determine success. Early on, the CajunBot software was designed so that modules could be developed independently from each other and communicate to each other using a common mechanism of shared memory data queues. By having producers and consumers of data and consumers with agreed upon protocols, parts of the system can be changed and upgraded over time.
[edit] Versions
Ensuring that only quality code is merged to the trunk of the system helps ensure safety. By designating which versions of the software are known to be testing and those that are considered stable helps to isolate software bugs. Major software changes are agreed upon ahead of time, and minor changes should not affect those working on separate parts of the system.
[edit] Simulation & Visualization
Simulation and visualization, though two distinct concepts, go hand in hand. Simulation refers to the physical approximation of how things behave, while visualization is being able to see data in a form that is understandable and informative. CajunBot uses an all-encompassing simulation system that enables every aspect of the vehicle and world to be simulated as if it were the real thing. Aside from a few configuration lines, the CajunBot software system does not know whether it is operating on the real vehicle or a simulated vehicle. This is tremendously powerful as it allows software changes to be tested over and over in the lab before ever being used in the field. Adequate visualization helps the programmer determine what is happening quickly and easily by simply watching the simulation unfold. Using these processes allows many hours of time to be saved.
[edit] Bug Reports
Diligently documenting the circumstances and inputs during a software crash can make the difference between a good bug report and a useless one. A good bug tracking system such as Bugzilla is extremely helpful in this process. By recreating the scenario in simulation, debugging time is significantly reduced for the developer. Additionally, these test scenarios can be scripted and added to a testing list to ensure that future changes to software do not introduce old bugs.
[edit] Exhaustive Automated Testing
The ability to run a simulation "headless" (without the visualization component and user interaction) is one of the keys to setting up an automated testing system. Iteratively defining goals for the vehicle, thousands upon thousands of test scenarios can be tested across a cluster of computing nodes and the results analyzed. Of course, this means that in the design phase of your software that you need to plan to be able to extract this information (software crashes or irregularities) so that this process can be streamlined. The CajunBot code has gone through miles and miles and hundreds of hours worth of simulated test runs to show that the vehicle can perform reliably under expected scenarios.
[edit] Specific Field Testing
No matter how perfect simulation is, there will always be some aspect of the real world that is not fully captured. For this reason, after software changes have been made and tested through hundreds of test runs in the simulator, it is time to do the actual field testing of the new features. There are extra peculiarities that are sometimes not accounted for that only show up once the system is really being tested. A controlled testing area with the ability to recreat the desired scenario is a must.
[edit] Dynamic Field Testing
Dynamically testing the vehicle is the next step in the process. Once the desired new feature has been rigorously tested both in simulation and in specific field testing, dynamic field testing that presents unexpected changes is very important. In the real world, nothing is going to happen precisely the same way every time when humans are involved. For this reason, ensuring that the vehicle behaves safely under unplanned scenarios is also of great concern. At all times, everyone involved with the field testing must keep open communication channels and resolve any ambiguities.
At the end of it all you still need to test over and over again.
[edit] Hardware Changes
To incorporate new hardware is very similar to incorporating new software. The design must include room for expandability, and it is usually a good idea to be able to abstract hardware changes in the software. Two different LIDAR models from different companies should be able to provide the same type of data and be handled in the same way. Accurate replication of the hardware's functionality through simulation must be verified so that simulation results can be trusted. Specific field testing of new hardware devices can help confirm that they function as expected.
