Testing
and Analysis
Testing
- A ten part primer on testing by ATA president
Barry Richard.
Part
1 - Introduction
Part2 - Understanding Requirements for a
Forensic Test
Part3 - Planning the Test and Writing the
Protocol
Part 4 - Selecting, Designing, Purchasing
and Assembling the Test Hardware
Part 5 - Conducting the Test Meeting
Part 6 - Pre-Test Setup and Preperation
Part7 - Conducting the Test
Part 8 - The Post Test Process
Part 9 - Post-Test Disassembly and Cleanup
Part 10 - Post-Test Data Analysis and Reporting
Testing
(Part 1)
By
Barry Richard, CSP
If
you have been in our facilities you know that ATA Associates does
a significant amount of unique failure analysis testing to support
our accident reconstruction activities. Additionally, our testing
activities extend to product testing. We frequently receive requests
to conduct tests to determine how a product failed, what caused
it to fail, or whether or not the failure was caused by overloading
that exceeded reasonable design parameters. Each time we are requested
to provide an estimate of what it will cost to conduct the test.
However, it is reasonable for a client to ask for an estimate of
the cost to conduct a test. Many times clients react in disbelief
at finding what the estimated cost of a test will be. Generally
it is because they do not fully understand what goes into planning
and conducting a test. Many might think that it is simply a process
of getting a few pieces and parts together and pushing them, pulling
them or re-enacting the accident to see what they do. In fact, it
may have actually been that simple in the past. Daubert has, however,
changed the face of tests forever. Now it is imperative that experts
conducting tests follow a methodology that has been documented and
reviewed to assure accuracy, sufficiency and repeatability in the
process. It is no longer sufficient to do the test, observe it and
testify to it. The expert must be able to prove the results and
provide raw data for the experts on the opposing side to review.
For that reason, I think it will be of interest and benefit to undertake
a series of discussions on what is involved in conducting a test.
Because
of space limitations I intend to cover this topic in a series of
articles. The first article will appear in our next issue. My purpose
this time will be to outline the primary activities necessary to
conduct a test and to set the stage for follow-on discussion.
I
have broken the test process into 9 phases of activity that will
provide subject matter for 9 issues. They are as follows:
- Understanding the requirements for the test.
- Planning the test and developing the protocol.
- Selecting, designing, purchasing and assembling test hardware.
- Conducting the pre-test meeting.
- Pretest set-up and preparation.
- Conducting the test.
- Post test data gathering.
- Post test disassembly and cleanup.
- Post test data analysis and reporting
.
Each phase may involve a small or significant amount of effort depending
on the scope, data requirements and complexity of the individual
test. Over the next few issues I will devote space to discussing
elements of each test phase while addressing factors affecting the
cost to the client in hopes of providing a little more understanding
of the entire process.
BACK
TO TOP
Testing
(Part 2)
Understanding Requirements For a Forensic Test
By
Barry Richard, CSP
The
first step toward effective testing is to understand the reason
for testing and what the test is supposed to achieve. This is usually
done through discussions with the client and experts involved in
the analysis of the event. It may not be possible to answer all
of the questions at the first session, but a clear understanding
of the issues and basics is a must before moving into the test planning
and development process where capital resources become committed.
This is necessarily a brief discussion of the elements, but the
testing organization and all parties involved must begin with the
following.
1. Understand the reason for the test – A major point to be
made is that testing should not be conducted with the express purpose
of proving or disproving a specific point. It should be conducted
to understand performance elements within a defined set of parameters.
2. Understand the basis for the test - What was the failure scenario?
Understand the sequence of events of the original failure as accurately
as possible based on available information. This may initially come
from the client, but must be verified through research into documents,
statements of witnesses and photographs. Ultimately, you will have
to explain your test plan and assumptions to a jury and you cannot
say that you took the client’s word on everything. It may
make you look as if your client manipulated you.
3. Understand the test parameters - Within what bounds is the test
to be conducted? Some criteria must be established to make the tasks
realistic. Naturally, you may not have the luxury of testing everything
about a given system, so you test the things most clearly at fault.
Questions that must be asked include: What components failed? What
were the environmental conditions? What phase of operation was occurring?
Were surface conditions a factor? What were the speeds? If there
was an impact, what was the angle and speed?
4. Understand data requirements - What information do you hope to
gain from the test? Ultimately, it is an understanding of how and
why a failure occurred, but that answer may not be clearly discernable
from the test. What may come from the test is a large amount of
raw data and information that must be analyzed to get the final
answer. So, what do you measure – temperature, pressure, speed,
force, acceleration, strain, voltage, current, flow or some other
parameters? Is normal or high-speed video needed? The decisions
on what to measure and how, have a decided effect on the complexity
of instrumentation and thus, the cost of the test.
5.
Understand equipment and personnel requirements - Once the data
requirements are understood, the equipment to gather the data can
be defined. Additionally, test fixtures, locations and special equipment
to handle the item(s) being tested can be defined. Finally, you
can determine how many people will be required to conduct the test
effectively.
6.
Agree that the test is worth doing and commit to the necessary costs
- You may not be able to get the cost down to the nickel at this
point, but the client needs to be “prepared” for the
realistic amount. It is better to overestimate the cost slightly
than to let the client go away with an unrealistically low perception
of the cost.
This
is certainly not an all-inclusive discussion of the elements or
time involved in this phase of the process, but it should give an
understanding of the process. In our next segment we will explore
the detailed planning for the test and development of the protocol
document.
BACK
TO TOP
Testing
(Part 3)
Planning the Test and Writing the Protocol
By
Barry Richard, CSP
Testing
is frequently more complicated than the first look would make you
think. As with any complicated operation, planning up front, discussing
what is to be done, and putting it down on paper, can save a lot
of stumbling and additional costs in making changes later on.
Important elements to this process:
1.
Understand what the test is about – at least from a preliminary
level, and make a list of all the people who need to be involved
in the planning process.
2.
Get the people together and have a meeting to talk about the test
requirements.
Important items for discussion at the meeting:
1.
What the objective of the test is.
2.
What kind of data is expected from the test?
3.
What equipment and support will be needed to support the test?
From
that information, you can begin to formulate the sequence of events
for the test, the location of the test, and how long the test would
be expected to take. The meeting then becomes a brainstorming session
with the group to organize and plan for all of the materials and
equipment that will be required to support the test and make it
successful.
As an integral part of the meeting, a rough outline of the protocol
should be drafted. As the meeting develops, the protocol should
come together as a natural part of the activities. You can discuss
the flow of the test, the information to be acquired, and talk about
details such as placement of sensors, measuring devices, cameras.
The draft protocol then becomes part of the product of the meeting.
It can be typed by the secretary or recorder, and then provided
in draft form to the technical expert who will be responsible for
reviewing the protocol, making any additions or changes to ensure
the accuracy and correctness of the information and the processes
contained in the protocol. At this point it becomes a final document.
That needs to occur as quickly as possible in the early stages of
the process because the protocol and its requirements ultimately
drive the requirements for the testing location, the photography
and number of cameras required, the instrumentation required, the
data logging and equipment required, and the time required for the
test, as well as determining the number of people that will be required
to support the test.
This
stage of the test process is often delayed until a few days before
the test, or is relegated to a single individual with total responsibility
for writing the protocol and determining what is to be done. That
is, in fact, the least effective, least efficient, and most costly
way, generally, to go about doing testing of any measurable complexity.
For a small, single-item test that requires only minimal video coverage
and manual measurement with standard measuring tools (such as rulers,
micrometers, or a basic scale), that process will work. But, for
any test requiring electronic data logging, coordination between
more than two people, and acquisition of equipment and materials,
it is imperative, from a cost and effectiveness standpoint, to have
the meeting and write the protocol (at least the draft) very early
in the process. Whether it is testing, or research and development,
testing for operational proof, testing for litigation to prove hypotheses
or to gather information for other purposes, the planning phase
is where the final cost of the test is determined. Not only should
the experts and the technicians who are being involved in the test
be there, but also the client, who is driving and, ultimately, paying
for the test.
Many people think of testing as something very simple that you just
put together and go out and do in a couple of days. And, in fact,
that can be done. It has been proven over and over again. What is
also true, and has been proven over and over again, is that except
for the simplest of tests, few people ever remember or think of
everything that is required, nor get all of the data that could
have been gotten, when they run their test in that manner.
The next issue will address the preparation and assembly of the
equipment for the testing and cover implementation issues in preparation
for the test.
BACK
TO TOP
Testing
(Part 4)
Selecting,
Designing, Purchasing and Assembling the Test Hardware
By
Barry Richard, CSP
Once
the planning phase and the preparation of the test protocol are
complete, the hands-on work of assembling the test begins. The steps
of preparation include:
1.
Selecting the components, parts, and support equipment to be used
in the test. This can be a simple two or three items, such as a
camera, a short stretch of road or parking lot, a decelerometer,
and a pad of paper, with a vehicle and an operator; or it can be
much more complex, including multiple vehicles, several cameras
for documentation from different angles, sensors, and recording
devices for temperature, pressure, speed, distance, acceleration,
and other factors.
2.
Once the equipment and components are selected specifically for
their usefulness to the particular test, those that are not on the
shelf must be ordered. Some items have lead times of as much as
two to eight weeks for delivery, and that must be considered when
thinking about planning for the test. It doesn’t do a lot
of good to plan a test for a week from now, when a vendor cannot
deliver a part for two weeks. Frequently, the task of finding certain
components that are required for a particular test can be a research
project of several hours, especially when specific operating parameters
and cost are factors.
Purchasing the test articles and hardware (nuts, bolts, metal, cars,
trailers, etc.) requires money, and that money must, generally,
come from a client, unless the testing organization has the resources
to purchase that hardware up-front. Once the hardware and all of
it components are received, the task of assembling them in the proper
manner, and assuring that they function properly every time, becomes
the focus of the task.
Many
times, especially for complex tests, the instrumentation of a vehicle
or component can take several days of assembly, positioning, fastening,
running wires, and securing equipment so that it does not become
a projectile or get damaged in the event of a failure or loss of
control during a test sequence.
BACK
TO TOP
Testing
(Part 5)
Conducting
the Pre-Test Meeting
By
Barry Richard, CSP
Meetings,
meetings, meetings. None of us really like them. They take away
time from doing the really important (or fun) things, but they are
very necessary to the efficiency of conducting complex operations
— especially when there is a large group of people involved.
When testing is involved, a pre-test meeting is an extremely effective
way to assure that everything is ready to go for the test. It is
the last check, the last discussion session, the last chance at
assuring that everything has been taken into account and will be
there for the test -- from complex equipment, vehicles and people,
all the way down to box lunches and enough water to keep folks from
getting sick on a hot day.
Attendance
at the pre-test meeting should be mandatory, either in person or
by telephone, for the test lead or director, each and every one
of the test participants, the client, and any other individuals
who may be participants or interested in the test, its conduct,
and its outcome.
This
pre-test meeting is the last opportunity, in most cases, for someone
to say, “I was thinking last night that I would also like
to have this particular device or information.” What does
everyone think?” It also is an opportunity for people to bring
up their thoughts on the process or procedure, and to understand
the sequence of events and timing of what will happen on the test
date.
Things
that should be specifically done at the pre-test meeting:
1.
A list of all participants should be finalized.
2.
The test protocol should be reviewed in detail from start to finish.
3.
A list of responsibilities for each person – what they are
to do, what they are to bring, and when they are to arrive at the
test site – should be prepared.
4.
And finally, a discussion of safety issues for the day of the test
must be conducted.
This
is an extremely important meeting to ensure that everything is where
it is supposed to be, when it is supposed to be there, so that the
test can proceed on time, efficiently, in a cost-effective manner.
Remember, time wasted, in many cases, is money wasted for the client
when you have a team of five or six people at anywhere from $50
to $250 an hour, standing around waiting because you forgot to get
some particular piece of equipment that is considered vital to the
test.
BACK
TO TOP
Testing
(Part 6)
Pre-Test
Setup and Preparation
By
Barry Richard, CSP
The
day of a test, or possibly starting the night before a test, organization
becomes a critical element in the success and smooth running of
any test process. Team members should have designated responsibilities
to check and prepare all equipment and supplies, and package them,
if necessary, for transport to the test site. Naturally, if a test
is being conducted in your facility or in a close area where transport
of items is not required, then it becomes a much easier task. However,
there is still a task to be performed if the test is going to run
smoothly when the observers arrive.
Equipment should be positioned, warmed up, checked-out, and tested
beforehand to assure that it is working properly and that there
are no malfunctions. This assures that when the test manager says,
“Let’s go do the test,” that it can start at that
time and proceed smoothly and effectively to a final conclusion.
The preparation process is important in that if it is not done,
and outside observers are coming to watch the test or to videotape
it, your failures are captured forever and become a “sore
thumb,” so to speak, in the documentation of the test that
shows your lack of preparation.
Pre-test
preparation and checkout is not practice for the test. It is assurance
that the test will run smoothly and that you will have the highest
assurance of obtaining the data the client and his experts need
to make an accurate evaluation.
BACK
TO TOP
Testing
(Part 7)
Conducting theTest
By
Barry Richard, CSP
The
day of testing is the proof of whether or not your planning, preparation
and expertise (skill and cunning) are adequate for the task. Keep
in mind that total success is not a guaranteed commodity but good
pre-test preparation and planning go a long way to stave off total
failure. Anyone who believes that failure can’t happen to
them is either very well prepared or very naive.
Start
the day of the test prepared for hard work and long hours if the
test is even moderately complex. The lead test engineer and the
logistics support lead should begin the day before the test by inventorying
all required supplies and equipment, assuring that all batteries
are charged and spares are available and assuring that all persons
involved in the test are prepared.
Be
at the testing location early to re-check everything. This is relatively
easy if your test is occurring in a controlled lab environment but
gets more demanding if you are testing at a remote site. The bottom
line is that you should start early enough to be completely ready
to test before your audience arrives. All of your planning and preparation
should be transparent to them.
Data
collection equipment should be installed or positioned. This includes
multiple video positions, 35mm photography, sensors and observer
positions. NEVER depend on a single, non-redundant data collection
method unless you have the luxury of running the test multiple times.
If it is a one-shot test, the cost of added data collection and
documentation capability is always justified.
Once
at the test site, there should be one person designated as being
in charge and having the final authority over the test conduct and
safety issues. Everyone should have a job to do and should be constantly
busy doing it.
Some
specific tips concerning the test process follow.
- Number each test segment for identification. Make markers or signs
that clearly identify each segment and record the segment number
at every video position so that the videotapes can be matched.
- Use a flag or flash to put a start-run marker on the video for
each segment so that the tapes can be time coded and coordinated
if required.
- Put as many timing and distance markers as reasonably possible
in the area of the test. They may be invaluable later if your electronic
data logging equipment goes south on you. This author has seen more
than once where manual analysis tools allowed usable data to be
extracted from video cameras when the fancy stuff failed. Remember,
even partial success is better than no success at all.
- Download data after the first segment and check it to assure that
the equipment is working properly. After that, download after every
four or five segments and save recorded data to disk in case there
is logging equipment failure or damage.
- Have all film and video turned in to a single point for processing.
Everyone who needs it should get copies but the master set should
be in one location.
- Treat test articles as evidence if litigation is involved. Until
determined otherwise by the litigating attorneys, the data and damaged
articles are exactly that, evidence.
Note
from the author: This is a necessarily brief overview due to space
limitations. More detailed information can be found in SAE and ASTM
guidelines and texts on the subject. I will endeavor to provide
a detailed list of references as a final article for this series.
BACK
TO TOP
Testing
(Part 8)
The Post Test Process
By
Barry Richard, CSP
It
seems to be a generally recognized requirement that test conductors
gather data during the test. After all, that is why we test. Still,
a lot of data and information is lost during the test due to lack
of preparation and attention to detail. Interestingly, it is not
so frequently recognized that there is also significant amounts
of data that are lost after the test is “over”. The
reason for this is that many testers end the “test”
phase too early.
After
the crash . . . after the bang . . . after the crunch, everyone's
desire is to move in and look, touch, and inspect. Too much haste
and too little control of the test site can lead to parts being
moved too soon, before they are properly documented. Post-test configuration,
instrument and sensor condition, parts location and overall orientation
and relationship of items should be documented by a minimum advance
team first, before the crowd is unleashed to get their looks and
pictures. The closer to the point of impact, prime failure area,
or primary test component, the stricter the control should be until
the primary investigator says he/she is finished and the area is
released.
The
post-test process includes photography, video, reading mechanical
sensors and measures, retrieving data loggers, and assuring that
all critical measurements are completed before anything is moved.
BACK
TO TOP
Testing
(Part 9)
Post-test Disassembly and Clean-up
By
Barry Richard, CSP
“The
Job Is Not Finished Until The Work Area Is Cleaned Up!!!!”
Ever heard it? From dad to bosses it is a necessity that the test
area is cleaned up. Quite often there are complex test jigs or course
layout markings. There may be ground securements, signs, cups, snack
wrappers, wires, marker cones, tables, chairs, or debris of some
other sort. It should be cleaned up to leave the site in as near
the same condition as it was when the test team arrived. This is
not only environmentally necessary but it creates good will with
the owners of the test site. No one likes cleaning up after guests.
In
addition, inspection of the test apparatus and jigs during the disassembly
process may reveal information about the loads encountered in the
test.
BACK
TO TOP
Testing
(Part 10)
Post-test Data Analysis and Reporting
By
Barry Richard, CSP
You
have spent $50,000 dollars conducting a test. It is of absolutely
no good unless the meaning of the data is revealed and documented
for others to see and understand. This may take as little as 3 or
4 hours for a simple test where data recording was done manually
and by video. On the other end of the spectrum when high-rate electronic
data logging of multiple sensors is involved, the analysis process
may take 40 to 100 hours of intense effort. This is a point frequently
overlooked when attorneys or other clients request elaborate data
sensing and recording technology. I am not discouraging the use
of more sophisticated data collection techniques, they are often
necessary and yield much more accurate data. The point is that all
involved parties must realize the time and effort required to bring
the data to a point where it is useful for writing a report. This
is the time when the pre-test discussion of what and how to gather
data pays off. Good planning up front means that you do not have
to decide what to use and what to disregard in the analysis phase.
If
the planning was good and the analysis only has to address pertinent
data then the reporting of the results becomes significantly easier.
Once
the final report is filed with all of the data attached;
THEN
THE TEST IS OVER!
BACK
TO TOP
BACK
TO TOP
BACK
TO TOP