dangquangminh
14-07-2006, 10:30 PM
Q&A with Cisco's CCIE lab exam developer
How Cisco develops its CCIE lab exams
IT Careers and Training Newsletter By Linda Leung, Network World, 05/24/06 Continuing with our fascination of the Cisco Certified Internetworking Expert (CCIE) lab exam, which is apparently harder to pass than the State Bar of California or the Certified Public Accounting Exam (read my colleague's article), I thought I'd pin down the Cisco employees who are responsible for creating those 8-hour lab exams.
I wanted to find out how they create those exam questions. I wanted to know how they feel about putting candidates through so much stress and whether they themselves passed first try. Here's my Q&A with Cisco's Elaine Lopes, CCIE program manager (and multiple CCIE). She oversees the exam development process for the CCIE team.
Q: What are the qualifications of the subject matter experts (SME) who help to create the exams?
A: The SMEs involved in CCIE exam development are technical experts with significant experience in either technical support, sales support, support escalations or product development. They are familiar with the use of Cisco gear in real-world customer requirements. Many of them are CCIE certified and all of them are familiar with the certification process and Cisco's standards for exam development.
Q: What is the process of developing a CCIE lab exam?
A: The SMEs collaborate to develop a lab blueprint of topics an expert should be knowledgeable about. Feedback on the blueprint and related question objectives is sought from Cisco's technology groups and business units, Cisco's technical support organization, Cisco's technical sales team, Customer Advisory Boards, and customer surveys. Cisco uses industry standard exam development practices, rigorous quality control, and a highly-controlled project management process to develop the labs. All exams are pretested before being released into production with real candidates. After release, we continue to monitor exams to ensure they are performing as required.
Q: How many SMEs work on the CCIE lab exam?
A: Lab exams are created using a virtual team of experts from across Cisco and the size of the team varies depending on the topic areas and the breadth of the experts on the team.
Q: How do you design the questions/scenarios?
A: We use a virtual team and collaborative approach to develop questions based on real-world situations the team has encountered in their work. One scenario will form the basis for a series of interrelated questions that cover various topics on the exam blueprint.
Q: How do you figure out the different paths that students can take in each question? Is there just one single path that is correct or can someone who started down the wrong path get back on track and solve the problem?
A: Multiple solutions are acceptable on the lab exam as long as candidates achieve a working solution. Candidates are requested to observe particular constraints such as 'you must use this' or 'you must not use that', but the solutions are graded based on the outcome.
Q: How long does it take to design a CCIE lab exam?
A: At any one time multiple exams are in production at different stages of development. The development time varies but a single exam can take up to three months to complete.
Q: How do you know when the lab exam development is finished? Do you have to tweak/change any lab exams and when would you have to do so?
A: Actually the CCIE lab exams are never really finished. We monitor the performance of each version of each exam at the question level. We pay particularly close attention during the first few months in production. If a question's performance begins to fall outside our expected parameters, we will modify it or replace it.
We monitor exam performance throughout its life cycle. We have a schedule for regular maintenance of exams and goals for the age of exams in production. Exams are constantly in production, coming online and being retired.
Q: Has there ever been an occasion when a question has just been too hard for any one to answer correctly?
A: As mentioned, we continually monitor the performance of each question to ensure it is performing as expected. We modify or replace questions as necessary.
Q: Who beta tests the lab exams?
A: We test the lab exams using internal CCIE candidates and the CCIE proctor team before putting them into production.
Q: How do you feel about putting people through so much stress? How was the CCIE test for you when you took it?
A: The exams are meant to present candidates with the most challenging situations encountered in the real-world experience of our SMEs. The reality is that networking in today's environment is complex and stressful. The most successful and competent technicians already know this. Believe it or not, we sometimes get feedback from candidates that the exam is too easy!
I prepared extensively for my first attempt and even though I failed, I was able to recognize improvement areas and build an 'attack plan' to cover them. I passed on my second attempt and I view the certification process as a growth experience, not only as an engineer but as a person. I believe we have to be humble enough to recognize that even doing your best, there's always room for improvement.
Q: Why do you think that the CCIE lab exam is viewed in the industry as a tough and stressful exam that some candidates have to take multiple times to pass?
A: The CCIE lab exam is the closest we can make it to the real world. Not everyone has the skills to deal with the most challenging networking situations. It's a worthy goal, but it takes experience and effort to get there.
Q: What has been the best and worst feedback you've had from students about the lab exam?
A: The best is when a candidate passes and calls you the next day to thank you for being supportive. Expert certification is a journey and it's great to see a candidate make progress to the point s/he passes. The most disappointing feedback is when they say they will give it up, which is too bad because they have the potential but don't go the extra mile to get what they want.
Q: What is your advice for people preparing for the lab exam?
A: The successful candidate is the one who has extensive hands-on knowledge in the track of his/her choosing, and recognizes their strengths and development areas so s/he can strike a balance between learning and putting what they've learned into practice. No single learning method is good for every candidate and for every improvement area, so a combination of preparation resources is best. Most important is to sit in a lab environment and practice by exploring configuration variances, how to accomplish the same thing differently, and how to use debugs and show commands that yield clues in case configurations are not working. The CCIE Program strongly encourages candidates to have at least three to five years of job experience before attempting certification, and study time will depend on candidate's experience and access to preparation resources.
Q: What's the secret to passing first time?
A: My advice is to know the subject matter thoroughly, practice time-efficient trouble-shooting and don't panic during the exam. Additionally, to help candidates better prepare for the lab exam, Cisco has developed the CCIE Assessor lab - a four-hour practice lab session accessed via a Web interface, which provides detailed feedback on a candidate's performance.
(source: http://www.networkworld.com/newsletters/edu/2006/0522ed1.html?page=1)
How Cisco develops its CCIE lab exams
IT Careers and Training Newsletter By Linda Leung, Network World, 05/24/06 Continuing with our fascination of the Cisco Certified Internetworking Expert (CCIE) lab exam, which is apparently harder to pass than the State Bar of California or the Certified Public Accounting Exam (read my colleague's article), I thought I'd pin down the Cisco employees who are responsible for creating those 8-hour lab exams.
I wanted to find out how they create those exam questions. I wanted to know how they feel about putting candidates through so much stress and whether they themselves passed first try. Here's my Q&A with Cisco's Elaine Lopes, CCIE program manager (and multiple CCIE). She oversees the exam development process for the CCIE team.
Q: What are the qualifications of the subject matter experts (SME) who help to create the exams?
A: The SMEs involved in CCIE exam development are technical experts with significant experience in either technical support, sales support, support escalations or product development. They are familiar with the use of Cisco gear in real-world customer requirements. Many of them are CCIE certified and all of them are familiar with the certification process and Cisco's standards for exam development.
Q: What is the process of developing a CCIE lab exam?
A: The SMEs collaborate to develop a lab blueprint of topics an expert should be knowledgeable about. Feedback on the blueprint and related question objectives is sought from Cisco's technology groups and business units, Cisco's technical support organization, Cisco's technical sales team, Customer Advisory Boards, and customer surveys. Cisco uses industry standard exam development practices, rigorous quality control, and a highly-controlled project management process to develop the labs. All exams are pretested before being released into production with real candidates. After release, we continue to monitor exams to ensure they are performing as required.
Q: How many SMEs work on the CCIE lab exam?
A: Lab exams are created using a virtual team of experts from across Cisco and the size of the team varies depending on the topic areas and the breadth of the experts on the team.
Q: How do you design the questions/scenarios?
A: We use a virtual team and collaborative approach to develop questions based on real-world situations the team has encountered in their work. One scenario will form the basis for a series of interrelated questions that cover various topics on the exam blueprint.
Q: How do you figure out the different paths that students can take in each question? Is there just one single path that is correct or can someone who started down the wrong path get back on track and solve the problem?
A: Multiple solutions are acceptable on the lab exam as long as candidates achieve a working solution. Candidates are requested to observe particular constraints such as 'you must use this' or 'you must not use that', but the solutions are graded based on the outcome.
Q: How long does it take to design a CCIE lab exam?
A: At any one time multiple exams are in production at different stages of development. The development time varies but a single exam can take up to three months to complete.
Q: How do you know when the lab exam development is finished? Do you have to tweak/change any lab exams and when would you have to do so?
A: Actually the CCIE lab exams are never really finished. We monitor the performance of each version of each exam at the question level. We pay particularly close attention during the first few months in production. If a question's performance begins to fall outside our expected parameters, we will modify it or replace it.
We monitor exam performance throughout its life cycle. We have a schedule for regular maintenance of exams and goals for the age of exams in production. Exams are constantly in production, coming online and being retired.
Q: Has there ever been an occasion when a question has just been too hard for any one to answer correctly?
A: As mentioned, we continually monitor the performance of each question to ensure it is performing as expected. We modify or replace questions as necessary.
Q: Who beta tests the lab exams?
A: We test the lab exams using internal CCIE candidates and the CCIE proctor team before putting them into production.
Q: How do you feel about putting people through so much stress? How was the CCIE test for you when you took it?
A: The exams are meant to present candidates with the most challenging situations encountered in the real-world experience of our SMEs. The reality is that networking in today's environment is complex and stressful. The most successful and competent technicians already know this. Believe it or not, we sometimes get feedback from candidates that the exam is too easy!
I prepared extensively for my first attempt and even though I failed, I was able to recognize improvement areas and build an 'attack plan' to cover them. I passed on my second attempt and I view the certification process as a growth experience, not only as an engineer but as a person. I believe we have to be humble enough to recognize that even doing your best, there's always room for improvement.
Q: Why do you think that the CCIE lab exam is viewed in the industry as a tough and stressful exam that some candidates have to take multiple times to pass?
A: The CCIE lab exam is the closest we can make it to the real world. Not everyone has the skills to deal with the most challenging networking situations. It's a worthy goal, but it takes experience and effort to get there.
Q: What has been the best and worst feedback you've had from students about the lab exam?
A: The best is when a candidate passes and calls you the next day to thank you for being supportive. Expert certification is a journey and it's great to see a candidate make progress to the point s/he passes. The most disappointing feedback is when they say they will give it up, which is too bad because they have the potential but don't go the extra mile to get what they want.
Q: What is your advice for people preparing for the lab exam?
A: The successful candidate is the one who has extensive hands-on knowledge in the track of his/her choosing, and recognizes their strengths and development areas so s/he can strike a balance between learning and putting what they've learned into practice. No single learning method is good for every candidate and for every improvement area, so a combination of preparation resources is best. Most important is to sit in a lab environment and practice by exploring configuration variances, how to accomplish the same thing differently, and how to use debugs and show commands that yield clues in case configurations are not working. The CCIE Program strongly encourages candidates to have at least three to five years of job experience before attempting certification, and study time will depend on candidate's experience and access to preparation resources.
Q: What's the secret to passing first time?
A: My advice is to know the subject matter thoroughly, practice time-efficient trouble-shooting and don't panic during the exam. Additionally, to help candidates better prepare for the lab exam, Cisco has developed the CCIE Assessor lab - a four-hour practice lab session accessed via a Web interface, which provides detailed feedback on a candidate's performance.
(source: http://www.networkworld.com/newsletters/edu/2006/0522ed1.html?page=1)