fix the "multiple-right-answer" issue by having questions vetted by non-ISC2 experts

bennetthaselton · September 2018

I provisionally passed yesterday. But I was left with the distinct feeling that many of the questions had more than one valid answer, without one of them clearly being "best", and that I had to guess multiple times what the question author was thinking, and that if I had guessed differently, I might not have passed. (Of course, the NDA prohibits giving examples of specific questions.)

But here's a simple suggestion to vet for these kind of questions, that wouldn't cost any more effort than ISC2 is already making anyway:

Have questions vetted by non-ISC2-affiliated security experts -- people whose security credentials are impeccable, but who, for whatever reason, are not CISSP-certified and don't have any affiliation with ISC2.

My theory is that when questions appear that have no clear best answer, but ISC2 has an answer in mind as the "best", one possible cause is "rolling groupthink". An initial critical mass of ISC2 members decide on an answer to a particular problem as the "best answer". The answer becomes standard in study materials, which means new candidates study the materials for the CISSP exam, and it becomes ingrained in them as the "right answer" which they then teach to new candidates. So can answer can become entrenched as the "best answer" even if it is not objectively the actual best answer.

So, this is the reason to have the questions vetted by non-ISC2 experts. If the "best answer" actually is the objectively best answer, then most of the non-ISC2 experts ought to agree with it as well, and the question is valid. On the other hand, if multiple non-ISC2 experts say "There is more than one right answer here, and no clear best one, so the question is invalid", then the "right answer" is an example of groupthink and the question should be modified or scrapped.

Since this would retain the good questions and get rid of the bad ones, is there any particular reason why ISC2 should not go ahead and do this?

Tekn0logy · September 2018

The scene in iRobot where the robot has to make a decision which human to save first comes to mind. Technically there are no wrong answers, but you have to demonstrate a logical thought process to arrive at the best solution. Changing the format of questioning would make the CISSP questions answerable by memorization, transforming it into yet another paper cert with diminishing value.

lucky0977 · September 2018

I took the 250q version and felt the questions were fair and obviously there were 25 questions in there that confused the hell out of you. The exam is designed in such a way that even if you've read every single piece of study material out there, you still need to use your brain and working experience. If they made the exam any easier, it would be just another Security+ exam.
With almost every question, there appears to be more than one correct answer but in the world of ISC2, there can only be one.

cyberguypr · September 2018

I have worked with several vendors as a SME developing exams. Your theory is simply wrong. Group thinking is not a thing in exam development workshops. What I want to point out is that the only thing ISC2 does is giving us tons of food and paying our expenses. All the material is exclusively developed and vetted by the SMEs. We create the questions, we review them, we accept them, recommend they are rewritten, or ask them to just be dropped and move on because they suck. ISC2 personnel in the room usually have no expertise in the topic at hand. People who handle questions later have zero power to change them. All questions are peer reviewed and everyone in the room (mostly strangers who met there for the first time) has a chance to object. As a result you can't say "but ISC2 has an answer in mind as the "best"".

I don't see how adding non-ISC2 credential holders will add any value. It simply wouldn't change a thing. Remember that the questions are NOT created from ISC2 material. They are created from industry accepted sources like the NIST publications and other frameworks/standards. When you say "when questions appear that have no clear best answer" that means most likely that the were created at a higher cognitive level in the taxonomy being used and therefore they require deep thinking and extensive analysis in order to discern subtle differences between the answers. Again, SMEs are the sole force behind this. ISC2 has no say and doesn't restrict sources.

Finally I am curious as to what your security background is. I could be wrong but I have a feeling the complaint can be a byproduct of limited experience int he field.

josephandre · September 2018

so that's where the pr comes from

JDMurray · September 2018

Realize that items on exams are weighted for a certain level of difficulty to fit a point scoring system, such as from 1 (very easy) to 5 (very difficult). How the exam item writers actually construct an exam item to fit into a specific weight (usually on a curve) is not an exact science. Using "two or more answer options are technically correct, but one item is more correct for the scenario given" is a technique used to make items that are weighted more heavily on the exam.

Also consider cyberguypr's last point that if you had the same level of understanding about the item's topics that the SMEs who wrote the item did, you would probably find the item much easier to answer.

PC509 · September 2018

While those questions were rough, I found them to be more real world and applied rather than just rote memorization. Yes, two answers may have been technically correct, but what one is best in the situation given. That's the test I was taking, that's what was expected, and that's what I want. I didn't want an exam that was just plain memorization of a book and terms and technical stuff. I wanted that information to be able to be applied.

Those are the questions that made me think harder, that made me question if I got it right, that made me really think like a security professional and not just a student that studied for an exam.

Yea, some of those were a pain in the ass. But, that's what I wanted. We passed. Even with those questions. We tackled the beast.

Late next year, I'm hitting the OSCP. I feel their motto has kept me going through the CISSP as well - Try Harder.

I'd like to add - anyone can memorize and take a test. Not everyone can know the concepts and pick out the best answer with multiple ones being correct... Sec+ can be done by cramming for a week. The CISSP not only requires experience to get the cert credentials, it's also required (mostly...) to pass the exam. $700 is a bit of cash for me. I feel like I invested it wisely in a certification I feel good about. If it could be non-situation specific and able to be passed using a memorize and pass style, I'd expect the exam to be half that cost.

sfportaro · September 2018

You and Lucky0977 are exactly correct.

Info_Sec_Wannabe · September 2018

bennetthaselton wrote: »

Have questions vetted by non-ISC2-affiliated security experts -- people whose security credentials are impeccable, but who, for whatever reason, are not CISSP-certified and don't have any affiliation with ISC2.

It sounds to me as if you are questioning the credibility of the exam. With that said, mind if I ask your motivation for sitting for it?

Also, if these "experts" you are looking for are not ISC2 affiliated, there is a possibility that they are affiliated elsewhere like CompTIA, ISACA, SANS or similar institutions. If you have taken any of their exams, did you not feel the same way?

Danielm7 · September 2018

Some great answers here. I took the 250 question version as well. For how much everyone kept going on about the questions were mind bending I was prepared for it to be a lot worse. Just read them clearly and realize that if you dig deep enough there is always one most correct answer. It may not apply to your personal workflow or how your last job did it, but it's likely the actual bet practice.

That Random Guy · September 2018

I literally just posted my first thread because of this exact issue. I provisionally passed the exam today and noticed the same crap. To me, that constitutes a poorly written exam and was not properly tested in ways that it should be been. It's almost as if a random group of know-it-all's got into a room and decided to come up with the most ambiguous and vexing of questions. I can't believe I'm praising my university professor's for their better example of how exams should be made. Not all professor's deserve that praise, but a handful did (from me). Questioning the credibility of the exam? Yes, I am! That was a poor excuse of an exam! Saying that we should go with our instinct—based on our own experiences—is bull for the very reason that one's experience could be wrong! That is not to say that failure cannot teach, but that everyone's experiences will be different. You can't ask ridiculous questions and then provide ambiguous answers that themselves don't entirely answer the question flat-out! It's like giving someone context and then telling them that anything they do in that given scenario is wrong simply because they thought of an unconventional yet logical approach. I am not happy about this!

JDMurray · September 2018

"provisionally passed" means you passed, unless you gave some seriously wonky answers that makes their psychometric algos infer that you were cheating, or there were technical glitches during the exam that were beyond anyone's control. So congratz!

fix the "multiple-right-answer" issue by having questions vetted by non-ISC2 experts

Comments