How to catch a cheater?

mikedisd2

Checked out the SNIA site for the first time. They state the following:

SNIA Exam Security/Cheating Policy
It is the policy of SNIA to revoke or refuse a credential to any person suspected of using unauthorized materials obtained using websites that advertise exact copies of SNIA exams. Psychometric data from a person's testing session will isolate patterns of cheating and gives SNIA the right to take legal action.

Do cheaters leave traceable patterns? Do they think differently to the rest of us? Or is this just twaddle to scare cheaters into learning the subject.

astorrs

mikedisd2 wrote: »

Do cheaters leave traceable patterns? Do they think differently to the rest of us? Or is this just twaddle to scare cheaters into learning the subject.

Absolutely. Psychometrics play a huge part in both certification development and integrity, all the major cert vendors and testing providers have psychometricians on staff. It is possible to use the data collected during a computerized test to determine the probability that cheating occurred (it's all about statistics).

wd40

mikedisd2 wrote: »

Do cheaters leave traceable patterns? Do they think differently to the rest of us? Or is this just twaddle to scare cheaters into learning the subject.

If a person finish the exam that should take 2 Hours in under 30 minutes with a score more than 95% ==> 99% chance of cheating.

1% ==> he /she has some kind of photo memory + super Fast readeing skills.


off topic:
We had a financial controller like that, it will take him 3 seconds to read a financial statement/report and spot any mistakes.

He is some kind of a group finance manager now making much more than our CEO.

jjbrog

They should have at least one ccna person on staff to give them a small pop quiz after their done, the look on the faces of the cheaters when they can't answer obvious questions would be priceless

mikedisd2

That makes sense. If you can answer 40 questions in 15 minutes, it looks real suss.

wd40 wrote: »

We had a financial controller like that, it will take him 3 seconds to read a financial statement/report and spot any mistakes.

How cool would that be? I could read the MSPress book in half hour.

tiersten

Ehh. Finishing quick with a high score doesn't mean very much. I've walked out less than an hour into a 3 hour exam and still scored within the top few percentile. Some people are just quicker than others.

What would be highly suspicious is if somebody made exactly the same mistakes as a ****. Alter a bunch of questions just enough that a cheater would get it wrong. If they flunk out on those and pass unaltered ones then something odd is going on and their paper would warrant a review.

dynamik

tiersten wrote: »

Ehh. Finishing quick with a high score doesn't mean very much. I've walked out less than an hour into a 3 hour exam and still scored within the top few percentile. Some people are just quicker than others.

What would be highly suspicious is if somebody made exactly the same mistakes as a ****. Alter a bunch of questions just enough that a cheater would get it wrong. If they flunk out on those and pass unaltered ones then something odd is going on and their paper would warrant a review.

Yea. I can't find my Vista pass post, but I annihilated that thing. The amount of time given was ridiculous.

Their detection algorithms are much more advanced than simply the duration of the exam though.

eMeS

mikedisd2 wrote: »

Do cheaters leave traceable patterns?

Yes. I took psychometric classes in both undergrad and graduate studies. In both classes we spent quite a bit of time discussing methods for detecting this type of behavior on various types of exams, and then legally what can be done about it.

I think the real issue here comes down to a question that the legal system has grappled with for years; is it worse to let a guilty man go free, or is it worse to convict someone that is innocent?

The risk with revoking a certification or basing legal action on a detected statistical pattern is that even if the chance is 1 in 1 billion (or 100 billion), you are admitting that the pattern observed is possible under natural conditions, however unlikely. Whether anyone likes it or not, 1 in 1 billion things happen all the time. As many have mentioned, there are some people out there who process things more quickly, are better at taking exams, etc...

The second major problem with many certification exams at the moment is that they are not standardized ("standardized" meaning that they are administered and scored consistently). How many posts have we seen here about computers failing at Prometric sites, dirty test centers, noisy test centers, terrible questions, etc... All of these items are variables which potentially affect test scores and patterns. IMO these variables are easily controllable, and with few exceptions very little has been done to control for these things.

Another way to look at this is if the test cannot be consistently delivered, then how can an assumption be made that statistics gathered during the test were consistently collected?

I believe we're in the midst of the industry's response to this these last couple of years, with the highest-level certifications often being granted based on more subjective qualifiers.

mikedisd2 wrote: »

Do they think differently to the rest of us?

Just an opinion here but I would say probably not. I think it's somewhat human nature to take shortcuts, some of us however are not willing to underpin our careers on some shortcuts that others have no problem taking. In fact, what I think is more likely the case is that at some point taking the shortcut was reinforced by positive outcomes, and the cheater continued down this path because it was the easiest way to continue to get positive outcomes.

mikedisd2 wrote: »

Or is this just twaddle to scare cheaters into learning the subject.

Twaddle. Funny word. In many cases I'm afraid the answer is yes. Having worked in corporate America for the past 20+ years I am both a little jaded and pessimistic. I've rarely seen sound logic and statistics win out over "gut-feelings", politics, and whatever else happens in large organizations (and keep in mind, I come primarily from an industry-financial services- that often claims to make all decisions based on sound statistics, logic, and methods). Vendors and organizations that offer certification do not have unlimited money to put into this, and statisticians and psychometricians are not inexpensive. Ultimately how an organization approaches all of this comes down to a cost/benefit analysis.

I ask everyone to consider that for the most part we are talking about profit-minded organizations who are most likely not running certification programs at a loss. I particularly wonder about those exams that if passed, allow the candidate to pursue some higher level class or certification. The incentive for the organization is clear; their potential market is subject to how many people pass the lower-level exam. Even more questionable are the organizations that exist primarily to award certifications.

It's a balancing act for all of these entities. On one hand they would love to increase their market share by having more and more people pass; on the other hand, the more people that pass the more potentially worthless their credential becomes. I must kill my host to live, but if I kill my host am I also killing myself? It's an enigma that all virii must accept or overcome....

IMO the bar to reject someone's exam results in most cases is likely set very high. This will minimize the rate of false positives, but will also increase the rate of false negatives.

Detecting those false negatives is your job, when you interview them for that position they want....

MS

eMeS

mikedisd2 wrote: »

That makes sense. If you can answer 40 questions in 15 minutes, it looks real suss.

This year I was granted "Accredited Supervisor" status with Exin. What this means is that I am able to administer various ITIL exams at all levels (some are basic multiple choice, some are complex multiple choice, and some are essay-based).

I do quite a bit of ITIL work and classes, and the classes all end the same way, with an exam.

At the foundation level the exams are 40 questions, with 1 hour given to take the exam. I rarely see anyone finish before 30 minutes have passed.

About 1 month ago I was doing a class. I can usually get a good feel for how people will do based on their performance during the class. There was one guy who seemed to not really comprehend the material, and I thought for sure that he would likely fail the exam.

About 10 minutes into the exam the guy gets up, brings his papers, etc.. to me and I sign him out, thinking that he just surrendered and did a pretty "Van Halen" logo on the response form.

That's around 15 seconds per question! I probably couldn't take a foundation exam that quickly, and I know that material cold.

A couple of days later when the results came in, I found out that he answered more than 30 questions correctly! That's a good score for someone who really knows the material.

All I could think was that the guy must have purchased a ****...there's really no other explanation, unless he's some kind of savant...

Before someone asks how I handled this...about all I can really say is that I am bound by NDA and cannot discuss those details.

MS

kevozz

Well written eMeS

I took a few classes at an "Academy" here in GA for my MCSA. One student took all 7 MCSE 2000 tests in under 2 weeks. Two of the tests he took in under 6 minutes (I was there working on Transcenders) which he bragged about. Shortly after he quit IT and wanted to get into the medical field because he couldn't hack it teaching the classes or get a job.

Surprised this didn't get the attention of M$, but this was back in '02.

eMeS

kevozz wrote: »

Two of the tests he took in under 6 minutes

That's amazing even for cheating...I still fail to see how even if you know the answers that this amount of time is enough to complete one of these exams. Or, why if you were a cheater you wouldn't think that something like this might raise a red flag....

The guy that finished in 10 minutes really stumped me...I just don't see how that's enough time to even look at the question, pick the answer that you already know, and write that answer on the exam form...

MS

Hyper-Me

I just don't get cheating on something like this.

Im always scared sh*tless that i won't know what im doing and will look like a dumbass. I cant imagine trying to get by knowing nothing and hoping to not look like an idiot.

Then again 95% of the people i work with dont know crap so I guess it works for them.

L0gicB0mb508

Hyper-Me wrote: »

I just don't get cheating on something like this.

Im always scared sh*tless that i won't know what im doing and will look like a dumbass. I cant imagine trying to get by knowing nothing and hoping to not look like an idiot.

Then again 95% of the people i work with dont know crap so I guess it works for them.

People just don't care. They have that "I'll learn it on the job" attitude. They view the certs as a way to get there, so they **** through them. What's awesome is when they get crushed on the job because they can't handle simple things. *evil laugh*

Mrock4

L0gicB0mb508 wrote: »

People just don't care. They have that "I'll learn it on the job" attitude. They view the certs as a way to get there, so they **** through them. What's awesome is when they get crushed on the job because they can't handle simple things. *evil laugh*

Unfortunately some of them don't get crushed on the job. THAT is the part that sucks. I've heard a story or two about CCIE's who couldn't handle CCNA-level tasks..but they got the paycheck and kept their job.

SmootCIS

Hyper-Me wrote: »

I just don't get cheating on something like this.

Im always scared sh*tless that i won't know what im doing and will look like a dumbass. I cant imagine trying to get by knowing nothing and hoping to not look like an idiot.

Then again 95% of the people i work with dont know crap so I guess it works for them.

Yeah it would be nice if it did work that way, I have a tech that works under me, One day I forgot a sheet on the paper work not a big deal I was going to have Help Desk print it out on the network printer, when I look over at him he is attaching a printer to his laptop trying to get it to print, I asked him if he installed the drivers and he just gave me a confused look, that guy has a MCSE, and has never heard of drivers. He did eventually get fired but it was not for incompetence but appearance, he came to work looking all dirty and they said that was to much.

I am all about a professional look but they just seem to have their priorities messed up there.

SmootCIS

I actually meant to quote the guy above me not the guy above him sorry.

hypnotoad

Mrock4 wrote: »

Unfortunately some of them don't get crushed on the job. THAT is the part that sucks. I've heard a story or two about CCIE's who couldn't handle CCNA-level tasks..but they got the paycheck and kept their job.

Or the A+'s who can't put in a PCI card...
Or the MCSE's who can't run DNS...
Or the B.S. graduates who can't write effectively in their native language...
Or the Ph.D.'s who teach math at your school and botch simple problems...

It's kinda scary man. I've met them all. They aren't going away any time soon.

Solaris_UNIX

Mrock4 wrote: »

Unfortunately some of them don't get crushed on the job. THAT is the part that sucks. I've heard a story or two about CCIE's who couldn't handle CCNA-level tasks..but they got the paycheck and kept their job.

I think the part you are missing about the "CCIE who can't handle CCNP / CCNA level tasks" is that CISCO's technology has changed A LOT since William Yeager from Sun Microsystems wrote the very first IOS back in 1987. A lot of people who used to be good at CISCO networking back in the day have slacked off and didn't learn the newer technologies and didn't keep re-testing / re-certifying themselves to keep their certifications current.

CISCO started giving out CCIE exams and certifications back in 1993, so it's possible that someone could have taken the exam in 1994 and passed and then let their CCIE cert expire. Someone like this might (wrongfully?) still refer to themselves as "CCIE number 2443" but there's a chance that they might not be familiar with newer networking technologies that didn't exist back in 1993, such as MPLS or ADSL in much the same way that someone who passed their CCNA in 2009 might not be familiar with ancient / arcane old networking technology like X.25, token ring, and Novell Netware / IPX routing.


There's only something like 17,000 currently certified CCIE's in the world in 2009, but I think the newest CCIE numbers that they're giving out are above 20,000 (i.e. I think a CCIE that just passed everything this month would probably have a number that is somewhere around the 25,000 range). So if you do the math, that means that there are somewhere around 8,000 CCIE's that let their certifications expire (the owner of the company in the data center across the hall from where I work is one of them).

So the first thing you should do when you meet someone who claims to be a CCIE is ask them what their name and CCIE number is so you can check to see if their certification is still current, because I suspect that a lot of them have slacked off and let their cert expire, yet they're still working out in the IT field somewhere informally passing themselves off as a "CCIE".

dynamik

Did you write all that just so you could name-drop Sun?

Seriously though, I don't think he's missing anything. The people he is referring to more than likely did everything in their power to obtain their certification illegitimately. Also, while I don't disagree with what you're saying, I don't see how that undermines his point either.

Solaris_UNIX

dynamik wrote: »

Did you write all that just so you could name-drop Sun?

For some strange reason, I am no longer capable of remembering anything about this "Sun" company of which you speak. Perhaps you were referring to Oracle?

My point still stands though that I see lots of old "expired CCIE's" in the field who haven't re-certified or kept up to speed with newer technology. I guess a lot of them are in senior management positions (i.e. owner of company) where they don't even touch a router anymore, so I guess that gives them an excuse to not keep up with things.

You can't braindump your way through a CCIE lab exam though, so I'm sure they were real hard-core experts at 1990's era CISCO networking at some time in the past just like Mohammad Ali / Cassius Clay was a great boxer 40 years ago.

dynamik

Not to the point where you just memorize Q&As, but after enough people "share" their experiences, the difficulty definitely decreases. That's one of the reasons Cisco has (or will shortly) added an interview component to exam.

Mrock4

I could absolutely see a CCIE who let his cert expire be one of the people I'm talking about (although that wasn't the group I was targeting). It could be likened to several ex-CCxx's I've met who go on and on about their Cisco expertise..only to find out they let their CCxx expire 5 years ago. Letting the cert expire is one thing, but giving the impression you're a *currently* certified CCxx is quite another.

There was one fellow in general who I was referring to, that was a relatively young guy, and did pass the lab, but myself, and many others who know him, can confidently say he did "****" it. Like dynamik said, maybe not by memorizing questions, but there's definitely some 'sharing' going on in cases like that. Enough so, that Cisco graced us with the v4.0 material, infact, and of course, the OEQ's.

hypnotoad- Hey, I think I know that MCSE guy!