Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
> The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
So true. I am aware of classes where everyone who didn't use AI cheated.
The simple reality is that if AI makes better answers than a student, and exam scores are normalized, then students who don't use it will fail as soon as a decent proportion of students do use it.
If the purpose of a system is what it does, what is the purpose of this education system that seems to uniformly and spectacularly fail to educate children?
It's not supposed to be a competition, but there should be incentives and oversight and controls and all the features you'd want to be able to reward outliers and foster excellence and all the good things while minimizing the bad.
When I was grading labs as a TA, the intent was communicated to me rather as "per university teaching guidelines we mustn't have too many students get the top grade but we also mustn't have too many students fail"
It also helps to avoid populist teachers that give everyone a A+++ to avoid students complains, and also idiots that give everyone a C because only God is A and only the teacher is B.
(We don't use that method here, we use other method to try to avoid both problems.)
An even-simpler reality is that, to the extent AI helps you cheat on your professional exams, you're about to enter a dead-end profession that will no longer exist a few years from now.
The common thinking of often a mental pattern of that intersects somewhere between laziness and comfort.
Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
> Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
I can't tell - are you suggesting these aren't good practices/traits to be learning when people are still in the "fundamentals of education/learning" stages of their lives?
I did all my basic differential and integral calculus studying by mind only. I don't do it that way in my career day to day now - nor could I without some serious practice. But the efforts I took in learning this way in undergrad made me a much stronger student and made me much more comfortable leveraging calculus in more application driven fields of study.
But in practice, having another human cheating for you was often unpractical: people don't usually like helping cheater, and simply trying to find an accomplice may get you in trouble. Because of that, it is relatively inefficient and therefore not a real problem and not a real impact on the final quality of the evaluation.
LLM is indeed just the same, except that finding an accomplice is now easy and without risk.
The rich and connected adjacent to accounting and willing to cheat would have most probably already worked somewhere else money-related and making an order of magnitude more money.
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
Not really. An open book exam still requires you to know which book to bring in, understand the concepts, and be able to reference them on the fly to answer questions. Basically, you need a reasonable grounding in the material to know where to start figuring out your answer.
What’s different with at-home exams is there’s nothing stopping your ringing your friend to ask for the answer, or looking it up on Google (now ChatGPT), or asking your parents who happen to be in the industry, if you want to go really old school!
I've had plenty of open book exams where the prof knew you would fail if you grabbed the book for more than a second. It's pretty much the same as the exams where you get to write your own cheatsheet: if you need it too much you are screwed.
The act of writing the cheat sheet is often enough to remember I find. It's yet another repetition of the material, just like doing labs and practice exams. And if you wrote the cheat sheet yourself, you also often know "where to look" for something specific, even if it's just to be sure you didn't remember something incorrectly and you really do only need to look at it for a few seconds.
So in my book (pun intended :P), allowing and actually encouraging a "cheat sheet" is a good thing. Open book is worse, as it's usually way too large and badly indexed. And who's gonna use an actual book in their actual job anyway?
Cheat sheets have an extra bonus, they are a great way to trick students into studying without realizing it is studying. By giving them a limited size, the student has to consider all of what they know and decide which areas they are the weakest on that need to be included, which they then have to organize into a compact and quick to reference chart. It doesn't replace the more boring phases of studying, but it does create a one off that gets better engagement and is more personalized than a fillable study guide or example test.
Seriously. Kids are going to cheat. It's already easy enough to just throw the test material into the LLM and get a bunch of flash cards on relevant content and memorize that. I Wish I had AI in college.
From watching slightly younger than college age kids adapt to the current world, I think you should be glad you did’t have access to LLMs during your learning years.
It’s too easy to slip from the idea that you’re just going to use the LLM to generate study materials into thinking that you’re just going to let the LLM do this homework assignment because your tired and then into a routine where ChatGPT is doing everything because you’ve come to rely on it. Then the students get slapped in the face with a sudden bad grade because the exams are in-person and they got all the way to the end of the semester with A-graded homework despite very little understanding of the material.
I'm in an online degree program in mathematics in my forties and this temptation is very real. The LLMs have memorized every textbook and every exercise so it's easy to have the kinds of conversations that before I could only have with TAs during office hours, and skip the mental struggle.
At least in my most recent class, it's also wrecked the class discussion forums that I previously found very helpful. By the end half the students were just slop-posting entire conceptual explanations and exercises, complete with different terminology, notation, and methods than the class text. So you just skip those and look for the few students you know are actually trying.
> It’s too easy to slip from the idea that you’re just going to use the LLM to generate study materials into thinking that you’re just going to let the LLM do this
This is exactly what people who know better are figuring out with vibe coding.
It’s extremely tempting for me to ask Claude to “do this thing that would take me three hours, but you only seconds”.
Many people are coming around to the realization that while that sometimes does work great, most of the time you ARE going to spend those three hours… you’re just going to spend it fixing, debugging, refactoring, instead of writing to begin with.
Using a tool to help you study isn't cheating. Using a tool to take the test for you, without regard to your own skills or knowledge of the subject under test, is.
The younger generations already struggle with technology because the guts have been hidden away their whole lives. They never had to understand a directory structure or a configuration file just to get a game running.
Having an LLM would turn that up to 11. Wishing you had AI in college is like wishing you had a car to train for a marathon. It’ll help a lot, if you ignore the actual goal of the work.
Absoluteky not. Actually having to contruct the flashcards embeds the information in your head to deeper level than 10 reviews could
Same with taking notes in class. You can never look at them again but the most benefit comes from having to organize the information in the first palce
I think it depends on the student, but I think you are probably overall correct. As someone who hated reading most of my textbooks, there is absolutely no way I am going to effectively extract relevant flash card material out of them better than an LLM can. I'm going to get bored and my mind will probably wonder and start thinking about other things while I am "reading".
This is a very concerning statement given the implications of your post.
AI can be a tool for learning or a tool for passing. Only one of those things is beneficial for society and it's not the one short minded students in crunch time will, on average, care about.
I also wish I had AI in college. I would have used it to descramble the unintelligible utterances of the calculus lecturers who had minimal or no English language skills.
Those poor calculus lecturers are most likely required to teach in order to earn their PHD. It is unfortunate that most students do not get to learn higher level math because of it. I was the type of student who did better when the professor was difficult, but engaging.
For example, I hated English growing up and then I had a college English course with a professor who was absolutely passionate about it and made it fun. Now, I hate English a little less and could appreciate it more. We need more people like that for other subjects.
For the last two decades, YouTube (or better, MIT's OpenCourseWare) has provided instruction that sets a baseline.
I'm positive that college lecturers fall below this baseline, but there's plenty of alternatives that a moderately motivated student could use.
Part of the problem is that the typical ~20 year old student has little idea how to learn something and little opinion about what their education should produce, to guide them.
> It's already easy enough to just throw the test material into the LLM and get a bunch of flash cards on relevant content and memorize that
LLM summarisation is broken, so I wouldn't expect them to get very far with this (see this comment on lobste.rs: https://lobste.rs/c/je7ve5 )
Also, memorizing flashcards is actually, to some point, learning the material. There's a reason why Anki is popular for students.
Ultimately, however, this comes down to the 20th+21st century problem of "students learning only for the test", which we can see has critical problems that are well-known:
Maybe it's different for higher education, but at least for my more memorization-centric high school courses (religion, science, civics), I find that I get good-enough grades by just feeding ChatGPT the test reviews and having it create Anki flashcards, making a few edits[1], and then reviewing them for a few weeks prior to the test on the toilet, bus, before bed, etc. If they're inaccurate, somebody should probably let the test know. So far it's been enough to bring my grades from low to mid 80s to high 90s. Spending an extra hour or two to squeeze out another 1 or 2 percentage points just doesn't seem worth it. I don't personally think that it's cheating, because IMO how I decide to study for the test is of no concern to the teacher, as long as I'm not getting outside help during the test itself[2].
A feeling I've been having a lot recently is that I have no idea why I actually want good grades in school. When I was a kid, I was told that life went:
good grades in high school -> good university -> good job -> lots of money -> being able to provide for your family
But now, it sort of feels like everything's been shaken up. Grade inflation means that good grades in high school aren't sufficient to get into university, and then you see statistics like "15% of CS grads can't find jobs", and that makes me think "is university really sufficient to get a good job?" And then getting requests by randos on the internet to do contract work for their start-up or whatever, with no formal CS or programming knowledge, and a grade 8 education, because of my projects, for entry-level wages, makes me think that a university degree really isn't even necessary for a good job. On the other hand, you see the richest people being the ones that make a big start-up then get acquired, is a good job even necessary for lots of money?
Sorry, this is rambling, but I should probably get back to work, so I'm not going to edit it.
[^1] Especially this semester, my religion teacher tends to use analogies in class that seem to be new, which messes up ChatGPT.
[^2] I feel less guilty using this method of studying for religion, specifically because in conversations with my religion teachers in the past, they've admitted to using ChatGPT to make and/or grade our tests. I know that HN people say "Oh, well, teachers are forced to use AI" or whatever, but I know that there are other teachers in my school who do not use AI.
I remember reading something when I was studying for AWS certs (might’ve been from AWS itself): the goal of the certifying bodies is to make as much money as possible. For this to happen, the exam can’t be so hard that nobody takes it, but it can’t be so easy that everyone takes it and it loses its value.
Organizations have been coasting on their pre-Covid reputations for a while. Now it’s time for them to adjust the slider the other way.
I don't know about this part. Years ago, my friend in college was taking all kinds of Microsoft certification exams and passing them with near perfect score. Thing is, he had no clue about most of the topics he passed, he had never worked with those tech. He just spent a bunch of time collecting questions (which wasn't that hard to find) and memorizing the answers. They could've made it difficult enough so just rote memorization wouldn't work, but they didn't (don't know if it has changed now).
Companies had long figured out these certifications are just easy money. It is hard to resist the temptation to just charge hundreds of dollars for a test and add it as a "profit center"
I don't want to sound heartless, especially as I'm in the high-risk category myself, but I think it's important to recognise that while COVID hasn't gone away, it is no longer a pandemic.
It is now endemic instead, and needs to be managed as such.
I have a dog in this fight as a professor, but I think the AI era may actually (and ironically) help reestablish colleges as a useful tool for employers. We have a significant amount of legacy infrastructure to support in-person testing, and non-digital written exams may be the best way to determine actual competency going forward.
I have historically done my computer science classes entirely online, but I am switching to in-person on-paper tests and increasing their weight in my classes to deal with the cheating.
In person exams make sense yes, but pen and paper?
If you want high integrity, exams should be done on school computers with extra integrity monitoring software. In the current cheating paradigm, pen and paper are as easy to cheat as students using their own laptops. Note that students can take pictures of exam with their phones, glasses, even their pens. There are pens with cameras and LED screens on them that connect via Wi-Fi to chatgpt.
Oral exams make for the best integrity but they are a pain to grade.
Exam room is covered in mirrors on all sides. Randomly and on demand, you have to hold up a spherical mirror that shows an infinitely complex reflection, such that AI can't generate such complexity, and have your camera zoom in on it to verify that it's real life, not an AI-generated video feed. Audio and perhaps clothing restrictions are also needed.
And what kind of skills would you test with this method?
Colleges are clearly not working, as evidenced by the number of unemployed graduates. Some will blame AI, but the reality is that any graduate would require training to be productive in the job, something they didn't learn in college.
My point is, if colleges could adapt to the job market, they wouldn't be in their current state.
At some point in my life, schools got tasked with teaching everything you need as a person. Things your parents or a business or the community or your employer taught you.
I don't know when, but it wasn't always like this.
> Colleges are clearly not working, as evidenced by the number of unemployed graduates.
What are you talking about? College-educated unemployment is 2.9% while for highschool-only it's 4.4%. Neither is high, but college-educated is definitely lower.
I plan on a mix of conceptual questions, pseudocode and code annotation. I think this may be on of those historical ironies where the slowness of universities to adapt actually works in its favor. I do agree that most universities do a bad job of preparing computer science students and I’m working to fix that as best I can where I teach. We will see how things pan out.
My wife is a teacher of physics and math for an online highschool. Its very common for kids to go into the in person exam with a mark in the 80s and 90s and get a failing grade on the exam.
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
When I took quantum mechanics in grad school, I struggled through the weekly (and intense) homework sets. My TA was a hardass, I’d spend hours on some problem, several few pages of math work just for one problem, and make some dumb mistake in an integral somewhere, being off by a factor of 2 at the end and only getting 2 of 4 points.
It was painful, and I felt like a dumbass seeing the other kids regularly getting perfect scores.
Then the midterm came and I blew them all out of the water. I hadn’t realised they somehow had the solutions manual so just got perfect scores all along but clearly didn’t learn the material like I did.
Yeah, I had this happen to me in an algorithms course. Tests were 80% of the grade and we had the guy who had been organizing mass homework "study" groups taking up increasingly larger sections of the class time desperately trying to figure out how to convince the professor to switch up the grading to be more homework based.
I figure that the professor had to know what was going on because he kept giving the same philosophical handwavey reasons for why the tests were staying at 80%.
That last sentence reminded me of a story one of my religion teachers told us last year: he was in university, and the way the course was structured was something like there were two exams, mid-term and final, each worth like 50%, but you could choose to not do the mid-term, and have the final be worth 100%. He chose the latter, and ended up being stressed out of the mine preparing for the exam. I can't remember if there was supposed to be a moral to the story, but it was a funny story, at least. It was probably funnier in-person than it will be for the person reading this comment.
This comment made me laugh, because I was looking into doing an online highschool, and while looking for discussions on the pros/cons, I stumbled upon the Reddit, which was all "Does anybody have the answers to the Unit 3 test for Mr. ${LAST_NAME}'s MCR3U class? I have $20." or "Selling the answers to the Unit 4 test for Mrs. ${LAST_NAME}'s ENG4U class for $30." That scared me off of doing high school online.
having "homework / coursework" count for the final score is what surprised me the most when learning about schooling in the US, in my university 100% of the score was the final, typically written test first, then oral in front of a blackboard (and usually the oral portion could move the needle of the written only +20%, but could definitely have you fail completely).
The one course that had something similar was microelectronics where during Christmas holidays we were given an optional assignment where we could design IIRC a NAND gate (2um process I think, most people ended up with a 5ft x 5ft sheet of paper at the end) which took a long time, but would give you up to +5% at the final (only one person got the full 5%, due to their creative use of the diffusion layer for interconnects). I don't remember any other course having anything along those lines, although to be honest you could slightly influence the difficulty of the oral final questions depending on how hard you worked / your behavior in class (of course only in years 4-5 where courses had only 20-30 students, no chance in year 1-2 with 400+)
It was extremely high stress, as you can imagine, but basically impossible to cheat. Every year a significant percentage of the students had to drop out, so by the time the 5th year thesis came around I think less than 20% of first years graduated at all. You were allowed to retake course finals if you wanted a different score (available 3x year typically, no guarantee you'd do better tho), but if you failed enough times you had to retake the course from scratch. You also were not allowed to enroll in the next year's courses until you passed all the prerequisites.
I failed one exam in my final year of uni (marginally), but passed the module because of excellent coursework. I put in an order of magnitude more work into the coursework for that class than I did any other class because I knew I was going to struggle in the exam.
In all honesty I shouldn’t have passed that course but it is what it is - and as far as I was (and still am) concerned, it was a bolt on course that I am ok being limited in my knowledge of.
80-100% of the grade imo. You could always tell which teachers were serious and teaching serious subjects by how little they cared about your attendance and homework assignments. In math classes, you could tell in an instant when they only assigned the problems that had answers in the back of the book. Not doing your homework in a serious subject is just punishing yourself when the exam comes in and it looks like it's written in a different language.
If you don't do your homework, or show up to class, but you ace the exams, you were just paying for the certification and to me that's totally legitimate.
I went to school with a bunch of working class immigrants who were working full time and going to school full time. If they had to miss every other class because of work but wanted to make up for it by studying all night, that seemed admirable to me. Nothing I hated more than participation points. It reminds me of management desperate to increase their headcount. It's the insistence that the focus of the class is the master-shifu at the front and center. It's a 300-level math class, dude; it's nothing that most people couldn't learn on their own.
I don't know, I've known many people that struggle with exams even if they know the material and even more people that excel with exams that learn nothing. Falling back on any kind of exam is just a recipe for more rote learning and that doesn't create better people (although possibly better readers, which we need).
(Preface: I am not a teacher, and I understand this is a hot take). At the end of the day there's an unwillingness from every level of education (parents, teachers, administrators, school boards, etc) to fight against the assault on intelligence by tech.
I don't think kids should have access to the public internet until they're adults, and certainly should never have it in schools except in controlled environments. Schools could create a private networks of curated sites and software. Parents don't have to give their kids unfettered access to computers. It's entirely in the realm of possibility to use computers and information networks in schools, accessed by children, designed to make it impossible to cheat while maximizing their ability to learn in a safe environment.
We don't build it because we don't want to. Parents don't care enough, teachers are overworked, administrators are inept, and big tech wants to turn them into little consumers who don't have critical thinking and addicted to their software.
I see this line of argument more and more over the last decade and it makes me feel heartless for my opinion.
But if you know the material but cannot apply it in an examination then you either don't actually know the material or don't have the emotional (for lack of better term) control to apply it in critical situations. Both are valid reasons to be marked down.
> don't have the emotional (for lack of better term) control to apply it in critical situations
No, not really, it just means you couldn't apply it in this one particular anxiety-inducing situation.
If someone finds it easier to display their knowledge in a certain way then school should strive to accommodate that as best they can (obviously there are practical limitations to this).
Mental health should be left to mental health professionals because you won't achieve anything by punishing students for their mental health struggles, you just make them hate you, hate school, and make their anxiety even worse.
I would argue that "knowledge" is an almost meaningless concept on its own. What assessments measure is a more complex form of "competency", and the competency of being able to write an essay on a topic is different from the competency of passing an MCQ quiz about it and both are different from being able to apply it in the field.
I don't have a clear solution, other than to have the assessments depend on what we're preparing people for. As an extreme example, I don't care how good of an essay a surgeon or anesthesiologist can write if they can't apply that under pressure.
But on the topic of test anxiety: I think intentionally causing emotional distress to children for the purposes of making a bad evaluation of their studies is cruel. It's a kind of cycle of trauma - "I did this, so you must to." We use grades to make value judgements of the quality of our children, when what we should be measuring is the ability of our schools to educate them and not how well-educated _the kids are_. The system is backwards, basically, and the fact it causes distress as a side effect is something that _should_ be managed - not ignored.
However anxiety exists and teaching children not to manage it is also bad. One of the really good things I've seen locally is that my school districts (the same that I went through as a child) focus on emotional education at the grade school level much more than when I was a kid, and I notice that the kids have much better emotional regulation than my generation.
In the end of the day academia in general should stop relying on exams based on memorization of random facts and start using real world examples of what kind of work student would be working with as an employee. And if student can deliver correct result even when using AI or any other method, and then explain why those results are the way they are, then student has passed.
In real world outside of academia, nobody cares how did you get to the result, only thing which matters is if result is correct and if you can explain why it is correct.
Academia should be for exploration, research, or preservation/archival. It’s about knowledge, not profit.
Academic attainment should be about the subject.
It’s business that should deal with the application, the short cuts, the “ends” rather than the means.
I’m sure there’s a formulation of this which also allows for AI in acedemia. I’m not srguing for that kind of purity. But I am saying that acedemia shouldn’t be treated as the training ground for employment.
If academia was solely about job training it would take at most like 6 weeks. It is about getting a rounded education. It is why engineers need to take liberal arts classes.
It can be hard to prevent cheating in person too: A criminal enterprise was uncovered in 2019 in Sweden. They had targeted the local SAT variant (högskoleprovet).
Their end customer equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers via voice.
The answers had been provided by some back office team based on a copy of the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
Locks arent meant to prevent doors from being opened, theyre meant to make it harder
If thats the sort of james bond shit that has to be done to cheat on in person exams, then theres a way higher chance of being caught. Using chatgpt at home is way harder to catch
In person exams are useless too. I was at a UK university as a mature student about 8 years ago. When exams came around a significant number of people went to the bathroom repeatedly during the 2-3 hour exam clearly to check notes on their phones. There isn't really anything that can be done to stop this other than doing some sort of spot check/search for phones on people mid-exam which would obviously be horribly disruptive.
Until quite recently, it was trivial to cheat on remotely proctored exams. All you had to do is spin up a VM, take the exam inside the VM, and use the host system to look up answers. I believe the main proctoring services now have crude VM checks. You can probably still use a KVM switch or a DP splitter and a buddy...
It's a dimension of neglect. If I run a service advertising itself as preventing people from harming themselves or each other (e.g. a mental health institution), then it would be criminally negligent of me to not limit people's access to sharp knives.
That is an excellent point. My recent coursework at Penn State, there were guardrails around cheating using Honor Lock, I am guessing a motivated student could find ways around it, but the system was better than trusting students to do the right thing.
My point is that since it is so incredibly easy to cheat (despite countermeasures that are essentially theater), returning to in person exams is probably a good thing.
The point you're making has nothing to do with anything the person you're responding to said, or with the OP. It's just a gratuitous description of sadism as a virtue-signalling imitation of seriousness.
You should find somebody who said cheating is fun and good to do, and explain your violent fantasies to them.
"We are doing what we can to hang on to relevancy as gatekeepers who already held way too much authority over a field". They are going to use AI on the job anyway.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Certifications have always been irrelevant for me, but that's only because my goal has always been what I'm capable of doing on my own AND (this one is a biggie) I was unbelievably fortunate to have several people in my career who trusted that I could get the job done.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Isn't that what the CPA and Bar exams, to use US analogs, do? They are an in-depth test or sets of tests that prove a person has a useful set of knowledge in a given domain.
I don’t think it will be too long before the pendulum swings back towards “real people who actually know the subject”. At that point, I might feel bad for everyone who coasted on AI.
Using AI is a different skill set that allows you to dive into topics that you otherwise aren’t ready for. I just used it to do a task that would have taken me a couple days of reading up on a different software system that I wasn’t already familiar. Now I have no need to ever really know that system, is that a good thing or not? I don’t know yet. But I had to know lots of basics about how those systems work in general to get the AI to do the thing I wanted, snd it wasn’t a one shot prompt, rather it was an iterative prompt process.
I touch LaTeX once every 10 years. I'm not going to learn it because I'm not fond of debugging macro processors and have never had a good experience with the language where you have to invoke a stew of packages that will mysteriously stomp on each other. I generated a script the other day to prepare a document in the format I needed. It mostly worked, but the LLM also stumbled on the packages until I could coax a working solution out of it. They're good for these problems where you only need shallow knowledge.
Get back to me when AI is actually reliably correct about any technical field.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
At the end of the day the job market will correct itself accordingly which is what most people who bother going to university or collecting any certificate care about. And right now it is already looking bleak. https://accountancyage.com/2025/09/29/pwcs-graduate-glow-up-...
Might be time we start adapting the pipeline into employment and start revising the importance of some of these gatekeepers before more people fall into unnecessary debt.
I've had no end of problems with accountants regardless of their certifications, they operate in a domain with an incoherent body of contradictory and highly subjective rules yet make it out to be a science.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
One of the main issues I had when I took accounting was that you often couldn't figure out things from first principles because the "right" way was whatever the relevant financial accounting standards board said it was. But following that standard is what companies need to do--and therefore has value--even if it's arguably arbitrary (within some general framework).
Yeah ... that's kind of the point. The money doesn't exist, but the violence people will use if their money is misappropriated is very real. Accounting is loophole patch stacked on loophole patch for thousands and thousands of years.
It's not intellectually enriching, but like it has the weight of society going back forever with dire consequences when it fails. That's not nothing even if it's boring from a technological point of view.
I think of it sort of like git. Technically, any sort of distributed version control would have served our industry just fine. Git didn't need to win, but things are vastly simplified having basically one version control framework to rule them all.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
So true. I am aware of classes where everyone who didn't use AI cheated.
The simple reality is that if AI makes better answers than a student, and exam scores are normalized, then students who don't use it will fail as soon as a decent proportion of students do use it.
This never should've been done to begin with. Education isn't supposed to be a competition.
It's not supposed to be a competition, but there should be incentives and oversight and controls and all the features you'd want to be able to reward outliers and foster excellence and all the good things while minimizing the bad.
What we have is tragic and absurd.
It's part of reproducing the labor-captial relationship.
(We don't use that method here, we use other method to try to avoid both problems.)
Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
I can't tell - are you suggesting these aren't good practices/traits to be learning when people are still in the "fundamentals of education/learning" stages of their lives?
I did all my basic differential and integral calculus studying by mind only. I don't do it that way in my career day to day now - nor could I without some serious practice. But the efforts I took in learning this way in undergrad made me a much stronger student and made me much more comfortable leveraging calculus in more application driven fields of study.
LLMs make this way easier but you can pay someone who gives private lessons in any subject and they can easily take an exam for you.
But in practice, having another human cheating for you was often unpractical: people don't usually like helping cheater, and simply trying to find an accomplice may get you in trouble. Because of that, it is relatively inefficient and therefore not a real problem and not a real impact on the final quality of the evaluation.
LLM is indeed just the same, except that finding an accomplice is now easy and without risk.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
What’s different with at-home exams is there’s nothing stopping your ringing your friend to ask for the answer, or looking it up on Google (now ChatGPT), or asking your parents who happen to be in the industry, if you want to go really old school!
I had used Illustrator to lay it all out. Lots of well type set diagrams and graphs alongside the equations.
Of course after spending the better part of 2 days making it I barely had to refer to it during the test!
So in my book (pun intended :P), allowing and actually encouraging a "cheat sheet" is a good thing. Open book is worse, as it's usually way too large and badly indexed. And who's gonna use an actual book in their actual job anyway?
Who sits in front of the PC, who is nearby?
The rest is kind of besides the point then.
From watching slightly younger than college age kids adapt to the current world, I think you should be glad you did’t have access to LLMs during your learning years.
It’s too easy to slip from the idea that you’re just going to use the LLM to generate study materials into thinking that you’re just going to let the LLM do this homework assignment because your tired and then into a routine where ChatGPT is doing everything because you’ve come to rely on it. Then the students get slapped in the face with a sudden bad grade because the exams are in-person and they got all the way to the end of the semester with A-graded homework despite very little understanding of the material.
At least in my most recent class, it's also wrecked the class discussion forums that I previously found very helpful. By the end half the students were just slop-posting entire conceptual explanations and exercises, complete with different terminology, notation, and methods than the class text. So you just skip those and look for the few students you know are actually trying.
This is exactly what people who know better are figuring out with vibe coding.
It’s extremely tempting for me to ask Claude to “do this thing that would take me three hours, but you only seconds”.
Many people are coming around to the realization that while that sometimes does work great, most of the time you ARE going to spend those three hours… you’re just going to spend it fixing, debugging, refactoring, instead of writing to begin with.
We are in a new era of ”no free lunch”.
Having an LLM would turn that up to 11. Wishing you had AI in college is like wishing you had a car to train for a marathon. It’ll help a lot, if you ignore the actual goal of the work.
Same with taking notes in class. You can never look at them again but the most benefit comes from having to organize the information in the first palce
This is a very concerning statement given the implications of your post.
AI can be a tool for learning or a tool for passing. Only one of those things is beneficial for society and it's not the one short minded students in crunch time will, on average, care about.
For example, I hated English growing up and then I had a college English course with a professor who was absolutely passionate about it and made it fun. Now, I hate English a little less and could appreciate it more. We need more people like that for other subjects.
I'm positive that college lecturers fall below this baseline, but there's plenty of alternatives that a moderately motivated student could use.
Part of the problem is that the typical ~20 year old student has little idea how to learn something and little opinion about what their education should produce, to guide them.
LLM summarisation is broken, so I wouldn't expect them to get very far with this (see this comment on lobste.rs: https://lobste.rs/c/je7ve5 )
Also, memorizing flashcards is actually, to some point, learning the material. There's a reason why Anki is popular for students.
Ultimately, however, this comes down to the 20th+21st century problem of "students learning only for the test", which we can see has critical problems that are well-known:
https://matheducators.stackexchange.com/a/8203
https://www.youtube.com/watch?v=J6lyURyVz7k
A feeling I've been having a lot recently is that I have no idea why I actually want good grades in school. When I was a kid, I was told that life went:
good grades in high school -> good university -> good job -> lots of money -> being able to provide for your family
But now, it sort of feels like everything's been shaken up. Grade inflation means that good grades in high school aren't sufficient to get into university, and then you see statistics like "15% of CS grads can't find jobs", and that makes me think "is university really sufficient to get a good job?" And then getting requests by randos on the internet to do contract work for their start-up or whatever, with no formal CS or programming knowledge, and a grade 8 education, because of my projects, for entry-level wages, makes me think that a university degree really isn't even necessary for a good job. On the other hand, you see the richest people being the ones that make a big start-up then get acquired, is a good job even necessary for lots of money?
Sorry, this is rambling, but I should probably get back to work, so I'm not going to edit it.
[^1] Especially this semester, my religion teacher tends to use analogies in class that seem to be new, which messes up ChatGPT.
[^2] I feel less guilty using this method of studying for religion, specifically because in conversations with my religion teachers in the past, they've admitted to using ChatGPT to make and/or grade our tests. I know that HN people say "Oh, well, teachers are forced to use AI" or whatever, but I know that there are other teachers in my school who do not use AI.
There will be a lot of COVID-era qualifications that are treated with a hint of suspicion in the future.
Take a look at A-level scores: https://schoolsweek.co.uk/a-level-results-2024-future-exams-...
( direct link to graph: https://schoolsweek.co.uk/wp-content/uploads/2023/04/Overall... )
It's unfortunate for those affected either way. It was a difficult time when drastic measures needed to be taken at short notice.
It's right to go back to in-person testing if there is a problem keeping remote exams fair.
Organizations have been coasting on their pre-Covid reputations for a while. Now it’s time for them to adjust the slider the other way.
I don't know about this part. Years ago, my friend in college was taking all kinds of Microsoft certification exams and passing them with near perfect score. Thing is, he had no clue about most of the topics he passed, he had never worked with those tech. He just spent a bunch of time collecting questions (which wasn't that hard to find) and memorizing the answers. They could've made it difficult enough so just rote memorization wouldn't work, but they didn't (don't know if it has changed now).
Companies had long figured out these certifications are just easy money. It is hard to resist the temptation to just charge hundreds of dollars for a test and add it as a "profit center"
The pandemic isn't actually over, at least, not for disabled people.
It is now endemic instead, and needs to be managed as such.
https://en.wikipedia.org/wiki/Endemic_COVID-19
I have historically done my computer science classes entirely online, but I am switching to in-person on-paper tests and increasing their weight in my classes to deal with the cheating.
As paul graham said: do things that don't scale.
If you want high integrity, exams should be done on school computers with extra integrity monitoring software. In the current cheating paradigm, pen and paper are as easy to cheat as students using their own laptops. Note that students can take pictures of exam with their phones, glasses, even their pens. There are pens with cameras and LED screens on them that connect via Wi-Fi to chatgpt.
Oral exams make for the best integrity but they are a pain to grade.
Exam room is covered in mirrors on all sides. Randomly and on demand, you have to hold up a spherical mirror that shows an infinitely complex reflection, such that AI can't generate such complexity, and have your camera zoom in on it to verify that it's real life, not an AI-generated video feed. Audio and perhaps clothing restrictions are also needed.
I would love to do oral exams but yes, hard to do with 100 students.
Colleges are clearly not working, as evidenced by the number of unemployed graduates. Some will blame AI, but the reality is that any graduate would require training to be productive in the job, something they didn't learn in college.
My point is, if colleges could adapt to the job market, they wouldn't be in their current state.
I don't know when, but it wasn't always like this.
What are you talking about? College-educated unemployment is 2.9% while for highschool-only it's 4.4%. Neither is high, but college-educated is definitely lower.
https://www.bls.gov/charts/employment-situation/unemployment...
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
When I took quantum mechanics in grad school, I struggled through the weekly (and intense) homework sets. My TA was a hardass, I’d spend hours on some problem, several few pages of math work just for one problem, and make some dumb mistake in an integral somewhere, being off by a factor of 2 at the end and only getting 2 of 4 points.
It was painful, and I felt like a dumbass seeing the other kids regularly getting perfect scores.
Then the midterm came and I blew them all out of the water. I hadn’t realised they somehow had the solutions manual so just got perfect scores all along but clearly didn’t learn the material like I did.
I figure that the professor had to know what was going on because he kept giving the same philosophical handwavey reasons for why the tests were staying at 80%.
Never heard that, some deep Yorkshire saying? Or typo?
The one course that had something similar was microelectronics where during Christmas holidays we were given an optional assignment where we could design IIRC a NAND gate (2um process I think, most people ended up with a 5ft x 5ft sheet of paper at the end) which took a long time, but would give you up to +5% at the final (only one person got the full 5%, due to their creative use of the diffusion layer for interconnects). I don't remember any other course having anything along those lines, although to be honest you could slightly influence the difficulty of the oral final questions depending on how hard you worked / your behavior in class (of course only in years 4-5 where courses had only 20-30 students, no chance in year 1-2 with 400+)
It was extremely high stress, as you can imagine, but basically impossible to cheat. Every year a significant percentage of the students had to drop out, so by the time the 5th year thesis came around I think less than 20% of first years graduated at all. You were allowed to retake course finals if you wanted a different score (available 3x year typically, no guarantee you'd do better tho), but if you failed enough times you had to retake the course from scratch. You also were not allowed to enroll in the next year's courses until you passed all the prerequisites.
In all honesty I shouldn’t have passed that course but it is what it is - and as far as I was (and still am) concerned, it was a bolt on course that I am ok being limited in my knowledge of.
If you don't do your homework, or show up to class, but you ace the exams, you were just paying for the certification and to me that's totally legitimate.
I went to school with a bunch of working class immigrants who were working full time and going to school full time. If they had to miss every other class because of work but wanted to make up for it by studying all night, that seemed admirable to me. Nothing I hated more than participation points. It reminds me of management desperate to increase their headcount. It's the insistence that the focus of the class is the master-shifu at the front and center. It's a 300-level math class, dude; it's nothing that most people couldn't learn on their own.
(Preface: I am not a teacher, and I understand this is a hot take). At the end of the day there's an unwillingness from every level of education (parents, teachers, administrators, school boards, etc) to fight against the assault on intelligence by tech.
I don't think kids should have access to the public internet until they're adults, and certainly should never have it in schools except in controlled environments. Schools could create a private networks of curated sites and software. Parents don't have to give their kids unfettered access to computers. It's entirely in the realm of possibility to use computers and information networks in schools, accessed by children, designed to make it impossible to cheat while maximizing their ability to learn in a safe environment.
We don't build it because we don't want to. Parents don't care enough, teachers are overworked, administrators are inept, and big tech wants to turn them into little consumers who don't have critical thinking and addicted to their software.
I see this line of argument more and more over the last decade and it makes me feel heartless for my opinion.
But if you know the material but cannot apply it in an examination then you either don't actually know the material or don't have the emotional (for lack of better term) control to apply it in critical situations. Both are valid reasons to be marked down.
No, not really, it just means you couldn't apply it in this one particular anxiety-inducing situation.
If someone finds it easier to display their knowledge in a certain way then school should strive to accommodate that as best they can (obviously there are practical limitations to this).
Mental health should be left to mental health professionals because you won't achieve anything by punishing students for their mental health struggles, you just make them hate you, hate school, and make their anxiety even worse.
I don't have a clear solution, other than to have the assessments depend on what we're preparing people for. As an extreme example, I don't care how good of an essay a surgeon or anesthesiologist can write if they can't apply that under pressure.
But on the topic of test anxiety: I think intentionally causing emotional distress to children for the purposes of making a bad evaluation of their studies is cruel. It's a kind of cycle of trauma - "I did this, so you must to." We use grades to make value judgements of the quality of our children, when what we should be measuring is the ability of our schools to educate them and not how well-educated _the kids are_. The system is backwards, basically, and the fact it causes distress as a side effect is something that _should_ be managed - not ignored.
However anxiety exists and teaching children not to manage it is also bad. One of the really good things I've seen locally is that my school districts (the same that I went through as a child) focus on emotional education at the grade school level much more than when I was a kid, and I notice that the kids have much better emotional regulation than my generation.
> I think intentionally causing emotional distress to children for the purposes of making a bad evaluation of their studies is cruel.
Is this ever the intended purpose?
My guess is the number of exceptional circumstances is about to explode...
In real world outside of academia, nobody cares how did you get to the result, only thing which matters is if result is correct and if you can explain why it is correct.
Academia should be for exploration, research, or preservation/archival. It’s about knowledge, not profit.
Academic attainment should be about the subject.
It’s business that should deal with the application, the short cuts, the “ends” rather than the means.
I’m sure there’s a formulation of this which also allows for AI in acedemia. I’m not srguing for that kind of purity. But I am saying that acedemia shouldn’t be treated as the training ground for employment.
Their end customer equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers via voice.
The answers had been provided by some back office team based on a copy of the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
If thats the sort of james bond shit that has to be done to cheat on in person exams, then theres a way higher chance of being caught. Using chatgpt at home is way harder to catch
Calculations must be getting accurate now. Not only questions about vocabulary or domain concepts.
Triviality is not a dimension of ethics as far as I have come to understand it.
You should find somebody who said cheating is fun and good to do, and explain your violent fantasies to them.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Much like how if you stop going gym you lose muscle mass, the same happens with knowledge and understanding with the brain.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
For exams and other tutorial like material* the LLMs have enough public training data for it to be good enough.
* all those vibe coded apps that are 95% boilerplate.
And no one is financing anything but LLMs at the moment.
Might be time we start adapting the pipeline into employment and start revising the importance of some of these gatekeepers before more people fall into unnecessary debt.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
It's not intellectually enriching, but like it has the weight of society going back forever with dire consequences when it fails. That's not nothing even if it's boring from a technological point of view.
I think of it sort of like git. Technically, any sort of distributed version control would have served our industry just fine. Git didn't need to win, but things are vastly simplified having basically one version control framework to rule them all.