Just grade the process, not the result. It's literally that simple.

AI can talk the talk, but it can't walk the walk.

A simple "explain why this is the best solution" will make it shit the bed 10 out 10 times.

Thanks for sharing. Interesting to see how teachers are adapting to LLMs. I agree that trying to ban AI use is futile.

I was thinking that one of the great ironies of artificial intelligence is that it will make good teaching even more labour intensive. Those kids that can afford the one to one tuition required to push past our natural inclination to be lazy will learn things, while everyone else will succumb to outsourcing all their thinking.

There’s no way I’d have learnt everything I know now about software in today’s environment. So much of software development - debugging techniques, architectural decisions, structuring data etc is learned through trial and error. As the models get better is becoming easier and easier to just not look too closely at their output. I suspect it’s human nature.

They do scale if the examiner is an LLM.

The country of Argentina, despite its crippling debt, is able to provide free university education where the majority of classes for most majors are graded with an oral exam.

They've shown that it is entirely possible to scale a system like that, as long as the society sincerely values the role of an educator.

Use more AI to solve the problem of AI?

The oral exams scale better than one might expect, at least if you're not just doing multiple choice exams and you have to actually grade the exam. The time spent on grading the exam could have also been spent on an oral exam... For bachelor level it's usually 15-20min (usually quicker for the well prepared students), and for master courses it is 30min.

Usually you know after a few minutes if it's going to be a fail, and then otherwise you only need to figure out where on the passing scale the student ends up.

When I was a student, nobody cared if you did your home work. The teacher would go through the home work in the next class, and if you didn't do it, you would just lose the opportunity.

Then there was the written exam at the end of each course, which constituted 100% of the grade. The only reason that got abolished was because it made too many students fail, and the board hates that. So that became a race to the bottom, and apparently, we can still go lower.

It's reaching levels where most uni diplomas don't matter any longer. The only thing that will then matter is your network. That's the end of social mobility.

It's terrifying that some people actually do think this. Not you obviously, but I had enough conversations with other people.

on the off chance you’re being sincere: the article mentions that they allow free retakes of the oral exam

I agree.

As was sort of offhandedly mentioned in the article, all pre-LLM research into what results in students actually learning the material (as a distinct thing from passing the class) has landed pretty solidly on the side of frequent, low-stakes exams having better outcomes than rare high-stakes exams. The best students learn the material regardless of how you evaluate them; some weaker students will benefit from more frequent nudges to keep up rather than trying to cram at the end, some students simply won't realize they're falling behind without regular feedback, and some students successfully cram at the end and pass the class without retaining much of the material. The final group often expresses a preference for the one-exam-at-the-end model despite being disserved by it if the goal is to learn and not merely pass.

This makes me think of exams — I always did well on open-book exams, but I didn't do well on exams where I couldn't look at the book. Whether it's because of ADHD or just a lack of memorization ability.

Similarly, I did well in interviews where I could use the internet, but I didn't do well in interviews where I couldn't use the internet.

Everyone has a testing format they're good at. I'm curious about how this kind of thing is evaluated, and whether there are any papers related to this.

This is for American schools, unfortunately. And it's not a good development.

"you fail them" can mean both "you give them a failing grade" and "you fail to provide the support they need", right?

Yes, but as written the latter would not be a reasonable parse. For that sense: “you have failed them”, sure; “you are failing them”, sure; even “in acting so, you fail them”; but “and then you fail them” as an entire sentence would be very weird. The emphasis I placed on “fail” also doesn’t feel particularly compatible with that sense of the word.

I've been in uni classes which used this concept (about 10 years ago though). It also requires that content is given in the right size and complexity each week, otherwise there will also be people coming to class who did do their homework but didn't understand it properly and just like the people who didn't read it need full explanations.

I also felt having multiple explanations (in a regular class where you could also read the book in advance) from regular lecturers helped me in gaining understanding because having something explained in multiple ways made sure there would be at least one way, or a combination, which made it 'click'.

I always liked the regular lectures where you would be required to do some pre class reading and a smaller set of exercises the most, as you would be engaged in the class itself and would be getting repetition with different explanations.

One of my colleagues tried it for a few terms. Unfortunately, it didn't work well. Most students don't care about learning, they are here for grades :(

We still have developers because we can't outsource responsibility to AI. If shit breaks, we still need someone to fix it. Maybe you think that can also be automated. I have a counter-example. AI will rarely use the existing state when solving a bug. It will tack on it's own event queue, it will make up it's own messaging system, it is a cancerous growth. And this observation is based on both opus5.5 and gpt5.6-astra. The results are way past magical at this point, but you still have to weigh in pretty heavily for orderly code growth where solutions are succint, focused and preserve the existing structure. They are still noobs at fixing/adding stuff.

If you do not want to learn, and take every shortcut available to avoid it, university might not be the path for you.

UK (and the wider Anglo-American system) is different from the continent, where in many countries the MA is the basic degree -- a mere BA will get you nowhere -- and this easily takes 5-6 years to achieve.

Having two random people picked each time might cause all of them to at least pay attention.

Recently it has been reported that many Chinese students in Moscow are no longer learning Russian during their years at the university (to the dismay of Russian officials who hoped this would be a form of soft power): even for in-person lectures their Bluetooth earbuds will just translate the lecturer's Russian into Chinese in real time. So, perhaps people around the world will start feeling that, thanks to technology, exams testing language skills are a mere formality and can be safely cheated on.

Just don't make it open note then? Or you provide the only notes or cheat sheet available to them at the start of each test. That's what some math tests I've had had done before and I believe the SAT also does the same thing with formulas in the back of the test to reference.

I think you might be letting perfect be the enemy of good here. Right now some students are using an LLM to do all of their work. I don't suggest for a moment my method is perfect but I do claim it could be better than the status quo (or at least warrants further investigation in my opinion).

Anyone that knows the subject matter well enough will spot the LLMs mistakes and correct them. If you don't know the subject matter well enough to spot a mistake then you don't know the subject matter well enough.

That will either lead to students becoming frustrated after spending inordinate amounts of time on the unsolvable problem or students giving up too early on the solvable problems.

Put a mixture of questions in that are impossible to answer

Students will complain to the admins and waste more of your time.

When I was in engineering school ~20 years ago, some lecturers openly said that they gave the assignments just enough credit to be worth doing. Something like 10-15% of your final grade would be from assignments, the rest from the exam. So we didn't blow off the assignments to go drinking.

Of course, this a lot easier for some subjects than others. Subjects like Film Studies relied on exams much less, and assignments much more.

According to the article, #1 leads to "failure and drop-out rates of 50 to 80%". #2 and #3 might work for organized and motivated students, but graded assignments from other classes will be prioritized by most.

1. is not a terrible idea, if structured correctly, but it doesn't adequately incentive the learning process, which requires continuous practice. For 1, I'd say put 80% of the grade in three to four exams taken over the semester, and allow one midterm exam grade to be dropped or replaced by the final exam grade. The remaining 20% of the grade is to incentivize practicing problems: homework graded based on completeness not correctness, short in-class quizzes randomly sampling a homework problem to incentivize students to legitimately complete the homework that are graded for correctness, and student participation in classtime activities.

It would be a waste of time. LLMs have become better but not perfect. Personal experience; it's easier/faster to grade for me than have the LLMs grade and cross-check each one.

Many of them just dumped the document into an LLM, asked it to produce some "exam guide", and showed up with that thing printed out, asking me during the test to explain what LLM output meant.

That student presumably performed poorly on the test. Your approach is working as intended here, isn't it?

Also, is the AI-generated exam guide all that different to a human-written guide they could have downloaded a decade ago?

It must be tiresome to be a student these days… with each teacher having different pet theories about AI use and sometimes trying to “trick” students who use AI, or implement some draconian rules about the way homework or exams must be done. It’s tiresome for you, but just imagine getting varying lectures on AI use as you go from class to class as a student. Must be hell

This is a nice idea, but if you follow the Carnegie model of the number of hours per week it takes to learn a subject per credit hour, which is estimated at something between 6-9 hours total including classtime and out-of-class study per week for a normal-semester 3 credit course, then you can't meet the amount of time students need to be exposed to lecture (explanations, demonstrations, derivations), to do the background readings which provide necessary context that can't always be adequately covered in lecture, and to practice the concepts and tools on their own.

I teach math-heavy economics courses. I found that grading homeworks for completeness and pairing the homework due date with an in-class quiz on the problem set material (taken in class the day the problem set is due, after the set is turned in) has reduced the incentive for students to use LLMs to complete the homework. Exam questions closely mirror what's on the homework, and exam grades correlate with students who make legitimate attempts on their homework. I can sus out legit attempts typically by handwriting. No erasing and fast writing means it was copied.

For grade weights, I weight the homework the lowest of any grading category, with in-class quizzes next, and exams scores the highest.

I think a possible solution is to make homework carry no grade weight, and simply mark it as complete or incomplete, like a presence check.

Homework still provide valuable practice for students who are truly committed to learning. An automated system could provide an initial assessment and feedback. However, students can request human feedback paired with an in-person meeting. Such requests would require a mandatory explanation of what feedback they want or what the issue is. Professors and TAs will therefore only read and answer such requests themselves and will not waste time on AI output. At the same time, the system will still support students who are genuinely interested in learning.

Students who use AI to do their homework are not interested in such feedback anyway, and they will also avoid in-person meetings because it will naturally reveal that they are unprepared and did not do the homework themselves, which is humiliating.

To make this system work, we should present the in-person meetings as collaborative 'working sessions' or group office hours. Encouraging students to sign up in pairs or small groups also reduces individual anxiety.

The main channels for grading remain exams and oral presentations.

Thinking too small. The interactive testing needs to be done by AI. Then the transcript is reviewed by the teacher.

It’s not just evaluating. Students used to learn skills while doing an assignment. LLMing through assignments teaches copy-paste at best.

It’s like copy-paste Wikipedia presentations 2 decades ago. People used to learn things doing research for a project. Then suddenly it became copying off Wikipedia and people not even pre-reading what they copied into PPT.

What you’re calling evaluating is a part of the learning process for the student, not just about confirming that the student knows the thing. LLMs are actively undermining learning in this way. There was useful friction which has been removed.

Teaching and evaluation are of course not the same thing, but evaluation is an important part of teaching. There is a mismatch between students’ overall desire to learn things and the moment-to-moment experience that learning is hard.

Even devoid of all the economic and social pressures that grades impose students would compromise their learning experience. Ever regretted looking up the solution to a puzzle in a video game or sneaked a peak at the crossword puzzle solutions?

One of the hardest part of teaching is keeping students from self-sabotage. Evaluation helps with that.

Beyond that evaluation is of course also useful just as feedback.

Teaching is imo mostly about techniques that battle laziness. Teaching intrinsically motivated intelligent students has always been trivial. All progress in education is about scaling it to work for people who only reluctantly engage with the material.

Pretty strong opinion that overlooks all those students that study or attend courses specifically to learn.

That outlook is way too bleak. As a person who has taught at universities for fifteen years and has run lots of open educational formats and infrastructure with other public institutions and citizen groups, my experience is the opposite. If anything, the desire to learn and curiosity are among the strongest common traits humans possess. Grades are just a tool and a proxy that most people working in formalized educational settings have to work with at specific points.

It may come in different styles. But what do you do with students who don’t have motivation beyond grading/punishment? Some people just do not have motivation unless somebody standing by them with a stick.

The least paragraph... has always been the case. My parents told me stories about teachers denying facts not to lose authority. I distinctly recall my teacher pretending that peripheral vision doesn't exist not to lose authority.

If they had a fire under their ass that made them realize if they don't study well, they will forever be locked into flipping burgers or being homeless, most of them would move their assess.

That strikes me as backward. The higher the expected monetary value of good grades, the more students will prioritise grades over deep learning.

This is the way it's always been, sadly. Before AI submissions I had to regularly look for where my materials were posted on sites like Course Hero. Students copy-pasted from Wiki, articles, blogs, etc without attribution. Before that, they were buying or finding solutions books and copying out of there, or hiring people to do their homework/write papers/sit exams for them.

The primary difference now is that the both costs of cheating and ability to verify cheating on the instructor side have decreased dramatically, in tandem. There's also been a kind of mass effect, a cultural change in the attitude of students, where the sheer prevalence of cheating among their peers has made them less likely to feel fear, shame, or guilt over cheating. Students in the past who were caught were likely to confess immediately. Students now are increasingly belligerent and more likely to deny cheating, not just because they believe the professor can't 100% prove it, but also because they believe being caught out for cheating when most of their peers are getting away with it is unfair.

20 years ago my department head was saying homework was pretty pointless because everyone was copying each other. They assigned a token percentage for homework so at least the students were submitting homework.

The most interesting part here is that AI isn't replacing the learning goals, it's replacing the evidence instructors used to measure them

You're not learning anything if you don't put in your hours. Maybe you're fine with that but make sure you realize what you're getting into.

I teach mostly smaller courses but my approach to grading has been the same for years: A combination of a semester-long group project (ending in a paper, a prototype and a group presentation) with individual oral exams about aspects of the project and of the lecture. The lecture is flanked by optional TA sessions as well as access to labs which, for most hours of the day, have some experienced/teaching people being around (this results in a lot of ad-hoc learning situations and a general diffusion of knowledge and practices).

This gives me/us a lot of angles to understand what progress has been made throughout the semester as well as to consider and react to individual differences between students. In my book, it’s also resistant to faking it convincingly throughout all modalities. However, scaling that approach to big entry level courses would require more money for teaching than any research group in Europe I’ve ever seen has available.

"Everyone leaves their phones at the door"...

— watches, calculators, discrete technological devices. I've seen kids encode text as patterns of dashes on their desks, sharing the cipher solution with their desk mates in other periods.

The problem isn't skirting the effort. The problem is systemic. Why does the system reward taking shortcuts? I think it's because most areas are focused on short horizon tasks, min-maxing effort:output. It's not a problem with those technical fields. It's a problem with the current crop of the C-Suite across the entire occupational plane. They're short sighted.

It sad that it has come to this, but your approach is correct - it is the act of writing (code or prose) that is important, not the finished output itself.

Interestingly, this applies not just for learning, but also to a great extent in production/work contexts too.

I cannot shake the feeling that we are chasing the wrong goose.

The end goal of education is to help you understand how things work, with problem-solving being a means of developing and applying that understanding. LLMs shortcut the former and turn the latter into a pointless arms race that we cannot win.

I loved the example of increasing the complexity of learning projects, but I still feel like one question is missing: are people really trying to learn the same thing we were trying to teach them in the first place?

very simple. All exams to be paper and pen.

In my experience a significant proportion of "evidence-based best pedagogy practices" is hogwash. I'm pretty anti-AI, but going against education-school dogma isn't a reason to avoid AI.

We don’t

I managed to push AI to make me a math solver: https://nonconfirmed.com/mathsolver-n/

I don't like it. Now, the TAs have high cognitive load. Earlier, you only need to know the material of the course. But now, you're forced to navigate AI output. And you got to keep with all new model developments.