SeanPaulMurphyVille: Grading ChatGPT, Part 1: "Wings of Courage"

Sunday, June 18, 2023

Grading ChatGPT, Part 1: "Wings of Courage"

My fascination with artificial intelligence continues, especially since the use of AI is a major factor in the negotiations between the studios and the striking WGA,w. I believe AI will be a threat to a number of professions in the future. But is it a threat to writers now? I decided to do a little experiment.

I asked ChatGPT to write a script for me, which I would then grade using the same criterion I employed with my students when I taught at Towson University.

I taught the class Broadcast/Film Writing for a few semesters at the University before COVID temporarily dampened enrollment. It was a 300 level course in the Electronic Media and Film Department. The class had two pre-requisites, and the textbook was The Short Screenplay by Daniel Gurskis. The syllabus, which I did not write, outlined a number of laudable goals for the class. However, its real purpose was to help the students write good, shootable scripts for their advanced production classes.

The class had six graded assignments.

1). Nonfiction Treatment Proposal.

2). Nonfiction Script

3). 2-4 Minute Short Treatment

4). 2-4 Minute Screenplay

5). 5-8 Minute Short Treatment

6). 5-8 Minute Screenplay (Final Project)

Here's the grading rubric for the fiction projects:

EMF 377 GRADING RUBRIC –

FICTION SCRIPT ASSIGNMENTS

Rubric Elements	Unsatisfactory 1 point	Average 2 points	Above Average 3 points	Excellent 4 points	Total
Protagonist Description, Motivation, Action	insufficient or lackluster portrayal of the protagonist	adequate portrayal of the protagonist	effective portrayal of the protagonist	vivid, superior work
Antagonist/ Source of Conflict Description, Motivation, Action	insufficient or lackluster portrayal of the antagonist/ conflict	adequate portrayal of the antagonist/ source of conflict	effective portrayal of the antagonist/source of conflict	vivid, superior work
Dramatic Structure Setup, Development& Climax/ Resolution	poor or ineffective use of the required dramatic structural elements	adequate use of the required dramatic structural elements	effective use of the required dramatic structural elements	superior use of the required elements
Format Script format as specified by the instructor & text	flawed, numerous errors	adequate, some errors	effective, very few errors.	professional& error-free
Writing Grammar, Punctuation, Spelling, Visual storytelling	flawed, many errors	adequate, with some errors	effective, very few errors.	professional & error-free
A = 17-20 B+ = 16 B = 14-15 C+ = 11-13 C = 9-10 D+ = 7-8 D = 6 F= 5 & below					Score

How did ChatGPT do? Here's the command I gave it: "Write a screenplay about a World War II pilot worried about dying on a mission."

Here's the result:

Title: "Wings of Courage"

INT. PILOT'S QUARTERS - NIGHT

A dimly lit room, filled with memorabilia of World War II. JAMES, a seasoned pilot in his early thirties, sits on the edge of his bed, a troubled expression on his face. He flips through a photo album, filled with pictures of his comrades, some crossed out with a pen.

JAMES (whispering to himself) How many more missions can I survive? Will tomorrow be the day I don't make it back?

INT. MESS HALL - DAY

James sits at a table, surrounded by fellow PILOTS. Laughter and conversation fill the air. But James remains distant, lost in his own thoughts. His eyes wander, observing the camaraderie around him.

PILOT 1 Hey, James, you've been quiet lately. Everything alright?

James snaps back to reality, forcing a smile.

JAMES Yeah, just preoccupied with the upcoming mission, I guess.

PILOT 2 You worry too much, mate. We've flown countless missions, and we're still here.

INT. AIRFIELD - DAY

James walks towards his P-51 MUSTANG plane, his trusted companion. He runs his hand along the fuselage, feeling the cold metal beneath his fingers. His anxiety builds.

JAMES (whispering) Come on, old girl. Bring me back safely, just one more time.

INT. READY ROOM - DAY

James and his comrades sit in a briefing room, maps and mission details projected on the wall. The COMMANDER, a stern man in his fifties, stands at the front.

COMMANDER Gentlemen, our mission tomorrow is critical. We're heading deep into enemy territory to neutralize a high-priority target.

James feels a knot forming in his stomach. His hands tremble slightly.

COMMANDER (CONT'D) We've trained for this. We know our jobs, and we know what's at stake. We're the best damn pilots out there.

James steals glances at his fellow pilots, his nerves still evident.

COMMANDER (CONT'D) Trust in your training, trust in your comrades, and most importantly, trust in yourselves. Now, let's bring the fight to them!

The pilots cheer, but James's enthusiasm is restrained.

INT. SQUADRON BRIEFING ROOM - NIGHT

James sits alone at a table, sipping coffee, his face etched with worry. Suddenly, the door opens, revealing SARAH, a young nurse in her twenties. She walks over to James, concern in her eyes.

SARAH You seem troubled, James. Is there something bothering you?

James hesitates, then opens up to Sarah.

JAMES I can't shake this feeling, Sarah. Every mission, the fear of not making it back haunts me. I've seen too many friends go down.

Sarah reaches out, placing a comforting hand on James's.

SARAH It's natural to be scared, James. But remember why you're fighting. Remember the lives you've saved, the freedom you're defending. Have faith in your skills and the strength of your spirit.

James gazes at Sarah, finding solace in her words.

INT. AIRPLANE COCKPIT - DAY

James sits in the cockpit of his Mustang, gripping the control stick tightly. He takes a deep breath, steeling himself for what lies ahead.

JAMES (to himself) I can do this. I will come back.

EXT. SKIES ABOVE ENEMY TERRITORY - DAY

James's squadron soars through the skies, their engines roaring. Enemy flak erupts around them, the sky filled with chaos. James skillfully maneuvers through the onslaught, determination etched on his face.

INT. AIRPLANE COCKPIT - DAY

James scans the skies, his heart pounding. Suddenly, an enemy fighter appears, heading straight toward him. Fear grips James, but he remembers Sarah's words, finding strength in them.

EXT. SKIES ABOVE ENEMY TERRITORY - DAY

James engages the enemy fighter, engaging in a fierce dogfight. Bullets whiz past him, but he maintains his focus, his skills honed through countless missions.

INT. AIRPLANE COCKPIT - DAY

James's plane shudders from a near-miss, smoke filling the cockpit. Fear and doubt threaten to overwhelm him, but he takes a deep breath, remembering his purpose.

JAMES (out loud) I will make it back. I must.

EXT. SKIES ABOVE ENEMY TERRITORY - DAY

James maneuvers his plane, deftly evading enemy fire. With a well-placed shot, he takes down the enemy fighter, the victory surging through his veins.

INT. AIRPLANE COCKPIT - DAY

A smile breaks across James's face, his confidence restored. He adjusts his course, heading back towards home.

EXT. AIRFIELD - DAY

James's plane lands on the airfield, rolling to a stop. The canopy opens, and James climbs out, greeted by cheers from his comrades.

COMMANDER (slapping James's back) Well done, James! You brought us all back.

James looks around, relief and gratitude filling his eyes.

JAMES (with a mix of relief and joy) We did it. We made it back.

Sarah stands in the crowd, her eyes meeting James's. They share a moment of profound understanding and connection.

FADE OUT.

Okay. If this script came from a student, how would I rank it. Note: I am giving the ChatGPT the benefit of the doubt concerning screenplay formatting. I instructed all of my students to use screenplay writing programs like Final Draft or Celtx, so, if ChatGPT attended my class, it would have used the proper formatting. As it stands now, ChatGPT simply doesn't have the ability to write in traditional screenwriting format. (Plus, I can't execute proper screenplay formatting in this blog either!)

1). Protagonist: Description, Motivation, Action. Although James' description was weak, overall it was a adequate portrayal. 2 points.

2). Antagonist/Conflict: Description, Motivation, Action. The conflict was essentially internal, but it was effective. 3 points.

3). Dramatic Structure: Setup, Development & Climax/Resolution. The script was structurally sound with a beginning, middle and end. I would rate it effective. 3 points.

4). Format: Script format was specified by the instructor & text. For reasons I mentioned earlier, I am giving it 4 points.

5). Writing: Grammar, Punctuation, Spelling, Visual storytelling. It's grammar punctuation and spelling was better than most first drafts I got from my students! That said, ChatGPT did not excel at visual storytelling. 3 points.

Total Score: 15 points. That's a B!

But was it any good? Absolutely not! Although it hit all the marks demanded by the grading rubric, the script was little more than a trite collection of cliches and tropes. The difference between adequate words on a page and great screenwriting is something I call sparkle. ChatGPT didn't come within a mile of sparkle. However, I am happy to report that there was always at least one student in my class that possessed it.

I honestly believe a few of my students had the raw talent needed to become professional screenwriters, if they stuck with it. I am less optimistic about ChatGPT, but there's no denying it would pass my class.

If I taught today, I suspect a lot of the assignments I would get will have been written by AI and simply reformatted. That's kind of scary when you think about it. How could a teacher possibly guard against that?

I've beaten ChatGPT up quite a bit. Now let's see how Bard would do with the same prompt: "Write a screenplay about a World War II pilot worried about dying on a mission."

Here's the Bard version:

Scene 1

INT. PLANE - DAY

A young pilot, JACK, sits in the cockpit of a World War II fighter plane. He is sweating and his hands are shaking.

JACK (to himself) I can't do this. I can't die.

Scene 2

EXT. SKY - DAY

The plane takes off and soars into the sky. Jack looks down at the ground below and sees the vastness of the ocean. He feels a wave of nausea wash over him.

JACK (to himself) I'm going to die.

Scene 3

INT. PLANE - DAY

Jack looks over at his co-pilot, who is calm and collected.

JACK (to co-pilot) How can you be so calm?

CO-PILOT Because I've done this a hundred times before.

JACK But what if something goes wrong? What if we get shot down?

CO-PILOT Then we die. But that's a risk we all take when we sign up to fly these missions.

JACK I don't want to die.

CO-PILOT No one wants to die, Jack. But sometimes it's a necessary sacrifice.

Jack looks out the window again and sees the enemy ships in the distance. He takes a deep breath and tries to calm his nerves.

JACK I'm ready.

Scene 4

EXT. SKY - DAY

The plane dives towards the enemy ships. Jack opens fire with his machine guns. The enemy ships return fire, but the plane is unharmed.

JACK We're doing it!

Scene 5

INT. PLANE - DAY

The plane flies back to base. Jack and his co-pilot are greeted as heroes.

JACK (to co-pilot) I can't believe we made it.

CO-PILOT I told you you'd be fine.

Jack smiles and shakes his head. He's glad to be alive.

JACK Thank you.

CO-PILOT For what?

JACK For saving my life.

CO-PILOT It was my pleasure.

Jack and his co-pilot walk off the runway, arm in arm. They've survived another mission, and they're both one step closer to home.

Wow. Not as good. No real descriptions. Less detail. Not as accurate. I mean, what kind of fighter was this? There were plenty of World War II two-seater fighters, but I don't think there were any with co-pilots sitting side-by-side. Also, I would have to downgrade the formatting simply because of the scene numbering. Also, how did the Co-Pilot save our hero's life? With the most mediocre pep talk in history? Nah, I don't buy this at all.

Let's grade Bard:

1). Protagonist: Description, Motivation, Action. Lackluster portrayal. 1 point.

2). Antagonist/Conflict: Description, Motivation, Action. Lackluster. 1 point.

3). Dramatic Structure: Setup, Development & Climax/Resolution. It had a beginning middle and end, but it was utterly uncompelling and the resolution felt unearned. 2 points.

4). Format: Script format was specified by the instructor & text. For reasons I mentioned earlier, I am giving the program the benefit of the doubt. However, I am downgrading it one point for the inappropriate scene numbering. 3 points.

5). Writing: Grammar, Punctuation, Spelling, Visual storytelling. Grammar punctuation and spelling were fine. No attempt at visual storytelling. 3 points.

Total Score: 10 points. That's still a C!

Bard would pass my class too. Yikes. That's scary.

Final Thoughts. Neither story rose to the sparkle level. Not even close. I didn't find anything really compelling or genuinely heartfelt in either of them. I think the programs will eventually be able to generate compelling plots, but I don't see the beginning of true human characterization in them yet. Compelling characters speak to our deepest needs and concerns. We want to see ourselves in the story. That's the talent great writers provide.

My old creative director at the advertising agency Smith Burke & Azzam, Jeff Millman, had a little placard on his desk which read: Who Cares? If you handed him a script that he didn't find compelling, he would simply point to it.

I know where Jeff would be pointing if he read these scripts.....

Still, the current abilities of ChatGPT are somewhat disconcerting. Still, many of my creative friends console themselves because they say the program cannot come up with the initial ideas. We'll see about that in an upcoming blog!

My novel Chapel Street is now available! You can buy the Kindle and paperback at Amazon and the Nook, paperback and hardcover at Barnes & Noble. Chapel Street is the tale of a young man battling a demonic entity that has driven members of his family to suicide for generations. It was inspired by an actual haunting.

Learn more about the book, click Here.

Listen to me read some chapters here:

Chapel Street - Prologue - My Mother

Chapel Street - Chapter 1 - RestingPlace.com

Chapel Street - Chapter 2 - Elisabetta

Chapel Street - Chapter 3 - The Upload

Chapel Street - Chapter 4 - The Kobayashi Maru

Read about the true haunting that inspired the novel here: