class: center, middle <!-- Formatting must occur immediately after the slide-start declaration --> <!-- https://github.com/gnab/remark/wiki --><!-- TODO: Darya - Notes to self: make google docs for landing, plus x 2 for final challenge - Make sure to mention we'll publish google docs later - Make sure to have more visuals, even given limitation of tech stack TODO: Brian - bloom lecture notes - review lecture notes in general --> # Lesson co-development using reverse design: a worked example  Darya Vanichkina PhD FHEA, University of Sydney Brian Ballsun-Stanton PhD, Macquarie University --- # Links: * Presentation: http://bit.ly/metameta-slides * Etherpad: http://bit.ly/metameta-etherpad * Google Docs for groupwork: http://bit.ly/metameta-drive * Timer: https://cuckoo.team/metameta * Github repo for presentation: https://github.com/dvanic/metametalesson * Minute Cards: http://bit.ly/metameta-feedback * Zoom call: https://carpentries.zoom.us/my/carpentriesroom3 --- <!-- /$$$$$$$ /$$ /$$ | $$__ $$ | $$ /$$$$ | $$ \ $$ /$$$$$$ /$$$$$$ /$$$$$$ |_ $$ | $$$$$$$/|____ $$ /$$__ $$|_ $$_/ | $$ | $$____/ /$$$$$$$| $$ \__/ | $$ | $$ | $$ /$$__ $$| $$ | $$ /$$ | $$ | $$ | $$$$$$$| $$ | $$$$/ /$$$$$$ |__/ \_______/|__/ \___/ |______/ ##################################### # Agenda: Part 1 ###################################### 1. Who are we? What are we doing here? What's going on? 1. Restate our objective 1. Disclaimer: this is our alpha run (remember this for later) Removed 1. Aside: Common traps 1. D What to teach: Making a target audience 1. D What to teach: Making learner profiles 1. D What to teach: Mapping skills and goals from learners 1. B Aside: Designing a carpentries workshop versus designing a lesson 1. B Challenges: Finding data 1. D Exercise: Form groups, make personae, make summative task to test goal. --> # Introduction: who are we? <table> <tr> <td style="width:50%"> <b> Darya </b> <img style="width:100%" src="images/daryaVhead1_1Mb.jpg"/> </td> <td> <b> Brian </b> <img style="width:90%" src="images/BrianSuit.jpg"/> </td> </tr></table> --- # Purpose of this workshop <i> In this session, we demonstrate how the constraints of reverse design reduce the uncertainty of a blank page titled 'Lesson 1'. Nowhere does the Carpentries demonstrate the full cycle of instructional design. We propose this workshop - co-developed by two experienced instructors - using this approach. The process of the development of this workshop is documented and shared as the content of the workshop. Attendees will see how learner personas, faded examples, concept maps, formative and summative assessment create requirements for lesson design. We provide a worked example of the reverse design process without any assumptions of prior computational knowledge. After the workshop, attendees can combine their experience of the lesson we developed with the process of development, which we share with them, to apply this framework to their own training work. </i> ??? Darya --- # Agenda <table style="margin:auto;margin-top:auto;"> <tr > <th style="width:25%;border-bottom: 1px solid black"> Timing </th><th style="width:75%;border-bottom:1px solid black;"> Task</th> </tr> <tr> <td>20 minutes</td><td> "Who and what" </td> </tr> <tr> <td>10 minutes</td><td> Group Formative Challenge 1 </td> </tr> <tr><td> 10 minutes </td><td> Break </td></tr> <tr> <td>20 minutes</td><td> "Narrative and Challenges"</td> </tr> <tr> <td>10 minutes</td><td> Group Formative Challenge 2 </td> </tr> <tr><td> 10 minutes </td><td> Break </td></tr> <tr> <td>20 minutes</td><td> "Timing and Talking" </td> </tr> <tr> <td>15 minutes</td><td> Meta-Group Summative Challenge </td> </tr> <tr> <td>5 minutes</td><td> Minute Cards </td> </tr> <tr> <td>post-workshop discussion</td><td> Discussion and Feedback </td> </tr> </table> --- # The agenda/bucket list/what I want to teach: GET IT OUT! .center[] ??? Darya --- # Real step 1: Who are your learners?  - Reverse design: start with who your learners are and what they should be able to DO at the end of the course. - Everything is developed from that premise ??? Darya --- # Draft personas for this workshops  - Alex - works as a trainer embedded at a Uni. Has done a lot of training but never developed any courses before. - Ali - attended Toby's talk, Carpentries instructor wanting to stretch their wings and begin contributing to some of the incubator lessons. Does not "code". - Uri - another instructor trainer with extensive course developer experience. Interested in what we have to say "just because". - Karen - watching session later on YouTube, not familiar with Carpentries philosophy or teaching style - ... (note: all we got from Central was ~60 attendees registered 😢) ??? Darya --- # Add "flesh" to the personas: Alex <!-- Link to rest of personas fleshed out here Also need a photo --> <!-- https://github.com/gnab/remark/issues/72 --> <table> <tr> <td style="width:30%">  </td> <td> <ul> <li> Alex is a PhD-qualified particle physicist who learned to use python for research work. During PhD, delivering introductory programming training with a cross-university training organisation as a casual tutor. </li> <li> After graduation, found employment with this organisation as a full-time trainer. </li> <li> Would like to grow in role by developing new training, and doesn't really know where to start.</li> <li> Has tried by making a list of topics to cover in a new course.</li> <li> Hasn't been exposed to any learning theory, received any training in instruction beyond that learned from peers on the job - and doesn't feel comfortable reaching out to trainers around who do have some of this background for support, as most of them are not employed on a full-time contracts (imposter syndrome). </li> <li> Deeply concerned about climate change. </li> </td> </tr> </table> ??? Darya --- # Add "flesh" to the personas: Alex <!-- Link to rest of personas fleshed out here Also need a photo --> <table> <tr> <td style="width:30%"> <ul> <li style="color: #648FFF">General "stuff" </li> <li style="color:#DC267F"> Starting point </li> <li style="color:#FE6100">Needs </li> <li style="color:#FFB000">Considerations </li> </ul> </td> <td> <ul> <li style="color: #648FFF"> Alex is a PhD-qualified particle physicist who learned to use python for research work. <span style="color:#DC267F">During PhD, delivering introductory programming training with a cross-university training organisation as a casual tutor. </span></li> <li style="color: #648FFF">After graduation, found employment with this organisation as a full-time trainer. </li> <li style="color:#FE6100">Would like to grow in role by developing new training, and doesn't really know where to start.</li> <li style="color:#DC267F">Has tried by making a list of topics to cover in a new course.</li> <li style="color:#DC267F"> Hasn't been exposed to any learning theory, received any training in instruction beyond that learned from peers on the job- <span style="color:#FFB000">and doesn't feel comfortable reaching out to trainers around who do have some of this background for support, as most of them are not employed on a full-time contracts (imposter syndrome). </span></li> <li style="color: #648FFF">Deeply concerned about climate change. </li> </td> </tr> </table> ??? Darya --- # The rest of the personas <!-- Removed the second reference to not a real person, as it was producing the same image. --> <a href="https://github.com/dvanic/metametalesson/blob/master/personas.md">https://tinyurl.com/personasdemo2</a> ??? Darya --- # Think about what learners will be able to DO at the end - To have a sense of what the reverse design process looks like, and - by referring to this presentation and using other resources - begin to develop a lesson using its core principles ### How do we know learners have been able to do this? - <b>Summative assessment</b>: how is the learner - you - going to show us that you've learned what we're trying to teach? - Well, to develop a course plan or a mini-course, as that's more feasible in < 2 hours - It will be a group task: peer learning ??? Darya Note: give it a go; then go to resources! --- # Talk-at-you aside 1: Finding datasets and making them teachable **Inclusivity in technical training is tricky: start with real data and real scenarios and see the researchers you are teaching.** - Dataset, scenario, and tools chosen informs: - Cognitive load - Sense of identification with the problem - Motivations ??? Brian * Your learners are not you * They need to discover the utility of the methods, not just be told * Pulling from a real dataset, linked to their methods or discipline is one way to do this * It also helps with cognitive load, * they don't need to learn what the terms mean -- <!-- Incremental slide --> - Performance of FAIR data - When we use and cite "real" data, we demonstrate "reproducible research" to participants - Participants must understand point of data and nature of data in about 5 minutes. **Simplify ruthlessly.** - Make sure there's a license which allows this. ??? * Real data also allows us to show the utility of reproducible research * "Real" allows them to understand in 5 minutes. Cut it down. -- - Meet learners where they are - Carpentries participants have *hugely different* skill sets. - A local lesson can assume shared context, jargon, and competencies. ??? * Skills known by postgrads on the other side of the world, in different faculties should not be assumed. * Starting with discipline specific data builds on expected norms of discipline --- # Our summative task for today: group development of mini-courses in < 2 hours! <table> <tr> <td style="width:30%">  </td> <td> <ul> <li> Starting a new job off on the right foot (the first month) </li> <li> Diet and exercise to stay healthy</li> </td> </tr> </table> ??? Darya Each of us reflects on multiple groups in same google doc --- # Our summative task for today: why did we choose these? - Starting a new job off on the right foot (the first month) - Diet and exercise to stay healthy -- ### Assess relevance/alignment of summative assessment with persona learning goals! | Persona | Reflection | Challenges | |------------|----------------------------------|----------------| | Alex | Can contribute to both; good practice/aligned with goal | Imposter syndrome | | Ali | Happy that tasks don't require programming: can participate | - | | Uri | May find topics too "little" of a challenge | Could "dominate" | | Karen | Won't be able to participate in learning tasks | - | ??? Darya --- # Our summative task for today: why did we choose these? - Starting a new job off on the right foot (the first month) - Diet and exercise to stay healthy ### Other notes: - "Generic" topics: most people will be able to contribute - Relatively consistent recommendations across cultures and life experiences (although nuances can be QUITE different) ??? Darya --- # Formative Challenge 1: In your breakout rooms (10 min or until 1st break?): 1. Open https://cuckoo.team/metameta in a new tab and enter a name or pesudonym 2. Introduce yourselves briefly (Who you are, what you do, one interesting thing you study/do, and your favourite tool) 3. Decide on a topic: * [Starting a new job off on the right foot (the first month)](https://docs.google.com/document/d/1stNyGlXlMF0q6iGbFECW7PyUCX602h0LulUxx24lOx0/edit) * [Diet and exercise to stay healthy](https://docs.google.com/document/d/1s7fwbwI-Fpia7mkRT59gUwGeRdGpeZS6LWJOvBJxIDI/edit) 4. All group members should open the document and find the group labeled with your breakout room number. 5. Draft two short personas for your courses 6. Agree on a summative assessment task 7. Reflect on the alignment of that assessment task with learning goals 8. Be prepared to rejoin the main room when the timer sounds (and then we will go on break) Group documents available at: http://bit.ly/metameta-drive Timer: https://cuckoo.team/metameta ??? Darya --- class:center, middle # Break time 10 minutes Timer: https://cuckoo.team/metameta ## Links: Presentation: http://bit.ly/metameta-slides Etherpad: http://bit.ly/metameta-etherpad Google Docs for groupwork: http://bit.ly/metameta-drive Github repo for presentation: https://github.com/dvanic/metametalesson Minute Cards: http://bit.ly/metameta-feedback Zoom call: https://carpentries.zoom.us/my/carpentriesroom3 ??? Come back. Summary of highlights from each group (in chat if HEAPS of people//time) --- <!-- /$$$$$$$ /$$ /$$$$$$ | $$__ $$ | $$ /$$__ $$ | $$ \ $$ /$$$$$$ /$$$$$$ /$$$$$$ |__/ \ $$ | $$$$$$$/|____ $$ /$$__ $$|_ $$_/ /$$$$$$/ | $$____/ /$$$$$$$| $$ \__/ | $$ /$$____/ | $$ /$$__ $$| $$ | $$ /$$ | $$ | $$ | $$$$$$$| $$ | $$$$/ | $$$$$$$$ |__/ \_______/|__/ \___/ |________/ ############################################# # Agenda: Part 2 ##########################################333 1. DB Reflection: reflect on group pages 1. B Narrative: What do people need to know to complete each formative assessment () 1. B First draft: Write an outline of your narrative for each code chunk 1. B Aside: Brian's theory of outlines. What to say is different from how to say 1. B First draft: Allocate skills as learning objectives to lessons. 1. B First draft: Allocate challenge+code+narrative to each episode to satisfy learning objectives 1. D Reminder: formative assessments 1. D Challenges: Making a challenge to exercise a skill 1. D Challenges: Designing a good MCQ 1. D Challenges: Parson's problems 1. D Challenges: Fill in the blank 1. D Challenges: Change context 1. B Exercise: Design formative assessments needed to accomplish summative task. --> # Formative Reflection 1 .center[  ] --- ## What are the key tasks needed to achieve this? 1. Personas * 2. Learning outcome - a summative assessment task * 3. Breaking that summative assessment into subtasks (this step! - usually a list of things you need to do to get from A to B) ### For this course: 2. Identifying formative assessments that allow us to practice/build up to being able to execute those subtasks 3. [Knowing the arsenal of types of formative assessments that can be combined for effective learning, and the pros/cons of each] 4. Identifying the minimum necessarily information for getting students "to" the assessment 5. Reviewing the series through the persona lens & identifying time constraints/skippable-if-need-be formative assessments ??? 5 minutes ??? D Need to note somewhere that formative assessments are also helpful for us to diagnose current learner mental models, and that they help us reduce cognitive load --- # Skill decomposition, explanation, and setup * Research problems are formed out of many small, shared, tasks (and the occasional flash of insight) * Each formative assessment will expect learners to perform a common task which the summative assessment requires. * The narrative around a task is: "What does a learner need to know in order to complete this formative assessment" * Intended outcome * How do they know they have achieved success? * What pattern is the assessment demonstrating? * How does this link to the bigger picture? * Live coding **is a very accessible formative assessment.** ??? * Research problems are unique * They can be broken apart into simpler tasks * We can teach to these tasks * Each of these split-apart tasks should reduce to a formative assessment * Learners need to be able to perform the task * Know when they're done * Know what to expect * Learn norms of error correction * Relate the small task to larger context * Sum up: Break the large summative assessment into small common tasks * Each task should have a clear end state which forms part of the summative assessment * Must teach useful patterns in professional practice * Write narrative to situate task: * Pattern of "Tell, then show, then do" * Useful for planning narrative --- # Aside: Bloom's taxonomy * Decomposed skills should use Bloom's taxonomy  .center[Recognise → Explain → Implement → Differentiate → Critique → Generate] <div style="font-size:70%"> Anderson, Lorin W., David R. Krathwohl, and Benjamin S. Bloom. <i>A Taxonomy for Learning, Teaching, and Assessing: a Revision of Bloom’s Taxonomy of Educational Objectives</i>. Editors, Lorin W. Anderson, David Krathwohl ; Contributors, Peter W. Airasian ... [et Al.]. Complete ed. New York: Longman, 2001. Print.</div> ??? Brian https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/ * This pyramid has layers that build on each other. Each layer provides a tool used as *part* of processing on a higher level. * The foundation supports higher order. * In order to create, we need to be able to evaluate options. * In order to evaluate, we need to be --- # Allocate skills as learning objectives to the workshop. * 45 minutes ago, we said to: "Think about what learners will be able to DO at the end" * What *specific* skills will they need to do that? * Use Bloom's taxonomy words here when listing out the skills they will need. * Don't split them apart yet, just get a sense of prerequisites * These skills are both a functional decomposition of the final goal and specific takeaways that learners will know they have achieved during the workshop. * You will be writing challenges to have learners demonstrate to themselves that they can perform each of these skills. ??? Brian --- # Talk-at-you aside 2: For text in the lesson, Start with an outline, not a script * After writing skills and challenges, you'll need to teach to them by writing text around them. This should not be a full script, but essential points. * An outline specifies *what* you want to say, and how thoughts are arranged * Writing a text or script specifies *how* you want to communicate thoughts previously arranged. * When designing a work, outline at roughly a 1:4 ratio (1 word of outline for every 4 expected words of script) * No sentences!  ??? * An outline focuses on relationships between thoughts * Key points * Organisation and flow -- * We are going to end up throwing most of the text away. * Focus on how ideas relate to each other, not how to express them. * Transitions, callouts, and asides come later. ??? * Differentiate what from how * Expect most words to be thrown away * Don't waste writing * This slide has sentences and is a bad example of an outline. * Narrative writing * No Transitions, * No callouts * For now, identify only what they need to know in order to accomplish the assessment, how they know they've succeeded, and how it links to the final goal. --- # Making a challenge to exercise a skill .center[] ## [Doing the laundry](https://www.thespruce.com/how-to-do-laundry-2146149) 1. Read the labels 2. Sort into whites, coloureds, heavily soiled; subsort by type of fabric (denim vs t-shirts, delicates, wool)/"lintiness" 5. Pre-treat stains before washing, using the appropriate method 4. Select appropriate type and quantity of detergent 6. Pick water temperature and wash cycle 7. Load properly to 2/3 full 8. After wash, hang or place in dryer, as appropriate to fabric and life ??? 1. D Challenges: Making a challenge to exercise a skill --- # Aside: why laundry? - Complex process that is done nearly everyday, by ALL of us. - Relatively consistent among cultures/countries (unlike cooking!). | Persona | Reflection | Challenges | |------------|----------------------------------|----------------| | Alex | No extra cognitive load; can understand | 🧺 | | Ali | Not really relevant as laundry done by partner | 😭 | | Uri | Has never thought so much about laundry in their lives | 🤔 | | Karen | No issues with understanding/participating | 🧺 | ??? D --- # Designing a good multiple choice question (MCQ) Which symbol means washing in warm water (40C/105F)? <ul> <li> <img src="images/wash4.png" alt="Washing symbol 1" width="60" height="50"> </li> <li> <img src="images/wash2.png" alt="Washing symbol 2" width="60" height="50"> </li> <li> <img src="images/wash3.png" alt="Washing symbol 3" width="60" height="50"> </li> <li> <img src="images/wash1.png" alt="Washing symbol 4" width="60" height="50"> </li> <li> <img src="images/wash5.png" alt="Washing symbol 5" width="60" height="50"> </li> </td> </tr> ??? Darya A good MCQ tests for conceptual misunderstanding rather than simple factual knowledge. If you are having a hard time coming up with diagnostic distractors, then either you need to think more about your learners’ mental models, or your question simply isn’t a good starting point for an MCQ. https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/ --- # Designing a good multiple choice question (MCQ) Which symbol means washing in warm water (40C/105F)? <ul> <li> <img src="images/wash4.png" alt="Washing symbol 1" width="60" height="50"> </li> <li> <img src="images/wash2.png" alt="Washing symbol 2" width="60" height="50"> </li> <li> <img src="images/wash3.png" alt="Washing symbol 3" width="60" height="50"> </li> <li> <img src="images/wash1.png" alt="Washing symbol 4" width="60" height="50"> *</li> <li> <img src="images/wash5.png" alt="Washing symbol 5" width="60" height="50"> </li> </td> </tr> ??? A good MCQ tests for conceptual misunderstanding rather than simple factual knowledge. If you are having a hard time coming up with diagnostic distractors, then either you need to think more about your learners’ mental models, or your question simply isn’t a good starting point for an MCQ. The question must probe for conceptual understanding and misconceptions (rather than check simple factual knowledge). https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/ --- # Why was that a "good" MCQ? Which symbol means washing in warm water (40C/105F)? <ul> <li> <img src="images/wash4.png" alt="Washing symbol 1" width="60" height="50"> Distractor: confused about dryer symbol vs washing</li> <li> <img src="images/wash2.png" alt="Washing symbol 2" width="60" height="50"> Distractor: confused about washing intensity vs temperature </li> <li> <img src="images/wash3.png" alt="Washing symbol 3" width="60" height="50"> Distractor: number of dots == temperature </li> <li> <img src="images/wash1.png" alt="Washing symbol 4" width="60" height="50"> Correct answer</li> <li> <img src="images/wash5.png" alt="Washing symbol 5" width="60" height="50"> Distractor: confused about ironing symbol vs washing</li> </td> </tr> ??? Darya 1. Each distractor is diagnostic 2. We are "assessing" understanding of only ONE aspect of the giant multi-pronged approach to doing laundry outlined above 3. Make it a think-pair-share!!! Peer learning/repetition/justification to group --- # Additional resources for developing MCQs:</b></p> - [Vanderbilt University checklist for writing good MCQs](https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/ ) - [Carpentries Instructor Training](https://carpentries.github.io/instructor-training/02-practice-learning/index.html) ## Blooms and MCQs  --- # Parson's problems: laundry Rearrange the steps below to be in the correct order: <table> <tr> <td style="width:30%"">  </td> <td> <ol> <li>Choose the detergent you want to use </li> <li>Pre-treat the stains with stain remover</li> <li>Sort out woolen items into a separate pile</li> <li>Sort out dark colours into a separate pile</li> <li>Look at the labels on clothes and sort into piles based on indicated temperature and recommended wash cycle</li> <li>Sort out heavily soiled items into a separate pile</li> <li>Sort out delicates into a separate pile</li> </ol> </td> </tr> </table> ??? Darya 1. D Challenges: Parson's problems --- # Parson's problems: laundry Rearrange the steps below to be in the correct order: <table> <tr> <td style="width:30%">  </td> <td> <ol> <li>Look at the labels on clothes and sort into piles based on indicated temperature and recommended wash cycle</li> <li>Sort out heavily soiled items into a separate pile</li> <li>Sort out dark colours into a separate pile</li> <li>Sort out woolen items into a separate pile</li> <li>Sort out delicates into a separate pile (may not be needed!)</li> <li> Choose the detergent you want to use </li> <li>Pre-treat the stains with stain remover</li> <li><s>Go crazy because no one quite has enough laundry to warrant running so many separate machines</s></li> </ol> </td> </tr> </table> ??? Darya 1. D Challenges: Parson's problems --- # Worked examples: start with fill in the blank .center[] You are sorting your to-be-washed-at 40 degrees pile of white clothes. You notice a tomato stain on your favourite (cotton) t-shirt. You rub in 5 ml of a <i>?????????????</i> stain remover to pre-treat the stain before setting up a 40C permanent pressure ("mix") washing cycle with 800 rpm spin. ??? Darya --- # Worked examples: start with fill in the blank .center[] You are sorting your to-be-washed-at 40 degrees pile of white clothes. You notice a tomato stain on your favourite (cotton) t-shirt. You rub in 5 ml of an [enzymatic] stain remover to pre-treat the stain before setting up a 40C permanent pressure ("mix") washing cycle with 800 rpm spin. <i>Note: if this were actually a course about laundry I wouldn't ASK you the question, I'd give you the pre-sorted piles and asked you to do it hands-on!</i> ??? Darya 1. D Challenges: Fill in the blank --- # Worked examples: continue with "modify/adapt the problem" > <span style="color:grey">You have sorted your to-be-washed-at 40 degrees pile of white clothes. You noticed a tomato stain on your favourite (cotton) t-shirt, and rubbed in 5 ml of an enzymatic stain remover to pre-treat the stain before setting up a 40C permanent pressure ("mix") washing cycle with 800 rpm spin. </span> That cycle has finished, and now you are setting the next load of white clothes at 60 degrees, without stains. Which settings would you select on your washing machine? [Python novice lessons have a lot of good examples of combining problems in order](https://swcarpentry.github.io/python-novice-inflammation/02-numpy/index.html) ??? Darya --- # Blooms and worked examples  ??? Darya --- # "Actual" problems .center[] Provide a pile of laundry (without staining that requires treatment). Ask to sort into piles based on garment labels and colours. Make sure to get group feedback first before presenting to "instructor" ??? Darya D Remind about cognitive load and interleavement of spaced retrieval practice --- # Challenges: Change context - <i>"Once learners have had an opportunity to practice using a concept in the same context as it was originally taught (direct application challenges), it’s time to stretch their understanding by asking them to apply the same concept in a different context."</i> - [More in the CDH](https://cdh.carpentries.org/designing-challenges.html#use-the-concept-in-a-different-context) - [Greg Wilson's programming problem types](https://third-bit.com/2017/10/16/exercise-types.html)  ??? Darya 1. D Challenges: Change context --- # Aside: How does live coding factor into this? * Live coding is an assessment * Attention and accuracy check. * Are they aware of *exactly* what is being written? * Errors * Expectations of the medium (precision and syntax) * It is a formative assessment * Learners use sticky notes to indicate state * Green = done, comprehended * Red = help, error * none = still working A constant refrain for sticky notes during a workshop is a constant refrain of formative assessments. ??? Brian --- # Formative Challenge 2: Design formative assessments needed to accomplish summative task. **10 minutes** In your groups, go back to the Google docs. Start to design some formative assessments you might use to build towards your summative task. Make sure you design at least one: - MCQ - Parson's problems - Fill in the blank - Faded example - Actual problem - Context-changing problem Group documents available at: http://bit.ly/metameta-drive Timer: https://cuckoo.team/metameta ??? Brian --- class:center, middle # Break time 10 minutes Timer: https://cuckoo.team/metameta ## Links: Presentation: http://bit.ly/metameta-slides Etherpad: http://bit.ly/metameta-etherpad Google Docs for groupwork: http://bit.ly/metameta-drive Github repo for presentation: https://github.com/dvanic/metametalesson Minute Cards: http://bit.ly/metameta-feedback Zoom call: https://carpentries.zoom.us/my/carpentriesroom3 <!-- /$$$$$$$ /$$ /$$$$$$ | $$__ $$ | $$ /$$__ $$ | $$ \ $$ /$$$$$$ /$$$$$$ /$$$$$$ |__/ \ $$ | $$$$$$$/|____ $$ /$$__ $$|_ $$_/ /$$$$$/ | $$____/ /$$$$$$$| $$ \__/ | $$ |___ $$ | $$ /$$__ $$| $$ | $$ /$$ /$$ \ $$ | $$ | $$$$$$$| $$ | $$$$/ | $$$$$$/ |__/ \_______/|__/ \___/ \______/ ################################# # Agenda: Part 3 ################################## 1. DB Reflection: Reflect on choice of formative assessments. 1. B NEW Added first draft allocation of challenge+code+narrative to satisfy objectives. 1. B Aside: Temporal budgets and overkill 1. B Aside: How does live coding factor into this? Removed? Live coding is the simplest possible formative assessment: follow along. 1. B Time yourself talking about the code you're typing, and then *do* your challenges. This *2 is your minimum time (you already know the answers). 1. D Run a pilot: Teach your lesson from outline to your helpers and co-instructors 1. D Run a pilot: While issues: iterate 1. D Run a pilot: Flesh out narrative from outline, move outline into instructor notes 1. B Summative exercise: proving that we've designed a workshop?? --> --- # Reflection on formative assignments 1. What were you testing in the assignment? 1. How did wrong answers help you identify errors in thinking? 1. What knowledge did the right answer require? ??? Darya Reflection: Reflect on choice of formative assessments. --- # Time Yourself Doing and Talking * Challenges and narrative are only half the battle. * Exercises are slow, live-coding is slow * For each challenge plus its outline: * Time yourself talking through the outline and doing each challenge * Multiply that time by two: * You know what you mean * You know what the answers are * You are not interrupted and likely not speaking at a slow presentation pace * This is your minimum time ??? Brian --- # Talking to you aside 3: Temporal budgets and overkill * You will write too much * You will design too much * What you design will suffer from the "expert awareness gap" * Aim low and outline Lesson 1. *Don't overprepare:* spend time on what you know you will present Lesson 2. *Don't crowd the lesson:* allocating time for answering questions about the content and discussion about the lesson during your alpha preparations provides a "crumple zone" for emergencies (a challenge is too hard or an explanation needs to be redone.) ??? Brian Overprepare: you know your content and can unpack it and discuss it more thoroughly through ad-libbing. Adding content ahead of time that you know you won't get to wastes your time. Crowd the lesson: As they note in instructor training, it is better to go too slowly and bore some of the more expert members of your audience than it is to go too quickly and alienate those who are unsure. This heuristic is even more critical in an alpha lesson where you don't know how useful any of your planned explanations are. Aiming for 80% of your target time during initial development at the very worst allows for discussion at the end of an episode where your class can reflect on what went well and what needs improvement. --- # Make a concept map  ??? Brian <table> <tr> <td style="width:100%"> A concept map is a directed graph which represents the relationships and **dependencies** of concepts in the workshop. <ul> <li>Your summative assessment is your goal/activity at the center.</li> <li>That activity depends on skills taught by the formative assessments</li> <li>Each assessment has necessary prior concepts which must be explained</li> </ul> </td> </tr> </table> --- # Throw stuff out * Are there any challenges that don't directly support the summative assessment? * Is there any explanatory text which is an *aside*? Theory for the sake of theory? * If something does not directly support the final goal, or a skill needed for the final goal, throw it out. * (Use proper version control so you can get it back later if you have time to fill) ??? Brian --- # Episodes, assessments, narrative, and time-budget * Now you have assessments, talking points, and a sense of how long each assessment + words took you * You also have a list of learner goals phrased using Bloom's taxonomy * You know generally how long you have to teach. (Online or offline) ## Now you get to make the lesson! * Allocate assignments to "episodes" * Make sure each episode has crumple-zones, sections and challenges you can skip depending on your audience, or extra time for questions. * Make sure you have a break at most every 90 minutes. * Title the episodes * Assign the learning objectives from each assignment to the episode ??? Brian This is where the agenda was handy, to see what we ended up with in relation to the original document we were teaching from. Having the context of the exercises allowed us to see which of the agenda points was important, which were forgotten and needed to be added (concept maps) and which could be ignored (technical details of running.) --- # Throw stuff out, again * Getting through three fifths of a carpentry workshop is normal. * If some other instructor gets through three fifths of your lesson, is the goal still (roughly) achieved? * If not, keep throwing things out. It's all in version control, so delete with abandon. ??? Brian --- # Run a pilot <table> <tr> <td style="width:30%"> <img src="images/pilot.png" alt="Two male pilots :(" width="200"> <span style="font-size:75%;font-style:italic">Two male pilots :( Unsplash</span> </td> <td> <ol> <li> Teach the lesson early! <ul><li> If you have the time/bandwidth, teach/work through your lesson with your co-instructors and helpers </li></ul></li> <li> The first delivery of every course is just as much about you learning as it is about your learners: reflect on everything! <ul> <li>Github issues for the advanced</li> Paper notes/post-its work too!</li> <li>Take notes on timing, moments of confusion, mismatches between actual students and personified expectations, "argh" failures and successes.</li> <li>Get LOTS of <a href="https://carpentries.github.io/instructor-training/06-feedback/index.html">feedback</a>: minute cards, one-up-one-down, surveys, unstructured/structured interviews etc</li> </li> </ol> </td> </tr> </table> ??? Darya --- # Summative challenge: Using the combined (across groups now!) group doc of your topic, plan a workshop. **15 minutes** We will assign everyone to two breakout rooms, based on which topic your group has chosen. In this large meta-group, design a workshop by pulling from your prior contributions to the document. * This workshop should have: * Personae, * Skill goals * 1 summative assessment, * 2 formative assessments, * 2 bulletpoints per formative assessment, * and be sorted into episodes. Group documents available at: http://bit.ly/metameta-drive ??? Brian --- # Other stuff we did (without being explicit) 1. Tried to combine visual and auditory information 2. Didn't given the whole picture at once (but this can be useful as well) 3. Taught for peer learning 4. Showed (and admitted to) how to detect, diagnose, and correct common mistakes (also: made them!) ## Wrapping up * Please give us feedback: http://bit.ly/metameta-feedback * Discussion of summative challenges * Questions from the session ??? Darya and Brian <!-- Put links here for bib and resources --> --- # Other resources * CDH: http://cdh.carpentries.org/ * Rstudio instructor training * [Vanderbilt university teaching guides](https://cft.vanderbilt.edu/guides-sub-pages/) * Wilson G. *Ten quick tips for creating an effective lesson.* PLoS Comput Biol. 2019;15(4):e1006915. Published 2019 Apr 11. doi:10.1371/journal.pcbi.1006915: ## Links: * Presentation: http://bit.ly/metameta-slides * Etherpad: http://bit.ly/metameta-etherpad * Google Docs for groupwork: http://bit.ly/metameta-drive * Github repo for presentation: https://github.com/dvanic/metametalesson