MOOCS, Not The Smashing Success

Back when MOOCs first started I was highly skeptical. As they evolved I became convinced that my initial assessment was correct. Now there are some good ones, yet they are far and few between. Science published a review of them in its recent issue.

They note:

It was clear from the first few years of MOOC research that MOOCs disproportionately drew their learners from affluent countries and neighborhoods, and markers of socioeconomic status were correlated with greater persistence and certification. In 2012 to 2013, 80% of learners came from countries rated with high or very high United Nations Human Development Index ratings. That proportion grew slightly through 2015 to 2016, so that the majority of new registrations and certifications came from the world's most affluent countries. Rather than creating new pathways at the margins of global higher education, MOOCs are primarily a complementary asset for learners within existing systems. Last, MOOCs' low completion rate has barely budged, despite 6 years of investment in course development and learning research. A strategy that depends on bringing new learners into higher education cannot succeed if educational institutions cannot support learners in converting their time and financial investment into completing a course to earn a credential with labor market value....MOOCs will not transform higher education and probably will not disappear entirely. Rather, they will provide new supports for specific niches within already existing education systems, primarily supporting already educated learners. The 6-year saga of MOOCs provides a cautionary tale for education policy-makers facing whatever will be the next promoted innovation in education technology, be it artificial intelligence or virtual reality or some unexpected new entrant. New education technologies are rarely disruptive but instead are domesticated by existing cultures and systems.

Here are my observations:

 1. My first one was an MIT course which I had actually taught decades before. The problem was that the software for the exams was defective, the information for using it was not there, and the overall structure was banal at best.

2. I took an Epigenetics course. After overcoming the language issue, Australian versus US, the most disturbing part was the usage of "peer evaluation" This means one is evaluated by those who know less and are often motivated to suppress those they feel are superior. The worst of the culture exploitation. The instructor was clearly in my opinion clueless on this dynamic. Peer Evaluation is one of the worst concepts in education.

3. Some instructors in courses were totally incomprehensible. Totally. One chemistry course had a few folks rambling across a board of stuff and one wondered if they just wanted to have a MOOC presence.

On the positive side:

1. A Hong Kong medical course was spot on, as was a US chemistry course, as was a Harvard Med Immunology course.

2. In contrast the Harvard Med Genetics course was driven by cartoons, yes real cartoons. It was as costly as the immunology one bust in my opinion targeted for the millennial. 

Overall I have tried well over two dozen of these. A recent couple from Mt Sinai in New York was exceptionally well done. The were short, on a specific topic, well articulated, and kept on topic. 

I found Cousera much better whereas EdX was all too often an overdone attempt as a semester level course. Often missing the target.

The drop rate is significant yet there does not seem to any attempt to find out why.  With such an attitude one sadly sees them going by the wayside.

The best course was the MIT one on introductory Biology with Eric Lander. It was just a well done recording of his lectures with exceptionally well thought out problems. In contrast when a colleague tried a genetics course it lacked the vibrancy of the Lander lectures.

The MOOC Fallacy

The MOOC fallacy is that "students" can grade other students; honestly, fairly, and competently. I bemoaned a course I took a year ago where some 10-20% of one's grade was via "peer review" grading.

Namely the concept is to take students, often not having English as a first language, or even second, and let them grade other students. Now besides the language barrier this means people who have no knowledge grade those with some knowledge. Furthermore what I also observed was gaming of the system by ganging up on those grades of what was clearly say Americans by those from other countries to suppress the American grades.

This year the course has raised the percent of the grade subject to this less than ethical practice to a level of some 36%. That means collections of students can target others and suppress grades for their benefit.

Anyone who has ever taught knows gaming. This system however verges in my opinion on the immoral. One subjects themselves to the rigor of the material only to have it grades by some one having no understanding of the material along with an interest in them succeeding and you failing. This is a wonderful example of a rigged game. It also in my opinion is an example why no one really trusts these systems.

Perhaps This is why MOOCs Have Problems?

I just read a note on the edX blog extolling the benefits of a MOOC in place of in person lectures. Sometimes the words can also be interpreted to demonstrate the problems more so than extol the thought of benefits.

Let me explain. First they state:

The course attracted over 7,500 students from 147 countries in its first run and was a prized specimen for researchers, data analysts and online course developers alike. By studying the dataset, correlations among different student demographics, learning patterns (e.g., clicking and seeking) and performance can be examined. An interesting finding was that students who performed best in assignments did not have high engagement in forums, probably meaning that they did not require much peer assistance in understanding the contents.

The problem is that the background and interest of the students bias the results greatly. In addition the data set is most likely in error. People report what they want to report and the reliability of anything in a MOOC is highly suspect.

They continue:

Although producing a MOOC takes more effort than face-to-face (F2F) lecturing, in Dr. Kajimoto’s point of view, the benefits outweigh the costs. As every single word in the videos has to be scripted, having a solid script helped Dr. Kajimoto realize how repetitive on-campus lectures are and how time in class can be better spent. Flipping the lectures also resulted in better activities engagement and quality of discussion, as students had much more time and motivation to prepare for tutorials. Peer pressure was also a factor as students fear lagging behind their classmates.

There is a key point in the above, namely that having a solid script helped Dr. Kajimoto realize how repetitive on-campus lectures are and how time in class can be better spent..

Repetitive is often what is needed in class. Saying or presenting a new concept once in one way often does not work. I recall again my first MOOC, if you will, a College Chemistry course on Television in 1959. 6AM and every day for 45 min. Somewhere along the way I missed the definition of a mole, not the ones on skin, but a chemical metric. So for a few weeks, not having access to anyone who knew, I wandered until I found it out. Then, back on track. The same problem befalls MOOCs today. For example a recent MOOC told you to open R, a statistical package, with 4 screens. Well try and find the other 2! It is that type of brick wall that redundant teaching, inefficient teaching if you will, eliminates.

Redundancy allows, enables, conversation and thought. If all you want to do is transfer your facts then the MOOC works, but remember that the other side may not get them. 

Thus this Blog note states a long list of problems, much more so than any benefits.

MOOCS and The Disconnect

I figured I would try another MOOC, this time R, since I have been doing statistics for fifty years and would like to see what they are proposing.

Well like so many of these things, you download R, then see the video, which does not in any way tell you how to load the R windows. The woman giving the lesson just proceeds assuming you know how to work the software!

If you cannot load the required 4 windows how in God's name do you follow the instructions? Well this is a U Texas program so why expect any more. I guess these folks do not do any programming. Remember, avoid ambiguity of expectations folks!

What does some poor person in Romania do here....spend hours flicking keys! Well it is Texas after all. Back to the Alamo!

How NOT to Learn a Language!

I just tried out a MOOC from that college west of Boston famed for a Presidential Candidate. It was Italian. Now being raised on Staten Island, Italian was a second language, Spanish a third. You see Italian was very important. The best priests to go to for Confession were the Italian ones down the street, not the old Irish guys at the local parish. But you had to speak Italian. At least enough.

Then I for some reason had to take Italian with a US State Department group. Then I took Italian from a woman from Trieste; try that accent on after Sicilian. Then I was in Italy. Fact, there is no one Italian and on Staten Island it is like Sicily. So do not try it in Florence.

Now how does this relate to the Italian MOOC. Well, it was the worst possible course ever to try and teach any language. It was literally all over the place. It was high Italian, Milan most likely, and in my picking up five other languages the key is simple:

1. Get the 100 most important words; here, there, numbers, please, thank you, my name is, where is the bathroom, how much is it, etc. You can survive a little,

2. Learn the present tense of about 30 key verbs including to have and to be.

3. Learn 20-50 new words each day. Get index cards and walk around memorizing them.

4. Read the newspaper each day.

5. Watch 2 hours of television each day in the language. Watching the Sopranos in Greek or Sex in the City in German really adds to your insight. On the other hand French TV is all intellectual stuff, like Sartre on steroids.

6. Speak to cab drivers. You can even do this in New York.

7. Try it out on waiters. This may work sometimes.

8. Shop, shop, shop. Read the signs. Find a restroom. It was amazing in Greece, Anthropos, Gynekon, yes anthropology and Gynecologist. Limbic valence on the spot.

9. Never waste time on a MOOC!

The course was almost cartoon like. The exams forced you to answer in a specific way. No one speaks that way. You learn to use phrases to express ideas, requests etc. My favorite experience is that French uses subjunctive, je voudrais, English uses I want. You never say I want in French, unless you do not want it. Language is culture, culture is expression, that is what a language is. Not what this MOOC does.

MOOC Assessments

The assessments of MOOCs has been a spotty effort yet MIT has published some detail in 2014 and it has been announced by MIT that they will be publishing an up to date assessment. We highlight and comment on what is made available:

First, while many MOOC creators and providers have increased access to learning opportunities, those who are accessing MOOCs are disproportionately those who already have college and graduate degrees. 

 That is interesting in a way. First I have observed many via Discussion groups who have little or no experience. Further those who may have limited in the fields in which they are trying to gain knowledge. Perhaps a better sampling at more detailed levels is necessary. For example how many PhDs in what fields, and when it was obtained. I suspect that many spend some time getting some knowledge but often drop out because of time spent and also because of poor teaching techniques.

Second, if improving online and on-campus learning is a priority, then “the flow of pedagogical innovations needs to be formalized,” Chuang says. For example, many of the MOOCs in the study used innovations from their campus counterparts, like physics assessments from MIT and close-reading practices from Harvard’s classics courses....The real potential is in the fostering of feedback loops between the two realms,” Chuang says. “In particular, the high number of teacher participants signals great potential for impact beyond Harvard and MIT, especially if deliberate steps could be taken to share best practices...

 Indeed feedback would be useful. For example, a classic presentation is Lander's. Others could learn from video style. Others could also learn from presentation style. An interesting analysis would be a critique of the instructors and then cross tabbing the instructor with the demographics of the ones doing the critique. There is a great deal of improvement and learning in style of presentation. The instructor who just "presents" as the always do and then assumes that a video of it is sufficient is truly making a mistake. The Lander presentation is wonderful because on the one hand he acknowledges and works with the students in class and also acknowledges the person online, albeit displaced in space and time. In many ways the MOOCs must learn the way early television did, there must be producers and directors, and style is as important as content.

Also the feedback is critical. Now, frankly, there is none. Discussion groups may be there but the TA or equivalent is monitoring at best. Focus Groups would be essential, online focus groups with the instructors would be a substantial benefit. This is especially true when one has such a diverse group of participants.

Third, advancing research through MOOCs may require a more nuanced definition of audience. Much of the research to date has done little to differentiate among the diverse participants in these free, self-paced learning environments.

 Marketing was always a part of television. They needed advertisers and the advertisers really wanted to know what the audience watched and wanted. Perhaps that element is necessary. Market segmentation has always been a part of the research and that segmentation should be part of what the MOOCs do. If all else fails listen to the customer!

More on MOOCs

As I do from time to time I examine new MOOCs as they come out. A recent one was of great interest but I was sorely disappointed. Here are the reasons:

1. The Video was Chaos. The Instructor had the habit of walking from left to right and then back again, while writing on a blackboard. The video tracked him and his one was seeing a constant flash of the marks on the board while watching the movement and attempting to understand what he said.

2. There is no text and there are no notes. A student asked why and in a slightly surly manner a TA responded that they felt you learned better by taking notes. Perhaps but also perhaps someone would tell the video person that the note should be visible and not just to follow the Brownian motion of the Instructor. I believe that having some form of content on line would help. Especially for international students. But there seems to be a bit of arrogance that is less than helpful.

3. Chaotic Content: This has to deal with DNA analysis using various tool kits available to the bench biologist. Now the result is akin to using a CSHL Lab book, following the protocol, and hopefully filling in the gaps.

4. Quizzical Class: The Instructor would ask a question of the MIT students and apparently there were few if any that even tried to answer. This feedback should have an affect on the Instructor. It did not.

5. Why? Here is where the engineer separates from the scientist. We engineers ask why, bench people ask what and how. Why does a certain protocol work, and why one is better than another. Such a discussion is missing.

6. Lander and his exposition is still a sine qua non. Lander is like listening to a symphony, like listening to Shakespeare, and this was like listening to some reality TV program. Why? The Instructor is quite well known, he has even written the book that the on campus folks use. Why the disconnect? Good question. Perhaps there should be some feedback.

Remember, if all else fails, listen to the customer.

MOOCs and Results

In a recent article in Science there was a discussion about obtaining performance data from MOOCs. Two things struck me:

1. The fact that it seems that there was limited if any thought as to how well they MOOC performed. One suspects that performance measures would have been at the heart of the effort. Namely is all this money worth it, and worth what? Apparently not.

2. Also the article in my opinion seems to ramble almost everywhere except in articulating any semblance of content or context. One wonders what the purpose of all those words were.

For example: 

Using engagement data rather than waiting for learning data, using data from individual courses rather than waiting for shared data, and using simple plug-in experiments versus more complex design research are all sensible design decisions for a young field. Advancing the field, however, will require that researchers tackle obstacles elided by early studies.These challenges cannot be addressed solely by individual researchers. Improving MOOC research will require collective action from universities, funding agencies, journal editors, conference organizers, and course developers. At many universities that produce MOOCs, there are more faculty eager to teach courses than there are resources to support course production. Universities should prioritize courses that will be designed from the outset to address fundamental questions about teaching and learning in a field. Journal editors and conference organizers should prioritize publication of work conducted jointly across institutions, examining learning outcomes rather than engagement outcomes, and favoring design research and experimental designs over post hoc analyses. Funding agencies should share these priorities, while supporting initiatives—such as new technologies and policies for data sharing—that have potential to transform open science in education and beyond. 

 OK, now try to parse this one. First what is engagement data? Second what is learning data? I have now played around with over a dozen MOOCs. Some are good, most are horrible. My last attempt was a Materials Science course at MIT. The lectures were spent watching the instructor write the lecture notes, which we had, on a large chalk board in total silence except for the clicking of the chalk. So why? Second the tests were really tests in reading comprehension, did you use the right units, and did you copy the value properly. Any errors were errors of transcription and not comprehension. How is that measured?

Frankly it seems that MOOC management has been stumbling around in some directionless manner. If there is no way to determine what the "return" on the investment is then why invest.

Now the biggest problem I have with MOOCs is the discussion functions. In my experience I saw anonymous  discussants who were a few steps from shall we say rather anti-social actions. The article states: 

The most common MOOC experimental interventions have been domain-independent “plug-in” experiments. In one study, students earned virtual “badges” for active participation in a discussion forum . Students randomly received different badge display conditions, some of which caused more forum activity than others. 

My experience, and anyone seeking proof need look no further than any anonymous discussion on the web, is that it facilitates the worst and possible near real anti-social behavior. Why would anyone want to participate. I tried once to make an observation and some, in my opinion, socially in-adept person made comments that had me remove my remark. The remark from another made one shudder! Then there is the near diabolical "peer review" method of having people who know nothing from disparate cultures attack others work. One wonders who invented that scheme!

Thus one may wonder if the writer of the piece has had any hands on experience. Apparently from what is written the answer is they have not. Yet the ability to measure effectiveness is critical. When will someone credibly address that issue?

MOOCs Again

There is a piece in the New Yorker on MOOCs. The author states:

The other major problem is that MOOCs tend to be set up in a way that minimizes frustration for students (who might drop out at any moment). There often aren’t pop quizzes or the kinds of challenges that can alienate students in traditional settings. The problem here is that easy learning does not make good learning. In fact, the very tools that we believe make for better education may also make students more likely to quit. More frequent testing, for instance, can improve memory, learning, and retention. And, sometimes, the best test of all is the test that you fail: recent work from the cognitive psychologist Elizabeth Ligon Bjork has shown that pre-testing on never-before-seen materials helps students perform better in a subsequent course covering that material. In general, Bjork has found, speed bumps in learning are good—desirable difficulties, she calls them. MOOCs would likely be more effective if they didn’t shy away from challenging students, rather than presenting a fluid experience which gives the false impression of the learning and retention.

 Frankly the problem with MOOCs is that those who do so in English do not know English. For example in a recent MIT exam for a materials course they asked a question which required height and width. Except they stated:

a width, b, of 20μm and a depth, h, of 1μm

Now what is depth? One was given the length and allegedly the width but one needed height. Is depth a new word for height? Words mean something.  Now I am facile in six languages, so I tried translating into each but came to the same conclusion, there must be some language wherein depth equals height. I think.

This is the problem of MOOCs, students and teachers who don't read what the wrote, at least in the language it is taught!

As to the above suggestion, my response is clap trap! The MOOCs maximize frustrations. Again in the same course one must enter equations in a format that is hidden in the bowels of the course. One spends hours finding out how to do the computer side of the course, NOT the material taught.

In addition the approach to grading is an all or none approach. One must go through all the equations, convert them to a program, run the numbers and then make certain the the units are consistent. To complicate things the units in the problems are disparate to say the least.

Finally the New Yorker shows Lander at MIT, strangely the only existing MOOC instructor who has a course that works. One wonders if the New Yorker writer had the slightest clue!I suspect as a psychology major she does not!

Then there is the Harvard Medical School course on Anatomy. Apparently the TAs has disappeared. The answers are wrong and the students in the discussion group are running wild. The overhead on a good MOOC is significant, and costly. Lander has figured it out, as he has with everything he touches. Almost all the others have not. Some seem clueless.

UPDATE: Well I just found out that depth=height. Amazing what people will come up with.

MOOCs: There Has to Be a Better Way

I am again examining the MOOCs and their approach to education. One recent course is an MIT course on what used to be called Strength of Materials. Some observations:

1. The instructor spends well more than half the time writing her note on the board without any real additional insight. The board should be a stage prop, not a means to re-write the notes. If you have them thus why re-write them?

2. The exam questions are interesting. Let me explain.

a. Some fifty plus years ago, with paper, pencil and slide rule, students attacked the same issues. Most likely it was done that way for a few centuries. One received an exam question on paper and then the student used what was called Engineering paper, a grid like sheet of paper, upon which you prepared the solution. You restated the givens, you then drew whatever diagrams you needed, then you proceeded through the equations in the analysis, then you inserted the numbers, used your slide rule, underlined the answers and so on. You interacted with the problem so as to get to know and understand it in a linear and proximate fashion. The pencil, paper and slide rule became an integrated whole.

b. In today's world the approach is dramatically different. The computer gives the problem on the screen, you must somehow capture that and convey it to whatever mode of processing you choose. I first started out with paper but bad mistake. It appears that using Excel spread sheets as a substitute for your Engineering paper works better. But beware, you must copy exactly. For example a 1.59 may appear as a 1.50 or even get copied that way.  Then the answer is inserted into the same screen from which you got the information. Watch as you units and what I have found essential is to copy from Excel into the computer screen. The problem is that the computer never understands your level of thought and your only feedback is the right or wrong mark. There is no way to understand a mistake, say a sign, and remedy it. The exam measure more the ability to copy in correctly and copy out correctly and to be correct in one's units! It does not measure understanding and it does not facilitate understanding.

Thus the change from half a century ago to today actually may defeat the purpose of education.

3. The Discussion groups are oftentimes chaotic. It is a mass of often divergent cultural approaches to learning, or worse, the participants may be actually assuming they are getting an MIT education. In reality this may be a move to checking reading skills and short term memory. Any mild dyslexia could and does cause havoc, one always writes the wrong number, close but not exact. Thus one spends time checking the transcriptions. One is not learning the material but fine tuning the process. The participants do not seem to understand that, unless they come from a world in which that is the way they are educated, which many do.

MOOC Philosophy?

In a recent New York Review of Books piece the author writes about the film Ivory Tower, a work commenting on contemporary higher education. At one point he makes the observation:

At the heart of the MOOC model is the idea that education is a mediated but unsocial activity. This is as strange as the idea—shared by ecstatic communities of faith—that the discovery of truth is a social but unmediated activity.

 This is a truly powerful observation. It is McLuhan in insight. It is the essence of the "peer review" process.

Do Instructor have value, besides the "stars". My classic example is Lander and Biology, he has no equal. But what of those others whose influence made us what we are today. I recall two, both in my secondary school, both sort of Mr Chips, at two extremes. Brother Edward set my mathematics career aflight, and Mr Brown made me think. Mr Brown made us work in Middle English, do Chaucer, see the future through past. Brother Edward taught how mathematics really was, discovering cause and effect.

Would a Lander on video have replaced them? No. They were essential. Without Mr Brown taking us for drinks before my final French exam, I would never have become fluent in French, and as for Brother Edward, he created the ability to "see" the answer.

The group, the mass, the consensus of opinion, it is useless, it is the insight of the individual into something new. The person willing to use their intellect to see the world in a different manner is the person who creates a difference. Truth is all too often destroyed by the social and unmediated activity. The individual is destroyed, the individual is silenced by the opinions of those without knowledge. Worse those without knowledge are empowered to apply their ignorance as truth. That is the evil in "peer reviews" by those without knowledge. Just a thought for the end of July.

MOOC Style Preference

MIT just posted a note as to what "styles" were working best for EdX. They argue:

In a paper published this spring, the CSAIL team outlined some key findings on what online learners want from videos. These include:

• Brevity (viewers generally tune out after six minutes)
• Informality, with professors seated at a desk, not standing behind a podium
• Lively visuals rather than static PowerPoint slides
• Fast talkers (professors seen as the most engaging spoke at 254 words per minute)
• More pauses, so viewers can soak in complex diagrams
• Web-friendly lessons (existing videos broken into shorter chunks are less effective than ones crafted for online audiences)

Now this should be really segmented by the market demographics. Any credible market research effort segments the demographics by enough dimensions to make it work and then seeks what is most desirable for each segment of the demographic. Usually a sophisticated task. Yet this report just assumed that everyone was the same. The results are generally interesting but a good demographic and psychographic set of dimensions would make it useful.

In my experience, I will leave my demographic and psychographic to the reader, I want:

1. A short lecture and six minutes is good.

2. A Lander approach with the Prof in front of a class, makes you feel it is real and you are pert of the process. Also it paces the instructor. It appears that Lander was the only one I used that did this.

3. I really liked the white board approach. I had the notes and could then annotate them as the lecture proceeded.

4. The rate at which a person speaks is not as critical and the rate which the transfer information. Again Lander is sine qua non, he had one to three points to get across and he did it simply.

5. Pauses, if the lecture is in front of a class they take control.

6. "Web friendly", again Lander was spot on, you did something with genes, you learned by your mistakes, my dyslexia came to the fore so I had to do a work around. This did Python program to read and translate the gene structures.

It would really be worthwhile to do real market research on these courses and not just a computer science approach. Yet that means you have to know your "students".

Peer Review, MOOCs and Total Chaos

I am not a fan of peer review in MOOCs since it makes no sense. There are no peers as would say be the case in submitting to Science or Nature. It is assumed the Editor in each case selects a true peer who has their own track record. Yet in a MOOC setting one gets a total and random collections of people who have generally no experience or knowledge of any depth in the subject. In addition, the cultural difference results in chaos.

But the real issue is what the so called peers see as Plagiarism. Here they walk a very fine line. It appears that the "peers" can assign a paper as plagiarized with no basis for saying so. There is no due process, there is no meeting your accuser. But there is a basis for defamation. Why the companies who allow this have not considered the legal consequences I cannot fathom. You have students from cultures where it is fine to defame others for their own achievements and practicing that randomly online and assuming there will be no consequences. That is unrealistic.

Plagiarism is a bit difficult to prove, unless one finds the source and then demonstrates the exact extraction without referring to the source. We try somewhat diligently herein to be certain at all times to delineate fair use of segments with a reference and where we seek comments. That generally is acceptable. On the other hand accusing someone of plagiarism without basis is de facto defamation.

Why MOOCs allow this practice is beyond me. It may come back to haunt them.In fact it is my opinion that it facilitates a potentially antisocial trend in a class of individuals and the result may be catastrophic.

MOOCs have a long way to go before they work. A small few, again I use Lander's course as the sine qua non example, but there have been frankly none since then. The potential is there but the anonymous discussions create cadres of the strangest types. They may be well worth a study.