(2022-02-23) Chin Ill-structured Domains Aren't Necessarily Wicked

Cedric Chin: Ill-Structured Domains Aren't Necessarily Wicked. Two weeks ago we discussed Cognitive Flexibility Theory (CFT), a theory of adaptive expertise in ill-structured domains.

We talked about how the answers to that question gives us pointers to better note taking, explains how learning from history can be instrumentally useful, and explicates how experts in ill-structured domains reason by analogy (as opposed to first principles).

This week, we’ll take a look at some of the caveats.

Concepts ARE More Important in Well-Structured Domains

CFT makes all sorts of interesting, unorthodox recommendations for learning in ill-structured real-world domains, but don’t you go applying its ideas to math, or computer programming or, hell, the study of French.

The formal definition for ’ill-structured domain’ is “concept instantiation is highly variable for cases of the same nominal type” — in other words, your domain may have concepts like ‘competitive advantage’ or ‘returns to scale’ or ‘heart attack’, but the way these concepts show up is hugely dependent on the context.

As a result of all this variation, CFT reminds us that you often can’t generalise from case studies. (case study)

it argues that you should study cases in all its variegated glory without abstraction, and that you should hold a cluster of cases in your head so that you may reason by comparison to fragments of the cases you’ve already seen.

Much has been written about Elon Musk’s love for first principles — to the point where self-help sites proclaim first principle thinking to be the secret to Musk’s effectiveness and success. But the majority of concrete examples I’ve seen where Musk breaks something down to first principles and succeeds has often been in the domain of some more well-structured thing — hardcore rocket engineering, say, or upending decades of industry dogma by calculating the ‘actual’ bill of materials for the manufacture of a starship.

I think it is useful to think of structuredness as a spectrum

business would end up on the ill-structured end. Language learning lies somewhere in the middle

Ill-Structured Domains Aren’t Necessarily Wicked

In his 2001 book Educating Intuition, psychologist Robin Hogarth introduced the notion of ‘kind’ vs ‘wicked’ learning environments — that is, the idea that certain environments could lead to effective expert intuition.

(So the key point for me is that a domain might be wicked, but you might be able to craft a learning process that is kind.)

The ‘kind’ vs ‘wicked’ dichotomy was an idea that was reused by Gary Klein and Daniel Kahneman in their landmark Conditions for Intuitive Expertise paper, and it was again popularised in David Epstein’s popsci book Range.

Hogarth was making an argument about the relationship between kinds of learning structures and the development of intuitive expertise: intuitions that arose in wicked learning structures should be mistrusted, whereas expert intuitions that were built within kind learning structures may be regarded as valid.

kind learning structures and wicked learning structures are orthogonal to the kindness or wickedness of an operating domain.

Medicine may well be a kind learning domain (in that there are valid causal and statistical cues for disease; albeit with huge amounts of variability) — but you may find yourself in a wicked learning environment as a practicing doctor. (e.g. emergency room is wicked environment - see OP for reasons)

• see Medical Learning
• see (2016-03-08) Caulfield Trump University's Online Materials Are A Lot Better Than Your University's Online Materials

How do you identify a domain as kind or wicked?

he proposes the following series of questions when assessing your own intuitions:

Compare your learning structure with known examples

How many times have you made this kind of judgment

What kind of feedback have you received

Has this been clear, that is, unambiguous? A criterion such as relative success in a job, for instance, could be quite vague.

How immediate was the feedback?

Was feedback compromised in any way? For example, could it have been biased

What role has luck played?

Is this important?

These questions may be inverted to produce recommendations for finding or creating kind learning environments for yourself.

Structure your environment so you may receive clear feedback. That is: ensure that you have a unambiguous criterion for evaluating if your action was good or bad.

• How is this possible in Product Management?
  • Thinking in Bets may be key, but failure-rate is so high that it's hard to distinguish "right action that just didn't work" from "wrong action".
  • see Enabling environment and other (2020-02-20) Matuschak On Primer
  • PM is so immature/wicked that it's hard to take someone's expert opinion as gospel (unlike, say, being a medical resident). But honest dialog is still useful: Pair-Product-Managing

Recall that CFT is a learning theory that emerged from work done on accelerating medical learning/expertise.

here’s my point: ill-structuredness is a separate property from the kindness or wickedness of a domain

ill-structured domain but within a kinder learning environment

CFT gives you a set of guidelines for learning under conditions of ill-structuredness, but it’s still up to you to seek out kind learning environments, or to create those kind learning environments for yourself.