When information on websites is released or published by companies, governments, and anyone or entity that feels a need to "manage" the message, what impact does this have on the ability of Generative-AI to deliver anything other than bland, composite responses? Can we trust Generative-AI (using ChatGPT as the example here) to provide anything more than the "starter pack" from which real investigation and consideration should begin? Or will the results be sugar-coated?
The news from Auckland was dispiriting; a gunman killed two people and wounded another six, before being shot by police. His name suggests that he is Māori (1), though all reporting has been careful not to make such a statement. A 24-year-old with domestic violence convictions, on home arrest but allowed to go out to his place of work, where he went on his rampage.
I have no idea if the shooter is an “Urban Māori”, an entire class of Māori who are disconnected from their hapū and iwi, and therefore are not full participants of Māori-dom, and not part of European New Zealand. This entire group of people are disconnected from their whakapapa and traditions, and do not share in the benefits that are allocated to iwi and through them to hapū and Whanau.
This got me thinking. There must be programmes to connect Urban Māori with their whakapapa, and ensure they are full participants, or have the opportunity to fully participate in their Māori heritage, not to mention able to share in Treaty programmes designed to redress historical wrongs.
This, I thought, would be a great use for Generative-AI and ChatGPT in particular. Could it jump-start my learning on this? I really wanted to know not only what is being done, what how we can know if it is successful. Are there metrics that we can see that will tell us which programmes are the most successful? Can we identify programmes with a low return for the invested resources (tax money in many cases)?
So, I set myself the combined task of attempting to learn a bit about how effective programmes are for Urban Māori, and to see how ChatGPT could help. It did not go as well as I hoped.
Probably the biggest issue is that much of the information that I was hoping to learn about is probably “locked up” in non-publicly available sources, such as government agencies, Māori entities, or private companies and their internal records. Where information is not easily available, I suspect that the Generative-AI engines may have difficulty providing detailed information.
ChatGPT tells me that there are eight ways and programmes designed to help Urban Māori connect with their whakapapa and iwi. These include:
- Urban Māori Authorities (UMAs): UMAs are organizations that provide services and support to urban Māori communities. They often have cultural advisors, language classes, and programs that focus on reconnecting urban Māori with their whakapapa and traditional practices.
- Te Kōhanga Reo: Te Kōhanga Reo is a Māori language immersion program primarily for preschool-aged children. It emphasizes Māori language and culture, including teaching whakapapa, stories, and traditions.
- Te Kura Kaupapa Māori: These are Māori immersion schools that provide education in te reo Māori (Māori language) and incorporate Māori culture and history into their curriculum, helping urban Māori students learn about their whakapapa in an educational setting.
- Whānau Ora: Whānau Ora is a holistic approach to supporting Māori families' wellbeing. It provides access to a range of services, including those that help urban Māori connect with their whakapapa and cultural identity.
- Marae-Based Programs: Some urban areas have marae (traditional Māori meeting grounds) or community centers that offer programs, workshops, and events to teach urban Māori about their whakapapa and cultural heritage.
- Iwi and Hapū Outreach: Many iwi (tribes) and hapū (sub-tribes) have outreach programs specifically targeted at urban Māori. These initiatives may include gatherings, workshops, and online resources to connect urban Māori with their ancestral roots.
- Cultural Festivals and Events: Various cultural festivals and events are organized throughout New Zealand, providing opportunities for urban Māori to learn about their whakapapa, engage with Māori traditions, and participate in cultural practices.
- Genealogy and Whakapapa Workshops: Some organizations and community groups offer workshops and resources to help urban Māori research and explore their whakapapa.
I should have expected a long list of programmes, and certainly, no one or two programmes could possibly reach the large numbers of disassociated Māori. But a long list of programmes does not give me any confidence that these are effective programmes. Moreover, a list does not mean that the effective programmes are being identified and ineffective programmes are either terminated or modified.
Questioning ChatGPT further does not provide much confidence in these programmes. When asked about performance measures, virtually all the listed programmes came with “can be” measures, and no “is measured”.
Me: What metrics are they using to assess effectiveness of the programmes?
ChatGPT:
The metrics used to assess the effectiveness of programs aimed at connecting urban Māori with their whakapapa and cultural heritage can vary depending on the specific goals and objectives of each program. Some common metrics and evaluation methods include:
A paragraph of non-metrics was then provided for each of the eight, only aspirational statements.
Me: In addition to the qualitative, soft measures, what quantitative measures are used to assess the effectiveness of the programmes?
ChatGPT:
Quantitative measures used to assess the effectiveness of programs aimed at connecting urban Māori with their whakapapa and cultural heritage can provide more concrete and measurable data. Some quantitative metrics commonly employed include:
More non-metrics. My frustration was beginning to build.
Me: For almost all of your 10 items, the words "can be". Are there any examples of hard measures, not "can be"?
ChatGPT:
You are correct, and I apologize for the ambiguity in my response. The use of "can be" indicates that these metrics are potential options and may be used in various programs, but specific examples of hard, quantifiable measures used to assess the effectiveness of programs for connecting urban Māori with their whakapapa and cultural heritage can include:
- Language Proficiency Test Scores: Participants' scores on standardized language proficiency tests before and after the language immersion program can provide concrete data on language learning progress.
- Knowledge Assessment Scores: Quantitative scores from tests or quizzes designed to assess participants' knowledge of Māori culture, history, and whakapapa can be used to measure learning outcomes.
- Attendance Records with Percentages: Tracking attendance and calculating the percentage of participant engagement can provide a clear quantitative measure of program involvement.
- Survey Ratings and Likert Scale Data: Surveys may include rating scales or Likert scale questions, where participants provide numerical responses, allowing for quantifiable data on satisfaction levels, perceived impact, etc.
- Retention Rates as Percentages: Calculating the percentage of participants who continue to be involved in the program over time provides a straightforward quantitative measure of retention.
- Language Usage Frequency Counts: For language revitalization programs, counting the number of times participants use the language outside of the program setting can offer quantitative insights into language integration.
- Community Outreach Metrics: Quantitative data on the number of people reached, the number of events conducted, etc., provides hard measures of the program's community outreach impact.
- Educational Attainment Rates: For programs that include educational components, quantifying the percentage of participants who achieve educational milestones can be a hard measure of success.
- Qualification Completion Rates: For programs offering specific qualifications or certifications, calculating the percentage of participants who successfully complete the requirements is a clear quantitative metric.
These hard measures offer objective data that can be analyzed and compared to assess the effectiveness of the programs more precisely. Combining these hard measures with qualitative data and participant feedback can provide a comprehensive evaluation of the program's outcomes and impact.
It is difficult to lose your temper with a Generative-AI model, so question construction takes some consideration,
Me: It appears that there are no hard, quantitative measures that you know of. Could this not contribute to potential misallocation or poor allocation of resources, where there are inadequate quantitative measures of success?
ChatGPT:
You are correct, and I apologize for any confusion in my previous responses. As of my last update in September 2021, I do not have specific examples of hard, quantitative measures used to assess the effectiveness of programs aimed at connecting urban Māori with their whakapapa and cultural heritage. This lack of concrete, quantifiable metrics could indeed contribute to potential challenges in resource allocation and evaluating the overall success of these programs. Without clear, measurable data, it may be difficult for policymakers, funders, and program administrators to determine the impact of these initiatives accurately.
Having well-defined quantitative measures is essential for several reasons:
- Resource Allocation: Clear quantitative metrics help decision-makers allocate resources more effectively by identifying which programs yield the best results and which areas require additional support.
- Evidence-Based Planning: Having hard data allows for evidence-based planning, ensuring that programs are designed based on proven successful approaches.
- Accountability and Evaluation: Quantitative measures facilitate objective evaluations, enabling stakeholders to assess the efficiency and impact of programs accurately.
- Continuous Improvement: Concrete metrics provide a basis for continuous improvement of programs. Identifying areas of strength and weakness allows for targeted adjustments and enhancements to achieve better outcomes.
- Transparency and Communication: Having measurable results enhances transparency in reporting to stakeholders, including the public, funders, and the community, fostering trust and confidence in the initiatives.
To address this issue, it is essential for policymakers, program organizers, and researchers to work collaboratively to develop and implement robust evaluation frameworks that incorporate both qualitative and quantitative measures. By establishing clear objectives, defining specific quantitative indicators, and collecting relevant data, they can gain better insights into the effectiveness of these programs.
As there may have been new developments beyond my knowledge cutoff date, it is possible that more recent initiatives have incorporated concrete quantitative measures to evaluate the effectiveness of programs for connecting urban Māori with their whakapapa and cultural heritage. I recommend consulting up-to-date sources and reports from government agencies, Māori organizations, and academic institutions to gain a comprehensive understanding of the current evaluation practices for such programs.
There is much that can be learned from Generative-AI, but care needs to be taken in the interrogation of the tools, but much also that is either outside the range of their models (private information, government information subject to freedom of information requests, or classified government, or private entity information). With too much information not available, it is dangerous to take a Generative-AI's results at face value.
Something else that I am learning from this experience is that Generative-AI, at least in the guise of ChatGPT, is very good at generic, low content – high volume responses to some specific questions. Careful construction of questions and follow-on questions is important to ‘fine-tune’ the engine’s focus. Apologetic-AI (“I’m sorry Dave, I can’t do that”, or the version I received today “I apologize for the ambiguity in my response” and “I apologize for any confusion in my previous responses”) also results in longer-winded conversations, too frequently padding out a nothing answer with platitudes and waffle.
My conclusion then from this exercise?
ChatGPT has given me little or no confidence that there are effective programmes to integrate Uban Māori, although there are plenty of programmes. It also gave me no confidence that serious measurements of quantitative metrics are being provided and monitored. Finally, while it provides a discussion of what would be required to ensure the most (more?) effective application of limited resources, even this is phrased in “nice to have” and not “it is being done” type of language.
Those defining, debating, and implementing policy in New Zealand would do well to at least look at what ChatGPT’s statement on quantitative measures:
This lack of concrete, quantifiable metrics could indeed contribute to potential challenges in resource allocation and evaluating the overall success of these programs. Without clear, measurable data, it may be difficult for policymakers, funders, and program administrators to determine the impact of these initiatives accurately.
(1) I asked ChatGPT
“Are Matu, Tangi, and Matua names commonly used within Māori names?”
ChatGPT’s response:
As of my last update in September 2021, "Matu," "Tangi," and "Matua" are words of Māori origin, but they are not commonly used as standalone names in contemporary Māori names. Instead, they have specific meanings and are often incorporated as part of longer names or used in combination with other elements.
- Matu: "Matu" is a Māori word that means "spring" or "source of water." While it can be used as part of a name, it is not as prevalent as other Māori words used in names.
- Tangi: "Tangi" is a more common term in Māori culture and refers to a traditional Māori funeral or mourning ceremony. It is not commonly used as a standalone name but may appear as part of a longer name or be used metaphorically to signify mourning or remembrance.
- Matua: "Matua" is a Māori word that means "parent" or "elder." It is commonly used to refer to a parent or older family member. As a name element, "Matua" is occasionally used to reflect respect for an elder or as a way of acknowledging family heritage.
In Māori culture, names often carry deep cultural and familial significance. Māori names commonly include elements that reflect tribal connections, whakapapa (genealogy), historical events, natural elements, or personal traits. These elements are often combined to create unique and meaningful names for individuals.
It's important to note that naming conventions and practices may vary among Māori communities and individuals. If you encounter a name that includes "Matu," "Tangi," or "Matua," it would be best to consider its context and consult with individuals who have knowledge of Māori culture and language to better understand its significance.
