Ecosystem View: One way to think about data

(A guest post from Andrew Brice. Data analytics and visualisation is Andrew's passion, turning data into insights. Enjoy some fascinating insights here: http://www.zyan.co.nz/)

The calls were stacking up. They weren’t meant to be, this was just a normal call centre day. Call agents were racing, management hearts were racing, but no one was quite sure where the finish line was. Or why there was even a race.

Afterward, the vast swathes of data collected by the call system was able to be analysed and it started to become clear as to what had occurred and why. The needle in the haystack was found.

The thing is, it was all preventable. The needle could have been tracked if only they had been analysing their own data, ideally with an ecosystem perspective. If only.

OK, so how would that have worked?

Example dashboard for call flow monitoring

Two parts of the organisation already knew this was not a normal call centre day. It was an end-of-year filing date for a subset of clients (yes, it’s a Government agency). Not a familiar filing date, but a statutory one for these clients. The business knew and so did the call centre. Every year, on the same date, a spike occurred. But the call centre didn’t analyse call patterns at a deep enough granularity so they hadn’t picked this up. They were busy with “real” call volumes… The data was collected but it was never sufficiently deployed through to analytics. Of course, you kind of hope the business might have reminded the call centre. Sigh.

As the calls arrived, the call notes were not being augmented with robust, pre-determined, keywords which would have enabled rapid and clear analysis of that entered text on a near-real-time basis. The same effect could be achieved by simply asking call agents what all the calls were about, but (to be fair), the call agents were rather busy.

That keyword data might have shown that the callers were all identifying as “Directors” and most were asking (in quite elongated conversations), about how to log-in to one of the organisation’s systems. Which, just about then, went splat. That’s a technical IT term which means it stopped working for no particular reason. Now the keyword flow changed from password issues to “the system won’t work” issues.

Luckily, the super-heroes in IT noticed a red-flashing light (I jest) and rebooted the server. The system returned. However, the rest of the morning it kept splatting, being rebooted, … You get the idea.

The data that IT had could have painted a pretty clear picture that this was going to happen. It’s just that they didn’t really care too much about that one little server, they had lots of other more important ones. But their data (for the naughty server) showed annual usage growing at around 4% a year over the last few years and that, at its peak (which oddly only seemed to occur once a year), last year was at 88%. Do the maths, the poor server was over-run. But each month, IT drew beautiful charts showing all the organisations’ servers were paragons of good health. Sometimes, they even shared these charts with the business (not the call centre though).

The final piece of the jigsaw was understanding why logging-in was proving such a problem for this subset of clients and this particular service. Connecting call data to client CRM data quickly showed that this was almost the only interaction these clients had with the organisation each year. The call records then showed that “remembering their password” or “knowing what data to enter” were the key tags to describe the clients’ issues. Eminently common issues and ones that can readily be addressed with better user interface design, education of agents and users, reminder communications, and perhaps adopting external user-validation processes. Of course, the next crisis arrived so these sorts of improvements never actually happened.

So, what lessons do we learn from this? It might be useful to use a simple graphic to show one way of contemplating how to think about getting good data that enables quality analytics.

Data has different owners who, traditionally, impose their own standards, definitions, segmentation, and so on. Adopting co-ordinated governance practices (just like for finance or risk) is a massive enabler of good data.

Data is moving from controlled source systems to all-over-the-place. And there’s so much more of it than there ever used to be. Active and consistent curation is needed to enforce (in a friendly way) an organisation’s data governance model.

Data now lives in many places. Collecting that data together in coherent and consistent ways matters. It’s also quite difficult. This isn’t about data marts, it’s about descriptors and licences and metadata and data packaging. Building confidence in the quality of the data being collected.

Data never seems to arrive in perfect shape. It always needs consolidating or averaging or building into new fields or any one of a myriad number of ways that data gets augmented. It might need to be assembled with other data (IT data and call centre data for example) or shaped a certain way for a particular visual. But there are pretty consistent augmentation requirements and there are pretty consistent ways of standardising and automating such augmentation. Done right, we can join a call with a CRM entry and associate them with a system and then to a server. Now we can see much of the ecosystem. And we can keep rerunning the analysis as we make improvements to see if those improvements really are working.

We also have an endless need to categorise data, to tag data. This is a phone call, it’s a happy person on the line, it’s about this system, it is from that person. These are all ways of adding tags to data. Tags are incredibly useful because that’s how we can start joining things up, clumping them together, showing how processes flow, visualising pathways to outcomes. Tags are exciting. But if we all invent our own tags then value is lost. Governance and curation are how tags are kept moderated.

And, finally, we get to the fun bit. Actually, deploying the data in ways that enable rapid, effective, and (hopefully), elegant visuals that communicate the story inherent in the combined data and which enable audiences to quickly understand and react to complex ecosystems. It might also be deploying into AI systems to read call agents’ notes in real-time and then to tag calls automagically.

Deployment is the real value proposition. But only if you do something about what you’ve learned.

(Andrew Brice works with New Zealand government agencies on the visualisation of business ecosystems using complex, multi-faceted, data.)

Respecting the Police is incompatible with NOT punishing bad cops

BLM seems to bring out some false either/or arguments. Either you respect and support the police, Or you want bad cops punished. That’s not an either/or; that must be a both. It is not possible to respect and support the police if they accept and protect bad cops. Rejecting bad cops, and demanding that bad cops face criminal investigation and are tried in court just like any other person is a sign of respect for the police.

I was a member of the Wellington Civil Defence Police Unit for about a year, when it was formed and until it died of lack of interest (from the police mainly). The unit was set up with the objective of creating a cadre of semi-trained individuals who would be able to man (person?) cordons in the case of an earthquake or other emergency. We were sworn constables, and took an oath to server her majesty Queen Elizabeth the Second of New Zealand. We were issued warrant cards, and in the event of the declaration of a civil defence emergency, we assumed the full powers and authorities of a police constable. Really, the idea could best be summed up with an image: a constable can control about 40 meters of frontage of a collapsed or damaged buildings, but with one CD Police constable under his/her supervision, suddenly that distance doubles. Crowd control and traffic management are critical in an emergency, and having sworn (and trained) constables available is critical.

As a member of the unit, I would spend on shift per month with active police. One time this included spending a Sunday evening with a couple of constables in a police car. Two young men, probably 25 – 30 years old. Mostly it was a boring evening, except for two events.

The first was a call over the radio saying that a gang needed to be shut down at x location. Now I do not remember the radio traffic, but sure enough, lights on and noise, and we rushed across downtown Wellington, to arrive at a side road off Tinakori Ave. There were three other police cars there, blue lights on but the sound off. There were a number of Polynesia street people who were being, shall we say, manhandled; shoved against walls, held up by their shirt fronts with backs to the wall, police faces inches from theirs. 

My gut feeling was that there were a whole lot of civil rights being violated. I kept my mouth shut.

Once the situation settled down, and the gang types dispersed, we got back in the squad car and off we went for a cup of coffee. 

I asked the “plods” what just happened.

“Well”, they said, “there are a few rival gangs in town; some Samoan, some Maori, others Tongan. They do not get on, and while the most they have are sticks and maybe a cricket bat, they do try to beat the snot out of each other. Their favourite trick is to have one of their young ladies stand on a street corner where they know the Maori boys will pass, and entice them to follow her. Once she has inveigled the victims into the alley, the rest of the gang comes out and beat the crap of the poor victim.”

The police had received a report from a civilian that it looked like they were up to their old game, and the police came down on them like a ton of bricks, to use the expression.

The two police I was with explained this all to me, and while I nodded, I have to say that I was a bit perturbed by it all. This was a pretty blatant disregard for their rights, and grabbing someone by their shirt-front and banging them against a wall can hardly be considered “community policing”. I kept my mouth shut.

About an hour later, we received a call to attend a burglary.

Off we went, and when we arrived, the burglary, of course, was not in progress but had been reported by the occupants on their return to the city after a weekend away.

The two police blokes (they most certainly were blokes) inspected the scene, took notes, and then sat down with the owners, and took all the details they could. The owners had been away, and returned earlier in the evening. Things did not seem right at home, and the more they looked, the clearer it was that they had been burgled. 

These two blokes were the picture of respect and compassion, with a fair amount of professional courtesy included. The victims, a lesbian couple, were distraught, yet trusted these two blokes who were there, as official representatives of the state and police forces.

I was taken aback. Not two hours earlier I had watched them as participants in what appeared to be a violation of civil rights (in the pursuit of avoiding a vicious attack and breaking up the plot early), and now they were the very picture of the sympathetic arm of justice doing what they could for victims.

There were too many nuances in that one shift for me. One lesson was clear, however:

We ask Police to be both the protectors of law, de-escalators, sympathetic to victims, yet also the visible and powerful arm of society in halting or avoiding violence. My respect for the police increased immeasurably that evening. These two blokes, that’s what they were, were Power Projected to stop violence from happening, and there will the soft-spoken, almost consoling voice of authority promising that they would do what they could to reduce the suffering of those wronged, a couple who were the antithesis of what they as good Kiwi Blokes has been taught to respect. 

I can and do have the highest respect for the police. Their job is full of contradictions and is terribly difficult. Yet I also have no place for those who have learned that there is pleasure in exercising their power to oppress. The difficulty is in seeing when one turns into the other. The first situation I witnessed could have spiralled out of control, but it didn’t through the professionalism of the police. There was a situation of direct threat to the public, and the stomped on it. Civil rights were, um, put to one side for the moment, because the police knew the pattern that was playing out, and they said: “not tonight”.

So, I respect and support the police. Yet that does not in any way reduce my expectation, no, demand, that police who betray the trust and their authority must be stamped out, and must face justice. I’ve seen how it can and should work. I respect and trust the police. I demand criminals be removed from the police force. There is no contradiction.

An Amerikan Coup is now highly likely

It seems the gloves are off. 

Today General Mattis, former commander of the Marine Corps and former Secretary of Defence, declared President Trump a Nazi, and all-but called for a coup d'état in the United States of Amerika. 

If there is any lesson that dictators around the world learn, it is that you keep the military on-side, no matter what. If you mess with the interests of the military and the generals, then you risk being replaced, peacefully or otherwise. Time and again, coups find their genesis in infringement of or denigration of the military. Even in Fiji, the South Pacific Paradise, denigration of the military resulted in a coup in 1987.

There is no reason that the United States of Amerika should be any different. Well, okay, there are lots of reasons, the most obvious being 240 years of an apolitical military whose highest-ranking member is a civilian. Also, there has yet to be a documented successful military coup in the United States. 

But I will suggest that one is very possible now.

President Trump has acted with impunity for over three years, has insulted every institution and any individual who disagrees with him. Himself a draft-dodger (and a coward), he famously said that he prefers heroes who were not captured, and reviled the memory of that hero on his death. He publically insulted a Gold Star family (a family that has lost a son or daughter in the defence of the nation). Trump's policies in relation to allies and treaty obligations have been appalling. The betrayal of such an ally led to General Mattis' resignation as SECDEF.

In a time of national unrest that is a direct result of his calling publically for police to use more violence, he has placed the US military on the streets of Washington DC.

Trump has now alienated the US military. That is dangerous.

After first suggesting that he supports Trump's call for military intervention at the state level, Mark Esper, the current SECDEF, walks back his statements and says that he does not support the use of Amerikan troops on the ground inside the US. This after being recorded saying that Governors must “control the battlespace”. So Amerikan cities are now the “battlespace”. I can only guess that in the waning days of the Trump administration, even those that are on board for ideology are beginning to understand the personal cost that they will bear. Esper took the job as a logical stepping stone from industry to the top of government. Now he finds himself part of propping up a failed administration, at the same time trying to defend the US. 

General Mattis, former Secretary of Defence and retired Marine 4-star general, one of the most respected military leaders of the past decade, has finally fully broken with Trump and his administration. He has all-but called for a coup. Some of his comments include:

“When I joined the military, some 50 years ago, I swore an oath to support and defend the Constitution. Never did I dream that troops taking that same oath would be ordered under any circumstance to violate the Constitutional rights of their fellow citizens—much less to provide a bizarre photo op for the elected commander-in-chief, with military leadership standing alongside.”

The then invokes the struggles against Nazism and the invasion of Europe:

“Instructions given by the military departments to our troops before the Normandy invasion reminded soldiers that "The Nazi slogan for destroying us...was 'Divide and Conquer.' Our American answer is 'In Union there is Strength.'" We must summon that unity to surmount this crisis—confident that we are better than our politics.”  

And then caps that by saying in his next paragraph:

“Donald Trump is the first president in my lifetime who does not try to unite the American people—does not even pretend to try. Instead he tries to divide us. We are witnessing the consequences of three years of this deliberate effort.”

General Mattis has just called President Trump a Nazi, worthy of violent overthrow. Make no mistake, General Mattis knows exactly what he just said, and he has said it in such a way that he can never be accused of saying “overthrow the government” even though that is what he means. He has called on members of the armed forces to “Protect the Constitution” from a “Domestic Enemy”. 

General Mattis is retired and can say what he wants. He has remained largely silent since resigning/being fired for not agreeing with Trump's betrayal of allies in Northern Syria. While I did not and do not agree with the US intervention in Syria, when you make allies, you stand by them unless it becomes impossible; you do not discard them. You especially do not discard them to the direct benefit of your primary stated adversary.

General Mike Mullen, former Chairman of the Joint Chiefs of Staff also commented:

"It sickened me yesterday to see security personnel—including members of the National Guard—forcibly and violently clear a path through Lafayette Square to accommodate the president's visit outside St. John's Church. I have to date been reticent to speak out on issues surrounding President Trump's leadership, but we are at an inflection point, and the events of the past few weeks have made it impossible to remain silent."

"Whatever Trump's goal in conducting his visit, he laid bare his disdain for the rights of peaceful protest in this country, gave succor to the leaders of other countries who take comfort in our domestic strife, and risked further politicizing the men and women of our armed forces."

Yet his most damning statement comes later in his comments:

"Furthermore, I am deeply worried that as they execute their orders, the members of our military will be co-opted for political purposes."

These are the words of two of the most respected members of the US defence fraternity. They cannot speak without those that currently serve listening to them. They also can say what serving officers are unable to say, even if they think it. 

The next few days may well see the resignation of Trump as president. That will be better than seeing the assassination of Trump by one of his personal guards, and the resulting internal turmoil that will result. But Trump is finished. He has lost the Republicans who are not sycophants, he is losing the moderate Christians who at least saw him as a promoter of Christianity against pure secularism (though he will never lose the “Christians” who seek preachers who will use the Bible to find passages to justify their racism, homophobia and misogynies). Now he has alienated the elderly (let them die for the economy) who are natural conservative voters, and finally, he has alienated the military, a naturally conservative force within society. Republican Governors are turning on him. 

Soon I expect we see the Senate begin to turn, the die-hard Republicans who see that they are in real danger of losing their seats in November (those that are up for re-election this cycle).

If there is a coup, no one will see it. There will be no Muammar Gaddafi moment, no sudden arrests and the taking over of the news outlets. No general will stand in front of a microphone. But this will be a coup none the less. 

For once Trump might be right.

But this coming week will be pivotal. Trump may suffer a deadly heart attack, or he may resign. Or he may survive and push through. If Trump should suffer such a heart attack, you can bet that the official autopsy will demonstrate that it was from natural causes. 

If he survives this, the US will be weaker, and his only hope to retain power past January will be through massive voter fraud and suppression. If he survives.