You rolled out Power BI. Why the reports aren't better

Power BI has been in the house for a year. The workspace holds sixty reports, three department heads build their own, and IT has set up the gateway refresh properly. On Tuesday the management team meets, and the controller brings an Excel extract because revenue in the dashboard is two per cent off the P&L and nobody could say why. The CFO asks the question that comes up in many companies after the first year: we rolled out Power BI – why aren't the reports better?

Power BI is there, the impact is not

The situation is more common than vendor success stories suggest. Cindi Howson measured what "success" actually means: only 24 % of companies surveyed call their BI "very successful", and only 34 % see a significant impact on the business. Her conclusion is sober – it is possible to build a perfectly architected BI solution that has little effect on the business (Howson 2014, pp. 76–79). Usage fits the picture: on average only a quarter of employees have access, and successful companies have noticeably more than failed ones (Howson 2014, pp. 88–89).

How do you notice it in your own company? Through Excel. As long as department heads keep their analysis in Excel, Power BI is not the reporting system but a second truth beside it. Howson describes this frustration as the typical state before the turning point: executive meetings start with a debate about how the numbers are compiled and whose are correct, rather than with a discussion of what they mean (Howson 2014, p. 111). She counts it among the LOFT effect – Luck, Opportunity, Frustration, Threat – the catalysts that can turn a mediocre BI rollout into a success, if someone uses them (Howson 2014, ch. 5).

The cost: sixty reports that are maintained and touched every time the model changes; a controlling team that reconciles two reporting worlds instead of one; a management meeting that spends a third of its time on which figure applies. And the most expensive item: a management team that does not trust the report and keeps deciding by gut. The licence is the smallest line.

The most common findings from project reviews

In our reviews of Power BI environments the same findings recur. There are about ten, in three groups: purpose, model, reconciliation. None of them lives in the tool.

Purpose. The first finding is the most frequent: there is no decision behind the report. It was built because the data was there or someone asked for it – not because a person makes a choice on a given day and needs this figure to make it. The purpose question – what are these numbers meant to say, and what are they for? – goes unanswered for most reports. The second finding follows: nothing is ever retired. Reports get added, none leave, and after a year there are sixty. The third is the overview page with forty tiles that wants to be dashboard, report and work list at once and is none of them – see glasses, page, hammer earlier in this series.

Model. This is where the expensive findings sit. The classic is the rebuild: the existing Excel logic is transferred one-to-one into Power BI, helper column by helper column, until the dashboard looks like the old spreadsheet. In a BI rollout case study, Caviezel wrote the sentence we quote in every review: projects that map existing solutions and processes into a new technology are doomed to fail (Keimer/Egle 2020, p. 119). Rebuild Excel in Power BI and you get Excel with a refresh button – and lose the one advantage Excel had: everyone knew where the number came from. The next findings hang on that one. Measures without a definition: "Revenue" exists as a DAX expression, but nowhere does it say gross or net, with or without intercompany, by invoice date or delivery date. A separate model per report: three department heads each built a file with its own revenue table, and the three values differ because each applied a different filter. And self-service without a guardian: the business units got data access and a tool, but nobody looks after the data model. The TDWI working group puts it plainly: self-service is not no-service; a new tool and data access alone are rarely enough, because without external impulses users fall back on their trusted Excel solutions (TDWI 2022, p. 33). A BI team, internal or external, remains necessary – as a service provider and as guardian of the data model (TDWI 2022, p. 43).

Reconciliation. This group brings the Excel extract into the meeting. Reconciliation to the general ledger is missing: no report demonstrates that its revenue total agrees with the P&L, so the management team does not believe it. Our rule is old and unchanged: reconcilability before beauty. Next, the refresh on the wrong day: the dataset refreshes at two in the morning, but accruals are posted on working day three, and on meeting day the report shows a state from before the close. The report hangs on the close, not on the calendar – that belongs in the close schedule, not in the refresh settings. And finally the missing notation concept: every report has its own colours, its own order of actual, budget and prior year, its own way of showing a variance. What two pages of rules would have settled costs explanation time in every meeting.

Fictional but typical distribution: each report is assigned to its most serious finding. More than a third fails the purpose question before anyone talks about technology. Only two of sixty reports pass the review without a finding.

What a project review does in a few days

A project review takes a few days and starts at the management team's table, not in the tool. Four steps, in this order.

The test bench places every report in the three circles described by Ossola-Haring, Schlageter and Schöning: supply (what is technically available), demand (what someone wants) and need (what a decision requires). Only the intersection of all three is relevant for steering; the rest are zombie reports, wish lists and gaps (Ossola-Haring et al. 2019, pp. 94 f.). Per report we ask: which decision depends on it, who makes it, how often, and how does that person notice things are going wrong?

The test bench produces the retirement list. It causes the most resistance in the review and the most relief afterwards. Reports without a decision go; reports with a decision but without an owner get one; duplicate reports are merged. Sixty typically become fifteen to twenty.

The model check examines the reports that stay for their foundation: how many data models are there, how many of them calculate the same measure, which measures have a definition, and does the total agree with the general ledger? The goal is one semantic model per steering area, in which every measure exists exactly once – with description, source and reconciliation evidence. It helps when the reviewer can read both the P&L and the DAX. A controller who understands IT finds the cause of the two per cent variance in an hour; a controller and a developer translating for each other need a week.

The business case finally sets out what the remaining reports cost and what they carry: maintenance and reconciliation effort, meeting time, and against that the decisions made faster or better because of them. On that basis the management team decides whether to keep investing in Power BI – and where.

Eight findings from Power BI reviews: symptom, cause, action
Finding	Symptom	Cause	Action
No decision behind it	Report is opened, nothing follows	Built because data existed or someone wanted it	Ask the purpose question; without a decision, retire or move to appendix
Excel logic rebuilt	Dashboard looks like the old sheet, helper columns in DAX	Existing solution mapped into new technology	Derive afresh from the decision, don't translate sheet by sheet
Measures without definition	"Revenue" in three reports, three values	No measure profile, no description in the model	Profile per measure: definition, source, cadence, owner; description in the measure
Own model per report	Every PBIX file brings its own revenue table	Self-service without a shared model	One semantic model per steering area; reports connect to it
Self-service without guardian	Business units build, nobody checks	Tool and access distributed, role left vacant	Name a model owner; review gate before publishing
No reconciliation to GL	Management doesn't trust the report, Excel extract in the meeting	No reconciliation evidence to the close	Reconciliation page per model: report total = P&L total
Refresh on the wrong day	Report shows state before accruals	Refresh by calendar, not by close	Refresh as a step in the close schedule, after the last posting
No notation concept	Every report different colours and orders	Rules never written down	Two-page notation concept following IBCS®; theme file

Eight of the ten most common findings with symptom, cause and the action recorded in the review. None of the actions is a feature in the tool.

The reports that stay are checked against a short checklist we have used for years, following IBCS®:

Purpose question answered: which decision, which person, which cadence?
Variance instead of value: target as baseline, actual as signed variance, threshold ±5 %
Fixed comparison logic: prior year – actual – budget, time runs from left to right
One page per unit, same layout, same colours, same order
Reconciliation evidence to the close on every page that shows financial figures

This is what the revenue page looks like after the review: one model, one definition, prior year – actual – budget in fixed order, variance as bars on the right. Sites B and D fall below the ±5 % threshold and get a comment; the others do not. The total agrees with the P&L, and the page says so.

The tool was never the problem

Anyone who finds the reports no better after a year of Power BI usually looks for the cause in the tool: wrong licence, too little training, the next consultant. In reviews we almost never find it there. The reports are not better because they start where they always did – with the data and the wish – not with the decision. The framework from decision to KPI reverses the order: decision → question → KPI → data → report → impact → review. Power BI comes fifth, and there it is a good tool. First, it achieves nothing.

So the rollout was not wrong, only incomplete. What is missing is the work before the report – and that can be done afterwards without installing anything new.

From practice

In a project review at a technical group with five sites, Power BI had been in use for a good year, and the site managers still kept their monthly figures in Excel. The reason was on the table quickly: there were three data models for the same revenue figure – one from the ERP export, one from the project system, one from a sales Excel list – and in the group meeting three different values sat side by side, none of them reconciled to the P&L. I first recorded with group management the decisions they make each month, then checked each of the roughly fifty reports against that list; by day two the retirement list was longer than the keep list. Then we merged the three models into one, with a single revenue definition, a reconciliation page to the general ledger and the refresh as the last step in the close schedule. The DAX error causing the variance to the close was a date filter on delivery date instead of invoice date – found in an hour, because I could read both. The Excel lists were never banned; two months later they simply stopped being opened.

Keimer and Egle describe digital controlling as the interplay of five domains – data, technology, processes, methods, competencies – with one condition first: all five must reach a comparable maturity; technology alone, without data and competencies, achieves nothing (Keimer/Egle 2020, p. 7). The Power BI environment after a year is usually exactly that picture: technology at level four, data and methods at level one. The TDWI working group answers the question of what makes a good dashboard in three words: that it gets used (TDWI 2022, p. 7). That is our measure in the review too.

What you can do tomorrow

Open the usage metrics of your Power BI workspace and sort the reports by views over the last ninety days. Whatever was opened less than once a month is the first draft of the retirement list. Then check whether the revenue total of the five most-used reports agrees with the last close. If it doesn't, you know why Excel keeps running.

If you would like an outside view: our project review for existing Power BI environments takes a few days, comes at a fixed price and ends with a written second opinion – test bench, retirement list, model check and business case, with a clear recommendation on which reports stay, which go and how many models you need.

Sources

Howson, C. (2014): Successful Business Intelligence. Unlock the Value of BI & Big Data. 2nd edition, McGraw-Hill.
Keimer, I., Egle, U. (eds.) (2020): Die Digitalisierung der Controlling-Funktion. Anwendungsbeispiele aus Theorie und Praxis. Springer Gabler – including Caviezel: BI rollout (ch. 7).
TDWI e.V. (ed.) (2022): Self-Service Analytics. Best of TDWI Themenzirkel Self-Service & Analytics, e-book 2022.04. SIGS DATACOM, Troisdorf.
Ossola-Haring, C., Schlageter, A., Schöning, S. (2019): 11 Irrtümer über Kennzahlen. Mit den richtigen Erkenntnissen führen. Springer Gabler.