UK government trial of Microsoft's M365 Copilot finds no clear productivity boost
M365 Copilot fails to up productivity in UK government trial
cross-posted from: https://programming.dev/post/36865760
"speeding up some tasks yet making others slower due to lower quality outputs"
So use it for the tasks that were made more efficient, and stop using it for the ones that slowed down or were low quality.
Pretty sure its main function is to back up your data to cloud fully accessible by microsloth
Ugh, thought this could've referred to a Trial as in "All rise for the judge", not Trial as in "Your free trial has expired".
We're way overdue to put AIs on former trials.
I don’t see where a government would need a chatbot. Anyways, chances are that half the staff was already using some form of LLM before this trial.
Why wouldn't they want one? If it's a tool their employees want, they should provide it.
Yeah, no shit. But they nearly doubled the price. I canceled my membership, but I doubt enough did to actually matter.
I was fine paying $60 a year for Office. I was never gonna use the AI stuff. When they said it was $100, I bailed. So now they don't get the $60. But enough people will go on paying that they will actually make more money on Office in the next year, not less.
Not enough people are willing to vote with their wallets or even their feet to effect any meaningful change. At least not when it comes to their tech toys.
I have been using Libre office for a while now and it's superior to office in every way.
Not enough people are willing to vote with their wallets
That and most governments are wrapped up in Windows, and therefore kinda just captive to the insane pricing. I get everything I need out of LibreOffice, personally.
The sole reason I still pay the Microsoft tax is Excel. Other office suite components are generally good enough to fill in for their Microsoft counterparts. But, spreadsheet programs are one area where open source competitors need to get their shit together.
Most of them can do the basics but Excel is still in a class by itself for power users and advanced functionality. That's a real bummer because I would love to stop paying the Microsoft tax.
Its the vba. Its proprietary and not for sale anymore and theres not a good free replacement. Been writing a reporting system tha5needs scripting and have had to use javascrip amd heavily cover things for end users to even understamd what is happening.
I'm no dev, but would you consider writing up in detail the features/behaviour you're missing on libreoffice issue tracker?
I love that the only AI goal the oligarchy can focus on is making sure we can all use it to work more.
If you can be in three meetings at once with AI then every single one of those meetings could have been an email
Or a group chat
I mean basically. Call me a paranoid communist but given half the chance they'd fucking bring back slavery.
I think we are there, just under the name of capitalism vs slavery.
that's pretty much where we are now
shit minimum wage, corporations owning housing, and monopolies in pretty much every market. it's just slavery with the illusion of freedom because you can choose which shitty apartment building to live in for over half your income, and which franchise stores you shop at, while your essentials are getting price gouged and constantly worse quality for higher cost, yet the workers don't make more
that's just slavery with extra steps
There are bad things in communism - a reductionist model advertised as fitting for everything (a bit similar to Unix, that), and there are good things in communism - attention to balance of power. Revolutionary ideologies generally have advice for situations warranting a revolution more fleshed out.
She looks happy too!
I believe that's the "I spent six years in college and $150,000 for the 'privilege' of sitting in teams meetings all day." look.
I like to imagine we are witnessing malicious compliance from the model.
Kinda want to send this to my company lol
According to the M365 Copilot monitoring dashboard made available in the trial, an average of 72 M365 Copilot actions were taken per user.
"Based on there being 63 working days during the pilot, this is an average of 1.14 M365 Copilot actions taken per user per day," the study says. Word, Teams, and Outlook were the most used, and Loop and OneNote usage rates were described as "very low," less than 1 percent and 3 percent per day, respectively.
Yeah that probably won't have the intended effect...this basically just shows that AI assistants provide no benefit when they're not used and nothing else.
People probably tried it, found out that it's crap and stopped using it.
.this basically just shows that AI assistants provide no benefit when they're not used and nothing else.
so you think they may be useful but people just like to work harder? or perhps, they tried and saw no benefit at all and moved on?
We have it on our system at work. When we asked what management expected it to be used for they didn't have an answer.
We have a shell script that ingests a list of user IDs and resets their active directory passwords, then locks the account, then sends them an email telling them to contact the support desk to unlock the account. It a cron job that runs ever Monday morning.
Why do a need an AI for when we can just use that? A script that can be easily read understood and upgraded, with no concerns about it going off-piste and doing something random and unpredictable.
So yeah, they don't use it, because it won't work.
Worth noting the average includes the people who did use it a lot too.
So you can conclude people basically did not use it at all.
No shit ‐ the AI bubble provides no value, but it is exciting for the c suite and governments.
Lots of LLM shills in these comments. I hope your work doesn't value reality/accuracy.
I use Copilot for generating images of concepts for presentations at work. It helps me get my point across and no accuracy is needed because it is taking the place of clip art and Google image searches. There is absolutely a place for Generative AI in the workplace. Whether it is worth the cost and whether people are trusting it too much is another question.
It helps me get there more often than not, anywhere from programming I'm unfamiliar with to brainstorming in graphic design. I see a lot of anti-AI folks diss it without considering how it's actually used. It's a tool like any other, and you get what you make of it.
Removed by Moderator — Modlog
If they do, who would buy their stuff?
They really don’t think past the next quarter or two. And even if they did they aren’t lacking on new ways to exploit people.
I mean.. it's software. It's only as good as you leverage it.
From reading the study, it seems like the workers didn't even use it. Less than 2 queries per day? A third of participants used it once per week?
This is a study of resistance to change or of malicious compliance. Or maybe it's a study of how people react when you're obviously trying to take their jobs.
I don't think it's people being resistant to change I think it's people understanding the technology isn't useful. The tagline explains it best.
AI tech shows promise writing emails or summarizing meetings. Don't bother with anything more complex
It's a gimmick, not a fully fleshed out productivity tool, of course no one uses it. That's like complaining that no one uses MS paint for the production of a high quality graphics.
Absolutely, and it's a massive and undeserved cash cow for AI companies (e.g. Sam "Sister-Lovin'" Altman).
AI is never an investment for businesses or individual users. It's a bloated and unfulfillable promise that just makes users dumb, dependant, and destroys the very environment we need to survive.
It also produces bad products (it's easy to tell which devs use it from reviewing poor quality code).
Not to mention the centralisation of power with the rich who are the problem in this world.
The figures are the averages for the full trial period.
So it’s possible they were making more queries at the start of the trial, but then mostly stopped when if they found using Copilot was more a hindrance than a help.
I have a Copilot license at work. We also have an in house „ChatGPT clone“ - basically a private deployment of that model so that (hopefully) no input data gets used to train the models.
There are some usecases that are neat. E.g. we’re a multilingual team, so having it transcribe, translate (and summarize) a meeting so that it’s easier to finalize and check a protocol. Coming back from a vacation and just ask it summarize everything you missed for a specific area of your work (to get on track before just checking everything chronologically) can be nice, too.
Also we finetuned a model to assist us in writing and explaining code from a domain specific language with many strange quirks that we use for a tool and that has poor support from off the shelf LLMs.
But all of these cases have one thing in common: They do not replace the actual work and are things that will be checked anyways (even the code one, as we know there are still many flaws, but it’s usually great at explaining the code now - not so at writing it). It’s just a convenient method to check your own work - and LLM hallucinations will usually be caught anyway.
I think the issue is that there's usually competing targets here. From the top they wanted to see use of copilot. But direct reports likely expected the same performance.
So are you going to a) do it the way you've done it for 10 years and know will deliver on time or b) ask hal9000 for help potentially actually slowing you down, at least initially.
I think there's gains to be made for some jobs in some cases with ai. But most people probably just got on with their work.
Seeing a big uptake in use in the education sector. Teachers paying for their own ChatGPT pro license to lesson plan etc.
Can't comment at this point if that's right or wrong, you hope the teachers using it would identify hallucinations etc. But you can see there is already a change occurring.
I've show my coworkers some practical implementations of copilot and that was enough to kickstart the use.
If you're composing the same mails a lot, for example, you can ask copilot to make a template text and then when you have to compose the same email again you ask copilot to compose and personalize the mail for you. That's an awesome function.
I've made an agent that answers HR related questions of my team. This saves me and HR a lot of time and they are assured their questions are handled discrete.
If you're composing the same mails a lot, for example, you can ask copilot to make a template text and then when you have to compose the same email again you ask copilot to compose and personalize the mail for you. That's an awesome funtion.
Uhm, email templates are far older than LLM.
They also don't make random artifacts out of nowhere
It sounds like it's creating the template and also modifying it as needed. That is a step up.
This template adds or deletes links to relevant webpages and adds recent figures when needed.
We've been using templates for years but this adds personality and customisation
So there's this thing called drafts.
Because they don't know how to use it.
I work for the government and we're trialing Copilot too.
Yesterday I gave copilot several legal documents and our departments long term goals and asked to analyse those documents and find opportunities, legal complications and a matrix of proposed actions.
In less than 5 minutes I have a great overview to start talks with local politicians. This would have taken me at least a day before AI.
And you mad sure that there are no invented points?
Of course they didn't
Yes. If you feed the documents this goes well enough. It just speeds up the process where I start indexing and gathering information a lot.
In that case they would have spent "less than 5 minutes" more time than without this, or, one can say, "at least a day" plus "less than 5 minutes".
100%. I'm also trialing Copilot at a medium-sized corpo job and it saves me roughly 12-20 hours of work per week.
I use it often in PowerShell scripting. It occasionally hallucinates and makes up commands, so sometimes it takes a bit of back and forth to get it to do what I want, but it's still a hundred times easier than writing from scratch or tweaking+combining similar scripts I find online.
Probably my favorite part is being able to ask it "Where did I leave off with John on x issue last week?" And it will remind me that I'm supposed to do x and John is supposed to do y. Or even, "I helped a user with this specific issue six months ago. How did I fix it?" and it pulls the exact email and Teams chats outlining what we did, and I can click the link to open those messages and ensure it didn't misinterperate. Way easier than digging by hand.
Finally, I absolutely hate making PowerPoints so I've been having it make all of my rough drafts from transcription notes in meetings. Super nice time saver.
Something I'm concerned about and playing with this week is pronoun usage in transcripts. I'm working with our LGBTQ ERG to ensure that we can make Copilot use preferred pronouns for everyone. If it can't, we'll need to pull back certain features.
It's far from perfect but it genuinely makes my job a lot easier and I'd hate to lose it. I think it will only get better from here.
okay, but why did they used a guy beatboxing to illustrate their statement ?
Turns out it was a better use of his time, than trying to use Copilot.