In August 2020, the City of London, Ontario deployed to production an AI model that can predict the likelihood of someone falling into chronic homelessness. Matt Ross, head of AI for the city shares more on this project, as well as how others in local government can take advantage of the open source work in London.
A few weeks back we chatted with Mat Daley, CIO for the City of London, Ontario, and skimmed the surface of an innovative project using AI to predict the likelihood of someone falling into chronic homeless.
It’s a tactic that could benefit communities around the world, so now we’re going straight to the source. In this episode, I sit down with Matt Ross, Manager of Artificial Intelligence and IT for the City of London to dive into this groundbreaking project and the use of AI in local government more generally.
More info:
Featured government: London, ON
Episode guests: Matt Ross, Manager of Artificial Intelligence and IT
Visit govlaunch.com for more stories and examples of local government innovation.
Lindsay: (00:05)
Welcome to the Govlaunch Podcast. Govlaunch is the wiki for local government innovation and on this podcast we’re sharing the stories of local government innovators and their efforts to build smarter governments. I’m Lindsay Pica-Alfano, co-founder of Govlaunch and your host.
A few weeks back I chatted with Mat Daley, CIO for the City of London in Ontario and we skimmed the surface of an innovative project using AI to predict the likelihood of someone falling into chronic homeless.
It’s a tactic that could benefit communities around the world, so today we’re going straight to the source. In this episode, I sit down with Matt Ross, Manager of Artificial Intelligence and IT for the City of London to dive into this groundbreaking project and the use of AI in local government more generally.
More and more local governments globally are looking to AI to improve efficiencies. Today, Matt will share some creative ways AI is improving services in London. We’ll also chat more about how to get started on your journey toward AI deployment in your government. So let’s find out what Matt and his team are up to.
Hey, Matt, thanks so much for joining me today. Can you quickly introduce yourself and tell our listeners a bit about what you do?
Matt: (01:17)
Yeah, for sure. Thanks for having me. So my name is Matt Ross. I'm the manager of artificial intelligence and I run our artificial intelligence research and development lab here at the city of London in our IT department. And so largely what I do is along with kind of developing and deploying AI models and systems for the city of London, um, as well as, you know, building the operational structures to support AI delivery building, um, kind of governance policies and support policies for AI. And I also run a few of our smart cities projects, an ultra high speed fiber network in our Southern regions to help close that digital divide, 5G network pilots and a rebuild of our municipal website.
Lindsay: (01:56)
Wow. So you're busy. So how are things going in London's AI department today?
Matt: (02:04)
You know, despite some of the disruption caused by COVID-19 in 2020, we're able to do a fair number of projects still. And realistically, this is AI is new for the city of London. This area is one year old or a little under a year old. So we have a few solid projects in production, a few prototypes and some more projects that were kind of in discovery and planning right now. So it's good. We're chugging along.
Lindsay: (02:30)
You actually just recently deployed to production a pretty impressive AI model. That's likely going to have a big impact on social services around homelessness. Can you share a bit more about this project?
Matt: (02:41)
Yeah, for sure. So really excited about that project that launched at the end of August. So this was our chronic homelessness AI model or our CHAI model. So it consumes all of the shelter data, the usage of our shelters or housing services, food banks, all of these organizations merged their data together. We built a model on top of that, which predicts the probability of an individual in our shelter system becoming chronically homeless six months in the future. So chronic homelessness means greater than 180 days in the shelter system in a year and the model also employs explainable AI, which allows users to understand why the model makes the prediction it does. And the really interesting thing about this model is it's working exceptionally well. The state of the arts are kind of global state of the art, with a 92.1 recall, which means kind of the accuracy at which it can identify an individual who is at risk.
Matt: (03:38)
It is 92.1% accurate at doing so. An interesting business case behind this whole thing, you know, the why we did this is - a Calgary study showed that a chronically homeless individual costs the city $135,000 a year. A chronically homeless individual in our shelter system uses on average 534 days of shelter stays, where someone who isn't chronically homeless we'll use 45. So that's 12 times as many resources used by what is actually a small, like 4 to 6% of our shelter clients in the first place. So it's a really impactful project. If we can use those risk predictions to predict probability of chronic homelessness, then intervene with resources early to decrease chronic homelessness in the city.
Lindsay: (04:25)
How long did this project take from ideation to deployment?
Matt: (04:29)
So it began as an idea in April, 2019. So April 2019 to August for deployment, that included three iterations of the model architecture and then we deployed it to Azure Machine Learning Studios and integrated that into the application used by our homeless prevention division.
Lindsay: (04:46)
I think we all may be surprised to hear how you accomplished this with a team of essentially four people and not all full time. Can you share a bit more about the team and why you think these smaller teams are actually better when it comes to building AI models?
Matt: (04:59)
Yeah, absolutely. So it was a small team of four of us and a few other stakeholders around that, but just small teams in general, I'm a large fan because they can be nimble. They can make decisions quickly, iterate very, very quickly. And I think with projects like this, where there's very high uncertainties, like AI projects, you need to get to prototypes quickly and get them in front of customers and stakeholders. You end up then getting ideas for other requirements and you can then iterate based on that, as opposed to the usual kind of cascade water flow project management method. So that's why I love low small teams. Our team was so myself, a manager who is a data scientist who could connect with our data scientists, our database administrators, and the business, and kind of translate between the three, but then the other critical people are a data scientist who really builds the model, Blake who built our V2 and V3 of the model.
Then Ryan, our database administrator and business systems analyst in IT, he was essential to not only getting the data, getting that pipeline set up, but then if you actually want to productionize a model, you know, doing all the dev ops and security, getting that model into deployment, you need that person as well. Not just the data scientists. And I think that's in general, just the reason why prototypes in companies struggle getting into production. And then finally, you know, very, very critical person was Jonathan, a manager in homeless prevention. He helped us understand the data, make sense of the explanations we were getting out of the model and ultimately is owning the solution and implementing the predictions like, you know, getting the predictions is only step one.
Lindsay: (06:34)
Right. I want to segue into that. Like, obviously your focus is on the accuracy of the model. So this is a little outside of your realm here, this question. But there's a lot of work that needs to be done to then put these insights to good use, right? Can you break down for me what you, or the city invisions that looking like?
Matt: (06:51)
Yeah, absolutely. So as you said, that's the first step and I worry about building a good model, has great performance metrics we can maintain and support it, but ultimately now that it's in the hands of caseworkers and homeless prevention, that's where the rubber meets the road and where we actually find out if this hypothesis is correct. If we prioritize resources, can we decrease chronic homelessness? So, um, homeless prevention has this network of almost 20 shelters and different service agencies that they're now thinking on, you know, how do we create a standard of practice? How do we integrate this into the daily operational workflow of caseworkers? And do we actually look at a housing or rapid rehousing model? So those are identified as extremely high risk early on, are they rapidly rehoused? So that is they're beginning to do that kind of work. They're beginning to build those kinds of procedures now that they have the analytics in place.
Lindsay: (07:48)
Like any project, these models need maintenance and have to be trained on an ongoing basis to ensure accuracy. You've chosen to outsource this to a vendor. Can you tell us who you're using and why you went this route versus maintaining in house?
Matt: (08:01)
Yeah, for sure. We outsource to a vendor for two big reasons. The first is retention. Like frankly data science salaries are extremely high. It's more effective use of taxpayer dollars if we maintain this model through a vendor SLA rather than having an in house data scientist who a portion of their role is dedicated to maintenance and support of this model. Then the retention piece is the other, other prong in the strategy. So a lot of data scientists that I've met or managed in the past want to build models, they want to produce this value, not just do ongoing maintenance for break fix. So in trying to attract AI talent to the city, I think that's the best strategy so that they understand they're not just kind of being saddled with only maintaining these applications that may change in the future as you know, cost of data scientists come down, that those skills kind of spread out in the labor force, but at this time this is the best approach. We partnered with an organization called Dimensional Strategies Incorporated out of Mississauga. We've used them in the past they're our business intelligence vendor, uh, the really talented Microsoft partner. And so they're handling that annual retraining, break fix, and any ongoing performance monitoring, making sure all the explainable and ethical explainable AI and ethical AI development practices are still in place. Things like reducing bias, that sort of thing.
Lindsay: (09:23)
I love what you guys have built. And Govlaunch is all about how do we get this information in front of more people and accessible to others that wouldn't have these resources. So the model you've made available open source for use by any local government. Those in Canada on the HIFIS database, which stands for, I believe homeless individuals and families information system can implement, uh, with very little work using what you all have already built. Um, how would others access this?
Matt: (09:54)
Yeah, so the model is going to be available on our AI at CLN and github. And we're also going to post that link as a resource on our city of London Govlaunch page, just to try to get that out there as much as possible, because as you've mentioned, it's using a HIFIS application, it's a Canadian federal government built and distributed this application to cities. If they don't already have a homeless information management system. So this is something that dozens of municipalities use - dozens of municipalities can use this model in a pretty turnkey fashion to train their own model. Um, so that's why we thought it was really valuable to, to open source it. And along with that will come, uh, print publication and some pretty extensive documentation on how to train your own model, how to configure it, that, that sort of thing.
And just it's worth mentioning just why I value open source, not just because it's going to eliminate some of the duplicated work that happens across cities in the world and in Canada, but we built better code. We built a better model. When you have a target for open source, you need to think about someone who's never seen the model or the code before. So you document it better. You build it in a more modular fashion and more extensible fashion. And ironically, it actually ended up saving us money in our maintenance and support contract because it was so well-documented because it was ready for open source that the scope was very clear for our maintenance and support vendor to understand how to support it. So ended up saving us money in the long run, which was a great benefit of it.
Lindsay: (11:26)
You got a discount for doing their job for them. Great. You're also working on some pretty cool AI projects unrelated to this one. Can you tell me more about perhaps a recent one?
Matt: (11:39)
Yeah, for sure. So one that happened right in the mix of the beginning of the COVID-19 pandemic and that we're pretty proud of it's called COVID CXR, it's available on our github as well as a resource on Govlaunch. It was a model built over about four weeks, five weeks. It was a collaboration with the Center for Next Generation Networks, the head of our ICU at our local Victoria hospital here and some Waterloo researchers, it was a model which predicted whether you had COVID-19 or not from a chest X Ray. And so it had quite high accuracy or recall and precision and we opensourced it. And since then it's been used in over 20 countries. A lot of data scientists have, have forked the repo and worked on it or asked us questions privately, that kind of thing.
Then along that we're doing some demand forecasting for our water utility, uh, using some recurrent neural networks. I think forecasting in general will be a really big area of machine learning, um, for me as palliative in the years to come. So the cool thing about this model, um, is we have a ton of data for water demand and water usage across the municipality. And it's very critical for us in doing budgeting in trying to make sure we resource our infrastructure correctly, uh, for the demand in different areas of the city to understand, uh, solid forecasting and predictions of water usage, depending on different clients, whether they're commercial, residential, what size they are, that sort of thing. So often the modeling that's done is, um, you know, basic statistical modeling. So we'll be using what are called recurrent neural networks, which are better at forecasting time series data such as water demand and water supply. Then we'll project water usage into the future for the city. And then that'll give us much more highly accurate budget numbers and forecasting for our infrastructure.
Lindsay: (13:32)
Well, I think just touching on these projects really shows us the breadth of impact that AI could have. So this is a relatively new concept in local government and having an AI department is pretty progressive. You're obviously a huge advocate for the use of AI in public service. Can you explain why this is and the opportunity you see specifically for local government?
Matt: (13:54)
I'm quite thankful to be in this role in this city at this time. Cause there's, there's huge opportunities. I think a huge shout out needs to go out to someone though because the only reason this is possible is we have a fairly forward thinking, director of IT, Mat Daley. So he sees the value in these kinds of projects and has kind of held the space open to allow these projects to exist because AI projects don't tend to look like regular IT projects, especially from project management standpoint, there's far, far greater number of uncertainties in those projects and it tends to look like a much more iterative, agile process. So, um, huge credit to him to holding that space open to allow that and influencing when necessary. Um, because another strange artifact I think of when municipalities are just adopting AI is some of these projects are AI led rather than the business leading the business, meeting something and coming to IT to support them.
So that's a different interaction. And so it requires a different kind of leadership to make that a success. But where the value is for this, I mean, like personally I find artificial intelligence, fascinating, um, the opportunities to do good in government rather than just, you know, not to hate on the for-profit colleagues, but to get more ad clicks, like a lot of data science labor is expended in getting more ad clicks and the opportunity to do good for citizens and residents with AI is enormous. And why there's a huge opportunity to municipalities is what we have huge data sets. We have a large number of operational processes with large volume of tasks flowing through those. There's a great candidates for optimization and AI models to really tap some of the value there. Um, and I think it's interesting cause we're just at the early stages of it, there's a huge opportunity to leverage, you know, sharing technology and building things more collaboratively.
Lindsay: (15:52)
What do you think are some of the biggest concerns around AI use in local government?
Matt: (15:58)
Yeah, I think the big concern is ethically I development. Are we eliminating unintended bias, like hearing the word bias in AI is commonly brought up as the barrier to AI adoption in government. Are you using explainable AI? So it's not a black box model, where's the transparency or the human in the loop, that kind of thing. And I think actually this is probably why you see a lot of prototypes, never leaves the prototype stage at especially at governments, but even in, in other sectors as well because the amount of infrastructure you need to place around a model to deploy into production in an ethical way, especially if it's processing personally identifiable data, it's significant. So you really need to kind of have those ducks in a row first, if you're going to deploy a model.
And so it's, I don't think it's a barrier, it's a concern and something to address, and it can be addressed. There's ways to address all of those issues and build requirements and features that'll address them. The other big chunk is I think, literacy of end users. We're building for businesses that often don't understand AI, and we're accountable to the public. Most of them do not understand the inner workings of AI. So that gap makes it extremely difficult because there's so much hype about AI, but no understanding of it. So it seems opaque when really it's just another tool. And we, and including myself in this could do a better job at educating one another on how these tools work.
Lindsay: (17:29)
Yeah. There's some other things that this is reminiscent of, internet of things, 5G it's all, it's all very scary or exciting depending on who you ask.
Matt: (17:39)
Exactly in the end. The problem is that those two groups don't talk to each other enough. And an interesting segue on that is in the early kind of requirements gathering after I'd built a version one model, and we brought it to homeless prevention, we didn't have explainable AI. This was just a prototype. And we built all of these ethical AI requirements because when we talked to EDS of shelters, they were like, well, I don't trust this thing because how could I trust it if it doesn't explain why it made the decision? How could I explain that to clients in the shelter system? And so it kind of forced those requirements, uh, and it was kind of an awesome organic way of developing AI ethically actually.
Lindsay: (18:20)
And for those in leadership and the citizens that are particularly worried about the use of this technology, how would you respond to their concerns?
Matt: (18:29)
So I would respond saying that if you're concerned about this technology, you have a right to be concerned if it's not done well. I think that just at the base level, like if it's not done well, like we should be concerned as citizens because it can have bias and it can be unethically built if you're not thoughtful about it. We're public servants and we need to be accountable if we're deploying these systems that impact residents' lives. But the good news is that all of these concerns can be addressed. They're just requirements that need to be baked into the project, whether it's features or procedural, processes and how it's deployed, but we can address all of those problems.
Lindsay: (19:11)
And what is some advice you'd give local governments who are considering deploying AI or just starting an AI department?
Matt: (19:18)
So a few things come to mind. One I'm sure you've heard before is the classic garbage in garbage out. Machine learning models are only as good as the data they rely on. That means data governance and data hygiene are going to be critical. So from a strategic perspective, focusing on data governance and data hygiene will set you up for success for future AI projects. But that being said, data doesn't need to be perfect. You can prototype model and see what the performance metrics are to set a baseline and then improve quality further in the future. Another big one would be, and I mentioned a bit earlier, but early AI projects are gonna look a lot more like IT project influencing rather than it project management. Um, because you are probably going to be, you know, looking at database, seeing that it has the hygiene and you see a valuable AI application, and then you're kind of selling that to the business and explaining that to them.
Matt: (20:07)
So it's, it's it influencing out rather than just serving the businesses because of that, that gap in knowledge. So it's a different set of skills and some different members on the team you'll need. And then another big one is actually begin with highest value AI project. So whether that means, SROI social return on investment or ROI, um, rather than just the low hanging fruit projects. And this is to unlock kind of the Goodwill to do more projects, because for all the reasons I mentioned for all the, you know, to develop something and deploy it into production, there's all of this infrastructure you need to put around it. So if you only go for the kind of low value, low hanging fruit, that just it's easy to develop, but there might actually not be that much value. You're not going to want to put in all the effort to maintain and support it and deploy into production. So you might as well focus on the really high value problems and then that'll unlock the energy to do the next high value projects.
Lindsay: (21:03)
A few weeks back, we did an episode with the team in Edmonton and Ben Greedy shared that he thought we should go with the low hanging fruit. I think he mentioned the juicy, ripe fruit at the bottom of the tree, do more that are going to offer value versus picking a big project that's gonna take months and months. What would your response be to that?
Matt: (21:27)
Yeah, that's, that's awesome. Ben we sit on a community practice together. Awesome guy, super brilliant guy. Um, so I slightly disagree with a caveat, um, that the, you know, the low hanging fruits and need to be value adding, right? They need to be extremely valuable if they're low hanging fruit in so far as like all the data is there. It's very easy to access the structured data, but the value is going to be minimal. Then I think that is like a low hanging fruit that you need to dismiss. I still do really think it's valuable to put in the effort to run these, you know, year long projects or more to ground as long as you can iterate quickly. This homeless prevention, chronic homelessness pro AI model was a year and a half long model from prototyping to deployment.
But we quickly within about a month in month and a half had built a model like version one of the model and validated whether there was at least a path forward. if we hadn't of done that, I totally agree. It is not worth like waiting until a year and a half from now to find out whether there's value in the project. So you need to iterate quickly. But I think then it is worth doing this like multi year project, if the value is high enough, but it needs to be something you can kind of quickly iterate and validate the feasibility of the model.
Lindsay: (22:43)
Right. So do you have time for a few more quick questions? Um, what is the most interesting innovation project you've heard of lately?
Matt: (22:52)
So this was, um, back at a conference when you were allowed to have conferences in person. Um, it was a project in Mexico City and it was a crowdsource mapping of their informal transit system and it's a wild project. So they have an informal transit system that has 14 million daily rides. And it's like tens of thousands of buses and cars that are all informally mapped. Like there's no routing, that's kind of formalized by the government. It's all informal citizens, just running a bus service. Um, and so it's like, that's a wild problem to try to make that system more accessible to other citizens because you kind need to like ask around like, who knows what bus to take to get to X Y or is that location, that kind of thing. So they built an application, it was then gamified, so that teams could go out and if they shared their GPS coordinates to map different routes, they would get more points.
They get more points for like lower density than higher density areas to try to spread out the whole system. Um, within two weeks they mapped all like 5,000 routes. Um, you know, take a photo where the stop is, what the bus or a car or whatever it looks like. And it was wild. They mapped it the whole system. Then they did a hackathon to build an SMS application where you would say, here's where I am, here's where I'm going and it would send you like the directions to get through this, this wild informal system I saw. It was like such a cool example of like realistic crowdsourcing, you know, cause there was actually a user adoption strategy and then a hackathon to solve a problem for government that like the only other solution is regulating The informal transit system. And I just loved that. They found a way to not have to do that. That's very cool.
Lindsay: (24:34)
Wow. Awesome. And then of course we love talking about failure on Govlaunch because we think that's where you learn the most. What's something you all have tried that didn't work?
Matt: (24:46)
So early in the AI lab days, one of those low hanging fruits I mentioned was what we have all these structured its service desk tickets. Why don't we do some unsupervised clustering on it to maybe find out are there service desk objects that we're not identifying, they're getting kind of looped into the general service requests. Maybe we can improve some efficiencies there and it worked, but it just wasn't valuable enough to then like do all the work necessary to then deploy something like that into production. It was just a reminder to me that the low hanging fruit project, um, isn't necessarily going to be the valuable project that you're going to want to run forward with. Like if you have a barn, don't have a barn party, that kind of thing. Um, so I think that was one, um, a personal one was I tried to build a computer vision model that would tell me when I was slouching and texts me when I was, and that did not work for a lot of reasons, but taught me a lot about computer vision models
Lindsay: (25:42)
I'm sitting up straight or since you mentioned that.
Matt: (25:44)
Good.
Lindsay:
Actually that would be very valuable now with everybody working from home and slouching over on their zooms.
Matt: (25:53)
Alright, well maybe I'll revisit.
Lindsay: (25:55)
Hey, that's what you gotta do. You gotta come back after a failure. Try again. Can you tell me about a govtech product you'd strongly recommend and why?
Matt: (26:07)
Yeah, so the Govlaunch Wiki. I think in general, just the idea of creating a repository and form for government sharing different solutions is just a fantastic idea. And I think it'll add a ton of value for the work we do.
Lindsay: (26:20)
Yeah. Awesome. Well, we agree obviously. What's something that excites you about the future of AI for municipal use?
Matt (26:30)
So I think a few things, I mean, it's like a Greenfield there's, there's not a lot of AI work being done or at least it's very early days for a lot of municipalities. So there's huge opportunity to influence the direction locally, nationally, and globally like opportunity to collaborate with other cities. I mean we have very similar problems, we have similar services, we sometimes have the exact same applications. So there's a ton of opportunity to really early on start collaborating as a whole sector, as a group. And then I think we just have really interesting, I'll call it interesting data sets and interesting opportunities. A municipal government is like 110 businesses all running under one roof, like totally independent businesses with their own cultures, applications and operations. We have tons of infrastructure that you could run some really interesting AI projects on. So there's a lot to do, a lot of exciting projects to work on and it's just early days. Right.
Lindsay: (27:27)
Thanks again for coming on and uh, we'll talk soon.
Matt:
Awesome. Thanks for having me Lindsay.
Lindsay: (27:39)
The City of London proves that once again, they’re having a positive impact beyond their city limits. The work to make this open source will not only save them money on maintenance, but will be impactful as a turnkey solution for other Canadian municipalities. With a little work, local governments globally can put Matt and his team’s model to good use.
We encourage more local governments to work toward open source development and to search Govlaunch for London’s AI model and other open source solutions that may meet your needs rather than building from scratch in house.
Thanks again for Matt Ross for coming on to talk about his great work with Artificial Intelligence.
I'm Lindsay Pica-Alfano and this podcast was produced by Govlaunch, the wiki for local government innovation. You can subscribe to your more stories like this, wherever you get your podcasts. If you're a local government innovator, we hope you'll help us on our mission to build the largest free resource for local governments globally. You can join to search and contribute to the wiki at govlaunch.com. Thanks for tuning in. We hope to see you next time on the Govlaunch podcast.