Search form


Become a Member Today Sign Up

Open Cities Index Methodology 2016

Similar to the 2015 Open Cities Index methodology, the 2016 methodology adapts the model of the Open Data Barometer for application at the local level in Canada. The Open Data Barometer study, conducted by the World Wide Web Foundation, successfully benchmarked national open data initiatives around the world in 2013 and again in 2015.

In establishing the Open Cities Index, PSD first reviewed all relevant open data literature including studies from across Europe and North America. In addition to the Open Data Barometer, these studies included POPSIS (Pricing of Public Sector Information Study) conducted by the European Commission, McKinsey Global Institute’s Open Data Study, and OKFN’s Open Data Census.

 

The Survey

Seeking to gather highly accurate and valid data on municipal open data initiatives, PSD opted not to pursue the crowdsourcing model for data collection used by the Open Data Census (website). PSD set out to create a survey that municipalities would complete themselves and return to PSD for validation. The Open Cities Index survey was created using Survey Monkey. The 2016 Open Cities Index survey included more complex questions that allowed for a higher level of detailed data to be collected from municipalities.  

The survey was separated into three sections based on the key elements of an open data initiative: Readiness, Implementation, and Impact.

 

Readiness

The Readiness portion of the survey consisted of five unique questions designed to evaluate the extent to which a municipality is ready to build its open data capacity and foster positive outcomes through their open data initiative. This section begins with three unmarked questions. The first question looks at the total budget that has been allocated to the municipality’s open data initiative in 2016. The remaining two questions within this category reference staffing capacity in terms of number of employees and number of weekly employee hours devoted to an open data initiative. The entire readiness section is marked out of 40 possible points.  This score out of 40 is then converted to an overall score of 25, and is worth 16% of the overall grade.

 

Implementation

The Implementation section of the survey was designed to measure the extent to which a city has fulfilled its open data goals and ultimately, what data it has posted online. This section looks at 32 specific datasets, up from 20 datasets reviewed in the 2015 Open Cities Index.
 
·         Government budget
·         Election data
·         Council expenses
·         Council voting records
·         Census data
·         Lobbyist information
·         Public facilities and structures
·         Company register
·         Municipal permits
·         Code enforcement violations
·         Construction contracts
·         Procurement contracts
·         Property assessments
·         Park inventory
·         Public transit
·         Real-time transit
·         Traffic volumes
·         Traffic accidents
·         Road closures
·         Service requests (311)
·         Zoning (GIS)
·         Base GIS data (roads, addresses, etc.)
·         Restaurant inspections
·         Health performance
·         Education performance
·         Crime statistics
·         City services
·         Bylaw infractions
·         Environmental services
·         Air quality
·         Recreational programs
·         Web analytics
 
 
All 32 possible datasets were assessed against the following 11 variables in order to ascertain the quality of the data:
 
 
 
Points were allocated for all published datasets (to a maximum of 10 points per dataset reflecting the number of the above variables satisfied by the data).  Zero points were allocated if a municipality did not have access to that particular dataset. The variable “The Data Exists” was added to the 2016 Open Cities Index in order to capture the types of data sets that are commonly unavailable to municipalities across Canada. In total, a municipality had the opportunity to score 320 points in this section. This score was then adjusted and converted to an overall score of 100, and is worth 68% of the overall grade.

 

Impact

The last section of the Open Cities Index survey is the Impact section, worth 16% of the overall score. Here, nine questions were asked relating to the usage of published data and the measured impact of an open data initiative on the economy and community.  The total available score here was 58, converted to an overall score of 25 points.

 

Validation of Survey Responses

For 2016, the PSD analytics team audited all of the survey responses within the Implementation section for accuracy.  This consisted of browsing the open data websites/portals for each municipality to find the relevant data for the 32 datasets that the municipality indicated they had available.  The team also double checked for machine readability and presence of APIs. This validation step ensures accuracy of the results and rankings.

 

Updates in 2016

In order to provide municipalities with even greater insight into the relative progress of their open data initiatives, several changes were made between the 2015 and 2016 iterations of the Open Cities Index. PSD hosted several virtual roundtable sessions with municipalities of various sizes over the course of the last year, providing the opportunity to discuss open data challenges with municipal practitioners in the field. PSD also met with experts from the broader open data community in order to get the perspectives of academics, open data end-users, provincial and federal governments, and the non-profit sector. Many of the changes made to the Open Cities Index methodology stemmed from this extensive consultation process. After PSD drafted its revised methodology and survey for 2016, our municipal roundtable participants and stakeholders from the open data community were invited to provide additional comment.

The 2016 Open Cities Index survey included 25 questions, up from 10 in 2015. New questions were added to all three sections of the survey, requiring adjustments to the grading and weighting system of the Index as outlined above.

The Readiness section of the survey no longer consists of simply yes/no questions. This section was enhanced with additional quantitative questions looking at open data budgets, employee capacity, and staff hours.

The Implementation section of the survey was expanded significantly to provide for a more robust analysis of municipal open datasets. Point allocations were added for visualizations, APIs, format conversion, and clear indications as to whether or not a dataset is provided by participating index members. The number of variables for which each dataset is measured against increased from 5 in 2015 to 11 in 2016 (see table above).

It is quite evident from the 2015 OCI results that gauging the impact of a municipal open data initiative is one of the most challenging tasks for both municipalities and for those benchmarking initiatives. In order to develop more precise questions for the Impact section of the OCI survey, PSD researched what municipal governments around the world are doing to gauge the impact of their own open data programs. This analysis contributed to the expansion of the Impact section for the 2016 survey, increasing the number of questions by 150% over the previous year. Qualitative questions were also added to the Impact section in order to continue unearthing new best practices in performance measurement for open data initiatives.
 

Finally, the bonus point system used in 2015 was eliminated for the 2016 Index. In 2015, a municipality had the opportunity to gain 2 bonus points per 50 datasets published. The Open Cities Index was developed to track the progress municipalities across Canada are making in building open data capacity, expanding the quality and breadth of their open data inventories, and improving their ability to not only make a positive impact on their organization and community, but also more accurately measure that impact. There was a broad consensus that providing a bonus point system tied solely to the number of datasets published did not reflect the mission of the Index. Furthermore, there is a need for more standardization around the way governments track and report the number and format of datasets published. Removing the bonus point system did not impact the rankings for the 2016 Open Cities Index, but did ensure that the individual results more accurately represent the progress made by each municipality.

PSD plans on engaging municipalities throughout the next year to further refine the OCI survey and to develop an improved framework for open data performance measurement. 

 

Back to the 2016 Open Cities Index Results