Butnaru and others associated with the HAI center at Stanford set up a Agile Ethics workflow in the form of a Trello board. From left to right, the workflow walks you through relevant ethical considerations at the various steps of a machine learning pipeline. The phases are: Scope Consider ethical implications of the project Consider skill mapping (what’s the impact of AI on jobs)? Facilitates up-skilling or a change of strategy in the use of human talent Data audit Led by Chief Data Officer “Meet and plan” stage in Agile Helpful: Data Ethics Canvas Train Build stage in Agile Consider (tools for) transparency and fairness Analyse Benchmarks, including benchmarks related to e. Read more...

AlgorithmWatch is maintaining a searchable inventory of published frameworks that set out ethical AI values. They can be searched on sector/actor, type, region and location. AlgorithmWatch noted some common patterns here after publishing the first version of the index: “All include the similar principles on transparency, equality/non-discrimination, accountability and safety. Some add additional principles, such as the demand for AI be socially beneficial and protect human rights.” “Most frameworks are developed by coalitions, or institutions such as universities that then invite companies and individuals to sign up to these. Read more...

AI Now published the Algorithmic Accountability Policy Toolkit in 2018. It is specifically tailored towards advocates concerned with government use of algorithms. The toolkit provides a FAQ, an overview of various types of algorithms used by governments in specific application areas such as public health or criminal justice, and a comprehensive list of relevant literature. The fact that this is a toolkit and not a paper can be seen from the very practical guidance that is offered. Read more...

The Data Ethics Canvas is a tool developed by the Open Data Institute for providing ethical guidance to organizations doing any type of project involving data. That includes data collection, sharing, and its usage for example in machine learning applications. The tool is accompanied with a white paper and a brief practical guide for its usage. Page 3 of the practical guide lists some recommendations that are also relevant when you do not use this tool. Read more...

In analogy with nutrition labels on food products, the authors of this paper propose a way to create a Data Nutrition Label. The goal of this method is to asses data quality and mitigate potential problems early on before building models on the data. According to the authors, their approach is different from the datasheet in that the “proposed datasheet [i.e. by Gebru et al.] includes dataset provenance, key characteristics, relevant regulations and test results, but also significant yet more subjective information such as potential bias, strengths and weaknesses of the dataset, API, or model, and suggested uses. Read more...

A data statement, according to the authors, is … a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize,how software might be appropriately deployed,and what biases might be reflected in systems built on the software. (587) This paper specifically focuses on ethically responsive NLP technology. The authors argue that a data statement should be an integral part of work and writing on NLP. Read more...

The method described in this paper aids in documenting datasets to help avoid unwanted consequences of data usage. Abstract: The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains. To address this gap, we propose datasheets for datasets. In the electronics industry, every component, no matter how simple or complex, is accompanied with a datasheet that describes its operating characteristics, test results, recommended uses, and other information. Read more...

This toolkit developed by the Utrecht Data School supports data analysts, projectmanagers, and policy makers in identifying ethical values and issues in data projects and promoting accountability towards stakeholders. The toolkit is written in Dutch and includes a poster to support brainstorm sessions, an interactive survey, and an accompanying guide with further explanations. On the toolkit’s website you can also find several case studies that highlight ethical issues in data projects, as well as a version of the toolkit specifically for researchers. Read more...

FactSheets, as proposed by Arnold et al. from IBM Research, are similar to the model-card and datasheet , but are significantly more comprehensive because they focus on whole AI services. A main difference is that live AI services may comprise several trained models that interact with each other. The model card and datasheet instead concern a single model and the data it is trained on. However, Arnold et al. point out that an AI service with safe components is not necessarily safe overall and “so it is prudent to also consider transparency and accountability of services in addition to datasets and models” (p. Read more...

Model cards are an extension of the datasheet to machine learning models. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Read more...

Introduction A major downside of external auditing is that it typically only can be done after model deployment. This paper presents a methodology for internal algorithmic auditing as an integral part of the development process, end-to-end. Those who move fast and break things, beware: The audit process is necessarily boring, slow, meticulous and methodical—antithetical to the typical rapid development pace for AI technology. However, it is critical to slow down as algorithms continue to be deployed in increasingly high-stakes domains. Read more...