S-0744.2

SUBSTITUTE SENATE BILL 5116

BySenate State Government & Elections (originally sponsored by Senators Hasegawa, Hunt, Kuderer, and Wilson, C.)

AN ACT Relating to establishing guidelines for government procurement and use of automated decision systems in order to protect consumers, improve transparency, and create more market predictability; amending RCW 43.386.901; adding a new section to chapter 49.60 RCW; adding a new chapter to Title 43 RCW; and declaring an emergency.

BE IT ENACTED BY THE LEGISLATURE OF THE STATE OF WASHINGTON:

(1) Washington is a technology leader on a national and global level and holds a distinctive position in creating frameworks around technology that enhance innovation while protecting consumers and promoting fairness, accountability, and transparency for all Washingtonians.

(2) Automated decision systems are rapidly being adopted to make or assist in core decisions in a variety of government and business functions, including criminal justice, health care, education, employment, public benefits, insurance, and commerce.

(3) These automated decision systems are currently unregulated, may be deployed without public notice, and vendors selling the systems may require restrictive contractual provisions that undermine government transparency and accountability.

(4) The average Washington resident is unlikely to understand processes used by these automated decision systems, yet these systems are increasingly used to make core government and business decisions impacting the civil rights and liberties of Washingtonians, raising significant concerns around due process, fairness, accountability, and transparency.

(5) A growing body of research shows that reliance on automated decision systems without adequate transparency, oversight, or safeguards can undermine market predictability, harm consumers, and deny historically disadvantaged or vulnerable groups the full measure of their civil rights and liberties.

(6) Research has shown that even the most innocent looking management tools often incorporate and compound the assumptions of institutional racism and other unfounded stereotypes. It is a matter of good governance to ensure that agencies consider whether the technologies they use improperly advantage or disadvantage Washington residents.

(7) In order to enhance innovation and ensure the use of these systems in ways that benefit Washington residents, the legislature intends to ensure the fair, transparent, and accountable use of automated decision systems.

(1) "Agency" or "public agency" means any entity meeting the definition of "public agency" as defined in RCW 42.30.020.

(2) "Algorithm" means a computerized procedure consisting of a set of steps to accomplish a determined task.

(3) "Algorithmic accountability report" means the report with content enumerated in section 5(6) of this act.

(4) "Algorithmic accountability review office" means:

(a) For a state agency, the office of the chief information officer; and

(b) For local municipal entities including, but not limited to, cities and counties, the chief information officer for the county in which the agency is located or such other department head as may be designated by the executive of that county.

(5) "Artificial intelligence-enabled profiling" means the automated or semiautomated process by which the external or internal characteristics of an individual are analyzed to determine, infer, or characterize an individual's state of mind, character, propensities, protected class status, political affiliation, religious beliefs or religious affiliation, immigration status, or employability.

(6)(a) "Automated decision system" means any algorithm, including one incorporating machine learning or other artificial intelligence techniques, that uses data-based analysis or calculations to make or support government decisions, judgments, or conclusions that cause a Washington resident to be treated differently than another Washington resident in the nature or amount of governmental interaction with that individual including, without limitation, benefits, protections, required payments, penalties, regulations, timing, application, or process requirements.

(b) "Automated decision system" does not include tools that do not make or support governmental decisions, judgments, or conclusions that cause a Washington resident to be treated differently than another Washington resident in the nature or amount of government interaction with that individual including, without limitation, internal governmental computer server or electrical usage optimization, antivirus programs, and internal governmental space optimization programs.

(7) "Automated final decision system" means an automated decision system that makes final decisions, judgments, or conclusions without human intervention.

(8) "Automated support decision system" means an automated decision system that provides information to inform the final decision, judgment, or conclusion of a human decision maker.

(9) "Use" means to operate an automated decision system or to contract with a third party to operate an automated decision system to automate, aid, or replace any decision-making process that would otherwise be made by an agency.

(1) A public agency may not develop, procure, or use an automated decision system that discriminates against an individual, or treats an individual less favorably than another, in whole or in part, on the basis of one or more factors enumerated in RCW 49.60.010. A public agency may not develop, procure, or use an automated final decision system to make a decision impacting the constitutional or legal rights, duties, or privileges of any Washington resident, or to deploy or trigger any weapon.

(2) A public agency may not operate, install, or commission the operation or installation of equipment incorporating artificial intelligence-enabled profiling in any place of public resort, accommodation, assemblage, or amusement, as defined in RCW 49.60.040, or use artificial intelligence-enabled profiling to make decisions that produce legal effects or similarly significant effects concerning individuals. Decisions that include legal effects or similarly significant effects concerning consumers include, without limitation, denial or degradation of consequential services or support, such as financial or lending services, housing, insurance, educational enrollment, criminal justice, employment opportunities, health care services, and access to basic necessities, such as food and water.

(3) A public agency shall develop, procure, or use an automated decision system only after the public agency first completes an algorithmic accountability report, provided that for automated decision systems in use on the effective date of this section, a public agency has until January 1, 2023, to complete an algorithmic accountability report on such a system and to comply with the requirements of subsection (4) of this section.

(4) A public agency that develops, procures, or uses an automated decision system must follow any conditions set forth in the relevant approved algorithmic accountability report. In addition, the public agency must, at a minimum:

(a) Give clear notice in plain language to an individual impacted by the automated decision system of the following:

(i) The fact that the system is in use;

(ii) The system's name, vendor, and version;

(iii) What decision or decisions it will be used to make or support;

(iv) Whether it is an automated final decision system or automated support decision system and whether and through what process a human verifies or confirms decisions made by the automated decision system;

(v) What policies and guidelines apply to its deployment; and

(vi) How an individual may contest any decision made involving the automated decision system as required pursuant to this section;

(b) Ensure the automated decision system and the data used to develop the system are made freely available by the vendor before, during, and after deployment for agency or independent third-party testing, auditing, or research to understand its impacts, including potential bias, inaccuracy, or disparate impacts;

(c) Ensure that any decision made or informed by the automated decision system is subject to appeal, immediate suspension if a legal right, duty, or privilege is impacted by the decision, and potential reversal by a human decision maker through a timely process not to exceed 20 days, and clearly described and accessible to an individual impacted by the decision; and

(d) Ensure the agency can explain the basis for its decision to any impacted individual in terms understandable to a layperson including, without limitation, by requiring the vendor to create such an explanation.

(5) A procurement contract for an automated decision system entered into by a public agency after the effective date of this section must ensure that the minimum standards set forth in this section are able to be effectuated without impairment, including requiring the vendor to waive any legal claims that may impair these minimum standards. Such a contract may not contain nondisclosure or other provisions that prohibit or impair these minimum standards.

(2) A public agency intending to newly develop, procure, or use an automated decision system between the effective date of this section and January 1, 2024, must, at least one month prior to procurement of, or if internally developed, implementation of, such a system produce and file with the applicable algorithmic accountability review office an algorithmic accountability report for that system.

(3) An agency intending to develop, procure, or use an automated decision system for implementation after January 1, 2024, must submit an algorithmic accountability report to the applicable algorithmic accountability review office and obtain approval or conditional approval prior to any use of the automated decision system. The algorithmic accountability review office must post the algorithmic accountability report on the algorithmic accountability review office's public website and invite public comment on the algorithmic accountability report for a period of no less than 30 days.

(4) After receiving public comment, the algorithmic accountability review office must determine whether the intended use of the automated decision system meets the minimum standards set forth in section 4 of this act. On the basis of that determination, the algorithmic accountability review office may approve the use of the automated decision system in accordance with rules and procedures set forth in the algorithmic accountability report, deny it, or make changes to rules and procedures set forth in the algorithmic accountability report prior to approval.

(5) Except as provided in subsections (1) and (2) of this section, no agency may develop, procure, or use an automated decision system prior to obtaining the approval required in this section, and after approval, such an agency may use the automated decision system only in accordance with the policies and procedures set forth in the approved algorithmic accountability report.

(6) Each algorithmic accountability report must include clear and understandable statements of the following:

(a) The automated decision system's name, vendor, and version;

(b) A description of the automated decision system's general capabilities, including reasonably foreseeable capabilities outside the scope of the agency's proposed use and whether the automated decision system is used or may be used to deploy or trigger any weapon;

(c)(i) The type or types of data inputs that the technology uses; (ii) how that data is generated, collected, and processed; and (iii) the type or types of data the system is reasonably likely to generate;

(d) Whether the automated decision system has been tested by an independent third party, has a known bias, or is untested for bias;

(e) A description of the purpose and proposed use of the automated decision system, including: What decision or decisions it will be used to make or support; whether it is an automated final decision system or automated support decision system; and its intended benefits, including any data or research demonstrating those benefits;

(f) A description of how the agency plans to comply with each requirement set forth in section 4 of this act;

(g) Whether the automated decision system makes decisions affecting the constitutional or legal rights, duties, or privileges of any Washington resident;

(h) A description of any potential impacts of the automated decision system on civil rights and liberties and potential disparate impacts on marginalized communities, and a mitigation plan;

(i) Whether any of the decision criteria are mandated by statute and, if so, which criteria and by what statutes;

(j) A clear use and data management policy, including specific protocols for the following:

(i) How and when the automated decision system will be deployed or used and by whom including, but not limited to: The factors that will be used to determine where, when, and how the technology is deployed; and other relevant information, such as whether the technology will be operated continuously or used only under specific circumstances. If the automated decision system will be operated or used by another entity on the agency's behalf, the algorithmic accountability report must explicitly include a description of the other entity's access and any applicable protocols;

(ii) Any additional rules that will govern use of the automated decision system and what processes will be required prior to each use of the automated decision system;

(iii) How automated decision system data will be securely stored and accessed, and whether an agency intends to share access to the automated decision system or the data from that automated decision system with any other entity, and why;

(iv) How the agency will ensure that all personnel who operate the automated decision system or access its data are properly trained and able to ensure compliance with the use and data management policy prior to use of the automated decision system;

(v) A description of any public or community engagement held and any future public or community engagement plans in connection with the automated decision system; and

(vi) A description of the fiscal impact of the automated decision system, including: Initial acquisition costs; ongoing operating costs such as maintenance, licensing, personnel, legal compliance, use auditing, data retention, and security costs; any cost savings that would be achieved through the use of the technology; and any current or potential sources of funding, including any subsidies, incentives, or free products being offered by vendors or governmental entities.

(2)(a) For algorithmic accountability reports filed between the effective date of this section and January 1, 2024, each algorithmic accountability review office shall conduct selective audits of automated decision systems for which an algorithmic accountability report has been filed and shall take appropriate action, such as approval, conditional approval, or denial with regard to the systems so audited. The selective audits conducted must contain the elements described in (b) of this subsection. In selecting which systems to audit, the algorithmic accountability review office may take into account:

(i) The number of persons affected by the automated decision system, including systems in use by multiple jurisdictions;

(ii) The apparent likelihood that the system creates unintended, erroneous, or discriminatory results;

(iii) The severity of the effects of an unintended, erroneous, or discriminatory decision on the affected individual; and

(iv) Other criteria as the algorithmic accountability review office deems appropriate to a selective audit.

(b) Beginning January 1, 2024, each algorithmic accountability review office shall conduct an annual audit that includes the following:

(i) Whether each agency that uses, develops, or procures an automated decision system has complied with the terms of its approved algorithmic accountability report;

(ii) Descriptions of any known or reasonably suspected violations of any algorithmic accountability report policies;

(iii) Any systematic issues, such as bias, disproportionate impacts on marginalized or vulnerable communities, raised by use of automated decision systems;

(iv) Publishing recommendations, if any, relating to revision to this chapter or to specific automated decision system algorithmic accountability reports.

(c) By January 1, 2022, the Washington state chief information officer shall establish guidelines for the number or percentage of systems to be audited by each algorithmic accountability review office pursuant to (a) of this subsection.

(d) The first annual audit must be made publicly available on the algorithmic accountability review office's website by March 1, 2024, and annually thereafter on or before March 1st.

(3) Beginning January 1, 2023, each agency using an automated decision system must publish on its website annual metrics regarding the number of requests for human review of a decision rendered by the automated decision system it received and the outcome of the human review.

NEW SECTION.  Sec. 8. A new section is added to chapter 49.60 RCW to read as follows:

It is an unfair practice for any automated decision system to discriminate against an individual, or to treat an individual less favorably than another, in whole or in part, on the basis of one or more factors enumerated in RCW 49.60.010. For the purposes of this section, "automated decision system" has the same meaning as defined in section 2 of this act.

Sec. 10. RCW 43.386.901 and 2020 c 257 s 14 are each amended to read as follows:

Sections 1 through 9 and 11 through 13 of this act take effect July 1, ((2021))2026.

--- END ---