Artificial Intelligence Technology and the Law

  • Does Illinois’ Artificial Intelligence Video Interview Bill Fully Address Biometric Data Privacy Concerns?
    Developers of artificial intelligence-based video interviewing systems promote their technology as one that helps human resource professionals on-board new talent faster, less expensively, and with greater insight compared to traditional human-only interviewing techniques. They also contend that their systems can avoid some of the potential implicit biases that may appear before, during, and after interviews, thus reducing risks to companies while leveling the playing field for qualified job applicants. But because those AI system have the potential to collect, store, and use data reflecting a job candidate’s face and voice, lawmakers in Illinois passed the Artificial Intelligence Video Interview Act on May 29, 2019, by a vote of 114-0, placing restrictions on companies using AI video interviews without consent and adding to work place restrictions in Illinois that have been in place for over a decade following enactment of the state’s Biometric Information Privacy Act (BIPA) in 2008. It is reported that Governor Pritzker, a Democrat, will sign the new legislation. In doing so, he will solidify the state’s preference for so-called “hard law,” command-and-control style regulatory schemes over soft-law and outright bans that others have chosen when it comes to governing certain controversial AI technology use cases. Ostensibly, the law’s video destruction requirement looks good on paper, but privacy hawks are sure to point out that the law may not address destruction of sensitive behavioral data derived from AI-based video interview systems. Businesses using AI-based video interviewing systems to assess applicants for jobs based in Illinois may want to consider how they handle such “derived from” data as part of their compliance with the new law (assuming it’s signed into law), other applicable laws (such as BIPA and other state’s privacy laws), and their internal data assurance and data privacy policies. How AI Video Interviews Work Job applicants today may be asked by a potential employer to upload a recording of themselves answering a pre-determined set of questions. Applicant’s may also be asked to participate in live video-chat interviews and in-person interviews, both of which may be recorded. In either case, AI-based video interviewing technologies can help companies identify micro facial expressions and vocal intonations in those recordings, ones that might escape a human interviewer’s perception. Combined with other features about a candidate (for example, demographic information submitted on a job application), an AI system can help HR professionals make a binary hire/no-hire decision using a high-dimensional feature set unique to each candidate. As shown in Figure 1, such a system may take audio-video data collected during a job candidate’s interview and processes the audio (speech) into text using automated speech recognition (ASR) algorithms to create a transcript. The ASR algorithm can do this by extracting features from an acoustic image (a 2-dimentional image, time sliced into, say, 10 millisecond frames) corresponding to input sound. For example, a spectral image of the sound corresponding to the utterance “hello” may be processed by an ASR algorithm to generate the phonetic letters “h-e-l-o” and then the word “h-e-l-l-o.” Once the sound has been converted into text, a machine learning (ML) model (such as a cloud-based convolutional or recurrent neural network) can be used to classify (label) all the text according to its semantic and/or sentiment characteristics for purposes of estimating the speaker’s affective state. The interviewer can then seek insight into the applicant’s verbal responses, degree of emotional engagement, and level of positivity exhibited, which could be compared to other applicants. The ML algorithms may be trained, for example, using available speech datasets (labeled data).  As shown in Figure 1, data derived from the original audio-video recording may be stored in one or more databases. FIG. 1. Basic audio-video data work flow beginning with extracting features from time-series data (Xt), which may be stored and then used as input into a ML model to produce discrete labels (classifications) over time (Yt) (e.g., confusion/joy/surprise, or simply discrete binary hire/no hire). On the video side, the AI system may pre-processes the video data, if necessary, to align the speaker’s face toward a face-forward position so that all the traditional facial landmarks may be tagged. The “reconstructed” face and landmarks data may be stored. Then, a ML model (e.g., convolutional neural network) trained on face data sets can be used to classify time-series image slices for purposes of estimating the speaker’s current and changing affect state, such as classifying the candidate’s emotional state over time as “joy” or “confusion” or some other emotion during an interview, and the degree of those emotional states, which again may be compared to others. That assessed data may be stored for future use. Data collected during video interviews is considered biometric information, but, as described above, it also reflects a person’s behavioral traits, making the data itself, and algorithmic assessments of the data, arguably some of the most sensitive from a privacy perspective. The Artificial Intelligence Video Interview Act The Illinois law would place restrictions on businesses that (1) request applicants to agree to have their job interview recorded and (2) that use an artificial intelligence technology to process that recording to assess the candidate’s fitness for the job. A regulated business would be required to do the following when considering an applicant for positions based in Illinois before asking applicants to submit video interviews: (1) Notify each applicant before the interview that artificial intelligence may be used to analyze the applicant’s video interview and consider the applicant’s fitness for the position. (2) Provide each applicant with information before the interview explaining how the artificial intelligence works and what general types of characteristics it uses to evaluate applicants. (3) Obtain, before the interview, consent from the applicant to be evaluated by the artificial intelligence program as described in the information provided. An employer may not use artificial intelligence to evaluate applicants who have not consented to the use of artificial intelligence analysis. The Illinois law would prohibit an employer from sharing an applicant’s video, except with persons whose expertise or technology is necessary in order to evaluate an applicant’s fitness for a position. Upon request from the applicant, the law would require employers, within 30 days after receipt of the request, to delete an applicant’s recorded interview and instruct any other persons who received copies of the applicant video interview to also delete the video, including all electronically generated backup copies. The law would require such other persons to comply with the employer’s instructions. Leaving Personal Data Crumbs Behind Other than algorithmic decision making systems (not all of which employ an AI technology), AI use cases involving the collection, storage, and processing of face data have been targeted by lawmakers over most other AI use cases. This focus is driven in large part by privacy concerns surrounding human face data, and often results in strict measures surrounding collection, storage, and use of the data. Notably, the Illinois AI video interview law only requires deletion of an applicant’s video recording by a business and third parties that received the recording, upon request from the applicant. The law does not explicitly state that feature (vector) data extracted from the video, or the AI model’s output (i.e., “derived data”), has to be deleted, though it’s certainly possible that future court interpretations of the term “video interviews” and/or future implementing regulations could very well expand what it means to destroy a video. Absent such guidance, however, businesses that use AI-based video interviews may wish to consider developing clearly-defined terms of service and/or privacy policies that address how job interview audio-video recordings will be used, including how related derived data from recordings will be used, as part of the company’s notification and consent policies. While a business must delete job interview video recordings under the new law if asked by applicants, it may have reasons for wanting to retain feature data sets and model outputs, for example in the case of no-hire decisions (for purposes of potential litigation). In those instances where an applicant who agreed to an AI-processed video interview is hired for a job in Illinois, a business may need to consider BIPA’s requirements related to data derived from a biometric identifier that may apply to the new employee. A business that uses AI-based video interviews should also consider the different ways their systems may be deployed so that it may appropriately respond to an applicant’s video destruction request. For example, as reflected in Figure 2, an AI system may involve the interviewee’s laptop camera located in Texas, a video recording saved at the company’s server in Illinois, video data sent to and stored at a cloud server in Virginia for processing by a ML algorithm, and a company’s HR personnel accessing the feature data set and model output via a desktop application running on a computer in California, all related to a job “based in” Illinois. To that end, businesses that need to comply with the new law may need to track where the video recording and derived data are stored, and may also want to revisit the terms of service they agreed to with third parties who provide cloud-based AI platform services that process their job applicant interview videos to ensure that those terms of service (and third party privacy policies) are consistent with what the business is telling job candidates about video and data destruction.   FIG. 2. Hypothetical video interview scenario: Illinois (Company’s location and on-prem web/data servers); Texas (job seeker uses browser-based UI for video interview served from Co.’s data server); Virginia (cloud server for ML video data processing; local storage or sends to Co.’s database); California (company’s HR accesses video and output data on cloud platform and Co.’s database)   As lawmakers and stakeholders continue to scrutinize AI technology use cases, especially as they raise concerns about privacy, disparate impacts, and other civil rights concerns, regulated businesses that make or use AI technologies may need to observe where a law’s data handling and destruction requirements do not address all possible data privacy issues. In doing so, the company can decide how best to address gaps in relevant laws and regulations as part of its risk mitigation processes. Read more »
  • California Court Agrees With Autonomous Driving Company’s Source Code Misappropriation Claims, Issues Preliminary Injunction
    In WeRide Corp. v. Huang et al., the U.S. District Court for the Northern District of California issued an order granting a motion for a preliminary injunction and expedited source code discovery.  In doing so, the court concluded that plaintiff’s source code for Level 4 highly autonomous vehicles (HAVs) was a protected trade secret under the federal Defend Trade Secrets Act (DTSA) and California’s Uniform Trade Secrets Act (Cal-UTSA).  The court also concluded that plaintiff was likely to succeed on the merits of its trade secret misappropriation claim against one of its former employees and a China-based company that the employee went to work for.  WeRide Corp. v. Huang, slip. op. No. 5:18-cv-07233-EJD (N.D. Cal. April 1, 2019).  While some of the defendants’ alleged actions–including, copying files, wiping hard drives, placing radar sensors in the same location as plaintiff, among others–led to the court’s decision, WeRide nevertheless illustrates difficulties companies may face when trying to rely on trade secrets alone to protect their technologies in highly competitive and emerging markets like autonomous vehicles. Trade secrets are information that (1) derives independent economic value, actual or potential, from not being generally known to, or readily ascertainable by other people who can obtain economic value from its disclosure or use, and (2) is subject to reasonable efforts to maintain its secrecy.  See 18 U.S.C. § 1839(3); Cal. Civ. Code § 3426.1(d).  Source code can be a trade secret.  See Altavion, Inc. v. Konica Minolta Sys. Lab., Inc., 226 Cal. App. 4th 26, 60 (2014). The source code at issue in WeRide included plaintiff’s code for high definition mapping, sensor fusion-based localization, and state machines.  High definition mapping, the court summarized, refers to electronic maps formed as test vehicles repeatedly traverse roads collecting sensor data; sensor fusion-based localization refers to combining (fusing) data from the vehicle’s many sensors to pinpoint the vehicle’s location in a mapped area; and state machines are machine learning and other algorithmic-based decision systems defining what actions a vehicle should take given its current state.  The court recognized that these and other functions of autonomous vehicles are based on complex artificial intelligence technologies such as deep learning algorithms that require significant time and resources to develop.  Some autonomous vehicle companies, the court noted, invest significant sums of money and take weeks to develop just the basic software needed to begin road tests. Plaintiff claimed that its source code algorithms were trade secrets protectable under federal and state law, that it had invested heavily in developing its source code, maintained it as confidential and proprietary, and the code gave it an advantage over its competitors.  The court agreed, concluding that plaintiff had taken reasonable measures to maintain the secrecy of its source code, which had value in being secret and not generally known, and thus plaintiff was likely to succeed at showing the source code met the definition of “trade secrets” under the DTSA and Cal-UTSA. In particular, the court found that plaintiff controlled access to its offices, encrypted its source code, restricted access to the code base to employees through the use of usernames and passwords authentication (whether accessed on-site or off-site via a virtual private network), and permitted decryption only with the use of a unique username, password, and possession of a token.  The court also found that plaintiff required employees to sign an agreement stating that they would “hold in confidence and not disclose or, except within the scope of [their] employment, use any Proprietary Information,” including “business, technical and financial information. . .[the employee] develop[s], learn[s], or obtain[s] during the term of [their] employment that relates to Company or the business or demonstrably anticipated business of Company or that are received by or for Company.” Defendants argued that much of plaintiff’s source code was derived from available open sources, that published articles described the general functionality of plaintiff’s alleged trade secrets, and that third parties had developed code that performed essentially the same function as plaintiff’s code and thus plaintiff’s code was readily available in the public.  The court dismissed those arguments, pointing out that plaintiff’s trade secret was not in its source code’s functionality, but in the source code itself, which it had maintained as a secret giving the plaintiff a competitive advantage. Both the DTSA and Cal-UTSA, the court wrote, require a plaintiff to demonstrate that a defendant misappropriated a trade secret and the defendant’s conduct damaged the plaintiff.  See 18 U.S.C. § 1839(5) and Cal. Civ. Code § 3426.1(b)).  “Misappropriation” means the acquisition of a trade secret through means that the party knew or should have known were improper, such as theft or breach of a duty to maintain secrecy. 18 U.S.C. § 1839(5); Cal. Civ. Code § 3426.1(b). A party may be liable for misappropriation if they knew or had reason to know that a trade secret it uses was derived from a person who acquired it through improper means. 18 U.S.C. § 1839(5); Cal. Civ. Code § 3426.1(b). In WeRide, the court noted that one of the former employee-defendants did not refute taking plaintiff’s source code by copying it to a personal storage device after deciding to leave plaintiff’s employment, and in fact admitted to doing so while also acknowledging that the code was proprietary to the plaintiff.  While the court considered the defendant’s argument that plaintiff had not demonstrated actual use by the defendant of the source code, the court pointed out that “direct evidence of misappropriation is rare,” BladeRoom Grp. Ltd. v. Emerson Elec. Co., 331 F. Supp. 3d 977, 984 (N.D. Cal. 2018), and that plaintiff was not required to prove its case at the preliminary injunction stage.  Here, plaintiff met the high standard of showing it was likely to succeed on the merits. As to the corporate defendant, the court accepted plaintiff’s expert opinion that “it would have been impossible [for defendant] to independently develop the technology for [detecting a pedestrian about to cross the street at a crosswalk and passing a slower vehicle] in the 10 weeks between [defendant’s] last day at WeRide and the recruitment [of another of plaintiff’s employees] event.”  An implausibly fast development of technology can contribute to finding misappropriation, the court wrote (citing precedent).  Moreover, plaintiff’s expert opined that the placement of defendant’s radar component on test vehicles was the same as plaintiff’s location, which is not a typical placement in autonomous vehicles but would be a desirable location if one were using plaintiff’s source code.  According to the court, the plaintiff was likely to succeed on the merits against the corporate defendant. (The court also considered the likelihood plaintiff would prevail on the merits of its defamation, intentional interference with prospective economic advantage, breach of contract, and breach of fiduciary duty claims, a discussion of which is not provided here). With regard to the question of irreparable harm, the court, citing Waymo LLC v. Uber Technologies, Inc., 2017 WL 2123560, at *9 (N.D. Cal. May 15, 2017), found that plaintiff’s claimed harms were neither speculative nor conclusory.  As in Waymo, the plaintiff was likely to have its market position set back without an injunction, whereas defendants would receive an unfair boost in part because possessing plaintiff’s source code would improve its ability to attract investors and talented engineers from competitors, including from plaintiff.  Moreover, without an injunction, disclosure of the plaintiff’s trade secret among defendants increased the risk of broader disclosure to, for example, defendants’ colleagues and other employees. Regarding the hardship balancing test, the court found that the test weighed heavily in the plaintiff’s favor due to the “serious harms” that plaintiff faced absent an injunction.  The court was not persuaded by defendants’ argument that revealing its source code to the plaintiff (as part of expedited discovery) would alter the status quo and would probably violate China secrecy law.  Those concerns, the court noted, could be addressed by a suitable protective order restricting production of defendant’s source code to plaintiff’s attorneys on an eyes-only basis. On the question of public interests, the Court agreed with plaintiff that an injunction would be in the public interest because the public has a strong interest in protecting intellectual property rights and would also promote fair and lawful competition in an emerging market. Despite plaintiff prevailing on its preliminary injunction motion, WeRide illustrates some of the difficulties companies face when relying on trade secrets alone to protect software in highly-competitive markets and why companies sometimes chose patents to also protect their technologies.  When a tight job market leads to frequent moves by employees, the movement increases the risk that trade secret information will spread among competitors.  While most former employees never intend to disclose their previous employer’s secrets, their experiences and trade craft learned from previous jobs may inevitably (and innocently) find its way into the work they perform for new employers, even in cases where they signed confidentiality and non-disclosure agreements with former employers.  After all, there are only so many ways to, for example, read and process sensor data, create data frames, or write def functions in Python to handle particular computations.  While each unintentional act of using a former employer’s trade secret source code may not fall within the “knew or should have known were improper” definition of misappropriation (and clearly do not involve the level of deception alleged in WeRide), the incremental use of trade secret information may be death of the secret by a thousand tiny cuts.  And once the secret is out in the market, the company that previously held those secrets may be irreparably harmed without a sufficient basis to bring a cause of action.  See FMC Corp. v. Taiwan Tainan Giant Indus. Co., 730 F.2d 61, 63 (2d Cir. 1984) (“A trade secret once lost is, of course, lost forever.”). That is not to say that seeking patent protection is always a better alternative to trade secrets.  The quid pro quo of obtaining a patent is full disclosure of one’s technology, including, in the case of autonomous driving systems, how the system works.  Since patenting software today may be difficult in view of published open source code, published articles (in this case, articles about autonomous vehicle technology), and third-party software with similar functionality known to the public, a party seeking patent protection could reveal how its system works in a patent application but end up without suitable patent protection (or without any patent at all).  As noted by the court in WeRide, the disclosure could also prevent the assertion of trade secrets over aspects of the disclosing party’s technology.  See Comet Technologies USA, Inc. v. Beuerman, 2018 WL 1990226, at *3 (N.D. Cal. Mar. 15, 2018) (noting that a company’s competitive advantage “would evaporate if the public or its competitors could easily recreate” something protected by a trade secret)).  Accordingly, many companies rely on both trade secrets and patents: they protect their source code as trade secret information (where the code provides a competitive advantage), and they seek patent protection for the functionality that the source code provides. Read more »
  • Civil Litigation Discovery Approaches in the Era of Advanced Artificial Intelligence Technologies
    For nearly as long as computers have existed, litigators have used software-generated machine output to buttress their cases, and courts have had to manage a host of machine-related evidentiary issues, including deciding whether a machine’s output, or testimony based on the output, could fairly be admitted as evidence and to what extent. Today, as litigants begin contesting cases involving aspects of so-called intelligent machines–hardware/software systems endowed with machine learning algorithms and other artificial intelligence-based models–their lawyers and the judges overseeing their cases may need to rely on highly-nuanced discovery strategies aimed at gaining insight into the nature of those algorithms, the underlying source code’s parameters and limitations, and the various possible alternative outputs the AI model could have produced given a set of training data and inputs.  A well-implemented strategy will lead to understanding how a disputed AI system worked and how it may have contributed to a plaintiff’s alleged harm, which is necessary if either party wishes to present an accurate and compelling story to a judge or jury. Parties in civil litigation may obtain discovery regarding any non-privileged matter that is relevant to any party’s claim or defense and that is proportional to the needs of the case, unless limited by a court, taking into consideration the following factors expressed in Federal Rules of Civil Procedure (FRCP) Rule 26(b): The importance of the issues at stake in the action The amount in controversy The parties’ relative access to relevant information The parties’ resources The importance of the discovery in resolving the issues, and Whether the burden or expense of the proposed discovery outweighs its likely benefit. Evidence is relevant to a party’s claim or defense if it tends “to make the existence of any fact that is of consequence to the determination of the action more or less probable that it would be without the evidence.” See Fed. R. Evid. 401.  Even if the information sought in discovery is relevant and proportional, discovery is not permitted where no need is shown. Standard Inc. v. Pfizer Inc., 828 F.2d 734, 743 (Fed. Cir. 1987). Initial disclosures Early in a lawsuit, the federal rules require parties to make initial disclosures involving the “exchange of core information about [their] case.”  ADC Ltd. NM, Inc. v. Jamis Software Corp., slip op. No. 18-cv-862 (D. NM Nov. 5, 2018).  This generally amounts to preliminary identifications of individuals likely to have discoverable information, types and locations of documents, and other information that a party in good faith believes may be relevant to a case, based on each parties’ claims, counterclaims, facts, and various demands for relief set forth in their pleadings.  See FRCP 26(a)(1).  A party failing to comply with initial disclosure rules “is not allowed to use” the information or person that was not disclosed “on a motion, at a hearing, or at a trial, unless the failure was substantially justified or is harmless.”  Baker Hughes Inc. v. S&S Chemical, LLC, 836 F. 3d 554 (6th Cir. 2016) (citing FRCP 37(c)(1)).  In a lawsuit involving an AI technology, individuals likely to have discoverable information about the AI system may include: Data scientists Software engineers Stack engineers/systems architects Hired consultants (even if they were employed by a third party) A company’s data scientists may need to be identified if they were involved in selecting and processing data sets, and if they trained, validated, and tested the algorithms at issue in the lawsuit.  Data scientists may also need to be identified if they were involved in developing the final deployed AI model.  Software engineers, depending on their involvement, may also need to be disclosed if they were involved in writing the machine learning algorithm code, especially if they can explain how parameters and hyperparameters were selected and which measures of accuracy were used.  Stack engineers and systems architects may need to be identified if they have discoverable information about how the hardware and software features of the contested AI systems were put together.  Of course, task and project managers and other higher-level scientists and engineers may also need to be identified. Some local court rules require initial or early disclosures beyond what is required under Rule 26.  See Drone Technologies, Inc. v. Parrot SA, 838 F. 3d 1283, 1295 (Fed. Cir. 2016) (citing US District Court for the Western District of Pennsylvania local rule LPR3.1, requiring, in patent cases, initial disclosure of source code and other documentation… sufficient to show the operation of any aspects or elements of each accused apparatus, product, device, process, method or other instrumentality identified in the claims pled of the party asserting patent infringement…”)) (emphasis added).  Thus, depending on the nature of the relevant AI technology at issue in a lawsuit, and the jurisdiction in which the lawsuit is pending, a party’s initial disclosure burden may involve identifying the location of relevant source code (and who controls it), or they could be required to make source code and other technical documents available for inspection early in a case (more on source code reviews below).  Where the system is cloud-based operable on a machine learning as a service (MLaaS) platform, a party may need to identify the platform service where its API requests are piped. Written discovery requests Aside from the question of damages, in lawsuits involving an AI technology, knowing how the AI system made a decision or took an action may be highly relevant to a party’s case, and thus the party seeking to learn more may want to identify information about at least the following topics, which may be obtained through targeted discovery requests, assuming, as required by FRCP 26(b), the requesting party can justify a need for the information: Data sets considered and used (raw and processed) Software, including earlier and later versions of the contested version Software development process Sensors for collecting real-time observational data for use by the AI model Source code Specifications Schematics Flow charts Formulas Drawings Other documentation A party may seek that information by serving requests for production of document and interrogatories.  In the case of document requests, if permitted by the court, a party may wish to request source code to understand the underlying algorithms used in an AI system, and the data sets used to train the algorithms (if the parties’ relevant dispute turns on a question of a characteristic of the data, e.g., did the data reflect biases? Is it out of date? Is it of poor quality due to labeling errors? etc.).  A party may wish to review software versions and software development documents to understand if best practices were followed.  AI model development often involves trial and error, and thus documentation regarding various inputs used, algorithm architectures selected and de-selected, and the hyperparameters chosen for the various algorithms–anything related to product development–may be relevant and should be requested.  In a lawsuit involving an AI system that uses sensor data (e.g., cameras providing image data to a facial recognition system), a party may want to obtain documentation about the chosen sensor to understand its performance capabilities and limitations. With regard to interrogatories, a party may use interrogatories to ask an opposing party to explain its basis for assertions, made in its pleadings or contentions regarding a challenged AI system, such as: The basis underlying a contention about the foreseeability by a person (either the system’s developer or its end user) of an AI system’s errors The basis for the facts regarding the transparency of the system from the developer’s and/or a user’s perspective The reasonableness of an assertion that a person could foresee the errors made by the AI system The basis underlying a contention that a particular relevant technical standard is applicable to the AI system The nature and extent of the contested AI system’s testing conducted prior to deployment The basis for alleged disparate impacts from an automated decision system The identities and their involvement in making final algorithmic decisions leading to a disparate impact, and how and how much they relied on machine-based algorithmic outputs The modeled feature space used in developing an AI model and its relationship to the primary decision variables at issue (e.g., job promotion qualifications, eligibility for housing assistance) Who makes up the relevant scientific community for the relevant technology and what are the relevant industry standards to apply to the disputed AI system and its output Source code reviews Judges and/or juries are often asked to measure a party’s actions against a standard, which may be defined by one or more objective factors.  In the case of an AI system, judging whether a standard has been met may involve assessing the nature of the underlying algorithm.  Without that knowledge, a party with the burden of proof may only be able to offer evidence of the system’s inputs and results, but would have no information about what happened inside the AI system’s black box.  That may be sufficient when a party’s case rests on a comparison of the system’s result or impact with a relevant standard; but in some cases, understanding the inner workings of the system’s algorithms, and how well they model the real world, could help buttress (or undermine) a party’s case in chief and support (or mitigate) a party’s potential damages.  Thus a source code review may be a necessary component of discovery in some lawsuits. For example, assume a technical standard for a machine learning-based algorithmic decision system requires a minimum accuracy (i.e., recall, precision, and/or f1 score), and the developer’s documentation demonstrates that its model met that standard.  An inspection of the source code, however, might reveal that the “test size” parameter was set too low (compared to what is customary), meaning most of the available data in the data set was used to train the model and the model may suffer from overfitting (and maybe the developer forgot to cross-validate).  A source code review might reveal those potential problems.  a source code review might also reveal which features were used to create the model and how many features were used compared to the number of data observations, both of which might reveal that the developer overlooked an important feature or used a feature that caused the model to reflect an implicit bias in the data. Because of source code’s proprietary and trade secret nature, parties requested to produce their code may resist inspection over concerns about the code getting out into the wild.  The burden falls to the requestor to establish a need and that procedures will safeguard the source code.  Cochran Consulting, Inc. v. Uwatec USA, Inc., 102 F.3d 1224, 1231 (Fed. Cir. 1996) (vacating discovery order pursuant to FRCP 26(b) requiring the production of computer-programming code because the party seeking discovery had not shown that the code was necessary to the case); People v. Superior Court of San Diego County, slip op. Case D073943 (Cal. App. 4th October 17, 2018) (concluding that the “black box” nature of software is not itself sufficient to warrant its production); FRCP 26(c)(1)(G) (a court may impose a protective order for trade secrets specifying how they are revealed). Assuming a need for source code has been demonstrated, the parties will need to negotiate terms of a protective order defining what constitutes source code and how source code reviews are to be conducted.  See Vidillion, Inc. v. Pixalate, Inc., slip. op. No. 2:18-cv-07270 (C.D. Cal. Mar. 22, 2019) (describing terms and conditions for disclosure and review of source code, including production at a secure facility, use of non-networked standalone computers, exclusion of recording media/recording devices by inspectors during review, and handling of source code as exhibits during depositions). In terms of definitions, it is not unusual to define source code broadly, relying on principles of trade secret law, to include things that the producing party believes in good faith are not generally known to others and have significant competitive value such that unrestricted disclosure to others would harm the producing party, and which the producing party would not normally reveal to third parties except in confidence or has undertaken with others to maintain in confidence.  Such things may include: Computer instructions (reflected in, e.g., .jupyter or .py files) Data structures Data schema Data definitions (that can be sharable or expressed in a form suitable for input to an assembler, compiler, translator, or other data processing module) Graphical and design elements (e.g., SQL, HTML, XML, XSL, and SGML files) In terms of procedure, source code inspections are typically conducted at the producing party’s law firm or possibly at the developer’s facility, where the inspection can be monitored to ensure compliance with the court’s protective order.  The inspectors will typically comprise a lawyer for the requesting party along with a testifying expert who should be familiar with multiple programming languages and developer’s tools.  Keeping in mind that the inspection machine will not have access to any network, and no recordable media or recording devices will be allowed in the space where the inspection machine is located, the individuals performing the review will need to ensure they’ve requested all the resources installed locally to facilitate inspection testing, including applications to create virtual servers to simulate remote API calls, if that is an element of the lawsuit.  Thus, the reviewers might request in advance that the inspection machine be loaded with: The above-listed files Relevant data sets A development environment such as a Jupyter notebook or similar application to facilitate opening python or other source code files and data sets. In some cases, it may be reasonable to request a GPU-based machine to create a run-time environment for instances of the AI model to explore how the code operates and how the model handles inputs and makes decisions/takes actions. Depending on the nature of the disputed AI system, the relevant source code may be embedded on hardware devices (e.g., sensors) that the party’s do not have access to.  For example, in a case involving the cameras and/or lidar sensors installed on an autonomous vehicle or as part of a facial recognition system, the party seeking to review source code may need to obtain third-party discovery via a subpoena duces tecum, as discussed below. Subpoenas (Third-Party) Discovery If source code is relevant to a lawsuit, and neither party has access to it, one or both of them may turn to a third party software developer/authorized seller for production of the code, and seek discovery from that entity through a subpoena duces tecum. It is not unusual for third parties to resist production on the basis doing so would be unduly burdensome, but often as likely they will resist production on the basis that its software is protected by trade secrets and/or is proprietary and disclosing it to others would put their business interests at risk.  Thus, the party seeking access to the source code in a contested AI lawsuit should be prepared for discovery motions in the district where the third-party software developer/authorized seller is being asked to comply with a subpoena. A court “may find that a subpoena presents an undue burden when the subpoena is facially overbroad.” Wiwa, 392 F.3d at 818. Courts have found that a subpoena for documents from a non-party is facially overbroad where the subpoena’s document requests “seek all documents concerning the parties to [the underlying] action, regardless of whether those documents relate to that action and regardless of date”; “[t]he requests are not particularized”; and “[t]he period covered by the requests is unlimited.” In re O’Hare, Misc. A. No. H-11-0539, 2012 WL 1377891 at *2 (S.D. Tex. Apr. 19, 2012).  Additionally, FRCP 45(d)(3)(B) provides that, “[t]o protect a person subject to or affected by a subpoena, the court for the district where compliance is required may, on motion, quash or modify the subpoena if it requires: (i) disclosing a trade secret or other confidential research, development, or commercial information.” But, “the court may, instead of quashing or modifying a subpoena, order appearance or production under specified conditions if the serving party: (i) shows a substantial need for the testimony or material that cannot be otherwise met without undue hardship; and (ii) ensures that the subpoenaed person will be reasonably compensated.” FRCP 45(d)(3)(C). Thus, in the case of a lawsuit involving an AI system in which one or more of the parties can demonstrate it/they have a substantial need to understand how the system made a decision or took a particular action, a narrowly-tailored subpoena duces tecum may be used to gain access to the third party’s source code.  To assuage the producing party’s proprietary/trade secret concerns, the third party may seek a court-issued protective order outlining terms covering the source code inspection. Depositions Armed with the AI-specific written discovery responses, document production, and an understanding of an AI system’s source code, counsel should be prepared to ask questions of an opponent’s witnesses, which in turn can help fill gaps in a party’s understanding of the facts relevant to its case.  FRCP 30 governs depositions by oral examination of a party, party witness, or third party to a matter.  In a technical deposition of a fact or party witness, such as a data scientist, machine learning engineer, software engineer, or stack developer, investigating the algorithm behind an AI model will help answer questions about how and why a particular system caused a particular result that is material to the litigation.  Thus, the deposition taker would want to inquire about some of the following issues: Which algorithms were considered? Were they separately tested? How were they tested? Why was the final algorithm chosen? Did an independent third party review the algorithm and model output? With regard to the data set used to create the AI model, the deposition taker will want to explore the following: What data sets were used for training, validation, and testing of the algorithm? How was testing and validation conducted, and were alternatives considered? What sort of exploratory data analysis was performed on the data set (or sets) to assess usability, quality, and implicit bias? Was the data adequate for the domain that the developer was trying to model, and could other data have been used? With regard to the final model, the deposition taker may want to explore the following issues: How old is the model? If it models a time-series (e.g., a model based on historical data that tends to increase over time), has the underlying distribution shifted enough such that the model is now outdated? If newer data were not considered, why? How accurate is the model, and how is accuracy measured? Finally, if written discovery revealed an independent third party reviewed the model before it was deployed, the deposition taker may want to explore the details about the scope of the testing and its results.  If sensors are used as the source for new observational data fed to an AI model, the deposition taker may want to learn why those sensors were chosen, how they operate, their limitations, and what alternative sensors could have been used instead. In an expert deposition, the goal of the deposition shifts to exploring the expert’s assumptions, inputs, applications, outputs, and conclusions for weaknesses.  If an expert prepared an adversarial or counterfactual model to dispute the contested AI system or an opposing expert’s model, a litigator should keep in mind the factors in Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993) and FRE 702, when deposing the expert.  For example, the following issues may need to be explored during the deposition: How was the adversarial or counterfactual modeled developed? Can the expert’s model itself be challenged objectively for reliability? Was the model and technique used subject to peer review and/or publication? What was the model’s known or potential rate of error when applied to facts relevant to the lawsuit? What technical standards apply to the model? Is the model based on techniques or theories that have been generally accepted in the scientific community? Conclusion This post explores a few approaches to fact and expert discovery that litigants may want to explore in a lawsuit where an AI technology is contested, though the approaches here are by no means exhaustive of the scope of discovery that one might need in a particular case. Read more »
  • Congress, States Introduce New Laws for Facial Recognition, Face Data – Part 2
    In Part I, new proposed federal and state laws governing the collection, storage, and use of face (biometric) data in connection with facial recognition technology were described.  If enacted, those new laws would join Illinois’ Biometric Information Privacy Act (BIPA), California’s Consumer Data Privacy Act (CCPA), and Texas’ “biometric identifier” regulations in the governance of face-related data.  It is reasonable for businesses to assume that other state laws and regulations will follow, and with them a shifting legal landscape creating uncertainty and potential legal risks.  A thoughtful and proactive approach to managing the risks associated with the use of facial recognition technology will distinguish those businesses that avoid adverse economic and reputational impacts, from those that could face lawsuits and unwanted media attention. Businesses with a proactive approach to risk management will of course already be aware of the proposed new laws that were described in Part I.  S. 847 (the federal Commercial Facial Recognition Privacy Act of 2019) and H1654-2 (Washington State’s legislative house bill) suggest what’s to come, but biometric privacy laws like those in California, Texas, and Illinois have been around for a while.  Companies that do business in Illinois, for instance, already know that BIPA regulates the collection of biometric data, including face scans, and has created much litigation due to its private right of action provision.  Maintaining awareness of the status of existing and proposed laws will be important for businesses that collect, store, and use face data. At the same time, however, federal governance of AI technologies under the Trump Administration is expected to favor a policy and standards governance approach over a more onerous command-and-control-type regulatory agency rulemaking approach (which the Trump administration often refers to “barriers”).  The takeaway for businesses is that the rulemaking provisions of S. 847 may look quite different if the legislation makes it out of committee and is reconciled with other federal bills, adding to the uncertain landscape. But even in the absence of regulations (or at least regulations with teeth) and the threat of private lawsuits (neither S. 847 nor H1654-2 provide a private right of action for violations), managing risk may require businesses that use facial recognition technology, or that directly or indirectly handle face data, or otherwise use the result of a facial recognition technology, to at least minimally self-regulate.  Those that don’t, or those that follow controversial practices involving monetizing face data at the expense of trust, such as obfuscating transparency about how they use consumer data, are more likely to see a backlash. In most cases, companies handling data already have privacy policies and terms of service (TOS) or end-user licensing agreements that address user data and privacy issues.  Those documents could be frequently reviewed and updated to address face data and facial recognition technology concerns.  Moreover, “camera in use” notices are not difficult to implement in the case of entities that deploy cameras for security, surveillance, or other reasons.  Avoiding legalese and use of vague or uncertain terms in those documents and notices could help reduce risks.  H1654-2 provides that a meaningful privacy notice should include: (a) the categories of personal data collected by the controller; (b) the purposes for which the categories of personal data is used and disclosed to third parties, if any; (c) the rights that consumers may exercise, if any; (d) the categories of personal data that the controller shares with third parties, if any; and (e) the categories of third parties, if any, with whom the controller shares personal data.  In the case of camera notices, prominently displaying the notice is standard, but companies should also be mindful of the differences in S. 847 and H1654-2 concerning notice and implied consent: the former may require notice and separate consent, while the later may provide that notice alone equates to implied consent under certain circumstances. Appropriate risk management also means business entities that supply face data sets to others, for machine learning development, training, and testing purposes, understand the source of its data and the data’s potential inherent biases.  Those businesses will be able to articulate the same to users of the data (who may insist on certain assurances about the data’s quality and utility if not provided on an as-is basis).  Ignoring potential inherent biases in data sets is inconsistent with a proactive and comprehensive risk management strategy. Both S. 847 and H1654-2 refer to a human-in-the-loop review process in certain circumstances, such as in cases where a final decision based on the output of a facial recognition technology may result in a reasonably foreseeable and material physical or financial harm to an end user or if it could be unexpected or highly offensive to a reasonable person.  Although “reasonably foreseeable,” “harm,” and “unexpected or highly offensive” are undefined, a thoughtful approach to managing risk and mitigating damages might consider ways to implement human reviews mindful of federal and state consumer protection, privacy, and civil rights laws that could be implicated absent use of a human reviewer. The White House’s AI technology use policy and S. 847 refer to the National Institute of Standards and Technology (NIST), which could play a large role in AI technology governance.  Learning about NIST’s current standards-setting approach and its AI model evaluation process could help companies seeking to do business with the federal government.  Of course, independent third parties could also evaluate a business’ AI models for bias, problematic data sets, model leakiness, and to identify potential problems that might lead to litigation.  While not every situation may require such extra scrutiny, the ability to recognize and avoid risks might justify the added expense. As noted above, neither S. 847 nor SB 5376 include a private right of action like BIPA, but the new laws could allow for states attorneys general to bring civil actions against violators.  Businesses should consider the possibility of such legal actions, as well as the other potential risks from the use of facial recognition technology and face data collection when assessing the risk factors that must be discussed in certain SEC filings. Above are just a few of the factors and approaches that businesses could consider as part of a risk management approach to the use of facial recognition technology in the face of a changing legal landscape. Read more »
  • Congress, States Introduce New Laws for Facial Recognition, Face Data – Part I
    Companies developing artificial intelligence-based products and services have been on the lookout for laws and regulations aimed at their technology.  In the case of facial recognition, new federal and state laws seem closer than ever.  Examples include Washington State’s recent data privacy and facial recognition bill (SB 5376; recent action on March 6, 2019) and the federal Commercial Facial Recognition Privacy Act of 2019 (S. 847, introduced March 14, 2019).  If enacted, these new laws would join others like Illinois’ Biometric Information Privacy Act (BIPA) and California’s Consumer Privacy Act (CCPA) in governing facial recognition systems and the collection, storage, and use of face data.  But even if those new bills fail to become law, they underscore how the technology will be regulated in the US and suggest, as discussed below, the kinds of litigation risks organizations may confront in the future. What is Face Data and Facial Recognition Technology? Definitions of face data often involve information that can be associated with an identified or identifiable person.  Face data may be supplied by a person (e.g., an uploaded image), purchased from a third party (i.e., a data broker), obtained from publicly-available data sets, or collected via audio-video equipment (e.g., using surveillance cameras). Facial recognition refers to extracting data from a camera’s output signal (still image or video), locating faces in the image data (an object detection process typically done using machine learning algorithms), picking out unique features from the faces that can be used to tell them apart from other people (e.g., facial landmarks), and comparing those features to all the faces of people already known to see if there is a match. Advances in the field of computer vision, including a machine learning technique called convolutional neural networks (ConvNets or CNNs), have turned what used to be a laborious manual process of identifying faces in image data into a highly accurate and automated process performed by machines in near real-time.  Online face image sources such as Facebook, Flickr, Twitter, Instagram, YouTube, news media websites, other websites, as well as face data images collected by government agencies from, among other sources, airport cameras, provide the data used to train and test CNNs. Why are Lawmakers Addressing Facial Recognition? Among the several AI technologies attracting lawmakers’ attention, facial recognition seems to top the list due in part to its rapidly-expanding use, especially in law enforcement, and the civil and privacy rights implications associated with the collection and use of face data, often without consent, by both private and public organizations. From a privacy perspective, Microsoft’s President Brad Smith, writing in 2018, expressed a common refrain by those concerned about facial recognition: unconsented surveillance.  “Imagine a government tracking everywhere you walked over the past month without your permission or knowledge.  Imagine a database of everyone who attended a political rally that constitutes the very essence of free speech. Imagine the stores of a shopping mall using facial recognition to share information with each other about each shelf that you browse and product you buy, without asking you first. This has long been the stuff of science fiction and popular movies – like ‘Minority Report,’ ‘Enemy of the State’ and even ‘1984’ – but now it’s on the verge of becoming possible.” Beyond surveillance, others have expressed concerns about the security of face data. Unlike non-biometric data, which is replaceable (think credit card numbers or passwords), face data represent intimate, unique, and irreplaceable characteristics of a person.  Once hackers have maliciously exfiltrated a person’s face data from a business’ computer system, the person’s privacy is threatened. From a civil rights perspective, known problems with bias in facial recognition systems have been documented.  This issue became a headline in July 2018 when the American Civil Liberties Union (ACLU) reported that a widely-used, commercially-available facial recognition program “incorrectly matched 28 members of Congress, identifying them as other people who have been arrested for a crime.”  The report noted that the members of Congress who were “falsely matched with the mugshot database [] used in the test include Republicans and Democrats, men and women, and legislators of all ages, from all across the country.”  The report also found that the mismatches disproportionately involved members who are people of color, thus raising questions about the accuracy and quality of the tested facial recognition technique, as well as revealing its possible inherent bias. <>  Bias may arise if a face data set used to train a CNN contains predominantly white male face images, for example.  The facial recognition technology using that algorithm may not perform well (or “generalize”) when it’s asked to classify (identify) non-white male faces.  The bias issue has led many to call for the meaningful ethical-based design of AI systems. But even beneficial uses of facial recognition technology and face data collection and use have been criticized, in part because people whose face data are being collected and used are typically not given an opportunity to give their consent (and in many cases, do not even know their face data is being used).  Thus, automatically identifying people in uploaded images (a process called image “tagging”), improving a person’s experience at a public venue, providing access to a private computer system or a physical location, establishing an employee’s working hours and their movements for safety purposes, and personalizing advertisements and newsfeeds displayed on a computer user’s browser, while arguably beneficial uses of face data, are often conducted without a user’s consent (or its access/use is conditioned upon giving consent) and thus criticized. As much as the many concerns about facial recognition may have piqued lawmakers’ interest in regulating face data, legislation like those mentioned above is just as likely to arise because stakeholders and vocal opponents have called for more certainty in the legal landscape.  Microsoft, for one, in 2018 called for regulating facial recognition technology.  “The only effective way to manage the use of technology by a government is for the government proactively to manage this use itself,” Brad Smith wrote, his words clearly directed to Capitol Hill as well as state lawmakers in Olympia.  “And if there are concerns about how a technology will be deployed more broadly across society, the only way to regulate this broad use is for the government to do so. This in fact is what we believe is needed today – a government initiative to regulate the proper use of facial recognition technology, informed first by a bipartisan and expert commission.” Comparing the Washington, DC and Washington State Bills [For a summary of Illinois’ face data privacy law, click here.] If S. 847 becomes law, it would cover any person (other than the federal government, state and local governments, law enforcement agencies, a national security agency, or an intelligence agency) that collects, stores, or processes facial recognition data from still or video images, including any unique attribute or feature of the face of a person that is used by facial recognition technology for the purpose of assigning a unique, persistent identifier or for the unique personal identification of a specific individual.  SB 5376, in contrast, would cover any natural or legal persons which, alone or jointly with others, determines the purposes and means of the processing of personal data by a processor, including personal data from a facial recognition technology.  While the federal bill would not cover government agency use, SB 5376 would condition Washington state and local government agencies, including law enforcement agencies, to conduct ongoing surveillance of specified individuals in public spaces only if such use is in support of law enforcement activities and either (a) a court order has been obtained to permit the use of facial recognition services for that ongoing surveillance, or (b) where there is an emergency involving imminent danger or risk of death or serious physical injury to a person. On the issue of consent, S. 847 would require a business that knowingly uses facial recognition technology to collect face data to obtain from a person affirmative consent (opt-in consent).  To the extent possible, if facial recognition technology is present, a business must provide to the person a concise notice that facial recognition technology is present, and, if contextually appropriate, where the person can find more information about the use by the business of facial recognition technology and documentation that includes general information that explains the capabilities and limitations of the technology in terms that the person is able to understand.  SB 5376 would also require controllers to obtain consent from consumers prior to deploying facial recognition services in physical premises open to the public.  The placement of a conspicuous notice in physical premises that clearly conveys that facial recognition services are being used would constitute a consumer’s consent to the use of such facial recognition services when that consumer enters those premises that have such notice. Under S. 847, obtaining affirmative consent would be effective only if a business makes available to a person a notice that describes the specific practices of the business in terms that persons are able to understand regarding the collection, storage, and use of facial recognition data.  These include reasonably foreseeable purposes, or examples, for which the business collects and shares information derived from facial recognition technology or uses facial recognition technology, and information about the practice of data retention and de-identification, and whether a person can review, correct, or delete information derived from facial recognition technology.  Under SB 5376, processors that provide facial recognition services would be required to provide documentation that includes general information that explains the capabilities and limitations of face recognition technology in terms that customers and consumers can understand. S. 847 would prohibit a business from knowingly using a facial recognition technology to discriminate against a person in violation of applicable federal or state law (presumably civil rights laws, consumer protection laws, and others), repurpose facial recognition data for a purpose that is different from those presented to the person, and share the facial recognition data with an unaffiliated third party without affirmative consent (separate from the opt-in affirmative consent noted above). SB 5376 would prohibit processors of face data that provide facial recognition services from using such facial recognition services by controllers to unlawfully discriminate under federal or state law against individual consumers or groups of consumers. S. 847 would require meaningful human review prior to making any final decision based on the output of facial recognition technology if the final decision may result in a reasonably foreseeable and material physical or financial harm to an end user or may be unexpected or highly offensive to a reasonable person. SB 5376 would require controllers that use facial recognition for profiling must employ meaningful human review prior to making final decisions based on such profiling where such final decisions produce legal effects concerning consumers or similarly significant effects concerning consumers. Decisions producing legal effects or similarly significant effects include, but are not limited to, denial of consequential services or support, such as financial and lending services, housing, insurance, education enrollment, criminal justice, employment opportunities, and health care services. S. 847 would require a regulated business that makes a facial recognition technology available as an online service to make available an application programming interface (API) to enable an independent testing company to conduct reasonable tests of the facial recognition technology for accuracy and bias. SB 5376 would require providers of commercial facial recognition services that make their technology available as an online service for developers and customers to use in their own scenarios must make available an API or other technical capability, chosen by the provider, to enable third parties that are legitimately engaged in independent testing to conduct reasonable tests of those facial recognition services for accuracy and unfair bias. S. 847 would provide exceptions for certain facial recognition technology uses, including product or service designed for personal file management or photo or video sorting or storage, if the facial recognition technology is not used for unique personal identification of a specific individual, as well as uses involving the identification of public figures for journalistic media created for public interest. The law would also provide exceptions for the identification of public figures in copyrighted material for theatrical release, or use in an emergency involving imminent danger or risk of death or serious physical injury to an individual. The law would also provide certain exceptions for certain security applications.   Even so, the noted exceptions would not permit businesses to conduct the mass scanning of faces in spaces where persons do not have a reasonable expectation that facial recognition technology is being used on them. SB 5376 would provide exceptions in the case of complying with federal, state, or local laws, rules, or regulations, or with a civil, criminal, or regulatory inquiry, investigation, subpoena, or summons by federal, state, local, or other governmental authorities.  The law would also provide exemptions to cooperate with law enforcement agencies concerning conduct or activity that the controller or processor reasonably and in good faith believes may violate federal, state, or local law, or to investigate, exercise, or defend legal claims, or prevent or detect identity theft, fraud, or other criminal activity or verify identities.  Other exceptions or exemptions are also provided. Under S. 847, violating aspects of the law would be defined as an unfair or deceptive act or practice under Section 18(a)(1)(B) of the Federal Trade Commission Act (15 USC 57a(a)(1)(B)).  The FTC would regulate the new law and would have authority to assert its penalty powers pursuant to 15 USC 41 et seq.  Moreover, state attorneys general, or any other officer of a state who is authorized by the state to do so, may, upon notice to the FTC, bring a civil action on behalf of state residents if it believes that an interest of the residents has been or is being threatened or adversely affected by a practice by a business covered by the new law that violates one of the law’s prohibitions.  The FTC may intervene in such civil action.  SB 5376 would provide that the state’s attorney general may bring an action in the name of the state, or as parens patriae on behalf of persons residing in the state, to enforce the law. S. 847 would also require the FTC to consult with the National Institute of Standards and Technology (NIST) to promulgate regulations within 180 days after enactment describing basic data security, minimization, and retention standards; defining action that are harmful and highly offensive; expanding the list of exceptions noted above in cases where it is impossible for a business to obtain affirmative consent from, or provide notice to, persons. S. 847 would not preempt tougher state laws covering facial recognition technology and the collection and use of face data, or other state or federal privacy and security laws. In Part II of this post, facial recognition and face data regulation impact for businesses will be discussed. Read more »
  • Government Plans to Issue Technical Standards For Artificial Intelligence Technologies
    On February 11, 2019, the White House published a plan for developing and protecting artificial intelligence technologies in the United States, citing economic and national security concerns among other reasons for the action.  Coming two years after Beijing’s 2017 announcement that China intends to be the global leader in AI by 2030, President Trump’s Executive Order on Maintaining American Leadership in Artificial Intelligence lays out five principles for AI, including “development of appropriate technical standards and reduc[ing] barriers to the safe testing and deployment of AI technologies in order to enable the creation of new AI-related industries and the adoption of AI by today’s industries.”  The Executive Order, which lays out a framework for an “American AI Initiative” (AAII), tasks the White House’s National Science and Technology Council (NSTC) Select Committee on Artificial Intelligence, established in 2018, with identifying federal government agencies to develop and implement the technical standards (so-called “implementing agencies”). Unpacking the AAII’s technical standards principle suggests two things.  First, federal governance of AI under the Trump Administration will favor a policy and standards governance approach over a more onerous command-and-control-type regulatory agency rulemaking approach leading to regulations (which the Trump administration often refers to as “barriers”).  Second, no technical standards will be adopted that stand in the way of the development or use of AI technologies at the federal level if they impede economic and national security goals. So what sort of technical standards might the Select Committee on AI and the implementing agencies come up with?  And how might those standards impact government agencies, government contractors, and even private businesses from a legal perspective? The AAII is short on answers to those questions, and we won’t know more until at least August 2019 when the Secretary of Commerce, through the Director of the National Institute of Standards and Technology (NIST), is required by the AAII to issue a plan “for Federal engagement in the development of technical standards and related tools in support of reliable, robust, and trustworthy systems that use AI technologies.”  Even so, it is instructive to review some relevant technical standards and related legal issues in anticipation of what might lie ahead for the United States AI industry. A survey of technical standards used across a spectrum of different industries shows that they can take many different forms, but often they classify as prescriptive or performance-based.  Pre-determined prescriptive metrics may specify requirements for things like accuracy, quality, output, materials, composition, and consumption.  In the AI space, a prescriptive standard could involve a benchmark for classification accuracy (loss or error) using a standardized data set (i.e., how well does the system work), or a numerical upper limit on power consumption, latency, weight, and size.  Prescriptive standards can be one-size-fits-all, or they can vary. Performance-based standards describe practices (minimum, best, commercially reasonable, etc.) focusing on results to be achieved.  In many situations, the performance-based approach provides more flexibility compared to using prescriptive standards.  In the context of AI, a performance-based standard could require a computer vision system to detect all objects in a specified field of view, and tag and track them for a period of time.  How the developer achieves that result is less important in performance-based standards. Technical standards may also specify requirements for the completion of risk assessments to numerically compare an AI system’s expected benefits and impacts to various alternatives.  Compliance with technical standards may be judged by advisory committees who follow established procedures for independent and open review.  Procedures may be established for enforcement of technical standards when non-compliance is observed.  Depending on the circumstances, technical standards may be published for the public to see or they may be maintained in confidence (e.g., in the case of national security).  Technical standards are often reviewed on an on-going or periodic basis to assess the need for revisions to reflect changes in previous assumptions (important in cases when rapid technological improvements or shifts in priorities occur). Under the direction of the AAII, the White House’s Select Committee and various designated implementing agencies could develop new technical standards for AI technologies, but they could also adopt (and possibly modify) standards published by others.  The International Organization for Standards (ISO), American National Standards Institute (ANSI), National Institute of Standards and Technology (NIST), and the Institute for Electronics and Electrical Engineers (IEEE) are among the few private and public organizations that have developed or are developing AI standards or guidance.  Individual state legislatures, academic institutions, and tech companies have also published guidance, principles, and areas of concern that could be applicable to the development of technical and non-technical standards for AI technologies.  By way of example, the ISO’s technical standard for “big data” architecture includes use cases for deep learning applications and large scale unstructured data collection.  The Partnership on AI, a private non-profit organization whose board consists of representatives from IBM, Google, Microsoft, Apple, Facebook, Amazon, and others, has developed what it considers “best practices” for AI technologies. Under the AAII, the role of technical standards, in addition to helping build an AI industry, will be to “minimize vulnerability to attacks from malicious actors and reflect Federal priorities for innovation, public trust, and public confidence in systems that use AI technologies.”  It is hard to imagine a purely technical standard addressing trust and confidence, though a non-technical standards-setting process could address those issues by, for example, introducing measures related to fairness, accountability, and transparency.  Consider the example of delivering AI-based healthcare services at Veterans Administration facilities, where trust and confidence could be reflected in non-technical standards that provide for the publication of clear, understandable explanations about how an AI system works and how it made a decision that affected a patent’s care.  Addressing trust and confidence could also be reflected in requirements for open auditing of AI systems.  The IEEE’s “Ethically Aligned Design” reference considers these and related issues. Another challenge in developing technical standards is to avoid incorporating patented technologies “essential” to the standards adopted by the government, or if unavoidable, to develop rules for disclosure and licensing of essential patents.  As the court in Apple v. Motorola explained, “[s]ome technological standards incorporate patented technology. If a patent claims technology selected by a standards-setting organization, the patent is called an ‘essential patent.’ Many standards-setting organizations have adopted rules related to the disclosure and licensing of essential patents. The policies often require or encourage members of the organization to identify patents that are essential to a proposed standard and to agree to license their essential patents on fair, reasonable and nondiscriminatory terms to anyone who requests a license. (These terms are often referred to by the acronyms FRAND or RAND.)  Such rules help to insure that standards do not allow the owners of essential patents to abuse their market power to extort competitors or prevent them from entering the marketplace.”  See Apple, Inc. v. Motorola Mobility, Inc., 886 F. Supp. 2d 1061 (WD Wis. 2012).  Given the proliferation of new AI-related US patents issued to tech companies in recent years, the likelihood that government technical standards will encroach on some of those patents seems high. For government contractors, AI technical standards could be imposed on them through the government contracting process.  A contracting agency could incorporate new AI technical standards by reference in government contracts, and those standards would flow through to individual task and work orders performed by contractors under those contracts.  Thus, government contractors would need to review and understand the technical standards in the course of executing a written scope of work to ensure they are in compliance.  Sponsoring agencies would likely be expected to review contractor deliverables to measure compliance with applicable AI technical standards.  In the case of non-compliance, contracting officials and their sponsoring agency would be expected to deploy their enforcement authority to ensure problems are corrected, which could include monetary penalties assessed against contractors. Although private businesses (i.e., not government contractors) may not be directly affected by agency-specific technical standards developed under the AAII, customers of those private businesses could, absent other relevant or applicable technical standards, use the government’s AI technical standards as a benchmark when evaluating a business’s products and services.  Moreover, even if federal AI-based technical standards do not directly apply to private businesses, there is certainly the possibility that Congress could legislatively mandate the development of similar or different technical and non-technical standards and other requirements applicable to a business’ AI technologies sold and used in commerce. The president’s Executive Order on AI has turned an “if” into a “when” in the context of federal governance of AI technologies.  If you are a stakeholder, now is a good time to put resources into closely monitoring developments in this area to prepare for possible impacts. Read more »
  • Washington State Seeks to Root Out Bias in Artificial Intelligence Systems
    The harmful effects of biased algorithms have been widely reported.  Indeed, some of the world’s leading tech companies have been accused of producing applications, powered by artificial intelligence (AI) technologies, that were later discovered to exhibit certain racial, cultural, gender, and other biases.  Some of the anecdotes are quite alarming, to say the least.  And while not all AI applications have these problems, it only takes a few concrete examples before lawmakers begin to take notice. In New York City, lawmakers began addressing algorithmic bias in 2017 with the introduction of legislation aimed at eliminating bias from algorithmic-based automated decision systems used by city agencies.  That effort led to the establishment of a Task Force in 2018 under Mayor de Blasio’s office to examine the issue in detail.  A report from the Task Force is expected this year. At the federal level, an increased focus by lawmakers on algorithmic bias issues began in 2018, as reported previously on this website (link) and elsewhere.  Those efforts, by both House and Senate members, focused primarily on gathering information from federal agencies like the FTC, and issuing reports highlighting the bias problem.  Expect congressional hearings in the coming months. Now, Washington State lawmakers are addressing bias concerns.  In companion bills SB-5527 and HB-1655, introduced on January 23, 2019, lawmakers in Olympia drafted a rather comprehensive piece of legislation aimed at governing the use of automated decision systems by state agencies, including the use of automated decision-making in the triggering of automated weapon systems.  As many in the AI community have discussed, eliminating algorithmic-based bias requires consideration of fairness, accountability, and transparency, issues the Washington bills appear to address.  But the bills also have teeth, in the form of a private right of action allowing those harmed to sue. Although the aspirational language of legislation often only provides a cursory glimpse at how stakeholders might be affected under a future law, especially in those instances where, as here, an agency head is tasked with producing implementing regulations, an examination of automated decisions system legislation like Washington’s is useful if only to understand how  states and the federal government might choose to regulate aspects of AI technologies and their societal impacts. Purpose and need for anti-bias algorithm legislation According to the bills’ sponsors, in Washington, automated decision systems are rapidly being adopted to make or assist in core decisions in a variety of government and business functions, including criminal justice, health care, education, employment, public benefits, insurance, and commerce.  These systems, the lawmakers say, are often deployed without public knowledge and are unregulated.  Their use raises concerns about due process, fairness, accountability, and transparency, as well as other civil rights and liberties.  Moreover, reliance on automated decision systems without adequate transparency, oversight, or safeguards can undermine market predictability, harm consumers, and deny historically disadvantaged or vulnerable groups the full measure of their civil rights and liberties. Definitions, Prohibited Actions, and Risk Assessments The new Washington law would define “automated decision systems” as any algorithm, including one incorporating machine learning or other AI techniques, that uses data-based analytics to make or support government decisions, judgments, or conclusions.  The law would distinguish “automated final decision system,” which are systems that make “final” decisions, judgments, or conclusions without human intervention, and “automated support decision system,” which provide information to inform the final decision, judgment, or conclusion of a human decision maker. Under the new law, in using an automated decision system, an agency would be prohibited from discriminating against an individual, or treating an individual less favorably than another, in whole or in part, on the basis of one or more factors enumerated in RCW 49.60.010.  An agency would be outright prohibited from developing, procuring, or using an automated final decision system to make a decision impacting the constitutional or legal rights, duties, or privileges of any Washington resident, or to deploy or trigger any weapon. Both versions of the bill include lengthy provisions detailing algorithmic accountability reports that agencies would be required to produce and publish for public comment.  Among other things, these reports must include clear information about the type or types of data inputs that a technology uses; how that data is generated, collected, and processed; and the type or types of data the systems are reasonably likely to generate, which could help reveal the degree of bias inherent in a system’s black box model.  The accountability reports also must identify and provide data showing benefits; describe where, when, and how the technology is to be deployed; and identify if results will be shared with other agencies. An agency that deploys an approved report would then be required to follow conditions that are set forth in the report. Although an agency’s choice to classify its automated decision system as one that makes “final” or “support” decisions may be given deference by courts, the designations are likely to be challenged if the classification is not justified.  One reason a party might challenge designations is to obtain an injunction, which may be available in the case where an agency relies on a final decision made by an automated decision system, whereas an injunction may be more difficult to obtain in the case of algorithmic decisions that merely support a human decision-maker.  The distinction between the two designations may also be important during discovery, under a growing evidentiary theory of “machine testimony” that includes cross-examining machines witnesses by gaining access to source code and, in the case of machine learning models, the developer’s data used to train a machine’s model.  Supportive decision systems involving a human making a final decision may warrant a different approach to discovery. Conditions impacting software makers Under the proposed law, public agencies that use automated decision systems would be required to publicize the system’s name, its vendor, and the software version, along with the decision it will be used to make or support.  Notably, a vendor must make its software and the data used in the software “freely available” before, during, and after deployment for agency or independent third-party testing, auditing, or research to understand its impacts, including potential bias, inaccuracy, or disparate impacts.  The law would require any procurement contract for an automated decision system entered into by a public agency to include provisions that require vendors to waive any legal claims that may impair the “freely available” requirement.  For example, contracts with vendors could not contain nondisclosure impairment provisions, such as those related to assertions of trade secrets. Accordingly, software companies who make automated decision systems will face the prospect of waiving proprietary and trade secret rights and opening up their algorithms and data to scrutiny by agencies, third parties, and researchers (presumably, under terms of confidentiality).  If litigation were to ensue, it could be difficult for vendors to resist third-party discovery requests on the basis of trade secrets, especially if information about auditing of the system by the state agency and third-party testers/researchers is available through administrative information disclosure laws.  A vendor who chooses to reveal the inner workings of a black box software application without safeguards should consider at least financial, legal, and market risks associated with such disclosure. Contesting automated decisions and private right of action Under the proposed law, public agencies would be required to announce procedures how an individual impacted by a decision made by an automated decision system can contest the decision.  In particular, any decision made or informed by an automated decision system will be subject to administrative appeal, an immediate suspension if a legal right, duty, or privilege is impacted by the decision, and a potential reversal by a human decision-maker through an open due process procedure.  The agency must also explain the basis for its decision to any impacted individual in terms “understandable” to laypersons including, without limitation, by requiring the software vendor to create such an explanation.  Thus, vendors may become material participants in administrative proceedings involving a contested decision made by its software. In addition to administrative relief, the law would provide a private right of action for injured parties to sue public agencies in state court.  In particular, any person who is injured by a material violation of the law, including denial of any government benefit on the basis of an automated decision system that does not meet the standards of the law, may seek injunctive relief, including restoration of the government benefit in question, declaratory relief, or a writ of mandate to enforce the law. For litigators representing injured parties in such cases, dealing with evidentiary issues involving information produced by machines would likely follow Washington judicial precedent in areas of administrative law, contracts, tort, civil rights, the substantive law involving the agency’s jurisdiction (e.g., housing, law enforcement, etc.), and even product liability.  In the case of AI-based automated decision systems, however, special attention may need to be given to the nuances of machine learning algorithms to prepare experts and take depositions in cases brought under the law.  Although the aforementioned algorithmic accountability report could be useful evidence for both sides in an automated decision system lawsuit, merely understanding the result of an algorithmic decision may not be sufficient when assessing if a public agency was thorough in its approach to vetting a system.  Being able to describe how the automated decision system works will be important.  For agencies, understanding the nuances of the software products they procure will be important to establish that they met their duty to vet the software under the new law. For example, where AI machine learning models are involved, new data, or even previous data used in a different way (i.e., a different cross-validation scheme or a random splitting of data into new training and testing subsets), can generate models that produce slightly different outcomes.  While small, the difference could mean granting or denying agency services to constituents.  Moreover, with new data and model updates comes the possibility of introducing or amplifying bias that was not previously observed.  The Washington bills do not appear to include provisions imposing an on-going duty on vendors to inform agencies when bias or other problems later appear in software updates (though it’s possible the third party auditors or researchers noted above might discover it).  Thus, vendors might expect agencies to demand transparency as a condition set forth in acquisition agreements, including software support requirements and help with developing algorithmic accountability reports.  Vendors might also expect to play a role in defending against claims by those alleging injury, should the law pass.  And they could be asked to shoulder some of the liability either through indemnification or other means of contractual risk-shifting to the extent the bills add damages as a remedy. Read more »
  • What’s in a Name? A Chatbot Given a Human Name is Still Just an Algorithm
    Due in part to the learned nature of artificial intelligence technologies, the spectrum of things that exhibit “intelligence” has, in debates over such things, expanded to include certain advanced AI systems.  If a computer vision system can “learn” to recognize real objects and make decisions, the argument goes, its ability to do so can be compared to that of humans and thus should not be excluded from the intelligence debate.  By extension, AI systems that can exhibit intelligence traits should not be treated like mere goods and services, and thus laws applicable to such good and services ought not to apply to them. In some ways, the marketing of AI products and services using names commonly associated with humans, such as “Alexa,” “Sophia,” and “Siri,” buttresses the argument that laws applicable to non-human things should not strictly apply to AI.  For now, however, lawmakers and the courts struggling with practical questions about regulating AI technologies can justifiably apply traditional goods and services laws to named AI systems just as they do to non-named system.  After all, a robot or chatbot doesn’t become more humanlike and less like a man-made product merely because it’s been anthropomorphized.  Even so, when future technological breakthroughs suggest artificial general intelligence (AGI) is on the horizon, lawmakers and the courts will be faced with the challenge of amending laws to account for the differences between AGI systems and today’s narrow AI and other “unintelligent” goods and services.  For now, it’s instructive to consider why the rise in the use of names for AI system is not a good basis for triggering greater attention by lawmakers.  Indeed, as suggested below, other characteristics of AI system may be more useful in deciding when laws need to be amended.  To begin, the recent case of a chatbot named “Erica” is presented. The birth of a new bot In 2016, machine learning developers at Bank of America created a “virtual financial assistant” application called “Erica” (derived from the bank’s name America).  After conducting a search of existing uses of the name Erica in other commercial endeavors, and finding none in connection with a chatbot like theirs, BoA sought federal trademark protection for the ERICA mark in October 2016.  The US Patent and Trademark Office concurred with BoA’s assessment of prior uses and registered the mark on July 31, 2018.  Trademarks are issued in connection with actual uses of words, phrases, and logos in commerce, and in the case of BoA, the ERICA trademark was registered in connection with computer financial software, banking and financial services, and personal assistant software in banking and financial SaaS (software as a service).  The Erica app is currently described as possessing the utility to answer customer questions and make banking easier.  During its launch, BoA used the “she” pronoun when describing the app’s AI and predictive analytics capabilities, ostensibly because the name Erica is a stereotypical female gender name, but also because of the apparent female-sounding voice the app outputs as part of its human-bot interface. One of the existing uses of an Erica-like mark identified by BoA was an instance of “E.R.I.C.A,” which appeared in October 2010 when Erik Underwood, a Colorado resident, filed a Georgia trademark registration application for “E.R.I.C.A. (Electronic Repetitious Informational Clone Application).”  See Underwood v. Bank of Am., slip op., No. 18-cv-02329-PAB-MEH (D. Colo. Dec. 19, 2018).  On his application, Mr. Underwood described E.R.I.C.A. as “a multinational computer animated woman that has slanted blue eyes and full lips”; he also attached a graphic image of E.R.I.C.A. to his application.  Mr. Underwood later sought a federal trademark application (filed in September 2018) for an ERICA trademark (without the separating periods).  At the time of his lawsuit, his only use of E.R.I.C.A. was on a searchable movie database website. In May 2018, Mr. Underwood sent a cease-and-desist letter to BoA regarding BoA’s use of Erica, and then filed a lawsuit in September 2018 against the bank alleging several causes of action, including “false association” under § 43(a) of the Lanham Act, 15 U.S.C. § 1125(a)(1)(A).  Section 43(a) states, in relevant part, that any person who, on or in connection with any goods or services, uses in commerce a name or a false designation of origin which is likely to cause confusion, or to cause mistake, or to deceive as to the affiliation, connection, or association of such person with another person, or as to the origin, sponsorship, or approval of his or her goods, services, or commercial activities by another person, shall be liable in a civil action by a person who believes that he or she is likely to be damaged by such act.  In testimony, Mr. Underwood stated that the E.R.I.C.A. service mark was being used in connection with “verbally tell[ing] the news and current events through cell phone[s] and computer applications” and he described plans to apply an artificial intelligence technology to E.R.I.C.A.  Mr. Underwood requested the court enter a preliminary injunction requiring BoA to cease using the Erica name. Upon considering the relevant preliminary injunction factors and applicable law, the District Court denied Mr. Underwood’s request for an injunction on several grounds, including the lack of relevant uses of E.R.I.C.A. in the same classes of goods and services that BoA’s Erica was being used in. Giving AI a persona may boost its economic value and market acceptance Not surprisingly, the District Court’s preliminary injunction analysis rested entirely on perception and treatment of the Erica and E.R.I.C.A. systems as nothing more than services, something neither party disputed or challenged.  Indeed, each party’s case-in-chief depended on their convincing the court that their applications fit squarely in the definition of goods and services despite the human-sounding names they chose to attach to them.  The court’s analysis, then, illuminated one of the public policies underlying laws like the Lanham Act, which is the protection of the economic benefits associated with goods and services created by people and companies.  The name Erica provides added economic value to each party’s creation and is an intangible asset associated with their commercial activities. The use of names has long been found to provide value to creators and owners, and not just in the realm of hardware and software.  Fictional characters like “Harry Potter,” which are protected under copyright and trademark laws, can be intellectual assets having tremendous economic value.  Likewise, namesake names carried over to goods and services, like IBM’s “Watson”–named after the company’s first CEO, John Watson–provide real economic benefits that might not have been achieved without a name, or even with a different name.  In the case of humanoid robots, like Hanson Robotics’ “Sophia,” which is endowed with aspects of AI technologies and was reportedly granted “citizenship” status in Saudi Arabia, certain perceived and real economic value is created by distinguishing the system from all other robots by using a real name (as compared to, for example, a simple numerical designation). On the other end of the spectrum are names chosen for humans, the uses of which are generally unrestricted from a legal perspective.  Thus, naming one’s baby “Erica” or even “Harry Potter” shouldn’t land a new parent in hot water.  At the same time, those parents aren’t able to stop others from using the same names for other children.  Although famous people may be able to prevent others from using their names (and likenesses) for commercial purposes, the law only recognizes those situations when the economic value of the name or likeness is established (though demonstrating economic value is not always necessary under some state right of publicity laws).  Some courts have gone so far as to liken the right to protect famous personas to a type of trademark in a person’s name because of the economic benefits attached to it, much the same way a company name, product name, or logo attached to a product or service can add value. Futurists might ask whether a robot or chatbot demonstrating a degree of intelligence and that endowed with unique human-like traits, including a unique persona (e.g., name and face generated from a generative-adversarial network) and the ability to recognize and respond to emotions (e.g., using facial coding algorithms in connection with a human-robot interface), thus making them sufficiently differentiable from all other robots and chatbots (at least superficially), should have special treatment.  So far, endowing AI technologies with a human form, gender, and/or a name has not motivated lawmakers and policymakers to pass new laws aimed at regulating AI technologies.  Indeed, lawmakers and regulators have so far proposed, and in some cases passed, laws and regulations placing restrictions on AI technologies based primarily on their specific applications (uses) and results (impacts on society).  For example, lawmakers are focusing on bot-generated spread and amplification of disinformation on social media, law enforcement use of facial recognition, the private business collection and use of face scans, users of drones and highly automated vehicles in the wild, production of “deepfake” videos, the harms caused by bias in algorithms, and others.  This application/results-focused approach, which acknowledges explicitly or implicitly certain normative standards or criteria for acceptable actions, as a means to regulate AI technology is consistent with how lawmakers have treated other technologies in the past. Thus, marketers, developers, and producers of AI systems who personify their chatbots and robots may sleep well knowing their efforts may add value to their creations and alter customer acceptance and attitudes about their AI systems, but they are unlikely to cause lawmakers to suddenly consider regulating them. At some point, however, advanced AI systems will need to be characterized in some normative way if they are to be governed as a new class of things.  The use of names, personal pronouns, personas, and metaphors associating bots to humans may frame bot technology in a way that ascribes particular values and norms to it (Jones 2017).  These might include characteristics such as utility, usefulness (including positive benefits to society), adaptability, enjoyment, sociability, companionship, and perceived or real “behavioral” control, which some argue are important in evaluating user acceptance of social robots.  Perhaps these and other factors, in addition to some measure of intelligence, need to be considered when deciding if an advanced AI bot or chatbot should be treated under the law as something other than a mere good or service.  The subjective nature of those factors, however, would obviously make it challenging to create legally-sound definitions of AI for governance purposes.  Of course, laws don’t have to be precise (and sometimes they are intentionally written without precision to provide flexibility in their application and interpretation), but a vague law won’t help an AI developer or marketer know whether his or her actions and products are subject to an AI law.  Identifying whether to treat bots as goods and services or as something else deserving of a different set of regulations, like those applicable to humans, is likely to involve a suite of factors that permit classifying advanced AI on the spectrum somewhere between goods/services and humans. Recommended reading  The Oxford Handbook of Law, Regulation, and Technology is one of my go-to references for timely insight about topics discussed on this website.  In the case of this post, I drew inspiration from Chapter 25: Hacking Metaphors in the Anticipatory Governance of Emerging Technology: The Case of Regulating Robots, by Meg Leta Jones and Jason Millar. Read more »
  • The Role of Explainable Artificial Intelligence in Patent Law
    Although the notion of “explainable artificial intelligence” (AI) has been suggested as a necessary component of governing AI technology, at least for the reason that transparency leads to trust and better management of AI systems in the wild, one area of US law already places a burden on AI developers and producers to explain how their AI technology works: patent law.  Patent law’s focus on how AI systems work was not borne from a Congressional mandate. Rather, the Supreme Court gets all the credit–or blame, as some might contend–for this legal development, which began with the Court’s 2014 decision in Alice Corp. Pty Ltd. v. CLS Bank International. Alice established the legal framework for assessing whether an invention fits in one of the patent law’s patent-eligible categories (i.e., any “new and useful process, machine, manufacture, or composition of matter” or improvements thereof) or is a patent-ineligible concept (i.e., law of nature, natural phenomenon, or abstract idea).  Alice Corp. Pty Ltd. v. CLS Bank International, 134 S. Ct. 2347, 2354–55 (2014); 35 USC § 101. Understanding how the idea of “explaining AI” came to be following Alice, one must look at the very nature of AI technology.  At their core, AI systems based on machine learning models generally transform input data into actionable output data, a process US courts and the Patent Office have historically found to be patent-ineligible.  Consider a decision by the US Court of Appeals for the Federal Circuit, whose judges are selected for their technical acumen as much as for their understanding of the nuances of patent and other areas of law, that issued around the same time as Alice: “a process that employs mathematical algorithms to manipulate existing information to generate additional information is not patent eligible.”  Digitech Image Techs, LLC v. Elecs. v. Imaging, Inc., 758 F.3d 1344, 1351 (Fed. Cir. 2014).  While Alice did not specifically address AI or mandate anything resembling explainable AI, it nevertheless spawned a progeny of Federal Circuit, district court, and Patent Office decisions that did just that.  Notably, those decisions arose not because of notions that individuals impacted by AI algorithmic decisions ought to have the right to understand how those decisions were made or why certain AI actions were taken, but because explaining how AI systems works helps satisfy the quid pro quo that is fundamental to patent law: an inventor who discloses to the world details of what she has invented is entitled to a limited legal monopoly on her creation (provided, of course, the invention is patentable). The Rise of Algorithmic Scrutiny Alice arrived not long after Congress passed patent reform legislation called the America Invents Act (AIA) of 2011, provisions of which came into effect in 2012 and 2013.  In part, the AIA targeted a decade of what many consider a time of abusive patent litigation brought against some of the largest tech companies in the world and thousands of mom-and-pop and small business owners who were sued for doing anything computer-related.  This litigious period saw the term “patent troll” used more often to describe patent assertion companies that bought up dot-com-era patents covering the very basics of using the Internet and computerized business methods and then sued to collect royalties for alleged infringement. Not surprisingly, some of the same big tech companies that pushed for patent reform provisions now in the AIA to curb patent litigation in the field of computer technology also filed amicus curiae briefs in the Alice case to further weaken software patents.  The Supreme Court’s unanimous decision in Alice helped curtail troll-led litigation by formalizing a procedure, one that lower court judges could easily adopt, for excluding certain software-related inventions from the list of inventions that are patentable. Under Alice, a patent claim–the language used by inventors to describe what he or she claims to be his or her invention–falls outside § 101 when it is “directed to” one of the patent-ineligible concepts noted above.  If so, Alice requires consideration of whether the particular elements of the claim, evaluated “both individually and ‘as an ordered combination,'” add enough to “‘transform the nature of the claim'” into one of the patent-eligible categories.  Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353 (Fed.Cir. 2016) (quoting Alice, 134 S. Ct. at 2355).  While simple in theory, it took years of court and Patent Office decisions to explain how that 2-part test is to be employed, and only more recently how it applies to AI technologies.  Today, the Patent Office and courts across the US routinely find that algorithms are abstract (even though algorithms, including certain mental processes embodied in algorithmic form performed by a computer, are by most measures useful processes).  According to the Federal Circuit, algorithmic-based data collection, manipulation, and communication–functions most AI algorithms perform–are abstract. Artificial Intelligence, Meet Alice In a bit of ironic foreshadowing, the Supreme Court issued Alice in the same year that major advances in AI technologies were being announced, such as Google’s deep neural network architecture that prevailed in the 2014 ImageNet challenge (ILSVCR) and Ian Goodfellow’s generative adversarial network (GAN) model, both of which were major contributions to the field of computer vision. Even as more breakthroughs were being announced, US courts and the Patent Office began issuing Alice decisions regarding AI technologies and explaining why it’s crucial for inventors to explain how their AI inventions work to satisfy the second half of Alice’s 2-part test. In Purepredictive, Inc. v. H2O.AI, Inc., for example, the US District Court for the Northern District of California considered the claims of US Patent 8,880,446, which, according to the patent’s owner, involves “AI driving machine learning ensembling.”  The district court characterized the patent as being directed to a software method that performs “predictive analytics” in three steps.  Purepredictive, Inc. v. H2O.AI, Inc., slip op., No. 17-cv-03049-WHO (N.D. Cal. Aug. 29, 2017).  In the method’s first step, it receives data and generates “learned functions,” or, for example, regressions from that data. Second, it evaluates the effectiveness of those learned functions at making accurate predictions based on the test data. Finally, it selects the most effective learned functions and creates a rule set for additional data input. The court found the claims invalid on the grounds that they “are directed to the abstract concept of the manipulation of mathematical functions and make use of computers only as tools, rather than provide a specific improvement on a computer-related technology.” The claimed method, the district court said, is merely “directed to a mental process” performed by a computer, and “the abstract concept of using mathematical algorithms to perform predictive analytics” by collecting and analyzing information.  The court explained that the claims “are mathematical processes that not only could be performed by humans but also go to the general abstract concept of predictive analytics rather than any specific application.” In Ex Parte Lyren, the Patent Office’s Appeals Board, made up of three administrative law judges, rejected a claim directed to customizing video on a computer as being abstract and thus not patent-eligible.  In doing so, the board disagreed with the inventor, who argued the claimed computer system, which generated and displayed a customized video by evaluating a user’s intention to purchase a product and information in the user’s profile, was an improvement in the technical field of generating videos. The claimed customized video, the Board found, could be any video modified in any way.  That is, the rejected claims were not directed to the details of how the video was modified, but rather to the result of modifying the video.  Citing precedent, the board reiterated that “[i]n applying the principles emerging from the developing body of law on abstract ideas under section 101, … claims that are ‘so result-focused, so functional, as to effectively cover any solution to an identified problem’ are frequently held ineligible under section 101.”  Ex ParteLyren, No. 2016-008571 (PTAB, June 25, 2018) (citing Affinity Labs of Texas,LLC v. DirecTV, LLC, 838 F.3d 1253, 1265 (Fed. Cir. 2016) (quoting Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1356 (Fed. Cir, 2016)); see also Ex parte Colcernian et al., No. 2018-002705 (PTAB, Oct. 1, 2018) (rejecting claims that use result-oriented language as not reciting the specificity necessary to show how the claimed computer processor’s operations differ from prior human methods, and thus are not directed to a technological improvement but rather are directed to an abstract idea). Notably, the claims in Ex Parte Lyren were also initially rejected as failing to satisfy a different patentability test–the written description requirement.  35 USC § 112.  In rejecting the claims as lacking sufficient description of the invention, the Patent Office Examiner found that the algorithmic features of the inventor’s claim were “all implemented inside a computer, and therefore all require artificial intelligence [(AI)] at some level” and thus require extensive implementation details “subject of cutting-edge research, e.g.[,] natural language processing and autonomous software agents exhibiting intelligent behavior.” The Examiner concluded that “one skilled in the art would not be persuaded that Applicant possessed the invention” because “it is not readily apparent how to make a device [to] analyze natural language.”  The Appeals Board disagreed and sided with the inventor who argued that his invention description was comprehensive and went beyond just artificial intelligence implementations.  Thus, while the description of how the invention worked was sufficiently set forth, Lyren’s claims focused too much on the results or application of the technology and thus were found to be abstract. In Ex Parte Homere, claims directed to “a computer-implemented method” involving “establishing a communication session between a user of a computer-implemented marketplace and a computer-implemented conversational agent associated with the market-place that is designed to simulate a conversation with the user to gather listing information, the Appeals Board affirmed an Examiner’s rejection of the claims as being abstract.  Ex Parte Homere, Appeal No. 2016-003447 (PTAB Mar. 29, 2018).  In doing so, the Appeals Board noted that the inventor had not identified anything in the claim or in the written description that would suggest the computer-related elements of the claimed invention represent anything more than “routine and conventional” technologies.  The most advanced technologies alluded to, the Board found, seemed to be embodiments in which “a program implementing a conversational agent may use other principles, including complex trained Artificial Intelligence (AI) algorithms.”  However, the claimed conversational agent was not so limited.  Instead, the Board concluded that the claims were directed to merely using recited computer-related elements to implement the underlying abstract idea, rather than being limited to any particular advances in the computer-related elements. In Ex Parte Hamilton, a rejection of a claim directed to “a method of planning and paying for advertisements in a virtual universe (VU), comprising…determining, via the analysis module, a set of agents controlled by an Artificial Intelligence…,” was affirmed as being patent ineligible.  Ex Parte Hamilton et al., Appeal No.2017-008577 (PTAB Nov. 20, 2018).  The Appeals Board found that the “determining” step was insufficient to transform the abstract idea of planning and paying for advertisements into patent-eligible subject matter because the step represented an insignificant data-gathering step and thus added nothing of practical significance to the underlying abstract idea. In Ex Parte Pizzorno, the Appeals Board affirmed a rejection of a claim directed to “a computer implemented method useful for improving artificial intelligence technology” as abstract.  Ex Parte Pizzorno, Appeal No. 2017-002355 (PTAB Sep. 21, 2018).  In doing so, the Board determined that the claim was directed to the concept of using stored health care information for a user to generate personalized health care recommendations based on Bayesian probabilities, which the Board said involved “organizing human activities and an idea in itself, and is an abstract idea beyond the scope of § 101.”  Considering each of the claim elements in turn, the Board also found that the function performed by the computer system at each step of the process was purely conventional in that each step did nothing more than require a generic computer to perform a generic computer function. Finally, in Ex Parte McAfee, the Appeals Board affirmed a rejection of a claim on the basis that it was “directed to the abstract idea of receiving, analyzing, and transmitting data.”  Ex Parte McAfee, Appeal No. 2016-006896 (PTAB May 22, 2018).  At issue was a method that included “estimating, by the ad service circuitry, a probability of a desired user event from the received user information, and the estimate of the probability of the desired user event incorporating artificial intelligence configured to learn from historical browsing information in the received user information, the desired user event including at least one of a conversion or a click-through, and the artificial intelligence including regression modeling.”  In affirming the rejection, the Board found that the functions performed by the computer at each step of the claimed process was purely conventional and did not transform the abstract method into a patent-eligible one. In particular, the step of estimating the probability of the desired user event incorporating artificial intelligence was found to be merely “a recitation of factors to be somehow incorporated, which is aspirational rather than functional and does not narrow the manner of incorporation, so it may include no more than incorporating results from some artificial intelligence outside the scope of the recited steps.” The above and other Alice decisions have led to a few general legal axioms, such as: a claim for a new abstract idea is still an abstract idea; a claim for a beneficial abstract idea is still an abstract idea; abstract ideas do not become patent-eligible because they are new ideas, are not previously well known, and are not routine activity; and, the “mere automation of manual processes using generic computers does not constitute a patentable improvement in computer technology.”  Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1151 (Fed. Cir. 2016); Ariosa Diagnostics, Inc. v. Sequenom, Inc., 788 F.3d 1371, 1379-80 (Fed. Cir. 2015); Ultramercial, Inc. v. Hulu, LLC, 772 F.3d. 709, 715-16 (Fed. Cir. 2014); Credit Acceptance Corp. v. Westlake Servs., 859 F.3d 1044, 1055 (Fed. Cir. 2017); see also SAP Am., Inc. v. Investpic, LLC, slip op. No. 2017-2081, 2018 WL2207254, at *2, 4-5 (Fed. Cir. May 15, 2018) (finding financial software patent claims abstract because they were directed to “nothing but a series of mathematical calculations based on selected information and the presentation of the results of those calculations (in the plot of a probability distribution function)”); but see Apple, Inc. v.Ameranth, Inc., 842 F.3d 1229, 1241 (Fed. Cir. 2016) (noting that “[t]he Supreme Court has recognized that all inventions embody, use,reflect, rest upon, or apply laws of nature, natural phenomena, or abstractideas[ ] but not all claims are directed to an abstract idea.”). The Focus on How, not the Results Following Alice, patent claims directed to an AI technology must recite features of the algorithm-based system that represent how the algorithm improves a computer-related technology and is not previously well-understood, routine, and conventional.  In PurePredictive, for example, the Northern California district court, which sees many software-related cases due to its proximity to the Bay Area and Silicon Valley, found that the claims of a machine learning ensemble invention were not directed to an invention that “provide[s] a specific improvement on a computer-related technology.”  See also Neochloris, Inc. v. Emerson Process Mgmt LLLP, 140 F. Supp. 3d 763, 773 (N.D. Ill. 2015) (explaining that patent claims including “an artificial neural network module” were invalid under § 101 because neural network modules were described as no more than “a central processing unit – a basic computer’s brain”). Satisfying Alice, thus, requires claims focusing on a narrow application of how an AI algorithmic model works, rather than the broader and result-oriented nature of what the model is used for.  This is necessary where the idea behind the algorithm itself could be used to achieve many different results.  For example, a claim directed to a mathematical process (even one that is said to be “computer-implemented”), and that could be performed by humans (even if it takes a long time), and that is directed to a result achieved instead of a specific application, will seemingly be patent-ineligible under today’s Alice legal framework. To illustrate, consider an image classification system, one that is based on a convolutional neural network.  Such a system may be patentable if the claimed system improves the field of computer vision technology. Claiming the invention in terms of how the elements of the computer are technically improved by its deep learning architecture and algorithm, rather than simply claiming a deep learning model using results-oriented language, may survive an Alice challenge, provided the claim does not merely cover an automated process that human used to do.  Moreover, claims directed to the multiple hidden layers, convolutions, recurrent connections, hyperperameters, and weights could also be claimed. By way of another example, a claim reciting “a computer-implemented process using artificial intelligence to generate an image of a person,” is likely abstract if it does not explain how the image is generated and merely claims a computerized process a human could perform.  But a claim that describes a unique AI system that specifies how it generates the image, including the details of a generative adversarial network architecture and its various inputs provided by physical devices (not routine data collection), its connections and hyperparameters, has a better chance of passing muster (keeping in mind, this only addresses the question of whether the claimed invention is eligible to be patented, not whether it is, in fact, patentable, which is an entirely different analysis and requires comparing the claim to prior art). Uncertainty Remains Although the issue of explaining how an AI system works in the context of patent law is still in flux, the number of US patents issued by the Patent Office mentioning “machine learning,” or the broader term “artificial intelligence,” has jumped in recent years. Just this year alone, US machine learning patents are up 27% compared to the same year-to-date period in 2017 (thru the end of November), according to available Patent Office records.  Even if machine learning is not the focus of many of them, the annual upward trend in patenting AI over the last several years appears unmistakable. But with so many patents invoking AI concepts being issued, questions about their validity may arise.  As the Federal Circuit has stated, “great uncertainty yet remains” when it comes to the test for deciding whether an invention like AI is patent-eligible under Alice, this despite the large number of cases that have “attempted to provide practical guidance.”  Smart Systems Innovations, LLC v. Chicago Transit Authority, slip. op. No. 2016-1233 (Fed. Cir. Oct. 18, 2017).  Calling the uncertainty “dangerous” for some of today’s “most important inventions in computing,” specifically mentioning AI, the Federal Circuit expressed concern that perhaps the application of the Alice test has gone too far, a concern mirrored in testimony by Andrei Iancu, Director of the Patent Office, before Congress in April 2018 (stating, in response to Judiciary Committee questions, that Alice and its progeny have introduced a degree of uncertainty into the area of subject matter eligibility, particularly as it relates to medical diagnostics and software-related inventions, and that Alice could be having a negative impact on innovation). Absent legislative changes abolishing or altering Alice, a solution to the uncertainty problem, at least in the context of AI technologies, lies in clarifying existing decisions issued by the Patent Office and courts, including the decisions summarized above.  While it can be challenging to explain why an AI algorithm made a particular decision or took a specific action (due to the black box nature of such algorithms once they are fully trained), it is generally not difficult to describe the structure of a deep learning or machine learning algorithm or how it works. Even so, it remains unclear whether and to what extent fully describing how one’s AI technology and including “how” features in patent claims will ever be sufficient to “add[] enough to transform the nature of an abstract algorithm into a patent-eligible [useful process].” If explaining how AI works is to have a future meaningful role in patent law, the courts or Congress will need to provide clarity. Read more »
  • California Appeals Court Denies Defendant Access to Algorithm That Contributed Evidence to His Conviction
    One of the concerns expressed by those studying algorithmic decision-making is the apparent lack of transparency. Those impacted by adverse algorithmic decisions often seek transparency to better understand the basis for the decisions. In the case of software used in legal proceedings, parties who seek explanations about software face a number of obstacles, including those imposed by evidentiary rules, criminal or civil procedural rules, and by software companies that resist discovery requests. The closely-followed issue of algorithmic transparency was recently considered by a California appellate court in People v. Superior Court of San Diego County, slip op. Case D073943 (Cal. App. 4th October 17, 2018), in which the People sought relief from a discovery order requiring the production of software and source code used in the conviction of Florencio Jose Dominguez. Following a hearing and review of the record and amicus briefs in support of Dominguez filed by the American Civil Liberties Union, the American Civil Liberties Union of San Diego and Imperial Counties, the Innocence Project, Inc., the California Innocence Project, the Northern California Innocence Project at Santa Clara University School of Law, Loyola Law School’s Project for the Innocent, and the Legal Aid Society of New York City, the appeals court granted the People’s relief. In doing so, the court considered, but was not persuaded by, the defense team’s “black box” and “machine testimony” arguments. At issue on appeal was Dominguez’s motion to compel production of a DNA testing program called STRmix used by local prosecutors in their analysis of forensic evidence (specifically, DNA found on the inside of gloves). STRmix is a “probabilistic genotyping” program that expresses a match between a suspect and DNA evidence in terms the probability of a match compared to a coincidental match. Probabilistic genotyping is said to reduce subjectivity in the analysis of DNA typing results. Dominguez’s counsel moved the trial court for an order compelling the People to produce the STRmix software program and related updates as well as its source code, arguing that defendant had a right to look inside the software’s “black box.” The trial court granted the motion and the People sought writ relief from the appellate court. On appeal, the appellate court noted that “computer software programs are written in specialized languages called source code” and “source code, which humans can read, is then translated into [a] language that computers can read.” Cadence Design Systems, Inc. v. Avant! Corp., 29 Cal. 4th 215, 218 at fn.3 (2002). The lab that used STRmix testified that it had no way to access the source code, which it licensed from a software authorized seller.  Thus,  the court considered whether the company that created the software should produce it. In concluding that the company was not obligated to produce the software and source code, the court, citing precedent, found that the company would have had no knowledge of the case but for the defendant’s  subpoena duces tecum, and it did not act as part of the prosecutorial team such that it was obligated to turn over exculpatory evidence (assuming software itself is exculpatory, which the court was reluctant to find). With regard to the defense team’s “black box” argument, the appellate court found nothing in the record to indicate that the STRmix software suffered a problem, as the defense team argued, that might have affected its results. Calling this allegation speculative, the court concluded that the “black box” nature of STRmix was not itself sufficient to warrant its production. Moreover, the court was unpersuaded by the defense team’s argument that the STRmix program essentially usurped the lab analyst’s role in providing the final statistical comparison, and so the software program—not the analyst using the software—was effectively the source of the expert opinion rendered at trial. The lab, the defense argued, merely acted in a scrivener’s capacity for STRmix’s analysis, and since the machine was providing testimony, Dominguez should be able to evaluate the software to defend against the prosecution’s case against him. The appellate court disagreed. While acknowledging the “creativity” of the defense team’s “machine testimony” argument (which relied heavily on Berkeley law professor Andrea Roth’s “Machine Testimony” article (126 Yale L.J. 1972 (2017)), the panel noted the testimony that STRmix did not act alone, that there were humans in the loop: “[t]here are still decisions that an analyst has to make on the front end in terms of determining the number of contributors to a particular sample and determin[ing] which peaks are from DNA or from potentially artifacts” and that the program then performs a “robust breakdown of the DNA samples,” based at least in part on “parameters [the lab] set during validation.” Moreover, after STRmix renders “the diagnostics,” the lab “evaluate[s] … the genotype combinations … . to see if that makes sense, given the data [it’s] looking at.” After the lab “determine[s] that all of the diagnostics indicate that the STRmix run has finished appropriately,” it can then “make comparisons to any person of interest or … database that [it’s] looking at.” While the appellate court’s decision mostly followed precedent and established procedure, it could easily have gone the other way and affirmed the trial judge’s decision granting Defendant’s motion to compel the STRmix software and source code, which would have given Dominguez better insight into the nature of the software’s algorithms, its parameters and limitations in view of validation studies, and the various possible outputs the model could have produced given a set of inputs. In particular, the court might have affirmed the trial judge’s decision to grant access to the STRmix software if the policy of imposing transparency in STRmix’s algorithmic decisions were given more consideration from the perspective of actual harm that might occur if software and source code are produced. Here, the source code owner’s objection to production was based in part on trade secret and other confidentiality concerns; however, procedures already exist to handle those concerns. Indeed, source code reviews happen all the time in the civil context, such as in patent infringement matters involving software technologies. While software makers are right to be concerned about the harm to their businesses if their code ends up in the wild, the real risk of this happening can be low if proper procedures, embodied in a suitable court-issued Protective Order, are followed by lawyers on both sides of a matter and if the court maintains oversight and demands status updates from the parties to ensure compliance and integrity in the review process. Instead of following the trial court’s approach, however, the appellate court conditional access to STRmix’s “black box” on the demonstration of specific errors in the program’s results, which seems intractable: only by looking into the black box in the first place is a party able to understand whether problems exist that affect the result. Interestingly, artificial intelligence had nothing to do with the outcome of the appellate court’s decision, yet the panel noted that “We do not underestimate the challenges facing the legal system as it confronts developments in the field of artificial intelligence.” The judges acknowledged that the notion of “machine testimony” in algorithmic decision-making matters is a subject about which there are widely divergent viewpoints in the legal community, a possible prelude to what is ahead when artificial intelligence software cases make their way through the courts in criminal or non-criminal cases.  To that, the judges cautioned, “when faced with a novel method of scientific proof, we have required a preliminary showing of general acceptance of the new technique in the relevant scientific community before the scientific evidence may be admitted at trial.” Lawyers in future artificial intelligence cases should consider how best to frame arguments concerning machine testimony in both civil and criminal contexts to improve their chances of overcoming evidentiary obstacles. Lawyers will need to effectively articulate the nature of artificial intelligence decision-making algorithms, as well as the relative roles of data scientists and model developers who make decisions about artificial intelligence model architecture, hyperparameters, data sets, model inputs, training and testing procedures, and the interpretation of results. Today’s artificial intelligence systems do not operate autonomously; there will always be humans associated with a model’s output or result and those persons may need to provide expert testimony beyond the machine’s testimony.  Even so, transparency will be important to understanding algorithmic decisions and for developing an evidentiary record in artificial intelligence cases. Read more »
WordPress RSS Feed Retriever by Theme Mason

Author: hits1k

Leave a Reply