AI safety rules curb frontier models

Summary#

S-10731 is an introduced bill to amend New York’s General Business Law provisions for the Responsible AI Safety and Education (RAISE) Act. It would change the rules for large frontier AI developers, add independent examiners and possible computing limits, and expand reporting and enforcement. Its stated goal is to address catastrophic and security risks from frontier models; that goal is not a guaranteed result. The bill is not law, and the supplied material does not include the full current Act, so I could not verify every change against current law.

  • It would define a “large frontier developer” by its AI-related spending—more than $1 billion over 36 months—instead of the revenue threshold shown as current law in the bill. It would also lower the computing threshold for a “frontier model” from 10²⁶ to 10²⁵ operations.
  • Large developers would have to publish and follow a safety framework, report specified assessments to the state, and hire independent examiners. The bill adds required disclosures about internal AI use, automated AI research, and risk assessments.
  • The state’s Department of Financial Services office would have to set binding minimum safety standards by January 1, 2029. It could also make rules for ordering limits on certain computing use.
  • The bill would expand what counts as a critical safety incident and require developers to report incidents, generally within 72 hours. An incident posing an imminent risk of death or serious injury must be disclosed to an appropriate authority within 24 hours.
  • The Attorney General could seek penalties of up to $1 million per day for a first violation and $3 million per day for a later violation, and could ask a court for orders to stop activity or restrict a model’s use in New York.

What it means for you#

  • Large frontier AI developers: They would have to meet the new framework, reporting, recordkeeping, and examiner requirements. They would pay to retain examiners. The office could require a second examiner in specified cases.
  • Other frontier developers: They would have to publish transparency reports before or when deploying a new or substantially modified frontier model and report critical safety incidents. The bill’s expanded incident definition includes certain unauthorized access to unreleased model weights, model conduct that evades developer controls, and certain harm to outside computer systems.
  • Employees of large developers: Employers would have to provide notices about whistleblower rights, including to new and remote employees, and could not use workplace rules or contracts to block protected activity described in Labor Law.
  • The public: Anyone could report a critical safety incident to the office. The bill does not create a private right to sue for violations. Public examiner reports and state summaries could be redacted to protect trade secrets, security, safety, or national security.
  • The state: The office would have new rulemaking, review, and oversight duties. A court—not the office acting alone—could order a developer to stop an activity or restrict a model’s deployment, use, or availability in New York.

Money#

No publicly available information.

  • The bill would require large developers to pay for examiners, and the office to retain an examiner to check compliance with any computing-limit order. It gives no cost estimate.
  • It would require state regulations, reviews, and reports. The bill gives no staffing or budget estimate for this work.
  • No tax, fee, or public benefit change is specified.

What is unclear#

  • The bill makes the new safety standards due by January 1, 2029, but does not say when those regulations would take effect or what the standards would require in practice.
  • The office may order limits on computing use, but the bill leaves the circumstances and procedure to future regulations. It does not set a specific limit.
  • “Substantial increase” in reported risk or AI research automation is not defined in the bill.
  • The bill allows redactions and confidential submissions. How much information the public would see in practice is unclear.
  • The effective date is tied to Chapter 699 of the Laws of 2025. The supplied material does not explain that law’s effective date or how the timing works.

Case for#

  • The bill appears intended to make large developers assess and disclose risks not only before deployment, but also from internal use and AI-assisted research.
  • Independent examiners could provide the state and public with checks on a developer’s own safety assessments and public statements.
  • Faster incident reporting and the power to seek court orders could allow action when a serious risk is reported or believed to be imminent.
  • The bill directs the office to weigh risks against societal benefits and to consider whether a subject is understood well enough for regulation.

Case against#

  • The bill would give regulators and courts significant power to limit computing use or restrict a model in New York, while leaving important details to future rules and decisions.
  • Frequent independent reviews and extensive access to internal information could be costly and burdensome for large developers; the bill provides no cost estimate.
  • Broad incident definitions and penalties of up to millions of dollars per day could create uncertainty about what must be reported and what counts as a violation.
  • Public transparency has limits: reports may be redacted, and reports about incidents and internal-use assessments would generally be exempt from public-record disclosure.