by Cansu Yalim, PhD

In 1669, an alchemist in Hamburg named Hennig Brand heated the residue left after evaporating urine. He was pursuing the philosopher’s stone, not a new industrial material. Yet his experiment produced phosphorus, a substance whose glow made the discovery extraordinary. Brand had found something real, even though the theory guiding his search was wrong [1].

That distinction matters today. Artificial intelligence can produce useful results even when an organization cannot explain how reliably those results will hold up under new inputs, changed operating conditions, or a model update. This is not to say that AI research is alchemy; it is to say that some AI deployments still rely on practices closer to trial and error than to disciplined engineering.

For engineering managers, the difference is consequential. A capability that works in a demonstration is not necessarily a capability that can be staffed, maintained, audited, and trusted in operation. The pressing question is no longer simply, “Can the model do this?” It is, “Under which conditions does it work, how will we know when it stops working, and who is responsible for acting on that evidence?”

What a “Lavoisier Moment” Really Means

In 1787, Louis-Bernard Guyton de Morveau, Antoine Lavoisier, Claude-Louis Berthollet, and Antoine-François de Fourcroy published Méthode de nomenclature chimique, with contributions on chemical symbols by Jean-Henri Hassenfratz and Pierre-Auguste Adet. The work sought a more systematic language for describing substances and their relationships [2]. It did not resolve chemistry’s scientific debates overnight. It helped make claims easier to communicate, question, and compare.

AI engineering needs a comparable change in working habits: away from isolated demonstrations and undocumented prompt recipes, and toward explicit interfaces, repeatable evaluation, and accountable oversight. Three shifts would make that transition tangible.

1) From clever prompts to specified behavior

A carefully written prompt can be effective. The problem begins when the prompt becomes the entire specification for a business-critical function. Teams may know which wording produced a convincing answer but lack a stable description of accepted inputs, required evidence, permissible actions, and failure handling. A vendor model change can then become an uncontrolled change to the application itself.

Frameworks such as DSPy illustrate a more systematic approach. DSPy represents language-model tasks through declarative signatures and composable modules, and can optimize parts of a pipeline against a chosen evaluation metric. This reduces dependence on manually tuning long prompt strings [3, 4]. It does not prove correctness or guarantee the same behavior after a model change; the optimized system still has to be tested under the conditions in which it will be used.

The managerial lesson is broader than any one framework. Specify what enters an AI component, what it is allowed to return or do, what evidence must accompany the output, and when it must defer to a person. Version its prompts, model settings, retrieval sources, and test data as deliberately as other software dependencies.

2) From impressive examples to evidence of reliability

Many AI pilots are evaluated through a handful of persuasive examples. Those examples are useful for exploration, but they say little about failure rates, rare events, or changing operating regimes. Engineering decisions require more than a plausible answer on a good day.

An evaluation pipeline should begin with a documented task definition and a representative, versioned set of test cases. It should include ordinary cases, ambiguous requests, outdated or conflicting records, and situations in which abstaining is the correct response. Depending on the application, measures may include factual accuracy, support from retrieved evidence, unsafe recommendations, missed detections, and the rate at which human review is required. The NIST AI Risk Management Framework and its generative AI profile place measurement, monitoring, and risk response within a continuing management process [5, 6]. If an LLM is used to grade another model’s output, its judgments should also be checked against a sample of independently reviewed human assessments, because automated judges can exhibit systematic biases [7].

A single accuracy percentage is not a release decision. Missing a high-consequence fault can matter more than several benign errors, and results measured on routine operations may not transfer to startup, shutdown, or other unusual conditions. Acceptance criteria should therefore reflect the intended use, the severity of foreseeable failures, and the organization’s capacity to detect and recover from them.

3) From informal assurances to operational accountability

Standards cannot make a probabilistic model deterministic. They can, however, clarify who owns the risks and what evidence should accompany deployment. NIST AI RMF 1.0 is a voluntary framework organized around Govern, Map, Measure, and Manage [5]. ISO/IEC 42001:2023 specifies requirements for establishing and improving an organizational AI management system [8]. Neither should be mistaken for a guarantee that an individual model’s answers are correct.

Engineering leaders can translate those frameworks into practical controls: maintain an inventory of AI uses; assign a risk owner; record intended use and operating limits; establish review and escalation procedures; and require regression testing when models, data sources, or workflows change. Governance becomes useful when it changes how a system is built and operated, rather than merely adding documentation at the end.

Consider One Engineering Use Case

Imagine an AI assistant that summarizes maintenance records and recommends follow-up checks after a compressor vibration alarm. In a demonstration, it may produce a clear, technically plausible explanation. In an operational review, an engineering manager should ask different questions. Did it cite the correct sensor trend and work-order history? Did it confuse a correlated operating condition with an established root cause? Does it distinguish a suggested inspection from an authorized maintenance intervention? Does it flag missing evidence and refer uncertain cases to qualified personnel?

These are not hypothetical questions about model eloquence. They define the system’s engineering requirements. The assistant’s reliability depends on traceable inputs, well-defined authority, testing across relevant operating conditions, and a safe path when the available evidence does not support a conclusion. Predicting a fault label is not, by itself, proof of its cause.

What Engineering Managers Can Do Next

Start with one consequential AI workflow rather than a company-wide transformation program. First, document its intended decisions, data dependencies, failure consequences, and human owner. Second, build a small but credible evaluation set, including difficult cases and explicit abstention expectations. Third, put input-output requirements, version control, and regression tests around the workflow. Finally, set a review process for changes, incidents, and evidence that performance has drifted. Expand only when the team can explain not just what improved but what remains uncertain.

Brand’s experiment shows that a surprising discovery can emerge from an imperfect theory. The history of chemistry also shows why discovery alone is not an engineering system. The next advance in applied AI will not be another incantation disguised as a prompt. It will be the ability to say, with evidence, what an AI capability can be trusted to do, where that confidence ends, and how the organization will respond when reality changes.

References

[1] Royal Society of Chemistry. “Phosphorus,” History section. Periodic Table. https://periodic-table.rsc.org/element/15/phosphorus

[2] Guyton de Morveau, L.-B., Lavoisier, A.-L., Berthollet, C.-L., Fourcroy, A.-F., Hassenfratz, J.-H., & Adet, P.-A. (1787). Méthode de nomenclature chimique. Paris: Cuchet. Library of Congress. https://www.loc.gov/item/15011439/

[3] Khattab, O., et al. (2024). “DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines.” International Conference on Learning Representations (ICLR). https://proceedings.iclr.cc/paper_files/paper/2024/hash/f1cf02ce09757f57c3b93c0db83181e0-Abstract-Conference.html

[4] DSPy. “Signatures.” Official documentation. https://dspy.ai/3.2.0/learn/programming/signatures/

[5] Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. https://doi.org/10.6028/NIST.AI.100-1

[6] Autio, C., Schwartz, R., Dunietz, J., Jain, S., Stanley, M., Tabassi, E., Hall, P., & Roberts, K. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. https://doi.org/10.6028/NIST.AI.600-1

[7] Zheng, L., et al. (2023). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track). https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html

[8] ISO/IEC. (2023). ISO/IEC 42001:2023: Information technology — Artificial intelligence — Management system. https://www.iso.org/standard/42001

About the Author

Cansu Yalim is pursuing her Ph.D. in Engineering Management and Systems Engineering at Old Dominion University, where she also serves as a graduate research and teaching assistant. Before academia, she worked in the thermotechnology and automotive industries. Yalim’s research reframes industrial root-cause diagnosis as a causal inference problem, overcoming the correlational limits of traditional ML by integrating time, causal structure, and system dynamics to deliver trustworthy fault attribution in complex, changing environments.

LinkedIn: https://www.linkedin.com/in/cansu-y-63089a153/

BLOG DISCLAIMER: The opinions, views and content presented in this online blog solely represent the original creator and has no association with ASEM activities, products or services. The content is made available to the community for educational and informational purposes only. All blog posts are created voluntarily and are not sold, but may be used, shared and distributed free of charge. The blogs are not academic pieces, and therefore do not go through a peer-review process, and are not fact-checked. All errors belong to the creators.