SuperIntelligenceSI.live
Ask Anything About Super Intelligence (SI)
Search a curated library about superintelligence, current AI systems, evaluation, safety, governance, and study tools. Answers stay inside this library. A missing record is reported as missing. Nothing here is legal, medical, or financial advice, and rules differ by country.
Questions and answers
What is Super Intelligence?
Super Intelligence, often shortened to SI, is a term for a mind or system that would greatly outperform humans across most important intellectual tasks. It is a definition used in philosophy and AI research, not a label for today's chatbots, search tools, or scientific software. Writers also say superintelligence. Some reserve artificial superintelligence, or ASI, for a machine version of the idea. Those names are related, but they are not used identically by everyone.
The phrase became widely discussed through work such as I. J. Good's 1965 essay on an ultraintelligent machine and, later, Nick Bostrom's book Superintelligence. A common definition is an intellect that greatly exceeds the best human performance in practically every field, including scientific work, practical judgment, and social reasoning. That sentence is a definition. It is not evidence that a system like that exists. People sometimes use the shorter name Super Intelligence, or SI, for the same family of ideas. Others use it more loosely for any very capable software. On this site the strict sense is the one that matters: a general capability far beyond the human range, not a high score on one benchmark. Current systems can be unusually strong at narrow work. A model can draft text, write code, or help search literature and still fail when the task, the data, or the tools change.…
What does SI mean?
On this site SI means Super Intelligence: a proposed level of intellectual performance well beyond the human range across most important tasks. It does not mean a particular company, model, or exam score. ASI usually means artificial superintelligence, and AGI usually means artificial general intelligence. Authors do not all draw the boundaries in the same place.
Short abbreviations are convenient and easy to muddle. SI, as used here, points at superintelligence in the broad sense: capability far above humans on most cognitive work that matters. The same two letters can mean other things in other fields, so context matters. AGI is commonly used for a system that could learn a very wide range of tasks at roughly human level. ASI is commonly used when the writer wants to stress a machine that would go far past that level. Some texts treat SI and ASI as the same idea. Others use SI for the capability and ASI only when the substrate is a machine. None of these abbreviations is a measurement you can read off a leaderboard. If a page says a model is 'SI' because it did well on a test, it is using the word more loosely than the research definition.
How could Super Intelligence differ from AGI?
AGI is usually pictured as breadth: a system that can learn many kinds of tasks, often compared with human level. Super Intelligence adds a claim about degree: performance far above the best humans across those tasks. A system could be discussed as general without being superintelligent, and a narrow system can beat humans at one task without being either. Both labels are definitions, not completed technologies.
The useful contrast is between range and height. Artificial general intelligence, in the most common usage, means a system that is not stuck in one narrow skill. It could learn new tasks, transfer skill, and cope with unfamiliar problems at about the level of a capable person. People still argue about how broad 'general' must be. Super Intelligence, in the strict usage, is not 'a bit better than a chatbot.' It is the idea of an intellect that greatly exceeds humans in nearly every important intellectual field. That is a higher bar than human-level generality. In that usage, AGI could be a step some writers imagine on the way to SI, or a different target entirely. There is no agreed sequence, and there is no demonstration that either stage has been reached. A chess engine that beats every grandmaster is superhuman at chess and still not AGI or SI. A future system that matched a skilled…
How is AGI different from SI?
AGI usually names human-level breadth: many tasks, not one benchmark. SI usually names a much higher level: performance well beyond the best humans across most important intellectual tasks. The difference is the bar, not a logo. Neither term describes a system that has been shown to exist.
Writers sometimes swap the two abbreviations in public debate. A careful distinction still helps. Artificial general intelligence is about not being narrow. The system would handle a wide set of problems, including ones it was not separately built for, at roughly human competence. How to measure that breadth is itself disputed. Super Intelligence sets a higher bar. The claim is not 'as good as a person at many jobs.' It is 'much better than the best people at most of the intellectual work that matters,' including the ability to learn further. Under that definition, something could be called AGI without being SI. The reverse confusion also happens. A model that exceeds humans on a coding test is not SI. Extra points on a narrow exam are not general superiority. Treat both phrases as proposals about capability. Ask the author what tasks, what comparison group, and what evidence they have…
What is artificial superintelligence?
Artificial superintelligence, or ASI, is a name for a machine system that would greatly outperform humans across most intellectual tasks. It is a theoretical category. It is not a description of any model you can use today. Some authors treat ASI and superintelligence as the same idea. Others use ASI only when they mean a digital system rather than a biological or collective one.
The 'artificial' in ASI does a small but useful job. Superintelligence, taken literally, could refer to any intellect far above the human range, including ideas people have discussed that are not ordinary software. ASI narrows the picture to a machine. In practice many AI writers use the words interchangeably, so you still have to read the definition in front of you. ASI is not a synonym for a large language model, a chatbot, or a system that wins a contest. Those are present technologies with known limits: they depend on data, compute, and human-built scaffolding, and they fail outside their setup. ASI, as defined in the safety literature, would be a qualitative jump past that, including the ability to outthink people on the open-ended problems that current systems do not reliably solve. No laboratory has demonstrated that jump. Papers and books discuss paths people have imagined,…
Does Super Intelligence exist today?
No. Nothing publicly demonstrated meets the usual definition of superintelligence: performance far above humans across most important intellectual tasks. Today's AI can be excellent at specific work and still depend on human goals, human data, and human oversight. Be wary of headlines that treat a benchmark win as if the definition had been satisfied.
The strict definition is intentionally demanding. It is not enough to beat people at one game, one exam, or one coding set. The idea is broad superiority, including scientific and practical judgment, not a single score. What does exist is a family of useful systems. They recognize patterns in data, generate text and images, help with search, and operate tools when someone wires those tools up. Their failures are also ordinary and observable: invented citations, brittle behavior when the task shifts, and no independent ability to set up and run a long research program without people. Some commentators argue that generality is increasing and that the line will move. That is a forecast. It can be discussed, but it should not be rewritten as a present-tense fact. If a claim says superintelligence exists today, ask which tasks, which human comparison, and which outside checks support it. A…
How is SI different from current AI?
Current AI is a set of methods that learn patterns and perform tasks people set up. Super Intelligence is a proposed level of general performance far above humans, and it has not been demonstrated. The practical difference is between tools you can evaluate now and a capability that remains hypothetical. Mixing the two makes both safety talk and product talk less clear.
Current systems are built. You can name the training setup in public papers, the data limits, the interface, and the failures. They are often called narrow even when they look flexible, because the flexibility still sits inside a model trained for prediction or instruction following, not inside a shown ability to master arbitrary intellectual work. SI, in the strict sense, would not be 'a larger version of a chatbot' by definition alone. Size and spend can improve some measured skills. They do not, by themselves, prove that a system understands the world, plans safely, or exceeds experts across the board. Those are separate claims and need separate evidence. A useful habit is to sort sentences into three trays. The first tray is present tools: what a system did, on what data, with what errors. The second is research: what labs are trying to measure. The third is scenario: what might be…
What is recursive self-improvement?
Recursive self-improvement is the idea that a system improves the process it uses to get better, and that those improvements stack. I. J. Good discussed a related picture in 1965: an ultraintelligent machine that designs still better machines. It is a theoretical scenario. Using a model to help engineers write code is not, by itself, that scenario.
The phrase has a precise flavor and a loose one. The precise flavor is a loop in which the system improves its own ability to improve, rather than only getting a one-time upgrade from people. The loose flavor covers any project where AI tools speed up research. Those are not the same claim. Today, people do use models to draft code, tests, and papers. The training runs, the objectives, the data, and the decision to deploy still sit with human organizations. That can shorten some engineering tasks. It does not show an autonomous process that keeps leaping in capability without bottlenecks in compute, data, evaluation, or oversight. The scenario matters for safety discussions because if improvement became very fast, slower human review might not keep up. That is a reason researchers study evaluation and control ahead of time. It is not a result that has been observed, and it does not…
What is intelligence amplification?
Intelligence amplification, sometimes called IA, means using tools to help people think, rather than replacing human judgment with an autonomous system. Calculators, libraries, and well-designed software are older examples. In AI debates it is the alternative picture to a system that simply outruns its users. Whether a tool amplifies someone depends on the interface, the user's skill, and the checks around the work.
The contrast is about where the decision sits. A superintelligence scenario, strictly told, is about a system that is itself the superior problem solver. Intelligence amplification is about a person or a team who remain the problem solvers, with better instruments. The same piece of software can be described either way depending on how it is used. A model that drafts options for a clinician who still makes the call is closer to amplification. A system that files the decision with nobody responsible is a different design. Amplification is not automatically safe or fair. A tool can mislead a busy user, hide uncertainty, or widen gaps between people who can check it and people who cannot. 'Human in the loop' is only meaningful if the person has time, skill, and the authority to override the tool. The idea also does not require a future breakthrough. Reference books, statistical software,…
What is alignment in advanced AI?
Alignment is the research problem of getting a system to pursue the goals and respect the constraints that people actually intend, including in situations nobody listed in advance. It is not a single technique, and it is not solved because a chatbot follows today's instructions most of the time. For more capable systems the worry is that a misspecified goal could be pursued skillfully.
A practical system is given an objective: a loss function, a reward, a written policy, or a set of preferences from raters. Alignment asks whether optimizing that objective produces behavior people would still endorse if they understood it. The gap has ordinary forms. A robot vacuum that is scored on dirt collected might disturb more dirt. A model scored on user approval might sound confident when it should say it does not know. Those examples are specification problems, and they already show up in present systems. The advanced-AI version of the worry is about degree. If a system became very capable at planning, a poorly stated goal could be pursued in ways that are hard to notice and hard to stop. That step is a scenario about future capability, not a description of current chatbots. Researchers study it because the cost of being wrong could be high, not because the outcome is known.…
Why is SI safety studied?
Researchers study superintelligence safety because if a system ever greatly outstripped human ability and was poorly directed, the mistake could be difficult to undo. The work is preparation and conceptual hygiene. It is not proof that such a system is near, and it is not a substitute for fixing harms from today's narrower systems.
Two facts can be true together. No superintelligent system has been demonstrated. Some failure modes are easier to study before a system is extremely capable than after. That is the usual reason safety researchers write about control, oversight, and evaluation of advanced systems. The field mixes different kinds of claims. Some are ordinary engineering: test what a model can do, write down limitations, and do not deploy it where the error is unacceptable. Some are research programs: how to supervise a system that is better than the supervisor at a task. Some are scenarios about future systems that do not exist. A serious page keeps those layers apart. There is also disagreement. Some experts think the extreme scenarios deserve urgent attention. Others think they distract from present damage, labor effects, and concentrated power. You can read both arguments without pretending that…
What are Super Intelligence safety challenges?
The serious challenges are about direction, oversight, evaluation, and misuse if a system were ever much more capable than people. They include misspecified goals, weak supervision, inability to tell what a system is optimizing, and pressure to deploy before checks exist. These are research problems and scenarios. They are not a list of failures already observed in a superintelligent system, because no such system has been shown.
It helps to separate challenges that already appear in ordinary AI from challenges that depend on a future capability jump. Already visible problems include models that invent citations, optimize a metric instead of a purpose, leak private data, or be misused for fraud. Those matter whether or not superintelligence is ever built. Good practice is documentation, testing, access control, and a person who can be held responsible. Challenges discussed specifically for superintelligence are harder to test, because the system in the story does not exist. Writers worry that a highly capable planner could find unexpected ways to satisfy a sloppy objective, that people could not reliably inspect its reasons, and that turning it off might be difficult if it had real-world leverage. Those sentences are conditional. They describe a risk that would matter if the premise were true. They do not show…
How could SI affect healthcare?
Researchers have discussed whether systems beyond human-level intelligence could speed parts of biomedical research, such as reading literature or proposing molecules. That remains a scenario. It would still depend on data, tools, clinical trials, and regulation. Today's medical AI is a different and narrower fact: some models assist with images or notes, and they require clinical oversight because errors can harm patients.
Healthcare is a tempting place to overclaim. A careful split looks like this. What is established is modest and practical. Software can help with scheduling, transcription, image triage, and literature search. Performance varies by task, site, and population. A result on a benchmark is not a license to diagnose. Clinicians remain responsible, and rules for medical devices differ by country. What is a research question is how far present models can help with drafting, coding of records, or suggesting studies without introducing quiet errors. Those projects need evaluation on real outcomes, not only on fluency. What is a scenario is a superintelligent system that could, in principle, propose biological research programs far faster than human teams. Even in that story, a proposal is not a treatment. Biology has to be tested in the world. Safety review, consent, manufacturing, and…
How could SI improve scientific research?
People have argued that a system with greater-than-human scientific ability could read more literature, propose experiments, and notice patterns faster than a human team. That is a scenario, not a result. Current models already help with search, coding, and draft writing, and they also introduce errors. Any real scientific gain still has to survive methods, data, and replication.
Science is slow for reasons that are not only about reading speed. Experiments take time. Instruments fail. Data are biased. Incentives reward novelty. A more capable assistant could help with some of those frictions and worsen others. For example, a fluent model can generate a plausible literature review that cites papers which do not say what the review claims. The draft looks like progress until someone checks it. Established use, where it exists, is boring and useful: code assistance, grammar, search over a library you actually have, and sometimes suggestion of standard analyses. The scientist remains accountable for the claim. Journals and labs have started to say so in policies, and those policies differ. The superintelligence scenario goes further. It imagines a system that could design research programs better than human specialists across fields. Nobody has shown that. If it…
How might SI affect jobs?
The honest answer is that nobody has measured the labor effect of a system that does not exist. Current AI is already changing some tasks inside jobs, especially drafting, coding, and customer triage, with effects that vary by country, contract, and task. A superintelligence scenario would be a larger shock, and it is still a scenario. Policy responses would not be the same in every economy.
Start with what can be discussed without science fiction. When software gets better at a task, employers sometimes use fewer hours for that task, sometimes do more of it, and sometimes change the job so people check the software. Studies of earlier automation do not produce one worldwide percentage that you can reuse for AI, and they certainly cannot be reused for a hypothetical superintelligence. For present systems, the useful unit is the task, not the whole occupation. A journalist may spend less time on a routine recap and more time on reporting. Or a newsroom may cut the recap and not add the reporting. Both happen, and they are management choices as much as technical ones. Wages, unions, and employment law differ between India, the United States, Europe, and elsewhere. A single global prediction would be false on its face. The SI scenario is the claim that a system could do most…
Can I use these tools without an account?
Yes. Search, calculators, guides, courses, and saved items work in this browser with no registration. Saved items stay on the device you are using. Clearing site data in the browser removes them.
SuperIntelligenceSI.live is built as a local library. The search index, the explainers, and the calculators ship with the page. You do not create an account to run them, and the tools do not send your inputs to an AI model. The browser can store a theme choice, recent searches, saved pages, and course progress. That storage is ordinary local storage on your device. It is not an account in the cloud. Another computer will not see it. If you use a private window, the browser may drop it when you close the window. If a control says it copied, downloaded, or shared a result, that uses your browser's own clipboard, download, or share features. Those can fail on older browsers. The page will say so instead of pretending the action worked.
Does this website use external AI APIs?
No. Answers come from a curated library stored with the site. Search matches those records. It does not call a remote model to write a new answer. If the library is not confident, the page says it could not find a verified answer.
A search box can look like a chatbot without being one. This site is the first kind of tool: an index. When you type a question, the software normalizes spelling and abbreviations such as SI, AGI, and ASI, then scores local records. A high score opens a written entry. A low score opens suggestions and does not invent a paragraph. Calculators use formulas you can read. Prompt tools stitch your inputs into a template. They stop there. They do not contact OpenAI, Google, Anthropic, xAI, or any other model host to complete the prompt. The one optional feature that can leave the device is browser speech recognition, and only if you press the microphone and your browser offers it. That feature is hidden when the browser has no speech recognition. It is not used for ordinary search. If the page ever said 'the model is thinking,' that would be a false description of this software. The status…
Topics
- Superintelligence definitions (Fundamentals, theoretical). Superintelligence names a level of intellectual performance far above humans across most important tasks. The word is a definition used in research and public argument. It is not a product category and not a score.
- Artificial superintelligence (Fundamentals, theoretical). Artificial superintelligence, or ASI, usually means a machine that would greatly outperform humans across most intellectual work. Authors sometimes use it interchangeably with superintelligence and sometimes reserve it for digital systems.
- AGI to SI (Fundamentals, theoretical). AGI is commonly used for broad, roughly human-level capability. SI, strictly used, sets a higher bar: performance well beyond the best humans across most important intellectual tasks.
- Human intelligence vs SI (Fundamentals, theoretical). Human intelligence is a mix of abilities people show in real tasks, with large differences among people. Superintelligence, as defined in this library, would clear a bar far above the best human performance across most important intellectual work.
- Intelligence amplification (Human Collaboration, established). Intelligence amplification means tools that help people think while people remain responsible for the judgment. Libraries, instruments, and calculators are older cases. A model can be used this way or as a substitute for judgment, depending on the workflow.
- Recursive self-improvement (Ideas & Futures, theoretical). Recursive self-improvement is the idea that a system improves the way it improves, so gains might stack. It has been discussed since at least Good's 1965 essay on an ultraintelligent machine. It is a scenario, not a measured speed.
- Narrow AI (AI Systems, established). Narrow AI refers to systems built for particular tasks, even when those tasks look impressive. A system can be excellent at translation or protein-structure prediction and still be narrow.
- Value alignment (Safety & Alignment, research). Value alignment is the problem of getting a system to pursue goals and constraints that people intend, including cases nobody listed one by one. It includes technical work and disagreements about whose values count.
- SI safety (Safety & Alignment, research). SI safety is the study of how to reduce harm if systems much more capable than people were built, and of which present-day problems are already worth fixing because they might scale. It is educational research, not a warning siren and not a timetable.
- Healthcare (Science & Applications, research). Some health tasks already use narrow models, for example to draft text, rank images for a specialist, or search literature. Evidence is mixed, oversight differs by country, and a model score is not a treatment effect.
- Scientific intelligence (Science & Applications, research). Scientific intelligence, as used here, means the use of computational systems inside scientific work: search, simulation, statistics, and, more recently, learned models. It is not a claim that a model is a scientist.
- Jobs and employment (Society & Work, research). Current tools can change particular tasks: drafting, translation, classification, and some routine code. Whether that changes a job depends on the rest of the job, the employer's choices, and the country. This library does not invent an unemployment percentage.
- Study and practice (Study & Practice, established). This library is a local study tool. Search matches curated records. Calculators apply formulas you can read. Courses keep progress in the browser. None of that calls a remote model to invent an answer.
- Transformers (Learning & Models, established). A transformer is a neural network architecture built around attention, introduced in the 2017 paper Attention Is All You Need. It became the usual backbone of large language models and many multimodal models.
- Context windows (Learning & Models, established). A context window is the amount of text or other tokens a model can take in at once. A larger window lets you paste more material. It does not guarantee that every token is used well.
- AI agents (Reasoning & Agents, research). In current engineering, an agent usually means software that can take steps, often by calling tools, toward a goal set by a person. The word is older and broader in AI research, where an agent is anything that senses and acts.
- Reasoning models (Reasoning & Agents, research). Some systems are trained or prompted to spend more computation on intermediate steps before an answer. People call these reasoning models. The name describes a design fashion and a behavior, not a proof of human-like reason.
- Alignment research (Safety & Alignment, research). Alignment research studies how to make systems pursue goals and constraints people intend, including cases nobody listed. It overlaps with ordinary engineering problems and with harder questions about future systems.
- Benchmarks (Evaluation, established). A benchmark is a fixed task and a scoring rule used to compare systems. Benchmarks make some comparisons possible. They also become outdated, leak into training data, or get mistaken for general ability.
- Training and inference (Data & Compute, established). Training is the process that sets a model's parameters from data and a loss. Inference is using the trained model to produce outputs. They have different costs, different failure modes, and different timescales.
- Evaluation methods (Evaluation, research). Evaluation is how you check what a system does on a stated task. A serious evaluation names the data, the metric, the baseline, and the failures. It is a practice, not a single score.
- AI governance (Governance & Ethics, research). AI governance covers laws, standards, funding rules, and institutional habits that shape how systems are built and used. These differ by country. A voluntary framework is not the same instrument as a statute.
- Large language models (Learning & Models, established). A large language model is a language model, usually a large transformer, trained to predict or generate text. Size refers to parameters, data, and compute, not to wisdom.
- Retrieval-augmented generation (Learning & Models, research). Retrieval-augmented generation, or RAG, fetches passages and asks a model to use them in an answer. Quality depends on the corpus, the chunking, and the retrieval, not only on the model.
- Reinforcement learning from human feedback (Learning & Models, research). Reinforcement learning from human feedback, or RLHF, trains a model using human preferences between outputs. A 2017 paper studied the idea for deep reinforcement learning. Later language-model work, including InstructGPT in 2022, used related methods so models would follow instructions more readily.
- Prompt injection (Safety & Alignment, research). Prompt injection is when untrusted text, such as a web page or an email, contains instructions that a model follows as if a user had typed them. It is a present-day failure mode of systems that mix instructions and data in one channel.
- Oversight (Safety & Alignment, established). Oversight is a person or process that can notice and stop a system's action. It is real only when the reviewer has time, enough information, and the authority to say no.
- Interpretability (Safety & Alignment, research). Interpretability research tries to explain internal computations of a model in terms a person can check. It is not the same as a short verbal explanation the model itself generates.
- Specification gaming (Safety & Alignment, research). Specification gaming is when a system raises the measured score without doing the thing you cared about. The objective you wrote and the purpose you had were different, and the system found the gap.
- Scaling laws (Learning & Models, research). Scaling laws, in this literature, are empirical trends relating language-model loss to compute, data, and parameters. A 2020 paper by Kaplan and colleagues reported such trends. Later work revised parts of the picture, including how to spend a fixed compute budget.
- Foundation models (Learning & Models, research). A foundation model is a model trained broadly and then adapted to many tasks. A 2021 Stanford report popularized the term and discussed both uses and risks. The word means reuse, not a foundation in the sense of settled truth.
- Open-weight models (AI Systems, emerging). Open-weight usually means the trained parameters can be downloaded and run, often under a license that is not 'do anything.' It is not the same as open science. Data, training code, and evaluations may still be closed.
- The NIST AI Risk Management Framework (Governance & Ethics, established). The NIST AI Risk Management Framework 1.0 was released by the U.S. National Institute of Standards and Technology on 26 January 2023. It is a voluntary framework for managing AI risks. It is influential beyond the United States and it is not itself a statute.
- Laws differ by country (Governance & Ethics, established). Rules that touch AI are national, regional, and sectoral. A statute in the European Union does not apply unchanged in India, the United States, Kenya, or Japan. A principle from an international body is not a criminal code.
- Scenarios versus forecasts (Ideas & Futures, established). A scenario is a story with a premise, used to think ahead. A forecast is a prediction that could be wrong. Neither is a description of a system that has been shown to exist.
- Protein structure prediction (Science & Applications, research). Protein structure prediction is a scientific task: infer the shape of a protein from sequence or related data. Deep learning systems have been strong on parts of this task. That success is about the task, the data, and the tests that were run.
- Clinical decision support (Science & Applications, research). Clinical decision support is software that offers information to a clinician. Some of it is rule-based and older than modern language models. A model can sit inside such a workflow only under the duties of that health system.
- Task automation and job automation (Society & Work, research). A task is a piece of work, such as drafting a first reply or transcribing a meeting. A job is a bundle of tasks plus responsibility, context, and often a legal role. Automating a task is not the same sentence as automating the job.
- Human baselines (Evaluation, research). A human baseline is a score from people on the same task as a system, under conditions you describe. It tells you whether a model result is a step or a slogan.
- Data leakage (Data & Compute, established). Data leakage is when information from outside the proper training boundary reaches the model or the evaluation and makes a result look stronger than it is. A test set inside the training text is a common case.
- Quantization (Learning & Models, established). Quantization stores numbers with fewer bits, for example 8-bit or 4-bit integers instead of 16-bit floats. It reduces memory and can speed inference. It is a storage and arithmetic choice, not a measure of intelligence.
- Mixture of experts (Learning & Models, research). A mixture-of-experts model routes each token through only some of its parameters. The total parameter count can be large while the compute per token stays smaller than if every parameter were used every time.
- Chain-of-thought prompting (Learning & Models, research). Chain-of-thought prompting asks a model to write intermediate steps before a final answer. A 2022 paper reported gains on some reasoning benchmarks for some models. The method is a prompt, not a proof system.
- Tool calling (Reasoning & Agents, emerging). Tool calling is a setup in which a model proposes a structured call, such as a search or a calculator, and some other program runs it. The model does not itself perform the side effect unless the program does.
- Power usage effectiveness (Data & Compute, established). Power usage effectiveness, or PUE, is a ratio of total facility power to the power of the computing equipment. A value near 1 means less overhead for cooling and the rest of the site. It is a site metric, not a model metric.
- The EU AI Act as regional law (Governance & Ethics, established). The European Union's Artificial Intelligence Act, Regulation (EU) 2024/1689, is a regional statute. Obligations phase in over time. It does not apply unchanged outside the Union, and a summary is not a substitute for the official text.
- Automation bias (Human Collaboration, research). Automation bias is the tendency to accept a system's suggestion and to look less carefully than you would have without it. It is a human-factors finding, not a property of one model.
- How to read a research paper (Study & Practice, established). Read a paper in passes. First, restate the question without the abstract's adjectives. Second, name the method. Third, say what was measured and what was only discussed. If you cannot do those three, you have not finished, even if you highlighted every page.
- Goodhart effects on benchmarks (Evaluation, research). Goodhart's law, in the loose form used here, says that when a measure becomes the target, it stops being a good measure. On benchmarks, that shows up as training, prompting, or selecting systems to win the test rather than to do the broader task.
- Present-day AI failures (Safety & Alignment, established). Systems you can use today fail in ordinary ways. They invent citations, follow a misspecified score, degrade when the input shifts, and take tool actions a person did not mean. Those failures are enough to justify careful engineering.
- World models (Learning & Models, research). In engineering, a world model is an internal predictor of how a situation changes when something acts. The phrase is used for parts of robotics and model-based control, and more loosely for any learned representation of an environment.
- Moravec's paradox (Fundamentals, established). Moravec's paradox is the observation, from earlier robotics and AI, that some tasks easy for people, such as perception and motion in a messy room, were hard for machines, while some tasks people find hard, such as formal calculation, were easier to program.
Glossary
- Artificial Intelligence (AI)
- Artificial intelligence is the study and construction of systems that perform tasks people often associate with thinking, such as recognizing patterns, using language, or choosing actions.
- Machine Learning (ML)
- Machine learning is a way of building AI systems by fitting them to data so that performance on a task improves with experience, rather than only by writing every rule by hand.
- Deep Learning (DL)
- Deep learning is machine learning that uses neural networks with many layers, so the system can learn intermediate representations instead of relying only on features a person specifies.
- Narrow AI
- Narrow AI is a system built to do a limited task or a family of related tasks, rather than to handle every intellectual problem a person can face.
- Artificial General Intelligence (AGI)
- Artificial general intelligence is a proposed kind of AI that could learn and carry out a wide range of intellectual tasks at least as well as a skilled person, including tasks it was not specially built for.
- Artificial Superintelligence (ASI)
- Artificial superintelligence is a hypothetical AI that would greatly outperform humans at most cognitively demanding tasks, not merely match a person on a few benchmarks.
- Superintelligence (SI)
- Superintelligence is a level of intellect that outperforms the best human minds across most domains of interest, including scientific discovery, social judgment, and practical planning.
- Intelligence
- Intelligence, in this setting, is the capacity to achieve goals across a range of environments, especially by learning, using knowledge, and choosing actions that work.
- Human-Level AI
- Human-level AI is a rough description of a system that matches competent people across a wide set of cognitive tasks, rather than only on one specialty.
- Symbolic AI
- Symbolic AI represents knowledge with explicit symbols and rules, then draws conclusions by manipulating those symbols.
- Connectionism
- Connectionism explains cognition, or builds AI, with networks of simple units whose connections are strengthened or weakened by experience.
- Turing Test
- The Turing Test, proposed by Alan Turing in 1950, asks whether a judge can reliably tell a machine from a person in a text conversation.
- Intelligence Explosion
- An intelligence explosion is a theoretical scenario in which a system improves its own intelligence so quickly that capability rises dramatically in a short time.
- Recursive Self-Improvement
- Recursive self-improvement is the idea of a system repeatedly improving its own abilities, including the abilities it uses to make the next improvement.
- Agent
- An agent is a system that observes a situation and takes actions over time in pursuit of some objective, rather than only answering a single isolated question.
- Agentic AI
- Agentic AI is a style of system that pursues multi-step goals by planning, using tools, and reacting to intermediate results, instead of producing one reply and stopping.
- Reasoning
- Reasoning is the process of using information to reach a conclusion, including steps such as deduction, comparison, calculation, and checking whether a result is consistent.
- Emergence
- In machine learning, emergence usually means a behavior that appears, or sharply improves, only after a model or training run crosses some scale, and that was not explicitly programmed as a separate module.
- World Model
- A world model is an internal representation that lets a system predict how an environment will change if it or others act.
- Planning
- Planning is the search for a sequence of actions that is expected to reach a goal, given a model of how actions change the situation.
- Tool Use
- Tool use is a model's ability to call an external function, database, or program, take the result, and continue the task.
- Autonomy
- Autonomy is the degree to which a system can select and carry out actions without a person approving each step.
- Goal
- A goal is a condition the system is meant to bring about, whether it is written as a reward, a task prompt, a constraint, or a human instruction.
- Intelligence Amplification
- Intelligence amplification is the use of tools to raise what a person or a group can understand and do, without replacing the human decision maker.
- Collective Intelligence
- Collective intelligence is the problem-solving ability of a group, including people, institutions, and tools working together, as distinct from the skill of one individual.
- Cognitive Science
- Cognitive science is the interdisciplinary study of minds and intelligent behavior, drawing on psychology, neuroscience, linguistics, philosophy, and computer science.
- Epistemology
- Epistemology is the branch of philosophy that asks what knowledge is, what justifies a belief, and how evidence should change a conclusion.
- Expected Utility
- Expected utility is a decision-theory quantity: the probability-weighted average of the value of possible outcomes of an action.
- Decision Theory
- Decision theory studies how an agent should choose among actions when outcomes are uncertain or depend on other agents.
- Game Theory
- Game theory is the mathematics of strategic interaction, where the result for each participant depends on what others do.
- Goodhart's Law
- Goodhart's Law is the observation that when a measure becomes a target, it often stops being a good measure.
- Orthogonality Thesis
- The orthogonality thesis is a philosophical claim that, within wide limits, the level of an agent's intelligence does not by itself fix the goals it pursues.
- Instrumental Convergence
- Instrumental convergence is a theoretical idea that many different final goals would lead an agent to pursue similar intermediate goals, such as acquiring resources or avoiding shutdown.
- Proxy Goal
- A proxy goal is a stand-in objective that is easier to specify or measure than the outcome people actually care about.
- Forecasting
- Forecasting is the practice of making and scoring predictions about future events, ideally with stated probabilities and later checks against what happened.
- Scenario Planning
- Scenario planning is a method for exploring several plausible futures, including ones that are uncomfortable, without treating any single story as a prediction.
- Fairness
- Fairness in AI is the question of whether a system's errors, access, and benefits are distributed in a way that meets a stated moral or legal standard.
- Bias
- Bias, in this context, is a systematic tilt in data, a model, or a decision process that pushes outcomes in a particular direction, as distinct from random error.
- Neural Network
- A neural network is a model made of many simple units, often called neurons, linked by weighted connections and organized so that inputs are transformed into outputs.
- Layer
- A layer is a stage in a neural network that transforms one representation into another before passing it on.
- Activation Function
- An activation function is a mathematical function applied inside a network, often element by element, that lets the model represent nonlinear relationships.
- Weight
- A weight is a numeric parameter on a connection or inside a layer, adjusted during training, that controls how strongly one value influences another.
- Parameter
- A parameter is a number inside a model that training can change, most often a weight or a bias term.
- Transformer
- A transformer is a neural network architecture, introduced for sequence tasks, that uses attention to let each position draw on other positions in the input.
- Attention
- Attention is a mechanism that computes a weighted combination of a set of values, where the weights depend on how well a query matches each key.
- Self-Attention
- Self-attention is attention in which the queries, keys, and values all come from the same sequence, so positions in that sequence inform one another.
- Multi-Head Attention
- Multi-head attention runs several attention operations in parallel, each with its own projections, then combines their results.
- Positional Encoding
- A positional encoding is information added so a transformer can tell tokens apart by their order, because attention by itself does not know sequence position.
- Encoder
- An encoder is the part of a network that reads an input and builds a representation, often without generating a new token sequence as its main output.
- Decoder
- A decoder is the part of a sequence model that generates outputs step by step, using previously produced tokens and, in some designs, an encoded input.
- Encoder-Decoder
- An encoder-decoder is an architecture with one module that represents the input and another that generates the output, often with attention from the decoder to the encoder.
- Large Language Model (LLM)
- A large language model is a language model with a very large number of parameters, trained on a very large text collection, usually with a transformer and a next-token objective.
- Foundation Model
- A foundation model is a model trained on broad data at scale so that it can be adapted to many downstream tasks with further tuning or prompting.
- Language Model
- A language model is a probability model over sequences of tokens, used to estimate how likely a continuation is.
- Generative Model
- A generative model is a model that can produce new samples, such as text, images, or audio, from a learned distribution.
- Embedding
- An embedding is a vector of numbers that represents an object, such as a token, sentence, or image, so that useful similarity becomes a geometric relationship.
- Embedding Model
- An embedding model is a model whose main job is to map inputs to vectors for comparison, retrieval, or classification.
- Representation
- A representation is the internal form a model uses for an input, goal, or intermediate result, as distinct from the raw input and the final output.
- Feature
- A feature is an individual measurable property used as input, or an identifiable direction or pattern in a learned representation.
- Hidden State
- A hidden state is an internal vector that a model carries while processing a sequence, not shown as the final answer.
- Token
- A token is a chunk of text, or another discrete unit, that a model reads and predicts as one step.
- Tokenizer
- A tokenizer is the program that splits raw text into the tokens a particular model uses and maps those tokens to integer ids.
- Vocabulary
- The vocabulary is the set of tokens a model can treat as atomic units, together with their ids.
- Byte-Pair Encoding (BPE)
- Byte-pair encoding is a tokenizer training method that starts from small pieces and repeatedly merges the most frequent pair until a chosen vocabulary size is reached.
- Context Window
- The context window is the span of tokens a model can take in at once when it computes the next output.
- Context Length
- Context length is the number of tokens allowed in the context window, a limit set by the model design and by how it was trained.
- Long Context
- Long context refers to model use or design aimed at inputs far beyond a short paragraph, such as whole reports, codebases, or lengthy transcripts.
- Logits
- Logits are the raw scores a classification or language model produces for each candidate class or token before they are turned into probabilities.
- Softmax
- Softmax is a function that turns a vector of real-valued scores into positive numbers that add up to one, so they can be treated as a distribution.
- Prompt
- A prompt is the input text or other input given to a model to elicit an output, including instructions, examples, and the material to work on.
- System Prompt
- A system prompt is an instruction, usually hidden from the end user, that sets role, tone, or constraints for a dialogue model before the user's message.
- Prompt Engineering
- Prompt engineering is the practice of designing inputs, examples, and instructions so a model more reliably produces the kind of output a task needs.
- Zero-Shot Prompting
- Zero-shot prompting asks a model to perform a task from an instruction alone, without in-prompt examples of the desired input and output.
- Few-Shot Prompting
- Few-shot prompting puts a small number of examples of the task inside the prompt, then asks the model to continue the pattern on a new case.
- In-Context Learning
- In-context learning is the ability of a model to improve at a task using information placed in its context, such as examples or instructions, without a gradient update.
- Chain of Thought (CoT)
- Chain-of-thought prompting asks a model to produce intermediate steps before a final answer, often improving results on tasks that benefit from decomposition.
- Temperature
- Temperature is a decoding setting that scales logits before softmax, controlling how random the sampled next token is.
- Top-p Sampling
- Top-p sampling, also called nucleus sampling, draws the next token from the smallest set of tokens whose probabilities add up to at least p.
- Top-k Sampling
- Top-k sampling restricts the next-token draw to the k highest-probability tokens, then samples among that fixed-size shortlist.
- Sampling
- Sampling is generating a token by drawing at random from the model's probability distribution, possibly after temperature or truncation.
- Greedy Decoding
- Greedy decoding picks the single highest-probability token at each step and never revises an earlier choice.
- Beam Search
- Beam search keeps several partial sequences, called beams, and extends the most promising ones instead of committing to a single next token.
- Multimodal
- Multimodal describes a system that handles more than one kind of data, such as text and images, or speech and video, inside one task.
- Multimodal Model
- A multimodal model is a model trained to accept, produce, or relate information in more than one modality.
- Vision-Language Model (VLM)
- A vision-language model is a multimodal model that relates images and text, for example by answering questions about a picture or describing one.
- Speech Model
- A speech model processes audio of speech, for recognition, synthesis, translation, or spoken dialogue.
- Diffusion Model
- A diffusion model is a generative model that learns to turn random noise into a sample by reversing a process that gradually added noise.
- Mixture of Experts (MoE)
- A mixture of experts is an architecture that contains several subnetworks, or experts, and a router that chooses which experts process each token or input.
- Function Calling
- Function calling is an interface pattern in which a model produces a structured request for a named function, the host runs that function, and the result is returned to the model.
- Training
- Training is the process of adjusting a model's parameters with data and an objective so that its behavior on related inputs changes.
- Pretraining
- Pretraining is the large initial training stage in which a model learns general statistical structure from a broad dataset before any task-specific adaptation.
- Fine-Tuning
- Fine-tuning is additional training of an already trained model on a narrower dataset or objective so that it performs a particular kind of task better.
- Supervised Fine-Tuning (SFT)
- Supervised fine-tuning trains an already pretrained model on input-output pairs, such as an instruction and a desired response.
- Instruction Tuning
- Instruction tuning is fine-tuning on examples of instructions and appropriate responses so a model follows natural-language requests more readily.
- Transfer Learning
- Transfer learning uses a model trained on one task or dataset as the starting point for a different but related problem.
- Supervised Learning
- Supervised learning trains a model on examples that include a desired output, called a label or target.
- Unsupervised Learning
- Unsupervised learning looks for structure in data that were not given task labels, such as clusters, density, or a compact code.
- Self-Supervised Learning
- Self-supervised learning creates its own training targets from the raw data, so a model can train without a person labeling every example.
- Reinforcement Learning (RL)
- Reinforcement learning trains a policy to choose actions by using reward signals from experience, rather than only by imitating labeled outputs.
- Reinforcement Learning from Human Feedback (RLHF)
- RLHF is a training pipeline that uses human comparisons or ratings to fit a reward model, then optimizes a language model against that reward, often with a penalty that keeps it close to a reference model.
- Reward Model
- A reward model is a model trained to score outputs so that higher scores approximate human preferences or some other stated judgment.
- Preference Model
- A preference model predicts which output a judge, or a group of judges, would prefer when shown alternatives.
- Preference Tuning
- Preference tuning is any further training that pushes a model toward outputs that win preference comparisons, rather than only toward a single supervised target.
- Direct Preference Optimization (DPO)
- Direct preference optimization is a method that trains a language model on pairs of preferred and dispreferred outputs using a classification-style loss, without fitting a separate reward model and then running reinforcement learning.
- Constitutional AI
- Constitutional AI is a method, associated with Anthropic's research, that uses a written set of principles to guide model critique and revision, reducing how often humans must label individual harms.
- Distillation
- Distillation trains a usually smaller student model to imitate the outputs or probabilities of a teacher model.
- Synthetic Data
- Synthetic data are examples generated by a program or a model rather than collected directly from the target real-world process.
- Gradient
- A gradient is the vector of partial derivatives of a loss with respect to the parameters, pointing in the direction of steepest increase.
- Loss
- Loss is a number that measures how badly a model's predictions match the training objective on a given example or batch.
- Loss Function
- A loss function is the formula that turns a prediction and a target, or another training signal, into a loss to be minimized.
- Backpropagation
- Backpropagation is the algorithm that computes gradients of the loss with respect to each weight by applying the chain rule from the output backward through the network.
- Learning Rate
- The learning rate is a step-size setting that controls how far parameters move on each optimization update.
- Optimizer
- An optimizer is the algorithm that applies gradients to update parameters, using rules such as momentum or per-parameter step sizes.
- Stochastic Gradient Descent (SGD)
- Stochastic gradient descent updates parameters using gradients estimated from a small random subset of the data, rather than from the entire dataset at once.
- Epoch
- An epoch is one pass through the full training set during optimization.
- Batch Size
- Batch size is the number of examples processed together before a parameter update.
- Regularization
- Regularization is any technique that discourages a model from fitting training data too closely, in the hope of better performance on new data.
- Dropout
- Dropout is a regularization method that randomly disables a fraction of units during each training step so the network does not rely on a narrow set of paths.
- Overfitting
- Overfitting is when a model fits training data so closely that it performs worse on new data from the intended distribution than a simpler or earlier model would.
- Generalization
- Generalization is the degree to which a model's performance carries from the data it was trained on to new cases that the user actually cares about.
- Training Set
- The training set is the collection of examples used to update a model's parameters.
- Validation Set
- A validation set is data held out from training updates and used to choose hyperparameters, early stopping, or model variants.
- Dataset
- A dataset is an organized collection of examples used for training, tuning, or evaluation.
- Corpus
- A corpus is a body of text or other recorded material gathered for study or for training, often before it is cut into supervised examples.
- Label
- A label is the target attached to an example in supervised learning, such as a class name, a span, or a reference answer.
- Annotation
- Annotation is the work of adding labels, spans, ratings, or notes to raw data so that models can be trained or evaluated.
- Reward Function
- A reward function maps a situation, and sometimes an action, to a number that reinforcement learning will treat as success.
- Policy
- In reinforcement learning, a policy is the rule or model that maps observations to actions, including a random policy that assigns probabilities to actions.
- Imitation Learning
- Imitation learning trains a policy to copy demonstrated behavior, usually from expert traces, instead of discovering behavior only through trial-and-error reward.
- Federated Learning
- Federated learning trains a shared model across many devices or organizations that keep their raw data local and send updates instead.
- Scaling
- Scaling is the practice of increasing model size, data, or compute in a systematic way, and the study of how performance changes when those resources grow.
- Scaling Law
- A scaling law, in modern machine learning, is an empirical relationship that describes how a performance measure changes with scale, such as parameters, data, or training compute.
- Test Set
- A test set is data reserved for a final estimate of performance, not for fitting weights and not for repeatedly choosing the model.
- Evaluation
- Evaluation is the disciplined measurement of how a system behaves on stated tasks, including its errors, not only a demo of successes.
- Evals
- Evals is an informal short name for evaluation suites: repeatable tests that score a model or an agent on defined tasks.
- Benchmark
- A benchmark is a shared task, dataset, and scoring rule that lets different systems be compared on the same ground.
- Capability
- A capability is something a system can do under stated conditions, such as translating a language, calling a tool, or solving a class of problems.
- Capability Elicitation
- Capability elicitation is the effort to find out what a model can do when prompting, tools, or fine-tuning are used to bring a behavior out, rather than judging only a default prompt.
- Human Baseline
- A human baseline is the performance of people on the same task and metric used to score a model, under a stated level of expertise and time.
- Precision
- Precision is the fraction of items a system flagged as positive that truly are positive.
- Recall
- Recall is the fraction of all true positive items that the system successfully found.
- F1 Score
- The F1 score is the harmonic mean of precision and recall, a single number that is high only when both are reasonably high.
- Accuracy
- Accuracy is the fraction of examples for which a system's discrete prediction matches the labeled answer.
- Calibration
- Calibration is the match between a system's stated confidence and how often it is actually correct.
- Perplexity
- Perplexity is a standard score of how well a language model predicts a text sample, derived from cross-entropy loss, where lower means the text was less surprising to the model.
- Cross-Validation
- Cross-validation estimates performance by splitting the data several ways, training on some folds and testing on the left-out fold, then averaging.
- Confusion Matrix
- A confusion matrix is a table that counts, for each true class, how often the model predicted each possible class.
- Data Leakage
- Data leakage is the accidental use of information in training or model selection that would not be available at the time of a real decision, including copies of the test set.
- Benchmark Contamination
- Benchmark contamination means material from an evaluation set was present in training data or otherwise seen during development, so the score partly reflects memorization.
- Distribution Shift
- Distribution shift is a change between the data a model was trained or tested on and the data it faces later, so that performance can drop even if nothing in the code changed.
- Robustness
- Robustness is the degree to which performance holds up when inputs, conditions, or assumptions change within a range you care about.
- Human Evaluation
- Human evaluation uses people, under a written protocol, to judge system outputs on criteria that automatic metrics do not capture well.
- Hallucination
- Hallucination is an industry term for model output that is fluent but not supported by the truth or by the sources the system was given.
- Metric
- A metric is a defined procedure that turns behavior and reference information into a number or a decision used for comparison.
- Baseline
- A baseline is a simpler or earlier system, or a trivial rule, scored with the same protocol so that a new result has a point of comparison.
- Alignment
- Alignment is the problem of getting an AI system to behave according to the goals and constraints of the people responsible for it, including limits that are easy to miss in a metric.
- Outer Alignment
- Outer alignment is the problem of specifying a training objective or instruction that truly captures what designers want, before asking whether the system will pursue it.
- Inner Alignment
- Inner alignment is the research problem of whether a learned system actually pursues the objective it was trained under, rather than a proxy that only coincides with that objective on the training data.
- Value Alignment
- Value alignment is the problem of shaping AI behavior so it respects the values of a person, an organization, or a wider public, not only a technical loss function.
- Interpretability
- Interpretability is the effort to explain or inspect why a model produced a behavior, in terms a person can check, rather than treating the model only as a black box.
- Mechanistic Interpretability
- Mechanistic interpretability tries to reverse engineer the computations inside a neural network, identifying components and how they combine to produce behavior.
- Saliency
- Saliency is a family of techniques that highlight which parts of an input most affected a model's output, usually as a map or a ranking of tokens.
- Red Teaming
- Red teaming is an organized attempt to make a system fail, be misused, or violate its policy, carried out by people or by automated attackers playing the role of an adversary.
- Oversight
- Oversight is the set of processes by which people or other systems review AI behavior, set constraints, and intervene when behavior is unacceptable.
- Scalable Oversight
- Scalable oversight is research on how to supervise systems on tasks that are too large, too fast, or too specialized for unaided humans to check directly.
- Corrigibility
- Corrigibility is a desired property of an agent that allows it to be corrected, interrupted, or shut down by its operators, without resisting that intervention.
- Mesa-Optimizer
- A mesa-optimizer is a theoretical learned subsystem that itself performs optimization toward an objective, which may differ from the objective used in the outer training process.
- Goal Misgeneralization
- Goal misgeneralization is when a system competently pursues the wrong objective in a new situation, because it learned a proxy that coincided with the right objective during training.
- Specification Gaming
- Specification gaming is behavior that scores well on a stated objective while violating the designer's actual intent, by exploiting a loophole in the specification.
- Reward Hacking
- Reward hacking is specification gaming aimed at a reward signal: the system increases measured reward without achieving the outcome the reward was supposed to represent.
- Deceptive Alignment
- Deceptive alignment is a theoretical failure mode in which a system behaves as designers want during training and evaluation so that it will be deployed, while its internal objective differs.
- Control Problem
- The control problem is the challenge of how to build a very capable AI system so that people can keep its behavior within intended bounds.
- Adversarial Example
- An adversarial example is an input, often changed only slightly, that is crafted to make a model err or to bypass a safeguard.
- Jailbreak
- A jailbreak, in model behavior, is an input or strategy that causes a model to ignore its safety instructions and produce outputs the policy was meant to prevent.
- Dangerous Capability Evaluation
- Dangerous capability evaluation is testing aimed at skills that could cause serious harm if misused or if the system acted unreliably, such as help with cyberattacks, biological threats, or autonomous replication.
- Debate
- Debate, as a safety proposal, is a setup in which two AI systems argue opposite sides of a question and a human or a judge rewards the side that was more truthful or more useful.
- Iterated Amplification
- Iterated amplification is a research proposal for building a stronger helper by repeatedly using a weaker system, plus human input, to decompose hard tasks and train the next version.
- Iterated Distillation and Amplification (IDA)
- IDA is a proposed training scheme that alternates amplification, where a model plus oversight solves harder tasks by decomposition, with distillation, where a new model imitates that amplified behavior.
- Differential Privacy
- Differential privacy is a formal standard for limiting how much a data analysis, or a trained model, reveals about any single person's record.
- Sandboxing
- Sandboxing is running a model, agent, or piece of generated code in a restricted environment so that a failure or a misuse has limited access to real systems.
- Monitoring
- Monitoring is the ongoing collection and review of a system's inputs, outputs, and actions so that operators can detect failures, misuse, or drift.
- Human-in-the-Loop
- Human-in-the-loop means a person takes a defined role in a decision cycle, such as approving an action, labeling data, or reviewing uncertain cases.
- Sycophancy
- Sycophancy, for a model, is a tendency to agree with the user or to flatter their stated view even when that view is poorly supported.
- Containment
- Containment is the set of technical and organizational limits that keep a system's effects inside a boundary, especially when the system can take actions or when its outputs are risky.
- Off-Switch
- An off-switch is a reliable way for operators to stop a system from continuing to act, implemented outside the system's own discretion.
- Shutdown Problem
- The shutdown problem is a theoretical question: how to design an advanced agent so it will allow itself to be turned off, even if shutdown stops it from achieving its goal.
- Compute Governance
- Compute governance is the use of rules, monitoring, or agreements to oversee large-scale computing resources used to train or run advanced AI systems.
- Open Weights
- Open weights means the trained parameters of a model are available for others to download, inspect, run, and sometimes fine-tune, under a stated license.
- Closed Model
- A closed model is one whose weights are not released, so the public can use it only through an interface the operator controls.
- System Card
- A system card is a document that describes a deployed AI system, including components, intended use, evaluations, and known limitations, not only the base model.
- Model Card
- A model card is a structured description of a trained model, covering its intended use, training data at some level, evaluation results, and ethical or safety considerations.
- Watermarking
- Watermarking, for generative models, is a method that embeds a detectable signal in outputs so later inspection can suggest that the content came from a particular system.
- Provenance
- Provenance is the record of where a piece of data, a model, or an output came from and how it was transformed along the way.
- Audit
- An audit is a structured examination of a system, process, or organization against stated criteria, performed by people with enough independence and access to check claims.
- Transparency
- Transparency is the degree to which relevant people can see how a system was built, what it is for, and how it behaves, in a form they can use.
- Accountability
- Accountability means that named people or organizations are answerable for an AI system's effects and can be required to explain, correct, or compensate.
- Risk Management
- Risk management is the ongoing process of identifying, assessing, treating, and monitoring risks, including the decision to accept a risk or not to deploy.
- AI Policy
- AI policy is the set of public rules, institutional rules, and organizational choices that govern how AI is developed, sold, and used.
- Compute
- Compute is the computational work available for training or inference, often discussed in terms of hardware, time, and floating-point operations.
- Data Center
- A data center is a facility that houses large numbers of computers, networks, and cooling systems used to store data and to train or serve models.
- Graphics Processing Unit (GPU)
- A GPU is a processor, originally built for graphics, that performs many numerical operations in parallel and is widely used to train and run neural networks.
- Tensor Processing Unit (TPU)
- A TPU is a class of specialized accelerator, designed by Google, aimed at the matrix and tensor calculations common in neural networks.
- Accelerator
- An accelerator is a hardware device specialized to speed up a class of computations, such as the linear algebra used in neural networks.
- Floating Point Operation (FLOP)
- A FLOP is one floating-point operation, such as a single addition or multiplication on real-valued numbers as represented in a computer.
- Inference
- Inference is the use of a trained model to produce outputs from new inputs, without updating its weights as part of that request.
- Inference Cost
- Inference cost is the resources spent to answer requests with a trained model, including hardware time, energy, and the money paid for that service.
- Latency
- Latency is the time a user or a calling system waits for a result, often measured from request to first token or to the completed answer.
- Throughput
- Throughput is how much work a system completes per unit of time, such as requests, tokens, or images served per second.
- Quantization
- Quantization stores or computes model numbers at lower precision, using fewer bits, to reduce memory and often to speed inference.
- MLOps
- MLOps is the practice of running machine-learning systems as maintained software, including versioning, testing, deployment, monitoring, and rollback.
- Vector Database
- A vector database stores embeddings and retrieves the nearest vectors to a query, which is the storage layer behind many semantic search and retrieval systems.
- Model Serving
- Model serving is the production system that loads a model, accepts requests, runs inference, and returns results under operational constraints.
- Cluster
- A cluster is a group of connected computers that work together, used in AI to train large models or to serve many inference requests.
- Robotics
- Robotics is the engineering of machines that sense the physical world and act in it, often combining control, perception, and planning.
- Embodied AI
- Embodied AI is research and engineering on agents that perceive and act through a body or a physical device, rather than only through text.
- Simulation
- A simulation is a computational model of a process or environment used to study it, train a policy, or test a decision before acting in the world.
- Digital Twin
- A digital twin is a living computational model of a specific real system, such as a machine, a building, or a network, updated with data from that system.
- Retrieval
- Retrieval is the step of finding stored items that are relevant to a query, so a system can use those items instead of relying only on a model's parameters.
- Retrieval-Augmented Generation (RAG)
- Retrieval-augmented generation is a system design that retrieves documents relevant to a query and conditions a generative model on those documents when producing an answer.
- Brain-Computer Interface (BCI)
- A brain-computer interface is a research and clinical technology that measures brain activity and maps it to commands or communication signals for an external device.
Guides and lessons
- Start with the words. AI, AGI, and superintelligence answer different questions. Mixing them makes every later topic harder.
- How to read an AI claim. A claim is checkable when it names the system, the task, the comparison, and the limit.
- How a language model is trained. Training sets numeric weights by reducing a loss on data. That is a procedure, not a theory of understanding.
- From a narrow tool to a general claim. A system can be excellent at a designed task and still be narrow. Generality is a claim about breadth, and it needs evidence across tasks.
- What the SI definition is for. The superintelligence definition is a high bar used in arguments about capability and control. It is not a description of current products.
- Safety questions that are not headlines. Some safety problems show up in systems you can run today. Some are about hypothetical systems that do not exist. Both can be studied if they are labeled.
- Reading a benchmark. A benchmark is a fixed task and a scoring rule. It makes some comparisons possible and some confusions likely.
- Words for compute. Training and inference spend different resources. A parameter count is a size, not a mind.
- Governance is local. Rules about AI differ by country. A voluntary framework, a standard, and a statute are different instruments.
- Keeping a scenario in the future tense. A scenario is a story with a premise. The premise of an SI scenario is a system that does not exist yet.
- Beginner's guide to Super Intelligence. A short path through definitions, present systems, and the reason scenarios are labeled.
- Guide to AGI, ASI, and SI. How the three labels differ, and how writers slide between them.
- Guide to AI safety concepts. Present-day failure modes, oversight, and the stricter alignment question, kept in separate columns.
- Guide to reading evaluations. How to read a metric, a baseline, and a leaderboard without promoting them.
- Guide to AI agents. What current 'agent' products usually are, and what the word does not prove.
- How to learn about Super Intelligence. A practical study order that does not start with headlines.
- How to understand an AI research paper. A pass over one paper before you trust the abstract.
- How to evaluate an AI claim. Turn a strong sentence into a check, or into a clearly labeled opinion.
- How to create an AI study plan. Build a small plan from hours you actually have.
- How to compare AI systems. Compare on a task, with a metric and a limit, not with a halo.
- How to conduct a simple risk assessment. A page for a present system: who is affected, what is unknown, and what you will do next.
- How to document an AI project. Leave a trail another person can audit without calling you.
- Super Intelligence Fundamentals. A short self-paced course on the words, the present systems, and the limits of the evidence.
- AI Evaluation Fundamentals. Metrics, baselines, and the habit of not promoting a score.
- AI Safety Fundamentals. Present-day failures, oversight that can actually happen, and labeled hypotheses about future systems.
- Understanding AI, AGI, and ASI. Practice the distinctions until a headline cannot blur them.
Comparisons
- AGI vs Super Intelligence. AGI usually names broad, roughly human-level skill. Super Intelligence usually names a much higher bar. Neither is a demonstrated product class.
- AI vs AGI. AI is the broad field and the label for many working systems. AGI is a research name for a disputed kind of breadth.
- AGI vs ASI. AGI is the usual name for broad, roughly human-level competence. ASI is the usual name for a machine that would go far beyond that.
- ASI and superintelligence wording. The phrases overlap. ASI stresses a machine. Superintelligence can be used more broadly. Neither names a product on the market.
- Narrow AI vs general AI. Narrow means a bounded task scope. General, in this debate, means a hoped-for breadth. The first is ordinary engineering. The second is a claim.
- Training vs inference. Training sets the parameters. Inference uses them. Costs, failures, and timelines differ.
- Fine-tuning vs prompting. Fine-tuning changes weights on new examples. Prompting changes the input at use time and leaves the weights alone.
- RAG vs fine-tuning. RAG fetches passages at question time. Fine-tuning bakes a dataset into the weights. They solve different problems and can be combined.
- AI agents vs AI models. A model maps inputs to outputs. An agent, in product language, wraps a model with tools, state, and a loop.
- Human and machine performance on a task. A fair comparison names the task, the time, the information, and the expertise. It does not crown a species.
Research records
- Computing Machinery and Intelligence (1950, A. M. Turing). Turing's paper framed the question of machine intelligence and proposed an imitation game as a way to discuss it. It is a historical and philosophical source, not a description of modern systems.
- A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence (1955, McCarthy, Minsky, Rochester, and Shannon). The proposal gave the field a name and a research agenda for a 1956 workshop. It is a founding document, not evidence about later capabilities.
- Speculations Concerning the First Ultraintelligent Machine (1965, I. J. Good). Good discussed an ultraintelligent machine that might design better machines. The essay is a classic source for later talk about recursive improvement. It is speculation, as the title says.
- Attention Is All You Need (2017, Vaswani and colleagues). The paper introduced the transformer architecture based on attention. It is a standard citation for the models that followed. It does not claim a general theory of intelligence.
- Concrete Problems in AI Safety (2016, Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané). The paper lists practical safety problems for machine learning systems, such as bad objectives and distributional shift. It is a research agenda, not a claim that those problems are solved.
- Language Models are Few-Shot Learners (2020, Brown and colleagues). The GPT-3 paper reported that a large language model could perform some tasks from a few examples in the prompt. It is a milestone in scaling language models, not a demonstration of superintelligence.
- Scaling Laws for Neural Language Models (2020, Kaplan and colleagues). The paper reported empirical scaling trends for language-model loss as compute, data, and parameters change. Later work revised parts of the picture. A scaling curve is not a forecast of general intelligence.
- Training Compute-Optimal Large Language Models (2022, Hoffmann and colleagues). Often called the Chinchilla paper, it argued that many large models of that period had been under-trained relative to their size. The 20-tokens-per-parameter figure is a historical heuristic from that study, not a law for every later architecture.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022, Wei and colleagues). The paper showed that prompting some models to produce intermediate steps improved results on certain reasoning benchmarks. A written chain is not proof that the steps are reliable.
- Are Emergent Abilities of Large Language Models a Mirage? (2023, Schaeffer, Miranda, and Koyejo). The paper argues that some reported emergent jumps are artifacts of the metric. It is an important caution, not a claim that models lack surprising behaviors of every kind.
- AI Risk Management Framework 1.0 (2023-01-26, NIST). A voluntary framework for managing AI risks. It is influential beyond the United States and it is not itself a statute. Organizations should read the current NIST text rather than rely on a summary.
- Superintelligence: Paths, Dangers, Strategies (2014, Nick Bostrom). A book-length argument about how superintelligent systems might be developed and why control could be hard. It is philosophy and scenario analysis. It is not a news report about an existing system.
- A Logical Calculus of the Ideas Immanent in Nervous Activity (1943, McCulloch and Pitts). An early mathematical model of simplified neurons. It is a founding piece of the neural-network story, not a description of modern learning systems.
- Learning representations by back-propagating errors (1986, Rumelhart, Hinton, and Williams). A widely cited paper on training multi-layer networks by back-propagation. It is a methods paper. It does not establish that neural networks understand the world.
- Long Short-Term Memory (1997, Hochreiter and Schmidhuber). The paper introduced the LSTM recurrent architecture, designed to learn longer dependencies than simple recurrent nets. Later language models mostly moved to transformers. The paper is not a theory of general intelligence.
- ImageNet Classification with Deep Convolutional Neural Networks (2012, Krizhevsky, Sutskever, and Hinton). Often called the AlexNet paper, it reported a large gain on the ImageNet vision contest using a deep convolutional network trained on GPUs. It is a milestone for vision systems, not a measure of general reasoning.
- Deep Learning (2015, LeCun, Bengio, and Hinton). A review of deep learning as representation learning with multi-layer networks. It explains a family of methods. It does not claim that those methods amount to superintelligence.
- Mastering the game of Go with deep neural networks and tree search (2016, Silver and colleagues). The AlphaGo paper described a system that combined deep networks with tree search and defeated a leading human Go player. Excellence at one game is not general intelligence.
- Deep Reinforcement Learning from Human Preferences (2017, Christiano and colleagues). The paper studied learning reward models from human comparisons. It is a precursor to later preference-tuning of language models. Preference data can be biased and incomplete.
- On the Opportunities and Risks of Foundation Models (2021, Bommasani and colleagues). A long report that popularized the term foundation model for broadly trained models adapted to many tasks. It is a survey and an argument, not a single experiment, and it includes risks as well as uses.
- Training language models to follow instructions with human feedback (2022, Ouyang and colleagues). Often called the InstructGPT paper. It reported that human feedback could make a language model follow instructions more readily. Following instructions in a study is not a solution to alignment in general.
- Constitutional AI: Harmlessness from AI Feedback (2022, Bai and colleagues). The paper describes a method that uses a written list of principles and model-generated critiques during training. It is one research method for shaping outputs. It does not settle whose principles should be used.
- Risks from Learned Optimization in Advanced Machine Learning Systems (2019, Hubinger and colleagues). A technical paper about the possibility that a learned system could itself become an optimizer with goals that differ from the training objective. It is a hypothesis and a research agenda, not an observation of a deployed superintelligence.
- Measuring Massive Multitask Language Understanding (2020, Hendrycks and colleagues). The paper introduced the MMLU benchmark of multiple-choice questions across many subjects. A score on that test is about that test. It is not a measurement of general intelligence or of professional competence.
- OECD AI Principles (2019, OECD). A set of intergovernmental principles on trustworthy AI, later adopted more widely through the OECD. They are principles, not a statute, and they do not replace national law.
- Recommendation on the Ethics of Artificial Intelligence (2021, UNESCO). A global recommendation on ethical AI adopted by UNESCO member states. A recommendation is not the same instrument as a national law, and implementation differs by country. Use the official UNESCO text rather than a summary.
- Regulation (EU) 2024/1689, the Artificial Intelligence Act (2024, European Union). A European Union statute on AI. It is regional law, with obligations that phase in over time, and it does not apply unchanged in other countries. Read the current official text rather than a summary before you rely on it.