API Reference

Overview

Our REST API provides seamless access to Tavily Search, a powerful search engine for LLM agents, and Tavily Extract, an advanced web scraping solution optimized for LLMs.

The Tavily Search and Extract REST API can be accessed through the base URL: https://api.tavily.com/.

If you want to experiment with the API from your browser, head to our API Playground!

Endpoint: POST /search

Search for data based on a query.

Parameters

• query: string (required) - The search query you want to execute with Tavily.

• api_key: string (required) - Your Tavily API Key.

• search_depth: string (optional) - The depth of the search. It can be "basic" or "advanced". Default is "basic" unless specified otherwise in a given method.

• topic: string (optional) - The category of the search. This will determine which of our agents will be used for the search. Currently: only "general" and "news" are supported. Default is "general".

• days: int (optional) - The number of days back from the current date to include in the search results. This specifies the time frame of data to be retrieved. Please note that this feature is only available when using the "news" search topic. Default is 3.

• max_results: int (optional) - The maximum number of search results to return. Default is 5.

• include_images: bool (optional) - Include a list of query-related images in the response. Default is False.

• include_image_descriptions: bool (optional) - When include_images is set to True, this option adds descriptive text for each image. Default is False.

• include_answer: bool (optional) - Include a short answer to original query. Default is False.

• include_raw_content: bool (optional) - Include the cleaned and parsed HTML content of each search result. Default is False.

• include_domains: list<string> (optional) - A list of domains to specifically include in the search results. Default is [], which includes all domains.

• exclude_domains: list<string> (optional) - A list of domains to specifically exclude from the search results. Default is [], which doesn't exclude any domains.

Response

• answer: string - The answer to your search query.

• query: string - Your search query.

• response_time: float - Your search result response time.

• images: list - A list of query-related image URLs. If include_image_descriptions is set to True each entry will be a dictionary with url and description

• results: list - A list of sorted search results ranked by relevancy. Each result is in the following format:

  • title: string - The title of the search result URL.
  • url: string - The URL of the search result.
  • content: string - The most query related content from the scraped URL. We use proprietary AI and algorithms to extract only the most relevant content from each URL, to optimize for context quality and size.
  • raw_content: string - The parsed and cleaned HTML of the site. For now includes parsed text only.
  • score: float - The relevance score of the search result.
  • published_date: str (optional) - The publication date of the source. This is only available if you are using "news" as your search topic.

Example Usage

Request

{
    "api_key": "tvly-YOUR_API_KEY",
    "query": "Who is Leo Messi?",
    "search_depth": "basic",
    "include_answer": false,
    "include_images": true,
    "include_image_descriptions": true,
    "include_raw_content": false,
    "max_results": 5,
    "include_domains": [],
    "exclude_domains": []
}

Response

{
    "query": "Who is Leo Messi?",
    "follow_up_questions": null,
    "answer": null,
    "images": [
        {
            "url": "https://cdn.britannica.com/34/212134-050-A7289400/Lionel-Messi-2018.jpg",
            "description": "Lionel Messi, wearing a Barcelona jersey, is skillfully dribbling a soccer ball during a match."
        },
        {
            "url": "https://www.thefamouspeople.com/profiles/images/lionel-messi-2.jpg",
            "description": "A serious-looking athlete with a beard, wearing a light blue and white striped jersey associated with the Argentinian national team, appears to be focused on the game."
        },
        {
            "url": "https://sportsmatik.com/uploads/world-events/players/lionel-messi_1564492648.jpg",
            "description": "A focused Leo Messi, wearing a Barcelona jersey, displays a thoughtful expression during a match."
        },
        {
            "url": "https://xsportnet.com/wp-content/uploads/2021/03/d852182c8359ca2cdb4a5acff414514d_hi-res-159596853_crop_north.jpg",
            "description": "Lionel Messi is holding the Ballon d'Or trophy while wearing a Barcelona jersey, showcasing a moment of celebration and achievement."
        }
    ],
    "results": [
        {
            "title": "Leo Messi - Official FC Barcelona Website",
            "url": "https://www.fcbarcelona.com/en/card/2214377/leo-messi",
            "content": "Notifications Center\nNotifications Center\nFirst Team\nClub\nTickets & Museum\nCulers\nDownload the official FC Barcelona App\nLeo Messi\nLeo Messi's footballing career started in 1995 at Newell's Old Boys, where he played until the year 2000. Messi’s individual achievements are also unprecedented: six Ballon d’Or awards, six times Champions League top scorer, six times Golden Shoe winner, eight times ‘Pichichi’ (top scorer) in La Liga, Ballon d’Or winner at the 2014 World Cup, FIFA World Player of the Year in 2009 and FIFA The Best winner in 2019.\n He has also played in six Copa Américas (2007, 2011, 2015, 2016, 2019, 2021), losing in the final in both 2015 and 2016 to Chile on penalties before finally claiming a major honour for his country with the win over Brazil in the 2021 final. In the summer of 2021 the love affair between FC Barcelona and Leo Messi came to an end and the Barça number 10 brought an end to his career which has been him become a life long legend for FC Barcelona.\n Getting better every season, Messi and Barça won 35 trophies during the Argentine's time at the Club, including the six won in 2009 and the treble in 2015.\n",
            "score": 0.98492736,
            "raw_content": null
        },
        {
            "title": "Lionel Messi | Biography, Barcelona, PSG, Ballon d'Or, Inter Miami ...",
            "url": "https://www.britannica.com/biography/Lionel-Messi",
            "content": "In early 2009 Messi capped off a spectacular 2008–09 season by helping FC Barcelona capture the club’s first “treble” (winning three major European club titles in one season): the team won the La Liga championship, the Copa del Rey (Spain’s major domestic cup), and the Champions League title. Messi’s play continued to rapidly improve over the years, and by 2008 he was one of the most dominant players in the world, finishing second to Manchester United’s Cristiano Ronaldo in the voting for the 2008 Ballon d’Or. At the 2014 World Cup, Messi put on a dazzling display, scoring four goals and almost single-handedly propelling an offense-deficient Argentina team through the group stage and into the knockout rounds, where Argentina then advanced to the World Cup final for the first time in 24 years. After Argentina was defeated in the Copa final—the team’s third consecutive finals loss in a major tournament—Messi said that he was quitting the national team, but his short-lived “retirement” lasted less than two months before he announced his return to the Argentine team. Messi helped Barcelona capture another treble during the 2014–15 season, leading the team with 43 goals scored over the course of the campaign, which resulted in his fifth world player of the year honour.",
            "score": 0.8749346,
            "raw_content": null
        },
        {
            "title": "Lionel Messi: The life and times of the Barcelona, Paris Saint-Germain ...",
            "url": "https://www.nytimes.com/athletic/4783674/2023/08/18/lionel-messi-profile-soccer/",
            "content": "For Messi, it is major trophy number 44.. Despite turning 36 in June, he is as influential as ever. Here is the complete story of Lionel Andres Messi, widely regarded as one of the greatest ...",
            "score": 0.87321484,
            "raw_content": null
        },
        {
            "title": "Lionel Messi Biography",
            "url": "https://www.biographyonline.net/sport/football/lionel-messi.html",
            "content": "In the 2010 World Cup, Messi wore the number 10 shirt and played well to help Argentina reach the quarter-finals, but Messi struggled to score, and Argentina disappointingly lost 4-0 to Germany in the quarter-final. He is an incredible player, gigantic.”\n– Gerd Muller\nAt the start of 2013, in club football, Messi has scored 292 goals from a total of 359 appearances, and in international football, 31 goals from 76 appearances.\n The decision was contentious and not in keeping with Messi’s style of play which is generally clean and in the spirit of fair play; he has very rarely been accused of diving.\n (total Barca)\n– Lionel Messi\nAfter winning the Ballon d’Or for the fourth time in January 2013, Messi said:\n“To tell you the truth this is really quite unbelievable. Messi major honours\nBarcelona\nArgentina\nWealth and income\nMessi has frequently been the target of other football clubs with big transfer budgets, but he has remained loyal to Barcelona FC.",
            "score": 0.82375413,
            "raw_content": null
        },
        {
            "title": "Lionel Messi - Wikipedia",
            "url": "https://en.wikipedia.org/wiki/Lionel_Messi",
            "content": "He scored twice in the last group match, a 3–2 victory over Nigeria, his second goal coming from a free kick, as they finished first in their group.[423] Messi assisted a late goal in extra time to ensure a 1–0 win against Switzerland in the round of 16, and played in the 1–0 quarter-final win against Belgium as Argentina progressed to the semi-final of the World Cup for the first time since 1990.[424][425] Following a 0–0 draw in extra time, they eliminated the Netherlands 4–2 in a penalty shootout to reach the final, with Messi scoring his team's first penalty.[426]\nBilled as Messi versus Germany, the world's best player against the best team, the final was a repeat of the 1990 final featuring Diego Maradona.[427] Within the first half-hour, Messi had started the play that led to a goal, but it was ruled offside. \"[582] Moreover, several pundits and footballing figures, including Maradona, questioned Messi's leadership with Argentina at times, despite his playing ability.[583][584][585] Vickery states the perception of Messi among Argentines changed in 2019, with Messi making a conscious effort to become \"more one of the group, more Argentine\", with Vickery adding that following the World Cup victory in 2022 Messi would now be held in the same esteem by his compatriots as Maradona.[581]\nComparisons with Cristiano Ronaldo\nAmong his contemporary peers, Messi is most often compared and contrasted with Portuguese forward Cristiano Ronaldo, as part of an ongoing rivalry that has been compared to past sports rivalries like the Muhammad Ali–Joe Frazier rivalry in boxing, the Roger Federer–Rafael Nadal rivalry in tennis, and the Prost–Senna rivalry from Formula One motor racing.[586][587]\nAlthough Messi has at times denied any rivalry,[588][589] they are widely believed to push one another in their aim to be the best player in the world.[160] Since 2008, Messi has won eight Ballons d'Or to Ronaldo's five,[590] seven FIFA World's Best Player awards to Ronaldo's five, and six European Golden Shoes to Ronaldo's four.[591] Pundits and fans regularly argue the individual merits of both players.[160][592] On 11 July, Messi provided his 20th assist of the league season for Arturo Vidal in a 1–0 away win over Real Valladolid, equalling Xavi's record of 20 assists in a single La Liga season from 2008 to 2009;[281][282] with 22 goals, he also became only the second player ever, after Thierry Henry in the 2002–03 FA Premier League season with Arsenal (24 goals and 20 assists), to record at least 20 goals and 20 assists in a single league season in one of Europe's top-five leagues.[282][283] Following his brace in a 5–0 away win against Alavés in the final match of the season on 20 May, Messi finished the season as both the top scorer and top assist provider in La Liga, with 25 goals and 21 assists respectively, which saw him win his record seventh Pichichi trophy, overtaking Zarra; however, Barcelona missed out on the league title to Real Madrid.[284] On 7 March, two weeks after scoring four goals in a league fixture against Valencia, he scored five times in a Champions League last 16-round match against Bayer Leverkusen, an unprecedented achievement in the history of the competition.[126][127] In addition to being the joint top assist provider with five assists, this feat made him top scorer with 14 goals, tying José Altafini's record from the 1962–63 season, as well as becoming only the second player after Gerd Müller to be top scorer in four campaigns.[128][129] Two weeks later, on 20 March, Messi became the top goalscorer in Barcelona's history at 24 years old, overtaking the 57-year record of César Rodríguez's 232 goals with a hat-trick against Granada.[130]\nDespite Messi's individual form, Barcelona's four-year cycle of success under Guardiola – one of the greatest eras in the club's history – drew to an end.[131] He still managed to break two longstanding records in a span of seven days: a hat-trick on 16 March against Osasuna saw him overtake Paulino Alcántara's 369 goals to become Barcelona's top goalscorer in all competitions including friendlies, while another hat-trick against Real Madrid on 23 March made him the all-time top scorer in El Clásico, ahead of the 18 goals scored by former Real Madrid player Alfredo Di Stéfano.[160][162] Messi finished the campaign with his worst output in five seasons, though he still managed to score 41 goals in all competitions.[161][163] For the first time in five years, Barcelona ended the season without a major trophy; they were defeated in the Copa del Rey final by Real Madrid and lost the league in the last game to Atlético Madrid, causing Messi to be booed by sections of fans at the Camp Nou.[164]",
            "score": 0.8143483,
            "raw_content": null
        }
    ],
    "response_time": 2.09
}

Endpoint: POST /extract

Retrieve raw web content from specified URLs.

Parameters

• urls: string | list<string> (required) - The urls you want to extract with Tavily.

• api_key: string (required) - Your Tavily API Key.

Response

• results: list: A list of extracted content from the provided URLs. Each result is in the following format:
  • url: string - The URL of the webpage from which content was extracted.
  • raw_content- string - The raw content extracted from the provided URL.

• failed_results: list - A list of URLs that could not be processed, if any. Each result is in the following format:
  • url: string - The URL that failed to be processed.
  • error: string - An error message describing why the URL could not be processed.

• response_time: float - Your search result response time.

Example Usage

Request

{
    "api_key": "tvly-YOUR_API_KEY",
    "urls": [
        "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "https://en.wikipedia.org/wiki/Machine_learning",
        "https://en.wikipedia.org/wiki/Data_science",
        "https://en.wikipedia.org/wiki/Quantum_computing",
        "https://com"
    ]
}

Response

{
    "results": [
        {
            "url": "https://en.wikipedia.org/wiki/Artificial_intelligence",
            "raw_content": "Contents\nArtificial intelligence\nArtificial intelligence (AI) is the intelligence of machines or software, as opposed to the intelligence of humans or animals. It is a field of study in computer science that develops and studies intelligent machines. \"AI\" may also refer to the machines themselves.\nAI technology is widely used throughout industry, government and science. Some high-profile applications are: advanced web search engines (e.g., Google Search), recommendation systems (used by YouTube, Amazon, and Netflix), understanding human speech (such as Siri and Alexa), self-driving cars (e.g., Waymo), generative or creative tools (ChatGPT and AI art), and competing at the highest level in strategy games (such as chess and Go).[1]\nArtificial intelligence was founded as an academic discipline in 1956.[2] The field went through multiple cycles of optimism[3][4] followed by disappointment and loss of funding,[5][6] but after 2012, when deep learning surpassed all previous AI techniques,[7] there was a vast increase in funding and interest.\nThe various sub-fields of AI research are centered around particular goals and the use of particular tools. The traditional goals of AI research include reasoning, knowledge representation, planning, learning, natural language processing, perception, and support for robotics.[a] General intelligence (the ability to solve an arbitrary problem) is among the field's long-term goals.[8]\nTo solve these problems, AI researchers have adapted and integrated a wide range of problem-solving techniques, including search and mathematical optimization, formal logic, artificial neural networks, and methods based on statistics, operations research, and economics.[b] AI also draws upon psychology, linguistics, philosophy, neuroscience and many other fields.[9]\nGoals\nThe general problem of simulating (or creating) intelligence has been broken down into sub-problems. These consist of particular traits or capabilities that researchers expect an intelligent system to display. The traits described below have received the most attention and cover the scope of AI research.[a]\nReasoning, problem-solving\nEarly researchers developed algorithms that imitated step-by-step reasoning that humans use when they solve puzzles or make logical deductions.[10] By the late 1980s and 1990s, methods were developed for dealing with uncertain or incomplete information, employing concepts from probability and economics.[11]\nMany of these algorithms are insufficient for solving large reasoning problems because they experience a \"combinatorial explosion\": they became exponentially slower as the problems grew larger.[12]\nEven humans rarely use the step-by-step deduction that early AI research could model. They solve most of their problems using fast, intuitive judgments.[13]\nAccurate and efficient reasoning is an unsolved problem.\nKnowledge representation\nKnowledge representation and knowledge engineering[14] allow AI programs to answer questions intelligently and make deductions about real-world facts. Formal knowledge representations are used in content-based indexing and retrieval,[15] scene interpretation,[16] clinical decision support,[17] knowledge discovery (mining \"interesting\" and actionable inferences from large databases),[18] and other areas.[19]\nA knowledge base is a body of knowledge represented in a form that can be used by a program. An ontology is the set of objects, relations, concepts, and properties used by a particular domain of knowledge.[20] Knowledge bases need to represent things such as:\nobjects, properties, categories and relations between objects;\n[21]\nsituations, events, states and time;[22]\ncauses and effects;[23]\nknowledge about knowledge (what we know about what other people know);[24]\ndefault reasoning (things that humans assume are true until they are told differently and will remain true even when other facts are changing);[25] and many other aspects and domains of knowledge.\nAmong the most difficult problems in KR are: the breadth of commonsense knowledge (the set of atomic facts that the average person knows is enormous);[26] and the sub-symbolic form of most commonsense knowledge (much of what people know is not represented as \"facts\" or \"statements\" that they could express verbally).[13]\nKnowledge acquisition is the difficult problem of obtaining knowledge for AI applications.[c] Modern AI gathers knowledge by \"scraping\" the internet (including Wikipedia). The knowledge itself was collected by the volunteers and professionals who published the information (who may or may not have agreed to provide their work to AI companies).[29] This \"crowd sourced\" technique does not guarantee that the knowledge is correct or reliable. The knowledge of Large Language Models (such as ChatGPT) is highly unreliable -- it generates misinformation and falsehoods (known as \"hallucinations\"). Providing accurate knowledge for these modern AI applications is an unsolved problem.\nPlanning and decision making\nAn \"agent\" is anything that perceives and takes actions in the world. A rational agent has goals or preferences and takes actions to make them happen.[d][30]\nIn automated planning, the agent has a specific goal.[31] In automated decision making, the agent has preferences – there are some situations it would prefer to be in, and some situations it is trying to avoid. The decision making agent assigns a number to each situation (called the \"utility\") that measures how much the agent prefers it. For each possible action, it can calculate the \"expected utility\": the utility of all possible outcomes of the action, weighted by the probability that the outcome will occur. It can then choose the action with the maximum expected utility.[32]\nIn classical planning, the agent knows exactly what the effect of any action will be.[33]\nIn most real-world problems, however, the agent may not be certain about the situation they are in (it is \"unknown\" or \"unobservable\") and it may not know for certain what will happen after each possible action (it is not \"deterministic\"). It must choose an action by making a probabilistic guess and then reassess the situation to see if the action worked.[34]\nIn some problems, the agent's preferences may be uncertain, especially if there are other agents or humans involved. These can be learned (e.g., with inverse reinforcement learning) or the agent can seek information to improve its preferences.[35]\nInformation value theory can be used to weigh the value of exploratory or experimental actions.[36]\nThe space of possible future actions and situations is typically intractably large, so the agents must take actions and evaluate situations while being uncertain what the outcome will be.\nA Markov decision process has a transition model that describes the probability that a particular action will change the state in a particular way, and a reward function that supplies the utility of each state and the cost of each action. A policy associates a decision with each possible state. The policy could be calculated (e.g. by iteration), be heuristic, or it can be learned.[37]\nGame theory describes rational behavior of multiple interacting agents, and is used in AI programs that make decisions that involve other agents.[38]\nLearning\nMachine learning is the study of programs that can improve their performance on a given task automatically.[39]\nIt has been a part of AI from the beginning.[e]\nThere are several kinds of machine learning. Unsupervised learning analyzes a stream of data and finds patterns and makes predictions without any other guidance.[42]\nSupervised learning requires a human to label the input data first, and comes in two main varieties: classification (where the program must learn to predict what category the input belongs in) and regression (where the program must deduce a numeric function based on numeric input).[43]\nIn reinforcement learning the agent is rewarded for good responses and punished for bad ones. The agent learns to choose responses that are classified as \"good\".[44]\nTransfer learning is when the knowledge gained from one problem is applied to a new problem.[45] Deep learning is a type of machine learning that runs inputs through biologically inspired artificial neural networks for all of these types of learning.[46]\nComputational learning theory can assess learners by computational complexity, by sample complexity (how much data is required), or by other notions of optimization.[47]\nNatural language processing\nNatural language processing (NLP)[48] allows programs to read, write and communicate in human languages such as English.\nSpecific problems include speech recognition, speech synthesis, machine translation, information extraction, information retrieval and question answering.[49]\nEarly work, based on Noam Chomsky's generative grammar and semantic networks, had difficulty with word-sense disambiguation[f]\nunless restricted to small domains called \"micro-worlds\" (due to the common sense knowledge problem[26]).\nModern deep learning techniques for NLP include word embedding (representing words, typically as vectors encoding their meaning),[50] transformers (a deep learning architecture using an attention mechanism),[51] and others.[52] In 2019, generative pre-trained transformer (or \"GPT\") language models began to generate coherent text,[53][54] and by 2023 these models were able to get human-level scores on the bar exam, SAT, GRE, and many other real-world applications.[55]\nPerception\nMachine perception is the ability to use input from sensors (such as cameras, microphones, wireless signals, active lidar, sonar, radar, and tactile sensors) to deduce aspects of the world. Computer vision is the ability to analyze visual input.[56]\nThe field includes speech recognition,[57]\nimage classification,[58]\nfacial recognition, object recognition,[59]\nand robotic perception.[60]\nSocial intelligence\nAffective computing is an interdisciplinary umbrella that comprises systems that recognize, interpret, process or simulate human feeling, emotion and mood.[62]\nFor example, some virtual assistants are programmed to speak conversationally or even to banter humorously; it makes them appear more sensitive to the emotional dynamics of human interaction, or to otherwise facilitate human–computer interaction.\nHowever, this tends to give naïve users an unrealistic conception of how intelligent existing computer agents actually are.[63] Moderate successes related to affective computing include textual sentiment analysis and, more recently, multimodal sentiment analysis, wherein AI classifies the affects displayed by a videotaped subject.[64]\nGeneral intelligence\nA machine with artificial general intelligence should be able to solve a wide variety of problems with breadth and versatility similar to human intelligence.[8]\nTools\nAI research uses a wide variety of tools to accomplish the goals above.[b]\nSearch and optimization\nAI can solve many problems by intelligently searching through many possible solutions.[65] There are two very different kinds of search used in AI: state space search and local search.\nState space search searches through a tree of possible states to try to find a goal state.[66]\nFor example, Planning algorithms search through trees of goals and subgoals, attempting to find a path to a target goal, a process called means-ends analysis.[67]\nSimple exhaustive searches[68]\nare rarely sufficient for most real-world problems: the search space (the number of places to search) quickly grows to astronomical numbers. The result is a search that is too slow or never completes.[12]\n\"Heuristics\" or \"rules of thumb\" can help to prioritize choices that are more likely to reach a goal.[69]\nAdversarial search is used for game-playing programs, such as chess or Go. It searches through a tree of possible moves and counter-moves, looking for a winning position.[70]\nLocal search uses mathematical optimization to find a numeric solution to a problem. It begins with some form of a guess and then refines the guess incrementally until no more refinements can be made. These algorithms can be visualized as blind hill climbing: we begin the search at a random point on the landscape, and then, by jumps or steps, we keep moving our guess uphill, until we reach the top. This process is called stochastic gradient descent.[71]\nEvolutionary computation uses a form of optimization search. For example, they may begin with a population of organisms (the guesses) and then allow them to mutate and recombine, selecting only the fittest to survive each generation (refining the guesses).[72]\nDistributed search processes can coordinate via swarm intelligence algorithms. Two popular swarm algorithms used in search are particle swarm optimization (inspired by bird flocking) and ant colony optimization (inspired by ant trails).[73]\nNeural networks and statistical classifiers (discussed below), also use a form of local search, where the \"landscape\" to be searched is formed by learning.\nLogic\nFormal Logic is used for reasoning and knowledge representation.[74]\nFormal logic comes in two main forms: propositional logic (which operates on statements that are true or false and uses logical connectives such as \"and\", \"or\", \"not\" and \"implies\")[75]\nand predicate logic (which also operates on objects, predicates and relations and uses quantifiers such as \"Every X is a Y\" and \"There are some Xs that are Ys\").[76]\nLogical inference (or deduction) is the process of proving a new statement (conclusion) from other statements that are already known to be true (the premises).[77]\nA logical knowledge base also handles queries and assertions as a special case of inference.[78]\nAn inference rule describes what is a valid step in a proof. The most general inference rule is resolution.[79]\nInference can be reduced to performing a search to find a path that leads from premises to conclusions, where each step is the application of an inference rule.[80]\nInference performed this way is intractable except for short proofs in restricted domains. No efficient, powerful and general method has been discovered.[81]\nFuzzy logic assigns a \"degree of truth\" between 0 and 1 and handles uncertainty and probabilistic situations.[82]\nNon-monotonic logics are designed to handle default reasoning.[25]\nOther specialized versions of logic have been developed to describe many complex domains (see knowledge representation above).\nProbabilistic methods for uncertain reasoning\nMany problems in AI (including in reasoning, planning, learning, perception, and robotics) require the agent to operate with incomplete or uncertain information. AI researchers have devised a number of tools to solve these problems using methods from probability theory and economics.[83]\nBayesian networks[84]\nare a very general tool that can be used for many problems, including reasoning (using the Bayesian inference algorithm),[g][86]\nlearning (using the expectation-maximization algorithm),[h][88]\nplanning (using decision networks)[89]\nand perception (using dynamic Bayesian networks).[90]\nProbabilistic algorithms can also be used for filtering, prediction, smoothing and finding explanations for streams of data, helping perception systems to analyze processes that occur over time (e.g., hidden Markov models or Kalman filters).[90]\nPrecise mathematical tools have been developed that analyze how an agent can make choices and plan, using decision theory, decision analysis,[91]\nand information value theory.[92]\nThese tools include models such as Markov decision processes,\n[93]\ndynamic decision networks,[90]\ngame theory and mechanism design.[94]\nClassifiers and statistical learning methods\nThe simplest AI applications can be divided into two types: classifiers (e.g. \"if shiny then diamond\"), on one hand, and controllers (e.g. \"if diamond then pick up\"), on the other hand. Classifiers[95]\nare functions that use pattern matching to determine the closest match. They can be fine-tuned based on chosen examples using supervised learning. Each pattern (also called an \"observation\") is labeled with a certain predefined class. All the observations combined with their class labels are known as a data set. When a new observation is received, that observation is classified based on previous experience.[43]\nThere are many kinds of classifiers in use. The decision tree is the simplest and most widely used symbolic machine learning algorithm.[96] K-nearest neighbor algorithm was the most widely used analogical AI until the mid-1990s, and Kernel methods such as the support vector machine (SVM) displaced k-nearest neighbor in the 1990s.[97]\nThe naive Bayes classifier is reportedly the \"most widely used learner\"[98] at Google, due in part to its scalability.[99]\nNeural networks are also used as classifiers.[100]\nArtificial neural networks\nArtificial neural networks[100] were inspired by the design of the human brain: a simple \"neuron\" N accepts input from other neurons, each of which, when activated (or \"fired\"), casts a weighted \"vote\" for or against whether neuron N should itself activate. In practice, the input \"neurons\" are a list of numbers, the \"weights\" are a matrix, the next layer is the dot product (i.e., several weighted sums) scaled by an increasing function, such as the logistic function. \"The resemblance to real neural cells and structures is superficial\", according to Russell and Norvig.[101][i]\nLearning algorithms for neural networks use local search to choose the weights that will get the right output for each input during training. The most common training technique is the backpropagation algorithm.[102]\nNeural networks learn to model complex relationships between inputs and outputs and find patterns in data. In theory, a neural network can learn any function.[103]\nIn feedforward neural networks the signal passes in only one direction.[104]\nRecurrent neural networks feed the output signal back into the input, which allows short-term memories of previous input events. Long short term memory is the most successful network architecture for recurrent networks.[105]\nPerceptrons[106]\nuse only a single layer of neurons, deep learning[107] uses multiple layers.\nConvolutional neural networks strengthen the connection between neurons that are \"close\" to each other – this is especially important in image processing, where a local set of neurons must identify an \"edge\" before the network can identify an object.[108]\nDeep learning\nDeep learning[107]\nuses several layers of neurons between the network's inputs and outputs. The multiple layers can progressively extract higher-level features from the raw input. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits or letters or faces.[110]\nDeep learning has drastically improved the performance of programs in many important subfields of artificial intelligence, including computer vision, speech recognition, image classification[111]\nand others. The reason that deep learning performs so well in so many applications is not known as of 2023.[112]\nThe sudden success of deep learning in 2012–2015 did not occur because of some new discovery or theoretical breakthrough (deep neural networks and backpropagation had been described by many people, as far back as the 1950s)[j]\nbut because of two factors: the incredible increase in computer power (including the hundred-fold increase in speed by switching to GPUs) and the availability of vast amounts of training data, especially the giant curated datasets used for benchmark testing, such as ImageNet.[k]\nSpecialized hardware and software\nIn the late 2010s, graphics processing units (GPUs) that were increasingly designed with AI-specific enhancements and used with specialized TensorFlow software, had replaced previously used central processing unit (CPUs) as the dominant means for large-scale (commercial and academic) machine learning models' training.[121]\nHistorically, specialized languages, such as Lisp, Prolog, and others, had been used.\nApplications\nAI and machine learning technology is used in most of the essential applications of the 2020s, including: search engines (such as Google Search),\ntargeting online advertisements,[122]\nrecommendation systems (offered by Netflix, YouTube or Amazon),\ndriving internet traffic,[123][124]\ntargeted advertising (AdSense, Facebook),\nvirtual assistants (such as Siri or Alexa),[125]\nautonomous vehicles (including drones,\nADAS and self-driving cars),\nautomatic language translation (Microsoft Translator, Google Translate),\nfacial recognition (Apple's Face ID or Microsoft's DeepFace and Google's FaceNet) and\nimage labeling (used by Facebook, Apple's iPhoto and TikTok).\nThere are also thousands of successful AI applications used to solve specific problems for specific industries or institutions. In a 2017 survey, one in five companies reported they had incorporated \"AI\" in some offerings or processes.[126]\nA few examples are energy storage,[127]\nmedical diagnosis,\nmilitary logistics,\napplications that predict the result of judicial decisions,[128]\nforeign policy,[129]\nor supply chain management.\nGame playing programs have been used since the 1950s to demonstrate and test AI's most advanced techniques. Deep Blue became the first computer chess-playing system to beat a reigning world chess champion, Garry Kasparov, on 11 May 1997.[130] In 2011, in a Jeopardy! quiz show exhibition match, IBM's question answering system, Watson, defeated the two greatest Jeopardy! champions, Brad Rutter and Ken Jennings, by a significant margin.[131]\nIn March 2016, AlphaGo won 4 out of 5 games of Go in a match with Go champion Lee Sedol, becoming the first computer Go-playing system to beat a professional Go player without handicaps.[132] Then it defeated Ke Jie in 2017, who at the time continuously held the world No. 1 ranking for two years.[133][134][135] Other programs handle imperfect-information games; such as for poker at a superhuman level, Pluribus[l] and Cepheus.[137] DeepMind in the 2010s developed a \"generalized artificial intelligence\" that could learn many diverse Atari games on its own.[138]\nIn the early 2020s, generative AI gained widespread prominence. ChatGPT, based on GPT-3, and other large language models, were tried by 14% of Americans adults.[139] The increasing realism and ease-of-use of AI-based text-to-image generators such as Midjourney, DALL-E, and Stable Diffusion[140][141]\nsparked a trend of viral AI-generated photos. Widespread attention was gained by a fake photo of Pope Francis wearing a white puffer coat,[142]\nthe fictional arrest of Donald Trump,[143]\nand a hoax of an attack on the Pentagon,[144]\nas well as the usage in professional creative arts.[145][146]\nAlphaFold 2 (2020) demonstrated the ability to approximate, in hours rather than months, the 3D structure of a protein.[147]\nEthics\nAI, like any powerful technology, has potential benefits and potential risks. AI may be able to advance science and find solutions for serious problems: Demis Hassabis of Deep Mind hopes to \"solve intelligence, and then use that to solve everything else\".[148] However, as the use of AI has become widespread, several unintended consequences and risks have been identified.[149]\nAnyone looking to use machine learning as part of real-world, in-production systems needs to factor ethics into their AI training processes and strive to avoid bias. This is especially true when using AI algorithms that are inherently unexplainable in deep learning.[150]\nRisks and harm\nMachine learning algorithms require large amounts of data. The techniques used to acquire this data have raised concerns about privacy, surveillance and copyright.\nTechnology companies collect a wide range of data from their users, including online activity, geolocation data, video and audio.[151]\nFor example, in order to build speech recognition algorithms, Amazon others have recorded millions of private conversations and allowed temps to listen to and transcribe some of them.[152]\nOpinions about this widespread surveillance range from those who see it as a necessary evil to those for whom it is clearly unethical and a violation of the right to privacy.[153]\nAI developers argue that this is the only way to deliver valuable applications. and have developed several techniques that attempt to preserve privacy while still obtaining the data, such as data aggregation, de-identification and differential privacy.[154]\nSince 2016, some privacy experts, such as Cynthia Dwork, began to view privacy in terms of fairness -- Brian Christian wrote that experts have pivoted \"from the question of 'what they know' to the question of 'what they're doing with it'.\".[155]\nGenerative AI is often trained on unlicensed copyrighted works, including in domains such as images or computer code; the output is then used under a rationale of \"fair use\". Experts disagree about how well, and under what circumstances, this rationale will hold up in courts of law; relevant factors may include \"the purpose and character of the use of the copyrighted work\" and \"the effect upon the potential market for the copyrighted work\".[156] In 2023, leading authors (including John Grisham and Jonathan Franzen) sued AI companies for using their work to train generative AI.[157][158]\nYouTube, Facebook and others use recommender systems to guide users to more content. These AI programs were given the goal of maximizing user engagement (that is, the only goal was to keep people watching). The AI learned that users tended to choose misinformation, conspiracy theories, and extreme partisan content, and, to keep them watching, the AI recommended more of it. Users also tended to watch more content on the same subject, so the AI led people into filter bubbles where they received multiple versions of the same misinformation.[159] This convinced many users that the misinformation was true, and ultimately undermined trust in institutions, the media and the government.[160] The AI program had correctly learned to maximize its goal, but the result was harmful to society. After the U.S. election in 2016, major technology companies took steps to mitigate the problem.\nIn 2022, generative AI began to create images, audio, video and text that are indistinguishable from real photographs, recordings, films or human writing.\nIt is possible for bad actors to use this technology to create massive amounts of misinformation or propaganda.[161] AI pioneer Geoffrey Hinton expressed concern about AI enabling \"authoritarian leaders to manipulate their electorates\" on a large scale, among other risks.[162]\nMachine learning applications will be biased if they learn from biased data.[163]\nThe developers may not be aware that the bias exists.[164]\nBias can be introduced by the way training data is selected and by the way a model is deployed.[165][163] If a biased algorithm is used to make decisions that can seriously harm people (as it can in medicine, finance, recruitment, housing or policing) then the algorithm may cause discrimination.[166]\nFairness in machine learning is the study of how to prevent the harm caused by algorithmic bias. It has become serious area of academic study within AI. Researchers have discovered it is not always possible to define \"fairness\" in a way that satisfies all stakeholders.[167]\nOn June 28, 2015, Google Photos's new image labeling feature mistakenly identified Jacky Alcine and a friend as \"gorillas\" because they were black. The system was trained on a dataset that contained very few images of black people,[168] a problem called \"sample size disparity\".[169] Google \"fixed\" this problem by preventing the system from labelling anything as a \"gorilla\". Eight years later, in 2023, Google Photos still could not identify a gorilla, and neither could similar products from Apple, Facebook, Microsoft and Amazon.[170]\nCOMPAS is a commercial program widely used by U.S. courts to assess the likelihood of a defendant becoming a recidivist.\nIn 2016, Julia Angwin at ProPublica discovered that COMPAS exhibited racial bias, despite the fact that the program was not told the races of the defendants. Although the error rate for both whites and blacks was calibrated equal at exactly 61%, the errors for each race were different -- the system consistently overestimated the chance that a black person would re-offend and would underestimate the chance that a white person would not re-offend.[171] In 2017, several researchers[m] showed that it was mathematically impossible for COMPAS to accommodate all possible measures of fairness when the base rates of re-offense were different for whites and blacks in the data.[173]\nA program can make biased decisions even if the data does not explicitly mention a problematic feature (such as \"race\" or \"gender\"). The feature will correlate with other features (like \"address\", \"shopping history\" or \"first name\"), and the program will make the same decisions based on these features as it would on \"race\" or \"gender\".[174]\nMoritz Hardt said \"the most robust fact in this research area is that fairness through blindness doesn't work.\"[175]\nCriticism of COMPAS highlighted a deeper problem with the misuse of AI. Machine learning models are designed to make \"predictions\" that are only valid if we assume that the future will resemble the past. If they are trained on data that includes the results of racist decisions in the past, machine learning models must predict that racist decisions will be made in the future. Unfortunately, if an application then uses these predictions as recommendations, some of these \"recommendations\" will likely be racist.[176] Thus, machine learning is not well suited to help make decisions in areas where there is hope that the future will be better than the past. It is necessarily descriptive and not proscriptive.[n]\nBias and unfairness may go undetected because the developers are overwhelmingly white and male: among AI engineers, about 4% are black and 20% are women.[169]\nAt its 2022 Conference on Fairness, Accountability, and Transparency (ACM FAccT 2022) the Association for Computing Machinery, in Seoul, South Korea, presented and published findings recommending that until AI and robotics systems are demonstrated to be free of bias mistakes, they are unsafe and the use of self-learning neural networks trained on vast, unregulated sources of flawed internet data should be curtailed.[178]\nMost modern AI applications can not explain how they have reached a decision.[179] The large amount of relationships between inputs and outputs in deep neural networks and resulting complexity makes it difficult for even an expert to explain how they produced their outputs, making them a black box.[180]\nThere have been many cases where a machine learning program passed rigorous tests, but nevertheless learned something different than what the programmers intended. For example, Justin Ko and Roberto Novoa developed a system that could identify skin diseases better than medical professionals, however it classified any image with a ruler as \"cancerous\", because pictures of malignancies typically include a ruler to show the scale.[181] A more dangerous example was discovered by Rich Caruana in 2015: a machine learning system that accurately predicted risk of death classified a patient that was over 65, asthma and difficulty breathing as \"low risk\". Further research showed that in high-risk cases like this, the hospital would allocate more resources and save the patient's life, decreasing the risk measured by the program.[182] Mistakes like these become obvious when we know how the program has reached a decision. Without an explanation, these problems may not not be discovered until after they have caused harm.\nA second issue is that people who have been harmed by an algorithm's decision have a right to an explanation. Doctors, for example, are required to clearly and completely explain the reasoning behind any decision they make.[183] Early drafts of the European Union's General Data Protection Regulation in 2016 included an explicit statement that this right exists.[o] Industry experts noted that this is an unsolved problem with no solution in sight. Regulators argued that nevertheless the harm is real: if the problem has no solution, the tools should not be used.[184]\nDARPA established the XAI (\"Explainable Artificial Intelligence\") program in 2014 to try and solve these problems.[185]\nThere are several potential solutions to the transparency problem. Multitask learning provides a large number of outputs in addition to the target classification. These other outputs can help developers deduce what the network has learned.[186] Deconvolution, DeepDream and other generative methods can allow developers to see what different layers of a deep network have learned and produce output that can suggest what the network is learning.[187] Supersparse linear integer models use learning to identify the most important features, rather than the classification. Simple addition of these features can then make the classification (i.e. learning is used to create a scoring system classifier, which is transparent).[188]\nA lethal autonomous weapon is a machine that locates, selects and engages human targets without human supervision.[p] By 2015, over fifty countries were reported to be researching battlefield robots.[190] These weapons are considered especially dangerous for several reasons: if they kill an innocent person it is not clear who should be held accountable, it is unlikely they will reliably choose targets, and, if produced at scale, they are potentially weapons of mass destruction.[191] In 2014, 30 nations (including China) supported a ban on autonomous weapons under the United Nations' Convention on Certain Conventional Weapons, however the United States and others disagreed.[192]\nAI provides a number of tools that are particularly useful for authoritarian governments: smart spyware, face recognition and voice recognition allow widespread surveillance; such surveillance allows machine learning to classify potential enemies of the state and can prevent them from hiding; recommendation systems can precisely target propaganda and misinformation for maximum effect; deepfakes and generative AI aid in producing misinformation; advanced AI can make authoritarian centralized decision making more competitive with liberal and decentralized systems such as markets.[193]\nTerrorists, criminals and rogue states can use weaponized AI such as advanced digital warfare and lethal autonomous weapons.\nMachine-learning AI is also able to design tens of thousands of toxic molecules in a matter of hours.[194]\nFrom the early days of the development of artificial intelligence there have been arguments, for example those put forward by Weizenbaum, about whether tasks that can be done by computers actually should be done by them, given the difference between computers and humans, and between quantitative calculation and qualitative, value-based judgement.[195]\nEconomists have frequently highlighted the risks of redundancies from AI, and speculated about unemployment if there is no adequate social policy for full employment.[196]\nIn the past, technology has tended to increase rather than reduce total employment, but economists acknowledge that \"we're in uncharted territory\" with AI.[197] A survey of economists showed disagreement about whether the increasing use of robots and AI will cause a substantial increase in long-term unemployment, but they generally agree that it could be a net benefit if productivity gains are redistributed.[198] Risk estimates vary; for example, in the 2010s Michael Osborne and Carl Benedikt Frey estimated 47% of U.S. jobs are at \"high risk\" of potential automation, while an OECD report classified only 9% of U.S. jobs as \"high risk\".[q][200] The methodology of speculating about future employment levels has been criticised as lacking evidential foundation, and for implying that technology (rather than social policy) creates unemployment (as opposed to redundancies).[196]\nUnlike previous waves of automation, many middle-class jobs may be eliminated by artificial intelligence; The Economist stated in 2015 that \"the worry that AI could do to white-collar jobs what steam power did to blue-collar ones during the Industrial Revolution\" is \"worth taking seriously\".[201] Jobs at extreme risk range from paralegals to fast food cooks, while job demand is likely to increase for care-related professions ranging from personal healthcare to the clergy.[202]\nIn April 2023, it was reported that 70% of the jobs for Chinese video game illlustrators had been eliminated by generative artificial intelligence.[203][204]\nIt has been argued AI will become so powerful that humanity may irreversibly lose control of it. This could, as the physicist Stephen Hawking puts it, \"spell the end of the human race\".[205] This scenario has been common in science fiction, when a computer or robot suddenly develops a human-like \"self-awareness\" (or \"sentience\" or \"consciousness\") and becomes a malevolent character.[r] These sci-fi scenarios are misleading in several ways.\nFirst, AI does not require human-like \"sentience\" to be an existential risk. Modern AI programs are given specific goals and use learning and intelligence to achieve them. Philosopher Nick Bostrom argued that if one gives almost any goal to a sufficiently powerful AI, it may choose to destroy humanity to achieve it (he used the example of a paperclip factory manager).[207] Stuart Russell gives the example of household robot that tries to find a way to kill its owner to prevent it from being unplugged, reasoning that \"you can't fetch the coffee if you're dead.\"[208] In order to be safe for humanity, a superintelligence would have to be genuinely aligned with humanity's morality and values so that it is \"fundamentally on our side\".[209]\nSecond, Yuval Noah Harari argues that AI does not require a robot body or physical control to pose an existential risk. The essential parts of civilization are not physical. Things like ideologies, law, government, money and the economy are made of language; they exist because there are stories that billions of people believe. The current prevalence of misinformation suggests that an AI could use language to convince people to believe anything, even to take actions that are destructive.[210]\nThe opinions amongst experts and industry insiders are mixed, with sizable fractions both concerned and unconcerned by risk from eventual superintelligent AI.[211] Personalities such as Stephen Hawking, Bill Gates, Elon Musk have expressed concern about existential risk from AI.[212]\nIn the early 2010's, experts argued that the risks are too distant in the future to warrant research or that humans will be valuable from the perspective of a superintelligent machine.[213]\nHowever, after 2016, the study of current and future risks and possible solutions became a serious area of research.[214]\nAI pioneers including Fei-Fei Li, Geoffrey Hinton, Yoshua Bengio, Cynthia Breazeal, Rana el Kaliouby, Demis Hassabis, Joy Buolamwini, and Sam Altman have expressed concerns about the risks of AI and in 2023 many leading AI experts issued the joint statement that \"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war\".[215]\nEthical machines and alignment\nFriendly AI are machines that have been designed from the beginning to minimize risks and to make choices that benefit humans. Eliezer Yudkowsky, who coined the term, argues that developing friendly AI should be a higher research priority: it may require a large investment and it must be completed before AI becomes an existential risk.[216]\nMachines with intelligence have the potential to use their intelligence to make ethical decisions. The field of machine ethics provides machines with ethical principles and procedures for resolving ethical dilemmas.[217]\nThe field of machine ethics is also called computational morality,[217]\nand was founded at an AAAI symposium in 2005.[218]\nOther approaches include Wendell Wallach's \"artificial moral agents\"[219]\nand Stuart J. Russell's three principles for developing provably beneficial machines.[220]\nRegulation\nThe regulation of artificial intelligence is the development of public sector policies and laws for promoting and regulating artificial intelligence (AI); it is therefore related to the broader regulation of algorithms.[221]\nThe regulatory and policy landscape for AI is an emerging issue in jurisdictions globally.[222] According to AI Index at Stanford, the annual number of AI-related laws passed in the 127 survey countries jumped from one passed in 2016 to 37 passed in 2022 alone.[223][224]\nBetween 2016 and 2020, more than 30 countries adopted dedicated strategies for AI.[225]\nMost EU member states had released national AI strategies, as had Canada, China, India, Japan, Mauritius, the Russian Federation, Saudi Arabia, United Arab Emirates, US and Vietnam. Others were in the process of elaborating their own AI strategy, including Bangladesh, Malaysia and Tunisia.[225]\nThe Global Partnership on Artificial Intelligence was launched in June 2020, stating a need for AI to be developed in accordance with human rights and democratic values, to ensure public confidence and trust in the technology.[225] Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher published a joint statement in November 2021 calling for a government commission to regulate AI.[226]\nIn 2023, OpenAI leaders published recommendations for the governance of superintelligence, which they believe may happen in less than 10 years.[227]\nIn a 2022 Ipsos survey, attitudes towards AI varied greatly by country; 78% of Chinese citizens, but only 35% of Americans, agreed that \"products and services using AI have more benefits than drawbacks\".[223] A 2023 Reuters/Ipsos poll found that 61% of Americans agree, and 22% disagree, that AI poses risks to humanity.[228]\nIn a 2023 Fox News poll, 35% of Americans thought it \"very important\", and an additional 41% thought it \"somewhat important\", for the federal government to regulate AI, versus 13% responding \"not very important\" and 8% responding \"not at all important\".[229][230]\nIn November 2023, a global AI safety summit was held in Bletchley Park to discuss the near and far term risks of AI and the possibility of mandatory and voluntary regulatory frameworks.[231]\nHistory\nThe study of mechanical or \"formal\" reasoning began with philosophers and mathematicians in antiquity. The study of logic led directly to Alan Turing's theory of computation, which suggested that a machine, by shuffling symbols as simple as \"0\" and \"1\", could simulate both mathematical deduction and formal reasoning, which is known as the Church–Turing thesis.[232] This, along with concurrent discoveries in cybernetics and information theory, led researchers to consider the possibility of building an \"electronic brain\".[s][234] The first paper later recognized as \"AI\" was McCullouch and Pitts design for Turing-complete \"artificial neurons\" in 1943.[235]\nThe field of AI research was founded at a workshop at Dartmouth College in 1956.[t][2] The attendees became the leaders of AI research in the 1960s.[u] They and their students produced programs that the press described as \"astonishing\":[v] computers were learning checkers strategies, solving word problems in algebra, proving logical theorems and speaking English.[w][3]\nBy the middle of the 1960s, research in the U.S. was heavily funded by the Department of Defense[239] and laboratories had been established around the world.[240] Herbert Simon predicted, \"machines will be capable, within twenty years, of doing any work a man can do\".[241] Marvin Minsky agreed, writing, \"within a generation ... the problem of creating 'artificial intelligence' will substantially be solved\".[242]\nThey had, however, underestimated the difficulty of the problem.[x] Both the U.S. and British governments cut off exploratory research in response to the criticism of Sir James Lighthill[244] and ongoing pressure from the US Congress to fund more productive projects. Minsky's and Papert's book Perceptrons was understood as proving that artificial neural networks approach would never be useful for solving real-world tasks, thus discrediting the approach altogether.[245] The \"AI winter\", a period when obtaining funding for AI projects was difficult, followed.[5]\nIn the early 1980s, AI research was revived by the commercial success of expert systems,[246] a form of AI program that simulated the knowledge and analytical skills of human experts. By 1985, the market for AI had reached over a billion dollars. At the same time, Japan's fifth generation computer project inspired the U.S. and British governments to restore funding for academic research.[4] However, beginning with the collapse of the Lisp Machine market in 1987, AI once again fell into disrepute, and a second, longer-lasting winter began.[6]\nMany researchers began to doubt that the current practices would be able to imitate all the processes of human cognition, especially perception, robotics, learning and pattern recognition.[247] A number of researchers began to look into \"sub-symbolic\" approaches.[248] Robotics researchers, such as Rodney Brooks, rejected \"representation\" in general and focussed directly on engineering machines that move and survive.[y]. Judea Pearl, Lofti Zadeh and others developed methods that handled incomplete and uncertain information by making reasonable guesses rather than precise logic.[83][253] But the most important development was the revival of \"connectionism\", including neural network research, by Geoffrey Hinton and others.[254] In 1990, Yann LeCun successfully showed that convolutional neural networks can recognize handwritten digits, the first of many successful applications of neural networks.[255]\nAI gradually restored its reputation in the late 1990s and early 21st century by exploiting formal mathematical methods and by finding specific solutions to specific problems. This \"narrow\" and \"formal\" focus allowed researchers to produce verifiable results and collaborate with other fields (such as statistics, economics and mathematics).[256]\nBy 2000, solutions developed by AI researchers were being widely used, although in the 1990s they were rarely described as \"artificial intelligence\".[257]\nSeveral academic researchers became concerned that AI was no longer pursuing the original goal of creating versatile, fully intelligent machines. Beginning around 2002, they founded the subfield of artificial general intelligence (or \"AGI\"), which had several well-funded institutions by the 2010s.[8]\nDeep learning began to dominate industry benchmarks in 2012 and was adopted throughout the field.[7]\nFor many specific tasks, other methods were abandoned.[z]\nDeep learning's success was based on both hardware improvements (faster computers,[259] graphics processing units, cloud computing[260])\nand access to large amounts of data[261] (including curated datasets,[260] such as ImageNet).\nDeep learning's success led to an enormous increase in interest and funding in AI.[aa]\nThe amount of machine learning research (measured by total publications) increased by 50% in the years 2015–2019,[225]\nand WIPO reported that AI was the most prolific emerging technology in terms of the number of patent applications and granted patents[262]\nAccording to 'AI Impacts', about $50 billion annually was invested in \"AI\" around 2022 in the US alone and about 20% of new US Computer Science PhD graduates have specialized in \"AI\";[263]\nabout 800,000 \"AI\"-related US job openings existed in 2022.[264]\nIn 2016, issues of fairness and the misuse of technology were catapulted into center stage at machine learning conferences, publications vastly increased, funding became available, and many researchers re-focussed their careers on these issues. The alignment problem became a serious field of academic study.[214]\nPhilosophy\nDefining artificial intelligence\nAlan Turing wrote in 1950 \"I propose to consider the question 'can machines think'?\"[265]\nHe advised changing the question from whether a machine \"thinks\", to \"whether or not it is possible for machinery to show intelligent behaviour\".[265]\nHe devised the Turing test, which measures the ability of a machine to simulate human conversation.[266] Since we can only observe the behavior of the machine, it does not matter if it is \"actually\" thinking or literally has a \"mind\". Turing notes that we can not determine these things about other people[ab] but \"it is usual to have a polite convention that everyone thinks\"[267]\nRussell and Norvig agree with Turing that AI must be defined in terms of \"acting\" and not \"thinking\".[268] However, they are critical that the test compares machines to people. \"Aeronautical engineering texts,\" they wrote, \"do not define the goal of their field as making 'machines that fly so exactly like pigeons that they can fool other pigeons.'\"[269] AI founder John McCarthy agreed, writing that \"Artificial intelligence is not, by definition, simulation of human intelligence\".[270]\nMcCarthy defines intelligence as \"the computational part of the ability to achieve goals in the world.\"[271] Another AI founder, Marvin Minsky similarly defines it as \"the ability to solve hard problems\".[272] These definitions view intelligence in terms of well-defined problems with well-defined solutions, where both the difficulty of the problem and the performance of the program are direct measures of the \"intelligence\" of the machine—and no other philosophical discussion is required, or may not even be possible.\nAnother definition has been adopted by Google,[273] a major practitioner in the field of AI.\nThis definition stipulates the ability of systems to synthesize information as the manifestation of intelligence, similar to the way it is defined in biological intelligence.\nEvaluating approaches to AI\nNo established unifying theory or paradigm has guided AI research for most of its history.[ac] The unprecedented success of statistical machine learning in the 2010s eclipsed all other approaches (so much so that some sources, especially in the business world, use the term \"artificial intelligence\" to mean \"machine learning with neural networks\"). This approach is mostly sub-symbolic, soft and narrow (see below). Critics argue that these questions may have to be revisited by future generations of AI researchers.\nSymbolic AI (or \"GOFAI\")[275] simulated the high-level conscious reasoning that people use when they solve puzzles, express legal reasoning and do mathematics. They were highly successful at \"intelligent\" tasks such as algebra or IQ tests. In the 1960s, Newell and Simon proposed the physical symbol systems hypothesis: \"A physical symbol system has the necessary and sufficient means of general intelligent action.\"[276]\nHowever, the symbolic approach failed on many tasks that humans solve easily, such as learning, recognizing an object or commonsense reasoning. Moravec's paradox is the discovery that high-level \"intelligent\" tasks were easy for AI, but low level \"instinctive\" tasks were extremely difficult.[277]\nPhilosopher Hubert Dreyfus had argued since the 1960s that human expertise depends on unconscious instinct rather than conscious symbol manipulation, and on having a \"feel\" for the situation, rather than explicit symbolic knowledge.[278]\nAlthough his arguments had been ridiculed and ignored when they were first presented, eventually, AI research came to agree.[ad][13]\nThe issue is not resolved: sub-symbolic reasoning can make many of the same inscrutable mistakes that human intuition does, such as algorithmic bias. Critics such as Noam Chomsky argue continuing research into symbolic AI will still be necessary to attain general intelligence,[280][281] in part because sub-symbolic AI is a move away from explainable AI: it can be difficult or impossible to understand why a modern statistical AI program made a particular decision. The emerging field of neuro-symbolic artificial intelligence attempts to bridge the two approaches.\n\"Neats\" hope that intelligent behavior is described using simple, elegant principles (such as logic, optimization, or neural networks). \"Scruffies\" expect that it necessarily requires solving a large number of unrelated problems. Neats defend their programs with theoretical rigor, scruffies rely mainly on incremental testing to see if they work. This issue was actively discussed in the 70s and 80s,[282]\nbut eventually was seen as irrelevant. Modern AI has elements of both.\nFinding a provably correct or optimal solution is intractable for many important problems.[12] Soft computing is a set of techniques, including genetic algorithms, fuzzy logic and neural networks, that are tolerant of imprecision, uncertainty, partial truth and approximation. Soft computing was introduced in the late 80s and most successful AI programs in the 21st century are examples of soft computing with neural networks.\nAI researchers are divided as to whether to pursue the goals of artificial general intelligence and superintelligence directly or to solve as many specific problems as possible (narrow AI) in hopes these solutions will lead indirectly to the field's long-term goals.[283][284]\nGeneral intelligence is difficult to define and difficult to measure, and modern AI has had more verifiable successes by focusing on specific problems with specific solutions. The experimental sub-field of artificial general intelligence studies this area exclusively.\nMachine consciousness, sentience and mind\nThe philosophy of mind does not know whether a machine can have a mind, consciousness and mental states, in the same sense that human beings do. This issue considers the internal experiences of the machine, rather than its external behavior. Mainstream AI research considers this issue irrelevant because it does not affect the goals of the field: to build machines that can solve problems using intelligence. Russell and Norvig add that \"[t]he additional project of making a machine conscious in exactly the way humans are is not one that we are equipped to take on.\"[285] However, the question has become central to the philosophy of mind. It is also typically the central question at issue in artificial intelligence in fiction.\nDavid Chalmers identified two problems in understanding the mind, which he named the \"hard\" and \"easy\" problems of consciousness.[286] The easy problem is understanding how the brain processes signals, makes plans and controls behavior. The hard problem is explaining how this feels or why it should feel like anything at all, assuming we are right in thinking that it truly does feel like something (Dennett's consciousness illusionism says this is an illusion). Human information processing is easy to explain, however, human subjective experience is difficult to explain. For example, it is easy to imagine a color-blind person who has learned to identify which objects in their field of view are red, but it is not clear what would be required for the person to know what red looks like.[287]\nComputationalism is the position in the philosophy of mind that the human mind is an information processing system and that thinking is a form of computing. Computationalism argues that the relationship between mind and body is similar or identical to the relationship between software and hardware and thus may be a solution to the mind–body problem. This philosophical position was inspired by the work of AI researchers and cognitive scientists in the 1960s and was originally proposed by philosophers Jerry Fodor and Hilary Putnam.[288]\nPhilosopher John Searle characterized this position as \"strong AI\": \"The appropriately programmed computer with the right inputs and outputs would thereby have a mind in exactly the same sense human beings have minds.\"[ae]\nSearle counters this assertion with his Chinese room argument, which attempts to show that, even if a machine perfectly simulates human behavior, there is still no reason to suppose it also has a mind.[292]\nIf a machine has a mind and subjective experience, then it may also have sentience (the ability to feel), and if so it could also suffer; it has been argued that this could entitle it to certain rights.[293]\nAny hypothetical robot rights would lie on a spectrum with animal rights and human rights.[294]\nThis issue has been considered in fiction for centuries,[295]\nand is now being considered by, for example, California's Institute for the Future; however, critics argue that the discussion is premature.[296]\nFuture\nSuperintelligence and the singularity\nA superintelligence is a hypothetical agent that would possess intelligence far surpassing that of the brightest and most gifted human mind.[284]\nIf research into artificial general intelligence produced sufficiently intelligent software, it might be able to reprogram and improve itself. The improved software would be even better at improving itself, leading to what I. J. Good called an \"intelligence explosion\" and Vernor Vinge called a \"singularity\".[297]\nHowever, technologies can't improve exponentially indefinitely, and typically follow an S-shaped curve, slowing when they reach the physical limits of what the technology can do.[298]\nTranshumanism\nRobot designer Hans Moravec, cyberneticist Kevin Warwick, and inventor Ray Kurzweil have predicted that humans and machines will merge in the future into cyborgs that are more capable and powerful than either. This idea, called transhumanism, has roots in Aldous Huxley and Robert Ettinger.[299]\nEdward Fredkin argues that \"artificial intelligence is the next stage in evolution\", an idea first proposed by Samuel Butler's \"Darwin among the Machines\" as far back as 1863, and expanded upon by George Dyson in his book of the same name in 1998.[300]\nIn fiction\nThought-capable artificial beings have appeared as storytelling devices since antiquity,[301]\nand have been a persistent theme in science fiction.[302]\nA common trope in these works began with Mary Shelley's Frankenstein, where a human creation becomes a threat to its masters. This includes such works as Arthur C. Clarke's and Stanley Kubrick's 2001: A Space Odyssey (both 1968), with HAL 9000, the murderous computer in charge of the Discovery One spaceship, as well as The Terminator (1984) and The Matrix (1999). In contrast, the rare loyal robots such as Gort from The Day the Earth Stood Still (1951) and Bishop from Aliens (1986) are less prominent in popular culture.[303]\nIsaac Asimov introduced the Three Laws of Robotics in many books and stories, most notably the \"Multivac\" series about a super-intelligent computer of the same name. Asimov's laws are often brought up during lay discussions of machine ethics;[304]\nwhile almost all artificial intelligence researchers are familiar with Asimov's laws through popular culture, they generally consider the laws useless for many reasons, one of which is their ambiguity.[305]\nSeveral works use AI to force us to confront the fundamental question of what makes us human, showing us artificial beings that have the ability to feel, and thus to suffer. This appears in Karel Čapek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, as well as the novel Do Androids Dream of Electric Sheep?, by Philip K. Dick. Dick considers the idea that our understanding of human subjectivity is altered by technology created with artificial intelligence.[306]\nSee also\nExplanatory notes\nReferences\nAI textbooks\nThe two most widely used textbooks in 2023. (See the Open Syllabus).\nThese were the four the most widely used AI textbooks in 2008:\nLater editions.\nHistory of AI\nOther sources\nFurther reading\nExternal links"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Machine_learning",
            "raw_content": "Contents\nMachine learning\nMachine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can effectively generalize and thus perform tasks without explicit instructions.[1] Recently, generative artificial neural networks have been able to surpass many previous approaches in performance.[2][3] Machine learning approaches have been applied to large language models, computer vision, speech recognition, email filtering, agriculture and medicine, where it is too costly to develop algorithms to perform the needed tasks.[4][5]\nThe mathematical foundations of ML are provided by mathematical optimization (mathematical programming) methods. Data mining is a related (parallel) field of study, focusing on exploratory data analysis through unsupervised learning.[7][8]\nML is known in its application across business problems under the name predictive analytics. Although not all machine learning is statistically based, computational statistics is an important source of the field's methods.\nHistory and relationships to other fields[edit]\nThe term machine learning was coined in 1959 by Arthur Samuel, an IBM employee and pioneer in the field of computer gaming and artificial intelligence.[9][10] The synonym self-teaching computers was also used in this time period.[11][12]\nBy the early 1960s an experimental \"learning machine\" with punched tape memory, called Cybertron, had been developed by Raytheon Company to analyze sonar signals, electrocardiograms, and speech patterns using rudimentary reinforcement learning. It was repetitively \"trained\" by a human operator/teacher to recognize patterns and equipped with a \"goof\" button to cause it to re-evaluate incorrect decisions.[13] A representative book on research into machine learning during the 1960s was Nilsson's book on Learning Machines, dealing mostly with machine learning for pattern classification.[14] Interest related to pattern recognition continued into the 1970s, as described by Duda and Hart in 1973.[15] In 1981 a report was given on using teaching strategies so that a neural network learns to recognize 40 characters (26 letters, 10 digits, and 4 special symbols) from a computer terminal.[16]\nTom M. Mitchell provided a widely quoted, more formal definition of the algorithms studied in the machine learning field: \"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P,\nimproves with experience E.\"[17] This definition of the tasks in which machine learning is concerned offers a fundamentally operational definition rather than defining the field in cognitive terms. This follows Alan Turing's proposal in his paper \"Computing Machinery and Intelligence\", in which the question \"Can machines think?\" is replaced with the question \"Can machines do what we (as thinking entities) can do?\".[18]\nModern-day machine learning has two objectives, one is to classify data based on models which have been developed, the other purpose is to make predictions for future outcomes based on these models. A hypothetical algorithm specific to classifying data may use computer vision of moles coupled with supervised learning in order to train it to classify the cancerous moles. A machine learning algorithm for stock trading may inform the trader of future potential predictions.[19]\nArtificial intelligence[edit]\nAs a scientific endeavor, machine learning grew out of the quest for artificial intelligence (AI). In the early days of AI as an academic discipline, some researchers were interested in having machines learn from data. They attempted to approach the problem with various symbolic methods, as well as what were then termed \"neural networks\"; these were mostly perceptrons and other models that were later found to be reinventions of the generalized linear models of statistics.[21] Probabilistic reasoning was also employed, especially in automated medical diagnosis.[22]: 488\nHowever, an increasing emphasis on the logical, knowledge-based approach caused a rift between AI and machine learning. Probabilistic systems were plagued by theoretical and practical problems of data acquisition and representation.[22]: 488  By 1980, expert systems had come to dominate AI, and statistics was out of favor.[23] Work on symbolic/knowledge-based learning did continue within AI, leading to inductive logic programming, but the more statistical line of research was now outside the field of AI proper, in pattern recognition and information retrieval.[22]: 708–710, 755  Neural networks research had been abandoned by AI and computer science around the same time. This line, too, was continued outside the AI/CS field, as \"connectionism\", by researchers from other disciplines including Hopfield, Rumelhart, and Hinton. Their main success came in the mid-1980s with the reinvention of backpropagation.[22]: 25\nMachine learning (ML), reorganized and recognized as its own field, started to flourish in the 1990s. The field changed its goal from achieving artificial intelligence to tackling solvable problems of a practical nature. It shifted focus away from the symbolic approaches it had inherited from AI, and toward methods and models borrowed from statistics, fuzzy logic, and probability theory.[23]\nData mining[edit]\nMachine learning and data mining often employ the same methods and overlap significantly, but while machine learning focuses on prediction, based on known properties learned from the training data, data mining focuses on the discovery of (previously) unknown properties in the data (this is the analysis step of knowledge discovery in databases). Data mining uses many machine learning methods, but with different goals; on the other hand, machine learning also employs data mining methods as \"unsupervised learning\" or as a preprocessing step to improve learner accuracy. Much of the confusion between these two research communities (which do often have separate conferences and separate journals, ECML PKDD being a major exception) comes from the basic assumptions they work with: in machine learning, performance is usually evaluated with respect to the ability to reproduce known knowledge, while in knowledge discovery and data mining (KDD) the key task is the discovery of previously unknown knowledge. Evaluated with respect to known knowledge, an uninformed (unsupervised) method will easily be outperformed by other supervised methods, while in a typical KDD task, supervised methods cannot be used due to the unavailability of training data.\nMachine learning also has intimate ties to optimization: many learning problems are formulated as minimization of some loss function on a training set of examples. Loss functions express the discrepancy between the predictions of the model being trained and the actual problem instances (for example, in classification, one wants to assign a label to instances, and models are trained to correctly predict the pre-assigned labels of a set of examples).[24]\nGeneralization[edit]\nThe difference between optimization and machine learning arises from the goal of generalization: while optimization algorithms can minimize the loss on a training set, machine learning is concerned with minimizing the loss on unseen samples. Characterizing the generalization of various learning algorithms is an active topic of current research, especially for deep learning algorithms.\nStatistics[edit]\nMachine learning and statistics are closely related fields in terms of methods, but distinct in their principal goal: statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns.[25] According to Michael I. Jordan, the ideas of machine learning, from methodological principles to theoretical tools, have had a long pre-history in statistics.[26] He also suggested the term data science as a placeholder to call the overall field.[26]\nConventional statistical analyses require the a priori selection of a model most suitable for the study data set. In addition, only significant or theoretically relevant variables based on previous experience are included for analysis. In contrast, machine learning is not built on a pre-structured model; rather, the data shape the model by detecting underlying patterns. The more variables (input) used to train the model, the more accurate the ultimate model will be.[27]\nLeo Breiman distinguished two statistical modeling paradigms: data model and algorithmic model,[28] wherein \"algorithmic model\" means more or less the machine learning algorithms like Random Forest.\nSome statisticians have adopted methods from machine learning, leading to a combined field that they call statistical learning.[29]\nPhysics[edit]\nAnalytical and computational techniques derived from deep-rooted physics of disordered systems can be extended to large-scale problems, including machine learning, e.g., to analyze the weight space of deep neural networks.[30] Statistical physics is thus finding applications in the area of medical diagnostics.[31]\nTheory[edit]\nA core objective of a learner is to generalize from its experience.[6][32] Generalization in this context is the ability of a learning machine to perform accurately on new, unseen examples/tasks after having experienced a learning data set. The training examples come from some generally unknown probability distribution (considered representative of the space of occurrences) and the learner has to build a general model about this space that enables it to produce sufficiently accurate predictions in new cases.\nThe computational analysis of machine learning algorithms and their performance is a branch of theoretical computer science known as computational learning theory via the Probably Approximately Correct Learning (PAC) model. Because training sets are finite and the future is uncertain, learning theory usually does not yield guarantees of the performance of algorithms. Instead, probabilistic bounds on the performance are quite common. The bias–variance decomposition is one way to quantify generalization error.\nFor the best performance in the context of generalization, the complexity of the hypothesis should match the complexity of the function underlying the data. If the hypothesis is less complex than the function, then the model has under fitted the data. If the complexity of the model is increased in response, then the training error decreases. But if the hypothesis is too complex, then the model is subject to overfitting and generalization will be poorer.[33]\nIn addition to performance bounds, learning theorists study the time complexity and feasibility of learning. In computational learning theory, a computation is considered feasible if it can be done in polynomial time. There are two kinds of time complexity results: Positive results show that a certain class of functions can be learned in polynomial time. Negative results show that certain classes cannot be learned in polynomial time.\nApproaches[edit]\nMachine learning approaches are traditionally divided into three broad categories, which correspond to learning paradigms, depending on the nature of the \"signal\" or \"feedback\" available to the learning system:\nSupervised learning[edit]\nSupervised learning algorithms build a mathematical model of a set of data that contains both the inputs and the desired outputs.[37] The data is known as training data, and consists of a set of training examples. Each training example has one or more inputs and the desired output, also known as a supervisory signal.\nIn the mathematical model, each training example is represented by an array or vector, sometimes called a feature vector, and the training data is represented by a matrix. Through iterative optimization of an objective function, supervised learning algorithms learn a function that can be used to predict the output associated with new inputs.[38] An optimal function allows the algorithm to correctly determine the output for inputs that were not a part of the training data. An algorithm that improves the accuracy of its outputs or predictions over time is said to have learned to perform that task.[17]\nTypes of supervised-learning algorithms include active learning, classification and regression.[39] Classification algorithms are used when the outputs are restricted to a limited set of values, and regression algorithms are used when the outputs may have any numerical value within a range. As an example, for a classification algorithm that filters emails, the input would be an incoming email, and the output would be the name of the folder in which to file the email.\nSimilarity learning is an area of supervised machine learning closely related to regression and classification, but the goal is to learn from examples using a similarity function that measures how similar or related two objects are. It has applications in ranking, recommendation systems, visual identity tracking, face verification, and speaker verification.\nUnsupervised learning[edit]\nUnsupervised learning algorithms take a set of data that contains only inputs, and find structure in the data, like grouping or clustering of data points. The algorithms, therefore, learn from test data that has not been labeled, classified or categorized. Instead of responding to feedback, unsupervised learning algorithms identify commonalities in the data and react based on the presence or absence of such commonalities in each new piece of data. A central application of unsupervised learning is in the field of density estimation in statistics, such as finding the probability density function.[40] Though unsupervised learning encompasses other domains involving summarizing and explaining data features. Unsupervised learning algorithms streamlined the process of survey and graph large indel based haplotypes of a gene of interest from pan-genome.[41]\nCluster analysis is the assignment of a set of observations into subsets (called clusters) so that observations within the same cluster are similar according to one or more predesignated criteria, while observations drawn from different clusters are dissimilar. Different clustering techniques make different assumptions on the structure of the data, often defined by some similarity metric and evaluated, for example, by internal compactness, or the similarity between members of the same cluster, and separation, the difference between clusters. Other methods are based on estimated density and graph connectivity.\nSemi-supervised learning[edit]\nSemi-supervised learning falls between unsupervised learning (without any labeled training data) and supervised learning (with completely labeled training data). Some of the training examples are missing training labels, yet many machine-learning researchers have found that unlabeled data, when used in conjunction with a small amount of labeled data, can produce a considerable improvement in learning accuracy.\nIn weakly supervised learning, the training labels are noisy, limited, or imprecise; however, these labels are often cheaper to obtain, resulting in larger effective training sets.[42]\nReinforcement learning[edit]\nReinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. Due to its generality, the field is studied in many other disciplines, such as game theory, control theory, operations research, information theory, simulation-based optimization, multi-agent systems, swarm intelligence, statistics and genetic algorithms. In reinforcement learning, the environment is typically represented as a Markov decision process (MDP). Many reinforcements learning algorithms use dynamic programming techniques.[43] Reinforcement learning algorithms do not assume knowledge of an exact mathematical model of the MDP and are used when exact models are infeasible. Reinforcement learning algorithms are used in autonomous vehicles or in learning to play a game against a human opponent.\nDimensionality reduction[edit]\nDimensionality reduction is a process of reducing the number of random variables under consideration by obtaining a set of principal variables.[44] In other words, it is a process of reducing the dimension of the feature set, also called the \"number of features\". Most of the dimensionality reduction techniques can be considered as either feature elimination or extraction. One of the popular methods of dimensionality reduction is principal component analysis (PCA). PCA involves changing higher-dimensional data (e.g., 3D) to a smaller space (e.g., 2D). This results in a smaller dimension of data (2D instead of 3D), while keeping all original variables in the model without changing the data.[45]\nThe manifold hypothesis proposes that high-dimensional data sets lie along low-dimensional manifolds, and many dimensionality reduction techniques make this assumption, leading to the area of manifold learning and manifold regularization.\nOther types[edit]\nOther approaches have been developed which do not fit neatly into this three-fold categorization, and sometimes more than one is used by the same machine learning system. For example, topic modeling, meta-learning.[46]\nSelf-learning, as a machine learning paradigm was introduced in 1982 along with a neural network capable of self-learning, named crossbar adaptive array (CAA).[47] It is learning with no external rewards and no external teacher advice. The CAA self-learning algorithm computes, in a crossbar fashion, both decisions about actions and emotions (feelings) about consequence situations. The system is driven by the interaction between cognition and emotion.[48]\nThe self-learning algorithm updates a memory matrix W =||w(a,s)|| such that in each iteration executes the following machine learning routine:\nIt is a system with only one input, situation, and only one output, action (or behavior) a. There is neither a separate reinforcement input nor an advice input from the environment. The backpropagated value (secondary reinforcement) is the emotion toward the consequence situation. The CAA exists in two environments, one is the behavioral environment where it behaves, and the other is the genetic environment, wherefrom it initially and only once receives initial emotions about situations to be encountered in the behavioral environment. After receiving the genome (species) vector from the genetic environment, the CAA learns a goal-seeking behavior, in an environment that contains both desirable and undesirable situations.[49]\nSeveral learning algorithms aim at discovering better representations of the inputs provided during training.[50] Classic examples include principal component analysis and cluster analysis. Feature learning algorithms, also called representation learning algorithms, often attempt to preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before performing classification or predictions. This technique allows reconstruction of the inputs coming from the unknown data-generating distribution, while not being necessarily faithful to configurations that are implausible under that distribution. This replaces manual feature engineering, and allows a machine to both learn the features and use them to perform a specific task.\nFeature learning can be either supervised or unsupervised. In supervised feature learning, features are learned using labeled input data. Examples include artificial neural networks, multilayer perceptrons, and supervised dictionary learning. In unsupervised feature learning, features are learned with unlabeled input data.\nExamples include dictionary learning, independent component analysis, autoencoders, matrix factorization[51] and various forms of clustering.[52][53][54]\nManifold learning algorithms attempt to do so under the constraint that the learned representation is low-dimensional. Sparse coding algorithms attempt to do so under the constraint that the learned representation is sparse, meaning that the mathematical model has many zeros. Multilinear subspace learning algorithms aim to learn low-dimensional representations directly from tensor representations for multidimensional data, without reshaping them into higher-dimensional vectors.[55] Deep learning algorithms discover multiple levels of representation, or a hierarchy of features, with higher-level, more abstract features defined in terms of (or generating) lower-level features. It has been argued that an intelligent machine is one that learns a representation that disentangles the underlying factors of variation that explain the observed data.[56]\nFeature learning is motivated by the fact that machine learning tasks such as classification often require input that is mathematically and computationally convenient to process. However, real-world data such as images, video, and sensory data has not yielded attempts to algorithmically define specific features. An alternative is to discover such features or representations through examination, without relying on explicit algorithms.\nSparse dictionary learning is a feature learning method where a training example is represented as a linear combination of basis functions, and is assumed to be a sparse matrix. The method is strongly NP-hard and difficult to solve approximately.[57] A popular heuristic method for sparse dictionary learning is the K-SVD algorithm. Sparse dictionary learning has been applied in several contexts. In classification, the problem is to determine the class to which a previously unseen training example belongs. For a dictionary where each class has already been built, a new training example is associated with the class that is best sparsely represented by the corresponding dictionary. Sparse dictionary learning has also been applied in image de-noising. The key idea is that a clean image patch can be sparsely represented by an image dictionary, but the noise cannot.[58]\nIn data mining, anomaly detection, also known as outlier detection, is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.[59] Typically, the anomalous items represent an issue such as bank fraud, a structural defect, medical problems or errors in a text. Anomalies are referred to as outliers, novelties, noise, deviations and exceptions.[60]\nIn particular, in the context of abuse and network intrusion detection, the interesting objects are often not rare objects, but unexpected bursts of inactivity. This pattern does not adhere to the common statistical definition of an outlier as a rare object. Many outlier detection methods (in particular, unsupervised algorithms) will fail on such data unless aggregated appropriately. Instead, a cluster analysis algorithm may be able to detect the micro-clusters formed by these patterns.[61]\nThree broad categories of anomaly detection techniques exist.[62] Unsupervised anomaly detection techniques detect anomalies in an unlabeled test data set under the assumption that the majority of the instances in the data set are normal, by looking for instances that seem to fit the least to the remainder of the data set. Supervised anomaly detection techniques require a data set that has been labeled as \"normal\" and \"abnormal\" and involves training a classifier (the key difference to many other statistical classification problems is the inherently unbalanced nature of outlier detection). Semi-supervised anomaly detection techniques construct a model representing normal behavior from a given normal training data set and then test the likelihood of a test instance to be generated by the model.\nRobot learning is inspired by a multitude of machine learning methods, starting from supervised learning, reinforcement learning,[63][64] and finally meta-learning (e.g. MAML).\nAssociation rule learning is a rule-based machine learning method for discovering relationships between variables in large databases. It is intended to identify strong rules discovered in databases using some measure of \"interestingness\".[65]\nRule-based machine learning is a general term for any machine learning method that identifies, learns, or evolves \"rules\" to store, manipulate or apply knowledge. The defining characteristic of a rule-based machine learning algorithm is the identification and utilization of a set of relational rules that collectively represent the knowledge captured by the system. This is in contrast to other machine learning algorithms that commonly identify a singular model that can be universally applied to any instance in order to make a prediction.[66] Rule-based machine learning approaches include learning classifier systems, association rule learning, and artificial immune systems.\nBased on the concept of strong rules, Rakesh Agrawal, Tomasz Imieliński and Arun Swami introduced association rules for discovering regularities between products in large-scale transaction data recorded by point-of-sale (POS) systems in supermarkets.[67] For example, the rule\n{\no\nn\ni\no\nn\ns\n,\np\no\nt\na\nt\no\ne\ns\n}\n⇒\n{\nb\nu\nr\ng\ne\nr\n}\n{\\displaystyle \\{\\mathrm {onions,potatoes} \\}\\Rightarrow \\{\\mathrm {burger} \\}}\nfound in the sales data of a supermarket would indicate that if a customer buys onions and potatoes together, they are likely to also buy hamburger meat. Such information can be used as the basis for decisions about marketing activities such as promotional pricing or product placements. In addition to market basket analysis, association rules are employed today in application areas including Web usage mining, intrusion detection, continuous production, and bioinformatics. In contrast with sequence mining, association rule learning typically does not consider the order of items either within a transaction or across transactions.\nLearning classifier systems (LCS) are a family of rule-based machine learning algorithms that combine a discovery component, typically a genetic algorithm, with a learning component, performing either supervised learning, reinforcement learning, or unsupervised learning. They seek to identify a set of context-dependent rules that collectively store and apply knowledge in a piecewise manner in order to make predictions.[68]\nInductive logic programming (ILP) is an approach to rule learning using logic programming as a uniform representation for input examples, background knowledge, and hypotheses. Given an encoding of the known background knowledge and a set of examples represented as a logical database of facts, an ILP system will derive a hypothesized logic program that entails all positive and no negative examples. Inductive programming is a related field that considers any kind of programming language for representing hypotheses (and not only logic programming), such as functional programs.\nInductive logic programming is particularly useful in bioinformatics and natural language processing. Gordon Plotkin and Ehud Shapiro laid the initial theoretical foundation for inductive machine learning in a logical setting.[69][70][71] Shapiro built their first implementation (Model Inference System) in 1981: a Prolog program that inductively inferred logic programs from positive and negative examples.[72] The term inductive here refers to philosophical induction, suggesting a theory to explain observed facts, rather than mathematical induction, proving a property for all members of a well-ordered set.\nModels[edit]\nPerforming machine learning can involve creating a model, which is trained on some training data and then can process additional data to make predictions. Various types of models have been used and researched for machine learning systems.\nArtificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that constitute animal brains. Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with any task-specific rules.\nAn ANN is a model based on a collection of connected units or nodes called \"artificial neurons\", which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit information, a \"signal\", from one artificial neuron to another. An artificial neuron that receives a signal can process it and then signal additional artificial neurons connected to it. In common ANN implementations, the signal at a connection between artificial neurons is a real number, and the output of each artificial neuron is computed by some non-linear function of the sum of its inputs. The connections between artificial neurons are called \"edges\". Artificial neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Artificial neurons may have a threshold such that the signal is only sent if the aggregate signal crosses that threshold. Typically, artificial neurons are aggregated into layers. Different layers may perform different kinds of transformations on their inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), possibly after traversing the layers multiple times.\nThe original goal of the ANN approach was to solve problems in the same way that a human brain would. However, over time, attention moved to performing specific tasks, leading to deviations from biology. Artificial neural networks have been used on a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games and medical diagnosis.\nDeep learning consists of multiple hidden layers in an artificial neural network. This approach tries to model the way the human brain processes light and sound into vision and hearing. Some successful applications of deep learning are computer vision and speech recognition.[73]\nDecision tree learning uses a decision tree as a predictive model to go from observations about an item (represented in the branches) to conclusions about the item's target value (represented in the leaves). It is one of the predictive modeling approaches used in statistics, data mining, and machine learning. Tree models where the target variable can take a discrete set of values are called classification trees; in these tree structures, leaves represent class labels, and branches represent conjunctions of features that lead to those class labels. Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees. In decision analysis, a decision tree can be used to visually and explicitly represent decisions and decision making. In data mining, a decision tree describes data, but the resulting classification tree can be an input for decision-making.\nSupport-vector machines (SVMs), also known as support-vector networks, are a set of related supervised learning methods used for classification and regression. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that predicts whether a new example falls into one category.[74]\nAn SVM training algorithm is a non-probabilistic, binary, linear classifier, although methods such as Platt scaling exist to use SVM in a probabilistic classification setting. In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.\nRegression analysis encompasses a large variety of statistical methods to estimate the relationship between input variables and their associated features. Its most common form is linear regression, where a single line is drawn to best fit the given data according to a mathematical criterion such as ordinary least squares. The latter is often extended by regularization methods to mitigate overfitting and bias, as in ridge regression. When dealing with non-linear problems, go-to models include polynomial regression (for example, used for trendline fitting in Microsoft Excel[75]), logistic regression (often used in statistical classification) or even kernel regression, which introduces non-linearity by taking advantage of the kernel trick to implicitly map input variables to higher-dimensional space.\nA Bayesian network, belief network, or directed acyclic graphical model is a probabilistic graphical model that represents a set of random variables and their conditional independence with a directed acyclic graph (DAG). For example, a Bayesian network could represent the probabilistic relationships between diseases and symptoms. Given symptoms, the network can be used to compute the probabilities of the presence of various diseases. Efficient algorithms exist that perform inference and learning. Bayesian networks that model sequences of variables, like speech signals or protein sequences, are called dynamic Bayesian networks. Generalizations of Bayesian networks that can represent and solve decision problems under uncertainty are called influence diagrams.\nA Gaussian process is a stochastic process in which every finite collection of the random variables in the process has a multivariate normal distribution, and it relies on a pre-defined covariance function, or kernel, that models how pairs of points relate to each other depending on their locations.\nGiven a set of observed points, or input–output examples, the distribution of the (unobserved) output of a new point as function of its input data can be directly computed by looking like the observed points and the covariances between those points and the new, unobserved point.\nGaussian processes are popular surrogate models in Bayesian optimization used to do hyperparameter optimization.\nA genetic algorithm (GA) is a search algorithm and heuristic technique that mimics the process of natural selection, using methods such as mutation and crossover to generate new genotypes in the hope of finding good solutions to a given problem. In machine learning, genetic algorithms were used in the 1980s and 1990s.[77][78] Conversely, machine learning techniques have been used to improve the performance of genetic and evolutionary algorithms.[79]\nThe theory of belief functions, also referred to as evidence theory or Dempster–Shafer theory, is a general framework for reasoning with uncertainty, with understood connections to other frameworks such as probability, possibility and\nimprecise probability theories. These theoretical frameworks can be thought of as a kind of learner and have some analogous properties of how evidence is combined (e.g.,\nDempster's rule of combination), just like how in a pmf-based Bayesian approach would combine probabilities. However, there are many caveats to these beliefs functions when compared to Bayesian approaches in order to incorporate ignorance and Uncertainty quantification. These belief function approaches that are implemented within the machine learning domain typically leverage a fusion approach of various ensemble methods to better handle the learner's decision boundary, low samples, and ambiguous class issues that standard machine learning approach tend to have difficulty resolving.[3][5][10] However, the computational complexity of these algorithms are dependent on the number of propositions (classes), and can lead a much higher computation time when compared to other machine learning approaches.\nTraining models[edit]\nTypically, machine learning models require a high quantity of reliable data in order for the models to perform accurate predictions. When training a machine learning model, machine learning engineers need to target and collect a large and representative sample of data. Data from the training set can be as varied as a corpus of text, a collection of images, sensor data, and data collected from individual users of a service. Overfitting is something to watch out for when training a machine learning model. Trained models derived from biased or non-evaluated data can result in skewed or undesired predictions. Bias models may result in detrimental outcomes thereby furthering the negative impacts on society or objectives. Algorithmic bias is a potential result of data not being fully prepared for training. Machine learning ethics is becoming a field of study and notably be integrated within machine learning engineering teams.\nFederated learning is an adapted form of distributed artificial intelligence to training machine learning models that decentralizes the training process, allowing for users' privacy to be maintained by not needing to send their data to a centralized server. This also increases efficiency by decentralizing the training process to many devices. For example, Gboard uses federated machine learning to train search query prediction models on users' mobile phones without having to send individual searches back to Google.[80]\nApplications[edit]\nThere are many applications for machine learning, including:\nIn 2006, the media-services provider Netflix held the first \"Netflix Prize\" competition to find a program to better predict user preferences and improve the accuracy of its existing Cinematch movie recommendation algorithm by at least 10%. A joint team made up of researchers from AT&T Labs-Research in collaboration with the teams Big Chaos and Pragmatic Theory built an ensemble model to win the Grand Prize in 2009 for $1 million.[83] Shortly after the prize was awarded, Netflix realized that viewers' ratings were not the best indicators of their viewing patterns (\"everything is a recommendation\") and they changed their recommendation engine accordingly.[84] In 2010 The Wall Street Journal wrote about the firm Rebellion Research and their use of machine learning to predict the financial crisis.[85] In 2012, co-founder of Sun Microsystems, Vinod Khosla, predicted that 80% of medical doctors jobs would be lost in the next two decades to automated machine learning medical diagnostic software.[86] In 2014, it was reported that a machine learning algorithm had been applied in the field of art history to study fine art paintings and that it may have revealed previously unrecognized influences among artists.[87] In 2019 Springer Nature published the first research book created using machine learning.[88] In 2020, machine learning technology was used to help make diagnoses and aid researchers in developing a cure for COVID-19.[89] Machine learning was recently applied to predict the pro-environmental behavior of travelers.[90] Recently, machine learning technology was also applied to optimize smartphone's performance and thermal behavior based on the user's interaction with the phone.[91][92][93]\nLimitations[edit]\nAlthough machine learning has been transformative in some fields, machine-learning programs often fail to deliver expected results.[94][95][96] Reasons for this are numerous: lack of (suitable) data, lack of access to the data, data bias, privacy problems, badly chosen tasks and algorithms, wrong tools and people, lack of resources, and evaluation problems.[97]\nIn 2018, a self-driving car from Uber failed to detect a pedestrian, who was killed after a collision.[98] Attempts to use machine learning in healthcare with the IBM Watson system failed to deliver even after years of time and billions of dollars invested.[99][100]\nMachine learning has been used as a strategy to update the evidence related to a systematic review and increased reviewer burden related to the growth of biomedical literature. While it has improved with training sets, it has not yet developed sufficiently to reduce the workload burden without limiting the necessary sensitivity for the findings research themselves.[101]\nBias[edit]\nMachine learning approaches in particular can suffer from different data biases. A machine learning system trained specifically on current customers may not be able to predict the needs of new customer groups that are not represented in the training data. When trained on human-made data, machine learning is likely to pick up the constitutional and unconscious biases already present in society.[102] Language models learned from data have been shown to contain human-like biases.[103][104] Machine learning systems used for criminal risk assessment have been found to be biased against black people.[105][106] In 2015, Google photos would often tag black people as gorillas,[107] and in 2018 this still was not well resolved, but Google reportedly was still using the workaround to remove all gorillas from the training data, and thus was not able to recognize real gorillas at all.[108] Similar issues with recognizing non-white people have been found in many other systems.[109] In 2016, Microsoft tested a chatbot that learned from Twitter, and it quickly picked up racist and sexist language.[110] Because of such challenges, the effective use of machine learning may take longer to be adopted in other domains.[111] Concern for fairness in machine learning, that is, reducing bias in machine learning and propelling its use for human good is increasingly expressed by artificial intelligence scientists, including Fei-Fei Li, who reminds engineers that \"There's nothing artificial about AI...It's inspired by people, it's created by people, and—most importantly—it impacts people. It is a powerful tool we are only just beginning to understand, and that is a profound responsibility.\"[112]\nExplainability[edit]\nExplainable AI (XAI), or Interpretable AI, or Explainable Machine Learning (XML), is artificial intelligence (AI) in which humans can understand the decisions or predictions made by the AI.[113] It contrasts with the \"black box\" concept in machine learning where even its designers cannot explain why an AI arrived at a specific decision.[114] By refining the mental models of users of AI-powered systems and dismantling their misconceptions, XAI promises to help users perform more effectively. XAI may be an implementation of the social right to explanation.\nOverfitting[edit]\nSettling on a bad, overly complex theory gerrymandered to fit all the past training data is known as overfitting. Many systems attempt to reduce overfitting by rewarding a theory in accordance with how well it fits the data but penalizing the theory in accordance with how complex the theory is.[115]\nOther limitations and vulnerabilities[edit]\nLearners can also disappoint by \"learning the wrong lesson\". A toy example is that an image classifier trained only on pictures of brown horses and black cats might conclude that all brown patches are likely to be horses.[116] A real-world example is that, unlike humans, current image classifiers often do not primarily make judgments from the spatial relationship between components of the picture, and they learn relationships between pixels that humans are oblivious to, but that still correlate with images of certain types of real objects. Modifying these patterns on a legitimate image can result in \"adversarial\" images that the system misclassifies.[117][118]\nAdversarial vulnerabilities can also result in nonlinear systems, or from non-pattern perturbations. Some systems are so brittle that changing a single adversarial pixel predictably induces misclassification.[citation needed] Machine learning models are often vulnerable to manipulation and/or evasion via adversarial machine learning.[119]\nResearchers have demonstrated how backdoors can be placed undetectably into classifying (e.g., for categories \"spam\" and well-visible \"not spam\" of posts) machine learning models which are often developed and/or trained by third parties. Parties can change the classification of any input, including in cases for which a type of data/software transparency is provided, possibly including white-box access.[120][121][122]\nModel assessments[edit]\nClassification of machine learning models can be validated by accuracy estimation techniques like the holdout method, which splits the data in a training and test set (conventionally 2/3 training set and 1/3 test set designation) and evaluates the performance of the training model on the test set. In comparison, the K-fold-cross-validation method randomly partitions the data into K subsets and then K experiments are performed each respectively considering 1 subset for evaluation and the remaining K-1 subsets for training the model. In addition to the holdout and cross-validation methods, bootstrap, which samples n instances with replacement from the dataset, can be used to assess model accuracy.[123]\nIn addition to overall accuracy, investigators frequently report sensitivity and specificity meaning True Positive Rate (TPR) and True Negative Rate (TNR) respectively. Similarly, investigators sometimes report the false positive rate (FPR) as well as the false negative rate (FNR). However, these rates are ratios that fail to reveal their numerators and denominators. The total operating characteristic (TOC) is an effective method to express a model's diagnostic ability. TOC shows the numerators and denominators of the previously mentioned rates, thus TOC provides more information than the commonly used receiver operating characteristic (ROC) and ROC's associated area under the curve (AUC).[124]\nEthics[edit]\nMachine learning poses a host of ethical questions. Systems that are trained on datasets collected with biases may exhibit these biases upon use (algorithmic bias), thus digitizing cultural prejudices.[125] For example, in 1988, the UK's Commission for Racial Equality found that St. George's Medical School had been using a computer program trained from data of previous admissions staff and this program had denied nearly 60 candidates who were found to be either women or had non-European sounding names.[102] Using job hiring data from a firm with racist hiring policies may lead to a machine learning system duplicating the bias by scoring job applicants by similarity to previous successful applicants.[126][127] Responsible collection of data and documentation of algorithmic rules used by a system thus is a critical part of machine learning.\nAI can be well-equipped to make decisions in technical fields, which rely heavily on data and historical information. These decisions rely on objectivity and logical reasoning.[128] Because human languages contain biases, machines trained on language corpora will necessarily also learn these biases.[129][130]\nOther forms of ethical challenges, not related to personal biases, are seen in health care. There are concerns among health care professionals that these systems might not be designed in the public's interest but as income-generating machines.[131] This is especially true in the United States where there is a long-standing ethical dilemma of improving health care, but also increasing profits. For example, the algorithms could be designed to provide patients with unnecessary tests or medication in which the algorithm's proprietary owners hold stakes. There is potential for machine learning in health care to provide professionals an additional tool to diagnose, medicate, and plan recovery paths for patients, but this requires these biases to be mitigated.[132]\nHardware[edit]\nSince the 2010s, advances in both machine learning algorithms and computer hardware have led to more efficient methods for training deep neural networks (a particular narrow subdomain of machine learning) that contain many layers of non-linear hidden units.[133] By 2019, graphic processing units (GPUs), often with AI-specific enhancements, had displaced CPUs as the dominant method of training large-scale commercial cloud AI.[134] OpenAI estimated the hardware computing used in the largest deep learning projects from AlexNet (2012) to AlphaZero (2017), and found a 300,000-fold increase in the amount of compute required, with a doubling-time trendline of 3.4 months.[135][136]\nNeuromorphic/Physical Neural Networks[edit]\nA physical neural network or Neuromorphic computer\nis a type of artificial neural network in which an electrically adjustable material is used to emulate the function of a neural synapse. \"Physical\" neural network is used to emphasize the reliance on physical hardware used to emulate neurons as opposed to software-based approaches. More generally the term is applicable to other artificial neural networks in which a memristor or other electrically adjustable resistance material is used to emulate a neural synapse.[137][138]\nEmbedded Machine Learning[edit]\nEmbedded Machine Learning is a sub-field of machine learning, where the machine learning model is run on embedded systems with limited computing resources such as wearable computers, edge devices and microcontrollers.[139][140][141] Running machine learning model in embedded devices removes the need for transferring and storing data on cloud servers for further processing, henceforth, reducing data breaches and privacy leaks happening because of transferring data, and also minimizes theft of intellectual properties, personal data and business secrets. Embedded Machine Learning could be applied through several techniques including hardware acceleration,[142][143] using approximate computing,[144] optimization of machine learning models and many more.[145][146]\nSoftware[edit]\nSoftware suites containing a variety of machine learning algorithms include the following:\nFree and open-source software[edit]\nProprietary software with free and open-source editions[edit]\nProprietary software[edit]\nJournals[edit]\nConferences[edit]\nSee also[edit]\nReferences[edit]\nSources[edit]\nFurther reading[edit]\nExternal links[edit]"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Data_science",
            "raw_content": "Contents\nData science\nData science is an interdisciplinary academic field[1] that uses statistics, scientific computing, scientific methods, processes, algorithms and systems to extract or extrapolate knowledge and insights from potentially noisy, structured, or unstructured data.[2]\nData science also integrates domain knowledge from the underlying application domain (e.g., natural sciences, information technology, and medicine).[3] Data science is multifaceted and can be described as a science, a research paradigm, a research method, a discipline, a workflow, and a profession.[4]\nData science is \"a concept to unify statistics, data analysis, informatics, and their related methods\" to \"understand and analyze actual phenomena\" with data.[5] It uses techniques and theories drawn from many fields within the context of mathematics, statistics, computer science, information science, and domain knowledge.[6] However, data science is different from computer science and information science. Turing Award winner Jim Gray imagined data science as a \"fourth paradigm\" of science (empirical, theoretical, computational, and now data-driven) and asserted that \"everything about science is changing because of the impact of information technology\" and the data deluge.[7][8]\nA\ndata scientist is a professional who creates programming code and combines it with statistical knowledge to create insights from data.[9]\nFoundations[edit]\nData science is an interdisciplinary field[10] focused on extracting knowledge from typically large data sets and applying the knowledge and insights from that data to solve problems in a wide range of application domains. The field encompasses preparing data for analysis, formulating data science problems, analyzing data, developing data-driven solutions, and presenting findings to inform high-level decisions in a broad range of application domains. As such, it incorporates skills from computer science, statistics, information science, mathematics, data visualization, information visualization, data sonification, data integration, graphic design, complex systems, communication and business.[11][12] Statistician Nathan Yau, drawing on Ben Fry, also links data science to human–computer interaction: users should be able to intuitively control and explore data.[13][14] In 2015, the American Statistical Association identified database management, statistics and machine learning, and distributed and parallel systems as the three emerging foundational professional communities.[15]\nRelationship to statistics[edit]\nMany statisticians, including Nate Silver, have argued that data science is not a new field, but rather another name for statistics.[16] Others argue that data science is distinct from statistics because it focuses on problems and techniques unique to digital data.[17] Vasant Dhar writes that statistics emphasizes quantitative data and description. In contrast, data science deals with quantitative and qualitative data (e.g., from images, text, sensors, transactions, customer information, etc.) and emphasizes prediction and action.[18] Andrew Gelman of Columbia University has described statistics as a non-essential part of data science.[19]\nStanford professor David Donoho writes that data science is not distinguished from statistics by the size of datasets or use of computing and that many graduate programs misleadingly advertise their analytics and statistics training as the essence of a data-science program. He describes data science as an applied field growing out of traditional statistics.[20]\nEtymology[edit]\nEarly usage[edit]\nIn 1962, John Tukey described a field he called \"data analysis\", which resembles modern data science.[20] In 1985, in a lecture given to the Chinese Academy of Sciences in Beijing, C. F. Jeff Wu used the term \"data science\" for the first time as an alternative name for statistics.[21] Later, attendees at a 1992 statistics symposium at the University of Montpellier  II acknowledged the emergence of a new discipline focused on data of various origins and forms, combining established concepts and principles of statistics and data analysis with computing.[22][23]\nThe term \"data science\" has been traced back to 1974, when Peter Naur proposed it as an alternative name to computer science.[6] In 1996, the International Federation of Classification Societies became the first conference to specifically feature data science as a topic.[6] However, the definition was still in flux. After the 1985 lecture at the Chinese Academy of Sciences in Beijing, in 1997 C. F. Jeff Wu again suggested that statistics should be renamed data science. He reasoned that a new name would help statistics shed inaccurate stereotypes, such as being synonymous with accounting or limited to describing data.[24] In 1998, Hayashi Chikio argued for data science as a new, interdisciplinary concept, with three aspects: data design, collection, and analysis.[23]\nDuring the 1990s, popular terms for the process of finding patterns in datasets (which were increasingly large) included \"knowledge discovery\" and \"data mining\".[6][25]\nModern usage[edit]\nIn 2012, technologists Thomas H. Davenport and DJ Patil declared \"Data Scientist: The Sexiest Job of the 21st Century\",[26] a catchphrase that was picked up even by major-city newspapers like the New York Times[27] and the Boston Globe.[28] A decade later, they reaffirmed it, stating that \"the job is more in demand than ever with employers\".[29]\nThe modern conception of data science as an independent discipline is sometimes attributed to William S. Cleveland.[30] In a 2001 paper, he advocated an expansion of statistics beyond theory into technical areas; because this would significantly change the field, it warranted a new name.[25] \"Data science\" became more widely used in the next few years: in 2002, the Committee on Data for Science and Technology launched the Data Science Journal. In 2003, Columbia University launched The Journal of Data Science.[25] In 2014, the American Statistical Association's Section on Statistical Learning and Data Mining changed its name to the Section on Statistical Learning and Data Science, reflecting the ascendant popularity of data science.[31]\nThe professional title of \"data scientist\" has been attributed to DJ Patil and Jeff Hammerbacher in 2008.[32] Though it was used by the National Science Board in their 2005 report \"Long-Lived Digital Data Collections: Enabling Research and Education in the 21st Century\", it referred broadly to any key role in managing a digital data collection.[33]\nThere is still no consensus on the definition of data science, and it is considered by some to be a buzzword.[34] Big data is a related marketing term.[35] Data scientists are responsible for breaking down big data into usable information and creating software and algorithms that help companies and organizations determine optimal operations.[36]\nData science and data analysis[edit]\nData science and data analysis are both important disciplines in the field of data management and analysis, but they differ in several key ways. While both fields involve working with data, data science is more of an interdisciplinary field that involves the application of statistical, computational, and machine learning methods to extract insights from data and make predictions, while data analysis is more focused on the examination and interpretation of data to identify patterns and trends.[37][38]\nData analysis typically involves working with smaller, structured datasets to answer specific questions or solve specific problems. This can involve tasks such as data cleaning, data visualization, and exploratory data analysis to gain insights into the data and develop hypotheses about relationships between variables. Data analysts typically use statistical methods to test these hypotheses and draw conclusions from the data. For example, a data analyst might analyze sales data to identify trends in customer behavior and make recommendations for marketing strategies.[37]\nData science, on the other hand, is a more complex and iterative process that involves working with larger, more complex datasets that often require advanced computational and statistical methods to analyze. Data scientists often work with unstructured data such as text or images and use machine learning algorithms to build predictive models and make data-driven decisions. In addition to statistical analysis, data science often involves tasks such as data preprocessing, feature engineering, and model selection. For instance, a data scientist might develop a recommendation system for an e-commerce platform by analyzing user behavior patterns and using machine learning algorithms to predict user preferences.[38][39]\nWhile data analysis focuses on extracting insights from existing data, data science goes beyond that by incorporating the development and implementation of predictive models to make informed decisions. Data scientists are often responsible for collecting and cleaning data, selecting appropriate analytical techniques, and deploying models in real-world scenarios. They work at the intersection of mathematics, computer science, and domain expertise to solve complex problems and uncover hidden patterns in large datasets.[38]\nDespite these differences, data science and data analysis are closely related fields and often require similar skill sets. Both fields require a solid foundation in statistics, programming, and data visualization, as well as the ability to communicate findings effectively to both technical and non-technical audiences. Both fields benefit from critical thinking and domain knowledge, as understanding the context and nuances of the data is essential for accurate analysis and modeling.[37][38]\nIn summary, data analysis and data science are distinct yet interconnected disciplines within the broader field of data management and analysis. Data analysis focuses on extracting insights and drawing conclusions from structured data, while data science involves a more comprehensive approach that combines statistical analysis, computational methods, and machine learning to extract insights, build predictive models, and drive data-driven decision-making. Both fields use data to understand patterns, make informed decisions, and solve complex problems across various domains.\nHistory[edit]\nSee also[edit]\nReferences[edit]"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Quantum_computing",
            "raw_content": "Contents\nQuantum computing\nA quantum computer is a computer that takes advantage of quantum mechanical phenomena.\nAt small scales, physical matter exhibits properties of both particles and waves, and quantum computing leverages this behavior, specifically quantum superposition and entanglement, using specialized hardware that supports the preparation and manipulation of quantum states.\nClassical physics cannot explain the operation of these quantum devices, and a scalable quantum computer could perform some calculations exponentially faster (with respect to input size scaling)[2] than any modern \"classical\" computer. In particular, a large-scale quantum computer could break widely used encryption schemes and aid physicists in performing physical simulations; however, the current state of the art is largely experimental and impractical, with several obstacles to useful applications. Moreover, scalable quantum computers do not hold promise for many practical tasks, and for many important tasks quantum speedups are proven impossible.\nThe basic unit of information in quantum computing is the qubit, similar to the bit in traditional digital electronics. Unlike a classical bit, a qubit can exist in a superposition of its two \"basis\" states, which loosely means that it is in both states simultaneously. When measuring a qubit, the result is a probabilistic output of a classical bit, therefore making quantum computers nondeterministic in general. If a quantum computer manipulates the qubit in a particular way, wave interference effects can amplify the desired measurement results. The design of quantum algorithms involves creating procedures that allow a quantum computer to perform calculations efficiently and quickly.\nPhysically engineering high-quality qubits has proven challenging. If a physical qubit is not sufficiently isolated from its environment, it suffers from quantum decoherence, introducing noise into calculations. Paradoxically, perfectly isolating qubits is also undesirable because quantum computations typically need to initialize qubits, perform controlled qubit interactions, and measure the resulting quantum states. Each of those operations introduces errors and suffers from noise, and such inaccuracies accumulate.\nNational governments have invested heavily in experimental research that aims to develop scalable qubits with longer coherence times and lower error rates. Two of the most promising technologies are superconductors (which isolate an electrical current by eliminating electrical resistance) and ion traps (which confine a single ion using electromagnetic fields).\nIn principle, a non-quantum (classical) computer can solve the same computational problems as a quantum computer, given enough time. Quantum advantage comes in the form of time complexity rather than computability, and quantum complexity theory shows that some quantum algorithms for carefully selected tasks require exponentially fewer computational steps than the best known non-quantum algorithms. Such tasks can in theory be solved on a large-scale quantum computer whereas classical computers would not finish computations in any reasonable amount of time. However, quantum speedup is not universal or even typical across computational tasks, since basic tasks such as sorting are proven to not allow any asymptotic quantum speedup. Claims of quantum supremacy have drawn significant attention to the discipline, but are demonstrated on contrived tasks, while near-term practical use cases remain limited.\nOptimism about quantum computing is fueled by a broad range of new theoretical hardware possibilities facilitated by quantum physics, but the improving understanding of quantum computing limitations counterbalances this optimism. In particular, quantum speedups have been traditionally estimated for noiseless quantum computers, whereas the impact of noise and the use of quantum error-correction can undermine low-polynomial speedups.\nHistory[edit]\nFor many years, the fields of quantum mechanics and computer science formed distinct academic communities.[3] Modern quantum theory developed in the 1920s to explain the wave–particle duality observed at atomic scales,[4] and digital computers emerged in the following decades to replace human computers for tedious calculations.[5] Both disciplines had practical applications during World War II; computers played a major role in wartime cryptography,[6] and quantum physics was essential for the nuclear physics used in the Manhattan Project.[7]\nAs physicists applied quantum mechanical models to computational problems and swapped digital bits for qubits, the fields of quantum mechanics and computer science began to converge.\nIn 1980, Paul Benioff introduced the quantum Turing machine, which uses quantum theory to describe a simplified computer.[8]\nWhen digital computers became faster, physicists faced an exponential increase in overhead when simulating quantum dynamics,[9] prompting Yuri Manin and Richard Feynman to independently suggest that hardware based on quantum phenomena might be more efficient for computer simulation.[10][11][12]\nIn a 1984 paper, Charles Bennett and Gilles Brassard applied quantum theory to cryptography protocols and demonstrated that quantum key distribution could enhance information security.[13][14]\nQuantum algorithms then emerged for solving oracle problems, such as Deutsch's algorithm in 1985,[15] the Bernstein–Vazirani algorithm in 1993,[16] and Simon's algorithm in 1994.[17]\nThese algorithms did not solve practical problems, but demonstrated mathematically that one could gain more information by querying a black box with a quantum state in superposition, sometimes referred to as quantum parallelism.[18]\nPeter Shor built on these results with his 1994 algorithms for breaking the widely used RSA and Diffie–Hellman encryption protocols,[19] which drew significant attention to the field of quantum computing.[20]\nIn 1996, Grover's algorithm established a quantum speedup for the widely applicable unstructured search problem.[21][22] The same year, Seth Lloyd proved that quantum computers could simulate quantum systems without the exponential overhead present in classical simulations,[23] validating Feynman's 1982 conjecture.[24]\nOver the years, experimentalists have constructed small-scale quantum computers using trapped ions and superconductors.[25]\nIn 1998, a two-qubit quantum computer demonstrated the feasibility of the technology,[26][27] and subsequent experiments have increased the number of qubits and reduced error rates.[25]\nIn 2019, Google AI and NASA announced that they had achieved quantum supremacy with a 54-qubit machine, performing a computation that is impossible for any classical computer.[28][29][30] However, the validity of this claim is still being actively researched.[31][32]\nThe threshold theorem shows how increasing the number of qubits can mitigate errors,[33] yet fully fault-tolerant quantum computing remains \"a rather distant dream\".[34] According to some researchers, noisy intermediate-scale quantum (NISQ) machines may have specialized uses in the near future, but noise in quantum gates limits their reliability.[34]\nInvestment in quantum computing research has increased in the public and private sectors.[35][36]\nAs one consulting firm summarized,[37]\n... investment dollars are pouring in, and quantum-computing start-ups are proliferating. ... While quantum computing promises to help businesses solve problems that are beyond the reach and speed of conventional high-performance computers, use cases are largely experimental and hypothetical at this early stage.\nWith focus on business management’s point of view, the potential applications of quantum computing into four major categories are cybersecurity, data analytics and artificial intelligence, optimization and simulation, and data management and searching.[38]\nQuantum information processing[edit]\nComputer engineers typically describe a modern computer's operation in terms of classical electrodynamics.\nWithin these \"classical\" computers, some components (such as semiconductors and random number generators) may rely on quantum behavior, but these components are not isolated from their environment, so any quantum information quickly decoheres.\nWhile programmers may depend on probability theory when designing a randomized algorithm, quantum mechanical notions like superposition and interference are largely irrelevant for program analysis.\nQuantum programs, in contrast, rely on precise control of coherent quantum systems. Physicists describe these systems mathematically using linear algebra. Complex numbers model probability amplitudes, vectors model quantum states, and matrices model the operations that can be performed on these states. Programming a quantum computer is then a matter of composing operations in such a way that the resulting program computes a useful result in theory and is implementable in practice.\nAs physicist Charlie Bennett describes the relationship between quantum and classical computers,[39]\nA classical computer is a quantum computer ... so we shouldn't be asking about \"where do quantum speedups come from?\" We should say, \"well, all computers are quantum. ... Where do classical slowdowns come from?\"\nQuantum information[edit]\nThe qubit serves as the basic unit of quantum information.\nIt represents a two-state system, just like a classical bit, except that it can exist in a superposition of its two states.[40]\nIn one sense, a superposition is like a probability distribution over the two values.[41]\nHowever, a quantum computation can be influenced by both values at once, inexplicable by either state individually.\nIn this sense, a \"superposed\" qubit stores both values simultaneously.[18]\nA two-dimensional vector mathematically represents a qubit state. Physicists typically use Dirac notation for quantum mechanical linear algebra, writing |ψ⟩ 'ket psi' for a vector labeled ψ. Because a qubit is a two-state system, any qubit state takes the form α|0⟩ + β|1⟩, where |0⟩ and |1⟩ are the standard basis states,[a] and α and β are the probability amplitudes. If either α or β is zero, the qubit is effectively a classical bit; when both are nonzero, the qubit is in superposition. Such a quantum state vector acts similarly to a (classical) probability vector, with one key difference: unlike probabilities, probability amplitudes are not necessarily positive numbers.[41] Negative amplitudes allow for destructive wave interference.[b]\nWhen a qubit is measured in the standard basis, the result is a classical bit.\nThe Born rule describes the norm-squared correspondence between amplitudes and probabilities—when measuring a qubit α|0⟩ + β|1⟩, the state collapses to |0⟩ with probability |α|2, or to |1⟩ with probability |β|2.\nAny valid qubit state has coefficients α and β such that |α|2 + |β|2 = 1.\nAs an example, measuring the qubit 1/√2|0⟩ + 1/√2|1⟩ would produce either |0⟩ or |1⟩ with equal probability.\nEach additional qubit doubles the dimension of the state space.\nAs an example, the vector 1/√2|00⟩ + 1/√2|01⟩ represents a two-qubit state, a tensor product of the qubit |0⟩ with the qubit 1/√2|0⟩ + 1/√2|1⟩.\nThis vector inhabits a four-dimensional vector space spanned by the basis vectors |00⟩, |01⟩, |10⟩, and |11⟩.\nThe Bell state 1/√2|00⟩ + 1/√2|11⟩ is impossible to decompose into the tensor product of two individual qubits—the two qubits are entangled because their probability amplitudes are correlated.\nIn general, the vector space for an n-qubit system is 2n-dimensional, and this makes it challenging for a classical computer to simulate a quantum one: representing a 100-qubit system requires storing 2100 classical values.\nUnitary operators[edit]\nThe state of this one-qubit quantum memory can be manipulated by applying quantum logic gates, analogous to how classical memory can be manipulated with classical logic gates. One important gate for both classical and quantum computation is the NOT gate, which can be represented by a matrix\nThe mathematics of single qubit gates can be extended to operate on multi-qubit quantum memories in two important ways. One way is simply to select a qubit and apply that gate to the target qubit while leaving the remainder of the memory unaffected. Another way is to apply the gate to its target only if another part of the memory is in a desired state. These two choices can be illustrated using another example. The possible states of a two-qubit quantum memory are\nIn summary, quantum computation can be described as a network of quantum logic gates and measurements. However, any measurement can be deferred to the end of quantum computation, though this deferment may come at a computational cost, so most quantum circuits depict a network consisting only of quantum logic gates and no measurements.\nQuantum parallelism[edit]\nQuantum parallelism refers to the ability of quantum computers to evaluate a function for multiple input values simultaneously. This can be achieved by preparing a quantum system in a superposition of input states, and applying a unitary transformation that encodes the function to be evaluated. The resulting state encodes the function's output values for all input values in the superposition, allowing for the computation of multiple outputs simultaneously. This property is key to the speedup of many quantum algorithms.[18]\nQuantum programming[edit]\nThere are a number of models of computation for quantum computing, distinguished by the basic elements in which the computation is decomposed.\nA quantum gate array decomposes computation into a sequence of few-qubit quantum gates. A quantum computation can be described as a network of quantum logic gates and measurements. However, any measurement can be deferred to the end of quantum computation, though this deferment may come at a computational cost, so most quantum circuits depict a network consisting only of quantum logic gates and no measurements.\nAny quantum computation (which is, in the above formalism, any unitary matrix of size\n2\nn\n×\n2\nn\n{\\displaystyle 2^{n}\\times 2^{n}}\nover\nn\n{\\displaystyle n}\nqubits) can be represented as a network of quantum logic gates from a fairly small family of gates. A choice of gate family that enables this construction is known as a universal gate set, since a computer that can run such circuits is a universal quantum computer. One common such set includes all single-qubit gates as well as the CNOT gate from above. This means any quantum computation can be performed by executing a sequence of single-qubit gates together with CNOT gates. Though this gate set is infinite, it can be replaced with a finite gate set by appealing to the Solovay-Kitaev theorem.\nA measurement-based quantum computer decomposes computation into a sequence of Bell state measurements and single-qubit quantum gates applied to a highly entangled initial state (a cluster state), using a technique called quantum gate teleportation.\nAn adiabatic quantum computer, based on quantum annealing, decomposes computation into a slow continuous transformation of an initial Hamiltonian into a final Hamiltonian, whose ground states contain the solution.[43]\nA topological quantum computer decomposes computation into the braiding of anyons in a 2D lattice.[44]\nA quantum Turing machine is the quantum analog of a Turing machine.[8] All of these models of computation—quantum circuits,[45] one-way quantum computation,[46] adiabatic quantum computation,[47] and topological quantum computation[48]—have been shown to be equivalent to the quantum Turing machine; given a perfect implementation of one such quantum computer, it can simulate all the others with no more than polynomial overhead. This equivalence need not hold for practical quantum computers, since the overhead of simulation may be too large to be practical.\nCommunication[edit]\nQuantum cryptography enables new ways to transmit data securely; for example, quantum key distribution uses entangled quantum states to establish secure cryptographic keys.[49] When a sender and receiver exchange quantum states, they can guarantee that an adversary does not intercept the message, as any unauthorized eavesdropper would disturb the delicate quantum system and introduce a detectable change.[50] With appropriate cryptographic protocols, the sender and receiver can thus establish shared private information resistant to eavesdropping.[13][51]\nModern fiber-optic cables can transmit quantum information over relatively short distances. Ongoing experimental research aims to develop more reliable hardware (such as quantum repeaters), hoping to scale this technology to long-distance quantum networks with end-to-end entanglement. Theoretically, this could enable novel technological applications, such as distributed quantum computing and enhanced quantum sensing.[52][53]\nAlgorithms[edit]\nProgress in finding quantum algorithms typically focuses on this quantum circuit model, though exceptions like the quantum adiabatic algorithm exist. Quantum algorithms can be roughly categorized by the type of speedup achieved over corresponding classical algorithms.[54]\nQuantum algorithms that offer more than a polynomial speedup over the best-known classical algorithm include Shor's algorithm for factoring and the related quantum algorithms for computing discrete logarithms, solving Pell's equation, and more generally solving the hidden subgroup problem for abelian finite groups.[54] These algorithms depend on the primitive of the quantum Fourier transform. No mathematical proof has been found that shows that an equally fast classical algorithm cannot be discovered, but evidence suggests that this is unlikely.[55] Certain oracle problems like Simon's problem and the Bernstein–Vazirani problem do give provable speedups, though this is in the quantum query model, which is a restricted model where lower bounds are much easier to prove and doesn't necessarily translate to speedups for practical problems.\nOther problems, including the simulation of quantum physical processes from chemistry and solid-state physics, the approximation of certain Jones polynomials, and the quantum algorithm for linear systems of equations have quantum algorithms appearing to give super-polynomial speedups and are BQP-complete. Because these problems are BQP-complete, an equally fast classical algorithm for them would imply that no quantum algorithm gives a super-polynomial speedup, which is believed to be unlikely.[56]\nSome quantum algorithms, like Grover's algorithm and amplitude amplification, give polynomial speedups over corresponding classical algorithms.[54] Though these algorithms give comparably modest quadratic speedup, they are widely applicable and thus give speedups for a wide range of problems.[22]\nSimulation of quantum systems[edit]\nSince chemistry and nanotechnology rely on understanding quantum systems, and such systems are impossible to simulate in an efficient manner classically, quantum simulation may be an important application of quantum computing.[57] Quantum simulation could also be used to simulate the behavior of atoms and particles at unusual conditions such as the reactions inside a collider.[58] In June 2023, IBM computer scientists reported that a quantum computer produced better results for a physics problem than a conventional supercomputer.[59][60]\nAbout 2% of the annual global energy output is used for nitrogen fixation to produce ammonia for the Haber process in the agricultural fertilizer industry (even though naturally occurring organisms also produce ammonia). Quantum simulations might be used to understand this process and increase the energy efficiency of production.[61] It is expected that an early use of quantum computing will be modeling that improves the efficiency of the Haber–Bosch process[62] by the mid 2020s[63] although some have predicted it will take longer.[64]\nPost-quantum cryptography[edit]\nA notable application of quantum computation is for attacks on cryptographic systems that are currently in use. Integer factorization, which underpins the security of public key cryptographic systems, is believed to be computationally infeasible with an ordinary computer for large integers if they are the product of few prime numbers (e.g., products of two 300-digit primes).[65] By comparison, a quantum computer could solve this problem exponentially faster using Shor's algorithm to find its factors.[66] This ability would allow a quantum computer to break many of the cryptographic systems in use today, in the sense that there would be a polynomial time (in the number of digits of the integer) algorithm for solving the problem. In particular, most of the popular public key ciphers are based on the difficulty of factoring integers or the discrete logarithm problem, both of which can be solved by Shor's algorithm. In particular, the RSA, Diffie–Hellman, and elliptic curve Diffie–Hellman algorithms could be broken. These are used to protect secure Web pages, encrypted email, and many other types of data. Breaking these would have significant ramifications for electronic privacy and security.\nIdentifying cryptographic systems that may be secure against quantum algorithms is an actively researched topic under the field of post-quantum cryptography.[67][68] Some public-key algorithms are based on problems other than the integer factorization and discrete logarithm problems to which Shor's algorithm applies, like the McEliece cryptosystem based on a problem in coding theory.[67][69] Lattice-based cryptosystems are also not known to be broken by quantum computers, and finding a polynomial time algorithm for solving the dihedral hidden subgroup problem, which would break many lattice based cryptosystems, is a well-studied open problem.[70] It has been proven that applying Grover's algorithm to break a symmetric (secret key) algorithm by brute force requires time equal to roughly 2n/2 invocations of the underlying cryptographic algorithm, compared with roughly 2n in the classical case,[71] meaning that symmetric key lengths are effectively halved: AES-256 would have the same security against an attack using Grover's algorithm that AES-128 has against classical brute-force search (see Key size).\nSearch problems[edit]\nThe most well-known example of a problem that allows for a polynomial quantum speedup is unstructured search, which involves finding a marked item out of a list of\nn\n{\\displaystyle n}\nitems in a database. This can be solved by Grover's algorithm using\nO\n(\nn\n)\n{\\displaystyle O({\\sqrt {n}})}\nqueries to the database, quadratically fewer than the\nΩ\n(\nn\n)\n{\\displaystyle \\Omega (n)}\nqueries required for classical algorithms. In this case, the advantage is not only provable but also optimal: it has been shown that Grover's algorithm gives the maximal possible probability of finding the desired element for any number of oracle lookups. Many examples of provable quantum speedups for query problems are based on Grover's algorithm, including Brassard, Høyer, and Tapp's algorithm for finding collisions in two-to-one functions,[72] and Farhi, Goldstone, and Gutmann's algorithm for evaluating NAND trees.[73]\nProblems that can be efficiently addressed with Grover's algorithm have the following properties:[74][75]\nFor problems with all these properties, the running time of Grover's algorithm on a quantum computer scales as the square root of the number of inputs (or elements in the database), as opposed to the linear scaling of classical algorithms. A general class of problems to which Grover's algorithm can be applied[76] is a Boolean satisfiability problem, where the database through which the algorithm iterates is that of all possible answers. An example and possible application of this is a password cracker that attempts to guess a password. Breaking symmetric ciphers with this algorithm is of interest to government agencies.[77]\nQuantum annealing[edit]\nQuantum annealing relies on the adiabatic theorem to undertake calculations. A system is placed in the ground state for a simple Hamiltonian, which slowly evolves to a more complicated Hamiltonian whose ground state represents the solution to the problem in question. The adiabatic theorem states that if the evolution is slow enough the system will stay in its ground state at all times through the process. Adiabatic optimization may be helpful for solving computational biology problems.[78]\nMachine learning[edit]\nSince quantum computers can produce outputs that classical computers cannot produce efficiently, and since quantum computation is fundamentally linear algebraic, some express hope in developing quantum algorithms that can speed up machine learning tasks.[79][34]\nFor example, the quantum algorithm for linear systems of equations, or \"HHL Algorithm\", named after its discoverers Harrow, Hassidim, and Lloyd, is believed to provide speedup over classical counterparts.[80][34] Some research groups have recently explored the use of quantum annealing hardware for training Boltzmann machines and deep neural networks.[81][82][83]\nDeep generative chemistry models emerge as powerful tools to expedite drug discovery. However, the immense size and complexity of the structural space of all possible drug-like molecules pose significant obstacles, which could be overcome in the future by quantum computers. Quantum computers are naturally good for solving complex quantum many-body problems[23] and thus may be instrumental in applications involving quantum chemistry. Therefore, one can expect that quantum-enhanced generative models[84] including quantum GANs[85] may eventually be developed into ultimate generative chemistry algorithms.\nEngineering[edit]\nAs of 2023,[update] classical computers outperform quantum computers for all real-world applications. While current quantum computers may speed up solutions to particular mathematical problems, they give no computational advantage for practical tasks. For many tasks there is no promise of useful quantum speedup, and some tasks provably prohibit any quantum speedup in the sense that any speedup is ruled out by proved theorems. Scientists and engineers are exploring multiple technologies for quantum computing hardware and hope to develop scalable quantum architectures, but serious obstacles remain.[86][87]\nChallenges[edit]\nThere are a number of technical challenges in building a large-scale quantum computer.[88] Physicist David DiVincenzo has listed these requirements for a practical quantum computer:[89]\nSourcing parts for quantum computers is also very difficult. Superconducting quantum computers, like those constructed by Google and IBM, need helium-3, a nuclear research byproduct, and special superconducting cables made only by the Japanese company Coax Co.[90]\nThe control of multi-qubit systems requires the generation and coordination of a large number of electrical signals with tight and deterministic timing resolution. This has led to the development of quantum controllers that enable interfacing with the qubits. Scaling these systems to support a growing number of qubits is an additional challenge.[91]\nOne of the greatest challenges involved with constructing quantum computers is controlling or removing quantum decoherence. This usually means isolating the system from its environment as interactions with the external world cause the system to decohere. However, other sources of decoherence also exist. Examples include the quantum gates, and the lattice vibrations and background thermonuclear spin of the physical system used to implement the qubits. Decoherence is irreversible, as it is effectively non-unitary, and is usually something that should be highly controlled, if not avoided. Decoherence times for candidate systems in particular, the transverse relaxation time T2 (for NMR and MRI technology, also called the dephasing time), typically range between nanoseconds and seconds at low temperature.[92] Currently, some quantum computers require their qubits to be cooled to 20 millikelvin (usually using a dilution refrigerator[93]) in order to prevent significant decoherence.[94] A 2020 study argues that ionizing radiation such as cosmic rays can nevertheless cause certain systems to decohere within milliseconds.[95]\nAs a result, time-consuming tasks may render some quantum algorithms inoperable, as attempting to maintain the state of qubits for a long enough duration will eventually corrupt the superpositions.[96]\nThese issues are more difficult for optical approaches as the timescales are orders of magnitude shorter and an often-cited approach to overcoming them is optical pulse shaping. Error rates are typically proportional to the ratio of operating time to decoherence time, hence any operation must be completed much more quickly than the decoherence time.\nAs described by the threshold theorem, if the error rate is small enough, it is thought to be possible to use quantum error correction to suppress errors and decoherence. This allows the total calculation time to be longer than the decoherence time if the error correction scheme can correct errors faster than decoherence introduces them. An often-cited figure for the required error rate in each gate for fault-tolerant computation is 10−3, assuming the noise is depolarizing.\nMeeting this scalability condition is possible for a wide range of systems. However, the use of error correction brings with it the cost of a greatly increased number of required qubits. The number required to factor integers using Shor's algorithm is still polynomial, and thought to be between L and L2, where L is the number of digits in the number to be factored; error correction algorithms would inflate this figure by an additional factor of L. For a 1000-bit number, this implies a need for about 104 bits without error correction.[97] With error correction, the figure would rise to about 107 bits. Computation time is about L2 or about 107 steps and at 1 MHz, about 10 seconds. However, the encoding and error-correction overheads increase the size of a real fault-tolerant quantum computer by several orders of magnitude. Careful estimates[98][99] show that at least 3 million physical qubits would factor 2,048-bit integer in 5 months on a fully error-corrected trapped-ion quantum computer. In terms of the number of physical qubits, to date, this remains the lowest estimate[100] for practically useful integer factorization problem sizing 1,024-bit or larger.\nAnother approach to the stability-decoherence problem is to create a topological quantum computer with anyons, quasi-particles used as threads, and relying on braid theory to form stable logic gates.[101][102]\nQuantum supremacy[edit]\nPhysicist John Preskill coined the term quantum supremacy to describe the engineering feat of demonstrating that a programmable quantum device can solve a problem beyond the capabilities of state-of-the-art classical computers.[103][104][105] The problem need not be useful, so some view the quantum supremacy test only as a potential future benchmark.[106]\nIn October 2019, Google AI Quantum, with the help of NASA, became the first to claim to have achieved quantum supremacy by performing calculations on the Sycamore quantum computer more than 3,000,000 times faster than they could be done on Summit, generally considered the world's fastest computer.[29][107][108] This claim has been subsequently challenged: IBM has stated that Summit can perform samples much faster than claimed,[109][110] and researchers have since developed better algorithms for the sampling problem used to claim quantum supremacy, giving substantial reductions to the gap between Sycamore and classical supercomputers[111][112][113] and even beating it.[114][115][116]\nIn December 2020, a group at USTC implemented a type of Boson sampling on 76 photons with a photonic quantum computer, Jiuzhang, to demonstrate quantum supremacy.[117][118][119] The authors claim that a classical contemporary supercomputer would require a computational time of 600 million years to generate the number of samples their quantum processor can generate in 20 seconds.[120]\nClaims of quantum supremacy have generated hype around quantum computing,[121] but they are based on contrived benchmark tasks that do not directly imply useful real-world applications.[86][122]\nSkepticism[edit]\nDespite high hopes for quantum computing, significant progress in hardware, and optimism about future applications, a 2023 Nature spotlight article summarised current quantum computers as being \"For now, [good for] absolutely nothing\".[86] The article elaborated that quantum computers are yet to be more useful or efficient than conventional computers in any case, though it also argued that in the long term such computers are likely to be useful. A 2023 Communications of the ACM article[87] found that current quantum computing algorithms are \"insufficient for practical quantum advantage without significant improvements across the software/hardware stack\". It argues that the most promising candidates for achieving speedup with quantum computers are \"small-data problems\", for example in chemistry and materials science. However, the article also concludes that a large range of the potential applications it considered, such as machine learning, \"will not achieve quantum advantage with current quantum algorithms in the foreseeable future\", and it identified I/O constraints that make speedup unlikely for \"big data problems, unstructured linear systems, and database search based on Grover's algorithm\".\nThis state of affairs can be traced to several current and long-term considerations.\nIn particular, building computers with large numbers of qubits may be futile if those qubits are not connected well enough and cannot maintain sufficiently high degree of entanglement for long time. When trying to outperform conventional computers, quantum computing researchers often look for new tasks that can be solved on quantum computers, but this leaves the possibility that efficient non-quantum techniques will be developed in response, as seen for Quantum supremacy demonstrations. Therefore, it is desirable to prove lower bounds on the complexity of best possible non-quantum algorithms (which may be unknown) and show that some quantum algorithms asymptomatically improve upon those bounds.\nSome researchers have expressed skepticism that scalable quantum computers could ever be built, typically because of the issue of maintaining coherence at large scales, but also for other reasons.\nBill Unruh doubted the practicality of quantum computers in a paper published in 1994.[125] Paul Davies argued that a 400-qubit computer would even come into conflict with the cosmological information bound implied by the holographic principle.[126] Skeptics like Gil Kalai doubt that quantum supremacy will ever be achieved.[127][128][129] Physicist Mikhail Dyakonov has expressed skepticism of quantum computing as follows:\nCandidates for physical realizations[edit]\nA practical quantum computer must use a physical system as a programmable quantum register.[132] Researchers are exploring several technologies as candidates for reliable qubit implementations.[133] Superconductors and trapped ions are some of the most developed proposals, but experimentalists are considering other hardware possibilities as well.[134]\nTheory[edit]\nComputability[edit]\nAny computational problem solvable by a classical computer is also solvable by a quantum computer.[135] Intuitively, this is because it is believed that all physical phenomena, including the operation of classical computers, can be described using quantum mechanics, which underlies the operation of quantum computers.\nConversely, any problem solvable by a quantum computer is also solvable by a classical computer. It is possible to simulate both quantum and classical computers manually with just some paper and a pen, if given enough time. More formally, any quantum computer can be simulated by a Turing machine. In other words, quantum computers provide no additional power over classical computers in terms of computability. This means that quantum computers cannot solve undecidable problems like the halting problem, and the existence of quantum computers does not disprove the Church–Turing thesis.[136]\nComplexity[edit]\nWhile quantum computers cannot solve any problems that classical computers cannot already solve, it is suspected that they can solve certain problems faster than classical computers. For instance, it is known that quantum computers can efficiently factor integers, while this is not believed to be the case for classical computers.\nThe class of problems that can be efficiently solved by a quantum computer with bounded error is called BQP, for \"bounded error, quantum, polynomial time\". More formally, BQP is the class of problems that can be solved by a polynomial-time quantum Turing machine with an error probability of at most 1/3. As a class of probabilistic problems, BQP is the quantum counterpart to BPP (\"bounded error, probabilistic, polynomial time\"), the class of problems that can be solved by polynomial-time probabilistic Turing machines with bounded error.[137] It is known that\nB\nP\nP\n⊆\nB\nQ\nP\n{\\displaystyle {\\mathsf {BPP\\subseteq BQP}}}\nand is widely suspected that\nB\nQ\nP\n⊊\nB\nP\nP\n{\\displaystyle {\\mathsf {BQP\\subsetneq BPP}}}\n, which intuitively would mean that quantum computers are more powerful than classical computers in terms of time complexity.[138]\nThe exact relationship of BQP to P, NP, and PSPACE is not known. However, it is known that\nP\n⊆\nB\nQ\nP\n⊆\nP\nS\nP\nA\nC\nE\n{\\displaystyle {\\mathsf {P\\subseteq BQP\\subseteq PSPACE}}}\n; that is, all problems that can be efficiently solved by a deterministic classical computer can also be efficiently solved by a quantum computer, and all problems that can be efficiently solved by a quantum computer can also be solved by a deterministic classical computer with polynomial space resources. It is further suspected that BQP is a strict superset of P, meaning there are problems that are efficiently solvable by quantum computers that are not efficiently solvable by deterministic classical computers. For instance, integer factorization and the discrete logarithm problem are known to be in BQP and are suspected to be outside of P. On the relationship of BQP to NP, little is known beyond the fact that some NP problems that are believed not to be in P are also in BQP (integer factorization and the discrete logarithm problem are both in NP, for example). It is suspected that\nN\nP\n⊈\nB\nQ\nP\n{\\displaystyle {\\mathsf {NP\\nsubseteq BQP}}}\n; that is, it is believed that there are efficiently checkable problems that are not efficiently solvable by a quantum computer. As a direct consequence of this belief, it is also suspected that BQP is disjoint from the class of NP-complete problems (if an NP-complete problem were in BQP, then it would follow from NP-hardness that all problems in NP are in BQP).[139]\nSee also[edit]\nNotes[edit]\nReferences[edit]\nFurther reading[edit]\nTextbooks[edit]\nAcademic papers[edit]\nExternal links[edit]"
        }
    ],
    "failed_results": [
        {
            "url": "https://com",
            "error": "Validation Error: Invalid URL format"
        }
    ],
    "response_time": 0.02
}

Error Codes

• 400: Bad Request — Your request is invalid.
• 401: Unauthorized — Your API key is wrong.
• 403: Forbidden — The endpoint requested is hidden for administrators only.
• 404: Not Found — The specified endpoint could not be found.
• 405: Method Not Allowed — You tried to access an endpoint with an invalid method.
• 422: Unprocessable Content - You submitted a request with invalid data that the server was unable to process.
• 429: Too Many Requests — You're sending too many requests! Slow down!
• 500: Internal Server Error — We had a problem with our server. Try again later.
• 503: Service Unavailable — We're temporarily offline for maintenance. Please try again later.
• 504: Gateway Timeout — We're temporarily offline for maintenance. Please try again later.

Authentication

Tavily Search and Extract use API keys to allow access to the API. You can register a new API key on the Tavily Platform.

Tavily API Credit Deduction Overview

• Tavily Search
  • Each basic search (search_depth:"basic") will deduct 1 API credit from your account.
  • Each advanced search (search_depth:"advanced") will deduct 2 API credits from your account.
• Tavily Extract
  • For every 5 successful URL extractions, 1 API credit will be deducted from your account.

Rate Limiting

The Tavily Search and Extract APIs have a rate limit of 100 requests per minute.

Support

For questions, support, or to learn more, please visit the Tavily Developer Community.