User contributions for Speedboys

20 September 2024

• 10:2910:29, 20 September 2024 diff hist +1,262‎ Talk:Neural network (machine learning) ‎ Answers to User:Cosmia Nebula, Michaelmalak, and User:North8000 current

19 September 2024

• 12:1712:17, 19 September 2024 diff hist +6,486‎ Talk:Neural network (machine learning) ‎ first proposed edit: deep learning breakthroughs in the 1960s and 1970s
• 11:5811:58, 19 September 2024 diff hist +2,284‎ Talk:Neural network (machine learning) ‎ Answers to User:Cosmia Nebula, SamuelRiv, and Tryptofish

18 September 2024

• 14:2114:21, 18 September 2024 diff hist +4,521‎ Talk:Neural network (machine learning) ‎ Answer to User:Cosmia Nebula with references copied from other articles

17 September 2024

• 21:3621:36, 17 September 2024 diff hist +389‎ User talk:North8000 ‎ →‎History section of Neural network (machine learning): important references, mostly resurrected
• 21:2421:24, 17 September 2024 diff hist +1,321‎ Talk:Neural network (machine learning) ‎ History section: request to approve edits of 15-16 September 2024
• 16:4216:42, 17 September 2024 diff hist +1,254‎ User talk:North8000 ‎ →‎History section of [[Neural network (machine learning)]]

16 September 2024

• 20:0720:07, 16 September 2024 diff hist +1,579‎ Neural network (machine learning) ‎ →‎Recurrent networks: copied origins of recurrent neural networks from deep learning: the Ising model by Wilhelm Lenz and Ernst Ising (1925), its adaptive RNN version by Shun'ichi Amari (1972) and Hopfield (1982), Kaoru Nagano's RNN (1972), Alan Turing's unpublished work (1948) on "Intelligent Machinery" containing "ideas related to artificial evolution and learning RNNs." Tag: Reverted
• 19:5519:55, 16 September 2024 diff hist +3,028‎ Neural network (machine learning) ‎ →‎History: Dear User:Cosmia Nebula, while compressing the text, you deleted the important references to Jürgen Schmidhuber's 1991 work on self-supervised pre-training, neural network distillation, GANs, and unnormalized linear Transformers. I reinserted them. Tag: Reverted
• 17:5517:55, 16 September 2024 diff hist −9‎ Transformer (deep learning architecture) ‎ →‎Predecessors: Dear User:Cosmia Nebula, I appreciate your efforts at compressing the text, but please don't overdo it by inserting errors. You wrote about the fast weight controller: "one neural network outputs the weights of another neural network" although it outputs *weight changes*. That's a big difference: any multiplicative gate outputs a weight. And why delete the important part about linear scaling? Corrected.

15 September 2024

• 19:2419:24, 15 September 2024 diff hist +153‎ Neural network (machine learning) ‎ Early work: the terminology "artificial neural networks" was introduced much later in the context of modelling information processing in biological systems, inspired by Santiago Ramón y Cajal's discoveries in the Cerebellum (around 1900). Tag: Reverted
• 19:0619:06, 15 September 2024 diff hist +841‎ Neural network (machine learning) ‎ →‎History: copying missing references from deep learning Tag: Reverted
• 18:4518:45, 15 September 2024 diff hist −910‎ Neural network (machine learning) ‎ →‎Convolutional neural networks: copying the improved text from deep learning and improving the context. Tag: Reverted
• 18:3418:34, 15 September 2024 diff hist +1,034‎ Neural network (machine learning) ‎ →‎Backpropagation: copying the improved text from deep learning Tag: Reverted
• 18:2918:29, 15 September 2024 diff hist +720‎ Neural network (machine learning) ‎ The "neural network winter" (if any) was limited to the US. Deep learning breakthroughs in the 1960s and 1970s: Ivakhnenko (1965), Amari (1967), Fukushima (1969, 1979). Tag: Reverted
• 17:4317:43, 15 September 2024 diff hist +602‎ Neural network (machine learning) ‎ →‎Early work: The first perceptrons did not have learning hidden units. However, R. D. Joseph (1960) discussed multilayer perceptrons with an adaptive hidden layer. Rosenblatt (1962) cited and adopted these ideas, also crediting work by H. D. Block and B. W. Knight. Unfortunately, these early efforts did not lead to a working learning algorithm for hidden units. Tag: Reverted
• 16:1216:12, 15 September 2024 diff hist +436‎ Neural network (machine learning) ‎ →‎Early work: R. D. Joseph (1960) mentions an even earlier perceptron-like device by Farley and Clark: "Farley and Clark of MIT Lincoln Laboratory actually preceded Rosenblatt in the development of a perceptron-like device." However, "they dropped the subject." Tag: Reverted
• 15:5015:50, 15 September 2024 diff hist +1,189‎ Neural network (machine learning) ‎ →‎Early work: Deleted paragraph: some connectionist models do separate memory and processing. Resurrecting deleted text of 30 July 2024: neural networks with linear activation functions are identical to the method of least squares introduced over 2 centuries ago by Legendre and Gauss. Tag: Reverted
• 15:0315:03, 15 September 2024 diff hist +609‎ Deep learning ‎ →‎Before 1980: References: other early recurrent neural networks were published by Kaoru Nakano in 1971.
• 14:5814:58, 15 September 2024 diff hist +68‎ Deep learning ‎ →‎History: References: recurrent networks appeared before the 1980s
• 14:4914:49, 15 September 2024 diff hist +255‎ Deep learning ‎ →‎Before 1980: G.M. Ostrovski et al. republished backpropagation in 1971.
• 14:3314:33, 15 September 2024 diff hist +369‎ Deep learning ‎ →‎Before 1980: Paul Werbos applied backpropagation to neural networks in 1982. His 1974 PhD thesis, reprinted in a 1994 book, did not yet describe the algorithm. Rumelhart et al. (1986) popularised backpropagation but did not cite the original work.

14 September 2024

• 17:2917:29, 14 September 2024 diff hist +471‎ Deep learning ‎ →‎Before 1980: Already in 1948, Alan Turing produced work "related to artificial evolution and learning RNNs" that was not published in his lifetime.
• 17:0717:07, 14 September 2024 diff hist +1,596‎ Deep learning ‎ →‎Before 1980: Elman networks and Jordan networks are mentioned, but not the origins of RNNs. Resurrecting deleted text of 6 Aug 2024: In the 1920s, Wilhelm Lenz and Ernst Ising created and analyzed the Ising model, the first RNN architecture. It did not learn. In 1972, Shun'ichi Amari made it adaptive. This became the foundation of the Hopfield network and the Boltzmann Machine and many other recurrent models.
• 16:4516:45, 14 September 2024 diff hist −493‎ Deep learning ‎ →‎Deep learning revolution: GANs are based on the 1991 curiosity principle. Highway nets (May 2015) were published 7 months before ResNets (Dec 2015). Deleted "concurrently." ResNet is like an open-gated Highway Net.
• 16:2716:27, 14 September 2024 diff hist +2,728‎ Deep learning ‎ →‎1980s-2000s: Corrections based on deleted text of 6 Aug 2024: in 1991, the pre-trained history compressor with knowledge distillation was proposed not by Sepp Hochreiter but by his advisor Juergen Schmidhuber, who also published the GAN principle in 1991. Fixed URLs in references.

13 September 2024

• 17:1617:16, 13 September 2024 diff hist +343‎ Deep learning ‎ →‎1980s-2000s: Alex Waibel's 1987 reference copied from Convolutional Neural Network. Temporal order of CNN applications corrected: Wei Zhang et al. 1988, Yann LeCun et al. 1989, CNN optical hardware by Zhang in 1990.
• 16:5616:56, 13 September 2024 diff hist +307‎ Deep learning ‎ →‎History: 1979 reference: Fukushima's CNN architecture moved from the 1980s to the 1970s.
• 16:4516:45, 13 September 2024 diff hist +17‎ Deep learning ‎ →‎History: Kunihiko Fukushima's ReLU moved from the 1980s to the 1960s.

10 September 2024

• 17:0317:03, 10 September 2024 diff hist +360‎ Deep learning ‎ →‎Before 1980: Alexey Grigorevich Ivakhnenko and Valentin Lapa in 1965: layer by layer training through regression analysis, superfluous hidden units pruned using a separate validation set, first deep networks with multiplicative units or "gates"
• 16:0316:03, 10 September 2024 diff hist +48‎ Deep learning ‎ →‎Before 1980: Frank Rosenblatt (1962) cites an earlier network by R. D. Joseph (1960) "functionally equivalent to a variation of" his four-layer system "with adaptive preterminal networks." Rosenblatt's book mentions Joseph over 30 times. Should one consider Joseph (1960) as the originator of adaptive multilayer perceptrons? Unfortunately, the learning algorithm was not a functional one, and fell into oblivion.

8 September 2024

• 16:2516:25, 8 September 2024 diff hist +698‎ Transformer (deep learning architecture) ‎ →‎Predecessors: References: the first multiplicative units in deep networks date back to the work of Alexey Grigorevich Ivakhnenko and Valentin Lapa in the 1960s.
• 16:0916:09, 8 September 2024 diff hist +23‎ Transformer (deep learning architecture) ‎ Missing name for the reference to the paper by Krzysztof Choromanski and colleagues (2020)
• 16:0416:04, 8 September 2024 diff hist +880‎ Transformer (deep learning architecture) ‎ →‎Predecessors: Resurrecting additional important references on this. The terminology "linear Transformer" was introduced only much later, in 2020, by Angelos Katharopoulos and colleagues. The mathematical equivalence of the Fast Weight Controller and the unnormalized linear Transformer was pointed out one year later by Imanol Schlag and colleagues. The terminology "learning internal spotlights of attention" was introduced in 1993.
• 15:5015:50, 8 September 2024 diff hist +847‎ Transformer (deep learning architecture) ‎ →‎Predecessors: Juergen Schmidhuber's unnormalised linear Transformer (1992) differs from the previous fast weight approaches since 1981: a "slow" neural network learns by gradient descent to output the weight changes of a separate "fast" neural network, in end-to-end differentiable fashion.
• 15:2515:25, 8 September 2024 diff hist +452‎ Transformer (deep learning architecture) ‎ →‎History: References: the 1987 fast weight paper by Geoffrey Hinton and David Plaut (1987) cites the earlier work on fast weights or dynamic links by Christoph Malsburg (1981) and Jerome Feldman (1982)

7 September 2024

• 15:3915:39, 7 September 2024 diff hist +419‎ Highway network ‎ →‎Related Work: Adding the reference to Frank Rosenblatt's work on skip connections. current
• 15:3015:30, 7 September 2024 diff hist +45‎ Residual neural network ‎ Correcting the Lang & Witbrock reference
• 15:2215:22, 7 September 2024 diff hist +687‎ Highway network ‎ →‎Related Work: Citing the "short-cut connection" or "skip connection" by Kevin Lang & Michael Witbrock (1988). Every residual connection is a skip connection, but almost all skip connections are not residual connections.
• 15:0415:04, 7 September 2024 diff hist +43‎ Highway network ‎ Highway nets (May 2015) were published 7 months before ResNets (Dec 2015). Deleted "concurrently." ResNet is like an open-gated Highway Net.
• 14:4614:46, 7 September 2024 diff hist −6‎ Residual neural network ‎ Every residual connection is a skip connection, but almost all skip connections are not residual connections that map x to x + F(x)
• 14:2614:26, 7 September 2024 diff hist +241‎ Residual neural network ‎ The "residual connection" is a special case of the "skip connection" by Lang & Witbrock (1988) where x maps to f(x) + Ax and the randomly initialized weight matrix A does not have to be the identity mapping.
• 14:0514:05, 7 September 2024 diff hist +11‎ Residual neural network ‎ Better URL for the 1988 paper on skip connections by Kevin Lang and Michael Witbrock

31 December 2023

• 14:1214:12, 31 December 2023 diff hist +398‎ ChatGPT ‎ →‎Existential risk: Artificial Intelligence: Yann LeCun "scoffs at his peers’ dystopian scenarios of supercharged misinformation and even, eventually, human extinction."
• 14:0814:08, 31 December 2023 diff hist +398‎ Artificial intelligence ‎ →‎Existential risk: Yann LeCun "scoffs at his peers’ dystopian scenarios of supercharged misinformation and even, eventually, human extinction."
• 13:5413:54, 31 December 2023 diff hist +1,915‎ Artificial intelligence ‎ →‎Existential risk: Material from ChatGPT: Juergen Schmidhuber and Andrew Ng spoke in favor of a less dystopian view of AI
• 13:2813:28, 31 December 2023 diff hist +471‎ ChatGPT ‎ →‎Existential risk: AI pioneer Andrew Ng also argued that "it’s a mistake to fall for the doomsday hype on AI — and that regulators who do will only benefit vested interests."
• 12:5612:56, 31 December 2023 diff hist +1,482‎ ChatGPT ‎ →‎Existential risk: Copied references from Juergen Schmidhuber who spoke in favor of a less dystopian view of AI: in 95% of all cases, AI research is about making "human lives longer and healthier and easier."
• 12:3012:30, 31 December 2023 diff hist +374‎ Generative pre-trained transformer ‎ →‎Initial developments: Schmidhuber's 1992 reference on pre-training
• 12:2312:23, 31 December 2023 diff hist +1,738‎ Generative pre-trained transformer ‎ →‎Initial developments: References: while the unnormalized linear Transformer dates back to 1992, the modern transformer (machine learning model)|transformer architecture was not available until 2017 when it was published by employees at Google.