Using curvature information for fast stochastic search

Genevieve B. Orr; Todd K. Leen

Using curvature information for fast stochastic search

Genevieve B. Orr, Todd K. Leen

Biomedical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

24 Scopus citations

Abstract

We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes effective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks.

Original language	English (US)
Title of host publication	Advances in Neural Information Processing Systems 9 - Proceedings of the 1996 Conference, NIPS 1996
Publisher	Neural information processing systems foundation
Pages	606-612
Number of pages	7
ISBN (Print)	0262100657, 9780262100656
State	Published - 1997
Event	10th Annual Conference on Neural Information Processing Systems, NIPS 1996 - Denver, CO, United States Duration: Dec 2 1996 → Dec 5 1996

Publication series

Name	Advances in Neural Information Processing Systems
ISSN (Print)	1049-5258

Other

Other	10th Annual Conference on Neural Information Processing Systems, NIPS 1996
Country/Territory	United States
City	Denver, CO
Period	12/2/96 → 12/5/96

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

Using curvature information for fast stochastic search. / Orr, Genevieve B.; Leen, Todd K.
Advances in Neural Information Processing Systems 9 - Proceedings of the 1996 Conference, NIPS 1996. Neural information processing systems foundation, 1997. p. 606-612 (Advances in Neural Information Processing Systems).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Orr, GB & Leen, TK 1997, Using curvature information for fast stochastic search. in Advances in Neural Information Processing Systems 9 - Proceedings of the 1996 Conference, NIPS 1996. Advances in Neural Information Processing Systems, Neural information processing systems foundation, pp. 606-612, 10th Annual Conference on Neural Information Processing Systems, NIPS 1996, Denver, CO, United States, 12/2/96.

@inproceedings{66f19f9306554eacad7b204a5e8eaeed,

title = "Using curvature information for fast stochastic search",

abstract = "We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes effective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks.",

author = "Orr, {Genevieve B.} and Leen, {Todd K.}",

year = "1997",

language = "English (US)",

isbn = "0262100657",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "606--612",

booktitle = "Advances in Neural Information Processing Systems 9 - Proceedings of the 1996 Conference, NIPS 1996",

}

TY - GEN

T1 - Using curvature information for fast stochastic search

AU - Orr, Genevieve B.

AU - Leen, Todd K.

PY - 1997

Y1 - 1997

N2 - We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes effective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks.

AB - We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes effective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks.

UR - http://www.scopus.com/inward/record.url?scp=84898987060&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84898987060&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84898987060

SN - 0262100657

SN - 9780262100656

T3 - Advances in Neural Information Processing Systems

SP - 606

EP - 612

BT - Advances in Neural Information Processing Systems 9 - Proceedings of the 1996 Conference, NIPS 1996

PB - Neural information processing systems foundation

T2 - 10th Annual Conference on Neural Information Processing Systems, NIPS 1996

Y2 - 2 December 1996 through 5 December 1996

ER -

Using curvature information for fast stochastic search

Abstract

Publication series

Other

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this