NUMDIFFTOOLS(1)

NAME
numdifftools - numdifftools 0.9.13.post0.dev36+g29bd164.dirty Code Health

This is the documentation of Numdifftools 0.9.13.post0.dev36+g29bd164.dirty.

Bleeding edge available at: https://github.com/pbrod/numdifftools.

Official releases are available at: http://pypi.python.org/pypi/Numdifftools.

INTRODUCTION TO NUMDIFFTOOLS

Numdifftools is a suite of tools written in Python to solve automatic numerical differentiation problems in one or more variables. Finite differences are used in an adaptive manner, coupled with a Richardson extrapolation methodology to provide a maximally accurate result. The user can configure many options like; changing the order of the method or the extrapolation, even allowing the user to specify whether complex, multicomplex, central, forward or backward differences are used. The methods provided are:

Derivative:
    Computates the derivative of order 1 through 10 on any scalar function.

Gradient:
    Computes the gradient vector of a scalar function of one or more variables.

Jacobian:
    Computes the Jacobian matrix of a vector valued function of one or more variables.

Hessian:
    Computes the Hessian matrix of all 2nd partial derivatives of a scalar function of one or more variables.

Hessdiag:
    Computes only the diagonal elements of the Hessian matrix

    All of these methods also produce error estimates on the result.

    Numdifftools also provide an easy to use interface to derivatives calculated with AlgoPy. Algopy stands for Algorithmic Differentiation in Python. The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs.

    Documentation is at: http://numdifftools.readthedocs.org/

    Code and issue tracker is at https://github.com/pbrod/numdifftools.

    Latest stable release is at http://pypi.python.org/pypi/Numdifftools.

    To test if the toolbox is working paste the following in an interactive python session:

    import numdifftools as nd
    nd.test(coverage=True, doctests=True)

  Getting Started

Compute 1'st and 2'nd derivative of exp(x), at x == 1:

>>> import numpy as np
>>> import numdifftools as nd
>>> fd = nd.Derivative(np.exp)        # 1'st derivative
>>> fdd = nd.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nd.Jacobian(fun)
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nd.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

Compute the same with the easy to use interface to AlgoPy:

>>> import numdifftools.nd_algopy as nda
>>> import numpy as np
>>> fd = nda.Derivative(np.exp)        # 1'st derivative
>>> fdd = nda.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nda.Jacobian(fun, method='reverse')
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nda.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

  See also

scipy.misc.derivative

NUMERICAL DIFFERENTIATION

  Introduction

The general problem of differentiation of a function typically pops up in three ways in Python.

•   The symbolic derivative of a function.

•   Compute numerical derivatives of a function defined only by a sequence of data points.

•   Compute numerical derivatives of a analytically supplied function.

    Clearly the first member of this list is the domain of the symbolic toolbox SymPy, or some set of symbolic tools. Numerical differentiation of a function defined by data points can be achieved with the function gradient, or perhaps by differentiation of a curve fit to the data, perhaps to an interpolating spline or a least squares spline fit.

    The third class of differentiation problems is where Numdifftools is valuable. This document will describe the methods used in Numdifftools and in particular the Derivative class.

  Numerical differentiation of a general function of one variable

Surely you recall the traditional definition of a derivative, in terms of a limit.

For small elta, the limit approaches f'(x). This is a one-sided approximation for the derivative. For a fixed value of elta, this is also known as a finite difference approximation (a forward difference.) Other approximations for the derivative are also available. We will see the origin of these approximations in the Taylor series expansion of a function f(x) around some point x_0.

Truncate the series in 2 to the first three terms, divide by elta and rearrange yields the forward difference approximation 1:

When elta is small, elta^2 and any higher powers are vanishingly small. So we tend to ignore those higher powers, and describe the approximation in 3 as a first order approximation since the error in this approximation approaches zero at the same rate as the first power of elta. [1] The values of f''(x_0) and f'''(x_0), while unknown to us, are fixed constants as elta varies.

Higher order approximations arise in the same fashion. The central difference 4 is a second order approximation.

  Unequally spaced finite difference rules

While most finite difference rules used to differentiate a function will use equally spaced points, this fails to be appropriate when one does not know the final spacing. Adaptive quadrature rules can succeed by subdividing each sub-interval as necessary. But an adaptive differentiation scheme must work differently, since differentiation is a point estimate. Derivative generates a sequence of sample points that follow a log spacing away from the point in question, then it uses a single rule (generated on the fly) to estimate the desired derivative. Because the points are log spaced, the same rule applies at any scale, with only a scale factor applied.

  Odd and even transformations of a function

Returning to the Taylor series expansion of f(x) around some point x_0, an even function [2] around x_0 must have all the odd order derivatives vanish at x_0. An odd function has all its even derivatives vanish from its expansion. Consider the derived functions f_{odd}(x) and f_{even}(x).

The Taylor series expansion of f_{odd}(x) around zero has the useful property that we have killed off any even order terms, but the odd order terms are identical to f(x), as expanded around x_0.

Likewise, the Taylor series expansion of f_{even}(x) has no odd order terms or a constant term, but other even order terms that are identical to f(x).

The point of these transformations is we can rather simply generate a higher order approximation for any odd order derivatives of f(x) by working with f_{odd}(x). Even order derivatives of f(x) are similarly generated from f_{even}(x). For example, a second order approximation for f'(x_0) is trivially written in 9 as a function of elta.

We can do better rather simply, so why not? 10 shows a fourth order approximation for f'(x_0).

Again, the next non-zero term 11 in that expansion has a higher power of elta on it, so we would normally ignore it since the lowest order neglected term should dominate the behavior for small elta.

Derivative uses similar approximations for all derivatives of f up to any order. Of course, it is not always possible for evaluation of a function on both sides of a point, as central difference rules will require. In these cases, you can specify forward or backward difference rules as appropriate. You can also specify to use the complex step derivative, which we will outline in the next section.

  Complex step derivative

The derivation of the complex-step derivative approximation is accomplished by replacing elta in 2 with a complex step i h:

Taking only the imaginary parts of both sides gives

Dividing with h and rearranging yields:

Terms with order h^2 or higher can safely be ignored since the interval h can be chosen up to machine precision without fear of rounding errors stemming from subtraction (since there are not any). Thus to within second-order the complex-step derivative approximation is given by:

Next, consider replacing the step elta in 8 with the complex step i^ac{1}{2} h:

Similarly dividing with h^2/2 and taking only the imaginary components yields:

This approximation is still subject to difference errors, but the error associated with this approximation is proportional to h^4. Neglecting these higher order terms yields:

See [LaiCrassidisCheng2005] and [Ridout2009] for more details. The complex-step derivative in numdifftools.Derivative has truncation error O(elta^4) for both odd and even order derivatives for n>1. For n=1 the truncation error is on the order of O(elta^2), so truncation error can be eliminated by choosing steps to be very small. The first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, the function to differentiate needs to be analytic. This method does not work if it does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. For this reason the central method is the default method.

  High order derivative

So how do we construct these higher order approximation formulas? Here we will deomonstrate the principle by computing the 6'th order central approximation for the first-order derivative. In order to do so we simply set f_{odd}(elta) equal to its 3-term Taylor expansion:

By inserting three different stepsizes into 12, eg elta, elta/2, elta/4, we get a set of linear equations:

The solution of these equations are simply:

The first row of 14a gives the coefficients for 6'th order approximation. Looking at at row two and three, we see also that this gives the 6'th order approximation for the 3'rd and 5'th order derivatives as bonus. Thus this is also a general method for obtaining high order differentiation rules. As previously noted these formulas have the additional benefit of beeing applicable to any scale, with only a scale factor applied.

  Richardson extrapolation methodology applied to derivative estimation

Some individuals might suggest that the above set of approximations are entirely adequate for any sane person. Can we do better?

Suppose we were to generate several different estimates of the approximation in 3 for different values of elta at a fixed x_0. Thus, choose a single elta, estimate a corresponding resulting approximation to f'(x_0), then do the same for elta/2. If we assume that the error drops off linearly as elta o 0, then it is a simple matter to extrapolate this process to a zero step size. Our lack of knowledge of f''(x_0) is irrelevant. All that matters is elta is small enough that the linear term dominates so we can ignore the quadratic term, therefore the error is purely linear.

The linear extrapolant for this interval halving scheme as elta o 0 is given by:

Since I've always been a big fan of convincing myself that something will work before I proceed too far, lets try this out in Python. Consider the function e^x. Generate a pair of approximations to f'(0), once at elta of 0.1, and the second approximation at 1/2 that value. Recall that ac{d(e^x)}{dx} = e^x, so at x = 0, the derivative should be exactly 1. How well will we do?

>>> from numpy import exp, allclose
>>> f = exp
>>> dx = 0.1
>>> df1 = (f(dx) - f(0))/dx
>>> allclose(df1, 1.05170918075648)
True

>>> df2 = (f(dx/2) - f(0))/(dx/2)
>>> allclose(df2, 1.02542192752048)
True

>>> allclose(2*df2 - df1, 0.999134674284488)
True

In fact, this worked very nicely, reducing the error to roughly 1 percent of our initial estimates. Should we be surprised at this reduction? Not if we recall that last term in 3. We saw there that the next term in the expansion was O(elta^2). Since elta was 0.1 in our experiment, that 1 percent number makes perfect sense.

The Richardson extrapolant in 16 assumed a linear process, with a specific reduction in elta by a factor of 2. Assume the two term (linear + quadratic) residual term in 3, evaluating our approximation there with a third value of elta. Again, assume the step size is cut in half again. The three term Richardson extrapolant is given by:

A quick test in Python yields much better results yet.

>>> from numpy import exp, allclose
>>> f = exp
>>> dx = 0.1

>>> df1 = (f(dx) - f(0))/dx
>>> allclose(df1,  1.05170918075648)
True

>>> df2 = (f(dx/2) - f(0))/(dx/2)
>>> allclose(df2, 1.02542192752048)
True

>>> df3 = (f(dx/4) - f(0))/(dx/4)
>>> allclose(df3, 1.01260482097715)
True

>>> allclose(1./3*df1 - 2*df2 + 8./3*df3, 1.00000539448361)
True

Again, Derivative uses the appropriate multiple term Richardson extrapolants for all derivatives of f up to any order [3]. This, combined with the use of high order approximations for the derivatives, allows the use of quite large step sizes. See [LynessMoler1966] and [LynessMoler1969]. How to compute the multiple term Richardson extrapolants will be elaborated further in the next section.

  Multiple term Richardson extrapolants

We shall now indicate how we can calculate the multiple term Richardson extrapolant for f_{odd}(elta)/elta by rearranging 12:

This equation has the form

where L stands for f'(x_0) and hi(elta) for the numerical differentiation formula f_{odd}(elta)/elta.

By neglecting higher order terms (a_3 elta^8) and inserting three different stepsizes into 18, eg elta, elta/2, elta/4, we get a set of linear equations:

The solution of these equations are simply:

The first row of 20 gives the coefficients for Richardson extrapolation scheme.

  Uncertainty estimates for Derivative

We can view the Richardson extrapolation step as a polynomial curve fit in the step size parameter elta. Our desired extrapolated value is seen as simply the constant term coefficient in that polynomial model. Remember though, this polynomial model (see 10 and 11) has only a few terms in it with known non-zero coefficients. That is, we will expect a constant term a_0, a term of the form a_1 elta^4, and a third term a_2 elta^6.

A neat trick to compute the statistical uncertainty in the estimate of our desired derivative is to use statistical methodology for that error estimate. While I do appreciate that there is nothing truly statistical or stochastic in this estimate, the approach still works nicely, providing a very reasonable estimate in practice. A three term Richardson-like extrapolant, then evaluated at four distinct values for elta, will yield an estimate of the standard error of the constant term, with one spare degree of freedom. The uncertainty is then derived by multiplying that standard error by the appropriate percentile from the Students-t distribution.

>>> import scipy.stats as ss
>>> allclose(ss.t.cdf(12.7062047361747, 1), 0.975)
True

This critical level will yield a two-sided confidence interval of 95 percent.

These error estimates are also of value in a different sense. Since they are efficiently generated at all the different scales, the particular spacing which yields the minimum predicted error is chosen as the best derivative estimate. This has been shown to work consistently well. A spacing too large tends to have large errors of approximation due to the finite difference schemes used. But a too small spacing is bad also, in that we see a significant amplification of least significant fit errors in the approximation. A middle value generally seems to yield quite good results. For example, Derivative will estimate the derivative of e^x automatically. As we see, the final overall spacing used was 0.0078125.

>>> import numdifftools as nd
>>> from numpy import exp, allclose
>>> f = nd.Derivative(exp, full_output=True)
>>> val, info = f(1)
>>> allclose(val, 2.71828183)
True
>>> allclose(info.error_estimate, 6.927791673660977e-14)
True
>>> allclose(info.final_step, 0.0078125)
True

However, if we force the step size to be artificially large, then approximation error takes over.

>>> f = nd.Derivative(exp, step=1, full_output=True)
>>> val, info = f(1)
>>> allclose(val, 3.19452805)
True
>>> allclose(val-exp(1), 0.47624622)
True
>>> allclose(info.final_step, 1)
True

And if the step size is forced to be too small, then we see noise dominate the problem.

>>> f = nd.Derivative(exp, step=1e-10, full_output=True)
>>> val, info = f(1)
>>> allclose(val, 2.71828093)
True
>>> allclose(val - exp(1), -8.97648138e-07)
True
>>> allclose(info.final_step, 1.0000000e-10)
True

Numdifftools, like Goldilocks in the fairy tale bearing her name, stays comfortably in the middle ground.

  Derivative in action

How does numdifftools.Derivative work in action? A simple nonlinear function with a well known derivative is e^x. At x = 0, the derivative should be 1.

>>> f = nd.Derivative(exp, full_output=True)
>>> val, info = f(0)
>>> allclose(val, 1)
True

>>> allclose(info.error_estimate, 5.28466160e-14)
True

A second simple example comes from trig functions. The first four derivatives of the sine function, evaluated at x = 0, should be respectively [cos(0), -sin(0), -cos(0), sin(0)], or [1,0,-1,0].

>>> from numpy import sin, allclose
>>> import numdifftools as nd
>>> df = nd.Derivative(sin, n=1)
>>> allclose(df(0), 1.)
True

>>> ddf = nd.Derivative(sin, n=2)
>>> allclose(ddf(0), 0.)
True

>>> dddf = nd.Derivative(sin, n=3)
>>> allclose(dddf(0), -1.)
True

>>> ddddf = nd.Derivative(sin, n=4)
>>> allclose(ddddf(0), 0.)
True

  Gradient and Hessian estimation

Estimation of the gradient vector (numdifftools.Gradient) of a function of multiple variables is a simple task, requiring merely repeated calls to numdifftools.Derivative. Likewise, the diagonal elements of the hessian matrix are merely pure second partial derivatives of a function. numdifftools.Hessdiag accomplishes this task, again calling numdifftools.Derivative multiple times. Efficient computation of the off-diagonal (mixed partial derivative) elements of the Hessian matrix uses a scheme much like that of numdifftools.Derivative, then Richardson extrapolation is used to improve a set of second order finite difference estimates of those mixed partials.

  Conclusion

numdifftools.Derivative is an a adaptive scheme that can compute the derivative of arbitrary (well behaved) functions. It is reasonably fast as an adaptive method. Many options have been provided for the user who wishes the ultimate amount of control over the estimation.

  Acknowledgments

The numdifftools package was originally a translation of an adaptive numerical differentiation toolbox written in Matlab by John D'Errico [DErrico2006].

Numdifftools has as of version 0.9 been extended with some of the functionality found in the statsmodels.tools.numdiff module written by Josef Perktold [Perktold2014].

  References

[LynessMoler1966]
    Lyness, J. M., Moler, C. B. (1966). Vandermonde Systems and Numerical Differentiation. Numerische Mathematik.

[LynessMoler1969]
    Lyness, J. M., Moler, C. B. (1969). Generalized Romberg Methods for Integrals of Derivatives. Numerische Mathematik.

[DErrico2006]
    D'Errico, J. R. (2006), Adaptive Robust Numerical Differentiation http://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation

[Perktold2014]
    Perktold, J (2014), numdiff package http://statsmodels.sourceforge.net/0.6.0/_modules/statsmodels/tools/numdiff.html

[LaiCrassidisCheng2005]
    K.-L. Lai, J.L. Crassidis, Y. Cheng, J. Kim (2005), New complex step derivative approximations with application to second-order kalman filtering, AIAA Guidance, Navigation and Control Conference, San Francisco, California, August 2005, AIAA-2005-5944.

[Ridout2009]
    Ridout, M.S. (2009) Statistical applications of the complex-step method of numerical differentiation. The American Statistician, 63, 66-74

[NAG]
    NAG Library. NAG Fortran Library Document: D04AAF

FOOTNOTES

[1] We would normally write these additional terms using O() notation, where all that matters is that the error term is O(elta) or perhaps O(elta^2), but explicit understanding of these error terms will be useful in the Richardson extrapolation step later on.

[2] An even function is one which expresses an even symmetry around a given point. An even symmetry has the property that f(x) = f(-x). Likewise, an odd function expresses an odd symmetry, wherein f(x) = -f(-x).

[3] For practical purposes the maximum order of the derivative is between 4 and 10 depending on the function to differentiate and also the method used in the approximation.

ALGORITHMIC DIFFERENTIATION

  Numdifftools.nd_algopy

This module provide an easy to use interface to derivatives calculated with AlgoPy. Algopy stands for Algorithmic Differentiation in Python.

The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs. Particular focus are functions that contain numerical linear algebra functions as they often appear in statistically motivated functions. The intended use of AlgoPy is for easy prototyping at reasonable execution speeds. More precisely, for a typical program a directional derivative takes order 10 times as much time as time as the function evaluation. This is approximately also true for the gradient.

  Algoritmic differentiation

Algorithmic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

Algorithmic differentiation is not:

Symbolic differentiation, nor Numerical differentiation (the method of finite differences). These classical methods run into problems: symbolic differentiation leads to inefficient code (unless carefully done) and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where the complexity and errors increase. Finally, both classical methods are slow at computing the partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Algoritmic differentiation solves all of these problems.

  Reference

Sebastian F. Walter and Lutz Lehmann 2013, "Algorithmic differentiation in Python with AlgoPy", in Journal of Computational Science, vol 4, no 5, pp 334 - 344, http://www.sciencedirect.com/science/article/pii/S1877750311001013

https://en.wikipedia.org/wiki/Automatic_differentiation

https://pythonhosted.org/algopy/index.html

LICENSE

Copyright (c) 2014, Per A. Brodtkorb, John D'Errico
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.
* Neither the name of the {organization} nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

DEVELOPERS

•   Per A. Brodtkorb <per.andreas.brodtkorb (at) gmail.com>

•   John D'Errico <woodchips (at) rochester.rr.com>

CHANGELOG

Created with gitcommand: git shortlog v0.9.12..v0.9.13

  Version 0.9.13, October 30, 2015

pbrod (21):

•   Updated README.rst and CHANGES.rst.

•   updated Limits.

•   Made it possible to differentiate complex functions and allow zero'th order derivative.

•   BUG: added missing derivative order, n to Gradient, Hessian, Jacobian.

•   Made test more robust.

•   Updated structure in setup according to pyscaffold version 2.4.2.

•   Updated setup.cfg and deleted duplicate tests folder.

•   removed unused code.

•   Added appveyor.yml.

•   Added required appveyor install scripts

•   Fixed bug in appveyor.yml.

•   added wheel to requirements.txt.

•   updated appveyor.yml.

•   Removed import matplotlib.

Justin Lecher (1):

•   Fix min version for numpy.

kikocorreoso (1):

•   fix some prints on run_benchmark.py to make it work with py3

  Version 0.9.12, August 28, 2015

pbrod (12):

•   Updated documentation.

•   Updated version in conf.py.

•   Updated CHANGES.rst.

•   Reimplemented outlier detection and made it more robust.

•   Added limits.py with tests.

•   Updated main tests folder.

•   Moved Richardson and dea3 to extrapolation.py.

•   Making a new release in order to upload to pypi.

  Version 0.9.11, August 27, 2015

pbrod (2):

•   Fixed sphinx-build and updated docs.

•   Fixed issue #9 Backward differentiation method fails with additional parameters.

  Version 0.9.10, August 26, 2015

pbrod (7):

•   Fixed sphinx-build and updated docs.

•   Added more tests to nd_algopy.

•   Dropped support for Python 2.6.

  Version 0.9.4, August 26, 2015

pbrod (7):

•   Fixed sphinx-build and updated docs.

  Version 0.9.3, August 23, 2015

Paul Kienzle (1):

•   more useful benchmark plots.

pbrod (7):

•   Fixed bugs and updated docs.

•   Major rewrite of the easy to use interface to Algopy.

•   Added possibility to calculate n'th order derivative not just for n=1 in nd_algopy.

•   Added tests to the easy to use interface to algopy.

  Version 0.9.2, August 20, 2015

pbrod (3):

•   Updated documentation

•   Added parenthesis to a call to the print function

•   Made the test less strict in order to pass the tests on Travis for python 2.6 and 3.2.

  Version 0.9.1, August 20,2015

Christoph Deil (1):

•   Fix Sphinx build

pbrod (47):

•

Total remake of numdifftools with slightly different call syntax.

•   Can compute derivatives of order up to 10-14 depending on function and method used.

•   Updated documentation and tests accordingly.

•   Fixed a bug in dea3.

•   Added StepsGenerator as an replacement for the adaptive option.

•   Added bicomplex class for testing the complex step second derivative.

•   Added fornberg_weights_all for computing optimal finite difference rules in a stable way.

•   Added higher order complex step derivative methods.

  Version 0.7.7, December 18, 2014

pbrod (35):

•   Got travis-ci working in order to run the tests automatically.

•   Fixed bugs in Dea class.

•   Fixed better error estimate for the Hessian.

•   Fixed tests for python 2.6.

•   Adding tests as subpackage.

•   Restructerd folders of numdifftools.

  Version 0.7.3, December 17, 2014

pbrod (5):

•   Small cosmetic fixes.

•   pep8 + some refactorings.

•   Simplified code by refactoring.

  Version 0.6.0, February 8, 2014

pbrod (20):

•   Update and rename README.md to README.rst.

•   Simplified call to Derivative: removed step_fix.

•   Deleted unused code.

•   Simplified and Refactored. Now possible to choose step_num=1.

•   Changed default step_nom from max(abs(x0), 0.2) to max(log2(abs(x0)), 0.2).

•   pep8ified code and made sure that all tests pass.

  Version 0.5.0, January 10, 2014

pbrod (9):

•   Updated the examples in Gradient class and in info.py.

•   Added test for vec2mat and docstrings + cosmetic fixes.

•   Refactored code into private methods.

•   Fixed issue #7: Derivative(fun)(numpy.ones((10,5)) * 2) failed.

•   Made print statements compatible with python 3.

  Version 0.4.0, May 5, 2012

pbrod (1)

•   Fixed a bug for inf and nan values.

  Version 0.3.5, May 19, 2011

pbrod (1)

•   Fixed a bug for inf and nan values.

  Version 0.3.4, Feb 24, 2011

pbrod (11)

•   Made automatic choice for the stepsize more robust.

•   Added easy to use interface to the algopy and scientificpython modules.

  Version 0.3.1, May 20, 2009

pbrod (4)

•   First version of numdifftools published on google.code

MODULES

  numdifftools package

  Subpackages

  numdifftools.tests package

  Submodules

  numdifftools.tests.conftest module

Dummy conftest.py for numdifftools.

If you don't know what this is for, just leave it empty. Read more about conftest.py under: https://pytest.org/latest/plugins.html

  numdifftools.tests.test_extrapolation module

class numdifftools.tests.test_extrapolation.TestExtrapolation(methodName='runTest')
    Bases: unittest.case.TestCase

setUp()

test_dea3_on_trapz_sin()

test_dea_on_trapz_sin()

test_richardson()

class numdifftools.tests.test_extrapolation.TestRichardson(methodName='runTest')
    Bases: unittest.case.TestCase

test_order_step_combinations()

  numdifftools.tests.test_hessian module

class numdifftools.tests.test_hessian.TestHessian(methodName='runTest')
    Bases: unittest.case.TestCase

test_hessian()

class numdifftools.tests.test_hessian.classicalHamiltonian
    Bases: object

    Hamiltonian

Parameters
    N : scalar number of ions in the chain

    w : scalar angular trap frequency

    C : scalar Coulomb constant times the electronic charge in SI units.

    m : scalar the mass of a single trapped ion in the chain

initialposition()
    Defines initial position as an estimate for the minimize process.

normal_modes(eigenvalues)
    Return normal modes

    the computed eigenvalues of the matrix Vx are of the form (normal_modes)2*m.

potential(positionvector)
    Return potential

    positionvector is an 1-d array (vector) of length N that contains the positions of the N ions

  numdifftools.tests.test_limits module

Created on 28. aug. 2015

@author: pab

class numdifftools.tests.test_limits.TestLimit(methodName='runTest')
    Bases: unittest.case.TestCase

test_derivative_of_cos()

test_difficult_limit()

test_residue()

test_sinx_divx()

  numdifftools.tests.test_multicomplex module

Created on 22. apr. 2015

@author: pab

class numdifftools.tests.test_multicomplex.BicomplexTester(methodName='runTest')
    Bases: unittest.case.TestCase

test_add()

test_arccos()

test_arcsin()

test_arg_c()

test_assign()

test_conjugate()

test_cos()

test_der_abs()

test_der_arccos()

test_der_arccosh()

test_der_arctan()

test_der_cos()

test_der_log()

test_division()

test_dot()

test_eq()

test_flat()

test_ge()

test_gt()

test_init()

test_le()

test_lt()

test_multiplication()

test_neg()

test_norm()

test_pow()

test_repr()

test_rpow()

test_rsub()

test_shape()

test_sub()

test_subsref()

class numdifftools.tests.test_multicomplex.DerivativeTester(methodName='runTest')
    Bases: unittest.case.TestCase

test_all_first_derivatives()

test_all_second_derivatives()

  numdifftools.tests.test_nd_algopy module

class numdifftools.tests.test_nd_algopy.TestDerivative(methodName='runTest')
    Bases: unittest.case.TestCase

test_derivative_cube()
    Test for Issue 7

test_derivative_exp()

test_derivative_on_log()

test_derivative_on_sinh()

test_derivative_sin()

test_fun_with_additional_parameters()
    Test for issue #9

test_high_order_derivative_cos()

class numdifftools.tests.test_nd_algopy.TestGradient(methodName='runTest')
    Bases: unittest.case.TestCase

test_on_scalar_function()

class numdifftools.tests.test_nd_algopy.TestHessdiag(methodName='runTest')
    Bases: unittest.case.TestCase

test_forward()

test_reverse()

class numdifftools.tests.test_nd_algopy.TestHessian(methodName='runTest')
    Bases: unittest.case.TestCase

test_hessian_cosIx_yI_at_I0_0I()

test_run_hamiltonian()

class numdifftools.tests.test_nd_algopy.TestJacobian(methodName='runTest')
    Bases: unittest.case.TestCase

test_on_scalar_function()

test_on_vector_valued_function()

class numdifftools.tests.test_nd_algopy.classicalHamiltonian
    Bases: object

    Hamiltonian

Parameters
    N : scalar number of ions in the chain

    w : scalar angular trap frequency

    C : scalar Coulomb constant times the electronic charge in SI units.

    m : scalar the mass of a single trapped ion in the chain

initialposition()
    Defines initial position as an estimate for the minimize process.

normal_modes(eigenvalues)
    Return normal modes

Computed eigenvalues of the matrix Vx are of the form
    (normal_modes)**2*m.

potential(positionvector)
    Return potential

    positionvector is an 1-d array (vector) of length N that contains the positions of the N ions

  numdifftools.tests.test_numdifftools module

Test functions for numdifftools module

class numdifftools.tests.test_numdifftools.TestDerivative(methodName='runTest')
    Bases: unittest.case.TestCase

test_backward_derivative_on_sinh()

test_central_and_forward_derivative_on_log()

test_default_scale()

test_derivative_cube()
    Test for Issue 7

test_derivative_exp()

test_derivative_of_cos_x()

test_derivative_sin()

test_fun_with_additional_parameters()
    Test for issue #9

test_high_order_derivative_cos()

test_infinite_functions()

class numdifftools.tests.test_numdifftools.TestFornbergWeights(methodName='runTest')
    Bases: unittest.case.TestCase

test_weights()

class numdifftools.tests.test_numdifftools.TestGlobalFunctions(methodName='runTest')
    Bases: unittest.case.TestCase

testdea3()

class numdifftools.tests.test_numdifftools.TestGradient(methodName='runTest')
    Bases: unittest.case.TestCase

testgradient()

class numdifftools.tests.test_numdifftools.TestHessdiag(methodName='runTest')
    Bases: unittest.case.TestCase

test_complex()

test_default_step()

test_fixed_step()

class numdifftools.tests.test_numdifftools.TestHessian(methodName='runTest')
    Bases: unittest.case.TestCase

test_hessian_cosIx_yI_at_I0_0I()

class numdifftools.tests.test_numdifftools.TestJacobian(methodName='runTest')
    Bases: unittest.case.TestCase

testjacobian()

class numdifftools.tests.test_numdifftools.TestRichardson(methodName='runTest')
    Bases: unittest.case.TestCase

test_central()

test_complex()

test_forward_backward()

class numdifftools.tests.test_numdifftools.TestStepGenerator(methodName='runTest')
    Bases: unittest.case.TestCase

test_default_base_step()

test_default_generator()

test_fixed_base_step()

  numdifftools.tests.test_numdifftools_docstrings module

numdifftools.tests.test_numdifftools_docstrings.load_tests(loader=None, tests=None, ignore=None)

numdifftools.tests.test_numdifftools_docstrings.suite()

  Module contents

  Submodules

  numdifftools.core module

numerical differentiation functions:

Derivative, Gradient, Jacobian, and Hessian

Author: Per A. Brodtkorb

Created: 01.08.2008 Copyright: (c) pab 2008 Licence: New BSD

Based on matlab functions derivest.m gradest.m hessdiag.m, hessian.m and jacobianest.m version 1.0 released 12/27/2006 by John D'Errico (e-mail: woodchips@rochester.rr.com)

Also based on the python functions approx_fprime, approx_fprime_cs, approx_hess_cs, approx_hess1, approx_hess2 and approx_hess3 in the statsmodels.tools.numdiff module released in 2014 written by Josef Perktold.

numdifftools.core.dea3(v0, v1, v2, symmetric=False)
    Extrapolate a slowly convergent sequence

Parameters
    v0, v1, v2 : array-like 3 values of a convergent sequence to extrapolate

Returns
    result : array-like extrapolated value

    abserr : array-like absolute error estimate

class numdifftools.core.Derivative(f, step=None, method='central', order=2, n=1, full_output=False)
    Bases: numdifftools.core._Derivative

    Calculate n-th derivative with finite difference approximation

Parameters
    f : function function of one array f(x, *args, **kwds)

    step : float, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=None) if step or method in in ['complex', 'multicomplex'], otherwise MaxStepGenerator(step_ratio=None, num_extrap=14) The results are extrapolated if the StepGenerator generate more than 3 steps.

    method : {'central', 'complex', 'multicomplex', 'forward', 'backward'} defines the method used in the approximation

    order : int, optional defines the order of the error term in the Taylor approximation used. For 'central' and 'complex' methods, it must be an even number.

    n : int, optional Order of the derivative.

    full_output : bool, optional If full_output is False, only the derivative is returned. If full_output is True, then (der, r) is returned der is the derivative, and r is a Results object.

Returns
    der : ndarray array of derivatives

    SEE ALSO: Gradient, Hessian Notes

    Complex methods are usually the most accurate provided the function to differentiate is analytic. The complex-step methods also requires fewer steps than the other methods and can work very close to the support of a function. The complex-step derivative has truncation error O(steps**2) for n=1 and O(steps**4) for n larger, so truncation error can be eliminated by choosing steps to be very small. Especially the first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, this method fails if f(x) does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. Central difference methods are almost as accurate and has no restriction on type of function. For this reason the 'central' method is the default method, but sometimes one can only allow evaluation in forward or backward direction.

    For all methods one should be careful in decreasing the step size too much due to round-off errors.

    Higher order approximation methods will generally be more accurate, but may also suffer more from numerical problems. First order methods is usually not recommended.

n

class numdifftools.core.Jacobian(f, step=None, method='central', order=2, full_output=False)
    Bases: numdifftools.core.Gradient

    Calculate Jacobian with finite difference approximation

Parameters
    f : function function of one array f(x, *args, **kwds)

    step : float, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=None) if step or method in in ['complex', 'multicomplex'], otherwise MaxStepGenerator(step_ratio=None, num_extrap=14) The results are extrapolated if the StepGenerator generate more than 3 steps.

    method : {'central', 'complex', 'multicomplex', 'forward', 'backward'} defines the method used in the approximation

    order : int, optional defines the order of the error term in the Taylor approximation used. For 'central' and 'complex' methods, it must be an even number.

    full_output : bool, optional If full_output is False, only the derivative is returned. If full_output is True, then (der, r) is returned der is the derivative, and r is a Results object.

Returns
    jacob : array Jacobian

    SEE ALSO: Derivative, Hessian, Gradient Notes

    Complex methods are usually the most accurate provided the function to differentiate is analytic. The complex-step methods also requires fewer steps than the other methods and can work very close to the support of a function. The complex-step derivative has truncation error O(steps**2) for n=1 and O(steps**4) for n larger, so truncation error can be eliminated by choosing steps to be very small. Especially the first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, this method fails if f(x) does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. Central difference methods are almost as accurate and has no restriction on type of function. For this reason the 'central' method is the default method, but sometimes one can only allow evaluation in forward or backward direction.

    For all methods one should be careful in decreasing the step size too much due to round-off errors.

    Higher order approximation methods will generally be more accurate, but may also suffer more from numerical problems. First order methods is usually not recommended.

    If f returns a 1d array, it returns a Jacobian. If a 2d array is returned by f (e.g., with a value for each observation), it returns a 3d array with the Jacobian of each observation with shape xk x nobs x xk. I.e., the Jacobian of the first observation would be [:, 0, :]

class numdifftools.core.Gradient(f, step=None, method='central', order=2, full_output=False)
    Bases: numdifftools.core.Derivative

    Calculate Gradient with finite difference approximation

Parameters
    f : function function of one array f(x, *args, **kwds)

    step : float, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=None) if step or method in in ['complex', 'multicomplex'], otherwise MaxStepGenerator(step_ratio=None, num_extrap=14) The results are extrapolated if the StepGenerator generate more than 3 steps.

    method : {'central', 'complex', 'multicomplex', 'forward', 'backward'} defines the method used in the approximation

    order : int, optional defines the order of the error term in the Taylor approximation used. For 'central' and 'complex' methods, it must be an even number.

    full_output : bool, optional If full_output is False, only the derivative is returned. If full_output is True, then (der, r) is returned der is the derivative, and r is a Results object.

Returns
    grad : array gradient

    SEE ALSO: Derivative, Hessian, Jacobian Notes

    Complex methods are usually the most accurate provided the function to differentiate is analytic. The complex-step methods also requires fewer steps than the other methods and can work very close to the support of a function. The complex-step derivative has truncation error O(steps**2) for n=1 and O(steps**4) for n larger, so truncation error can be eliminated by choosing steps to be very small. Especially the first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, this method fails if f(x) does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. Central difference methods are almost as accurate and has no restriction on type of function. For this reason the 'central' method is the default method, but sometimes one can only allow evaluation in forward or backward direction.

    For all methods one should be careful in decreasing the step size too much due to round-off errors.

    Higher order approximation methods will generally be more accurate, but may also suffer more from numerical problems. First order methods is usually not recommended.

class numdifftools.core.Hessian(f, step=None, method='central', full_output=False)
    Bases: numdifftools.core._Derivative

    Calculate Hessian with finite difference approximation

Parameters
    f : function function of one array f(x, *args, **kwds)

    step : float, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=None) if step or method in in ['complex', 'multicomplex'], otherwise MaxStepGenerator(step_ratio=None, num_extrap=14) The results are extrapolated if the StepGenerator generate more than 3 steps.

    method : {'central', 'complex', 'multicomplex', 'forward', 'backward'} defines the method used in the approximation

    full_output : bool, optional If full_output is False, only the derivative is returned. If full_output is True, then (der, r) is returned der is the derivative, and r is a Results object.

Returns
    hess : ndarray array of partial second derivatives, Hessian

    SEE ALSO: Derivative, Hessian Notes

    Complex methods are usually the most accurate provided the function to differentiate is analytic. The complex-step methods also requires fewer steps than the other methods and can work very close to the support of a function. The complex-step derivative has truncation error O(steps**2) for n=1 and O(steps**4) for n larger, so truncation error can be eliminated by choosing steps to be very small. Especially the first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, this method fails if f(x) does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. Central difference methods are almost as accurate and has no restriction on type of function. For this reason the 'central' method is the default method, but sometimes one can only allow evaluation in forward or backward direction.

    For all methods one should be careful in decreasing the step size too much due to round-off errors.

    Computes the Hessian according to method as: 'forward' 7, 'central' 9 and 'complex' 10:

    where e_j is a vector with element j is one and the rest are zero and d_j is a scalar spacing steps_j.

class numdifftools.core.Hessdiag(f, step=None, method='central', order=2, full_output=False)
    Bases: numdifftools.core.Derivative

    Calculate Hessian diagonal with finite difference approximation

Parameters
    f : function function of one array f(x, *args, **kwds)

    step : float, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=None) if step or method in in ['complex', 'multicomplex'], otherwise MaxStepGenerator(step_ratio=None, num_extrap=14) The results are extrapolated if the StepGenerator generate more than 3 steps.

    method : {'central', 'complex', 'multicomplex', 'forward', 'backward'} defines the method used in the approximationorder : int, optional defines the order of the error term in the Taylor approximation used. For 'central' and 'complex' methods, it must be an even number.

    full_output : bool, optional If full_output is False, only the derivative is returned. If full_output is True, then (der, r) is returned der is the derivative, and r is a Results object.

Returns
    hessdiag : array hessian diagonal

    SEE ALSO: Derivative, Hessian, Jacobian, Gradient Notes

    Complex methods are usually the most accurate provided the function to differentiate is analytic. The complex-step methods also requires fewer steps than the other methods and can work very close to the support of a function. The complex-step derivative has truncation error O(steps**2) for n=1 and O(steps**4) for n larger, so truncation error can be eliminated by choosing steps to be very small. Especially the first order complex-step derivative avoids the problem of round-off error with small steps because there is no subtraction. However, this method fails if f(x) does not support complex numbers or involves non-analytic functions such as e.g.: abs, max, min. Central difference methods are almost as accurate and has no restriction on type of function. For this reason the 'central' method is the default method, but sometimes one can only allow evaluation in forward or backward direction.

    For all methods one should be careful in decreasing the step size too much due to round-off errors.

    Higher order approximation methods will generally be more accurate, but may also suffer more from numerical problems. First order methods is usually not recommended.

class numdifftools.core.MinStepGenerator(base_step=None, step_ratio=2, num_steps=None, offset=0, scale=None, num_extrap=0, use_exact_steps=True, check_num_steps=True)
    Bases: object

    Generates a sequence of steps

    where steps = base_step * step_ratio ** (np.arange(num_steps) + offset)

Parameters
    base_step : float, array-like, optional Defines the base step, if None, then base_step is set to EPS**(1/scale)*max(log(1+|x|), 1) where x is supplied at runtime through the __call__ method.

    step_ratio : real scalar, optional, default 2 Ratio between sequential steps generated. Note: Ratio > 1 If None then step_ratio is 2 for n=1 otherwise step_ratio is 1.6

    num_steps : scalar integer, optional, default n + order - 1 + num_extrap defines number of steps generated. It should be larger than n + order - 1

    offset : real scalar, optional, default 0 offset to the base step

    scale : real scalar, optional scale used in base step. If not None it will override the default computed with the default_scale function.

class numdifftools.core.MaxStepGenerator(step_max=2.0, step_ratio=2.0, num_steps=15, step_nom=None, offset=0, num_extrap=0, use_exact_steps=False, check_num_steps=True)
    Bases: numdifftools.core.MinStepGenerator

    Generates a sequence of steps

where
    steps = base_step * step_ratio ** (-np.arange(num_steps) + offset) base_step = step_max * step_nom

Parameters
    max_step : float, array-like, optional default 2 Defines the maximum step

    step_ratio : real scalar, optional, default 2 Ratio between sequential steps generated. Note: Ratio > 1

    num_steps : scalar integer, optional, default n + order - 1 + num_extrap defines number of steps generated. It should be larger than n + order - 1

    step_nom : default maximum(log1p(abs(x)), 1) Nominal step.

    offset : real scalar, optional, default 0 offset to the base step: max_step * nom_step

class numdifftools.core.Richardson(step_ratio=2.0, step=1, order=1, num_terms=2)
    Bases: object

    Extrapolates as sequence with Richardsons method Notes

    Suppose you have series expansion that goes like this

    L = f(h) + a0 * h^p_0 + a1 * h^p_1+ a2 * h^p_2 + ...

    where p_i = order + step * i and f(h) -> L as h -> 0, but f(0) != L.

    If we evaluate the right hand side for different stepsizes h we can fit a polynomial to that sequence of approximations. This is exactly what this class does.

extrapolate(sequence, steps)

  numdifftools.extrapolation module

Created on 28. aug. 2015

@author: pab

class numdifftools.extrapolation.Dea(limexp=3)
    Bases: object

    LIMEXP is the maximum number of elements the epsilon table data can contain. The epsilon table is stored in the first (LIMEXP+2) entries of EPSTAB.

class numdifftools.extrapolation.Richardson(step_ratio=2.0, step=1, order=1, num_terms=2)
    Bases: object

    Extrapolates as sequence with Richardsons method Notes

    Suppose you have series expansion that goes like this

    L = f(h) + a0 * h^p_0 + a1 * h^p_1+ a2 * h^p_2 + ...

    where p_i = order + step * i and f(h) -> L as h -> 0, but f(0) != L.

    If we evaluate the right hand side for different stepsizes h we can fit a polynomial to that sequence of approximations. This is exactly what this class does.

extrapolate(sequence, steps)

numdifftools.extrapolation.convolve(sequence, rule, **kwds)
    Wrapper around scipy.ndimage.convolve1d that allows complex input.

numdifftools.extrapolation.dea3(v0, v1, v2, symmetric=False)
    Extrapolate a slowly convergent sequence

Parameters
    v0, v1, v2 : array-like 3 values of a convergent sequence to extrapolate

Returns
    result : array-like extrapolated value

    abserr : array-like absolute error estimate

numdifftools.extrapolation.test_dea()

  numdifftools.info module

  Introduction to Numdifftools

Numdifftools is a suite of tools written in Python to solve automatic numerical differentiation problems in one or more variables. Finite differences are used in an adaptive manner, coupled with a Richardson extrapolation methodology to provide a maximally accurate result. The user can configure many options like; changing the order of the method or the extrapolation, even allowing the user to specify whether complex, multicomplex, central, forward or backward differences are used. The methods provided are:

Derivative:
    Computates the derivative of order 1 through 10 on any scalar function.

Gradient:
    Computes the gradient vector of a scalar function of one or more variables.

Jacobian:
    Computes the Jacobian matrix of a vector valued function of one or more variables.

Hessian:
    Computes the Hessian matrix of all 2nd partial derivatives of a scalar function of one or more variables.

Hessdiag:
    Computes only the diagonal elements of the Hessian matrix

    All of these methods also produce error estimates on the result.

    Numdifftools also provide an easy to use interface to derivatives calculated with AlgoPy. Algopy stands for Algorithmic Differentiation in Python. The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs.

    Documentation is at: http://numdifftools.readthedocs.org/

    Code and issue tracker is at https://github.com/pbrod/numdifftools.

    Latest stable release is at http://pypi.python.org/pypi/Numdifftools.

    To test if the toolbox is working paste the following in an interactive python session:

    import numdifftools as nd
    nd.test(coverage=True, doctests=True)

  Getting Started

Compute 1'st and 2'nd derivative of exp(x), at x == 1:

>>> import numpy as np
>>> import numdifftools as nd
>>> fd = nd.Derivative(np.exp)        # 1'st derivative
>>> fdd = nd.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nd.Jacobian(fun)
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nd.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

Compute the same with the easy to use interface to AlgoPy:

>>> import numdifftools.nd_algopy as nda
>>> import numpy as np
>>> fd = nda.Derivative(np.exp)        # 1'st derivative
>>> fdd = nda.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nda.Jacobian(fun, method='reverse')
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nda.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

  See also

scipy.misc.derivative

numdifftools.info.test_docstrings()

  numdifftools.limits module

Created on 27. aug. 2015

@author: pab Author: John D'Errico e-mail: woodchips@rochester.rr.com Release: 1.0 Release date: 5/23/2008

class numdifftools.limits.Limit(f, step=None, method='above', order=4, full_output=False)
    Bases: object

    Compute limit of a function at a given point

Parameters
    f : callable function of one array f(z, *args, **kwds) to compute the limit for. The function, f, is assumed to return a result of the same shape and size as its input, z.

    step: float, complex, array-like or StepGenerator object, optional Defines the spacing used in the approximation. Default is MinStepGenerator(base_step=step, step_ratio=4)

    method : {'above', 'below'} defines if the limit is taken from above or below

    order: positive scalar integer, optional. defines the order of approximation used to find the specified limit. The order must be member of [1 2 3 4 5 6 7 8]. 4 is a good compromise.

Returns
    limit_fz: array like estimated limit of f(z) as z --> z0

    info: Only given if full_output is True and contains the following: error estimate: ndarray 95 uncertainty estimate around the limit, such that abs(limit_fz - f(z0)*(z-z0)) < error_estimate

final_step: ndarray
    final step used in approximation

class info
    Bases: tuple

error_estimate
    Alias for field number 0

final_step
    Alias for field number 1

index
    Alias for field number 2

Limit.limit(x, *args, **kwds)

class numdifftools.limits.MinStepGenerator(base_step=None, step_ratio=4.0, num_steps=None, offset=0, scale=1.2, use_exact_steps=True)
    Bases: object

    Generates a sequence of steps

    where steps = base_step * step_ratio ** (np.arange(num_steps) + offset)

Parameters
    base_step : float, array-like, optional Defines the base step, if None, then base_step is set to EPS**(1/scale)*max(log(1+|x|), 1) where x is supplied at runtime through the __call__ method.

    step_ratio : real scalar, optional, default 4 Ratio between sequential steps generated.

    num_steps : scalar integer, optional, default n + order - 1 + num_extrap defines number of steps generated. It should be larger than n + order - 1

    offset : real scalar, optional, default 0 offset to the base step

    scale : real scalar, optional scale used in base step. If not None it will override the default computed with the default_scale function.

numdifftools.limits.nom_step(x=None)
    Return nominal step

numdifftools.limits.test_docstrings()

numdifftools.limits.valarray(shape, value=nan, typecode=None)
    Return an array of all value.

  numdifftools.multicomplex module

Created on 22. apr. 2015

@author: pab

  References

A METHODOLOGY FOR ROBUST OPTIMIZATION OF LOW-THRUST TRAJECTORIES IN MULTI-BODY ENVIRONMENTS Gregory Lantoine (2010) Phd thesis, Georgia Institute of Technology

USING MULTICOMPLEX VARIABLES FOR AUTOMATIC COMPUTATION OF HIGH-ORDER DERIVATIVES Gregory Lantoine, Ryan P. Russell , and Thierry Dargent ACM Transactions on Mathematical Software, Vol. 38, No. 3, Article 16, April 2012, 21 pages,

M.E. Luna-Elizarraras, M. Shapiro, D.C. Struppa1, A. Vajiac (2012) CUBO A Mathematical Journal Vol. 14, No 2, (61-80). June 2012.

Computation of higher-order derivatives using the multi-complex step method Adriaen Verheyleweghen, (2014) Project report, NTNU

class numdifftools.multicomplex.bicomplex(z1, z2)
    Bases: object

    Creates an instance of a bicomplex object. zeta = z1 + j*z2, where z1 and z2 are complex numbers.

arccos()

arccosh()

arcsin()

arcsinh()

arctan()

arctanh()

arg_c()

arg_c1p()

static asarray(other)

conjugate()

cos()

cosh()

cot()

coth()

csc()

csch()

dot(other)

exp()

exp2()

expm1()

flat(index)

imag

imag1

imag12

imag2

log()

log10()

log1p()

log2()

logaddexp(other)

logaddexp2(other)

static mat2bicomp(arr)

mod_c()
    Complex modulus

norm()

real

sec()

sech()

shape

sin()

sinh()

size

sqrt()

tan()

tanh()

numdifftools.multicomplex.c_abs(z)

numdifftools.multicomplex.c_atan2(x, y)

numdifftools.multicomplex.c_max(x, y)

numdifftools.multicomplex.c_min(x, y)

  numdifftools.nd_algopy module

  Numdifftools.nd_algopy

This module provide an easy to use interface to derivatives calculated with AlgoPy. Algopy stands for Algorithmic Differentiation in Python.

The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs. Particular focus are functions that contain numerical linear algebra functions as they often appear in statistically motivated functions. The intended use of AlgoPy is for easy prototyping at reasonable execution speeds. More precisely, for a typical program a directional derivative takes order 10 times as much time as time as the function evaluation. This is approximately also true for the gradient.

  Algoritmic differentiation

Algorithmic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

Algorithmic differentiation is not:

Symbolic differentiation, nor Numerical differentiation (the method of finite differences). These classical methods run into problems: symbolic differentiation leads to inefficient code (unless carefully done) and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation. Both classical methods have problems with calculating higher derivatives, where the complexity and errors increase. Finally, both classical methods are slow at computing the partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Algoritmic differentiation solves all of these problems.

  Reference

Sebastian F. Walter and Lutz Lehmann 2013, "Algorithmic differentiation in Python with AlgoPy", in Journal of Computational Science, vol 4, no 5, pp 334 - 344, http://www.sciencedirect.com/science/article/pii/S1877750311001013

https://en.wikipedia.org/wiki/Automatic_differentiation

https://pythonhosted.org/algopy/index.html

class numdifftools.nd_algopy.Derivative(f, n=1, method='forward')
    Bases: numdifftools.nd_algopy._Common

    Calculate n-th derivative with Algorithmic Differentiation method

Parameters
    f : function function of one array f(x, *args, **kwds)

    n : int, optional Order of the derivative.

    method : string, optional {'forward', 'reverse'} defines method used in the approximation

Returns
    der : ndarray array of derivatives

    SEE ALSO: Gradient, Hessdiag, Hessian, Jacobian Notes

    Algorithmic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

class numdifftools.nd_algopy.Gradient(f, method='forward')
    Bases: numdifftools.nd_algopy._Common

    Calculate Gradient with Algorithmic Differentiation method

Parameters
    f : function function of one array f(x, *args, **kwds)

    method : string, optional {'forward', 'reverse'} defines method used in the approximation

Returns
    grad : array gradient

    SEE ALSO: Derivative, Jacobian, Hessdiag, Hessian Notes

    Algorithmic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

class numdifftools.nd_algopy.Hessdiag(f, method='forward')
    Bases: numdifftools.nd_algopy.Hessian

    Calculate Hessian diagonal with Algorithmic Differentiation method

Parameters
    f : function function of one array f(x, *args, **kwds)

    method : string, optional {'forward', 'reverse'} defines method used in the approximation

Returns
    hessdiag : ndarray Hessian diagonal array of partial second order derivatives.

    SEE ALSO: Derivative, Gradient, Jacobian, Hessian Notes

    Algorithmic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

class numdifftools.nd_algopy.Hessian(f, method='forward')
    Bases: numdifftools.nd_algopy._Common

    Calculate Hessian with Algorithmic Differentiation method

Parameters
    f : function function of one array f(x, *args, **kwds)

    method : string, optional {'forward', 'reverse'} defines method used in the approximation

Returns
    hess : ndarray array of partial second derivatives, Hessian

    SEE ALSO: Derivative, Gradient, Jacobian, Hessdiag Notes

    Algorithmic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

class numdifftools.nd_algopy.Jacobian(f, method='forward')
    Bases: numdifftools.nd_algopy._Common

    Calculate Jacobian with Algorithmic Differentiation method

Parameters
    f : function function of one array f(x, *args, **kwds)

    method : string, optional {'forward', 'reverse'} defines method used in the approximation

Returns
    jacob : array Jacobian

    SEE ALSO: Derivative, Gradient, Hessdiag, Hessian Notes

    Algorithmic differentiation is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

numdifftools.nd_algopy.test_docstrings()

  numdifftools.run_benchmark module

  numdifftools.test_functions module

Created on 17. mai 2015

@author: pab

numdifftools.test_functions.dcos(x)

numdifftools.test_functions.ddcos(x)

numdifftools.test_functions.get_function(fun_name, n=1)

  Module contents

  Introduction to Numdifftools

Numdifftools is a suite of tools written in Python to solve automatic numerical differentiation problems in one or more variables. Finite differences are used in an adaptive manner, coupled with a Richardson extrapolation methodology to provide a maximally accurate result. The user can configure many options like; changing the order of the method or the extrapolation, even allowing the user to specify whether complex, multicomplex, central, forward or backward differences are used. The methods provided are:

Derivative:
    Computates the derivative of order 1 through 10 on any scalar function.

Gradient:
    Computes the gradient vector of a scalar function of one or more variables.

Jacobian:
    Computes the Jacobian matrix of a vector valued function of one or more variables.

Hessian:
    Computes the Hessian matrix of all 2nd partial derivatives of a scalar function of one or more variables.

Hessdiag:
    Computes only the diagonal elements of the Hessian matrix

    All of these methods also produce error estimates on the result.

    Numdifftools also provide an easy to use interface to derivatives calculated with AlgoPy. Algopy stands for Algorithmic Differentiation in Python. The purpose of AlgoPy is the evaluation of higher-order derivatives in the forward and reverse mode of Algorithmic Differentiation (AD) of functions that are implemented as Python programs.

    Documentation is at: http://numdifftools.readthedocs.org/

    Code and issue tracker is at https://github.com/pbrod/numdifftools.

    Latest stable release is at http://pypi.python.org/pypi/Numdifftools.

    To test if the toolbox is working paste the following in an interactive python session:

    import numdifftools as nd
    nd.test(coverage=True, doctests=True)

  Getting Started

Compute 1'st and 2'nd derivative of exp(x), at x == 1:

>>> import numpy as np
>>> import numdifftools as nd
>>> fd = nd.Derivative(np.exp)        # 1'st derivative
>>> fdd = nd.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nd.Jacobian(fun)
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nd.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

Compute the same with the easy to use interface to AlgoPy:

>>> import numdifftools.nd_algopy as nda
>>> import numpy as np
>>> fd = nda.Derivative(np.exp)        # 1'st derivative
>>> fdd = nda.Derivative(np.exp, n=2)  # 2'nd derivative
>>> np.allclose(fd(1), 2.7182818284590424)
True
>>> np.allclose(fdd(1), 2.7182818284590424)
True

Nonlinear least squares:

>>> xdata = np.reshape(np.arange(0,1,0.1),(-1,1))
>>> ydata = 1+2*np.exp(0.75*xdata)
>>> fun = lambda c: (c[0]+c[1]*np.exp(c[2]*xdata) - ydata)**2
>>> Jfun = nda.Jacobian(fun, method='reverse')
>>> np.allclose(np.abs(Jfun([1,2,0.75])), 0) # should be numerically zero
True

Compute gradient of sum(x**2):

>>> fun = lambda x: np.sum(x**2)
>>> dfun = nda.Gradient(fun)
>>> dfun([1,2,3])
array([ 2.,  4.,  6.])

  See also

scipy.misc.derivative

•   genindex

•   modindex

•   search

COPYRIGHT
2009-2015, Per A Brodtkorb, John D'Errico