Build real-time stuff with Speedy Vision, a GPU-accelerated Computer Vision library for JavaScript.
Speedy Vision is developed independently by Alexandre Martins and released under the Apache-2.0 license.
For web-based Augmented Reality, check out my other project.
- Feature detection
- Harris corner detector
- FAST feature detector
- ORB feature descriptor
- Feature tracking
- KLT feature tracker
- LK optical flow
- Feature matching
- Fast approximate k-nearest neighbors (kNN)
- Brute force matching
- Geometric transformations
- Homography matrix
- Affine transform
- Image processing
- Convert to greyscale
- Convolution
- Gaussian blur, box & median filters
- Contrast and brightness adjustment
- Image normalization & warping
- Image pyramids
- Linear Algebra
- Beautiful matrix algebra with a fluent interface
- Efficient computations with WebAssembly
- Systems of linear equations
- QR decomposition
... and more in development!
There are plenty of demos available!
- Demos
- Usage
- Motivation
- The Pipeline
- API Reference
- Unit tests
Try the demos and take a look at their source code:
- Hello, world!
- Feature detection
- Feature tracking
- Feature matching
- Image processing
- Linear Algebra
Download the latest release of Speedy Vision and include it in the <head> section of your HTML page:
<script src="dist/speedy-vision.min.js"></script>Once you import the library, the Speedy object will be exposed. Check out the Hello World demo for a working example.
Add the following to the <head> of your HTML page:
<script src="https://cdn.jsdelivr.net/gh/alemart/speedy-vision@VERSION/dist/speedy-vision.min.js"></script>Replace VERSION by
Simply run:
npm install speedy-visionNext, import the Speedy object as follows:
import Speedy from 'speedy-vision';Detecting features in an image is an important step of many computer vision algorithms. Traditionally, the computationally expensive nature of this process made it difficult to bring interactive Computer Vision applications to the web browser. The framerates were unsatisfactory for a compelling user experience. Speedy, a short name for Speedy Vision, is a JavaScript library created to address this issue.
Speedy's real-time performance in the web browser is possible thanks to its efficient WebGL2 backend and to its GPU implementations of fast computer vision algorithms. With an easy-to-use API, Speedy is an excellent choice for real-time computer vision projects involving tasks such as: object detection in videos, pose estimation, Simultaneous Location and Mapping (SLAM), and others.
The pipeline is a central concept in Speedy. It's a powerful structure that lets you organize the computations that take place in the GPU. It's a very flexible, yet conceptually simple, way of working with computer vision and image processing. Let's define a few things:
- A pipeline is a network of nodes in which data flows downstream from one or more sources to one or more sinks.
- Nodes have input and/or output ports. A node with no input ports is called a source. A node with no output ports is called a sink. A node with both input and output ports transforms the input data in some way and writes the results to its output port(s).
- A link connects an output port of a node to an input port of another node. Two nodes are said to be connected if there is a link connecting their ports. Data flows from one node to another by means of a link. An input port may only be connected to a single output port, but an output port may be connected to multiple input ports.
- Input ports expect data of a certain type (e.g., an image). Output ports hold data of a certain type. Two ports may only be connected if their types match.
- Ports may impose additional constraints on the data passing through them. For example, an input port may expect an image and also impose the constraint that this image must be greyscale.
- Different nodes may have different parameters. These parameters can be adjusted and are meant to modify the output of the nodes in some way.
- Nodes and their ports have names. An input port is typically called
"in". An output port is typically called"out". These names can vary, e.g., if a node has more than one input / output port. Speedy automatically assigns names to the nodes, but you can assign your own names as well.
The picture below shows a visual representation of a pipeline that converts an image or video to greyscale. Data gets into the pipeline via the image source. It is then passed to the Convert to greyscale node. Finally, a greyscale image goes into the image sink, where it gets out of the pipeline.
Here's a little bit of code:
// Load an image
const img = document.querySelector('img');
const media = await Speedy.load(img);
// Create the pipeline and the nodes
const pipeline = Speedy.Pipeline();
const source = Speedy.Image.Source();
const sink = Speedy.Image.Sink();
const greyscale = Speedy.Filter.Greyscale();
// Set the media source
source.media = media; // media is a SpeedyMedia object
// Connect the nodes
source.output().connectTo(greyscale.input());
greyscale.output().connectTo(sink.input());
// Specify the nodes to initialize the pipeline
pipeline.init(source, sink, greyscale);
// Run the pipeline
const { image } = await pipeline.run(); // image is a SpeedyMedia
// Create a <canvas> to display the result
const canvas = document.createElement('canvas');
canvas.width = image.width;
canvas.height = image.height;
document.body.appendChild(canvas);
// Display the result
const ctx = canvas.getContext('2d');
ctx.drawImage(media.source, 0, 0);Speedy provides many types of nodes. You can connect these nodes in a way that is suitable to your application, and Speedy will bring back the results you ask for.
A SpeedyMedia object encapsulates a media object: an image, a video, a canvas or a bitmap.
Speedy.load(source: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | OffscreenCanvas | ImageBitmap, options?: object): SpeedyPromise<SpeedyMedia>
Tells Speedy to load source. The source parameter may be an image, a video, a canvas or a bitmap.
source: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | OffscreenCanvas | ImageBitmap. The media source.options: object, optional. Additional options for advanced configuration. See SpeedyMedia.options for details.
A SpeedyPromise<SpeedyMedia> that resolves as soon as the media source is loaded.
window.onload = async function() {
let image = document.getElementById('my-image'); // <img id="my-image" src="...">
let media = await Speedy.load(image);
}Speedy.camera(width?: number, height?: number): SpeedyPromise<SpeedyMedia>
Speedy.camera(constraints: MediaStreamConstraints): SpeedyPromise<SpeedyMedia>
Loads a camera stream into a new SpeedyMedia object. This is a wrapper around navigator.mediaDevices.getUserMedia(), provided for your convenience.
width: number, optional. The ideal width of the stream. The browser will use this value or a close match. Defaults to640.height: number, optional. The ideal height of the stream. The browser will use this value or a close match. Defaults to360.constraints: MediaStreamConstraints. A MediaStreamConstraints dictionary to be passed togetUserMedia()for complete customization.
A SpeedyPromise<SpeedyMedia> that resolves as soon as the media source is loaded with the camera stream.
// Display the contents of a webcam
window.onload = async function() {
const media = await Speedy.camera();
const canvas = createCanvas(media.width, media.height);
const ctx = canvas.getContext('2d');
function render()
{
ctx.drawImage(media.source, 0, 0);
requestAnimationFrame(render);
}
render();
}
function createCanvas(width, height)
{
const canvas = document.createElement('canvas');
canvas.width = width;
canvas.height = height;
document.body.appendChild(canvas);
return canvas;
}SpeedyMedia.release(): null
Releases internal resources associated with this SpeedyMedia.
Returns null.
SpeedyMedia.source: HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | OffscreenCanvas | ImageBitmap | ImageData, read-only
The media source associated with the SpeedyMedia object.
SpeedyMedia.type: string, read-only
The type of the media source. One of the following: "image", "video", "canvas", "offscreen-canvas", "bitmap", "data".
See also: SpeedyMedia.source.
SpeedyMedia.width: number, read-only
The width of the media source, in pixels.
SpeedyMedia.height: number, read-only
The height of the media source, in pixels.
SpeedyMedia.size: SpeedySize, read-only
The size of the media, in pixels.
SpeedyMedia.options: object, read-only
Read-only object defined when loading the media. Deprecated.
SpeedyMedia.clone(): SpeedyPromise<SpeedyMedia>
Clones the SpeedyMedia object.
A SpeedyPromise that resolves to a clone of the SpeedyMedia object.
const clone = await media.clone();SpeedyMedia.toBitmap(): SpeedyPromise<ImageBitmap>
Converts the media to an ImageBitmap.
A SpeedyPromise that resolves to an ImageBitmap.
Speedy.Pipeline.Pipeline(): SpeedyPipeline
Creates a new, empty pipeline.
A new SpeedyPipeline object.
SpeedyPipeline.init(...nodes: SpeedyPipelineNode[]): SpeedyPipeline
Initializes a pipeline with the specified nodes.
...nodes: SpeedyPipelineNode[]. The list of nodes that belong to the pipeline.
The pipeline itself.
const pipeline = Speedy.Pipeline(); // create the pipeline and the nodes
const source = Speedy.Image.Source();
const sink = Speedy.Image.Sink();
const greyscale = Speedy.Filter.Greyscale();
source.media = media; // set the media source
source.output().connectTo(greyscale.input()); // connect the nodes
greyscale.output().connectTo(sink.input());
pipeline.init(source, sink, greyscale); // add the nodes to the pipelineSpeedyPipeline.release(): null
Releases the resources associated with this pipeline.
Returns null.
SpeedyPipeline.run(): SpeedyPromise<object>
Runs this pipeline.
Returns a SpeedyPromise that resolves to an object whose keys are the names of the sinks of the pipeline and whose values are the data exported by those sinks.
const { sink1, sink2 } = await pipeline.run();SpeedyPipeline.node(name: string): SpeedyPipelineNode | null
Finds a node by its name.
name: string. Name of the target node.
Returns a SpeedyPipelineNode that has the specified name and that belongs to this pipeline, or null if there is no such node.
SpeedyPipelineNode.input(portName?: string): SpeedyPipelineNodePort
The input port of this node whose name is portName.
portName: string, optional. The name of the port you want to access. Defaults to"in".
The requested input port.
SpeedyPipelineNode.output(portName?: string): SpeedyPipelineNodePort
The output port of this node whose name is portName.
portName: string, optional. The name of the port you want to access. Defaults to"out".
The requested output port.
SpeedyPipelineNodePort.connectTo(port: SpeedyPipelineNodePort): void
Creates a link connecting this port to another port.
SpeedyPipelineNode.name: string, read-only
The name of the node.
SpeedyPipelineNode.fullName: string, read-only
A string that exhibits the name and the type of the node.
SpeedyPipelineNodePort.name: string, read-only
The name of the port.
SpeedyPipelineNodePort.node: SpeedyPipelineNode, read-only
The node to which this port belongs.
Speedy.Image.Source(name?: string): SpeedyPipelineNodeImageInput
Creates an image source with the specified name. If the name is not specified, Speedy will automatically generate a name for you.
media: SpeedyMedia. The media to be imported into the pipeline.
| Port name | Data type | Description |
|---|---|---|
"out" |
Image | An image corresponding to the media of this node. |
Speedy.Image.Sink(name?: string): SpeedyPipelineNodeImageOutput
Creates an image sink with the specified name. If the name is not specified, Speedy will call this node "image". A SpeedyMedia object will be exported from the pipeline.
mediaType: "bitmap" | "data". The desired type of the source of the exportedSpeedyMedia. Use"bitmap"to be able to draw the exported image to a canvas without undue latency, or"data"to be able to access its pixel data directly. Defaults to"bitmap".
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | An image to be exported from the pipeline. |
Speedy.Image.Pyramid(name?: string): SpeedyPipelineNodeImagePyramid
Generate a Gaussian pyramid. A pyramid is a texture with mipmaps.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | Gaussian pyramid. |
Speedy.Image.Multiplexer(name?: string): SpeedyPipelineNodeImageMultiplexer
An image multiplexer receives two images as input and outputs one of the them.
port: number. Which input image should be redirected to the output:0or1? Defaults to0.
| Port name | Data type | Description |
|---|---|---|
"in0" |
Image | First image. |
"in1" |
Image | Second image. |
"out" |
Image | Either the first or the second image, depending on the value of port. |
Speedy.Image.Buffer(name?: string): SpeedyPipelineNodeImageBuffer
An image buffer outputs at time t the input image received at time t-1. It's useful for tracking.
Note: an image buffer cannot be used to store a pyramid at this time.
frozen: boolean. A frozen buffer discards the input, effectively increasing the buffering time. Defaults tofalse.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image at time t. |
"out" |
Image | Output image: the input image at time t-1. |
Speedy.Image.Mixer(name?: string): SpeedyPipelineNodeImageMixer
An image mixer combines two images, image0 and image1, as follows:
output = alpha * image0 + beta * image1 + gamma
The above expression will be computed for each pixel of the resulting image and then clamped to the [0,1] interval. The dimensions of the resulting image will be the dimensions of the larger of the input images.
Note: Both input images must have the same format. If they're colored, the above expression will be evaluated in each color channel independently.
Tip: if you pick an alpha between 0 and 1, set beta to 1 - alpha and set gamma to 0, you'll get a nice alpha blending effect.
alpha: number. A scalar value. Defaults to 0.5.beta: number. A scalar value. Defaults to 0.5.gamma: number. A scalar value. Defaults to 0.0.
| Port name | Data type | Description |
|---|---|---|
"in0" |
Image | Input image: the image0 above |
"in1" |
Image | Input image: the image1 above |
"out" |
Image | Output image |
Speedy.Filter.Greyscale(name?: string): SpeedyPipelineNodeGreyscale
Convert an image to greyscale.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | The input image converted to greyscale. |
Speedy.Filter.SimpleBlur(name?: string): SpeedyPipelineNodeSimpleBlur
Blur an image using a box filter.
kernelSize: SpeedySize. The size of the convolution kernel: from 3x3 to 15x15. Defaults to 5x5.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | The input image, blurred. |
Speedy.Filter.SimpleBlur(name?: string): SpeedyPipelineNodeGaussianBlur
Blur an image using a Gaussian filter.
kernelSize: SpeedySize. The size of the convolution kernel: from 3x3 to 15x15. Defaults to 5x5.sigma: SpeedyVector2. The sigma of the Gaussian function in both x and y axes. If set to the zero vector, Speedy will automatically pick a sigma according to the selectedkernelSize. Defaults to (0,0).
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | The input image, blurred. |
Speedy.Filter.MedianBlur(name?: string): SpeedyPipelineNodeMedianBlur
Median filter.
kernelSize: SpeedySize. One of the following: 3x3, 5x5 or 7x7. Defaults to 5x5.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | A greyscale image. |
"out" |
Image | The result of the median blur. |
const median = Speedy.Filter.MedianBlur();
median.kernelSize = Speedy.Size(7,7);Speedy.Filter.Convolution(name?: string): SpeedyPipelineNodeConvolution
Compute the convolution of an image using a 2D kernel.
kernel: SpeedyMatrixExpr. A 3x3, 5x5 or 7x7 matrix.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | The result of the convolution. |
// Sharpening an image
const sharpen = Speedy.Filter.Convolution();
sharpen.kernel = Speedy.Matrix(3, 3, [
0,-1, 0,
-1, 5,-1,
0,-1, 0
]);Speedy.Filter.Normalize(name?: string): SpeedyPipelineNodeNormalize
Normalize the intensity values of the input image to the [minValue, maxValue] interval.
minValue: number. A value in [0,255].maxValue: number. A value in [0,255] greater than or equal tominValue.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Greyscale image. |
"out" |
Image | Normalized image. |
Speedy.Filter.Nightvision(name?: string): SpeedyPipelineNodeNightvision
Nightvision filter for local contrast stretching and brightness control.
gain: number. A value in [0,1]: the larger the number, the higher the contrast. Defaults to0.5.offset: number. A value in [0,1] that controls the brightness. Defaults to0.5.decay: number. A value in [0,1] specifying a contrast decay from the center of the image. Defaults to zero (no decay).quality: string. Quality level:"high","medium"or"low". Defaults to"medium".
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | Output image. |
Speedy.Transform.Resize(name?: string): SpeedyPipelineNodeResize
Resize an image.
size: SpeedySize. The size of the output image, in pixels. If set to zero,scalewill be used to determine the size of the output. Defaults to zero.scale: SpeedyVector2. The size of the output image relative to the size of the input image. This parameter is only applied ifsizeis zero. Defaults to (1,1), meaning: keep the original size.method: string. Resize method. One of the following:"bilinear"(bilinear interpolation) or"nearest"(nearest neighbors). Defaults to"bilinear".
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | Resized image. |
Speedy.Transform.PerspectiveWarp(name?: string): SpeedyPipelineNodePerspectiveWarp
Warp an image using a homography matrix.
transform: SpeedyMatrixExpr. A 3x3 perspective transformation. Defaults to the identity matrix.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Input image. |
"out" |
Image | Warped image. |
A keypoint is a small patch in an image that is somehow distinctive. For example, a small patch with significant intensity changes in both x and y axes (i.e., a "corner") is distinctive. If we pick two "similar" images, we should be able to locate a set of keypoints in each of them and then match those keypoints based on their similarity.
A descriptor is a mathematical object that somehow describes a keypoint. Two keypoints are considered to be "similar" if their descriptors are "similar". Speedy works with binary descriptors, meaning that keypoints are described using bit vectors of fixed length.
There are different ways to detect and describe keypoints. For example, in order to detect a keypoint, you may take a look at the pixel intensities around a point or perhaps study the image derivatives. You may describe a keypoint by comparing the pixel intensities of the image patch in a special way. Additionally, it's possible to conceive a way to describe a keypoint in such a way that, if you rotate the patch, the descriptor stays roughly the same. This is called rotational invariance and is usually a desirable property for a descriptor.
Speedy offers different options for processing keypoints in multiple ways. A novelty of this work is that Speedy's implementations have been either adapted from the literature or conceived from scratch to work on the GPU. Therefore, keypoint processing is done in parallel and is often very fast.
A SpeedyKeypoint object represents a keypoint.
SpeedyKeypoint.position: SpeedyPoint2
The position of the keypoint in the image.
SpeedyKeypoint.x: number
The x position of the keypoint in the image. A shortcut to position.x.
SpeedyKeypoint.y: number
The y position of the keypoint in the image. A shortcut to position.y.
SpeedyKeypoint.lod: number, read-only
The level-of-detail (pyramid level) from which the keypoint was extracted, starting from zero. Defaults to 0.0.
SpeedyKeypoint.scale: number, read-only
The scale of the keypoint. This is equivalent to 2 ^ lod. Defaults to 1.0.
SpeedyKeypoint.rotation: number, read-only
The rotation angle / orientation of the keypoint, in radians. Defaults to 0.0.
SpeedyKeypoint.score: number, read-only
The score is a measure associated with the keypoint. Although different detection methods employ different measurement strategies, the larger the score, the "better" the keypoint is considered to be. The score is always a positive value.
SpeedyKeypoint.descriptor: SpeedyKeypointDescriptor | null, read-only
The descriptor associated with the keypoint, if it exists.
A SpeedyKeypointDescriptor represents a keypoint descriptor.
SpeedyKeypointDescriptor.data: Uint8Array, read-only
The bytes of the keypoint descriptor.
SpeedyKeypointDescriptor.size: number, read-only
The size of the keypoint descriptor, in bytes.
SpeedyKeypointDescriptor.toString(): string
Returns a string representation of the keypoint descriptor.
A SpeedyTrackedKeypoint is a SpeedyKeypoint with the following additional properties:
SpeedyTrackedKeypoint.flow: SpeedyVector2, read-only
A displacement vector associated with the tracked keypoint.
A SpeedyMatchedKeypoint is a SpeedyKeypoint with the following additional properties:
SpeedyMatchedKeypoint.matches: SpeedyKeypointMatch[], read-only
A list of keypoint matches associated with the keypoint. They will be sorted by increasing distance (better matches come first).
See also: SpeedyKeypointMatch.
A SpeedyKeypointMatch represents a keypoint match.
SpeedyKeypointMatch.index: number, read-only
The non-negative index of the matched keypoint in a database of keypoints, or -1 if there is no match.
SpeedyKeypointMatch.distance: number, read-only
A distance metric between the keypoint and the matched keypoint. The lower the distance, the better the match. If there is no match, then this field will be set to infinity.
Speedy.Keypoint.Source(name?: string): SpeedyPipelineNodeKeypointSource
Creates a source of keypoints. Only the position, score and scale of the provided keypoints will be imported to the pipeline. Descriptors, if present, will be lost.
keypoints: SpeedyKeypoint[]. The keypoints you want to import.capacity: number. The maximum number of keypoints that can be imported to the GPU. If you have an idea of how many keypoints you expect (at most), use a tight bound to make processing more efficient. The default capacity is2048. It can be no larger than8192.
| Port name | Data type | Description |
|---|---|---|
"out" |
Keypoints | The imported set of keypoints. |
Speedy.Keypoint.Sink(name?: string): SpeedyPipelineNodeKeypointSink
Creates a sink of keypoints using the specified name. If the name is not specified, Speedy will call this node "keypoints". An array of SpeedyKeypoint objects will be exported from the pipeline.
turbo: boolean. Accelerate GPU-CPU transfers. You'll get the data from the previous frame. Defaults tofalse.includeDiscarded: boolean. Set discarded keypoints (e.g., by a tracker) tonullin the exported set. Defaults tofalse, meaning that discarded keypoints will simply be dropped from the exported set rather than being set tonull.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints to be exported from the pipeline. |
Speedy.Keypoint.Clipper(name?: string): SpeedyPipelineNodeKeypointClipper
Clips a set of keypoints, so that it outputs no more than a fixed quantity of them. When generating the output, it will choose the "best" keypoints according to their score metric. The keypoint clipper is a very useful tool to reduce processing time, since it can discard "bad" keypoints regardless of the sensitivity of their detector. The clipping must be applied before computing any descriptors.
size: number. A positive integer. No more than this number of keypoints will be available in the output.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"out" |
Keypoints | A set of at most size keypoints. |
Speedy.Keypoint.BorderClipper(name?: string): SpeedyPipelineNodeKeypointBorderClipper
Removes all keypoints within a specified border of the edge of an image. The border is specified in pixels as an ordered pair of integers: the first is the size of the horizontal border and the second is the size of the vertical border.
imageSize: SpeedySize. Image size, in pixels.borderSize: SpeedyVector2. Border size in both x and y axes. Defaults to zero, meaning that no clipping takes place.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"out" |
Keypoints | The clipped set of keypoints. |
Speedy.Keypoint.Mixer(name?: string): SpeedyPipelineNodeKeypointMixer
Mixes (merges) two sets of keypoints.
| Port name | Data type | Description |
|---|---|---|
"in0" |
Keypoints | A set of keypoints. |
"in1" |
Keypoints | Another set of keypoints. |
"out" |
Keypoints | The union of the two input sets. |
Speedy.Keypoint.Buffer(name?: string): SpeedyPipelineNodeKeypointBuffer
A keypoint buffer outputs at time t the keypoints received at time t-1.
frozen: boolean. A frozen buffer discards the input, effectively increasing the buffering time. Defaults tofalse.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints at time t. |
"out" |
Keypoints | The set of keypoints received at time t-1. |
Speedy.Keypoint.Multiplexer(name?: string): SpeedyPipelineNodeKeypointMultiplexer
A keypoint multiplexer receives two sets of keypoints as input and outputs one of the them.
port: number. Which input set of keypoints should be redirected to the output:0or1? Defaults to0.
| Port name | Data type | Description |
|---|---|---|
"in0" |
Image | First set of keypoints. |
"in1" |
Image | Second set of keypoints. |
"out" |
Image | Either the first or the second set of keypoints, depending on the value of port. |
Speedy.Keypoint.Transformer(name?: string): SpeedyPipelineNodeKeypointTransformer
Applies a transformation matrix to a set of keypoints.
transform: SpeedyMatrix. A 3x3 homography matrix. Defaults to the identity matrix.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"out" |
Keypoints | A transformed set of keypoints. |
Speedy.Keypoint.SubpixelRefiner(name?: string): SpeedyPipelineNodeKeypointSubpixelRefiner
Refines the position of a set of keypoints down to the subpixel level.
Note 1: filter the image to reduce the noise before working at the subpixel level.
Note 2: if there are keypoints in multiple scales, make sure to provide a pyramid as input.
Note 3: the position of the keypoints is stored as fixed-point. This representation may introduce a loss of accuracy (~0.1 pixel). This is probably enough already, but if you need higher accuracy, ignore the output keypoints and work with the displacement vectors instead. These are encoded as floating-point. In addition, use the upsampling methods.
method: string. The method to be used to compute the subpixel displacement. See the table below.maxIterations: number. The maximum number of iterations used by methods"bicubic-upsample"and"bilinear-upsample". Defaults to 6.epsilon: number. The threshold used to determine when the subpixel displacement has reached convergence. Used with methods"bicubic-upsample"and"bilinear-upsample". Defaults to 0.1 pixel.
Table of methods:
| Method | Description |
|---|---|
"quadratic1d" |
Maximize a 1D parabola fit to a corner strength function. This is the default method. |
"taylor2d" |
Maximize a second-order 2D Taylor expansion of a corner strength function. Method "quadratic1d" seems to perform slightly better than this, but your mileage may vary. |
"bicubic-upsample" |
Iteratively upsample the image using bicubic interpolation in order to maximize a corner strength function. Repeat until convergence or until a maximum number of iterations is reached. |
"bilinear-upsample" |
Analogous to bicubic upsample, but this method uses bilinear interpolation instead. |
| Port name | Data type | Description |
|---|---|---|
"image" |
Image | An image or pyramid from which you extracted the keypoints. |
"keypoints" |
Keypoints | Input set of keypoints. |
"out" |
Keypoints | Subpixel-refined output set of keypoints. |
"displacements" |
Vector2 | Displacement vectors (output). |
Speedy.Keypoint.DistanceFilter(name?: string): SpeedyPipelineNodeKeypointDistanceFilter
Given a set of pairs of keypoints, discard all pairs whose distance is above a user-defined threshold. This is useful for implementing bidirectional optical-flow.
The pairs of keypoints are provided as two separate sets, "in" and "reference". Keypoints that are kept will have their data extracted from the "in" set.
threshold: number. Distance threshold, given in pixels.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"reference" |
Keypoints | A reference set of keypoints. |
"out" |
Keypoints | Filtered set of keypoints. |
Speedy.Keypoint.HammingDistanceFilter(name?: string): SpeedyPipelineNodeKeypointHammingDistanceFilter
Given a set of pairs of keypoints with descriptors, discard all pairs whose Hamming distance between their descriptors is above a user-defined threshold.
The pairs of keypoints are provided as two separate sets, "in" and "reference". Keypoints that are kept will have their data extracted from the "in" set.
threshold: number. Distance threshold, an integer.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"reference" |
Keypoints | A reference set of keypoints. |
"out" |
Keypoints | Filtered set of keypoints. |
Speedy.Keypoint.Shuffler(name?: string): SpeedyPipelineNodeKeypointShuffler
Shuffles the input keypoints, optionally clipping the output set.
maxKeypoints: number. Maximum number of keypoints of the output set. If unspecified, the number of keypoints of the output set will be the number of keypoints of the input set.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
"out" |
Keypoints | The input set of keypoints, shuffled and possibly clipped. |
The following nodes expect greyscale images as input. They output a set of keypoints.
Speedy.Keypoint.Detector.FAST(name?: string): SpeedyPipelineNodeFASTKeypointDetector
FAST keypoint detector. Speedy implements the FAST-9,16 variant of the algorithm.
To use the multi-scale version of the algorithm, pass a pyramid as input, set the number of levels you want to scan and optionally set the scale factor. After scanning all levels and performing non-maximum suppression, the scale of the keypoints will be set by means of interpolation using the scale that maximizes a response measure and its adjacent scales.
threshold: number. An integer between0and255, inclusive. The larger the number, the "stronger" your keypoints will be. The smaller the number, the more keypoint you will get. Numbers between20and50are usually meaningful.levels: number. The number of pyramid levels you want to use. Defaults to1(i.e., no pyramid is used). When using a pyramid, a value such as7is a reasonable choice.scaleFactor: number. The scale factor between two consecutive levels of the pyramid. This is a value between1(exclusive) and2(inclusive). Defaults to the square root of two. This is applicable only when using a pyramid.capacity: number. The maximum number of keypoints that can be detected by this node. The default capacity is2048. It can be no larger than8192.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Greyscale image or pyramid. |
"out" |
Keypoints | Detected keypoints. |
Speedy.Keypoint.Detector.Harris(name?: string): SpeedyPipelineNodeHarrisKeypointDetector
Harris corner detector. Speedy implements the Shi-Tomasi corner response for best results.
To use the multi-scale version of the algorithm, pass a pyramid as input, set the number of levels you want to scan and optionally set the scale factor. After scanning all levels and performing non-maximum suppression, the scale of the keypoints will be set by means of interpolation using the scale that maximizes a response measure and its adjacent scales.
quality: number. A value between0and1representing the minimum "quality" of the returned keypoints. Speedy will discard any keypoint whose score is lower than the specified percentage of the maximum keypoint score found in the image. A typical value for this parameter is0.10(10%).levels: number. The number of pyramid levels you want to use. Defaults to1(i.e., no pyramid is used). When using a pyramid, a value such as7is a reasonable choice.scaleFactor: number. The scale factor between two consecutive levels of the pyramid. This is a value between1(exclusive) and2(inclusive). Defaults to the square root of two. This is applicable only when using a pyramid.capacity: number. The maximum number of keypoints that can be detected by this node. The default capacity is2048. It can be no larger than8192.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | Greyscale image or pyramid. |
"out" |
Keypoints | Detected keypoints. |
Speedy.Keypoint.Descriptor.ORB(name?: string): SpeedyPipelineNodeORBKeypointDescriptor
ORB descriptors. In order to improve robustness to noise, apply a Gaussian filter to the image before computing the descriptors.
| Port name | Data type | Description |
|---|---|---|
"image" |
Image | Input image. Must be greyscale. |
"keypoints" |
Keypoints | Input keypoints. |
"out" |
Keypoints | Keypoints with descriptors. |
/*
This is our pipeline:
Image ---> Convert to ---> Image ------> FAST corner -----> Keypoint ---> ORB ----------> Keypoint
Source greyscale Pyramid detector Clipper descriptors Sink
| ^
| |
+-------------------------> Gaussian ---------------------------+
Blur
*/
const pipeline = Speedy.Pipeline();
const source = Speedy.Image.Source();
const greyscale = Speedy.Filter.Greyscale();
const pyramid = Speedy.Image.Pyramid();
const fast = Speedy.Keypoint.Detector.FAST();
const blur = Speedy.Filter.GaussianBlur();
const clipper = Speedy.Keypoint.Clipper();
const descriptor = Speedy.Keypoint.Descriptor.ORB();
const sink = Speedy.Keypoint.Sink();
source.media = media;
blur.kernelSize = Speedy.Size(9, 9);
blur.sigma = Speedy.Vector2(2, 2);
fast.threshold = 50;
fast.levels = 8; // pyramid levels
fast.scaleFactor = 1.19; // approx. 2^0.25
clipper.size = 800; // up to how many features?
source.output().connectTo(greyscale.input());
greyscale.output().connectTo(pyramid.input());
pyramid.output().connectTo(fast.input());
fast.output().connectTo(clipper.input());
clipper.output().connectTo(descriptor.input('keypoints'));
greyscale.output().connectTo(blur.input());
blur.output().connectTo(descriptor.input('image'));
descriptor.output().connectTo(sink.input());
pipeline.init(source, greyscale, pyramid, blur, fast, clipper, descriptor, sink);Keypoint tracking is the process of tracking keypoints across a sequence of images. It allows you to get a sense of how keypoints are moving in time - i.e., how fast they are moving and where they are going.
Speedy uses sparse optical-flow algorithms to track keypoints in a video. Applications of optical-flow are numerous: you may get a sense of how objects are moving in a scene, estimate how the camera itself is moving, detect a transition in a film (a cut between two shots), and so on.
Speedy.Keypoint.SinkOfTrackedKeypoints(name?: string): SpeedyPipelineNodeTrackedKeypointSink
Creates a sink of tracked keypoints using the specified name. If the name is not specified, Speedy will call this node "keypoints". An array of SpeedyTrackedKeypoint objects will be exported from the pipeline.
See also: SpeedyTrackedKeypoint.
The same as SpeedyPipelineNodeKeypointSink.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints to be exported from the pipeline. |
"flow" |
Vector2 | A set of displacement vectors associated with each keypoint. |
Speedy.Keypoint.Tracker.LK(name?: string): SpeedyPipelineNodeLKKeypointTracker
Pyramid-based LK optical-flow.
windowSize: SpeedySize. The size of the window to be used by the feature tracker. The algorithm will read neighbor pixels to determine the motion of a keypoint. You must specify a square window. Typical sizes include: 7x7, 11x11, 15x15 (use positive odd integers). Defaults to 11x11.levels: number. Specifies how many pyramid levels will be used in the computation. The more levels you use, the faster the motions you can capture. Defaults to3.discardThreshold: number. A threshold used to discard keypoints that are not "good" candidates for tracking. The higher the value, the more keypoints will be discarded. Defaults to0.0001.numberOfIterations: number. Maximum number of iterations for computing the local optical-flow on each level of the pyramid. Defaults to30.epsilon: number. An accuracy threshold used to stop the computation of the local optical-flow of any level of the pyramid. The local optical-flow is computed iteratively and in small increments. If the length of an increment is too small, we discard it. This property defaults to0.01.
| Port name | Data type | Description |
|---|---|---|
"previousImage" |
Image | Input image at time t-1. Must be greyscale. |
"nextImage" |
Image | Input image at time t. Must be greyscale. |
"previousKeypoints" |
Keypoints | Input keypoints at time t-1. |
"out" |
Keypoints | Output keypoints at time t. |
"flow" |
Vector2 | Flow vectors (output) at time t. |
Note: you need to provide pyramids as input if levels > 1.
Keypoint matching is the process of matching keypoints based on their descriptors. A distance metric is established in descriptor space. Two keypoints are said to be "matched" if the distance between their respective descriptors is minimized according to some criteria. Since Speedy uses binary descriptors, in practice we use the Hamming distance, i.e., the number of differing bits in two descriptors of same size.
Keypoint matching is useful for object recognition, object tracking, rectification of images, and more.
Speedy.Keypoint.SinkOfMatchedKeypoints(name?: string): SpeedyPipelineNodeMatchedKeypointSink
Create a sink of matched keypoints using the specified name. If the name is not specified, Speedy will call this node "keypoints". An array of SpeedyMatchedKeypoint objects will be exported from the pipeline.
See also: SpeedyMatchedKeypoint.
The same as SpeedyPipelineNodeKeypointSink.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints to be exported from the pipeline. |
"matches" |
KeypointMatches | A set of keypoint matches associated with each keypoint. |
Speedy.Keypoint.Matcher.BFKNN(name?: string): SpeedyPipelineNodeBruteForceKNNKeypointMatcher
Brute-force k-nearest neighbors keypoint matcher.
k: number. The desired number of matches per keypoint. Defaults to1(i.e., it will get you only the best match for each keypoint). Setting it to two gets you the first and the second best matches, and so on.
| Port name | Data type | Description |
|---|---|---|
"keypoints" |
Keypoints | The input keypoints that you want to match. |
"database" |
Keypoints | A collection of keypoints to be matched against. |
"out" |
KeypointMatches | The k best matches for all elements of "keypoints". |
Note: I suggest using brute-force to match two sets containing no more than a few hundreds of keypoints. Your mileage may vary. If you need to match thousands of keypoints or more, consider using an approximate matcher.
Note 2: make sure that you use as input two sets of keypoints with the same type of descriptors.
Speedy.Keypoint.Matcher.LSHKNN(name?: string): SpeedyPipelineNodeLSHKNNKeypointMatcher
Fast approximate k-nearest neighbors keypoint matcher based on my own GPU-based variant of Locality Sensitive Hashing (LSH).
k: number. The desired number of matches per keypoint. Defaults to1.quality: string. The desired quality level for the search of the best matches. One of the following:"default","fastest"or"demanding". Changing this parameter impacts performance, and possibly the quality of the results.
| Port name | Data type | Description |
|---|---|---|
"keypoints" |
Keypoints | The input keypoints that you want to match. |
"lsh" |
LSHTables | LSH tables of the keypoints to be matched against (the "database"). |
"out" |
KeypointMatches | The k best matches (approximately) for all elements of "keypoints". |
Tip: the "default" quality is generally appropriate, but if you set it to "fastest", consider increasing the number of LSH tables (see below).
Speedy.Keypoint.Matcher.StaticLSHTables(name?: string): SpeedyPipelineNodeStaticLSHTables
Generate LSH tables based on a known, and potentially large, collection of keypoints. LSH tables can help you match up to hundreds of thousands of keypoints.
keypoints: SpeedyKeypoint[]. The known collection of keypoints to be used as a "database" for matching. Make sure that they have descriptors.numberOfTables: number. The number of LSH tables that you want to generate. Defaults to8. This parameter can be as low as4and as high as32. Increasing it may increase the quality of the results - at the expense of performance.hashSize: number. The size of a descriptor hash, in bits. Defaults to15. This parameter can be as low as10and as high as20. Increasing it will substantially increase VRAM usage.
| Port name | Data type | Description |
|---|---|---|
"out" |
LSHTables | LSH tables associated with the known collection of keypoints. |
Tip: Speedy generates logs based on the numerical parameters that you define and on the number of keypoints in your database. You may use these logs to help you tune the numerical parameters. That being said, the default parameters are generally good.
Portals let you create loops within a pipeline. They also let you transfer data between different pipelines.
A portal is defined by a set of nodes: a portal sink and one or more portal sources. The portal sink receives data from a node of a pipeline, which is then read by the portal source(s). The portal source(s) feed(s) one or more pipelines. The portal nodes may or may not belong to the same pipeline.
Speedy.Image.Portal.Source(name?: string): SpeedyPipelineNodeImagePortalSource
Create a source of an Image Portal.
source: SpeedyPipelineNodeImagePortalSink. A sink of an Image Portal.
| Port name | Data type | Description |
|---|---|---|
"out" |
Image | An image. |
Speedy.Image.Portal.Sink(name?: string): SpeedyPipelineNodeImagePortalSink
Create a sink of an Image Portal.
Note: pyramids can't travel through portals at this time.
| Port name | Data type | Description |
|---|---|---|
"in" |
Image | An image. |
Speedy.Keypoint.Portal.Source(name?: string): SpeedyPipelineNodeKeypointPortalSource
Create a source of a Keypoint Portal.
source: SpeedyPipelineNodeKeypointPortalSink. A sink of a Keypoint Portal.
| Port name | Data type | Description |
|---|---|---|
"out" |
Keypoints | A set of keypoints. |
Speedy.Keypoint.Portal.Sink(name?: string): SpeedyPipelineNodeKeypointPortalSink
Create a sink of a Keypoint Portal.
| Port name | Data type | Description |
|---|---|---|
"in" |
Keypoints | A set of keypoints. |
Matrix computations play a crucial role in computer vision applications. Speedy includes its own implementation of numerical linear algebra algorithms.
Matrix operations are specified using a fluent interface that has been crafted to be easy to use and to mirror how we write matrix algebra using pen-and-paper.
Since numerical algorithms may be computationally demanding, Speedy uses WebAssembly for extra performance. Most matrix-related routines are written in C language. Matrices are stored in column-major format. Typed Arrays are used for storage.
There are two basic classes you need to be aware of: SpeedyMatrix and SpeedyMatrixExpr. The latter represents a symbolic expression, whereas the former represents an actual matrix with data. A SpeedyMatrix is a SpeedyMatrixExpr. A SpeedyMatrixExpr may be evaluated to a SpeedyMatrix.
Speedy.Matrix(rows: number, columns: number, entries?: number[]): SpeedyMatrix
Speedy.Matrix(expr: SpeedyMatrixExpr): SpeedyMatrix
First form: create a new matrix with the specified size and entries.
Second form: synchronously evaluate a matrix expression and store the result in a new matrix.
rows: number. The number of rows of the matrix.columns: number, optional. The number of columns of the matrix. If not specified, it will be set torows(i.e., you'll get a square matrix).entries: number[], optional. The elements of the matrix in column-major format. The length of this array must berows * columns.expr: SpeedyMatrixExpr. The matrix expression to be evaluated.
A new SpeedyMatrix.
//
// We use the column-major format to specify
// the elements of the new matrix. For example,
// to create the 2x3 matrix (2 rows, 3 columns)
// below, we first specify the elements of the
// first column, then the elements of the second
// column, and finally the elements of the third
// column.
//
// M = [ 1 3 5 ]
// [ 2 4 6 ]
//
const mat = Speedy.Matrix(2, 3, [
1,
2,
3,
4,
5,
6
]);
// Alternatively, we may write the data in
// column-major format in a compact form:
const mat1 = Speedy.Matrix(2, 3, [
1, 2, // first column
3, 4, // second column
5, 6 // third column
]);
// Print the matrices to the console
console.log(mat.toString());
console.log(mat1.toString());Speedy.Matrix.Zeros(rows: number, columns?: number): SpeedyMatrix
Create a new matrix filled with zeros.
rows: number. The number of rows of the matrix.columns: number, optional. The number of columns of the matrix. If not specified, it will be set torows(square matrix).
A new rows x columns SpeedyMatrix filled with zeros.
// A 3x3 matrix filled with zeros
const zeros = Speedy.Matrix.Zeros(3);Speedy.Matrix.Ones(rows: number, columns?: number): SpeedyMatrix
Create a new matrix filled with ones.
rows: number. The number of rows of the matrix.columns: number, optional. The number of columns of the matrix. If not specified, it will be set torows(square matrix).
A new rows x columns SpeedyMatrix filled with ones.
Speedy.Matrix.Eye(rows: number, columns?: number): SpeedyMatrix
Create a new matrix with ones on the main diagonal and zeros elsewhere.
rows: number. The number of rows of the matrix.columns: number, optional. The number of columns of the matrix. If not specified, it will be set torows(identity matrix).
A new SpeedyMatrix with the specified configuration.
// A 3x3 identity matrix
const eye = Speedy.Matrix.Eye(3);SpeedyMatrixExpr.rows: number, read-only
The number of rows of the matrix expression.
SpeedyMatrixExpr.columns: number, read-only
The number of columns of the matrix expression.
SpeedyMatrixExpr.dtype: string, read-only
The constant "float32".
SpeedyMatrix.data: ArrayBufferView, read-only
Data storage.
SpeedyMatrix.step0: number, read-only
SpeedyMatrix.step1: number, read-only
Storage steps. The (i, j) entry of the matrix is stored at data[i * step0 + j * step1].
SpeedyMatrix.read(): number[]
Read the entries of the matrix.
An array containing the entries of the matrix in column-major format.
const mat = Speedy.Matrix(2, 2, [
1,
2,
3,
4
]);
const entries = mat.read();
console.log(entries); // [ 1, 2, 3, 4 ]SpeedyMatrix.at(row: number, column: number): number
Read a single entry of the matrix.
row: number. Index of the row of the desired element (0-based).column: number. Index of the column of the desired element (0-based).
The requested entry of the matrix, or a NaN if the entry is outside bounds.
const A = Speedy.Matrix(2, 2, [
1,
2,
3,
4
]);
const a00 = A.at(0, 0); // first row, first column
const a10 = A.at(1, 0); // second row, first column
const a01 = A.at(0, 1); // first row, second column
const a11 = A.at(1, 1); // second row, second column
console.log([ a00, a10, a01, a11 ]); // [ 1, 2, 3, 4 ]SpeedyMatrixExpr.toString(): string
Convert a matrix expression to a string. Entries will only be included if this expression is a SpeedyMatrix.
A string representation of the matrix expression.
SpeedyMatrix.setTo(expr: SpeedyMatrixExpr): SpeedyPromise<SpeedyMatrix>
Evaluate a matrix expression and store the result in this matrix.
expr: SpeedyMatrixExpr. A matrix expression.
A SpeedyPromise that resolves to this matrix after evaluating expr.
//
// Let's add two matrices:
//
// A = [ 1 3 ] B = [ 4 2 ]
// [ 2 4 ] [ 3 1 ]
//
// We'll set C to the sum A + B
//
const matA = Speedy.Matrix(2, 2, [
1, 2,
3, 4
]);
const matB = Speedy.Matrix(2, 2, [
4, 3,
2, 1
]);
// Set C = A + B
const matC = Speedy.Matrix.Zeros(2, 2);
await matC.setTo(matA.plus(matB));
//
// Print the result:
//
// C = [ 5 5 ]
// [ 5 5 ]
//
console.log(matC.toString());SpeedyMatrix.fill(value: number): SpeedyPromise<SpeedyMatrix>
Fill this matrix with a scalar.
value: number. Scalar value.
A SpeedyPromise that resolves to this matrix.
// Create a 5x5 matrix filled with twos
const twos = Speedy.Matrix.Zeros(5);
await twos.fill(2);Speedy provides synchronous writing methods for convenience.
Speedy.Matrix.ready(): SpeedyPromise<void>
This method lets you know that the matrix routines are initialized and ready to be used (the WebAssembly routines need to be loaded before usage). You should only use the synchronous writing methods when the matrix routines are ready.
A SpeedyPromise that resolves immediately if the matrix routines are already initialized, or as soon as they are initialized.
SpeedyMatrix.setToSync(expr: SpeedyMatrixExpr): SpeedyMatrix
Synchronously evaluate a matrix expression and store the result in this matrix.
expr: SpeedyMatrixExpr. A matrix expression.
Returns this matrix after setting it to the result of expr.
Speedy.Matrix.ready().then(() => {
const mat = Speedy.Matrix.Eye(3); // I := identity matrix
const pot = 4; // power-of-two
for(let i = 0; i < pot; i++)
mat.setToSync(mat.plus(mat)); // mat := mat + mat
console.log(mat.toString()); // mat will be (2^pot) * I
});SpeedyMatrix.fillSync(value: number): SpeedyMatrix
Synchronously fill this matrix with a scalar.
value: number. Scalar value.
Returns this matrix after filling it with the provided value.
Speedy lets you work with blocks of matrices. This is a very handy feature! Blocks share memory with the originating matrices. If you modify the entries of a block of a matrix M, you'll modify the corresponding entries of M. Columns and rows are examples of blocks.
SpeedyMatrix.block(firstRow: number, lastRow: number, firstColumn: number, lastColumn: number): SpeedyMatrix
Extract a lastRow - firstRow + 1 x lastColumn - firstColumn + 1 block from the matrix. All indices are 0-based. They are all inclusive. The memory of the matrix is shared with the block.
firstRow: number. Index of the first row (0-based).lastRow: number. Index of the last row (0-based). UselastRow >= firstRow.firstColumn: number. Index of the first column (0-based).lastColumn: number. Index of the last column (0-based). UselastColumn >= firstColumn.
A new SpeedyMatrix representing the specified block.
//
// We'll create the following 4x4 matrix:
// (a dot represents a zero)
//
// [ 5 5 5 . ]
// [ 5 5 5 . ]
// [ 5 5 5 . ]
// [ . . . . ]
//
const mat = Speedy.Matrix.Zeros(4);
await mat.block(0, 2, 0, 2).fill(5);
console.log(mat.toString());SpeedyMatrix.column(index: number): SpeedyMatrix
Extract a column of the matrix.
index: number. Index of the column (0-based).
A new SpeedyMatrix representing the specified column.
const mat = Speedy.Matrix(2, 3, [
1,
2,
3,
4,
5,
6
]);
const firstColumn = mat.column(0); // [1, 2]^T
const secondColumn = mat.column(1); // [3, 4]^T
const thirdColumn = mat.column(2); // [5, 6]^T
console.log(firstColumn.toString());
console.log(secondColumn.toString());
console.log(thirdColumn.toString());SpeedyMatrix.row(index: number): SpeedyMatrix
Extract a row of the matrix.
index: number. Index of the row (0-based).
A new SpeedyMatrix representing the specified row.
//
// We'll create the following matrix:
// [ 0 0 0 0 ]
// [ 1 1 1 1 ]
// [ 2 2 2 2 ]
// [ 0 0 0 0 ]
//
const mat = Speedy.Matrix.Zeros(4);
await mat.row(1).fill(1);
await mat.row(2).fill(2);
console.log(mat.toString());SpeedyMatrix.diagonal(): SpeedyMatrix
Extract the main diagonal of this matrix as a column vector.
A new SpeedyMatrix representing the main diagonal of this matrix.
//
// We'll create the following matrix:
// (a dot represents a zero)
//
// [ 5 . . . . ]
// [ . 5 . . . ]
// [ . . 5 . . ]
// [ . . . . . ]
// [ . . . . . ]
//
const mat = Speedy.Matrix.Zeros(5); // create a 5x5 matrix filled with zeros
const submat = mat.block(0, 2, 0, 2); // extract 3x3 submatrix at the "top-left"
const diag = submat.diagonal(); // extract the diagonal of the submatrix
await diag.fill(5); // fill the diagonal of the submatrix with a constant
console.log(mat.toString()); // print the entire matrix
// Alternatively, we may use this compact form:
await mat.block(0, 2, 0, 2).diagonal().fill(5);SpeedyMatrixExpr.transpose(): SpeedyMatrixExpr
Transpose this matrix expression.
A SpeedyMatrixExpr representing the tranpose of this matrix expression.
// Create a 2x3 matrix
const mat = Speedy.Matrix(2, 3, [
1, 2, // first column
3, 4, // second column
5, 6 // third column
]);
// We'll store the transpose of mat in matT
const matT = Speedy.Matrix.Zeros(mat.columns, mat.rows);
await matT.setTo(mat.transpose());
// Print the matrix and its transpose
console.log(mat.toString());
console.log(matT.toString());SpeedyMatrixExpr.plus(expr: SpeedyMatrixExpr): SpeedyMatrixExpr
Compute the sum between this matrix expression and expr. Both expressions must have the same shape.
expr: SpeedyMatrixExpr. Another matrix expression.
A SpeedyMatrixExpr representing the sum between this matrix expression and expr.
const matA = Speedy.Matrix(3, 3, [
1, 2, 3,
4, 5, 6,
7, 8, 9
]);
const ones = Speedy.Matrix.Ones(3);
// set B = A + 1
const matB = Speedy.Matrix.Zeros(3);
await matB.setTo(matA.plus(ones));SpeedyMatrixExpr.minus(expr: SpeedyMatrixExpr): SpeedyMatrixExpr
Compute the difference between this matrix expression and expr. Both expressions must have the same shape.
expr: SpeedyMatrixExpr. Another matrix expression.
A SpeedyMatrixExpr representing the difference between this matrix expression and expr.
SpeedyMatrixExpr.times(expr: SpeedyMatrixExpr): SpeedyMatrixExpr
SpeedyMatrixExpr.times(scalar: number): SpeedyMatrixExpr
Matrix multiplication.
In the first form, compute the matrix multiplication between this matrix expression and expr. The shape of expr must be compatible with the shape of this matrix expression.
In the second form, multiply this matrix expression by a scalar.
expr: SpeedyMatrixExpr. Matrix expression.scalar: number. A number.
A SpeedyMatrixExpr representing the result of the multiplication.
const col = Speedy.Matrix(3, 1, [0, 5, 2]);
const row = Speedy.Matrix(1, 3, [1, 2, 3]);
const dot = row.times(col); // 1x1 matrix expression: inner product
const out = col.times(row); // 3x3 matrix expression: outer product
const len = col.transpose().times(col); // 1x1 matrix expression: squared length of col
const mat = Speedy.Matrix.Zeros(1);
await mat.setTo(len); // evaluate len
console.log(mat.read()); // 29 = 0*0 + 5*5 + 2*2SpeedyMatrixExpr.compMult(expr: SpeedyMatrixExpr): SpeedyMatrixExpr
Compute the component-wise multiplication between this matrix expression and expr. Both matrices must have the same shape.
expr: SpeedyMatrixExpr. Matrix expression.
A SpeedyMatrixExpr representing the component-wise multiplication.
SpeedyMatrixExpr.inverse(): SpeedyMatrixExpr
Compute the inverse of this matrix expression. Make sure it's square.
A SpeedyMatrixExpr representing the inverse of this matrix expression.
SpeedyMatrixExpr.ldiv(expr: SpeedyMatrixExpr): SpeedyMatrixExpr
Left division this \ expr. This is equivalent to solving a system of linear equations Ax = b, where A is this and b is expr (in a least squares sense if A is not square). The number of rows of this must be greater or equal than its number of columns. expr must be a column vector.
expr: SpeedyMatrixExpr. Matrix expression.
A SpeedyMatrixExpr representing the left division.
Speedy.Matrix.solve(solution: SpeedyMatrix, A: SpeedyMatrix, b: SpeedyMatrix, options?: object): SpeedyPromise<SpeedyMatrix>
Solve a system of linear equations Ax = b for x, the solution, where A is a n x n square matrix, b is a n x 1 column vector and solution is a n x 1 column vector of unknowns. n is the number of equations and the number of unknowns.
solution: SpeedyMatrix. The output column vector.A: SpeedyMatrix. A square matrix.b: SpeedyMatrix. A column vector.options: object, optional. Options to be passed to the solver. Available keys:method: string. One of the following:"qr". Defaults to"qr".
A SpeedyPromise that resolves to solution.
//
// We'll solve the following system of equations:
// y - z = 9
// y + z = 6
//
// Let's write it in matrix form:
// [ 1 -1 ] [ y ] = [ 9 ]
// [ 1 1 ] [ z ] [ 6 ]
//
// The code below solves Ax = b for x, where
// x = (y, z) is the column vector of unknowns.
//
const A = Speedy.Matrix(2, 2, [
1, 1, // first column
-1, 1 // second column
]);
const b = Speedy.Matrix(2, 1, [
9, 6 // column vector
]);
// Solve Ax = b for x
const solution = Speedy.Matrix.Zeros(2, 1);
await Speedy.Matrix.solve(solution, A, b);
// get the result
console.log(solution.read()); // [ 7.5, -1.5 ]Speedy.Matrix.ols(solution: SpeedyMatrix, A: SpeedyMatrix, b: SpeedyMatrix): SpeedyPromise<SpeedyMatrix>
Ordinary least squares.
Given an overdetermined system of linear equations Ax = b, where A is a m x n matrix, b is a m x 1 column vector and solution x is a n x 1 column vector of unknowns, find a solution x that minimizes the Euclidean norm of the residual b - Ax.
m is the number of equations and n is the number of unknowns. We require m >= n.
solution: SpeedyMatrix. The output column vector.A: SpeedyMatrix. A matrix.b: SpeedyMatrix. A column vector.options: object, optional. Options to be passed to the solver. Available keys:method: string. One of the following:"qr". Defaults to"qr".
A SpeedyPromise that resolves to solution.
Speedy.Matrix.qr(Q: SpeedyMatrix, R: SpeedyMatrix, A: SpeedyMatrix, options?: object): SpeedyPromise<void>
Compute a QR decomposition of a m x n matrix A using Householder reflectors. Q will be orthogonal and R will be upper-triangular. We require m >= n.
Q: SpeedyMatrix. Output matrix (m x n if reduced, m x m if full).R: SpeedyMatrix. Output matrix (n x n if reduced, m x n if full).A: SpeedyMatrix. The matrix to be decomposed.options: object, optional. A configuration object that accepts the following keys:mode: string. Either"full"or"reduced". Defaults to"reduced".
Returns a SpeedyPromise that resolves as soon as the computation is complete.
// We'll find a QR decomposition of this matrix
const A = Speedy.Matrix(3, 3, [
0, 1, 0, // first column
1, 1, 0, // second column
1, 2, 3, // third column
]);
// Compute a QR decomposition of A
const Q = Speedy.Matrix.Zeros(3, 3);
const R = Speedy.Matrix.Zeros(3, 3);
await Speedy.Matrix.qr(Q, R, A);
// Print the result
console.log(Q.toString());
console.log(R.toString());
// Check the answer (A = QR)
const QR = await Speedy.Matrix.Zeros(Q.rows, R.columns).setTo(Q.times(R));
console.log(QR.toString());Speedy.Matrix.applyPerspectiveTransform(dest: SpeedyMatrix, src: SpeedyMatrix, transform: SpeedyMatrix): SpeedyPromise<SpeedyMatrix>
Apply a perspective transform to a set of 2D points described by src and store the results in dest.
dest: SpeedyMatrix. A 2 x n output matrix.src: SpeedyMatrix. A 2 x n matrix encoding a set of n points, one per column.transform: SpeedyMatrix. A 3x3 homography matrix.
A SpeedyPromise that resolves to dest.
const transform = Speedy.Matrix(3, 3, [
3, 0, 0, // first column
0, 2, 0, // second column
2, 1, 1, // third column
]);
const src = Speedy.Matrix(2, 4, [
0, 0,
1, 0,
1, 1,
0, 1,
]);
const dest = Speedy.Matrix.Zeros(src.rows, src.columns);
await Speedy.Matrix.applyPerspectiveTransform(dest, src, transform);
console.log(dest.toString());
//
// Result:
// [ 2 5 5 2 ]
// [ 1 1 3 3 ]
//Speedy.Matrix.perspective(homography: SpeedyMatrix, src: SpeedyMatrix, dest: SpeedyMatrix): SpeedyPromise<SpeedyMatrix>
Compute a homography matrix using four correspondences of points.
homography: SpeedyMatrix. A 3x3 output matrix.src: SpeedyMatrix. A 2x4 matrix with the coordinates of four points (one per column) representing the corners of the source space.dest: SpeedyMatrix. A 2x4 matrix with the coordinates of four points (one per column) representing the corners of the destination space.
A SpeedyPromise that resolves to homography.
const src = Speedy.Matrix(2, 4, [
0, 0, // first point
1, 0, // second point
1, 1, // third point
0, 1, // fourth point
]);
const dest = Speedy.Matrix(2, 4, [
0, 0,
3, 0,
3, 2,
0, 2,
]);
const homography = Speedy.Matrix.Zeros(3, 3);
await Speedy.Matrix.perspective(homography, src, dest);
console.log(homography.toString());Speedy.Matrix.findHomography(homography: SpeedyMatrix, src: SpeedyMatrix, dest: SpeedyMatrix, options?: object): SpeedyPromise<SpeedyMatrix>
Compute a homography matrix using a set of n >= 4 correspondences of points, possibly with noise.
homography: SpeedyMatrix. A 3x3 output matrix.src: SpeedyMatrix. A 2 x n matrix with the coordinates of n points (one per column) representing the corners of the source space.dest: SpeedyMatrix. A 2 x n matrix with the coordinates of n points (one per column) representing the corners of the destination space.options: object, optional. A configuration object.method: string. The method to be employed to compute the homography (see the table of methods below).
Table of methods:
| Method | Description |
|---|---|
"default" |
Normalized Direct Linear Transform (DLT). All points will be used to estimate the homography. Use this method if your data set is not polluted with outliers. |
"pransac" |
PRANSAC is a variant of RANSAC with bounded runtime that is designed for real-time tasks. It is able to reject outliers in the data set. |
Table of parameters:
| Parameter | Supported methods | Description |
|---|---|---|
reprojectionError: number |
"pransac" |
A threshold, measured in pixels, that lets Speedy decide if a data point is an inlier or an outlier for a given model. A data point is an inlier for a given model if the model maps its src coordinates near its dest coordinates (i.e., if the Euclidean distance is not greater than the threshold). A data point is an outlier if it's not an inlier. Defaults to 3 pixels. |
mask: SpeedyMatrix |
"pransac" |
An optional output matrix of shape 1 x n. Its i-th entry will be set to 1 if the i-th data point is an inlier for the best model found by the method, or 0 if it's an outlier. |
numberOfHypotheses: number |
"pransac" |
A positive integer specifying the number of models that will be generated and tested. The best model found by the method will be refined and then returned. If your inlier ratio is "high", this parameter can be set to a "low" number, making the algorithm run even faster. Defaults to 500. |
bundleSize: number |
"pransac" |
A positive integer specifying the number of data points to be tested against all viable models before the set of viable models gets cut in half, over and over again. Defaults to 100. |
A SpeedyPromise that resolves to homography.
//
// Map random points
// from [0,100] x [0,100]
// to [200,600] x [200,600]
//
const numPoints = 50;
const noiseLevel = 2;
const transform = x => 4*x + 200; // simulated model
const randCoord = () => 100 * Math.random(); // in [0, 100)
const randNoise = () => (Math.random() - 0.5) * noiseLevel;
const srcCoords = new Array(numPoints * 2).fill(0).map(() => randCoord());
const dstCoords = srcCoords.map(x => transform(x) + randNoise());
const src = Speedy.Matrix(2, numPoints, srcCoords);
const dst = Speedy.Matrix(2, numPoints, dstCoords);
const mask = Speedy.Matrix.Zeros(1, numPoints);
const homography = Speedy.Matrix.Zeros(3, 3);
await Speedy.Matrix.findHomography(homography, src, dst, {
method: "pransac",
mask: mask,
reprojectionError: 1
});
console.log('homography:', homography.toString());
console.log('mask:', mask.toString());
// Now let's test the homography using a few test points.
// The points need to be mapped in line with our simulated model (see above)
const tstCoords = Speedy.Matrix(2, 5, [
0, 0,
100, 0,
100, 100,
0, 100,
50, 50,
]);
const chkCoords = Speedy.Matrix.Zeros(2, 5);
await Speedy.Matrix.applyPerspectiveTransform(chkCoords, tstCoords, homography);
console.log(chkCoords.toString());Speedy.Matrix.applyAffineTransform(dest: SpeedyMatrix, src: SpeedyMatrix, transform: SpeedyMatrix): SpeedyPromise<SpeedyMatrix>
Apply an affine transform to a set of 2D points described by src and store the results in dest.
dest: SpeedyMatrix. A 2 x n output matrix.src: SpeedyMatrix. A 2 x n matrix encoding a set of n points, one per column.transform: SpeedyMatrix. A 2x3 affine transformation matrix.
A SpeedyPromise that resolves to dest.
const transform = Speedy.Matrix(2, 3, [
3, 0, // first column
0, 2, // second column
2, 1, // third column
]);
const src = Speedy.Matrix(2, 4, [
0, 0,
1, 0,
1, 1,
0, 1,
]);
const dest = Speedy.Matrix.Zeros(src.rows, src.columns);
await Speedy.Matrix.applyAffineTransform(dest, src, transform);
console.log(dest.toString());
//
// Result:
// [ 2 5 5 2 ]
// [ 1 1 3 3 ]
//Speedy.Matrix.affine(transform: SpeedyMatrix, src: SpeedyMatrix, dest: SpeedyMatrix): SpeedyPromise<SpeedyMatrix>
Compute an affine transform using three correspondences of points.
transform: SpeedyMatrix. A 2x3 output matrix.src: SpeedyMatrix. A 2x3 matrix with the coordinates of three points (one per column) representing the corners of the source space.dest: SpeedyMatrix. A 2x3 matrix with the coordinates of three points (one per column) representing the corners of the destination space.
A SpeedyPromise that resolves to transform.
const src = Speedy.Matrix(2, 3, [
0, 0, // first point
1, 0, // second point
1, 1, // third point
]);
const dest = Speedy.Matrix(2, 3, [
0, 0,
3, 0,
3, 2,
]);
const transform = Speedy.Matrix.Zeros(2, 3);
await Speedy.Matrix.affine(transform, src, dest);
console.log(transform.toString());Speedy.Matrix.findAffineTransform(transform: SpeedyMatrix, src: SpeedyMatrix, dest: SpeedyMatrix, options?: object): SpeedyPromise<SpeedyMatrix>
Compute an affine transform using a set of n >= 3 correspondences of points, possibly with noise.
transform: SpeedyMatrix. A 2x3 output matrix.src: SpeedyMatrix. A 2 x n matrix with the coordinates of n points (one per column) representing the corners of the source space.dest: SpeedyMatrix. A 2 x n matrix with the coordinates of n points (one per column) representing the corners of the destination space.options: object, optional. A configuration object.method: string. The method to be employed to compute the affine transform (see the table of methods below).
Table of methods:
- the same as Speedy.Matrix.findHomography().
Table of parameters:
- the same as Speedy.Matrix.findHomography().
A SpeedyPromise that resolves to transform.
Speedy.Vector2(x: number, y: number): SpeedyVector2
Creates a new 2D vector with the given coordinates.
x: number. The x-coordinate of the vector.y: number. The y-coordinate of the vector.
A new SpeedyVector2 instance.
const zero = Speedy.Vector2(0, 0);Speedy.Vector2.Sink(name?: string): SpeedyPipelineNodeVector2Sink
Creates a sink of 2D vectors using the specified name. If the name is not specified, Speedy will call this node "vec2". An array of SpeedyVector2 objects will be exported from the pipeline.
turbo: boolean. Accelerate GPU-CPU transfers. You'll get the data from the previous frame. Defaults tofalse.
| Port name | Data type | Description |
|---|---|---|
"in" |
Vector2 | A set of 2D vectors to be exported from the pipeline. |
SpeedyVector2.x: number
The x-coordinate of the vector.
SpeedyVector2.y: number
The y-coordinate of the vector.
SpeedyVector2.plus(offset: SpeedyVector2): SpeedyVector2
Vector addition.
A new vector corresponding to this + offset.
SpeedyVector2.minus(offset: SpeedyVector2): SpeedyVector2
Vector subtraction.
A new vector corresponding to this - offset.
SpeedyVector2.times(scalar: number): SpeedyVector2
Multiply a vector by a scalar.
A new vector corresponding to this * scalar.
SpeedyVector2.length(): number
Computes the length of the vector (Euclidean norm).
The length of the vector.
const v = Speedy.Vector2(3, 4);
console.log('Coordinates', v.x, v.y);
console.log('Length', v.length()); // 5SpeedyVector2.normalized(): SpeedyVector2
Returns a normalized version of this vector.
A new vector with the same direction as the original one and with length equal to one.
SpeedyVector2.dot(v: SpeedyVector2): number
Dot product.
v: SpeedyVector2. A vector.
The dot product between the two vectors.
SpeedyVector2.distanceTo(v: SpeedyVector2): number
Computes the distance between two vectors.
v: SpeedyVector2. A vector.
The Euclidean distance between the two vectors.
const u = Speedy.Vector2(1, 0);
const v = Speedy.Vector2(5, 0);
console.log(u.distanceTo(v)); // 4SpeedyVector2.toString(): string
Get a string representation of the vector.
A string representation of the vector.
SpeedyVector2.equals(v: SpeedyVector2): boolean
Equality comparison.
Returns true if the coordinates of this are equal to the coordinates of v, or false otherwise.
Speedy.Point2(x: number, y: number): SpeedyPoint2
Creates a new 2D point with the given coordinates.
x: number. The x-coordinate of the point.y: number. The y-coordinate of the point.
A new SpeedyPoint2 instance.
const p = Speedy.Point2(5, 10);SpeedyPoint2.x: number
The x-coordinate of the point.
SpeedyPoint2.y: number
The y-coordinate of the point.
SpeedyPoint2.plus(v: SpeedyVector2): SpeedyPoint2
Adds a vector to this point.
v: SpeedyVector2. A 2D vector.
A new SpeedyPoint2 instance corresponding to this point translated by v.
SpeedyPoint2.minus(p: SpeedyPoint2): SpeedyVector2
Subtracts point p from this.
p: SpeedyPoint2. A 2D point.
A new SpeedyVector2 instance such that p plus that vector equals this point.
SpeedyPoint2.equals(p: SpeedyPoint2): boolean
Equality comparison.
Returns true if the coordinates of this are equal to the coordinates of p, or false otherwise.
Speedy.Size(width: number, height: number): SpeedySize
Creates a new object that represents the size of a rectangle.
width: number. A non-negative number.height: number. A non-negative number.
A new SpeedySize instance.
const size = Speedy.Size(640, 360);SpeedySize.width: number
Width property.
SpeedySize.height: number
Height property.
SpeedySize.equals(anotherSize: SpeedySize): boolean
Checks if two size objects have the same dimensions.
Returns true if the dimensions of this and anotherSize are equal.
SpeedySize.toString(): string
Convert to string.
A string representation of the object.
Speedy includes its own implementation of Promises, called SpeedyPromises. SpeedyPromises can interoperate with standard ES6 Promises and are based on the Promises/A+ specification. The main difference between SpeedyPromises and standard ES6 Promises is that, under certain circunstances, SpeedyPromises can be made to run faster than ES6 Promises.
SpeedyPromises are specially beneficial when you have a chain of them. When (and if) their "turbocharged" mode is invoked, they will adopt a special (non-standard) behavior and skip the microtask queue when settling promises in a chain. This will save you a few milliseconds. While "a few milliseconds" doesn't sound much in terms of standard web development, for a real-time library such as Speedy it means a lot. Simply put, we're squeezing out performance. SpeedyPromises are used internally by the library.
Speedy.Promise: Function
Used to create a new SpeedyPromise object.
let promise = new Speedy.Promise((resolve, reject) => {
setTimeout(resolve, 2000);
});
promise.then(() => {
console.log(`The SpeedyPromise is now fulfilled.`);
}).catch(() => {
console.log(`The SpeedyPromise is now rejected.`);
}).finally(() => {
console.log(`The SpeedyPromise is now settled.`);
});Global settings.
Speedy.Settings.powerPreference: "default" | "low-power" | "high-performance"
Experimental. The desired power preference for the WebGL context. This option should be set before creating any pipelines. The browser uses this setting as a hint to balance rendering performance and battery life (especially on mobile devices).
Speedy.Settings.gpuPollingMode: "raf" | "asap"
Experimental. GPU polling mode. "asap" has slightly better performance than "raf", at the cost of higher CPU usage.
Speedy.Settings.logging: "default" | "none" | "diagnostic"
Speedy prints messages to the browser console according to the logging mode. The table below summarizes the available modes:
| Mode | Description |
|---|---|
"default" |
Shows warnings and some informative messages. |
"none" |
Hides all messages. |
"diagnostic" |
Enables the diagnostic mode, which lets you inspect the raw data traveling throughout the nodes of a pipeline. This has performance implications and is not meant to be used in production code. |
Extra utilities.
Speedy.version: string, read-only
The version of the library.
Speedy.fps: number, read-only
Speedy includes a frames per second (FPS) counter for testing purposes. It will be created as soon as you access it.
console.log(Speedy.fps);Speedy.isSupported(): boolean
Checks if Speedy is supported in this machine & browser.
Returns a boolean telling whether or not Speedy is supported in the client environment.
if(!Speedy.isSupported())
alert('This application is not supported in this browser. Please use a different browser.');Utilities to query information about the graphics driver. This information may or may not be available, depending on the privacy settings of the web browser. In addition, it may be more or less accurate in different browsers.
Speedy.Platform.renderer: string, read-only
Renderer string of the graphics driver.
Speedy.Platform.vendor: string, read-only
Vendor string of the graphics driver.