missing API against TH #70

soumith · 2014-11-06T05:50:20Z

clementfarabet · 2014-11-08T17:26:24Z

Hey @soumith , so running cutorch.test() currently reports errors on cumsum and cumprod (missing). That's on purpose?

soumith · 2014-11-08T17:27:42Z

Yes they are yet to be implemented on cutorch. I'll get to that next week. They can be done with a thrust:: scan

clementfarabet · 2014-11-08T18:13:41Z

Ok cool.

On Sat, Nov 8, 2014 at 12:27 PM, Soumith Chintala [email protected]
wrote:

Yes they are yet to be implemented on cutorch. I'll get to that next week.
They can be done with a thrust:: scan

—
Reply to this email directly or view it on GitHub
#70 (comment).

dominikgrewe · 2015-01-03T13:11:54Z

Does anyone have an implementation of std and var yet? Otherwise I'll have a go next week.
Afaict, we can't use THCudaTensor_reduceDim so we'll need something custom (but with a similar structure as reduceDim).

soumith · 2015-01-03T18:45:14Z

no, do not have it. this has been long overdue, but we should co-ordinate these, let me email you guys.

dominikgrewe · 2015-01-20T13:46:26Z

Any progress on cumsum and cumprod?

soumith · 2015-01-20T20:24:07Z

have not started them yet!

dominikgrewe · 2015-01-20T22:08:17Z

Okay, I might have a go soon then, because we'd like to use it. Just waiting for the THCState PR to be merged.

soumith · 2015-01-21T17:01:56Z

I will merge the THCState PR on Friday. All the patches have been prepared except for fbcunn, working on that as well.

dominikgrewe · 2015-02-19T11:03:28Z

@soumith For reductions along a single dimension, we still have the restriction that the tensor must not have more than 4 dimensions. Did you say you're working on a fix for that? If so, what's the progress on it?
If you're not working on it, we could easily change the code to what we do for std and var, where there's no restriction on dimensionality at all.

soumith · 2015-02-19T16:17:11Z

@wickedfoo already has PR for that internally. Its all implemented. I'm on
vacay till 26th, so I will sync those changes at the end of this month.
On Feb 19, 2015 4:34 PM, "Dominik Grewe" [email protected] wrote:

@soumith https://github.com/soumith For reductions along a single
dimension, we still have the restriction that the tensor must not have more
than 4 dimensions. Did you say you're working on a fix for that? If so,
what's the progress on it?
If you're not working on it, we could easily change the code to what we do
for std and var, where there's no restriction on dimensionality at all.

Reply to this email directly or view it on GitHub
#70 (comment).

soumith · 2015-02-19T16:19:37Z

His PR is for apply (and apply2) along an arbitrary dimension. Not reductions. It generalizes the copy kernels and changes all the tensor math to use these apply kernels where appropriate instead of make contiguous + thrust

soumith · 2015-02-19T16:20:17Z

That's the status on that. He did not work yet on arbitrary reductions. If you want to tackle that, go for it.

dominikgrewe · 2015-02-19T16:22:18Z

Thanks for the update. I'll have a go at the reductions kernel then.
Looking forward to the apply kernels!

wickedfoo · 2015-02-19T18:17:32Z

I'm on vacation too until next week but I don't think the generic apply
stuff got pushed yet, I think only the old version of the copy kernel. I
reimplemented all cutorch math (pointwise operators) in terms of it, so no
newContiguous calls are needed. I don't yet support reductions but it
shouldn't be too hard to add that in.

On Thursday, February 19, 2015, Dominik Grewe [email protected]
wrote:

Thanks for the update. I'll have a go at the reductions kernel then.
Looking forward to the apply kernels!

—
Reply to this email directly or view it on GitHub
#70 (comment).

dominikgrewe · 2015-03-02T19:29:39Z

If I remember correctly, you guys said you'd look into maskedFill etc, right? Any progress on that?

wickedfoo · 2015-03-02T22:50:49Z

for maskedFill etc. do you want the mask to be a float vector (because that's the only thing we have in cutorch at present), or do you want it to be 4 bytes packed into a float?

dominikgrewe · 2015-03-02T23:19:05Z

I guess float vectors make the most sense, because that's what logical functions (gt, ge etc) return.

wickedfoo · 2015-03-05T02:58:35Z

I have maskedFill/Copy/Select done, and sort() I have power-of-2 sizes at present (but on input with an arbitrary number of dimensions), so still working on that. maskedFill, maskedCopy and sort avoid newContiguous on the input, but maskedSelect I chickened out and just used two passes and temporary space with a Thrust prefix scan.

Re: "For reductions along a single dimension, we still have the restriction that the tensor must not have more than 4 dimensions. Did you say you're working on a fix for that? If so, what's the progress on it?" I have this fixed as well, took the copy kernel code and made a reduction kernel out of it, so no calls to newContiguous/copies etc. needed. Not a global reduction kernel (like a norm that reduces down to one point, but reduces along a dimension. sort() exploits similar code. I want to do the same shared memory optimization (so I can use coalesced reads) that you did if the reduction dimension is innermost/most contiguous though.

dominikgrewe · 2015-03-05T09:37:05Z

Cool, looking forward to that. Yes, using the shared memory approach for reductions along contiguous dimensions is vital.

dominikgrewe · 2015-03-05T14:13:24Z

When do you think you'll have a PR for maskedFill etc?

soumith · 2015-03-05T14:15:36Z

it's in review, and jeff is still working on revamping our code-base to the state argument based change.
Hopefully sometime next week.

And the cutorch TensorMath changes that remove most of the sync points on non-contiguous cases will also land at the same time.

jaderberg · 2015-03-12T12:15:26Z

Any progress on the masked* functions?

soumith · 2015-03-12T14:12:22Z

Theyre implemented. We are working on refactoring our code and syncing.with master, we will try to merge them this week.

soumith · 2015-03-13T14:24:49Z

An update:
Jeff has powered through our internal refactor, getting us back to parity with oss cutorch.
There are three PRs coming up.

masked* functions
sort
all of the math (where applicable) revamped to use our own pointwise and reduce kernels, so that non-contiguous tensors are no longer sync points.

It will either be EOD today or most likely Monday/Tuesday.

soumith · 2015-03-13T14:30:57Z

Looks like what's left is the small fish.

torch.diag, torch.eye, torch.trace can be handled with the same generic diagonal-apply kernel.
torch.randperm, linspace, logspace, range is a thrust::scan
logicalall, logicalany also apply kernel!?
tril and triu might be tricky

dominikgrewe · 2015-04-01T15:18:13Z

There are a number of linear algebra functions missing: symeig, eig, inverse etc. In Torch they seem to be implemented by wrapping Lapack. Could we do something similar for cutorch? There's MAGMA and CULA; does anyone have experience with these libraries?

soumith · 2015-04-01T15:22:27Z

MAGMA looks best, we built MAGMA internally and it looks reasonably good.

soumith · 2015-04-01T15:23:32Z

Also, on the CuDNN note, we can configure a header (like THGeneral.h.in ) if we find cudnn. Caffe has the cmake macros needed for finding cudnn already written: https://github.com/BVLC/caffe/blob/master/cmake/Cuda.cmake

for issue torch#70

dpfau · 2015-05-24T22:47:32Z

Hi guys. Noticed this thread when dealing with a script that needs diag, svd and eig on CudaTensors. I implemented diag myself in Lua using storage() and set(), but svd and eig are beyond my ken. What's the plan for that?

soumith · 2015-05-25T00:56:30Z

One of my colleagues @SamGross is working on it by interfacing the magma cuda library. It'll happen over the next month or so when he finishes it up and sends a PR.

wickedfoo · 2015-05-27T21:19:12Z

This is not on this list, but I'm in the process of implementing THCudaTensor_multinomial as well.

nicholas-leonard · 2015-05-28T15:16:01Z

@wickedfoo Awesome. Would love to see multinomial in cuda.

hughperkins · 2015-06-15T12:02:56Z

Just to confirm, scatter/gather arent implemented in cutorch, right?

dominikgrewe · 2015-06-15T12:42:54Z

That's right. I meant to do it, but haven't had the time yet, sorry.

hughperkins · 2015-06-23T00:52:50Z

For gather, which I suddenly realize could be useful for implementing ClassNLLCriterion.forward, without needing a custom kernel, I guess?, I suppose a simple naive first-cut could be:

use the isContiguous (toContiguous? asContiguous?) method to convert the tensor to contiguous format
simply assign one thread to each output location, and I dont think we need any local memory or anything right? So, just throw everything into warp-size blocks and it's basically done?

Does that sound about right? Anything else I should bear in mind if I write a naive gather along these lines? (I'll be targeting cltorch I confess, but it's quite easy to convert cutorch<->cltorch kernels I think?)

(Edit: what do you think is the most similar existing class/kernel to base this off? and/or thoughts on where to put this, ie filename(s)?)

hughperkins · 2015-06-23T10:11:49Z

One of my colleagues @SamGross is working on it by interfacing the magma cuda library. It'll happen over the next month or so when he finishes it up and sends a PR.

Magma looks cool. Has opencl version too it seems :-)

hughperkins · 2015-06-23T16:42:23Z

Here is a gather implementation. It's not very tested yet..

The kernel: https://github.com/hughperkins/cltorch/blob/master/lib/THCl/THClGather.cl
The driver: https://github.com/hughperkins/cltorch/blob/master/lib/THCl/THClGather.cpp
The lua wrapper: https://github.com/hughperkins/cltorch/blob/master/torch/generic/Tensor.c#L539

hughperkins · 2015-06-24T00:24:21Z

Shoe-horned the lua wrapper into TensorMath.lua: hughperkins/cltorch@0e469f4

hughperkins · 2015-06-24T10:50:06Z

Did scatter too, since seems like more of the same?

(Edit: and scatterFill, in same files)

abyravan · 2016-02-23T00:01:02Z

Is there any update on adding these functions? I'm in need of the cross product on cudatensors.

I can code it up if someone can point me towards the things that need to be done. All the layers I've written so far are on the lua side and I'm not sure how to make the connection between cuda and lua. Thanks!

soumith · 2016-02-28T01:09:01Z

@abyravan i could get to cross next week.

abyravan · 2016-02-28T01:17:12Z

That would be great. Thanks a lot! Is there any sort of a tutorial or howto on adding new functionality for tensors? Would be useful to have :)

soumith · 2016-02-28T01:19:00Z

You can look at existing PRs that are cross-linked in this thread.
Like:
#120
#96
#75

soumith · 2016-02-28T01:19:45Z

nonzero is being implemented by FB, should be out in a few days.

** NOTE on API changes and versioning ** Cutorch provides a CUDA backend for torch7. Cutorch provides the following: a new tensor type: torch.CudaTensor that acts like torch.FloatTensor, but all it's operations are on the GPU. Most of the tensor operations are supported by cutorch. There are a few missing ones, which are being implemented. The missing list can be found here: torch/cutorch#70 several other GPU tensor types, with limited functionality. Currently limited to copying/conversion, and several indexing and shaping operations. cutorch.* - Functions to set/get GPU, get device properties, memory usage, set/get low-level streams, set/get random number generator's seed, synchronization etc. They are described in more detail belo

pvtokmakov · 2017-06-26T16:35:34Z

Hi guys,

any hope on implementing the conv2 (or xcorr2 for that matter)?

ethanluoyc · 2017-08-15T21:25:25Z

Just adding a note that eye has been implemented here pytorch/pytorch#2395

sjain-stanford · 2017-12-22T16:56:58Z

Thanks for supporting many of the math functions in THC. I'm now waiting for histc!

soumith added the bug label Nov 6, 2014

This was referenced Nov 7, 2014

ones, zeros, numel, prod, reshape #75

Merged

rand,randn wrapping #76

Merged

soumith mentioned this issue Jan 6, 2015

Single-dimension std and var. #96

Merged

kashif added a commit to kashif/cutorch that referenced this issue Apr 15, 2015

initial go at histc implementation

0084198

for issue torch#70

soumith mentioned this issue May 4, 2015

Missing :diag() #146

Closed

fmassa mentioned this issue May 12, 2015

Small incompatibility between C/Cuda maskedFill / maskedSelect torch/torch7#231

Open

hughperkins mentioned this issue Feb 28, 2016

Annoying non-existent dependency file warnings #100

Open

JoostvDoorn mentioned this issue Oct 14, 2016

torch.ByteTensor and nonzero() method #538

Closed

hfxunlp mentioned this issue Mar 29, 2017

Will this library support range()? #738

Closed

kaiyuyue mentioned this issue Apr 28, 2017

torch.histc doesn't work if input tensor on gpu pytorch/pytorch#1382

Closed

missing API against TH #70

missing API against TH #70

Comments

soumith commented Nov 6, 2014 • edited Loading

clementfarabet commented Nov 8, 2014

soumith commented Nov 8, 2014

clementfarabet commented Nov 8, 2014

dominikgrewe commented Jan 3, 2015

soumith commented Jan 3, 2015

dominikgrewe commented Jan 20, 2015

soumith commented Jan 20, 2015

dominikgrewe commented Jan 20, 2015

soumith commented Jan 21, 2015

dominikgrewe commented Feb 19, 2015

soumith commented Feb 19, 2015

soumith commented Feb 19, 2015

soumith commented Feb 19, 2015

dominikgrewe commented Feb 19, 2015

wickedfoo commented Feb 19, 2015

dominikgrewe commented Mar 2, 2015

wickedfoo commented Mar 2, 2015

dominikgrewe commented Mar 2, 2015

wickedfoo commented Mar 5, 2015

dominikgrewe commented Mar 5, 2015

dominikgrewe commented Mar 5, 2015

soumith commented Mar 5, 2015

jaderberg commented Mar 12, 2015

soumith commented Mar 12, 2015

soumith commented Mar 13, 2015

soumith commented Mar 13, 2015

dominikgrewe commented Apr 1, 2015

soumith commented Apr 1, 2015

soumith commented Apr 1, 2015

dpfau commented May 24, 2015

soumith commented May 25, 2015

wickedfoo commented May 27, 2015

nicholas-leonard commented May 28, 2015

hughperkins commented Jun 15, 2015

dominikgrewe commented Jun 15, 2015

hughperkins commented Jun 23, 2015

hughperkins commented Jun 23, 2015

hughperkins commented Jun 23, 2015

hughperkins commented Jun 24, 2015

hughperkins commented Jun 24, 2015

abyravan commented Feb 23, 2016

soumith commented Feb 28, 2016

abyravan commented Feb 28, 2016

soumith commented Feb 28, 2016

soumith commented Feb 28, 2016

pvtokmakov commented Jun 26, 2017

ethanluoyc commented Aug 15, 2017

sjain-stanford commented Dec 22, 2017

soumith commented Nov 6, 2014 •

edited

Loading