Synchronously sort data while creating a set

I have two sets of data of the same size, one of is a list of points and the other contains generic data. I can create a set from the list of points to remove all duplicate data, but I also want to be able to see which items in the generic data list correspond to each of the unique items in the set. In other words each of the unique items in the first data set will have a corresponding list of items from the second data set.

My current process is to use the replace branches component to go back and extract the corresponding items, but that component is quite slow due I suspect to its having to search through the data (I have large amounts of data, 40-50k items)

I may be missing something obvious here, but is there no simpler / faster way? It would be a nice option for the Create Set component to accept synchronous data sets which are then filtered into paths corresponding to each of the unique set items.

Any ideas?

Cheers,

Luke

▶ Reply to This

Replies to This Discussion

Permalink Reply by Vicente Soler on April 5, 2012 at 5:41pm

Is this what you are looking for?

▶ Reply

Permalink Reply by Vicente Soler on April 5, 2012 at 5:49pm

I missed the part when you say you have 50k items and it's working slow.

Maybe a scripting component will make it faster, try the following:

Create a VB component and paste the following code:

Dim dt As New datatree(Of Object) For i As int32 = 0 To x.Count - 1 dt.add(x(i), New gh_path(y(i))) Next a = dt

Set both x and y inputs as "list" and set the y input as "integer". Connect the generic data to the x input and the M output of the create set component to the y input.

▶ Reply

Permalink Reply by Luke Epp on April 5, 2012 at 5:58pm

Thanks Vicente - I had tried the first method you showed but for some reason hadn't clued into the tree structure of the member index component - exactly what I wanted. Using those native components takes about 40% of the time of my original solution, and using your script is a fraction slower.

I guess my point with this all was to try to do it natively (as the information necessary must be created when the set is first formed) and thus avoid extra computation downstream. I still think adding an optional input to the create set component would be useful (could use the ZUI as well..) I wouldn't imagine it would be much more overhead and it would be very handy (similar to the sort component).

Cheers!

Luke

▶ Reply

Permalink Reply by Vicente Soler on April 6, 2012 at 8:52am

Is the script really slower? I just tested it out with 1 million items and the script was much much faster. (11 sec vs 2.4 min)

▶ Reply

Permalink Reply by Luke Epp on April 6, 2012 at 10:09am

No, the script is in actuality faster. For some reason on the first solution run I did it was slower (I've noticed the profiler does that occasionally - unless you change data somewhere, a recompute doesn't necessarily cause the profiler to show the right computation times).

Thanks again for the help.