ELK, huge amount of data operation

Replies to This Discussion

Permalink Reply by Armin Seltz on November 3, 2015 at 2:45am

Hi Archiheart.

You will have to attach the GH file you are using to load the data into, which shows the subsequent steps you are trying to take.

Also tell us how you are importing the data into GH. Have you tried with a small subset of the same data, like just one block of the same data? Did it work? Did you use Profiler to note which components are the most time-consuming? If you can see the total time that it takes for example for data that contains 100 values and the total set has 10.000 values than you can safely say, that it will take at least 100 times as long. Have you checked the Windows Task Manager while it is running to see if it is actually running out of Memory?

Until then I don't think we can help you. GH not responding is very common if you work with a lot of data and/or do a lot of operations that can take a long time. I dont know why it sometimes freezes completely, sometimes can be paused using Escape, sometimes saves a Recovery File and sometimes doensn't. I even have cases where I can't even force quit Rhino, but have to actually restart the computer. Maybe David Rutten could enlighten us about this, as I have always wondered about this and its so frustrating having to restart Rhino a lot.

To prevent crashes its also a good idea for example to Disable the Solver and then Recompute manually once you have done the changes you want. A lot of times GH freezes because you forgot to Graft or Flatten one of the Connections and then all the calculations get multiplied.

Permalink Reply by Archiheart on November 3, 2015 at 3:16pm

Hi Armin,

Thank you for responding,

Well, as I said , I am rather new to elk. My usage of ELK is limited to 

http://blog.alexwebb.com/?p=1343  and https://www.youtube.com/watch?v=8_uQj_Rr8eA 

Thus I import it through 'file path' , then 'Location'. That is the problem that it has an excessive amount of data inside it. And I don't know how to do to limit/select that amount (nor could I find an understandable way to be honest).

Concerning these questions: "Have you tried with a small subset of the same data, like just one block of the same data? Did it work? Did you use Profiler to note which components are the most time-consuming?" 
I'm not sure I understand you, but in case you're telling me to try to open it through a text editor, to see the data I need, I did try it. But it took allready too much time to open, thus finding the data I need was unbearable.

And yes, I did check the task manager, and it shows that I have no free ram memory while processing it. There are about (~700 000 values). 

Permalink Reply by Armin Seltz on November 4, 2015 at 2:23am

Alright, looking at that link it looks like it works with a smaller subset of the data. What I meant with "just one block" is that you use the same map of Paris, but don't zoom out so far and just try to see if it works with a small section.

But so tell me about the moment that it freezes. Does it freeze even when loading in the file? So if you just have the component that reads the data and dont connect anything to it, can you successfully load the data and see the output has 700 000 values inside? If this already works than you can simply use the list item component to just get a few of the values and then ramp it up, like this:

So what you can see here is at first I have a series with 1 million values and then I just take a small set of that using list item with another series. The data that goes into the List Item can be anything. Already just the list item takes 1.4 seconds and the whole thing takes 3.2 seconds to calculate, which is quite long for something so simple. Rhino reports 980 MB memory usage for this. If I crank it up to 10 million the calculate time is 28.6 seconds and memory goes up to 3.2 GB. So yes I can believe that what you are trying to achieve will push the computer really hard.

Can I ask what computer you have? How much memory does it have?

To do things for an entire huge city like Paris is starting to be something you wouldn't do on your laptop or an old PC, thats why researchers and scientists have expensive workstations with large amounts of RAM. Can't you just use a smaller city or just a part of Paris? Is there a reason it has to be ALL of Paris? In the end its what you do with the data that has to be cool and not just the fact that you used A LOT of data. So rather do something really cool with a small set of data than do something thats limited because there is so much data.

Permalink Reply by Archiheart on November 4, 2015 at 5:09am

It freezes at the moment when I try creating the a polyline for the 'buildings' feature key . It loads ok I guess, it does take a little while after connecting it to the 'Location' component, but not too long. Anyways I could get to see the number of components within the file.  

I have a Dell Inspiron 17R(5721). With the 8 gb ram. It works allright, with a little portion of the map, as in I took the region I would be working on, but I need the entire map for a research/presentation I am doing. Thus a vectorised map, and the possibility of extracting a certain information, would save me a great amount of time.(Unless figuring it out would take more time of course)

Permalink Reply by Armin Seltz on November 4, 2015 at 6:17am

Hm ok, well then it will just take a long time to calculate and need a lot of RAM. Once your physical RAM runs out and it has to start using the hard drive things will get really really slow. 

Send me the GH file you are using and I will download the map and try it out on my machine here. Its a top-of-the-range 27" iMac with 16GB of RAM. If it works I can send you it back as a Rhino File. If you do, then please put some notes of what goes where. I will download the maps data from the link you gave.

Permalink Reply by Armin Seltz on November 4, 2015 at 6:21am

Hm, actually openstreetmaps keeps giving me an error when trying to download the data :/ Maybe you can put the data on dropbox or wetransfer or something.

Permalink Reply by Archiheart on November 4, 2015 at 1:18pm

https://www.dropbox.com/s/vmxioz2gq132psj/Paris1_02_01.osm?dl=0

Attachments:

• Carte site.gh, 5 KB

Permalink Reply by Armin Seltz on November 5, 2015 at 3:31am

Ok, so I had a look at it. It seems ELK is just super slow for some reason. I downloaded a software called OSM IQ, which can read and export OSM data and that opened that file you sent me in a few seconds. I could then export the buildings, natural and highways to .dxf and import it into Rhino.

I am attaching a link to the .3dm file with buildings, highways and natural on layers. From there it is easy to get them into GH and do what you need with them. That way you wont need ELK at all and can concentrate on actually doing something cool with the data :)

https://www.dropbox.com/s/bybo4krodje6i0e/Paris_natural_highways_bu...

Its funny that you are using a map from Paris. I went to the Informed Matter Workshop in Paris a while ago and for our last little project called "lifeline" we were using data of paris, in this case the metrolines and population data. This is population density along the metro lines:

Maybe they will have a workshop again, it seems like you should go to it:

http://www.co-de-it.com/wordpress/informed-matter-gh-ws-paris.html

Permalink Reply by Archiheart on November 6, 2015 at 4:09am

Thank you very much :) although I would really like to know how to do it in ELK, but there might be a problem within the PC or maybe even ELK :( 

About that workshop: is it like a temporary program, or is it to get a masters degree? Because I looked through it, and it was a 12 month program for masters degree. so I'm a bit confused. (me being in 2nd year architecture school, don't think I could go through it. Although, having more knowledge about the computational design, would help me a lot, but being a 12 month program, might leave me out of time to focus on school project.)

Is it just me who got it wrong?

Permalink Reply by Armin Seltz on November 6, 2015 at 8:16am

Hi again. No it was not a masters program. They also offer that now and I would love to go, but can't this year, so will hopefully do it next year.

This was a 4 day workshop that was all about using Data in Grasshopper. So we learned a lot about data sources, environmental data, etc. and how to use them in GH. It was a great experience and was my start with GH. I knew it a little bit, but I learned so many things about it there. Not sure if they will do it again, but it was quite successful, so maybe they will.

I have not used ELK before, so can't comment on it, but I can confirm that it took a really long time. But for the most part even for a very large architectural project you wont need OSM data for a whole city, especially not one as big as Paris. So if you want to use ELK use it with normal size maps like a few hundred meters square, because 99% of projects wont be bigger than that. If they are then you will have the proper resources ;)