Firefly Plug-in

Firefly offers a set of comprehensive software tools dedicated to bridging the gap between Grasshopper (a free plug-in for Rhino) and the Arduino micro-controller. It allows near real-time data flow between the digital and physical worlds – enabling the possibility to explore virtual and physical prototypes with unprecedented fluidity.

As a generative modeling tool, Grasshopper offers a fluid visual interface for creating sophisticated parametric models, but by default, it lacks the ability to communicate with hardware devices such as programmable microcontrollers or haptic interfaces. Firefly fills this void. It is an extension to the Grasshopper’s parametric interface; combining a specialized set of components with a novel communication protocol (called the Firefly Firmata or Firmware) which together enable real-time communication between hardware devices and the parametric plug-in for Rhino.

A notable distinction for Firefly is that it is the first visual microcontroller programming environment designed specifically for a 3-D parametric CAD package (such as Rhino). This feature means that real-world data, acquired from various types of sensors or other input devices (video cameras, internet feeds, or mobile phone devices, etc.) can be used to explicitly define parametric relationships within a Grasshopper model. Firefly completes the communication feedback loop by allowing users the ability to send information from Grasshopper back to the microcontroller in order to incite specific actuations (ie. lights, motors, valves, etc). Ultimately, this workflow creates a new way to create interactive prototypes.

Arduino Related Components:
 

Ports Available - Check to see which COM Ports are curretly available. If you have one or more Arduino boards connected to your computer, this will return the integer associated with each port.  You can verify the COM number is correct by clicking on Tools>Serial Port inside the Arduino IDE (board must be connected to the computer).
 

Open/Close Port - Open or close the serial port connection.  To be used in tandem with the Serial Read and Write components to establish the connection between Grasshopper and the Arduino.
 

Uno Read - Using the FireFly Uno sketch, this component will read values for all corresponding digital and analog pins on the Arduino Uno, Duemilanove, Diecimilla, or Lillypad board. Note: This component is meant to be used in tandem with the FireFly Uno Firmata Sketch.

Uno Write - Using the FireFly Uno sketch, this component will write values for all corresponding pins on an Arduino Uno, Duemilanove, Diecimilla, or Lillypad board. Note: This component is meant to be used in tandem with the FireFly Uno Firmata Sketch.

Mega Read - Using the FireFly Mega sketch, this component will read values for the corresponding digital and analog pins on an Arduino Mega board. Note: This component is meant to be used in tandem with the FireFly Mega Firmata Sketch.

Mega Write - Using the FireFly Mega sketch, this component will write values to all corresponding digital pins on an Arduino Mega board. Note: This component is meant to be used in tandem with the FireFly Mega Firmata Sketch.

Serial Read (Generic) - Returns any string value coming over the Serial Port. This component does not have any specific Arduino sketch associated with it, which means you can create your own code to send a string value from the Arduino using the Serial.println command.
 

Serial Write (Generic) - Write any string value to the Serial Port. You will need to also provide some Arduino code to process the string value coming over the serial port.
 

Arduino Code Generator - This component will attempt to convert any Grasshopper definition into Arduino compatible code (C++) on the fly. Simply connect the Out string from either the Uno/Mega Write component to the input of the Code Generator and watch the code build as you create your Grasshopper definition. It works by detecting any component 'upstream' from the Uno/Mega Write component. When it detects a component, it checks whether or not it has a function for that component. If it does, it adds the function to the top of the code, and properly calls the function inside the main setup() or loop() functions. If the library of functions does not contain a definition for a particular Grasshopper component, it will provide a warning. This code can be simultaneously saved as a .pde file to be opened in the Arduino IDE.

Quad Stepper Motor - This component will format incoming data to control up to four stepper motors. Note: This component will format the data according to the Firefly Stepper Firmata. With this component you can control the number of steps to move the motors, the maximum speed of the motors, acceleration, and calibration setpoints.

Convert Degrees To Steps - This utility component will convert an angle value in degrees to the appropriate amount of steps to move a motor.

Upload to I/O Board (Temporarily Removed) - This component will attempt to upload any Arduino sketch to your I/O Board. If the Arduino sketch file path is valid (can be from the Code Generator), you can click on the arrow icon to upload the sketch to the board. Alot of stuff happens behind the scenes, but it should create a dynamic MakeFile and convert the .pde file into a .cpp (C++) file and then into .hex code (machine readable code for the microcontroller). NOTE: WinAVR is required to be installed in order for the Upload to I/O Board component to work properly. WinAVR (pronounced "whenever") is a free suite of executable, open source software development tools which includes avr-gcc (compiler), avrdude (programmer), among other things. You can download the latest version of WinAVR at http://sourceforge.net/projects/winavr/files/WinAVR/

Wii Nunchuck Read - This component will allow you to read all of the sensor values (accelerometers, joystick, C and Z buttons) from the Wii Nunchuck. Note: This component is meant to be used in tandem with the Wii Nunchuck Arduino Sketch.
 

Audio:
 

Sound Capture - Capture the audio frequencies from the left and right channel from an audio input (eg. microphone). Note: This component requires the use of the GH_Timer component.
 

Tones From Frequency - Generate audio tones based on incoming frequencies. Also control volume and duration of each tone.
 

Note To Frequency - Generate frequency from notes. For more information visit http://en.wikipedia.org/wiki/Piano_key_frequencies
 

Networking:
 

OSC Listener  - Open Sound Control (OSC) messages are essentially specially formatted User Datagram Protocols (UDP) transmissions. The OSC Listener component works by opening a UDP port and listening for any OSC message on the network. It receives a message it creates a data tree structure which contains the OSC name and the value (can be floats, integers, strings, etc.). If a new message is received with a different name, then a new branch is created for that message. However, if message is received and a branch already contains that name, then the value will updated. Use a List Item or Tree Item to retrieve the latest value for each message in the data tree. A reset toggle can be used to clear the data tree values.
 

OSC Sender - As stated above, OSC messages are specially formatted messages which can be broadcast over a network. In order to send an OSC message, you must know the IP address of the device you're trying to send the message to. You must also specify a unique UDP port. Once you have specified the IP address and the Port, you can connect any type of data (sliders, toggles, panels, etc.) and it will automatically format the data into an OSC message and broadcast that to the specified address.
 

UDP Listener - User Datagram Protocol (UDP) is a transmission model for sending/receiving messages over a network. The UDP Listener component simply opens a UDP port and listens for any incoming message. An IP address is not required for listening for messages (only for sending messages). The return value is a string.
 

UDP Sender - The UDP Sender component sends any string message to a specified IP address on the network through a given port. This can be useful in sending messages to other programs running on another computer (such as Processing) or other mobile devices.
 

Pachube Read - Read a file from any online Pachube feed. Will require an online URL and your pachube API key. Visit www.pachube.com for more information.
 

XML Search - This component allows you to search through an XML file (either from the web or from a local directory) for a specific element tag, returning the value of that tag in a list.
 

Utility Components:
 

AND Flip Flop - The AND flip-flop differs from the NOR flip-flop in the sense that the output gate is determined by its present output state as well the states of both of its inputs. Thus, in a AND flip-flop, the output will not change if both S and R are false, but will toggle to its complement if both inputs are set to true.
 

Bang - The Bang component registers edge conditions for a boolean input. The T-output will register a momentary True value when the input switches to a True state. The F-output will register a momentary True value when the input switches back to a False state. This component is the equivalent to a Bang component in other programs like Pd, Max/MSP, and VVVV. Note: This component requires the use of a Timer component (Params/Special/Timer).
 

Binary Blink - Oscillates 0's and 1's based on an incoming pattern of integers. Can be used to create intricate (HIGH/LOW) blinking patterns to be sent to the Serial Write components to control LED's, etc.
 

Buffer - Store a set of values in the buffer based on the buffer domain. For example, a buffer domain of 0 To 10 will store the last 10 values. However, if we used a buffer domain of 5 to 15 then the buffer will store the 10 values, but only those that occurred five frames before the current incoming value.
 

Constrain - Constrains a number to a specific numeric range (Domain).

 

Counter - Counts upwards and downwards. You can set the direction to count, step size, and constraint limits for the counter. Note: This component requires the use of the GH_Timer component (Params/Special/Timer).
 

Data Log - The Data Log component will store a set of values in a list. If the List Length is set to "0" then the log will store values indefinitely (or until you stop the recording or reset the log). Otherwise, the List Length will determine how many values to store in the log. If the Wrap input is set to True, then the values will begin re-recording over the previous log values after it has reached the end of the List Length.
 

Fader One Way - Fade from one value to another based on a single time interval (ms).
 

Fader Two Way - Fade between a minimum and maximum value based on the fade in and fade out time interval (ms).
 

Frame Rate - Returns the time in milliseconds since the data has last been updated, as well as an estimated frames per second interval.
 

Is Key Pressed - This component will register if a key on your keyboard has just been pressed. Simply specify a string value that you would like to test, and connect a Timer component to this component. The output will return True if the key was just pressed. This component recognizes all alpha numeric inputs, F-keys, and special keys (like up, down, left, right, control, shift, space, etc.)
 

Is Geometry Selected - The Is Geometry Selected component accepts a list of GUID numbers for specified geometry and will return a true boolean value if the geometry is currently selected. Note: This component requires the use of the GH_Timer component (Params/Special/Timer).
 

NOR Flip Flop - The NOR flip-flop has two inputs, namely, a Set input (S) and a Reset (R) input. A simple representation of an S-R flip-flop is a pair of cross-coupled NOR gates, i.e, the output of one gate is tied to one of the two inputs of the other gate and vice versa. Thus, setting the output gate to false requires R=true and S=false, while setting the output gate to true requires S=true and R=false.
 

Playback- The Playback component will retrieve values from a text file (acceptable file types: .txt, .csv, and .dat) and will begin returning individual values at a given frame rate(s). You can input multiple framerates to simultaneously keep track of multiple outputs at different intervals.
 

Smoothing - Smooth (or average) an incoming value based on a sampling level (number of values).
 

State Detection - The State Detection component is used when you want to do some action based on how many times a button is pushed. It checks the number of times the current value has switched from HIGH to LOW (or vice versa) and whether or not the modulo of that counter value equals zero. If so, it flips the state detection output. So, if the modulo value is 2, then every other time the button is pushed, the state value will switch. Or if the modulo value is 4, then every 4th push of the button will trigger the state value to change.
 

Stop Watch- The Stop Watch component attempts to replicate the functionality of a real stop-watch. When you set the start input toggle to true, the timer will start counting, and when you switch it back to false, then it will return the time in milliseconds for that interval. You can repeating this pattern, adding more time each time the stop watch is started and then stopped. You can reset the Stop Watch interval by toggling the reset input.
 

Vision:

Web Cam Video Stream - The Web Cam Video Stream component will automatically connect to your default video capturing device (if available) and begin returning a custom Firefly Bitmap which can be passed through various image filters. Right-click on the component to change the speed of the image capture.
 

Video Player - The Video Player component will provides basic playback functionality for Windows video file types. At the moment, it only supports .avi files. The video player outputs a Firefly Bitmap (just like the Web Cam Video Stream component) and all of the same filters/video analysis tools can be applied to the resultant bitmap.
 

Bitmap Painter - The Bitmap Painter component will visualize any Firely Bitmap on the Grasshopper canvas. You can pass a Firefly Bitmap through various filters/analysis components and visualize the image at any point using this component.
 

Blur Filter - This filter will blur a Firefly Bitmap. The intensity of the blur can be controlled through the number of iterations.
 

Brightness and Saturation Filter - This filter will modify the saturation and brightness of a Firefly Bitmap.
 

Color Correction Filter - This filter will modify the color properties (red, green, and blue) of a Firefly Bitmap.
 

Contrast Filter - This filter will modify the contrast of a Firefly Bitmap.

 

Custom Convolution Filter - A convolution kernel describes how to scale and add together the values of the neighbours of a pixel in order to compute a new value for that pixel. Using a custom 3x3 matrix, you can create all kinds of different image filters including: gaussian blur, edge detection, and sharpening etc. using this component.
 

Edge Detection Filter -Find edges in a Firefly Bitmap.

 

Emboss Filter - This filter will emboss a Firefly Bitmap.

 

Gamma Correction Filter - This filter will provide gamma correction of a Firefly Bitmap.
 

Invert Filter - This filter will invert a Firefly Bitmap.

 

Replace Colors Filter - This filter will replace any given color in a Firefly Bitmap with another color. The intensity can be controlled with a threshold input.
 

Sharpen Filter - This filter will sharpen a Firefly Bitmap.

 

Bitmap Sampler - Sample a Firefly Bitmap based on an incoming set of points for a given filter type (Color, Red, Green, Blue, Hue, Saturation, and Brightness). This component works similarly to the Grasshopper Bitmap Sampler, only this component is customized for Firefly images. The Bitmap Sampler will return values for a series of points mapped at normalized units. Thus, the lower left hand corner of the image can be found by sampling point (0.0, 0.0, 0.0) and the upper right hand corner represents point (1.0, 1.0, 0.0) - so if you want a normal mapping, you will need the domain of your point grid to fall within a u:{0.0 To 1.0} and v:{0.0 To 1.0} range.
 

Gradient Vector - The Gradient Vector component creates a vector field based on changes in brightness in a Firefly Bitmap. The derivatives dx and dy measure how fast the brightness values vary along the x and y axes of the image. If the image were a heigh field then they would measure the slope at each point along the x and y directions. The two derivatives dx and dy form the components of a vector called the gradient of the image. The gradient is a vector that at each point points toward the direction of maximum change.
 

Contour Vector - The field perpendicular to the gradient is the contour field and designates the directions of minimal or no change. It is simply [-dy, dx] since in 2 dimensions we can construct a vector normal to another vector just be switching the x and y components and flipping the sign of one of them.
 

Swirl Vector - Combining the gradient and contour fields we can move diagonally along the height field generating spiralling paths around peaks.
 

Optical Flow - Optical flow is an algorithm that seeks to extract the approximate direction that each pixel tends to move to between frames.
 

Mesh From Image - The height of the mesh can be controlled by the luminance of each pixel. The scale of influence can be controlled using the height slider (works in both positive and negative directions).
 

Video Averages - Find the average direction of movement and cetroids of color within the pixels defined by a bounding box within a Firefly bitmap. This component will search within this rectangle and determine the average direction of movement (vector) as well as the average colors. Mutiple bounding boxes can be defined as search areas within an image to determine the general direction of movement and average colors within that area. Note: Point (0.0, 0.0, 0.0) denotes the lower left hand corner of the image while Point (1.0, 1.0, 0.0) indicates the upper right hand corner of the image.
 

ReacTIVision Listener - Reactivision Listener draws the position and rotation of each Reactivision fiducial marker. This component is meant to be used in tandem with the Reactivision software, available for free from http://www.reactivision.com/downloads. Fiducial marker set available from http://reactivision.sourceforge.net/#files. Note: You must have the Reactivision software and working camera enabled to use this component.
 

Kinect Skeleton Tracker - This component allows you to track up to two people (skeletons) using the Xbox Kinect Sensor. Note: This component will only run if you have Windows 7, Rhino 5.0 beta 32-bit version. This component is being released as a separate download for now, and there are special instructions to work with this component included in this download.

To download the latest version of Firefly (and the Firefly Primer) visit our site: http://www.fireflyexperiments.com

If you've done something amazing with Firefly, we'd love to see it. Feel free to send us images or information about your project (info@fireflyexperiments.com) and we'll make sure we show the world!

If you are new to Arduino, here are a couple videos introducing the microcontroller:

• English version: http://vimeo.com/18539129
• Spanish version: http://vimeo.com/18390711