Resources
How to analyse sound input? Goto and download the free opensource language called PureData and open the following example with it. This is designed to track pitch of an incoming sounds, identify its loudness and get sinusoidal components of the spectrum.
Download the example here
***
How to determine a position and relationship of letters in a text? It's not an easy task, but with the following example you can have an impression of the occurances of different characters in a text. Just open "index.html" in any browser and type in the desired characters to find in the text.
Download Example here
***
How to control more LEDs than the amount of pins we have on an arduino? There is a way of hacking the flow of current using "charlieplexing". Just connect your LEDs with the proper resistors (around 250 Ohm) and see the results. More on this example can be found here. You need to setup the wires correctly, connect and upload the code to your arduino. More on the basics of arduino can be found here.
***
A way of analysing video that can be streamed to a small array of LEDs can be realized easily with the free and opensource Processing language. Download it from here, then run the following example within it.
***
Analyzing live video stream (webcam, and other video sources). The central, vertical line of pixels are tracked and if the color of a pixel falls below a darkness threshold, its position is passed to a sound engine. Therefore, the pitch and modulation of the sound is determined by the position of dark pixels in the (center line) of the image.
***
As a continuation of one of the previous examples ( where you can determine a spatial position of a letter in a string of characters ) we had to find out a method to match corresponding coordinates of characters within scanned documents. This has nothing to do with text based character matching (using reular expressions or the like) anymore. We had to figure out a method to match small parts within a large scanned image and store the corresponding coordinates of each of its occurences. We ended up with a nicely developed algorithm which is called template matching. Here, using computer vision strategies, a small part of the image can be compared to a whole larger image, and the result can be adjusted smoothly with different thresholding parameters.
The goal is to find each occurence of one specific letter on the original printed copy, and use their actual positions for further analysis (such as sonification, internal structural representations). For this we were using the free, open source computer vision library OpenCV with Python. After (some non-trivial, frustrating twisted) proper installing on the OS, the code is pretty straightforward:
import cv2
import numpy as np
img_rgb = cv2.imread('source_image.jpg')
img_target = cv2.imread('target_image.jpg')
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
template = cv2.imread('template_image.jpg',0)
w, h = template.shape[::-1]
res = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED)
threshold = 0.75
loc = np.where( res >= threshold)
for pt in zip(*loc[::-1]):
cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
cv2.rectangle(img_target, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
cv2.line(img_target,pt,(500,500),(0,0,255),2)
cv2.imwrite('result_image_with_original.png',img_rgb)
cv2.imwrite('result_with_white_background.png',img_target)