混搭文化 - 图片的声音
Remix Practice
The Sound of Image
WHAT DOES THE IMAGE “SOUND” LIKE?
Abstract
During the development of art, visual arts and sound have a close relationship with each other. People express their sincere feelings by singing, preforming, dancing, and painting, promoting the development and evolution of realm of art. However, in terms of the relationship between sound and images in the past, they are separated. More specifically, sound always coordinates with images, or images always coordinates with sound. Few people shift images into sound and know what the images really sounds like. For example, we do not know what Vincent Van Gogh’s The Starry Night (1980) sounds like if we do not shift it into any kinds of sound. I believe that, as Navas puts it, “everything is intertextual” (2011). Therefore, for my project, I really want to find the relationship between sound and images. What kinds of the sound generate from the image itself in this particular experiment? Is the visual impression similar to the sound of images? As an experiment, I placed the images into a sound application and created the sound. Then, in order to prove my assumption, I chose 50 people as my experimental subjects to test them via a questionnaire. I intended to use the sound application to generate the sound of images from a film, and combine the sound with that film.
Conceptually
-Inspiration
My inspiration came from a French programmer Michel Rouzic. He made a lot of projects about transforming the images into sound and uploaded the videos to YouTube. In recent years, he always finds and fixes the relationship between sound and images, creating a sound application, Photosounder, which allows people to transform any sound as an image and to create any possible sound from an image. It is the bridge between the graphical world and the audio world, but in my experiment, the sound is not music but the sound of frequency. There are two reasons I want to do this experiment. On the one hand, after I watched his video, I did an experiment for myself. When I listened to a sound without looking at the image, I found that I could almost imagine what the image looked like through the sound, but what I imagined was not totally same as the the image itself. Surprisingly, according to the sound of frequency, I could truly describe the shape and color saturation of the image. However, because the experiment for myself cannot represent others, I need to know if other people can recognize the images by listening to the sound of frequency. On the other hand, different images will have significant and different impact on the visual impression to people, Which seems that different settings will evoke different emotional responses as well. Moreover, everything has its own vibration. In other words, different frequency or vibration will evoke different emotional states. Therefore, sound could help people to understand things easier. As Kramer states, “listening to as well as looking at visual artifacts by ways of digital transpositions of data enables better close reading, more compelling interpretations, and deeper contextual understanding” (2021). Moreover, when people try to analyze a painting or photograph, they usually study it visually. However, we do not know if this painting or photograph has other “miraculous performances” or not behind it. As Kramer points out, “Is there more here than meets the eyes” ( 2021). Therefore, I believe that sound to images can lead to, as Campt puts it, an “ensemble of seeing, feeling, being affected, contacted, and moved beyond the distance of sight and observer” (2017).
Research
According to my research, images do have relationship with sound. In modern digital media society, it is achievable for people to transform visual images to sound or sound to visual images through analyzing them by digital advices. Kramer defines this kind of action as “digital image sonification” (2021). He thinks that “there are truths to discover, but when it comes to analyses of images and sounds, the truths are often multi-perspective, multifocal, and rarely self-evident” (2021). Therefore, the technique of digital image sonification can totally help film industry to make the background music, theme music, and etc. For example, for me, when I watched Interstellar (2014), a science fiction film, I found that all footages perfectly matched to the background music because it is not difficult for me to recall the scenes through listening the background music without looking at the film. Also, I found that when Joseph Cooper traveled in the vast universe with other astronauts, the frequency of the background music is relatively slow; when two space vehicles docked in space, the frequency of the background music is relatively fast, which makes two different hearing responses. Besides the film industry, many other industries are also willing to use similar mode to create scenes or sound such as clothing fashion.
On the other hand, compared with visual analysis, the sound analysis, as another method of acquiring information, could help people to understand data more accurately. As I mentioned before, digital image sonification allows people to sonify images in so many perspectives, which is similar to the Ferguson’s remix method. Generally, shaping an image into sound is transforming an original version to a new version, revealing the difference to the original. As Lisa Gitelman said, “raw data is an oxymoron, and all data are always already new” (2006). For example, we can determine the pitch of the instrument by listening to the vibration frequency of string. Take another example, Robert Hooke was able to determine how many strokes a fly makes with its wings by the stroking frequency during its flying. There are a lot of similar example about the images and sound in different scopes. In summary, different methods of information processing provide people with different types of information, and even help us to predict the future. The more information we get, the more things we can predict (Cao, 2008).
Experiment
-Methodology
Some people regard innovation as an fad. For me, innovation is not necessarily about the most advanced and cutting-edge technology. Instead, it is more about one’s mindset and goals. It is about doing something that has never been done before, as a means to bring forth new products, new technologies, new models, and new approaches. However, the new things do not represent the originality but “recombinations of previously existing building blocks” (Flath, et al., 2017). In this project, I used Kirby Ferguson’s three techniques: “Copy, Transform, and Combine” (2012). Basically, this technique is a method of recreation of music. However, not only it can be applied on making music, but also can create anything based on this technique. As Ferguson said in TED talk, “I think these aren’t just the components of remixing. I think these are the basic elements of all creativity. I think everything is remix, and I think this is a better way to conceive of creativity” (2012). In this experiment, I was mainly responsible for copying, and the sound application helped me to transform and combine.
-Procedure
According to Ferguson’s three techniques, I set each image as my copying object and used Photosounder to achieve the transformation from the image to sound. There were six questions in this experiment, 24 images in total, and the types of images in each question were totally different. I chose four line-pattern images for the first question, four three-dimensional images for the second question, four landscape photographs for the third question, four landscape paintings for the fourth question, four portraits for the fifth question, and four plant paintings for the sixth question. The patterns of these six types of images were from simple to complex. I transformed six images into sounds of frequency, and each image was from one of the six questions. For variables in Photosounder, I set spray width as 20, tool intensity as 25%, minimum frequency as 27.5, maximum frequency as 20000, logarithmic base as 2, and time resolution as 100. Then, I made a multiple choice questionnaire. Each question included one sound of frequency and four images, one of the image in each question represented that audio. Next, I chose 50 people as my experimental subjects to finish the multiple choice questionnaire. Finally, I placed the data into Excel and calculated the correct rate, sample deviations and sum of the squares of the deviations for each question, analyzing the result based on the calculation.
-Result
According to the data (figure1.1), the correct rates of questions 1-6 are 16%, 54%, 22%, 18%, 30%, and 44%, only the correct rate of the second question above 50%, indicating that it is not easy for people to know what the images sound like. Also, different types of images have influence on the experimental result. In terms of the histogram (figure 1.2), ignoring the types of images, the tendency of the correct rates of question 1 to 6 is increasing, so that people gradually know or find the pattern of the relationship between sound and images. Next, I calculated the sum of the squares of the deviations for each question. Based on the result (figure 1.3), the sum of the squares of the deviations for question 1 to 6 are 69, 414.25, 77, 155, 29, and 235. The larger the number, the more confidence people had when they were choosing the answers. Question 2 and 6 have two largest sum of the squares of the deviations, and the correct rates are also the largest compared with the rest of questions.
figure 1.1
figure 1.2
figure 1.3
Conclusion
According to the research, it is not hard for me to find the relationship between sound and images. By experiment, the result are somewhat at variance with the assumption and prediction. However, I believe that the experimental result can not represent that there is no relationship between sound and images because the tendency of the correct rates is increasing. If I make more questions, the correct rates may be higher, even 100%. Furthermore, the result may also relate to the types of images because during the process of transforming images into sound, I found that the brighter the image, the noisier the sound of frequency, which is difficult for people to recognize the style of images. In other words, if I change some variables in Photosounder, the change in frequency may be more apparent so that it may be easier for people to select the right answers. For another perspective, people’s educational background may also affects the experimental result. For people who have studied music, they have a greater ability and an acuter sense of hearing to recognize the images. Overall, the whole experiment need to modify and improve till it become a satisfied and feasible experimental education research. Finally, sound and images, as part of the digital society, “push forward research in digital history and archival thinking” because the sound can “bring one more deeply into the images less discernible aspects” (Kramer, 2021). Therefore, in order to extend the range of subjects taught, digital image sonification can help different scopes to create or find something and “activates the scholarly imagination in promising new ways” (Kramer, 2021).
Archive
-Experiment
When I thinking of archives, I initially intended to use some different logos, basic and complex graphs as my archive for the experiment. Then I planned to drew those logos and graphs as musical notes in Logic Pro. (Look at my Initial Experiment). However, after I did some research and found some examples. I pondered that drawing could not represent the original images or graphs because they were not accurate. Therefore, I decided to use Photosounder as my experiment tool. Although it can not produce the sound as instruments, the sound of frequency totally represent the original images or graphs, which makes the experiment more reasonable. Initially, I placed a lot of simple graphs into Photosounder such as the straight line, diagonal line, spiral, rectangle, and etc. I realized that making these graphs as parts of the test was totally meaningless because everyone who does not have hearing-impaired was able to select the correct answer, much as people know the police car driving through when they hear the siren blaring. Therefore, I searched some complex images. All archives were downloaded online. There were four images of line pattern, four 3-D images, four landscape photographs, four landscape paintings, four portrait paintings, and four plant paintings. The Image credit was shown below.
-Final Product
For the final product, the archives are two images and two clips from the film Interstellar (2014). Two images are Black Hole (02:13:39, Interstellar, 2014) (figure 2.1) and 5-D Space (02:22:45, Interstellar, 2014) (figure 2.2). Two clips are from Interstellar 2014 (02:20:25-02:20:54) and (00:59:25-01:00:16). I placed the images into Photosounder and generate two audios. (figure 2.3 and 2.4). Then, I combine these two sounds with two clips and create a new video.
Remix with Others
In this final remix, I considered if I could transform Sufi image to sound, so that people could generate different emotional responses. However, transforming Sufi image to sound is not my ultimate goal. According to my experiment, people is not easy to recognize what the image looks like. Therefore, I found a relative video about the Sufi Dancing. Comparing Sufi image with video, I found that the essential dancing movement is whirling, which looks same as the Sufi image. Next, after trying to transform the Sufi image to sound, I combined the sound with the video, The Sufi Whirling Dervishes-Istanbul, Turkey (2013). I realized that the sound of the Sufi image and the Sufi dance were a good match because the sound of Sufi image sounds like people are constantly whirling.