For the past eight or so years, I have been exploring issues at the intersection of computational thinking, functional problem solving, and image making. My main research question has been "How can we take problems from image making and use them to encourage people to think computationally and functionally?" The primary focus of the project has been developing interactive scripting environments that use the Scheme programming language. That project, called MediaScripting, has reached a fairly stable state - we have a robust environment that allows Racket, a dialect of Scheme, to script Gimp, an open-source alternative to Photoshop. This environment is extendible to other languages and applications, and I expect those extensions to provide some interesting projects in future summers.
However, this summer I'd like to explore these issues from a very different perspective. In particular, I plan to develop an ecosystem in which people can combine nonrepresentational images using arithmetic operations. In this model, every image can be understood to be a function that takes input in the range [-1 .. 1] and returns numbers in the same range. In the simplest version, -1 represents black, 1 represents white, and numbers in between represent shades of grey. (We can extend this model by using separate functions for red, green, and blue components, or for hue, saturation, and value.)
We can think of any image as a subset of the plane, with x and y
values ranging from -1 to 1. (As is the custom in computer graphics,
we flip the y coordinate system, so that values increase as you move
down the image.)
Here is x.
Here is y.
Here is add x and y. (If the sum is greater than 1, we
use 1. If the sum is less than -1, we use -1).
Here is sign of x.
Note that the sign function maps numbers less than 0 to -1, numbers greater than 0 to 1, and numbers sufficiently close to 0 to 0.
We can also render the image at different scales and in different
widths and heights. For example, here is the the result of multiplying
sign x by y.
The project has four primary foci:
Some additional functions come from basic mathematics. For example, we can negate a number in the range [-1 .. 1] and get a value in the same range, we can take the absolute value of a number in that range and get a value in the same range, we can take the maximum or minimum of with two or more values, and we can do something like sine and cosine.
Some additional functions are natural consequences of work we've seen in CSC 151, such as "rotating" the value (when you go beyond 1 you end up at -1, and vice versa).
Some functions will allow us to build evolving images or to take user input. For example, we will likely support getting the mouse x position, the mouse y position, the x position of the last click, the y position of the last click, and the current time (probably as a series of functions: second, minute, hour, day, month, year).
Designing and developing an appropriate set of functions is essential to the success of this project.
At the same time, we need to think about how these functions can represent images. The greyscale images example are one simple model. But we need models of color images, too. (The HSV and RGB approaches above are one strategy.) And we might also look at ways to design 3D functions.
I envision a graphical environment in which people can drag links between images and functions that combine values, thereby creating new images. In effect, a program becomes a network of functions and images.
Increasingly the social aspects of image-making and Web applications are as important as the technical capabilities. ImageMath users should be able to explore the works of other users, not only the images created, but also the "programs" that create those images. And, once they explore those programs, they should be able to use them in making their own images. (Of course, users should also be able to limit the use of their programs.)
I've developed a small "primer" to guide people through the basic functions. That primer includes a variety of problems, some of the form "predict the image you get by combining two images with this function", some of the form "figure out how to make this image". For example, how might one make a circle using the techniques given above? I think a larger set of problems, along with an appropriate narrative text, will encourage people not only to think more broadly about the capabilities of the environment, but also to think in new ways about computation and functions. Dr. Davis has suggested that we might also talk to high-school math teachers to see if there are ways we can use this system to help students better understand functions.
It is likely that we will do most of the development of the ImageMath ecosystem using Web tools. Students will need to know or learn Javascript, CSS, HTML, and such.
We may also explore using other graphics languages to render the computed images. It will be helpful if students have experience with languages like Postscript and Processing.
While we are unlikely to use Scheme directly, I do think that having good knowledge of the functional data model and key aspects of Scheme will help students design and build the system.
We will use Git and GitHub to manage our code.
Most of our work will be posted on a public Web site. In addition, the source code will be posted on a public GitHub code repository.
Any software written in this project will be distributed under an open-source license. I am not yet sure whether we will use GPL v3, LGPL v3, or another model. I am happy to discuss this choice with my research students.
The spring portion of this schedule mimics the schedule officially approved by the division. However, I will certainly understand if students accept a position with me and then later choose to take a position elsewhere. In that case, I will notify students on my waiting list.
The particular ten weeks we use for summer research are currently under discussion. I would like to start on May 24, but other issues may make that impossible. Hence, the schedule may move back a week.
I have very high expectations of my summer research students. Among other things, I expect my students to begin their "summer" research during spring semester and continue their "summer" research into fall semester (and sometimes beyond). By applying for summer research you are agreeing to meet these expectations if I take you on as a research student. You are unlikely to receive explicit credit or compensation for work in the spring and fall.
As some students from the past may have reported, I expect my students to be self-reliant. While I do my best to be around, I expect you to be able to do many things on your own or with a small group.
During the summer, you are expected to work full-time on the project (40-50 hours per week for ten weeks). Most of the work will be done in my research laboratory, during normal hours (8/9 to 4/5). The work will include scheduled daily group meetings.
I realize that some students may be unavailable in the fall, due to a study-abroad program or similar opportunity. Those students will do what they can of this work (e.g., they might present work in the spring, rather than in the fall).*
Applicants must meet with me for a fifteen minute interview.
Applicants must prepare responses to the Glimmer application form, available at http://www.cs.grinnell.edu/~rebelsky/Department/glimmer-app-2014.html.
Applicants must fill out the divisional summer research application, available at http://www.grinnell.edu/forms/summer-research-application. At one time, that form required you to agree to particular privacy policies. I am not sure if it still does. If the form requires you to agree to a privacy policy that you are uncomfortable with, you can discuss other options with Mary Lyons in the Science Division Office (Science 1232).
I am likely to distribute the essays (e.g., project description for upper-division students, essay questions for first-year students) and sample code from your applications anonymously to a group of trusted advisors. We may each evaluate each person's essay or essays on a 1-4 scale.
In some years, I have incorporated those evaluations in a scoring system. If I choose to score, I will use a formula something like the sum of the following:
I may discuss applicants with colleagues before and after the interview. While the interview and those discussions are not directly incorporated in that score, they provide additional information.
In the end, I will use both rankings and other information as advisory rather than as absolute determiners of students. For example, a poor interview may disqualify you, no matter how good your overall score is, and I may bump up a person with an excellent interview or who past experience suggests will fit well in my research team. I may also prioritize students with certain skill sets.
I do not use the scores in every year. In some years, I try to build an appropriate team using the knowledge I have garnered through interactions with students (including the interview). In the end, I strive to find students who will contribute to my project, who will benefit from working on my project, who I will enjoy working with, and who will work well together.
There are, unfortunately, a number of conflicting issues I must consider as I choose who to accept for summer research. While one goal is certainly to enhance my research, summer research also helps students, the department, and the institution. My approximate MAP priorities are as follows, although I will certainly take some students from "lower" categories before taking every student from a "higher" category.
Students in the class of 2015 on track to receive honors. One of the requirements for honors is that students receive external evaluation of their work. Research in CS is one way to complete that goal. (High scores on the Putnam and Modeling competition are others.) Students also need a 3.4 overall GPA, a 3.5 in CS, an extra 300-level Math or CS course, 213 and (211 or Physics 220), and probably some other things I've forgotten. I'd like to make sure that students who meet all the other requirements also meet the research requirement. Because international students are not eligible for REUs, I am more likely to take international students who meet the other honors requirements.
Students whose background shows they will be particularly productive members of the project team.
Students in the class of 2016 and 2017 likely to participate in the project for multiple summers. I'd like at least one student who will be in category 1 the following year.
Because women and domestic students of color are under-represented in computer science, I will strive for some equity in the mix of students I select. I hope this will encourage more members of these groups at Grinnell to participate in computer science and, as they go on to successful careers, that they will further encourage others.
In many years, I have imitated the early research experience program in Physics and will reserve a few slots for first-year women and domestic students of color who have expressed an interest in and aptitude for computer science. These may be the only first-year students I am able to take for paid positions.