I am a trained computer scientist. I am also a trained calligrapher with one leg deeply dipped into Chinese calligraphy and related art design work. In this article, I would like to combine my knowledge in these two areas and give my first impression on painting Chinese calligraphy by using VR devices and applications.
In recent years, Virtual Reality (VR) becomes a buzzword in application area. There are a lot articles talking about it that can be easily found online. VR devices have completed the evolution from simple devices like all those mobile phone based VR glasses to more advanced dedicated devices like HTC Vive and Oculus Rift. From my point of view, the VR devices also evolve from a tool for solely viewing to a tool for interaction with VR space. This is the foundation allows me to paint calligraphy and create artworks in a 3D virtual space.
This article will start with the discussion on hardware and software. Then it will continue with some discussion on limitations, possible improvements. Finally, it will finish with discussion on this new method for creative works from a more technique-oriented and artistic perspective.
Not all the VR devices in the market now allow me to paint calligraphy. It at least needs to allow me to interact with 3D spaces freely. This so-called freedom means that it should give me enough space to interact and can track at least the movement of my hand or arm. This will limit the devices to only HTC Vive and Oculus Rift. I know there is Sony PlayStation VR that may potentially achieve this. But at least, now, when I write this article, its website still shows “Greatness awaits”, not saying related software. Oculus Rift does not have the ability to track body movement in virtual space. It allows hand tracking by integrating another device – Leap Motion. I also possess one. However, for people who once used it, they probably know that it is not that reliable. Even after several updates, the accuracy of its tracking function still cannot let me to do any serious work. The concrete phenomenon is that, even when my hand does not move at all, the virtual hand will still shake and change its gesture randomly. This is unacceptable.
In a nutshell, HTC Vive is the only device that can match my basic requirements for painting Chinese calligraphy. It gives me an at least 1.5m*2m space for body movement (The largest space it can support is a rectangle space with a 5m diagonal. But regretfully, I do not have such a space at home for me to indulge). Also, it can relatively accurately track the movements of my arms and the position in a dedicated virtual space. The highlight is on its two wireless trackable controllers. They can act as brushes in VR painting.
In a VR accuracy analysis, Oliver Kreylos from UC Davis claimed that the ‘jitter’ of HTC Vive is around 0.3mm across all directions. Jitter means the imprecision of the measurements when the measured object is standing completely still. This amount of jitter is so small that it is imperceptible to our brains. Also, this system has a precision of 1.5mm and accuracy of 1.9mm. Here precision is “how close multiple subsequent measurements of the same point in space are to each other”, and accuracy is “how close a measurement of a point in space is to the actual position of that point in space”. Therefore, the virtual brush will have very high accuracy. To some extent, this enables the painting to reflect exactly the painters’ intentions. However, considering the size and scale of the system as well as the precision and accuracy, the painting cannot be too small. Based on my experience, painter cannot expect each of the character in the painting to be smaller than palm size. The larger the character is, the better it can match what the painter expects. The size probably will only be limited by your body size and the virtual space size.
Therefore, this article will focus on the experience from HTC Vive. With hardware decided, then I can only find two applications that in Vive’s store can allow me to paint in virtual space. They are PaintLab and Tilt Brush from Google.
This is a free software. It is a good software for graffiti style painting. However, other than that, this software does not show any attraction to me, as a calligrapher. The big problem is how does it models its brush. In my view, it should be called as a spray rather than brush. In 3D virtual space, each of its stroke will be in the form of a continuous tube. Although, painter can change the texture of the tube, but it does not change the nature that it is just a smooth tube. The impression of this tube-like stroke has too long distance from the impression of a stroke you can get from a real Chinese calligraphy work. It does not reflect the rough edge that one normally can observe in the stroke of Chinese calligraphy work. By simply changing the texture on the surface of the smooth 3D tube does not help at all on bringing back this impression. Thus, the review on this software stops here. It is indeed a good software for graffiti style painting. But it cannot be used for Chinese calligraphy painting at this stage.
Tilt Brush is developed for HTC Vive only. With Tilt Brush, user can paint in three-dimensional space. Because of the experience in PaintLab, the first thing I am concerning is how do they build their brushes. I have to say the results are better than what I expected. In most occasions, the brushes are still modelled as a 2D-like thin piece rather than the 3D tube in PaintLab. This gives the software a more concrete foundation to mimic the brushes finish in real world. For example, I use the ink brush the most. It gives a very similar feeling to the brush in real world. The random rough edge and “flying white”(hollowed) part in the stroke can be easily noticed. Besides, it also provides many other brushes. From my perspective, I categorise them into two groups: static brush and dynamic brush. The static brushes, such as ink, will not change its form and shape after being painted. Dynamic brushes will dynamically change their form and shape. There is a fire brush, whose stroke vibrates like real flame. There are a lot of them. You can easily get an idea about how do they dynamically change their form and shape by names, like plasma, smoke, snow and so on. This definitely provides more possibility to the art presentations to this new form of Chinses calligraphy in 3D space.
In Chinese calligraphy, the control on pressure you give to the paper is crucially important. This influences the shape of the stroke, the width of the stroke, the dynamics of the whole painting, which will finally form your own style. This needs to be satisfied in VR world. In Tile Brush, there are two ways to control the width of the stoke. You can adjust the medium width of your brush before you paint a stroke. Then when you are painting, you can use the trigger on wireless controller to further adjust the width around the medium value. But this generates new confusion to me. Let me call the first type of adjustment as static adjustment, and the second is dynamic. Both of the adjustments have their ranges. The dynamic one is even more narrow. It is not like that free and flexible in real world (Of the course, real world has far higher entropy). But there is definitely something can be improved. For certain brushes, the smallest value you can choose is already two fingers wide. That is also one reasons I like to use ink brush because its stoke diameter can be very small. Also, in order to change the width to a different one, which is not in the dynamic range the user is in now, the user needs to stop the current painting, reselect the medium value and re-paint. This means, if you want to paint a stroke with a wide dynamic range continuously, it is simply impossible.
In traditional Chinese calligraphy painting process, there is a secret tool always used by artists. Sometimes, they even do not realize that they are using it until this tool is taken away. It is the feedback. They most direct feedback is visual feedback. In addition, in the process of painting a stroke, painter can also feel the changing resistance from the paper by applying varying pressure. Besides, the sound generated by friction is also an important feedback. The feedbacks from these three channels allow artists to dynamically control and adjust arm and hand movements. In Tilt Brush, when stroke being painted, the software will generate corresponding frequencies of sounds based on the speed of brush movements. However, there is no feedback from the controller, such as different frequencies of vibration. Therefore, two of the three traditional feedback channels can be mimicked by Tilt Brush to some extent.
We cannot expect all the beholders of 3D calligraphy works will put on the VR glasses to only have a look of them. There must be some ways to allow beholders to approach easily. I will discuss about this perhaps in another article. But before we arrive there, the VR painting software must allow us to export the 3D model in a standard way. Fortunately, Tilt Brush can export 3D paintings to OBJ file. This is a geometry definition file format. The file format is open and has been adopted by plenty of 3D graphics application vendors. This makes the potential to present the 3D calligraphy in other ways be possible.
For the developers of Tilt Brush and any similar software’s developers, I have following suggestions for possible improvements if they want calligrapher to use it:
1. Please provide wide range to the dynamic adjustment of brush width/diameter.
2. It is better to provide more ways of feedback channels when painting. Not only from visual channels, the sense of touch and hearing are also very important.
3. Surely, it will be an obvious direction that the accuracy and precision still need improvement. This is also a suggestion to hardware developer.
New era of Chinese calligraphy
In the process of trying and reviewing the hardware and software, I am also trying to figure out the techniques about how to create this art form of lines and shapes in this new intuitive virtual space. As aforementioned, the first thing I noticed is that each character cannot be smaller than my palm. This is mostly limited by the accuracy and precision of hardware and the width of brush in software. Besides, I list my first thoughts about technique tips in following text:
It is well known that in traditional calligraphy work, the contrast between white paper as background and black ink is an essential part of this art form. Even when the calligraphy is carved on other surfaces, such as stone, the characters themselves will be painted with a highly contrasted colour. However, in software like Tilt Brush, the background is isolated from the 3D model. It can be changed. In such a circumstance, there is a possibility that the background of the calligraphy may be different from what the artist expected when it is presented. There are two ways to preserve this information. First, the background needs to be saved together with the 3D model in one file or at least associated. This needs to be done from software development level. Second, which is an easier way, you draw the background for yourself. I notice the paper brush in Tilt Brush can work very well on this.
2. Dynamic Brush:
This is an amazing function from Tilt Brush. Calligraphy is always a static art form with static lines. By using dynamic brushes, such as fire, smoke and plasma, the stroke of the character generates a special texture, which is eye-opening. The rough edge is dynamic. The “flying white” is also dynamic. Furthermore, with the combination of several dynamic brushes, it will offer more possibilities to the presentation of this art form. It is strongly recommended to try more on this new type of brushes. If there is a way to present this finish in real world, the effect will be stunning.
3. Different scripts, Different techniques:
Now, I have tried to write cursive script, semi-cursive script and regular script in Tilt Brush. I notice that because of the relatively higher randomness in cursive script’s strokes, the stroke or even the whole character (you do not need worry about that the ink runs out) can be painted in one go. This is similar to traditional calligraphy. But I found out that, to write regular script, it is very hard to write a stroke in one go because of the current accuracy, precision and brush width. I have to paint strokes and characters like drawing a picture. This is definitely not the original calligraphy technique. To write semi-cursive script, the feeling is some sort in the middle. Maybe I should practice to fit myself into this brush style. Or the software can be improved in order to support such subtle technique.
4. New 3D fonts:
No doubt, VR opens a new door. There is no reference at all for a 3D Chinese character. The strokes are not crossing with each on a 2D surface. They are wired together in a 3D space. I hardly can still call them as Chinese characters. Perhaps they are just some 3D shapes inspired by Chinese characters. No matter how do you call that, it cannot be denied that this new 3D space give artists more chances to create new 3D fonts with various styles that never existed.