Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
325 lines (261 loc) · 23.2 KB
/
Copy pathindex.html
File metadata and controls
325 lines (261 loc) · 23.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<link rel="stylesheet" type="text/css" href="https://fonts.googleapis.com/css?family=Merriweather:300">
<link rel="icon" type="image/png" href="https://android.processing.org/favicon.png">
<link rel="stylesheet" href="../../css/main.css">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<title>Processing for Android</title>
</head>
<body>
<nav class="topnav">
<ul class="left">
<li><a href="https://processing.org/">Processing</a></li>
<li><a href="https://p5js.org/">p5.js</a></li>
<li><a href="https://py.processing.org/">Processing.py</a></li>
<li><a href="https://pi.processing.org/">Processing for Pi</a></li>
</ul>
<ul class="foundation">
<li><a href="https://processingfoundation.org/">Processing Foundation</a></li>
</ul>
</nav>
<header>
<h1 class="title">Processing for Android</h1>
</header>
<div class="group">
<nav class="site">
<img class="logo" src="../../imgs/logo_processing_android.svg" alt="Processing for Android logo">
<ul class="leftnav">
<li><a href="../../index.html">Home</a></li>
<li><a href="../../install.html">Install</a></li>
<li><a href="../../reference/index.html">Reference</a></li>
<li><a id="selected" href="../../tutorials/index.html">Tutorials</a></li>
<li><a href="../../books/index.html">Books</a></li>
<li><a href="../../gallery/index.html">Gallery</a></li>
<li><a href="https://discourse.processing.org/c/processing-android" target="_black">Forum</a></li>
<li><a href="https://github.com/processing/processing-android" target="_black">GitHub</a></li>
</ul>
</nav>
<section class="container main-text">
<div class="lang">
<a id="selected" href="index.html">EN</a>
<a href="../../es/tutorials/ar_intro/index.html">ES</a>
</div>
<hr style="clear:both;">
<h2>AR in Processing</h2>
<p>
How to create Augmented Reality apps using Processing for Android.
</p>
<h3>ARCore</h3>
<p>
Google offers a framework to create Augmented Reality (AR) apps for Android, called <a href="https://developers.google.com/ar" target="_black">ARCore</a>. Phone-based AR experiences typically involve drawing digital 3D content on top of the physical world, as seen from the perspective of the phone's camera, in such a way that the digital contents convincingly appear to be part of the real-world environment. Doing this requires applying complex algorithms in real-time to recognize objects in the physical environment, such as walls, pieces of furniture, even people, and determining the pose (position and orientation) of the phone relative to the world over time. Fortunately for us, ARCore handles all of these calculations automatically, and provides an <a href="https://developers.google.com/ar/reference" target="_black">API</a> to access the real-world features and attach digital contents to them from our code. While it is possible to access ARCore directly from Processing, the Android mode includes an AR library that makes it easier to use ARCore in our Processing sketches and apply Processing's drawing API to create AR content.
</p>
<p><a href="https://experiments.withgoogle.com/hidden-world" target="_black"><img class="body-image" src="../../imgs/tutorials/ar_intro/hidden_world.gif" alt="Hidden World"></a></p>
<p>AR is a rapidly growing field, thanks to frameworks such as ARCore on Android (and <a href="https://developer.apple.com/augmented-reality/arkit/" target="_black">ARKit</a> on iOS) and to the fast pace of technical improvement of smartphones. The integration of real-world environments with digital contents is opening many new possibilities for phone-mediated experiences in the physical world. The <a href="https://experiments.withgoogle.com/collection/ar" target="_black">AR experiments</a> from Google and the <a href="https://github.com/olucurious/Awesome-ARKit" target="_black">Awesome ARKit</a> list of AR apps for iOS are good resources to find projects making creative use of AR.
</p>
<h3>Requirements</h3>
<p>In order to ensure a good AR experience, Google certifies devices to be supported by ARCore. The official list of supported devices is available <a href="https://developers.google.com/ar/discover/supported-devices" target="_black">here</a>. The devices in that list should work with Processing as well to create AR projects.</p>
<h3>Getting started</h3>
<p>
You need to select the AR option in the Android menu to make sure that your sketch is built as an AR app:
</p>
<p><img class="body-image" src="../../imgs/tutorials/ar_intro/ar_selection.png" alt="AR selection"></p>
<p>
While in VR you can make pretty much turn any 3D Processing sketch into VR simply by importing the VR library and using the VR renderer, using AR requires a few more steps. First of all, we need to add an ARTracker to our sketch, and call its <code>start() function</code> to get AR tracking going. The other two classes in the AR library are ARTrackble and ARAnchor, they represent a surface that can be tracked in real-world space (such as a table or a wall), and a point in space that remains fixed relative to a given trackable surface. These concepts will hopefully become clearer as we go through the tutorial. For the time being, a basic skeleton for an AR sketch in Processing could look like the following code, where we just launch tracking and print to the console the trackable objects detected in each frame:
</p>
<pre><code>
<span style="color: #33997E;">import</span> processing.ar.*;
<span style="color: #E2661A;">ARTracker</span> tracker;
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>setup</b></span>() {
<span style="color: #006699;">fullScreen</span>(<span style="color: #718A62;">AR</span>);
tracker = <span style="color: #33997E;">new</span> ARTracker(<span style="color: #33997E;">this</span>);
tracker.<span style="color: #006699;">start</span>();
<span style="color: #006699;">noStroke</span>();
}
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>draw</b></span>() {
<span style="color: #006699;">lights</span>();
<span style="color: #669900;">for</span> (<span style="color: #E2661A;">int</span> i = 0; i < tracker.count(); i++) {
<span style="color: #E2661A;">ARTrackable</span> t = tracker.<span style="color: #006699;">get</span>(i);
<span style="color: #006699;">println</span>(<span style="color: #7D4793;">"Trackable"</span>, i, t);
}
}
</code></pre>
<h3>Drawing trackables</h3>
<p>
Trackable objects in Processing are limited to plane surfaces, even though the underlying <a href="https://developers.google.com/ar/reference/java/arcore/reference/com/google/ar/core/Trackable" target="_black">trackable in ARCore</a> can represent other kind of features in physical space, such as cloud points. Each trackable object contains basic information about the physical entity it represents, including size and current status (whether is being tracked, paused, or stopped). In order to draw a trackable plane, we first need to apply the transformation that turns Processing's world coordinates into the trackable system, in this way, it is easy to represent points relative to the trackable plane that spans X and Z axis irrespective of its orientation with respect to Processing's coordinate system. We can retrieve the size of the trackable along each axis with the lengthX() and lengthZ() functions, and call the transform() function before making any drawing:
</p>
<pre><code>
<span style="color: #33997E;">import</span> processing.ar.*;
<span style="color: #E2661A;">ARTracker</span> tracker;
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>setup</b></span>() {
<span style="color: #006699;">fullScreen</span>(<span style="color: #718A62;">AR</span>);
tracker = <span style="color: #33997E;">new</span> ARTracker(<span style="color: #33997E;">this</span>);
tracker.<span style="color: #006699;">start</span>();
<span style="color: #006699;">noStroke</span>();
}
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>draw</b></span>() {
<span style="color: #006699;">lights</span>();
<span style="color: #669900;">for</span> (<span style="color: #E2661A;">int</span> i = 0; i < tracker.count(); i++) {
<span style="color: #E2661A;">ARTrackable</span> t = tracker.<span style="color: #006699;">get</span>(i);
<span style="color: #006699;">pushMatrix</span>();
t.transform();
<span style="color: #E2661A;">float</span> lx = t.lengthX();
<span style="color: #E2661A;">float</span> lz = t.lengthZ();
<span style="color: #006699;">fill</span>(255, 100);
<span style="color: #006699;">beginShape</span>(<span style="color: #718A62;">QUADS</span>);
<span style="color: #006699;">vertex</span>(-lx/2, 0, -lz/2);
<span style="color: #006699;">vertex</span>(-lx/2, 0, +lz/2);
<span style="color: #006699;">vertex</span>(+lx/2, 0, +lz/2);
<span style="color: #006699;">vertex</span>(+lx/2, 0, -lz/2);
<span style="color: #006699;">endShape</span>();
<span style="color: #006699;">popMatrix</span>();
}
}
</code></pre>
<p>
The sketch above should draw all the trackable planes being detected by the phone as we move around:
</p>
<p><img class="body-image" src="../../imgs/tutorials/ar_intro/ar_trackables.png" alt="trackable planes"></p>
<p>
It is easy to test if a trackable is being selected by the user using the touchscreen, all we need to is to call the <code>isSelected(x, y)</code> with the (x, y) coordinates of the touch point:
</p>
<pre><code>
...
t.transform();
<span style="color: #E2661A;">float</span> lx = t.lengthX();
<span style="color: #E2661A;">float</span> lz = t.lengthZ();
<span style="color: #669900;">if</span> (<span style="color: #D94A7A;">mousePressed</span> && t.isSelected(<span style="color: #D94A7A;">mouseX</span>, <span style="color: #D94A7A;">mouseY</span>)) {
<span style="color: #006699;">fill</span>(255, 0, 0, 100);
} <span style="color: #669900;">else</span> {
<span style="color: #006699;">fill</span>(255, 100);
}
<span style="color: #006699;">beginShape</span>(<span style="color: #718A62;">QUADS</span>);
<span style="color: #006699;">vertex</span>(-lx/2, 0, -lz/2);
...
</code></pre>
<p>
Adding the code above to our previous sketch should result in the following interaction when touching a plane in the camera image:
</p>
<p><img class="body-image" src="../../imgs/tutorials/ar_intro/ar_trackable_selection.png" alt="trackable selection"></p>
<h3>Using anchors</h3>
<p>
Once we have trackable surfaces in our AR scene, we can attach anchors to them. <a href="https://developers.google.com/ar/develop/developer-guides/anchors" target="_black">Anchors in ARCore</a> are essentially positions in space that are fixed relative to the trackable they are attached to, and allow us to make virtual objects appear to stay in place in the scene. Anchor objects in Processing are created with the relative coordinates they will have relative to their parent trackable. Extending our code above, we could add new anchors exactly when a trackable is detected for the first time. That event can be handled by adding the <code>trackableEvent()</code> function to our code, which will receive the trackable being detected as an argument. We need to keep track of all the anchors in our scene manually, for example by storing them in a list, and removing them when they are no longer being tracked. The following code does all of that:
</p>
<pre><code>
<span style="color: #33997E;">import</span> processing.ar.*;
<span style="color: #E2661A;">ARTracker</span> tracker;
<span style="color: #E2661A;">ArrayList</span><<span style="color: #E2661A;">ARAnchor</span>> trackAnchors = <span style="color: #33997E;">new</span> <span style="color: #E2661A;">ArrayList</span><<span style="color: #E2661A;">ARAnchor</span>>();
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>setup</b></span>() {
<span style="color: #006699;">fullScreen</span>(<span style="color: #718A62;">AR</span>);
tracker = <span style="color: #33997E;">new</span> ARTracker(<span style="color: #33997E;">this</span>);
tracker.<span style="color: #006699;">start</span>();
<span style="color: #006699;">noStroke</span>();
}
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>draw</b></span>() {
<span style="color: #006699;">lights</span>();
drawAnchors();
drawTrackables();
}
<span style="color: #33997E;">void</span> trackableEvent(<span style="color: #E2661A;">ARTrackable</span> t) {
<span style="color: #669900;">if</span> (trackAnchors.<span style="color: #006699;">size</span>() < 10 && t.isFloorPlane()) {
trackAnchors.<span style="color: #006699;">add</span>(<span style="color: #33997E;">new</span> ARAnchor(t, 0, 0, 0));
}
}
<span style="color: #33997E;">void</span> drawAnchors() {
<span style="color: #669900;">for</span> (<span style="color: #E2661A;">ARAnchor</span> anchor : trackAnchors) {
<span style="color: #669900;">if</span> (anchor.isTracking()) drawSphere(anchor, 0.05);
<span style="color: #669900;">if</span> (anchor.isStopped()) anchor.dispose();
}
tracker.clearAnchors(trackAnchors);
}
<span style="color: #33997E;">void</span> drawTrackables() {
<span style="color: #669900;">for</span> (<span style="color: #E2661A;">int</span> i = 0; i < tracker.count(); i++) {
<span style="color: #E2661A;">ARTrackable</span> t = tracker.<span style="color: #006699;">get</span>(i);
<span style="color: #006699;">pushMatrix</span>();
t.transform();
<span style="color: #E2661A;">float</span> lx = t.lengthX();
<span style="color: #E2661A;">float</span> lz = t.lengthZ();
<span style="color: #669900;">if</span> (<span style="color: #D94A7A;">mousePressed</span> && t.isSelected(<span style="color: #D94A7A;">mouseX</span>, <span style="color: #D94A7A;">mouseY</span>)) {
<span style="color: #006699;">fill</span>(255, 0, 0, 100);
} <span style="color: #669900;">else</span> {
<span style="color: #006699;">fill</span>(255, 100);
}
drawPlane(lx, lz);
<span style="color: #006699;">popMatrix</span>();
}
}
<span style="color: #33997E;">void</span> drawSphere(<span style="color: #E2661A;">ARAnchor</span> anchor, <span style="color: #E2661A;">float</span> r) {
anchor.attach();
<span style="color: #006699;">fill</span>(#CF79F5);
<span style="color: #006699;">sphere</span>(r);
anchor.detach();
}
<span style="color: #33997E;">void</span> drawPlane(<span style="color: #E2661A;">float</span> lx, <span style="color: #E2661A;">float</span> lz) {
<span style="color: #006699;">beginShape</span>(<span style="color: #718A62;">QUADS</span>);
<span style="color: #006699;">vertex</span>(-lx/2, 0, -lz/2);
<span style="color: #006699;">vertex</span>(-lx/2, 0, +lz/2);
<span style="color: #006699;">vertex</span>(+lx/2, 0, +lz/2);
<span style="color: #006699;">vertex</span>(+lx/2, 0, -lz/2);
<span style="color: #006699;">endShape</span>();
}
</code></pre>
<p>
In the code above, we create a new anchor at the point (0, 0, 0) relative to the trackable being detected, but only if is a horizontal surface (i.e.: a floor), which we can detect by calling <code>isFloorPlane()</code>. A similar function, <code>isWallPlane()</code>, allows us to know if the trackable is a vertical plane (i.e.: a wall). It is important to notice the dimensions of the objects in AR are rather small in comparison with a typical 3D sketch (for example, the radius of the spheres being drawn at the anchor positions is only 0.05).
</p>
<div class = "butterbar">
The reason is that ARCore's translation units <a href="https://developers.google.com/ar/reference/c/group/concepts#poses-and-coordinate-spaces" target="_black">are meters</a>, so everything in our AR sketch should be scaled to sizes that are consistent with the real-world dimensions, as expressed in the decimal system.
</div>
<p>It is also important that we dispose an anchor once is no longer tracked with the line <code>if (anchor.isStopped()) anchor.dispose();</code> to make sure we don't waste resources with anchors that are no longer active (anchors that are simply paused should not be disposed as they can be tracked again later, but those stopped are not going to be restarted). We should also remove the disposed anchors from our array list, we can do that in our sketch, or use the utility function <code>clearAnchors</code> provided in the ARTracker class. The result of our latest code should be the following:
</p>
<p><img class="body-image" src="../../imgs/tutorials/ar_intro/ar_anchors.png" alt="Anchor drawing"></p>
<h3>Hitting trackables</h3>
<p>
A typical interaction in AR is to move around an object along a trackable plane, for example to find its best placement interactively. The AR library in Processing offers functionality that makes implementing such interaction very easy. The ARTracker class has a <code>get()</code> function that takes an (x, y) touch position, and returns the trackable hit by that touch point. We can then create a new anchor for that hit trackable, and the anchor will be placed exactly at the position on the trackable that's at the intersection with the touch pointer. As the pointer moves around, we need to keep disposing the anchor at the previous position and creating another one at the new position. This is done with the following code (to be added to last version of our sketch):
<p>
<pre><code>
<span style="color: #33997E;">import</span> processing.ar.*;
<span style="color: #E2661A;">ARTracker</span> tracker;
<span style="color: #E2661A;">ArrayList</span><<span style="color: #E2661A;">ARAnchor</span>> trackAnchors = <span style="color: #33997E;">new</span> <span style="color: #E2661A;">ArrayList</span><<span style="color: #E2661A;">ARAnchor</span>>();
<span style="color: #E2661A;">ARAnchor</span> touchAnchor;
<span style="color: #E2661A;">PShape</span> obj;
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>setup</b></span>() {
<span style="color: #006699;">fullScreen</span>(<span style="color: #718A62;">AR</span>);
obj = <span style="color: #006699;">loadShape</span>(<span style="color: #7D4793;">"model.obj"</span>);
tracker = <span style="color: #33997E;">new</span> ARTracker(<span style="color: #33997E;">this</span>);
tracker.<span style="color: #006699;">start</span>();
<span style="color: #006699;">noStroke</span>();
}
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>draw</b></span>() {
<span style="color: #006699;">lights</span>();
drawObject(touchAnchor);
drawAnchors();
drawTrackables();
}
<span style="color: #33997E;">void</span> <span style="color: #006699;"><b>mousePressed</b></span>() {
<span style="color: #669900;">if</span> (touchAnchor != <span style="color: #33997E;">null</span>) touchAnchor.dispose();
<span style="color: #E2661A;">ARTrackable</span> hit = tracker.<span style="color: #006699;">get</span>(<span style="color: #D94A7A;">mouseX</span>, <span style="color: #D94A7A;">mouseY</span>);
<span style="color: #669900;">if</span> (hit != <span style="color: #33997E;">null</span>) touchAnchor = <span style="color: #33997E;">new</span> ARAnchor(hit);
<span style="color: #669900;">else</span> touchAnchor = <span style="color: #33997E;">null</span>;
}
...
<span style="color: #33997E;">void</span> drawObject(<span style="color: #E2661A;">ARAnchor</span> anchor) {
<span style="color: #669900;">if</span> (anchor != <span style="color: #33997E;">null</span>) {
anchor.attach();
<span style="color: #006699;">shape</span>(obj);
anchor.detach();
}
}
</code></pre>
<p>
Here we are using an OBJ model loaded from the sketch's data folder to draw at the placement of the hit anchor. If all is correct in the code, the output would look similar to the following:
</p>
<p><img class="body-image" src="../../imgs/tutorials/ar_intro/ar_objects.png" alt="Detecting trackable hitting"></p>
<p>The complete sketch code is available <a href="https://github.com/codeanticode/processing-android-tutorials/tree/master/ar/ex1_ar_intro" target="_black">here</a>.</p>
</section>
</div>
<footer class="footinfo">
<small>© The Processing Foundation. Processing for Android is a <a href="../../team.html">collaborative project</a>.</small>
</footer>
</body>
</html>