Figure 1: Results from [1]. The upper row shows the rendered images, and the lower row shows the reconstructed meshes when the quality of rendered images is acceptable. The rectangles over the images represent the viewable areas of the user.
Receiver-Driven View-Dependent Streaming of Multiple Progressive Meshes
Muhammad S Karim
Project Report for CS585: Computer Networks
University of New Mexico
mskarim@cs.unm.edu
ABSTRACT
Progressive mesh is very suitable and has been increasingly used in 3D networked applications, such as online games, virtual worlds, and digital museums. This technique enables users to view 3D meshes over the network with increasing level of details, by sending a coarse mesh initially followed by a series of refinements. Entertainment applications are increasingly requiring more gigabytes of data for geometry assets to be stored in disk in the receiver's (client's) machine, making streaming media more viable in a lot of cases where the scene description or the game world assets changes frequently. And geometry is still the most important asset in these online applications. So in this project I study a receiver-driven protocol for streaming progressive meshes and analyse the performance and impact especially on the receiver side on more realistic situations where several meshes are needed to be streamed simultaneously. To support several simultaneous requests for multiple mesh refinements I give an algorithm to prioritize the requests based on the visual contribution of the meshes.
Keywords
Progressive meshes, view-dependent streaming.
INTRODUCTION
In this project I studied a solution that is being researched by W. Cheng [1] which utilizes a receiver-driven protocol to stream progressive meshes. The receiver requests a coarse, base-level mesh with minimal geometry information initially, and they progressively requests refinements to increase the level of detail of the mesh. And this is also view-dependent meaning each time refinements are requested they are based on the viewing parameters, so only the parts of the mesh that are visible to the user is refined, and as more parts of the mesh comes to the view more refinements are requested. Their result of this technique is demonstrated in Figure 1.
Figure
1: Results from [1]. The upper row shows the rendered images, and the
lower row shows the reconstructed meshes when the quality of rendered
images is acceptable. The rectangles over the images represent the
viewable areas of the user.
Figure
2: An example of an online appliation with a sceen composed of
several objects (humans, houses, utility objects) where each object
requires a seperate geometry mesh (Image from Gamespot, Application
by Linden Labs)
It was shown in [1] and also a related work [2] that this receiver-driven approach is significantly more practical for scaling applicaitons for large number of users. [2] is and extension of [1] where the researchers applied peer-to-peer techniques to optimize and scale the sender's work to stream out progressive mesh requests from receivers. In favor of this receiver-driven approach [1] gives two strong arguments. Firstly, the usual sender-driven protocol is not scalable to many receivers, since determining visibility of vertices and sorting the visible vertex splits based on their visible contributionss are expensive operations. Moreover the sender needs to maintain the viewing state of each receiver. Secondly, the sender-driven approach cannot be scaled to support caching proxy and peer-to-peer architecture due to the huge computational requirements and the stateful design.
Though both [1] and [2] evaluates sender side performance and scalability, performance in the receiver side is also a significant issue to investigate. Experiments in [1] and [2] are tested in the client side using only one mesh. However, realistic environments such as in Figure 2 often have multiple, in several cases hundreds of meshes displayed in a scene at a time. The question that I investigated in this project is that how can this technique be implemented so that requesting refinements from multiple meshes simultaneously can be done efficiently.
To address this issue I propose an algorithm to process multiple requests for mesh refinements which uses a technique similar as [1] to determine visual importance of meshes in the receiver's screen. But in this case I evaluate visual importance of meshes instead of vertices and determine a way to compute the priority of each request to be send to the sender.
Figure
3: On the left is an object rendered using only its base mesh, and to
the right rendered as more refinements have been streamed in. On the
top the series of vertex splits are shown corresponding to the level
of refinement required for the rendered objects in the bottom
RELATED WORK
Progressive Mesh Streaming
The basis of the implementation in [1] is from Hoppe [3] that represents represents a vertex hierarchy among the vertices in a progressive mesh, which is basically a forest of binary trees. Here the root nodes are vertices in the base mesh, and leaf nodes are vertices in the original mesh. Cheng [1] uses a modified version of this algorithm and also uses [4] to determine vertex splits efficiently.
Receiver-Driven Mesh Streaming
The main challenge in designing a receiver-driven protocol is how a receiever can determine the visible vertices when it has not yet received a complete mesh. In their proposal, Cheng et. al. [1] exploit the progressive nature of meshes and assume that if a vertex is visible, the vertex from which it is split is also visible. Therefore, they split visible vertices to refine the mesh demonstrated in Figure 3. Their experiments showed that the visible vertices can be approximated with negligible errors using this assumption.
Once a receiver finds a visible vertex, it requests the vertex split to the sender to split this vertex. A unique identification (ID) id assigned to each vertex split to allow the receiver to explicitly request a vertex split. In a progressive mesh, vertex splits are hierarchically organized as a forest of binary trees which is based on Hoppe [3]. Here a parent vertex is a coarser representation of its two children and the root of the trees are the vertices in the base mesh, this is shown in Figure 4.
Cheng et. Al [1] use the method proposed by Kim and Lee [5] and assign vertex a bit string consisting of two parts – a tree ID and a node ID. Here the tree ID is a the sequence number of the root of this tree in the base mesh, and the node ID represents the path from the root to this vertex in the binary tree.
Figure
4: Vertex hierarchy showing the vertices for the base mesh and the
original mesh in the leaves
Figure
5: Rendered image of a mesh on receiver's screen where the shaded
region is the screen area of vertex
and
vertex
And the final important detail in [1] is the determination of visual importance based on which vertex splits are requested. They estimate the importance of the vertex splits to request for based on the received mesh, using the screen-space area of all the neighbor faces of a vertex as the metrix of its visual importance. As shown in Figure 5, if the screen area of a vertex is larger, it is likely the quality can be improved more by splitting this vertex. This technique can be efficiently computed in GPU, by simply counting the number of pixels in the framebuffer. Once the screen-space areas are computed, the receiver sends the requests following the descending order of the screen-space area. My proposed algorithm uses a similar GPU technique as detailed in the next section.
EXTENDED IMPLEMENATION WITH MULTIPLE MESHES
Multiple Mesh Transmission
When the reciever has multiple meshes in the screen-space view area, requesting refinements for these meshes in arbitrary order is not ideal. This is because not all meshes are positioned in important regions, and an arbitrary request ordering could update meshes completely far away or situtated in the corner of the screen before the more important one's which are closer and encompass a sigficant portion of the screen. So we need to determine a priority for each mesh based on their visual importance and use this priority to determine the request order. We also need a data structure that will queue the outstanding requests from different meshes and proecess the requests based on their priority. This visual importance for meshes is a little difference than the visual importance of vertex splits, which is detailed in the following sub sections.
Data Structure for Requesting Refinements
We need a Mesh-refinement Request Manager (MRM) that uses a priority queue data structure (Figure 6) to store incoming requests from the available meshes. Whenever a request is added it comes with a priority which is a real numbered value representing the visual importance of the mesh. Then, the MRM processes the requests one by one processing them in descending order by their priority. Number of requests per frame in interactive applications has to be limited to a small number so that this does not create a slowdown in rendering. And this process has to run in a separate thread than the rendering thread so it does not interfer with the frame rate.
Figure 6: The priority queue data-structure used by the the Mesh-refinement Request Manger (MRM)
Determining Order of Mesh Refinement Requests
The refinement order request for each mesh is based on their visual importance. To determine the visual importance I consider the screen-space contribution of a mesh as the key parameter. And also in most interactive visual applications the user is more attentive near the focal point or the center of the screen, and usually moves the view/camera so that the important objects are closer to the center of the screen. Another consideration is the distance of the objects from the users position in the world space. So the objects closer to the user are more important to the objects that are very far away. Taking these into considereatin I devised the following algorithm to determine visual contribution of a mesh.
The screen is divided in rectangles of different sizes that are proportional to the screen, but encompass a specific percentage of the total screen-space as shown in Figure 7. From the rendering buffer of the GPU we can compute the percentage contribution of each mesh rasterized to a particular rectangle by simply conting pixels from the render buffer. The visual contribution of a specific slice can determined by:
Figure 7: Screen is divided in several rectangular slices, and visual contribution of each object is determined by the sum of there weighted contribution in each slices.
Now to compute the contribution of Mesh i in a Rectangle of pixels out of several other meshes we need to assing an mesh ID to each pixel to determine which mesh a particular pixel belongs to. Now to do this we can do a separate rendering pass to render all the visible meshes from viewer's point of view into a render-buffer and color code IDs, that is to render the IDs as colors, so when we read from the buffer we read off a color of a pixel and treat it as IDs. Since the IDs need to be unsigned integers usually we are limitted to only 255 colors per channel. Since we have upto 4 channels (RGBA: 8x4 = 32-bits) per pixel we can represent binary representation of integers for values ranging [0, 2^32-1], which is more than enough for any possible scene.
So after calculating the visual contribution of mesh i in slice i we can compute the total visual contribution of mesh is for the entire screen:
Here
the visual contribution coefficient assigns a real number value for
each slice of the screen, such that the value is highest for the
center slice and decreases gradually in the slices that are further
from the center. To model the distribution of these coefficients in
relation to the number of slices a gaussian distribution of values
for
would
be ideal and shall be much better than a linear distribution. Where
the objects that may not be at center but close to it will still have
a significant contribution, but objects near the edge of the screen
will be much less significant.
Finally,
this
shall
be used as a priority for the i'th mesh when it will be inserted in
the MRM Queue. This process needs to be calculated in each frame if
the view is changed or any object's position is updated. But this is
not impractical with moderate GPUs that are a requirement in most
visual intractive applications.
CONCLUSION
In the area of research related to mesh streaming I studied on two approaches: [1] by Cheng et al. And [6] by He et al. While [1] is focused on streaming the vertices for mesh as raw data, [6] trasmits mesh data into a geometry image format called Spectral Geometry Image (SGI) which is based on Gu et al. [7]. However, I chose to pursue the former because even though [6] is appropriate in some cases, [1] is more practical for interactive applications since it implements progregressive streaming very well in addition to reducing the bandwidth requirements by dividing the mesh transmissions into receiver-driven requests. I believe the algorithm that I discussed here will be useful complement to the related research on server side scaling optimizations while implementing the solution on a real online application. But due to time constraints my project is only limited to analysing solutions to progressive streaming of geometry and devising an algorithm that can be practically implemented in GPU.
REFERENCES
[1] W. Cheng and W. T. Ooi. Receiver-driven view-dependent streaming of progressive mesh. In Proceedings of NOSSDAV'08, Brauschweig, Germany, May 2008.
[2] W. Cheng, D. Liu, W. T. Ooi. Peer-assisted view-dependent progressive mesh streaming. In Proceedings of ACM Multimedia'09, Beiging, China October 19-24, 2009.
[3] H. Hoppe. Progressive meshes, In Proceedings of SIGGRAPH'96, pages 99-108, New Orleans, USA August 1996.
[4] J. Kim, S. Choe, and S. Lee. Multiresolution random accessible mesh compression. Computer Graphics Forum, 25(3):323-331, September 2006.
[5] J. Kim and S. Lee. Truly selective refinement of progressive meshes. In proceedings of Graphics Interface 2001, pages 101-110, June 2001.
[6] Y. He, B. Chew, D. Wang, S. Hoi and L. Chau. Streaming 3D meshes using spectral geometry images. In Proceedings of ACM Multimedia'09, Beiging, China October 19-24, 2009.
[7] X. Gu, S. J. Gortler, and H. Hoppe. Geometry images. In Proceedings of SIGGRAPH'02, pages 355-361, 2002.