Connecting to the Real World with VR

Build next generation software applications and to manage information complexity in real world and virtual environments.

Connect to the Real World

nSpace’s paradigm of ‘connecting things’ will be familiar to Unreal developers who use ‘blueprints’, except changes occur in real time and don’t require compiling or restarting of the ‘application’ under development. nSpace’s scope is also much larger, being a bridge to the outside world and the Internet of Things. The ability to create a virtual world based on external data or simply controlling the outside world through the Unreal Engine using virtual reality will expand the possible uses of the engine for new applications such as real-time collaboration or command and control.

Ready for Virtual Reality

nSpace® VR™ is ready Virtual Reality with support for all popular HMDs, Kinect, Leap Motion and other trackers available in the market.  Imagine editing the behavior of your application in-world and seeing your changes in real-time without ever leaving the visual editor environment.  You can actually affect the application’s state and program execution by directly interacting with it.

Unleash Your Potential

nSpace Connection-Oriented Programming Framework and VR

Three Primary Requirements for VR Development Tools

1. Performance

 Effective immersive environments should be able to take advantage of all available resources on a system, such as processors and special graphics hardware. The development system itself should have as little overhead as possible, letting the hardware run at its maximum efficiency.  In nSpace, any feature that needs to be optimized can be handled in a behavior which is coded in a high-level language and can be executed as efficiently as native code.

If a computational bottleneck exists in the system you simply write a behavior and optimize for the one function.  Now in most applications, everything is about moving the data around so there are no computational bottlenecks, then you just end up waiting for disk/network/etc and nSpace currently runs faster than the I/O.

It is the other 99% of the application that nSpace graphs are effectively used for and that 99% is also where the requirements change, preferences change, etc.  nSpace allows you to change the things that change most often – easily and without writing code.

2. Flexibility 

The development environment should be able to adapt to many hardware and software configurations. If the environment cannot adapt to new configurations, applications will be limited in the scope of their usefulness. A developer should not be required to rewrite an application for every new configuration. In addition, the design of the system itself should not limit the sort of applications that can be developed with it. Developers should never hit a wall where the development environment restricts them from creating an application that works the way they have envisioned it.

The nSpace framework is a different type of implementation.  It provides a ‘step up’ in capability and abstraction without imposing any design decisions on the programmer.  It is still a ‘clean slate’ when it comes to designing the next application without restricting any potential that a standard language based library might impose.  This flexibility has allowed the nSpace engine and concepts to be applied to a number of diverse applications and what makes it ideal for virtual reality development.

3. Ease of Use

The development system should be easy to configure and to learn. The Application Programming Interfaces (APIs) and/or languages used to create applications should be cleanly designed and should hide as much of the system’s underlying complexity as possible.  nSpace-based applications and tools allow users to customize their experience through direct manipulation of context-sensitive information.  The framework also provides access to a general Application Programmer Interface (API) which provides a simple unified method to allow an unprecedented level of implicit information sharing between independent programs.

An ideal development system would score perfect marks in each category and we are striving to check all of these boxes.  In reality, one of the challenges of building developer tools is creating a visual programming environment that replaces the text file entry and is easy to use.  Since the namespace structure is very rich, creating the correct visualizations will need to be an iterative process.  This is where we intend to have a continual dialog with our developers in order to continually improve the tools.

 

nSpace VR Features

Cross-Platform Development

What happens if an application has twenty potential customers, but ten of them use Linux workstations and the rest prefer Windows or Mac OS X systems? What about mobile VR systems that are based on iOS and Android? Well-designed tools should hide platform-specific details well enough that porting an application between different platforms requires little or no change in the application. Since all of the ‘knowledge’ of an nSpace application is described in the form of graphs, portability becomes fairly simple.  Only the underlying nSpace engine and behaviors need to be ported to a new language or operating system.  All of the graphs that have been tested and running in completed applications will immediately be available in the new environment, provided the behaviors can be made to perform the same function in that environment.

Support for VR Hardware

What kind of special VR hardware is the application designed to use? Is it simple enough to use a joystick or wand for user input, or will a glove make it easier to control?  Is the finished product going to be used with projection systems, or HMDs, or both? Many groups already have large investments in hardware, so it is important to make sure that hardware is supported by the development system. In addition to immediate needs, future plans should be considered – will a glove or some kind of haptic output device be available in the foreseeable future?  nSpace provides ‘drivers’ for new VR hardware that can be added easily using new specific behaviors and graphs to support the latest features of devices as they become available.

Hardware Abstraction

Support for required hardware is mandatory, but almost as vital is how well the tool abstracts away the details of the hardware interfaces. Do devices of the same type share the same interface, or are there specific APIs for each one? This comes into play when replacing hardware. For example: If an application has been using tracking system A, but a newer, better tracking system B becomes available, will the application have to be modified to take advantage of it?

Reutilizing much of the same behaviors and sub-graphs to implement the new functionality insures that new VR hardware can be supported quickly by allowing application developers to directly access the new functionality without requiring any re-coding. A less generic interface allows the developer to be able to take advantage of a device’s unusual features, while making the application more portable and easier to upgrade, maintain, and understand. Using nSpace, major changes such as replacing a joystick with a glove, might require rethinking the user interface design, smaller changes, like switching one tracking system for another or changing between models of HMDs, does not require changes to the VR application itself.

Locally Distributed Applications

Locally distributed applications attempt to increase performance by dividing the workload between several computers on the same network. For example, two workstations might be used to generate the two images used by an HMD. nSpace relieves the burden on the developer to create a working distributed application. Since nSpace flows are inherently asynchronous the developer no longer has to worry about creating multiple threads of execution and synchronizing data between those threads to prevent race conditions, etc.  Threads are created ‘on demand’ implied by the use of certain behavior functionality such as timers or socket I/O availability and detection. This asynchronous capability allows parallel execution which has several advantages in addition to increasing the application’s frame rate, such as increasing the number of input devices or display channels available to an application, or allowing additional computers to be used for simulations and other computationally intensive tasks.

Distributed Environments

With application distribution, we think about connecting multiple machines at a single location. With distributed environments, we expand that to the idea of connecting machines – and users – at remote sites across a network. Development systems with this ability open up the possibility of bringing people together to collaborate in a virtual world. One example of such a system is the nSpace portal technology which is designed to allow multiple users to share a common namespace and to meet together to explore and review product designs. nSpace offers capabilities for this sort of networking, including control of interactions with multiple users and dealing with variable network latency.

Rapid Prototyping

To allow developers to quickly prototype an application, nSpace® Studio FX will provide a fully integrated visual programming environment for adding, modifying and debugging graphs that make up an nSpace application.  A graphical layout editor is included for quickly creating and assigning functionality of UI/UX designs directly on the target display over any support communications interface so editing it is entirely WYSIWYG and real-time.  While support for this sort of low-hardware setup is fairly universal, the cumbersomeness of the input simulation and the closeness with which it models the actual hardware such as gloves or body-tracking suits, will be provided by graphs and support the full range of inputs of the actual devices. nSpace® Studio FX will provide an environment that simulates the functionality of the actual VR hardware. The layout editor provides 2D rendering support with platform specific widgets and 3D rendering of custom models with built-in visual effects using popular toolkits like OpenGL to provide drawing the display in a window on a monitor and using the keyboard and mouse to simulate head tracking and other input devices.

nSpace® Studio FX provides other features that are important for rapid development, including an enhanced level of support for object behaviors and the existence of easy-to-use tools to control the virtual world or external real-world systems without ever leaving the virtual environment.

Extensibility

The nSpace VR graph library is easily extendable to new devices, new platforms, and new VR interfaces. The field of VR is constantly changing so nSpace graph libraries adapt easily to changes in order to keep from becoming obsolete. When the library is extended it should not require any changes to current applications.

Run-time Flexibility

Most VR applications often take considerable time to start up: processes must be created and synchronized, hardware initialized, databases loaded, and so on. Restarting an application every time a small change is required or a device needs to be restarted can consume a great deal of time. It is often desirable to be able to change hardware configurations on the fly, without restarting the application itself. Sometimes devices fail and need to be restarted or replaced, and sometimes the initial configuration is not what the user expected. Some aspects of configuration may require trial and error, such as finding the optimal distribution of processes and resources across several machines. Sometimes it is convenient to be able to switch rapidly between two devices, as perhaps between two tracking systems, to see which performs better.  Auto-detection of nSpace-enabled VR devices and a system level namespace browser will allow for quickly finding a device on the network and making real-time changes to it on-the-fly without having to upload a package or reboot.  nSpace® Studio FX will provide a visual programming approach that simplifies the task of configuring I/O, storage and integrated communications interfaces (i,e, USB, Bluetooth, Ethernet, etc.), graphic displays, and other capabilities greatly improving design productivity and reducing the time and cost for deploying and maintaining VR applications. The developer can move straight from viewing an application in a workstation simulator to viewing it on an HMD, without restarting it.

High-level and Low-level Interfaces

nSpace® Studio FX will provide a multi-platform development environment for designing, implementing, and debugging graphs as well as managing and making updates to a running VR application.  It provides a very high-level view, where applications can be created by linking custom behaviors that take on most of the responsibility of calculations, geometry, and interaction. nSpace also provides an interface just above the hardware level, by providing the ability to create behaviors that have access to well-known graphics APIs and programming languages to ensure the greatest performance and flexibility.  Using nSpace® Studio FX will enable faster development with a shallower learning curve but we realize a continuing need for these lower-level environments for implementing optimized functionality. Most of an nSpace solution is defined in graphs and not based in code but nSpace implements code-based algorithms or low-level interfaces within its various behaviors which are implemented in the language of your choice C, C++, Java, etc. These behaviors and the nSpace engine are the only code that needs to be ported to support a new computing platform.

Graphics Interfaces

nSpace is designed integrate with popular game engines like Unreal Engine 4 through its custom API for defining rendering of graphics elements. This popular gaming platform has several advantages in that there is a great deal of documentation and tutorials from diverse sources, and there are many programmers who are already familiar with them. Though specialized graphs, nSpace allows loading models generated in a variety of external modeling and Computer Aided Design (CAD) packages. If the application only needs to load the model of a car and allow a user to move around the environment to view the model, then nSpace behaviors and graphs will support loading models in the CAD format with which the car was produced and the hand-off the rendering to the game engine.

nSpace also has its own direct renderer which supports standard API’s such as OpenGL or Direct3D that can be used in other, possibly non-VR, projects. Furthermore, non-VR programs that use OpenGL (for example) can be easily rebuilt into VR applications – because the framework supports that API. nSpace support its own scene graph architectures, including rendering and collision detection. For scientific simulation, the developer may need to have access to the polygon level to render data for each frame which is provided through the integrated renderable elements stored directly in the namespace.

Interaction

The nSpace framework handles the details of user and program interaction through separate graphs that act as event handlers to deal with changes in the state of all input devices. The event could be anything from “User grabbed this object” to “These two objects just collided” to simply “New head tracker data is available.” The developer doesn’t have to write code to determine the current state of input devices but only has to link to the state value location in the namespace for processing interactions with the VR world.

Unreal Engine has its own scripting language, using ‘blueprints” for defining object interaction, and has a greater range of interaction capabilities. Blueprints have their own collision detection routines, and may even have built-in support for object physics (so that dropped items fall to or even bounce on the virtual floor, and so forth). Other environments that give developers the power to create their own graphics in OpenGL or another API will often give them much of the responsibility for handling user and object interactions. nSpace makes handing these interactions easier to implement.

APIs and Languages

nSpace is not a compiled ‘language’ but it still has the level of generality that a ‘meta language’ provides while implementing a ‘formal definition’ with which to describe computing solutions.

Because there is not one standard for platform, operating system and communication protocol between devices there is a current need for technology that can successfully “glue” together the multitude of APIs and protocols found in the market. Graphs behave like ‘glue logic’ which allows the data and logic of legacy systems to be migrated to new platforms and operating systems in a piecewise manner.  There is also a COM server and a namespace service available for programmers with a single general API that allows external programs to inject information and share state with running graphs in a local or hosted namespace which itself can be used as a service for existing code-based solutions.  If you are writing a flight simulator and you have thousands of lines of code in C++ that control the aircraft simulation, nSpace will allow you to use your current code. The new interface code is implemented in a behavior and is ‘handed’ messages/values from the running graph in the namespace.  The behavior takes the appropriate action and ’emits’ the proper output back into the graph.  This makes the code segments very concise and simple.

nSpace promotes more reuse of components since the task is no longer finding a ‘library’ that is compatible to the current language and operating system.  With nSpace, the developer can now just find the ‘behavior’ that is required and wire it into the higher level application.  Developers spend less time learning new code libraries and class hierarchies and more time focusing on implementing the desired functionality, thereby reducing development costs.  Since all of the ‘knowledge’ of an application is described in the form of graphs, portability becomes fairly simple. All of the graphs that have been tested and running in completed applications will immediately be available in the new environment, provided the behaviors can be made to perform the same function in the that environment. Because nSpace supports existing high-level languages such as C++, or Java, the learning curve is not as steep when implementing new behaviors.

Minimal Limitations

nSpace is a general-purpose programming environment and is different from other programming languages and development environments. The nSpace architecture provides an alternative method of writing distributed, multi-threaded, asynchronous and inherently parallel applications without the need to implement these features in the source code a priori.

Although the architecture enforces a system view in which the main emphasis is the proper definition of the problem to solved, you will no doubt see very familiar ideas played out in the details of implementation of program logic and constructs. Ultimately this generality is what makes nSpace such a diverse environment, although having no ‘built-in’ functionality means it is not a ‘build your application in two steps’ type of environment.  However, by creating new and reusing existing graph definitions (i.e. ‘subgraphs’) one can have a huge ‘library’ of pre-generated solutions from which to start.

Performance Monitoring

Utilizing the nSpace temporal database the developer automatically has access to a historical record of changes made to the namespace, both the state and structure of a running system.  This temporal storage is also available to the graphs for storage of any data in the system and is a straightforward way to enable ‘persistence’ in an application again without having to write additional logic either in code or in graphs.  Data is stored relative to its location within the namespace which also allows easier ‘discovery’ of available data via a file system like browser.  The result is ‘previous’ snap-shots in time can be brought up to ‘now’ in order to restore previous solutions and/or data. Temporal information is necessary in order to optimize software and hardware configurations, and to find bottlenecks. Given the collected performance data, the user may be able to reconfigure the system or change the application to maximize performance.  The framework’s integrated versioning of connections and context makes it easier to integrate real-time changes into the workflow and enables fast restoration of previous states of applications and data to also support rapid development iteration cycles helping the developer to optimize the design and also detect and correct programming and specification errors early.