Hololens - Manipulate Gestures and Resizing Windows

I had a bit of trouble finding documentation for how to make a re-sizable window for Hololens and thus want to document my findings here.  As this is all fairly new please share your findings in the comments.

Here is the goal... in 2D form:

On the device you can click and drag (manipulate) any of the blue cubes to scale the window.  And you can click and drag the center circle to re-position the window in 3D space.

If you have a device you can find an example of this by opening the Photo app and clicking resize in the top right corner.

Prep:
First and foremost make sure you are up-to-date with the latest Unity build which currently is 5.4b16: http://unity3d.com/pages/windows/hololens

Otherwise you may crash while trying to listen to the manipulate gesture.

Gestures:
If you aren't familiar with Gestures you can learn more about them in detail here: https://developer.microsoft.com/en-us/windows/holographic/gestures

Manipulate is going to be our focus since we are going to move/re-size based on the user's hand movements.

Tutorial:
Microsoft's Holograms 211 tutorial will get you most of the way there. Namely the GestureManager code though their example uses both Navigation and Manipulation which adds a bit of complexity as you can't use both at the same time.  Since we only need Manipulate we can nix the Transition code and just add the GesturesSettings.ManipulationTranslate to our recognizer we are using for tap.  I'll share my code below.

Integrating with HoloToolkit:
Microsoft's HoloToolkit code base's GestureManager is different than the one in the tutorial. I decided to try to pick and choose bits from the 211 tutorial and drop it into the Toolkit code but ended up with a half baked solution.  You could manipulate items while gazing at them but as soon as the item lost your gaze the manipulation would stop. The trick is to make sure you Override the focusedObject. Otherwise when the focus object changes the gesture is cancelled.


GestureManager Class:
This class mixes the HoloToolkit code with the Holograms 211 Tutorial and makes adjustments to handle the change in focusedObject.

	// Copyright (c) Microsoft Corporation. All rights reserved.
	// Licensed under the MIT License. See LICENSE in the project root for license information.
	using UnityEngine;
	using System.Collections;
	using UnityEngine.VR.WSA.Input;
	using System;
	namespace HoloToolkit.Unity
	{
	/// <summary>
	/// GestureManager creates a gesture recognizer and signs up for a tap gesture.
	/// When a tap gesture is detected, GestureManager uses GazeManager to find the game object.
	/// GestureManager then sends a message to that game object.
	/// </summary>
	[RequireComponent(typeof(GazeManager))]
	public class GestureManager : Singleton<GestureManager>
	{
	/// <summary>
	/// To select even when a hologram is not being gazed at,
	/// set the override focused object.
	/// If its null, then the gazed at object will be selected.
	/// </summary>
	public GameObject OverrideFocusedObject
	{
	get; set;
	}
	/// <summary>
	/// Gets the currently focused object, or null if none.
	/// </summary>
	public GameObject FocusedObject
	{
	get { return focusedObject; }
	}
	private HandsManager _handsManager;
	private HandsManager HandsManagerInstance
	{
	get
	{
	if (_handsManager == null) _handsManager = HandsManager.Instance;
	return _handsManager;
	}
	}
	[NonSerialized]
	private GestureRecognizer m_GestureRecognizer;
	// Manipulation gesture recognizer.
	// public GestureRecognizer ManipulationRecognizer { get; private set; }
	// Currently active gesture recognizer.
	public GestureRecognizer ActiveRecognizer { get; private set; }
	public bool IsManipulating { get; private set; }
	public Vector3 ManipulationPosition { get; private set; }
	private GestureRecognizer gestureRecognizer;
	private GameObject focusedObject;
	void Start()
	{
	// Create a new GestureRecognizer. Sign up for tapped events.
	gestureRecognizer = new GestureRecognizer();
	// gestureRecognizer.SetRecognizableGestures(GestureSettings.Tap);
	gestureRecognizer.TappedEvent += GestureRecognizer_TappedEvent;
	gestureRecognizer.SetRecognizableGestures(
	GestureSettings.Tap | GestureSettings.ManipulationTranslate );
	// Instantiate the ManipulationRecognizer.
	/// ManipulationRecognizer = new GestureRecognizer();
	// Add the ManipulationTranslate GestureSetting to the ManipulationRecognizer's RecognizableGestures.
	/// ManipulationRecognizer.SetRecognizableGestures( GestureSettings.ManipulationTranslate);
	// Register for the Manipulation events on the ManipulationRecognizer.
	gestureRecognizer.ManipulationStartedEvent += ManipulationRecognizer_ManipulationStartedEvent;
	gestureRecognizer.ManipulationUpdatedEvent += ManipulationRecognizer_ManipulationUpdatedEvent;
	gestureRecognizer.ManipulationCompletedEvent += ManipulationRecognizer_ManipulationCompletedEvent;
	gestureRecognizer.ManipulationCanceledEvent += ManipulationRecognizer_ManipulationCanceledEvent;
	ResetGestureRecognizers();
	}
	/// <summary>
	/// Revert back to the default GestureRecognizer.
	/// </summary>
	public void ResetGestureRecognizers()
	{
	// Default to the navigation gestures.
	Transition(gestureRecognizer);
	}
	/// <summary>
	/// Transition to a new GestureRecognizer.
	/// </summary>
	/// <param name="newRecognizer">The GestureRecognizer to transition to.</param>
	public void Transition(GestureRecognizer newRecognizer)
	{
	return;// ignore for now
	if (newRecognizer == null)
	{
	return;
	}
	if (ActiveRecognizer != null)
	{
	if (ActiveRecognizer == newRecognizer)
	{
	return;
	}
	ActiveRecognizer.CancelGestures();
	ActiveRecognizer.StopCapturingGestures();
	}
	newRecognizer.StartCapturingGestures();
	ActiveRecognizer = newRecognizer;
	}
	private void GestureRecognizer_TappedEvent(InteractionSourceKind source, int tapCount, Ray headRay)
	{
	if (focusedObject != null)
	{
	focusedObject.SendMessage("OnSelect");
	}
	}
	void LateUpdate()
	{
	GameObject oldFocusedObject = focusedObject;
	if (GazeManager.Instance.Hit &&
	OverrideFocusedObject == null &&
	GazeManager.Instance.HitInfo.collider != null)
	{
	// If gaze hits a hologram, set the focused object to that game object.
	// Also if the caller has not decided to override the focused object.
	focusedObject = GazeManager.Instance.FocusedObject;
	}
	else
	{
	// If our gaze doesn't hit a hologram, set the focused object to null or override focused object.
	focusedObject = OverrideFocusedObject;
	}
	if (focusedObject != oldFocusedObject)
	{
	// If the currently focused object doesn't match the old focused object, cancel the current gesture.
	// Start looking for new gestures. This is to prevent applying gestures from one hologram to another.
	gestureRecognizer.CancelGestures();
	gestureRecognizer.StartCapturingGestures();
	}
	}
	private void ManipulationRecognizer_ManipulationStartedEvent(InteractionSourceKind source, Vector3 position, Ray ray)
	{
	if (HandsManagerInstance.FocusedGameObject != null)
	{
	IsManipulating = true;
	ManipulationPosition = position;
	focusedObject = OverrideFocusedObject = HandsManagerInstance.FocusedGameObject;
	OverrideFocusedObject.SendMessageUpwards("PerformManipulationStart", position);
	}
	}
	private void ManipulationRecognizer_ManipulationUpdatedEvent(InteractionSourceKind source, Vector3 position, Ray ray)
	{
	if (OverrideFocusedObject != null)
	{
	IsManipulating = true;
	ManipulationPosition = position;
	//HandsManager.Instance.FocusedGameObject.SendMessageUpwards("PerformManipulationUpdate", position);
	}
	}
	private void ManipulationRecognizer_ManipulationCompletedEvent(InteractionSourceKind source, Vector3 position, Ray ray)
	{
	Debug.Log("Manipulation Completed");
	OverrideFocusedObject = null;
	IsManipulating = false;
	}
	private void ManipulationRecognizer_ManipulationCanceledEvent(InteractionSourceKind source, Vector3 position, Ray ray)
	{
	Debug.Log("Manipulation Cancelled");
	OverrideFocusedObject = null;
	IsManipulating = false;
	}
	void OnDestroy()
	{
	if (gestureRecognizer != null)
	{
	gestureRecognizer.StopCapturingGestures();
	gestureRecognizer.TappedEvent -= GestureRecognizer_TappedEvent;
	/* }
	if (ManipulationRecognizer != null)
	{ */
	// Unregister the Manipulation events on the ManipulationRecognizer.
	gestureRecognizer.ManipulationStartedEvent -= ManipulationRecognizer_ManipulationStartedEvent;
	gestureRecognizer.ManipulationUpdatedEvent -= ManipulationRecognizer_ManipulationUpdatedEvent;
	gestureRecognizer.ManipulationCompletedEvent -= ManipulationRecognizer_ManipulationCompletedEvent;
	gestureRecognizer.ManipulationCanceledEvent -= ManipulationRecognizer_ManipulationCanceledEvent;
	}
	}
	}
	}

view raw GestureManager hosted with ❤ by GitHub

Handling:
To handle these events you will need to wire something to catch the starts and then update while manipulating.  I have my corner buttons catching the start manipulate calls and then triggering StartScaleManipulation while my center button triggers StartManipulation. Code below:

	public enum ManipulationType
	{
	Position = 0,
	Scale = 1
	}
	private bool _manipulating;
	private Vector3 _lastManipulationPosition;
	private ManipulationType _manipulationType;
	public void StartLocationManipulation(Vector3 pos)
	{
	_manipulationType = ManipulationType.Position;
	StartManipulation(pos);
	}
	void Update()
	{
	if (_manipulating)
	{
	UpdateManipulation();
	_manipulating = GestureManagerInstance.IsManipulating;
	}
	}
	public void StartScaleManipulation(Vector3 pos)
	{
	_manipulationType = ManipulationType.Scale;
	StartManipulation(pos);
	}
	private void StartManipulation(Vector3 pos)
	{
	_lastManipulationPosition = pos;
	_manipulating = true;
	}
	public void UpdateManipulation()
	{
	Vector3 pos = GestureManagerInstance.ManipulationPosition;
	Vector3 diff = _lastManipulationPosition - pos;
	switch(_manipulationType)
	{
	case (ManipulationType.Scale):
	Vector3 currentScale = transform.localScale;
	currentScale += diff;
	transform.localScale = currentScale;
	break;
	case (ManipulationType.Position):
	Vector3 currentPos = transform.position;
	currentPos -= diff;
	transform.position = currentPos;
	break;
	}
	_lastManipulationPosition = pos;
	_manipulating = GestureManagerInstance.IsManipulating;
	}

view raw HandlerCode hosted with ❤ by GitHub

Custom Hand Manager:
I am not sure if we added the FocusedGameObject or if that was originally in the HoloToolkit and since removed. In either case, this is the HandManager we are using. Feel free to rename to HandsManager or customize how you see fit.

	// Copyright (c) Microsoft Corporation. All rights reserved.
	// Licensed under the MIT License. See LICENSE in the project root for license information.
	using System.Collections.Generic;
	using UnityEngine;
	using UnityEngine.VR.WSA.Input;
	namespace HoloToolkit.Unity
	{
	/// <summary>
	/// HandsDetected determines if the hand is currently detected or not.
	/// </summary>
	public class TQHandsManager : Singleton<TQHandsManager>
	{
	/// <summary>
	/// HandDetected tracks the hand detected state.
	/// Returns true if the list of tracked hands is not empty.
	/// </summary>
	public bool HandDetected
	{
	get { return trackedHands.Count > 0; }
	}
	public GameObject FocusedGameObject { get; private set; }
	private List<uint> trackedHands = new List<uint>();
	void Awake()
	{
	InteractionManager.SourceDetected += InteractionManager_SourceDetected;
	InteractionManager.SourceLost += InteractionManager_SourceLost;
	InteractionManager.SourcePressed += InteractionManager_SourcePressed;
	InteractionManager.SourceReleased += InteractionManager_SourceReleased;
	FocusedGameObject = null;
	}
	private void InteractionManager_SourceDetected(InteractionSourceState state)
	{
	// Check to see that the source is a hand.
	if (state.source.kind != InteractionSourceKind.Hand)
	{
	return;
	}
	trackedHands.Add(state.source.id);
	}
	private void InteractionManager_SourceLost(InteractionSourceState state)
	{
	// Check to see that the source is a hand.
	if (state.source.kind != InteractionSourceKind.Hand)
	{
	return;
	}
	if (trackedHands.Contains(state.source.id))
	{
	trackedHands.Remove(state.source.id);
	}
	ResetFocusedGameObject();
	}
	private void InteractionManager_SourcePressed(InteractionSourceState hand)
	{
	if (GazeManager.Instance.FocusedObject != null)
	{
	FocusedGameObject = GazeManager.Instance.FocusedObject;
	}
	}
	private void InteractionManager_SourceReleased(InteractionSourceState hand)
	{
	ResetFocusedGameObject();
	}
	private void ResetFocusedGameObject()
	{
	FocusedGameObject = null;
	}
	void OnDestroy()
	{
	InteractionManager.SourceDetected -= InteractionManager_SourceDetected;
	InteractionManager.SourceLost -= InteractionManager_SourceLost;
	InteractionManager.SourceReleased -= InteractionManager_SourceReleased;
	InteractionManager.SourcePressed -= InteractionManager_SourcePressed;
	}
	}
	}

view raw TQHandsManager hosted with ❤ by GitHub