Passing the limits of human intuition

Introduction

In 2017, as interest in artificial intelligence was beginning to build momentum, we founded New Cortex with a vision to solve complex technical problems through human-AI collaboration. Our flagship product, Factor, was designed to address the overwhelming complexity data scientists and engineers faced when optimizing distributed computing systems.

From the beginning, we were fortunate to have Peter Norvig, Google's Director of Research and co-author of the definitive AI textbook "Artificial Intelligence: A Modern Approach," on our board of advisors. His early guidance and introductions at Google helped us validate our approach with some of the world's most sophisticated technical teams.

Factor emerged at a time when most AI applications were focused on replacing human capabilities. We took a different approach, creating systems that augmented human expertise and helped technical professionals navigate complexity beyond human cognitive limits.

Definition & Discovery

The Problem: Technical systems managed by data scientists and engineers had reached a critical complexity threshold where human optimization alone was no longer effective. Engineers relied on intuition, trial-and-error, or community "folklore" to make configuration decisions—approaches that didn't scale with enterprise needs.

Through research with technical teams at leading companies (Nike, Uber, Facebook, Google), we identified how data professionals struggled specifically with Apache Spark optimization. The complexity barrier was preventing implementation of potentially valuable projects and driving excessive resource costs.

Key Insight: The complexity problem could be solved through a collaborative human-AI approach that leveraged machine intelligence for exploration while maintaining human agency over strategic decisions.

Exploration & Strategy

We developed a strategic framework for "collaborative AI" centered on three principles:

• Understanding: Making system complexity comprehensible through tactical simplification

• Agency: Identifying meaningful decision points for human intervention

• Trust: Building confidence through transparent communication of statistical probabilities

This framework informed three primary user workflows:

1. AI-recommended solution implementation

2. AI-guided exploration for novel problem spaces

3. User-driven modification based on AI insights

I led cross-functional alignment between our AI researchers, engineers, and trial customers to establish a product vision that balanced technical capabilities with real-world user needs.

Process & Production

I designed a card-based interface that translated complex statistical models into accessible visualizations. Each recommendation card featured:

• Histograms showing probability distributions of performance outcomes

• Plain-language summaries of technical recommendations

• Comparative metrics against baseline performance

The design system I developed included:

• Visual patterns for communicating uncertainty and confidence levels

• Information architecture that exposed increasing levels of technical detail on demand

• Interactive elements that allowed users to explore parameter relationships

  
  

Technical Implementation

Beyond the interface, I collaborated deeply with our AI team on technical implementation:

• Refined visualization approaches for communicating probability distributions, moving from standard histograms to more intuitive cumulative distribution function (CDF) visualizations

• Developed interaction patterns for exploring decision trees generated by Monte Carlo Tree Search algorithms

• Designed and implemented a comprehensive design system that standardized both visual elements and data representation

• Created novel interaction models for navigating complex parameter spaces, allowing users to explore relationships between configuration changes and performance outcomes

The system architecture connected these frontend components to our backend AI engines that used neural networks to evaluate program modifications and predict performance outcomes.

I personally implemented many of the Vue.js frontend components to ensure fidelity between design intent and execution, creating a cohesive experience that technical users found both powerful and accessible.